Three-dimensional digital human model training method and three-dimensional digital human generating method

Abstract

The embodiment of the invention discloses a three-dimensional digital human model training method and a three-dimensional digital human generation method, and the method comprises the steps: obtaining sample videos corresponding to different personalized parameters, and the personalized parameters comprise at least one of identity information and emotion information; corresponding sample data are obtained from sample videos corresponding to the individual parameters, the sample data comprise sample voice, emotion of a sample user and head parameters of the sample user, and the head parameters of the sample user are target head key points in a calibrated digital human established according to the sample videos; and calibrating a to-be-calibrated point on the initial digital person to obtain the initial digital person. According to the sample data corresponding to each personalized parameter, training a sound and picture generation model to obtain a personalized digital human generation model corresponding to each personalized parameter; and performing normalization processing on different personalized digital human generation models to obtain a universal digital human generation model. The stability of the three-dimensional digital human can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to image processing technology, and relate to a training method for a three-dimensional digital human model and a method for generating a three-dimensional digital human. Background Art

[0002] A three-dimensional digital human is a virtual image that can be displayed in a client interface. Voice input can be used to make the three-dimensional digital human perform corresponding actions, allowing the audience to visually perceive that the voice is emitted by the three-dimensional digital human.

[0003] In the related art, the rendering method used is unstable. For example, the head of the generated three-dimensional digital human is prone to shaking, deformation, and separation of the head and torso, resulting in poor stability of the three-dimensional digital human obtained by this method. Summary of the invention

[0004] The embodiment of the present application provides a method for training a three-dimensional digital human model and a method for generating a three-dimensional digital human, which can improve the stability of the three-dimensional digital human. The method for training a three-dimensional digital human model and a method for generating a three-dimensional digital human provided in the embodiment of the present application are implemented as follows:

[0005] In one aspect of the embodiments of the present application, a training method for a three-dimensional digital human model is provided in the embodiments of the present application, the method comprising:

[0006] Obtaining sample videos corresponding to different personalized parameters, where the personalized parameters include: at least one of identity information and emotion information;

[0007] Obtaining corresponding sample data from the sample videos corresponding to the individualized parameters, the sample data including sample voice, sample user's emotions, and sample user's head parameters, wherein the sample user's head parameters are obtained after calibration based on the positions of the target head key points of the sample user in the sample video;

[0008] According to the sample data corresponding to each personalized parameter, the audio and video generation model is trained to obtain the personalized digital human generation model corresponding to each personalized parameter;

[0009] Different personalized digital human generation models are normalized to obtain a universal digital human generation model.

[0010] In this scheme, the sample data in the sample video includes the head parameters of the sample user. Since the head parameters are obtained by calibrating the points to be calibrated on the initial digital human based on the head key points of the calibrated digital human established in the sample video, the personalized digital human generation model and the universal digital human generation model finally obtained can generate a digital human with a more stable head, which is closer to the head movement of the user in the sample video, thereby enhancing the stability of the digital human.

[0011] In one embodiment, corresponding sample data is obtained from sample videos corresponding to each personalized parameter, including: establishing a corresponding training digital human according to the sample video, and determining the head parameters of the sample user according to the head parameters of the training digital human; identifying the emotions of the sample user through the facial parameters of the training digital human; determining the sample speech corresponding to each personalized parameter, as well as the emotion of the sample user and the head parameters of the sample user corresponding to each speech frame of the sample speech.

[0012] In this solution, by establishing a method of training digital humans, the head parameters and emotions of sample users can be obtained more quickly and accurately, thereby obtaining sample data more quickly and accurately.

[0013] In one embodiment, a sample video includes multiple video frames, and a corresponding training digital human is established based on the sample video, including: establishing a calibrated digital human through multiple video frames in the sample video, the calibrated digital human including multiple frames; calibrating the initial digital human according to the target head key points of the calibrated digital human to obtain a training digital human.

[0014] In this scheme, a more accurate calibrated digital human can be obtained through multiple video frames in the sample video, and then the initial digital human can be calibrated based on the target head key points in the calibrated digital human, so that a more accurate training digital human can be obtained, thereby improving the accuracy and stability of the sample data.

[0015] In one embodiment, the target head key points include a first head key point and a second head key point, wherein the second head key point is determined based on optical flow information in a sample video, and the initial digital human is calibrated based on the target head key points of the calibration digital human to obtain a training digital human, including: based on a first loss function and a position of the first head key point in the calibration digital human, performing a first calibration on a first point to be calibrated on the initial digital human to obtain a first calibrated digital human; wherein the first head key point has a corresponding relationship with the first point to be calibrated on the initial digital human; based on a second loss function and the position of the second head key point in the calibration digital human, performing a second calibration on a second point to be calibrated on the first calibrated digital human to obtain a training digital human; wherein the second head key point has a corresponding relationship with the second point to be calibrated on the first calibrated digital human.

[0016] In one embodiment, the first calibrated digital human is a digital human obtained when the first loss function is minimized during the first calibration of the first point to be calibrated on the initial digital human; the training digital human is a digital human obtained when the second loss function is minimized during the second calibration of the second point to be calibrated on the digital human after the first calibration.

[0017] In one embodiment, determining a second head key point based on optical flow information in a sample video includes: obtaining light density and color in the sample video, and determining a head region of the sample user based on the light density and color; determining a second head key point from a plurality of pixels in the head region based on the brightness of each pixel in the head region and / or the contrast with adjacent pixels, wherein the brightness of the second head key point and / or the contrast with adjacent pixels is greater than a preset threshold.

[0018] In the above scheme, by determining the first loss function and the second loss function in sequence, and then calibrating the initial digital human through the calibration digital human, a more accurate training digital human can be obtained, and further, sample data can be obtained more accurately.

[0019] In one embodiment, the emotions of the sample user are identified by training the facial parameters of the digital person, including: calculating the similarity between the facial parameters of the training digital person and the facial parameters of each pre-set emotional avatar model; determining the emotional label of the training digital person based on the similarity, the emotional label includes the emotion and weight of the target emotional avatar model, the target emotional avatar model is the emotional avatar model corresponding to the similarity greater than or equal to a threshold, and the emotion of the sample user includes the emotional label of the training digital person.

[0020] In this scheme, by determining the emotion label, the emotion of each training digital person can be determined in more detail. In addition to the emotion, the weight corresponding to the emotion can also be determined, so that more accurate and detailed emotion labels can be obtained.

[0021] In one embodiment, the method further includes: determining the weight of the sample data according to the degree of matching between the head parameters of the sample user in the sample video and the sample voice corresponding to the sample video, wherein the weight of the sample data is positively correlated with the degree of matching; training the audio and video generation model according to the sample data corresponding to each personalized parameter to obtain a personalized digital human generation model corresponding to each personalized parameter, including: training the audio and video generation model according to the sample data whose weight corresponding to each personalized parameter is greater than a preset weight threshold to obtain a personalized digital human generation model corresponding to each personalized parameter.

[0022] In this solution, by setting the weights of sample data, better quality and more effective sample data can be obtained, thereby improving the accuracy of the model obtained through training.

[0023] Another aspect of the embodiments of the present application further provides a method for generating a three-dimensional digital human, the method comprising:

[0024] Obtaining target personalized parameters; the target personalized parameters include at least one of target identity information and target emotion information;

[0025] Determine the target adjustment parameter based on the target personalized parameter and the mapping relationship; the mapping relationship includes a one-to-one correspondence between multiple personalized parameters and multiple adjustment parameters, and the adjustment parameter is used to indicate the deviation of the head parameters between the animated digital human obtained by inputting the same speech into the personalized digital human generation model and the general digital human generation model;

[0026] Based on the target adjustment parameters and the animation initial digital human, an animation target digital human is generated, where the animation initial digital human is generated based on the first voice input to the universal digital human generation model.

[0027] In this solution, by using the digital human generation model, a more stable three-dimensional digital human can be obtained, thereby improving the stability of the three-dimensional digital human.

[0028] In a possible implementation, the first speech includes multiple speech frames, the target adjustment parameters include multiple frame adjustment parameters, and each frame adjustment parameter corresponds to a speech frame; based on the target adjustment parameters and the initial animation digital human, an animation target digital human is generated, specifically including: based on a frame adjustment parameter corresponding to each speech frame, each frame of the initial animation digital human corresponding to the speech frame is adjusted to generate an animation target digital human.

[0029] In this solution, by adjusting each frame of the initial digital human of the animation, the adjustment of the three-dimensional digital human can be achieved more comprehensively. Moreover, since each frame adjustment parameter corresponds to a voice frame, adjusting each frame of the initial digital human of the animation based on the frame adjustment parameter corresponding to each voice frame can also obtain a more realistic three-dimensional digital human that better matches the content of the first voice.

[0030] In a possible implementation, each frame of the digital human includes multiple grid units in a three-dimensional space, and each frame adjustment parameter includes: an identifier of a grid unit to be adjusted among the multiple grid units and an offset value of each grid unit to be adjusted; based on each frame adjustment parameter, each frame of the animated initial digital human corresponding to the voice frame is adjusted to generate an animated target digital human, specifically including: based on the identifier of the grid unit to be adjusted in each frame adjustment parameter and the offset value of each grid unit to be adjusted, the three-dimensional space coordinates of the grid unit to be adjusted in the animated initial digital human corresponding to the voice frame are adjusted to generate an animated target digital human corresponding to the voice frame.

[0031] In this solution, by using the identification and offset value of the grid unit to be adjusted, the grid units that need to be adjusted and the values ​​that need to be adjusted for each grid unit that needs to be adjusted can be determined more accurately and quickly, so that the adjustment of the grid units can be achieved more quickly and accurately, and a more accurate animated target digital human can be obtained.

[0032] In a possible implementation, the method further includes: sending the animated target digital human to a client, so that the client renders the animated target digital human.

[0033] In this solution, by transmitting the target animated digital human, the client can render the animated target digital human more accurately and quickly, and then the client can display the rendered three-dimensional digital human.

[0034] In a possible implementation, sending the animated target digital human to the client specifically includes: sending the target digital human and target deviation parameters corresponding to the first speech frame in the first speech to the client; wherein the target deviation parameters are at least part of multiple deviation parameters; each deviation parameter includes the deviation of the head parameters of the animated target digital human corresponding to adjacent speech frames in the animated target digital human.

[0035] In this solution, by determining the target digital human and the target deviation parameters, the client can more accurately render the three-dimensional digital human.

[0036] In a possible implementation, each deviation parameter includes: an identifier of a grid unit to be adjusted among multiple grid units of a target digital person corresponding to adjacent speech frames and an offset value of each grid unit to be adjusted; sending the target digital person and the target deviation parameter corresponding to the first speech frame in the first speech to the client, including: sending multiple target deviation parameters to the client in sequence according to the time sequence of each speech frame in the first speech.

[0037] In this solution, the transmission of multiple target deviation parameters is implemented in the order of the voice frames in the first voice, which can more accurately realize the rendering of the three-dimensional digital human by the client, thereby improving the real-time performance and accuracy of the three-dimensional digital human rendering by the client.

[0038] In a possible implementation, sending the target deviation parameter to the client includes: determining the target deviation parameter from a plurality of deviation parameters according to preset conditions, the preset conditions including bandwidth configuration information of the client, display requirements of the client, and at least one of weights of facial areas corresponding to the deviation parameters; and sending the target deviation parameter to the client.

[0039] In this scheme, the number of grid cells with differences in deviation parameters can be reduced by determining the target deviation parameters from the deviation parameters, thereby reducing the amount of information transmitted between the client and the server, reducing the amount of computation on the client during the rendering process, and improving rendering efficiency.

[0040] The computing device provided in the embodiment of the present application includes a memory and a processor. The memory stores a computer program that can be run on the processor. When the processor executes the program, the method of the embodiment of the present application is implemented.

[0041] The computer-readable storage medium provided in the embodiment of the present application has a computer program stored thereon, and when the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0042] In the training method of the three-dimensional digital human model and the three-dimensional digital human generation method provided in the embodiments of the present application, sample videos corresponding to different personalized parameters can be obtained; corresponding sample data can be obtained from the sample videos corresponding to each personalized parameter; the audio and video generation model can be trained according to the sample data corresponding to each personalized parameter to obtain the personalized digital human generation model corresponding to each personalized parameter; the different personalized digital human generation models are normalized to obtain the universal digital human generation model. Among them, since the head parameters of the sample user are obtained by calibrating the points to be calibrated on the initial digital human according to the target head key points in the calibration digital human established by the sample video, the audio and video generation model can be trained based on more accurate and stable sample data, so that a more stable and accurate universal digital human generation model can be obtained, and a more stable three-dimensional digital human can be obtained through the universal digital human generation model, thereby improving the stability of the three-dimensional digital human. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 This is a schematic diagram showing an interface of a method for generating a three-dimensional digital human provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of an application scenario of the method for generating a three-dimensional digital human provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of a flow chart of a training method for a three-dimensional digital human model provided in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of a process for determining sample data provided in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of a process for obtaining a trained digital human provided in an embodiment of the present application;

[0049] Figure 6 A schematic diagram of a head point cloud provided in an embodiment of the present application;

[0050] Figure 7 A schematic diagram of a process for calibrating an initial digital human provided in an embodiment of the present application;

[0051] Figure 8 A schematic diagram of a process for identifying sample user emotions provided in an embodiment of the present application;

[0052] Fig. 9 A schematic diagram of an emotional avatar model provided in an embodiment of the present application;

[0053] Fig.10 A schematic diagram of the process of generating a three-dimensional digital human provided in an embodiment of the present application;

[0054] Fig.11 This is a schematic diagram of the head grid of the three-dimensional digital human provided in the embodiment of this application. Please refer to Fig.11 ;

[0055] Fig.12 A schematic diagram of interface changes for rendering a three-dimensional digital human based on deviation parameters provided in an embodiment of the present application;

[0056] Fig.13 A schematic diagram of the structure of a training device for a three-dimensional digital human model provided in an embodiment of the present application;

[0057] Fig.14 This is a schematic diagram of the structure of the three-dimensional digital human generation device provided in the embodiment of the present application. Please refer to Fig.14 ;

[0058] Fig.15 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0062] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are used to distinguish similar or different objects, and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0063] The following is an explanation of the three-dimensional digital human provided in the embodiments of the present application and its practical application scenarios.

[0064] First of all, the three-dimensional digital human can be a virtual image displayed on the client, such as a virtual anchor, a virtual teacher, etc.

[0065] Optionally, the three-dimensional digital human is a virtual character image developed based on computer technology and artificial intelligence technology, with the characteristics of high-quality character image and exquisite visual expression. During use, the user can input sound, and make the three-dimensional digital human make corresponding actions, expressions, etc. according to the sound, so that the audience mistakenly believes that the sound is made by the three-dimensional digital human.

[0066] This technology can be used in different fields to meet corresponding needs. For example, three-dimensional digital humans can be used to simulate real anchors to conduct live streaming of goods, live games, or live chatting. Or, three-dimensional digital humans can be used to simulate real teachers to conduct online teaching.

[0067] The three-dimensional digital humans used in the above scenarios are mainly generated in real time. In actual use, they may also need to be pre-generated and then rendered in non-real time. For example, three-dimensional digital humans are used in movies and TV series to replace actors, and corresponding videos are produced based on three-dimensional digital humans. There is no specific restriction here. You can choose to obtain the three-dimensional digital human by real-time rendering or pre-rendering according to actual needs.

[0068] Among them, in the real-time rendering scene, taking the above-mentioned virtual anchor as an example, the three-dimensional digital human can be displayed in the live broadcast screen, and according to the input voice, the three-dimensional digital human can show different actions and expressions, such as happy actions and expressions, angry actions and expressions, etc., without specific restrictions here.

[0069] In the application process of three-dimensional digital humans, in order to reduce the pressure on the client, the client can perform rendering, and the server can generate the data needed in the rendering process and send it to the client.

[0070] For the client, it can display the corresponding three-dimensional digital human in the corresponding display interface. The following explains the scene of the three-dimensional digital human displayed in the client interface provided in the embodiment of the present application.

[0071] Figure 1 This is a schematic diagram of the interface display of the three-dimensional digital human generation method provided in the embodiment of this application. Please refer to Figure 1 , Figure 1 The interface shown is a live broadcast interface, in which a three-dimensional digital human 110 can be displayed, wherein the image of the three-dimensional digital human can be customized by the user, for example: a simulated real person image, a cartoon image, etc., which is not specifically limited here.

[0072] In this live broadcast scene, relevant parameters can be given to the three-dimensional digital human by inputting voice or text, so that the three-dimensional digital human can perform corresponding actions.

[0073] For example, if the user inputs a voice expressing exclamation, the corresponding three-dimensional digital human can show an expression of exclamation according to the voice, and the lips can also make exclamation mouth shapes.

[0074] In the application of three-dimensional digital humans, in addition to the client needing to render and display the corresponding three-dimensional digital humans, the process of calculating the relevant parameters of each action of the three-dimensional digital human can be completed by the server. The following explains the actual application scenario of the three-dimensional digital human generation method provided in the embodiment of the present application.

[0075] In this scenario, the server can be, for example, a server, one or more servers, or a server cluster. The client can be various terminal devices, mobile phones, personal computers, tablet computers, etc., and is not specifically limited here. The client and the server are connected in communication.

[0076] Figure 2 This is a schematic diagram of an application scenario of the three-dimensional digital human generation method provided in the embodiment of the present application. Please refer to Figure 2 , the scenario may include a server 210 and multiple clients 220 corresponding to the server 210 .

[0077] Among them, the service end 210 can be an end running on a server, for example: a cloud server, an AI (Artificial Intelligence) server, a rack server, etc. that can realize the relevant parameter calculations of a three-dimensional digital human. Among them, the AI ​​server can be a server for performing artificial intelligence calculations, and the rack server can be, for example, a server installed in a standardized network equipment rack, and no specific restrictions are made here.

[0078] The client 220 can be a terminal device used by the user, for example, including but not limited to mobile phones, wearable devices (such as smart watches, smart bracelets, smart glasses, etc.), tablet computers, laptops, car terminals, PCs (Personal Computers), etc., without specific limitations here.

[0079] It should be noted that in an application scenario, the server 210 can synchronously implement data interaction for multiple clients 220. For example, the client sends the relevant parameters of the three-dimensional digital human to the server, and the server calculates the relevant parameters based on the relevant parameters to obtain the relevant data required for rendering, and then sends these data to the corresponding client, allowing the client to complete the rendering work.

[0080] In this scenario, an embodiment of the present application provides a method for training a three-dimensional digital human model and a method for generating a three-dimensional digital human.

[0081] Figure 3 This is a flow chart of the training method of the three-dimensional digital human model provided in the embodiment of the present application. Please refer to Figure 3 , the method comprising:

[0082] S310: Obtain sample videos corresponding to different personalized parameters.

[0083] Optionally, the sample videos may be pre-acquired 2D videos (two-dimensional videos), and these sample videos may be classified in advance to obtain sample videos corresponding to each personalized parameter.

[0084] Among them, the personalized parameters may include: at least one of identity information and emotional information. The identity information may be the identity represented by the three-dimensional digital person, such as age, gender, etc.; the emotional information may be the emotions that the three-dimensional digital person can express, such as: happiness, anger, sadness, etc.

[0085] It should be noted that the corresponding sample videos can be configured according to the required personalized parameters, or the personalized parameters in these sample videos can be determined through the collected sample videos, and no specific limitation is made here.

[0086] For example, if 100 different sets of personalized parameters are configured, a corresponding sample video may be set for each set of personalized parameters. Alternatively, if 20 sample videos are acquired in advance, the personalized parameters existing in the 20 sample videos may be extracted.

[0087] It should be noted that the two-dimensional video may be a video with a real person speaking, for example, a video of a teacher giving a lecture. In this sample video, there needs to be a head image of a real person, preferably a face image.

[0088] For any two-dimensional video, in the actual implementation process, the real person may show different emotions, for example, changing from happy to sad, or the real person in the video may change, for example, the first half of the video is shot with a male, and the second half changes to a female, etc.

[0089] Before obtaining sample videos, these two-dimensional videos can be segmented. For example, segments with different emotions or identities in the video can be identified by clustering, and multiple segments can be obtained, and each segment can be used as a sample video.

[0090] Optionally, multiple sample videos may be obtained by segmenting the two-dimensional video, and each sample video may correspond to a set of personalized parameters. For example, the personalized parameters corresponding to a sample video are: male, 50 years old, happy; and the personalized parameters corresponding to another sample video are: sad.

[0091] It should be noted that the correspondence between the sample videos and the personalized parameters can be presented in the form of Table 1.

[0092] Table 1

[0093] Sample Video Personalized parameters Sample Video 1 male Sample Video 2 female Sample Video 3 Male, 20 years old … … Sample Video N-1 Male, 50 years old, happy Sample Video N Male, happy

[0094] Please refer to Table 1. Each sample video may correspond to a set of different personalized parameters. The personalized parameters may be one or more, and there is no specific limitation here.

[0095] S320: Obtain corresponding sample data from the sample videos corresponding to each personalized parameter.

[0096] The sample data includes sample voice, sample user's emotions and sample user's head parameters.

[0097] Optionally, after obtaining the sample video, corresponding sample data can be determined from the sample video. For the same sample video, multiple groups of sample data can be obtained, wherein each group of sample data can include sample voice, sample user's emotions, and sample user's head parameters.

[0098] The sample voice may be a part of the voice corresponding to the sample video, the emotion of the sample user may be the emotion of the real person in the sample video, and the head parameters of the sample user may be the head parameters of a three-dimensional model (that is, the sample user) established in three-dimensional space based on the head image of the real person in the sample video.

[0099] For each sample video corresponding to a personalized parameter, the above-mentioned multiple groups of sample data can be obtained.

[0100] It should be noted that the head parameters of the sample user may specifically include the head contour parameters, facial parameters, lip parameters, etc. of the sample user. These parameters may specifically be the positions of the grid units at the corresponding positions. For example, the head contour parameters may be the positions of the grid units that constitute the head contour of the entire three-dimensional digital human; the facial parameters may be the positions of the grid units that constitute the face of the three-dimensional digital human; and the lip parameters may be the positions of the grid units that constitute the lips of the three-dimensional digital human.

[0101] Optionally, the head parameters and the emotions of the sample users may correspond to the sample voices, and each voice frame may correspond to the head parameters and the emotions of the sample users.

[0102] For example: if a sample speech in a sample data has 100 speech frames, there may be 100 emotions of the sample user and 100 head parameters of the sample user, and the emotion and head parameter of each frame of the sample user corresponds to a speech frame of the sample speech.

[0103] In one embodiment, the head parameters of the sample user are obtained by calibrating the points to be calibrated on the initial digital human according to the target head key points in the calibration digital human established by the sample video.

[0104] It should be noted that the target head key points of the sample user refer to some key points screened from the head of the sample user, wherein the target head key points may be multiple key points of the calibrated digital human head.

[0105] The initial digital human may include multiple points to be calibrated, each of which may correspond to a target head key point, and the position of the point to be calibrated corresponding to the target head key point may be calibrated according to the position of the target head key point.

[0106] After calibrating the corresponding points to be calibrated based on each target head key point, a calibrated digital human can be obtained, and the head parameters of the calibrated digital human can be used as the head parameters of the sample user.

[0107] It should be noted that the calibrated digital human can be a digital human model without specific modeling, for example, it can be a model frame, the calibrated digital human only has the coordinates of each target head key point, but does not have the corresponding specific appearance and style in three-dimensional space. The calibrated digital human can be obtained based on multiple frames of two-dimensional images in the sample video.

[0108] The initial digital human may be a digital human with a specific appearance and style, and each point to be calibrated of the initial digital human has an actual position in the three-dimensional space. The initial digital human may be a digital human with a specific appearance and style that is pre-generated in the three-dimensional space.

[0109] In one embodiment, the various points to be calibrated of the initial digital human can be calibrated by calibrating the positions of the key points of the target head in the digital human, that is, adjusting the positions of the various points to be calibrated of the initial digital human, thereby obtaining a calibrated digital human with a specific appearance and style in three-dimensional space, and the calibrated digital human can be used as a training digital human.

[0110] S330: Training the audio and video generation model according to the sample data corresponding to each personalized parameter to obtain a personalized digital human generation model corresponding to each personalized parameter.

[0111] Optionally, sample data corresponding to each personalized parameter may be obtained in the above manner, and then the audio-visual generation model may be trained based on the sample data.

[0112] Among them, the audio and video generation model can be an audio and video encoder, or a neural network structure, whose input can be sample sounds and the emotions of sample users, and whose output can be a three-dimensional digital human. The head parameters of the output three-dimensional digital human can be compared with the head parameters of the sample user. After a large amount of training, if the audio and video generation model converges, for example: when the degree of match between the head parameters of the output three-dimensional digital human and the head parameters of the sample user is greater than a preset similarity threshold, it can be determined that the audio and video generation model converges, thereby obtaining a personalized digital human generation model corresponding to the personalized parameters.

[0113] It should be noted that the above-mentioned training process can be performed for a set of personalized parameters corresponding to each sample video, so as to obtain a personalized digital human generation model corresponding to each set of personalized parameters. That is to say, multiple personalized digital human generation models are obtained by training the same audio and video generation model based on different sample data.

[0114] S340: Normalizing different personalized digital human generation models to obtain a universal digital human generation model.

[0115] It should be noted that after obtaining multiple personalized digital human generation models, these personalized digital human generation models can be normalized.

[0116] It should be noted that there are many methods for normalization processing. For example, the normalization processing of input features can be used, that is, before training these personalized digital human generation models, the input data is normalized to ensure that different sample data have similar scales. Alternatively, inter-layer normalization processing can be used. Since each personalized digital human generation model is trained based on the same audio and video generation model, these personalized digital human generation models have a certain degree of similarity. For example, the number of neural network layers is the same, and normalization can be applied after each neural network layer is generated, especially before the activation function. Alternatively, the weight normalization method can be used to re-parameterize the weights of each layer in the neural network, decouple the length of the weight vector and its direction, and normalize these personalized digital human generation models.

[0117] In the actual implementation process, any one of the above-mentioned normalization methods can be used to implement the normalization processing of multiple personalized digital human generation models, so as to obtain a universal digital human generation model.

[0118] The general digital human generation model can be obtained by normalizing all personalized digital human generation models.

[0119] In actual application, the input parameter of the universal digital human generation model may be a first voice, and the output may be a three-dimensional digital human.

[0120] In the training method of the three-dimensional digital human model provided in the embodiment of the present application, sample videos corresponding to different personalized parameters can be obtained; corresponding sample data can be obtained from the sample videos corresponding to each personalized parameter; the audio and video generation model can be trained according to the sample data corresponding to each personalized parameter to obtain the personalized digital human generation model corresponding to each personalized parameter; the different personalized digital human generation models are normalized to obtain the universal digital human generation model. Among them, since the head parameters of the sample user are obtained by calibrating the points to be calibrated on the initial digital human according to the target head key points in the calibration digital human established by the sample video, the audio and video generation model can be trained based on more accurate and stable sample data, so that a more stable and accurate universal digital human generation model can be obtained, and a more stable three-dimensional digital human can be obtained through the universal digital human generation model, thereby improving the stability of the three-dimensional digital human.

[0121] The following is an explanation of the implementation process of determining sample data based on sample videos provided in the embodiments of the present application.

[0122] Figure 4 Please refer to the flowchart of determining sample data provided in the embodiment of this application. Figure 4 For the sample video, the corresponding sample speech in each frame of the sample video can be determined, and the three-dimensional model established based on the video image of the real person can be used to determine the emotion and head parameters of the three-dimensional model established in the three-dimensional space, and the head parameters of the model are used as the head parameters of the sample user, and the facial emotions of the model are used as the emotions of the sample user.

[0123] Among them, a three-dimensional model can be first established based on multiple frames of two-dimensional images in a two-dimensional video, that is, a digital human is trained, and the head parameters of the sample user are determined based on the head parameters of the trained digital human, and then the emotions of the trained digital human are identified to obtain the emotions of the sample user.

[0124] In one embodiment, corresponding sample data is obtained from sample videos corresponding to each personalized parameter, including: establishing a corresponding training digital human according to the sample video, and determining the head parameters of the sample user according to the head parameters of the training digital human; identifying the emotions of the sample user through the facial parameters of the training digital human; determining the sample speech corresponding to each personalized parameter, as well as the emotion of the sample user and the head parameters of the sample user corresponding to each speech frame of the sample speech.

[0125] It should be noted that the speech in the sample video can be extracted and used as the sample speech in the sample data.

[0126] Optionally, the head area of ​​the real person in the video can be determined from the sample video, and then the head parameters of the sample user in the two-dimensional image can be determined from the head area. A conversion method of multiple frames of two-dimensional images to three-dimensional modeling is adopted to establish a training digital human and obtain the head parameters of the training digital human. The head parameters of the training digital human can be considered to be accurate parameters, that is, the head parameters of the training digital human can be used as the head parameters of the sample user in the sample data.

[0127] After obtaining the trained digital human, the sample user's emotions can be identified based on the facial parameters of the trained digital human. For example, the sample user's emotions can be determined by comparing the differences between the facial parameters of the three-dimensional digital human and the facial parameters of a preset three-dimensional digital human with emotional tendencies.

[0128] For each speech frame, there can be a corresponding training digital human. The head parameters of the sample user corresponding to the training digital human are the head parameters of the sample user corresponding to the speech frame. Correspondingly, the emotions of the sample user corresponding to the training digital human are the emotions of the sample user corresponding to the speech frame.

[0129] Through the above steps, the head parameters of the sample user and the emotion of the sample user corresponding to each speech frame of the sample speech in the sample data can be obtained respectively.

[0130] It should be noted that the training digital human can be a digital human obtained by converting a two-dimensional graphic into a three-dimensional graphic. For example, projection modeling can be used to obtain the depth information of the two-dimensional image through different angles of the real person's head in multiple frames of two-dimensional images, and then a training digital human can be established based on the depth information of the two-dimensional image and the real person's facial image displayed in the two-dimensional image.

[0131] In a training method for a three-dimensional digital human model provided in an embodiment of the present application, a corresponding training digital human can be established based on a sample video, and the head parameters of a sample user can be determined based on the head parameters of the training digital human; the emotions of the sample user can be identified by the facial parameters of the training digital human; the sample voice corresponding to each personalized parameter, and the sample user's emotions and the head parameters of the sample user corresponding to each voice frame of the sample voice can be determined. Among them, by establishing a training digital human, the head parameters and emotions of the sample user can be obtained more quickly and accurately, so that the sample data can be obtained more quickly and accurately.

[0132] The specific implementation process of determining the training of a three-dimensional digital human provided in the embodiment of the present application is explained below.

[0133] Figure 5 This is a flow chart of obtaining a trained digital human provided in the embodiment of the present application. Please refer to Figure 5 , Figure 5 In the process shown in , a calibration digital human can be first established based on the sample video, and then the initial digital human can be calibrated based on the calibration digital human to obtain a training digital human. The specific implementation steps are as follows:

[0134] In one embodiment, a sample video includes multiple video frames, and a corresponding training digital human is established based on the sample video, including: establishing a calibrated digital human through multiple video frames in the sample video, the calibrated digital human including multiple frames; calibrating the initial digital human according to the target head key points of the calibrated digital human to obtain a training digital human.

[0135] It should be noted that the calibrated digital human can be established based on the sample video. For example, the depth information of the sample user's head can be calculated based on multiple video frames of the sample video, and then the calibrated digital human can be established based on the two-dimensional image of the sample user in the sample video and the depth information of the sample user's head. The calibrated digital human can be a digital human without a specific appearance and style, for example, a framework of a three-dimensional digital human.

[0136] In the process of calculating the depth information, the depth information can be calculated based on the difference in the two-dimensional images of the sample user between multiple video frames. For example, one of the two-dimensional images is the front face of the sample user, and the other two-dimensional image is the side face of the sample user. Based on the position difference between the front face and the side face, the depth information of a part of the points in the head area of ​​the sample user can be determined. The above steps can be performed based on any two frames of sample video, and the depth information of the sample user's head can be calculated.

[0137] Optionally, in the actual implementation process, a three-dimensional model can be set in space, and multiple frames of two-dimensional images can be mapped onto the three-dimensional model to obtain the depth information of each point in the head area of ​​the sample user, thereby realizing the process of converting the two-dimensional image into a three-dimensional model. The obtained three-dimensional model framework can be used as a calibration digital human.

[0138] It should be noted that the depth information of the sample user may refer to the depth of each point in the head area in the two-dimensional image of the sample user. After the depth information is calculated in the above manner, a calibrated digital human may be established based on the two-dimensional image of the sample user and the depth information of the sample user's head.

[0139] Optionally, after obtaining the calibrated digital human, an initial digital human can be established. The initial digital human can be a digital human without any personalized parameter tendencies, for example, it can be a digital human without gender tendencies, emotional tendencies, or age tendencies, or it can be any digital human with certain emotional information or certain identity information. There is no specific limitation here, and you can choose to set an initial digital human according to actual needs.

[0140] Among them, the initial digital human can be a digital human with specific modeling, for example: a digital human can be randomly generated in a three-dimensional space, or it can be a digital human established in advance without personalized parameter tendencies. When it is necessary to establish an initial digital human, the established digital human can be called.

[0141] In one embodiment, the positions of the corresponding points to be calibrated in the initial digital human can be calibrated based on the positions of the target head key points of the calibration digital human, so that the calibrated initial digital human is used as the training digital human.

[0142] In a training method for a three-dimensional digital human model provided in an embodiment of the present application, a calibration digital human can be established through multiple video frames in a sample video, and the calibration digital human includes multiple frames; the initial digital human is calibrated according to the target head key points of the calibration digital human to obtain a training digital human. Among them, a more accurate calibration digital human can be obtained through multiple video frames in the sample video, and then the initial digital human can be calibrated based on the target head key points in the calibration digital human, so that a more accurate training digital human can be obtained, and then the accuracy and stability of the sample data can be improved.

[0143] In one embodiment, in the process of establishing a calibrated digital human based on a sample video, head features may be extracted by means of a head point cloud, thereby determining depth information.

[0144] It should be noted that a head point cloud can be established through the head area in the two-dimensional image of the sample user in multiple frames of the sample video. These head point clouds can be a collection of points with depth information in the head area of ​​the sample user. For example, the depth information of multiple points can be determined based on multiple frames of video, and the collection formed by these points is the head point cloud.

[0145] The head point cloud may be a method of representing the head features of the sample user in the two-dimensional image by means of multiple points. For example, the head contour and facial details of the sample user may be formed by means of multiple points.

[0146] After obtaining the head point cloud, these head point clouds can be converted into corresponding grid units in three-dimensional space according to the depth information of the head point cloud, thereby realizing the conversion of two-dimensional image to three-dimensional image. The three-dimensional digital human obtained by this conversion method is the above-mentioned calibrated digital human.

[0147] In order to explain the head point cloud more clearly, an example distribution of the head point cloud is explained below.

[0148] Figure 6 This is a schematic diagram of the head point cloud provided in the embodiment of this application, please refer to Figure 6 , Figure 6 The head outline and facial expression of the sample user are composed of multiple points in the head point cloud. The conversion from two-dimensional space to three-dimensional space can be performed based on these points. Since each point has depth information, the corresponding three-dimensional digital human can be established in three-dimensional space based on the distribution of these points in the two-dimensional plane and their corresponding depth information. This three-dimensional digital human is also the above-mentioned calibrated digital human.

[0149] The specific implementation process of calibrating the initial digital human based on the calibrated digital human provided in the embodiment of the present application is explained below.

[0150] Figure 7 This is a schematic diagram of a process for calibrating an initial digital human provided in an embodiment of the present application. Please refer to Figure 7 The target head key points include a first head key point and a second head key point, wherein the second head key point is determined according to the optical flow information in the sample video, and the initial digital human is calibrated according to the target head key points of the calibration digital human to obtain the training digital human, including:

[0151] S710: Based on the first loss function and the position of the target head key point in the calibrated digital human, a first calibration is performed on the first point to be calibrated on the initial digital human to obtain a digital human after the first calibration.

[0152] The first head key point corresponds to the first point to be calibrated on the initial digital human.

[0153] According to the position of the first head key point in the calibrated digital human, the point to be calibrated corresponding to the first head key point on the initial digital human is determined, and based on the first loss function, the first point to be calibrated on the initial digital human is calibrated for the first time to obtain the digital human after the first calibration.

[0154] It should be noted that the first head key point may be any target head key point, and a certain number of grid cells may be randomly selected from the head grid cells of the calibrated digital human as the first head key point.

[0155] The initial digital human and the calibrated digital human can be two three-dimensional digital humans composed of the same number of grid units. The position of the first point to be calibrated corresponding to the first head key point in the initial digital human can be determined based on the number of the grid units, and then the position of the first point to be calibrated corresponding to the initial digital human can be calibrated first based on the position of the first head key point in the calibrated digital human. The first calibration process can be a process of calculating the result of the first loss function of both the initial digital human and the calibrated digital human.

[0156] In one embodiment, the first calibrated digital human is a digital human obtained when the first loss function is minimized during the first calibration of the first to-be-calibrated point on the initial digital human.

[0157] Among them, the calculation formula of the first loss function is as follows:

[0158] L init =∑ i ||P i -K i ||2;

[0159] In this formula, L init It is the first loss function, i is a number of coordinates randomly initialized in three-dimensional space, which can be used to represent the number of the first head key points, and the position of the first head key point in the calibration digital human is P i , the position of the point to be calibrated on the initial digital human is K i .

[0160] For each frame of the digital human, the first loss function mentioned above can be used for calibration, and in each frame, the first loss function L init In the smallest case, each K i The digital human formed by the value of is the digital human after the first calibration, that is, the digital human after the first calibration can be determined when the first loss function is minimized.

[0161] S720: Based on the second loss function and the position of the second head key point in the calibrated digital human, perform a second calibration on the second to-be-calibrated point on the digital human after the first calibration to obtain a training digital human.

[0162] The second head key point has a corresponding relationship with the second to-be-calibrated point on the digital human after the first calibration.

[0163] It should be noted that the second head key point can be the target head key point obtained after screening based on optical flow information, wherein the optical flow information refers to the motion trajectory of the key point with obvious position changes in any two adjacent frames in the sample video, and the key point that generates the motion trajectory can be used as the second head key point.

[0164] The obvious position change refers to the pixel with the highest brightness in the head area of ​​the sample user in the sample video, or may be a pixel with the largest contrast.

[0165] In the process of determining the second head key point, the following methods can be used:

[0166] Optionally, determining a second head key point based on optical flow information in a sample video includes: obtaining light density and color in the sample video, and determining a head region of the sample user based on the light density and color; determining a second head key point from a plurality of pixels in the head region based on the brightness of each pixel in the head region and / or the contrast with adjacent pixels, wherein the brightness of the second head key point and / or the contrast with adjacent pixels is greater than a preset threshold.

[0167] The light density may refer to the degree of brightness of different regions in each frame of the sample video, which is related to the contrast and brightness of the image. The color may refer to the color of each pixel in each frame of the sample video, which may be represented by the weights of the three colors RGB (red, green and blue).

[0168] The head area of ​​the sample user can be determined based on the above two types of information. For example, the light density in the head area is usually within a certain range, and the color of the pixels in the area can also be determined within a certain range based on the real person's skin color and scene lighting. Based on these two ranges, each pixel in the video frame image can be traversed to determine the corresponding head area.

[0169] It should be noted that the second head key point may be a pixel in the head area of ​​the sample user in the sample video, for example, it may be a pixel with the highest brightness in the head area of ​​the sample user in the sample video, or it may be a pixel with the greatest contrast.

[0170] The maximum contrast ratio may refer to the maximum difference in color or brightness relative to other pixel points around the pixel point.

[0171] Optionally, in addition to selecting the method with the highest brightness and the largest contrast, a brightness threshold and a contrast threshold may be set to select pixels whose brightness is greater than the brightness threshold and whose contrast with adjacent pixels is greater than a preset threshold as the second head key point.

[0172] Optionally, after determining the second head key point, the position of the second head key point in the calibrated digital human may be determined.

[0173] Then, based on the position of the second head key point in the calibrated digital human and the second loss function, the position of the second point to be calibrated on the digital human after the first calibration can be calibrated for the second time. When the second loss function is minimized, each second point to be calibrated on the digital human after the first calibration is K i The value can be used as the result of the second calibration, and the trained digital human can be obtained according to the result of the second calibration.

[0174] In one embodiment, the training digital human is a digital human obtained when the second loss function is minimized during the second calibration of the second to-be-calibrated point on the digital human after the first calibration.

[0175] Among them, the calculation formula of the second loss function is as follows:

[0176] L sec =∑ i ||P i (R,T)-K i ||2;

[0177] In this formula, L sec That is, the second loss function, i is a number of coordinates randomly initialized in three-dimensional space, which can be used to represent the number of the second head key points, and the position of the second head key points in the calibration digital human is P i , the position of the second calibration point on the digital human after the first calibration is K i .

[0178] It should be noted that R can be a rotation vector, T can be a displacement vector, and the second loss function can represent the position of point Pi in three-dimensional space according to the coordinate representation of the rotation vector and the position vector.

[0179] For each frame of the digital human, the second loss function can be used for calibration, and in each frame, the second loss function L sec In the smallest case, each K i The initial digital human formed by the value of is the digital human after the second calibration, and the digital human after the second calibration is the training digital human, that is, the training digital human can be determined when the second loss function is minimized.

[0180] Optionally, the number of the second head key points may be smaller than the number of the first head key points.

[0181] In the actual implementation process, coarse calibration can be achieved through the first calibration method, that is, calibration is performed by randomly selecting a large number of head key points. After the coarse calibration is completed, fine calibration can be achieved through the second calibration method, that is, calibration is performed by directionally selecting a small number of head key points. After the first calibration and the second calibration are performed in sequence, the calibration of the initial digital human can be completed.

[0182] It should be noted that after the above-mentioned first calibration and second calibration are performed in sequence, the digital human after the second calibration when the second loss function is minimized can be used as the training digital human.

[0183] In the training of the three-dimensional digital human model provided in the embodiment of the present application, the first point to be calibrated on the initial digital human can be first calibrated based on the first loss function and the position of the first head key point in the calibration digital human to obtain the first calibrated digital human; based on the second loss function and the position of the second head key point in the calibration digital human, the second point to be calibrated on the first calibrated digital human can be second calibrated to obtain the training digital human. Among them, by determining the first loss function and the second loss function in sequence, and then calibrating the initial digital human through the calibration digital human, a more accurate training digital human can be obtained, and further, the sample data can be obtained more accurately.

[0184] It should be noted that after obtaining the trained digital human, the emotion of the trained digital human can be determined, and the determination can be performed in the following manner.

[0185] Figure 8 Please refer to the flowchart of identifying the sample user's emotions provided in the embodiment of this application. Figure 8 , by training the facial parameters of the digital human, the emotions of the sample user are recognized, including:

[0186] S810: Calculate the similarity between the facial parameters of the training digital human and the facial parameters of each preset emotional avatar model.

[0187] Optionally, multiple emotional avatar models can be pre-stored on the server side, each of which can be a three-dimensional digital human with an expression tendency. Each three-dimensional digital human with an expression tendency can represent an emotion, such as happiness, sadness, anger, etc., and the facial expression of each emotional model is matched with the corresponding emotion, such as the expression corresponding to the happy emotion, the expression corresponding to the sad emotion, the expression corresponding to the angry emotion, etc., without specific limitation here.

[0188] It should be noted that the facial parameters of each emotion model may be parameters corresponding to the emotion represented by the model.

[0189] For example, different emotional avatar models can be set for different emotions. For example, human emotions can be divided into 52 different emotions, and an emotional avatar model can be set for each emotion.

[0190] It should be noted that the number and distribution of grid units in all emotional avatar models may be the same as the number and distribution of grid units in training digital humans.

[0191] After obtaining the trained digital human, the facial parameters of the trained digital human can be compared with the facial parameters of the emotional avatar model, for example, similarity can be calculated, wherein the similarity can be determined by comparing the relative positions of each grid unit corresponding to the facial parameters in the trained digital human and the emotional avatar model.

[0192] The relative position may be the relative position between multiple grid units in the training digital human and the emotional avatar model, for example, the deviation between the position of any grid unit in the training digital human and the position in the emotional avatar model.

[0193] It should be noted that the number and distribution of grid cells in the emotional avatar model can be the same as the number and distribution of grid cells in the training digital human. In the actual implementation process, the similarity of facial parameters between the training digital human and each pre-set emotional avatar model can be determined based on the degree of matching of the relative positions.

[0194] S820: Determine the emotion label of the training digital human according to the similarity.

[0195] Among them, the emotion label includes the emotion and weight of the target emotion avatar model, the target emotion avatar model is the emotion avatar model corresponding to the similarity greater than or equal to the threshold, and the emotion of the sample user includes the emotion label of the training digital person.

[0196] In one embodiment, facial parameters of the training digital human can be compared with each pre-set emotional avatar model, and a target emotional avatar model with the highest similarity in the comparison results can be determined. The emotion and weight of the target emotional avatar model can be used as the emotion in the emotional label of the training digital human and the weight obtained by comparison.

[0197] It should be noted that the similarity can be expressed in the form of weights. For example, if the similarity between the training digital human and the emotional avatar model corresponding to the emotion of happiness is compared, the weight of the emotion of happiness in the training digital human can be obtained. For example, if the similarity is 60%, the weight of the emotion of happiness can be obtained as 0.6.

[0198] Optionally, the similarity between the training digital human and each emotional avatar model can be determined in the above manner, thereby obtaining the emotional label of the three-dimensional digital human.

[0199] Among them, emotional labels can be recorded in the form of "emotion + weight", for example: "happy 0.6, sad 0.1, angry 0.1" can mean that in the expression corresponding to the emotional label, the weight of the emotion of happiness is 0.6, the weight of the emotion of sadness is 0.1, and the weight of the emotion of anger is 0.1.

[0200] It should be noted that the weight may represent the degree of similarity between the training digital human and the emotional avatar model of the corresponding emotion. The above-mentioned emotional label may record that the expression has a similarity of 0.6 to the emotion of happiness, a similarity of 0.1 to the emotion of sadness, and a similarity of 0.1 to the emotion of anger.

[0201] In addition, multiple weights may be recorded in the emotion tag in a certain comparison order with multiple emotion avatar models, for example, in the order of "0.6, 0.1, 0.1, 0.5, ..., 0.2". If 52 emotion avatar models are compared, the order may have 52 similarity weights.

[0202] It should be noted that in the actual process of determining emotion labels, the cases with lower similarity can be discarded and only the results with higher similarity can be retained. For example, for the label "happy 0.6, sad 0.1, angry 0.1" in the aforementioned example, if the similarity threshold is 0.5, the emotion label can be changed to the result "happy 0.6".

[0203] In the actual process of expressing emotional labels, if there is only one label and its weight is 100%, the emotion corresponding to the label can be used as the label to express it, for example: "happy", "sad", etc.; if there is only one label but its weight is not 100%, the emotion corresponding to the label and its weight can also be used as the label to express it, for example: "happy 0.6", "sad 0.5", etc.; if there are multiple labels and each label has a corresponding weight, they can be expressed in a combined manner, for example: "happy 0.6, sad 0.2", "angry 0.6, sad 0.5, happy 0.1".

[0204] That is to say, after determining the similarity, a target emotional avatar model, that is, an emotional avatar model with a similarity greater than or equal to a threshold, can be determined from the emotional avatar models, thereby obtaining a corresponding emotional label.

[0205] Optionally, the sample user's emotions may include emotion labels of the trained digital human, and the emotion labels may be used to represent the sample user's emotions. Alternatively, the emotion labels and other relevant data may be combined to represent the sample user's emotions, which is not specifically limited here.

[0206] It should be noted that the emotions of sample users can be expressed in the form of emotion labels, that is, the emotions of sample users in the same frame are not unique, and may be mixed with both "happy" and "angry". In this case, the emotions of sample users in a certain frame can be expressed in the form of emotion labels with weights. For the emotions of sample users in multiple frames, multiple emotion labels can be used to express them.

[0207] In the training method of the three-dimensional digital human model provided in the embodiment of the present application, the similarity between the facial parameters of the training digital human and each pre-set emotional avatar model can be calculated; and the emotional label of the training digital human can be determined according to the similarity. Among them, by determining the emotional label, the emotion of each training digital human can be determined in more detail, and in addition to the emotion, the weight corresponding to the emotion can also be determined, so that a more accurate and detailed emotional label can be obtained.

[0208] In one embodiment, the sample video includes multiple video frames, the training digital human includes multiple frames, each frame of the training digital human corresponds one-to-one to a video frame, and the emotion of the sample user includes an emotion label of the training digital human in each frame.

[0209] Optionally, the sample speech in the sample video may include multiple speech frames, each speech frame may correspond to a video frame in the sample video, and accordingly, may correspond to a frame of training digital human, and the emotions of the training digital human corresponding to each video frame may be different. For example, the expression of the training digital human in the first video frame is a happy expression, and the expression of the training digital human in the tenth video frame becomes a sad expression. In this case, the emotion label of each video frame can be recorded, and these multiple emotion labels together constitute the emotion of the sample user, that is, the emotion of the above-mentioned sample user.

[0210] It should be noted that in the process of training the model, the emotions of the sample users can be trained with one or more specific emotions, or, the weight of the emotions can be added to the specific emotions for training. For example, the emotions of the sample users in each frame can be input into the audio and video generation model in the form of an emotion label for training. No specific restrictions are made here.

[0211] Through the above steps, we can obtain sample speech, sample user's emotions and sample user's head parameters respectively, so as to obtain more accurate and detailed sample data. After training the audio and video generation model according to these sample data, we can obtain a personalized digital human generation model.

[0212] It should be noted that a universal digital human generation model can be obtained after normalization processing based on the personalized digital human generation model.

[0213] Correspondingly, the emotional avatar model explained above can also be a three-dimensional digital human with preset expressions, and the three-dimensional digital human can be displayed in the following manner.

[0214] Fig. 9 This is a schematic diagram of the emotional avatar model provided in the embodiment of the present application, please refer to Fig. 9 , Fig. 9 The 3D digital human on the left can be the emotional avatar model corresponding to the emotion of surprise. Fig. 9 The 3D digital human on the right can be the emotional avatar model corresponding to the emotion of calmness. Fig. 9 Only the emotional avatar models corresponding to these two emotions are taken as examples. In the actual implementation process, different emotional avatar models can be set for different emotions.

[0215] In one embodiment, the method further includes: determining the weight of the sample data according to the degree of matching between the head parameters of the sample user in the sample video and the sample voice corresponding to the sample video, wherein the weight of the sample data is positively correlated with the degree of matching.

[0216] Among them, the weight of the sample data can be a measurement condition used to indicate the accuracy of the sample data. If the weight of the sample data is greater than a preset weight threshold, it can be determined that the accuracy of the sample data is high. If the weight of the sample data is less than or equal to the preset weight threshold, it can be determined that the accuracy of the sample data is low.

[0217] It should be noted that the matching degree between the head parameters of the sample user and the sample voice corresponding to the sample video can be manually annotated, or can be obtained by recognition through a corresponding neural network model.

[0218] If a manual representation method is used, the corresponding sample voice in each sample video and the lip movement of the sample user in the video can be determined in advance to determine whether they match. If they match, the matching degree can be determined to be 1. If they do not match, the matching degree can be determined to be 0. An average value can be calculated based on the overall matching degree in a video. For example, for a video lasting 1 minute, the matching degree of the first 30 seconds is 1, and the matching degree of the last 30 seconds is 0, then the overall matching degree can be determined to be 0.5.

[0219] If the model recognition method is adopted, the video can be input into a pre-trained recognition model, and the model can determine whether the sound of each frame matches the lip movement of the sample user. Then, a similar labeling method can be used to determine the matching degree of a video.

[0220] It should be noted that the weight of the sample data can be set according to the matching degree. The higher the matching degree, the greater the weight of the sample data; the lower the matching degree, the smaller the weight of the sample data.

[0221] After obtaining the weights of the sample data, the sample data can be screened based on the weights during training based on the sample data.

[0222] Optionally, according to the sample data corresponding to each personalized parameter, the audio and video generation model is trained to obtain the personalized digital human generation model corresponding to each personalized parameter, including: according to the sample data corresponding to each personalized parameter whose weight is greater than a preset weight threshold, the audio and video generation model is trained to obtain the personalized digital human generation model corresponding to each personalized parameter.

[0223] For example, if the preset weight threshold is 0.8, sample data with a weight less than 0.8 can be discarded, and these samples are not used in the training process of the personalized digital human generation model.

[0224] In the training method of the three-dimensional digital human model provided in the embodiment of the present application, the weight of the sample data can be determined according to the matching degree between the head parameters of the sample user in the sample video and the sample voice corresponding to the sample video; the audio and video generation model is trained according to the sample data whose weight corresponding to each personalized parameter is greater than the preset weight threshold, and the personalized digital human generation model corresponding to each personalized parameter is obtained. Among them, by setting the weight of the sample data, more high-quality and effective sample data can be obtained, and the accuracy of the model obtained by training is improved.

[0225] The above process is a training process for a universal three-dimensional digital human model. In the actual implementation process, after obtaining the universal three-dimensional digital human model, corresponding applications can be performed, for example, realizing the generation of a three-dimensional digital human. The implementation process of the three-dimensional digital human generation method provided in the embodiment of the present application is explained below.

[0226] Fig.10 This is a flow chart of the method for generating a three-dimensional digital human provided in the embodiment of the present application. Please refer to Fig.10 , the method comprising:

[0227] S1010: Obtain target personalized parameters.

[0228] It should be noted that the target personalized parameter includes at least one of target identity information and target emotion information.

[0229] Optionally, the personalized parameters may be parameters corresponding to the selected presets after the user selects multiple presets in the client, or may be parameters determined by the user through input. There is no specific limitation here and the parameters may be set according to actual needs.

[0230] The target personalized parameter may be a personalized parameter sent by the client to the server, or a default personalized parameter in the server, which is not specifically limited here and can be set accordingly according to the actual scenario.

[0231] For example: if you need to use the preset expressions on the server to generate the corresponding three-dimensional digital human, you can use the default personalized parameters in the server; if you need to generate three-dimensional digital humans corresponding to different expressions according to the needs of the client, you can use the personalized parameters sent by the receiving client.

[0232] It should be noted that if the personalized parameters sent by the client are used, multiple personalized parameters that can be pre-selected, generated in advance or generated in real time can be set for each client. The personalized parameters corresponding to the three-dimensional digital human actively selected by the user or displayed by default on the client can be used as the above-mentioned target personalized parameters.

[0233] Among them, in the target personalized parameters, the target identity information can refer to the identity of the three-dimensional digital person, such as age, gender, race, etc., and the target emotional information can refer to the emotions that the three-dimensional digital person can express, such as happiness, anger, sadness, etc. There is no specific restriction here, and corresponding settings can be made based on the emotions possessed by humans.

[0234] It should be noted that the target personalized parameter may be a personalized parameter determined by the client and sent to the server after the user performs a selection operation or an input operation through the client.

[0235] In one embodiment, the target personalization parameters may include one or more parameters.

[0236] For example, the target personalization parameters can be represented by Table 2 and Table 3:

[0237] Table 2

[0238] gender age Ethnicity mood male / / /

[0239] Table 2 shows the target personalized parameters with only one parameter, for example: gender male, and other parameters are not limited. Table 2 shows only one example. In the actual implementation process, there may also be only one other parameter such as age, race or emotion.

[0240] Table 3

[0241] gender age Ethnicity mood female 20 Yellow people Happy

[0242] Table 3 shows target personalized parameters with multiple parameters, for example: gender is female, age is 20, race is yellow, and emotion is happy. Table 3 takes four different parameters as an example to form the target personalized parameters. In the actual implementation process, there may also be two, three, five or more parameters, which are not specifically limited here.

[0243] It should be noted that the emotions in Table 2 and Table 3 can be a fixed emotion, a combination of multiple emotions, or a changing emotion, which can be set accordingly according to actual needs, for example: an emotion in which happiness accounts for 0.5 and sadness accounts for 0.5, or a changing emotion of first being sad and then being happy, etc., and the duration of sadness and happiness can be set accordingly, without specific restrictions here, and the emotion parameter can be set accordingly according to actual needs.

[0244] When characterizing the target personalized parameters, each type of parameter (including parameters without specific types, such as age in Table 2) can be represented as target personalized parameters in the manner of Tables 2 and 3 above. For example, the target personalized parameters corresponding to Table 2 can be characterized as: male, unlimited age, unlimited race, and unlimited emotion; or, only parameters with specific types can be characterized as target personalized parameters, for example, the target personalized parameters corresponding to Table 2 can be characterized as: male.

[0245] S1020: Determine target adjustment parameters based on the target personalized parameters and the mapping relationship.

[0246] The mapping relationship may include a one-to-one correspondence between multiple personalized parameters and multiple adjustment parameters.

[0247] The adjustment parameters are used to indicate the deviation of the head parameters between the animated digital human obtained by inputting the same speech into the personalized digital human generation model and the general digital human generation model.

[0248] Among them, the personalized digital human generation model can correspond to at least one personalized parameter. It should be noted that, since the target personalized parameter can be one parameter or multiple parameters, and the target personalized parameters can be different, multiple personalized digital human generation models can be set on the server side, and each target personalized parameter can correspond to a personalized digital human generation model.

[0249] The personalized digital human generation model can be a model used to generate a corresponding three-dimensional digital human according to at least one personalized parameter corresponding to it. For example, a speech can be input into the personalized digital human generation model to obtain a three-dimensional digital human with the characteristics of at least one personalized parameter.

[0250] The universal digital human generation model can be a model used to generate a three-dimensional digital human without personalization. For example, it can be a model used to generate a gender-independent, age-independent, race-independent, and emotion-independent model. Through this model, a universal three-dimensional digital human can be obtained. The universal three-dimensional digital human has no gender preference, no age preference, no race preference, and no emotional preference.

[0251] The same speech can be input into a personalized digital human generation model and a universal digital human generation model to obtain a three-dimensional digital human with at least one personalized parameter feature and a universal three-dimensional digital human, respectively. The head parameters of the three-dimensional digital human with at least one personalized parameter feature and the head parameters of the universal three-dimensional digital human in three-dimensional space can be determined, and the deviation of the two head parameters can be calculated.

[0252] Among them, the head parameters can be the position coordinates of the key points of the head in the three-dimensional space. For example, N points can be selected from the head of a three-dimensional digital human with at least one personalized parameter feature, and N points can be selected from the corresponding positions of the head of a universal three-dimensional digital human. The differences in the positions of these N points in the spatial coordinate systems where the two three-dimensional digital humans are located are compared, so as to obtain the deviation of the head parameters. The deviation of the position of each key point of the head can be used as one of the adjustment parameters. If there are differences in the above N points, there can be N adjustment parameters corresponding to the at least one personalized parameter.

[0253] The mapping relationship may include a one-to-one correspondence between multiple personalized parameters and multiple adjustment parameters. The server may determine the target adjustment parameter corresponding to the target personalized parameter by searching the mapping relationship.

[0254] In the actual implementation process, the target adjustment parameter corresponding to the target personalized parameter can be determined from the mapping relationship through the above mapping relationship and the target personalized parameter.

[0255] S1030: Generate an animation target digital human based on the target adjustment parameters and the animation initial digital human.

[0256] The initial digital human being of the animation is obtained based on the first voice input into the general digital human generation model.

[0257] Optionally, the first voice may be a voice input by a user through a client, or a voice obtained by converting input text into voice using a preset text-to-speech conversion tool, wherein the text-to-speech conversion tool may be run on a client or on a server. If it is run on a client, the client may obtain the text and convert the text into voice and send it to the server; if it is run on a server, the client may send the text to the server, which converts the text into voice.

[0258] After the server obtains the first voice, the first voice can be input into the general digital human generation model to obtain an animated initial digital human.

[0259] After the server determines the initial digital human of the animation, it can adjust the initial digital human of the animation through the target adjustment parameters to obtain the target digital human of the animation. Among them, the target adjustment parameters can record the position deviation of some key points of the head. The head parameters of the initial digital human of the animation can be adjusted according to the position deviation of these key points of the head, that is, the position of the corresponding key points of the head can be adjusted. After all adjustments are completed, the target digital human of the animation can be obtained.

[0260] In a three-dimensional digital human generation method provided in an embodiment of the present application, a target personalized parameter can be obtained, and a target adjustment parameter can be determined based on the target personalized parameter and a mapping relationship, wherein the mapping relationship includes a one-to-one correspondence between multiple personalized parameters and multiple adjustment parameters, and the adjustment parameter is used to indicate the deviation of the head parameters between the animated digital human obtained by inputting the same speech into the personalized digital human generation model and the universal digital human generation model, and then the animated target digital human can be generated based on the target adjustment parameter and the animated initial digital human. In the process of generating a three-dimensional digital human, since the universal digital human generation model is used, a more accurate and stable animated target digital human can be obtained.

[0261] A feasible implementation process of generating an animated target digital human provided in an embodiment of the present application is explained below.

[0262] Optionally, the first speech includes multiple speech frames, the target adjustment parameter includes multiple frame adjustment parameters, and each frame adjustment parameter corresponds to one speech frame.

[0263] Among them, the first voice can be composed of multiple voice frames, the initial digital human of the animation can also be composed of multiple frames, each frame of the initial digital human of the animation can be adjusted by a frame adjustment parameter, and each frame adjustment parameter can correspond to a voice frame, that is, each frame of the initial digital human of the animation can also correspond to a voice frame.

[0264] In one embodiment, generating an animated target digital human based on the target adjustment parameter and the animated initial digital human specifically includes: based on a frame adjustment parameter corresponding to each speech frame, adjusting each frame of the animated initial digital human corresponding to the speech frame to generate the animated target digital human.

[0265] It should be noted that, in the process of adjusting the initial animation digital human, each frame of the initial animation digital human can be adjusted based on a frame adjustment parameter corresponding to each voice frame. For example, the first voice can be divided into 100 voice frames, each voice frame can correspond to a frame adjustment parameter, and the 100 frames of the initial animation digital human can be adjusted based on these 100 frame adjustment parameters. After the adjustment is completed, the target animation digital human can be obtained.

[0266] Among them, the frame adjustment parameters can be used to adjust some head parameters of the initial animation digital human, for example: adjusting the positions of some selected head key points, so as to achieve adjustment of a certain frame. After the initial animation digital human of the corresponding frame is adjusted based on the frame adjustment parameters corresponding to each voice frame, the target animation digital human can be obtained.

[0267] In a method for generating a three-dimensional digital human provided by an embodiment of the present application, each frame of an initial animation digital human corresponding to a speech frame can be adjusted based on a frame adjustment parameter corresponding to the speech frame to generate an animation target digital human. Among them, by adjusting each frame of the initial animation digital human, the adjustment of the three-dimensional digital human can be more comprehensively achieved, and since each frame adjustment parameter corresponds to a speech frame, adjusting each frame of the initial animation digital human based on the frame adjustment parameter corresponding to each speech frame can also obtain a more realistic three-dimensional digital human that better matches the content of the first speech.

[0268] A feasible implementation process of adjusting each frame of the initial digital human animation provided in the embodiment of the present application is explained below.

[0269] In one embodiment, each frame of the digital human includes a plurality of grid units in a three-dimensional space, and each frame adjustment parameter includes: an identifier of a grid unit to be adjusted among the plurality of grid units and an offset value of each grid unit to be adjusted.

[0270] It should be noted that both the initial animation digital human and the target animation digital human may include multiple frames, and each frame may include multiple grid units in the three-dimensional space.

[0271] Among them, the grid unit can be a grid point in a grid structure that constitutes a triangular network or a quadrangular network. These grid structures are spread over the head of the three-dimensional digital human, thereby forming a head skeleton of the three-dimensional digital human. The head parameters of the three-dimensional digital human can be represented on the head skeleton in the form of these grid units. The head parameters may include, for example: head contour parameters, facial parameters, lip parameters, etc.

[0272] Fig.11 This is a schematic diagram of the head grid of the three-dimensional digital human provided in the embodiment of this application. Please refer to Fig.11 , Fig.11 The three-dimensional digital human shown in the figure may be the above-mentioned initial animation digital human, for example, it may be a digital human output by a general digital human generation model, or it may be a digital human with personalized characteristics output by a personalized digital human generation model, and no specific limitation is made here.

[0273] It should be noted that the grid unit to be adjusted can be any one or more grid units selected from the grid units of the three-dimensional digital human's head. There is no specific limitation here and it can be selected according to actual needs. For example, if facial movements need to be adjusted, the grid units of the three-dimensional digital human's face can be selected; if the lip shape needs to be adjusted, the grid units of the three-dimensional digital human's lips can be selected.

[0274] Among them, the identifier of the grid unit to be adjusted can be the label corresponding to the grid unit, for example: it can be represented by a preset digital serial number, letter serial number, etc., and specific restrictions are made here. Each grid unit to be adjusted can be represented by the label.

[0275] The offset value of the grid unit to be adjusted may be used to indicate the offset amount of the grid unit, for example, the amount of change of the three-dimensional coordinates of the grid unit in the xyz axis of the three-dimensional space.

[0276] In one embodiment, based on each frame adjustment parameter, each frame of the animated initial digital human corresponding to the voice frame is adjusted to generate an animated target digital human, specifically including: based on the identifier of the grid unit to be adjusted in each frame adjustment parameter and the offset value of each grid unit to be adjusted, the three-dimensional space coordinates of the grid unit to be adjusted in the animated initial digital human corresponding to the voice frame are adjusted to generate an animated target digital human corresponding to the voice frame.

[0277] It should be noted that during the adjustment process, the three-dimensional space coordinates of the grid unit to be adjusted in the animated initial digital human corresponding to the frame can be adjusted based on the identification of the grid unit to be adjusted in the adjustment parameters of each frame and the offset value of the grid unit to be adjusted.

[0278] For example: In the actual implementation process, the grid unit to be adjusted can be first determined from multiple grid units according to the identifier of the grid unit to be adjusted in the adjustment parameters of each frame, and then the coordinates of the corresponding grid unit to be adjusted in the initial animation digital human can be adjusted based on the offset value of the grid unit to be adjusted. The offset value can indicate the change amount of the x-axis, the change amount of the y-axis and the change amount of the z-axis. The positions of each grid unit to be adjusted of the initial animation digital human can be adjusted based on the adjustment method indicated by the offset value, so as to obtain the animation target digital human corresponding to the voice frame.

[0279] Based on the above method, the initial animation digital human corresponding to each speech frame can be adjusted respectively to obtain the target animation digital human.

[0280] In a three-dimensional digital human generation method provided in an embodiment of the present application, the three-dimensional spatial coordinates of the grid units to be adjusted in the animation initial digital human corresponding to the voice frame can be adjusted based on the identifiers of the grid units to be adjusted in each frame adjustment parameter and the offset values ​​of each grid unit to be adjusted, so as to generate an animation target digital human corresponding to the voice frame. In this method, the grid units to be adjusted and the values ​​to be adjusted of each grid unit to be adjusted can be determined more accurately and quickly according to the identifiers of the grid units to be adjusted and the offset values, so that the adjustment of the grid units can be achieved more quickly and accurately, and a more accurate animation target digital human can be obtained.

[0281] The following is an explanation of the implementation steps that can be performed after obtaining the animation target digital human in the three-dimensional digital human generation method provided in the embodiment of the present application.

[0282] The method also includes: sending the animated target digital human to the client, so that the client renders the animated target digital human.

[0283] It should be noted that after the server adjusts the initial animation digital human in the above manner and obtains the animation target digital human, the animation target digital human can be sent to the client, wherein the client can be a client that provides target personalized parameters to the server, or a client that provides the first voice, and no specific restrictions are made here.

[0284] After the client obtains the animation target digital human, it can render the animation target digital human.

[0285] Optionally, rendering may include rendering in a variety of different ways, for example, the skeleton and texture files sent by the server may be received in the form of a data stream, and then rendering may be performed according to the configuration file of the digital human, wherein the configuration file of the digital human indicates the coordinates of the skeleton of the digital human in three-dimensional space, and may indicate the specific positions of each texture on the skeleton of the digital human, and may also indicate the type of texture to be used to render the digital human.

[0286] In addition, during the rendering process, you can also use lighting and shadows to simulate real effects, and use the rendering engine to generate a three-dimensional digital human displayed on the client's monitor, which is the rendered three-dimensional digital human.

[0287] In the three-dimensional digital human generation method provided in the embodiment of the present application, the animated target digital human can be sent to the client so that the client can render the animated target digital human. By transmitting the target animated digital human, the client can more accurately and quickly render the animated target digital human, and then the client can display the rendered three-dimensional digital human.

[0288] The following is an explanation of the implementation process of sending the animated target digital human to the client in the three-dimensional digital human generation method provided in the embodiment of the present application.

[0289] Sending the animated target digital human to the client specifically includes: sending the target digital human and the target deviation parameter corresponding to the first speech frame in the first speech to the client.

[0290] The target deviation parameter is at least part of a plurality of deviation parameters; each deviation parameter includes a deviation of head parameters of the animated target digital human corresponding to adjacent speech frames in the animated target digital human.

[0291] It should be noted that the target digital human can be the digital human corresponding to the first voice frame in the first voice, the target digital human can be the first frame in the animated target digital human, or it can be another frame. There is no specific limitation here, and it can be set according to the correspondence between the frame of the animated target digital human and the voice frame of the first voice.

[0292] Optionally, the deviation parameter may be the deviation of the head parameters of the animated target digital human in two adjacent frames corresponding to every two adjacent speech frames in the animated target digital human. The target deviation parameter may be a part of the head deviation parameters, for example, if the deviation parameter includes the position deviation of 100 key points of the head, the target deviation parameter may be the position deviation of 50 key points of the head.

[0293] For example, if the first voice includes 200 frames, there may be a deviation of the head parameters of a corresponding animated target digital human in each two adjacent frames, that is, there are deviations of the head parameters of 199 animated target digital humans. The target digital human corresponding to the first voice frame and the remaining 199 target deviation parameters may be sent to the client.

[0294] It should be noted that the deviation parameters can be recorded in the form of position differences of key points of the head. For example, if there is a position difference of 20 grid units between two adjacent animated target digital humans, the deviation parameters can record the numbers of these 20 grid units and the deviation values ​​of these 20 grid units.

[0295] Optionally, based on the target digital human, the client can start from the second voice frame and render each voice frame according to the target deviation parameters of the animated target digital human corresponding to the voice frame and the next voice frame.

[0296] In the method for generating a three-dimensional digital human provided in the embodiment of the present application, the target digital human and the target deviation parameter corresponding to the first voice frame in the first voice can be sent to the client. The target deviation parameter is at least part of a plurality of deviation parameters; each deviation parameter includes the deviation of the head parameters of the animated target digital human corresponding to adjacent voice frames in the animated target digital human. Among them, by determining the target digital human and the target deviation parameter, the client can more accurately render the three-dimensional digital human.

[0297] Optionally, the deviation parameters of different frames may not be sent to the client at one time, but new deviation parameters may be obtained in real time and sent to the client in sequence according to the order corresponding to the voice frames.

[0298] In one embodiment, each deviation parameter includes: an identifier of a grid unit to be adjusted among multiple grid units of a target digital person corresponding to adjacent speech frames and an offset value of each grid unit to be adjusted; sending the target digital person and the target deviation parameter corresponding to the first speech frame in the first speech to the client, including: sending multiple target deviation parameters to the client in sequence according to the time sequence of each speech frame in the first speech.

[0299] Optionally, the target deviation parameters in the deviation parameters corresponding to each voice frame may be sent to the client in sequence according to the order of the voice frames in the first voice. The deviation parameters corresponding to each voice frame may be the deviation of the head parameters of the animated target digital human corresponding to the voice frame and the next frame of the voice frame.

[0300] Correspondingly, the client can also obtain the corresponding target deviation parameters in sequence, and then perform rendering.

[0301] For the client, in each rendering process, the position of the corresponding grid unit in the three-dimensional digital human rendered in the previous frame can be adjusted according to the position difference of the grid unit in the target deviation parameter.

[0302] In the method for generating a three-dimensional digital human provided in the embodiment of the present application, multiple target deviation parameters can be sent to the client in sequence according to the time sequence of each voice frame in the first voice. Among them, the transmission of multiple target deviation parameters according to the order of the voice frames in the first voice can more accurately realize the rendering of the three-dimensional digital human by the client, thereby improving the real-time performance and accuracy of the rendering of the three-dimensional digital human by the client.

[0303] In the process of sending the deviation parameters, all the deviation parameters can be sent to the client, or only a part of the data in the deviation parameters can be sent to the client. For example, if the deviation parameters record the difference values ​​of 20 grid units, among which the difference of 5 grid units has a lower impact on the overall three-dimensional digital human, then the difference of the remaining 15 grid units can be sent to the client as the target deviation parameter, and the difference of these 5 grid units will not be transmitted accordingly.

[0304] In one embodiment, sending a target deviation parameter to a client includes: determining the target deviation parameter from a plurality of deviation parameters according to preset conditions, the preset conditions including at least one of bandwidth configuration information of the client, display requirements of the client, and weights of facial areas corresponding to the deviation parameters; and sending the target deviation parameter to the client.

[0305] It should be noted that bandwidth configuration information refers to the size of the transmission bandwidth between the client and the server, which can be used to characterize the amount of information that can be transmitted during a communication process. If the bandwidth is large, more information can be transmitted, and if the bandwidth is small, less information can be transmitted. In this case, if the bandwidth is small, the difference of the grid cells in the deviation parameter that have a lower impact on the three-dimensional digital human can be deleted and not transmitted, that is, only the difference of some of the grid cells is transmitted.

[0306] Among them, the grid units with lower influence may be grid units at certain positions, such as grid units at the forehead and ears of the digital human. The specific grid units with lower influence may be actively set by the user, and no specific limitation is made here.

[0307] The display requirement of the client may be a personalized configuration on the client. For example, due to reasons such as resolution, the number of grid units that can be displayed by the three-dimensional digital human is reduced. In this case, only some of the grid units can be displayed on the client. In this case, only the differences in the grid units that can be displayed on the client may be transmitted. For the grid units that cannot be displayed, the difference values ​​will not be transmitted.

[0308] The weight of the facial area corresponding to the deviation parameter may refer to the weight of the facial area position where the head key point corresponding to the deviation parameter is located. For example, the weights of the cheek and mouth positions in the facial area of ​​the three-dimensional digital human may be set to be higher, while the weights of other positions may be lower. If the head key point corresponding to a certain deviation parameter is at a position with a higher weight, it may be determined that the deviation parameter is transmitted; if the head key point corresponding to a certain deviation parameter is at a position with a lower weight, it may be determined that the deviation parameter is not transmitted. It should be noted that the weight may be set according to actual needs and is not a fixed weight.

[0309] In the actual transmission process, the target deviation parameter can be determined from the deviation parameter according to any one or more of the above three conditions, and the determined target deviation parameter can be transmitted to the client.

[0310] In the method for generating a three-dimensional digital human provided in the embodiment of the present application, a target deviation parameter can be determined from a plurality of deviation parameters according to a preset condition, and the target deviation parameter can be sent to the client. The preset condition includes at least one of the bandwidth configuration information of the client, the display requirements of the client, and the weight of the facial area corresponding to the deviation parameter. The number of grid cells with differences in the deviation parameters can be reduced by determining the target deviation parameter from the deviation parameters, thereby reducing the amount of information transmitted between the client and the server, reducing the amount of computation of the client in the rendering process, and improving the rendering efficiency.

[0311] In order to more clearly demonstrate the process of rendering a three-dimensional digital human using the deviation parameters provided in the embodiment of the present application, the rendering process is explained below through the changes of two frames of three-dimensional digital humans.

[0312] Fig.12 This is a schematic diagram of the interface changes of rendering a three-dimensional digital human based on deviation parameters provided in the embodiment of the present application. Please refer to Fig.12 , Fig.12 The display interface on the left is the display interface corresponding to the Nth frame of the three-dimensional digital human, and the display interface on the right is the display interface corresponding to the N+1th frame of the three-dimensional digital human. N can be a positive integer greater than or equal to 1.

[0313] according to Fig.12 It can be seen from the three-dimensional digital human shown in that there is a certain difference between the movements of the three-dimensional digital human on the left and the movements of the three-dimensional digital human on the right. This difference can be obtained by rendering the three-dimensional digital human corresponding to the Nth frame according to the target deviation parameter corresponding to the N+1th frame.

[0314] For example, if N is 1, then Fig.12The three-dimensional digital human in the left display interface is the target digital human corresponding to the first voice frame transmitted by the server, and the three-dimensional digital human in the right interface is the three-dimensional digital human obtained by rendering the target digital human according to the target deviation parameter of the animated target digital human corresponding to the first voice frame and the second voice frame.

[0315] For example, Fig.12 The emotion of the three-dimensional digital person on the left side of the middle is sadness, and the emotion of the three-dimensional digital person on the right side is fear.

[0316] It should be noted that in addition to executing the above rendering process, the client can also pre-load the skeleton, textures and configuration files of the three-dimensional digital human during the initialization process, so that it can render more quickly after receiving the three-dimensional digital human to be displayed sent by the server.

[0317] It should be understood that, although the steps in the above-mentioned flowcharts are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flowcharts may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0318] Based on the foregoing embodiments, the embodiments of the present application provide a three-dimensional digital human rendering device. The modules included in the device and the units included in each module can be implemented by a processor; of course, they can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0319] Fig.13 This is a structural diagram of the training device for the three-dimensional digital human model provided in the embodiment of the present application. Please refer to Fig.13 , a training device for a three-dimensional digital human model, comprising: a video acquisition module 1310, a data acquisition module 1320, a training module 1330 and a normalization module 1340;

[0320] The video acquisition module 1310 is used to acquire sample videos corresponding to different personalized parameters, where the personalized parameters include: at least one of identity information and emotion information;

[0321] The data acquisition module 1320 is used to acquire corresponding sample data from the sample videos corresponding to the individualized parameters, wherein the sample data includes sample voice, sample user's emotions and sample user's head parameters, wherein the sample user's head parameters are obtained by calibrating the points to be calibrated on the initial digital human according to the target head key points in the calibration digital human established by the sample video;

[0322] The training module 1330 is used to train the audio-visual generation model according to the sample data corresponding to each personalized parameter, and obtain the personalized digital human generation model corresponding to each personalized parameter;

[0323] The normalization module 1340 is used to perform normalization processing on different personalized digital human generation models to obtain a universal digital human generation model.

[0324] In one embodiment, the data acquisition module 1320 is specifically used to establish a corresponding training digital human based on the sample video, and determine the head parameters of the sample user based on the head parameters of the training digital human; identify the emotions of the sample user through the facial parameters of the training digital human; determine the sample speech corresponding to each personalized parameter, as well as the sample user's emotions and the head parameters of the sample user corresponding to each speech frame of the sample speech.

[0325] In one embodiment, the sample video includes multiple video frames, and the data acquisition module 1320 is specifically used to establish a calibrated digital human through multiple video frames in the sample video, and the calibrated digital human includes multiple frames; the initial digital human is calibrated according to the target head key points of the calibrated digital human to obtain a trained digital human.

[0326] In one embodiment, the target head key point includes a first head key point and a second head key point, wherein the second head key point is determined according to the optical flow information in the sample video, and the data acquisition module 1320 is specifically used to perform a first calibration on the first point to be calibrated on the initial digital human based on the first loss function and the position of the first head key point in the calibrated digital human, to obtain a first calibrated digital human; wherein the first head key point has a corresponding relationship with the first point to be calibrated on the initial digital human; based on the second loss function and the position of the second head key point in the calibrated digital human, perform a second calibration on the second point to be calibrated on the first calibrated digital human, to obtain a training digital human; wherein the second head key point has a corresponding relationship with the second point to be calibrated on the first calibrated digital human.

[0327] In one embodiment, in the device, the first calibrated digital human is a digital human obtained when the first loss function is minimized during the first calibration of the first point to be calibrated on the initial digital human; the training digital human is a digital human obtained when the second loss function is minimized during the second calibration of the second point to be calibrated on the digital human after the first calibration.

[0328] In one embodiment, the data acquisition module 1320 is specifically used to obtain the light density and color in the sample video, and determine the head area of ​​the sample user based on the light density and color; based on the brightness of each pixel point in the head area and / or the contrast with the adjacent pixel points, determine a second head key point from the head area including multiple pixels, and the brightness of the second head key point and / or the contrast with the adjacent pixel points is greater than a preset threshold.

[0329] In one embodiment, the data acquisition module 1320 is specifically used to calculate the similarity between the facial parameters of the training digital human and the facial parameters of each pre-set emotional avatar model; based on the similarity, determine the emotional label of the training digital human, the emotional label includes the emotion and weight of the target emotional avatar model, the target emotional avatar model is the emotional avatar model corresponding to the similarity greater than or equal to the threshold, and the emotion of the sample user includes the emotional label of the training digital human.

[0330] In one embodiment, the training module 1330 is also used to determine the weight of the sample data based on the degree of matching between the head parameters of the sample user in the sample video and the sample voice corresponding to the sample video, and the weight of the sample data is positively correlated with the degree of matching; based on the sample data whose weights corresponding to each personalized parameter are greater than a preset weight threshold, the audio and video generation model is trained to obtain a personalized digital human generation model corresponding to each personalized parameter.

[0331] Fig.14 This is a schematic diagram of the structure of the three-dimensional digital human generation device provided in the embodiment of the present application. Please refer to Fig.14 , a three-dimensional digital human generation device, the device comprising: an acquisition module 1410, a determination module 1420 and a generation module 1430;

[0332] The acquisition module 1410 is used to acquire target personalized parameters; the target personalized parameters include at least one of target identity information and target emotion information;

[0333] The determination module 1420 is used to determine the target adjustment parameter based on the target personalized parameter and the mapping relationship; the mapping relationship includes a one-to-one correspondence between multiple personalized parameters and multiple adjustment parameters, and the adjustment parameter is used to indicate the deviation of the head parameters between the animated digital human obtained by inputting the same speech into the personalized digital human generation model and the universal digital human generation model;

[0334] The generation module 1430 is used to generate an animation target digital human based on the target adjustment parameters and the animation initial digital human, where the animation initial digital human is generated based on the first voice input to the universal digital human generation model.

[0335] In one embodiment, the first speech includes multiple speech frames, the target adjustment parameters include multiple frame adjustment parameters, and each frame adjustment parameter corresponds to a speech frame; the generation module 1430 is specifically used to adjust each frame of the animated initial digital human corresponding to the speech frame based on a frame adjustment parameter corresponding to each speech frame, so as to generate an animated target digital human.

[0336] In one embodiment, each frame of the digital human includes multiple grid units in a three-dimensional space, and each frame adjustment parameter includes: an identifier of a grid unit to be adjusted among the multiple grid units and an offset value of each grid unit to be adjusted; a generation module 1430 is specifically used to adjust the three-dimensional space coordinates of the grid unit to be adjusted in the animated initial digital human corresponding to the voice frame based on the identifier of the grid unit to be adjusted and the offset value of each grid unit to be adjusted in each frame adjustment parameter, so as to generate an animated target digital human corresponding to the voice frame.

[0337] In one embodiment, the generation module 1430 is further used to send the animated target digital human to the client, so that the client renders the animated target digital human.

[0338] In one embodiment, the generation module 1430 is specifically used to send the target digital human and target deviation parameters corresponding to the first speech frame in the first speech to the client; wherein the target deviation parameters are at least part of multiple deviation parameters; each deviation parameter includes the deviation of the head parameters of the animated target digital human corresponding to adjacent speech frames in the animated target digital human.

[0339] In one embodiment, each deviation parameter includes: an identifier of a grid unit to be adjusted among multiple grid units of a target digital person corresponding to adjacent speech frames and an offset value of each grid unit to be adjusted; a generation module 1430 is specifically used to send multiple target deviation parameters to the client in sequence according to the time sequence of each speech frame in the first speech.

[0340] In one embodiment, the determination module 1420 is further used to determine a target deviation parameter from a plurality of deviation parameters according to preset conditions, wherein the preset conditions include at least one of the bandwidth configuration information of the client, the display requirements of the client, and the weight of the facial area corresponding to the deviation parameter; the generation module 1430 is further used to send the target deviation parameter to the client.

[0341] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0342] It should be noted that in the embodiments of this application Fig.13 as well as Fig.14 The division of modules in the training device for the three-dimensional digital human model and the three-dimensional digital human generation device shown is schematic and is only a logical functional division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. It may also be implemented in the form of a combination of software and hardware.

[0343] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0344] Fig.15 For a schematic diagram of the structure of the computing device provided in the embodiment of the present application, please refer to Fig.15 , the present application embodiment provides a computing device, which may be the above-mentioned server, and is not specifically limited here. Its internal structure diagram may be as follows Fig.15As shown. The computing device includes a processor 1520, a memory and a network interface 1540 connected via a system bus 1510. Among them, the processor 1520 of the computing device is used to provide computing and control capabilities. The memory of the computing device includes a non-volatile storage medium 1531 and an internal memory 1532. The non-volatile storage medium 1531 stores an operating system, a computer program and a database. The internal memory 1532 provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium 1531. The database of the computing device is used to store data. The network interface 1540 of the computing device is used to communicate with an external terminal through a network connection. For example, communication between a client and a server can be achieved through the network interface 1540. When the computer program is executed by the processor 1520, the above method is implemented.

[0345] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the above embodiment are implemented.

[0346] An embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps of the method provided in the above method embodiment.

[0347] Those skilled in the art will understand that Fig.15 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the computing device to which the scheme of the present application is applied. The specific computing device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0348] In one embodiment, the training device for the three-dimensional digital human model and the generating device for the three-dimensional digital human provided in the present application can be implemented in the form of a computer program. The computer program can be used in the following manner: Fig.15 The computer program composed of each program module enables the processor to execute the steps of the method of each embodiment of the present application described in this specification.

[0349] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0350] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. For the sake of brevity, this article will not repeat them.

[0351] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0352] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0353] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0354] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0355] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0356] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0357] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0358] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0359] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0360] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0361] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A training method for a three-dimensional digital human model, characterized in that: The method comprises: Obtaining sample videos corresponding to different personalized parameters, where the personalized parameters include: at least one of identity information and emotion information; Acquire corresponding sample data from the sample videos corresponding to the individualized parameters, the sample data including sample voice, sample user's emotions and sample user's head parameters, wherein the sample user's head parameters are obtained by calibrating the points to be calibrated on the initial digital human according to the target head key points in the calibration digital human established by the sample videos; According to the sample data corresponding to each personalized parameter, the audio and video generation model is trained to obtain the personalized digital human generation model corresponding to each personalized parameter; The different personalized digital human generation models are normalized to obtain a universal digital human generation model.

2. The method according to claim 1, characterized in that The obtaining corresponding sample data from the sample videos corresponding to the individualized parameters includes: Establishing a corresponding training digital human according to the sample video, and determining the head parameters of the sample user according to the head parameters of the training digital human; Recognizing the emotion of the sample user through the facial parameters of the trained digital human; The sample speech corresponding to each personalized parameter, as well as the sample user's emotion and the sample user's head parameter corresponding to each speech frame of the sample speech are determined.

3. The method according to claim 2, characterized in that The sample video includes a plurality of video frames, and the step of establishing a corresponding training digital human according to the sample video includes: Creating a calibrated digital human through a plurality of video frames in the sample video, wherein the calibrated digital human comprises a plurality of frames; The initial digital human is calibrated according to the target head key points of the calibrated digital human to obtain the training digital human.

4. The method according to claim 3, characterized in that The target head key points include a first head key point and a second head key point, wherein the second head key point is determined according to the optical flow information in the sample video, and the initial digital human is calibrated according to the target head key points of the calibration digital human to obtain the training digital human, including: Based on the first loss function and the position of the first head key point in the calibrated digital human, a first calibration is performed on the first point to be calibrated on the initial digital human to obtain a first calibrated digital human; wherein the first head key point has a corresponding relationship with the first point to be calibrated on the initial digital human; Based on the second loss function and the position of the second head key point in the calibrated digital human, a second calibration is performed on the second point to be calibrated on the digital human after the first calibration to obtain the training digital human; wherein the second head key point has a corresponding relationship with the second point to be calibrated on the digital human after the first calibration.

5. The method according to claim 4, characterized in that The first calibrated digital human is a digital human obtained when the first loss function is minimized during the first calibration of the first to-be-calibrated point on the initial digital human; The training digital human is a digital human obtained when the second loss function is minimized during the second calibration of the second to-be-calibrated point on the digital human after the first calibration.

6. The method according to claim 4, characterized in that The determining the second head key point according to the optical flow information in the sample video includes: Acquire the light density and color in the sample video, and determine the head area of ​​the sample user according to the light density and color; According to the brightness of each pixel in the head area and / or the contrast with adjacent pixels, the second head key point is determined from the multiple pixels in the head area, and the brightness of the second head key point and / or the contrast with adjacent pixels is greater than a preset threshold.

7. The method according to claim 2, characterized in that The step of identifying the emotion of the sample user by using the facial parameters of the trained digital human comprises: Calculating the similarity between the facial parameters of the training digital human and the facial parameters of each preset emotional avatar model; According to the similarity, the emotion label of the training digital human is determined, the emotion label includes the emotion and weight of the target emotion avatar model, the target emotion avatar model is the emotion avatar model corresponding to the similarity greater than or equal to the threshold, and the emotion of the sample user includes the emotion label of the training digital human.

8. The method according to claim 1, characterized in that The method further comprises: Determining the weight of the sample data according to the matching degree between the head parameters of the sample user in the sample video and the sample voice corresponding to the sample video, wherein the weight of the sample data is positively correlated with the matching degree; The method of training the audio-visual generation model according to the sample data corresponding to each personalized parameter to obtain the personalized digital human generation model corresponding to each personalized parameter includes: The audio-visual generation model is trained according to sample data whose weights corresponding to various personalized parameters are greater than a preset weight threshold, so as to obtain a personalized digital human generation model corresponding to various personalized parameters.

9. A method for generating a three-dimensional digital human, characterized in that: The method comprises: Obtaining target personalized parameters; the target personalized parameters include at least one of target identity information and target emotion information; Determine a target adjustment parameter based on the target personalized parameter and the mapping relationship; the mapping relationship includes a one-to-one correspondence between a plurality of personalized parameters and a plurality of adjustment parameters, the adjustment parameter being used to indicate a deviation in head parameters between an animated digital human obtained by inputting the same speech segment into a personalized digital human generation model and a universal digital human generation model as claimed in any one of claims 1 to 8; Based on the target adjustment parameters and the animation initial digital human, an animation target digital human is generated, wherein the animation initial digital human is generated based on a first voice input to the universal digital human generation model.

10. A computing device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the method according to claim 1 to 8 or claim 9 is implemented.

Images