Three-dimensional digital human generation method and three-dimensional digital human model training method

Abstract

The embodiment of the invention discloses a three-dimensional digital human generation method and a three-dimensional digital human model training method. The method comprises the steps that target personalized parameters are acquired; the target personalized parameter comprises at least one of target identity information and target emotion information; determining a target adjustment parameter based on the target personalized parameter and a mapping relationship; the mapping relationship comprises a one-to-one correspondence relationship between a plurality of personalized parameters and a plurality of adjustment parameters; and generating an animation target digital person based on the target adjustment parameter and an animation initial digital person, the animation initial digital person being generated based on inputting a first voice to the universal digital person generation model. Different digital people can be rendered, and the diversity of rendering results can be increased.

Description

Technical Field

[0001] The embodiments of the present application relate to image processing technology, and relate to a method for generating a three-dimensional digital human and a method for training a three-dimensional digital human model. Background Art

[0002] A three-dimensional digital human is a virtual image that can be displayed in a client interface. Voice input can be used to make the three-dimensional digital human perform corresponding actions, allowing the audience to visually perceive that the voice is emitted by the three-dimensional digital human.

[0003] In the related art, the rendering method used can usually only render a specific digital human, resulting in the digital humans rendered by different users being the same. It can be seen that the rendering results of the digital human are relatively simple and the user experience is not good. Summary of the invention

[0004] The embodiment of the present application provides a method for generating a three-dimensional digital human and a method for training a three-dimensional digital human model, which can render different digital humans and increase the diversity of rendering results. The method for generating a three-dimensional digital human and the method for training a three-dimensional digital human model provided in the embodiment of the present application are implemented as follows:

[0005] In one aspect of an embodiment of the present application, a method for generating a three-dimensional digital human is provided, the method comprising:

[0006] Obtaining target personalized parameters; the target personalized parameters include at least one of target identity information and target emotion information;

[0007] Determine the target adjustment parameter based on the target personalized parameter and the mapping relationship; the mapping relationship includes a one-to-one correspondence between multiple personalized parameters and multiple adjustment parameters, and the adjustment parameter is used to indicate the deviation of the head parameters between the animated digital human obtained by inputting the same speech into the personalized digital human generation model and the general digital human generation model;

[0008] Based on the target adjustment parameters and the animation initial digital human, an animation target digital human is generated, where the animation initial digital human is generated based on the first voice input to the universal digital human generation model.

[0009] In a possible implementation, the first speech includes multiple speech frames, the target adjustment parameters include multiple frame adjustment parameters, and each frame adjustment parameter corresponds to a speech frame; based on the target adjustment parameters and the initial animation digital human, an animation target digital human is generated, specifically including: based on a frame adjustment parameter corresponding to each speech frame, each frame of the initial animation digital human corresponding to the speech frame is adjusted to generate an animation target digital human.

[0010] In this solution, by adjusting each frame of the initial digital human of the animation, the adjustment of the three-dimensional digital human can be achieved more comprehensively. Moreover, since each frame adjustment parameter corresponds to a voice frame, adjusting each frame of the initial digital human of the animation based on the frame adjustment parameter corresponding to each voice frame can also obtain a more realistic three-dimensional digital human that better matches the content of the first voice.

[0011] In a possible implementation, each frame of the digital human includes multiple grid units in a three-dimensional space, and each frame adjustment parameter includes: an identifier of a grid unit to be adjusted among the multiple grid units and an offset value of each grid unit to be adjusted; based on each frame adjustment parameter, each frame of the animated initial digital human corresponding to the voice frame is adjusted to generate an animated target digital human, specifically including: based on the identifier of the grid unit to be adjusted in each frame adjustment parameter and the offset value of each grid unit to be adjusted, the three-dimensional space coordinates of the grid unit to be adjusted in the animated initial digital human corresponding to the voice frame are adjusted to generate an animated target digital human corresponding to the voice frame.

[0012] In this solution, by using the identification and offset value of the grid unit to be adjusted, the grid units that need to be adjusted and the values ​​that need to be adjusted for each grid unit that needs to be adjusted can be determined more accurately and quickly, so that the adjustment of the grid units can be achieved more quickly and accurately, and a more accurate animated target digital human can be obtained.

[0013] In a possible implementation, the method further includes: sending the animated target digital human to a client, so that the client renders the animated target digital human.

[0014] In this solution, by transmitting the target animated digital human, the client can render the animated target digital human more accurately and quickly, and then the client can display the rendered three-dimensional digital human.

[0015] In a possible implementation, sending the animated target digital human to the client specifically includes: sending the target digital human and target deviation parameters corresponding to the first speech frame in the first speech to the client; wherein the target deviation parameters are at least part of multiple deviation parameters; each deviation parameter includes the deviation of the head parameters of the animated target digital human corresponding to adjacent speech frames in the animated target digital human.

[0016] In this solution, by determining the target digital human and the target deviation parameters, the client can more accurately render the three-dimensional digital human.

[0017] In a possible implementation, each deviation parameter includes: an identifier of a grid unit to be adjusted among multiple grid units of a target digital person corresponding to adjacent speech frames and an offset value of each grid unit to be adjusted; sending the target digital person and the target deviation parameter corresponding to the first speech frame in the first speech to the client, including: sending multiple target deviation parameters to the client in sequence according to the time sequence of each speech frame in the first speech.

[0018] In this solution, the transmission of multiple target deviation parameters is implemented in the order of the voice frames in the first voice, which can more accurately realize the rendering of the three-dimensional digital human by the client, thereby improving the real-time performance and accuracy of the three-dimensional digital human rendering by the client.

[0019] In a possible implementation, sending the target deviation parameter to the client includes: determining the target deviation parameter from a plurality of deviation parameters according to preset conditions, the preset conditions including bandwidth configuration information of the client, display requirements of the client, and at least one of weights of facial areas corresponding to the deviation parameters; and sending the target deviation parameter to the client.

[0020] In this scheme, the number of grid cells with differences in deviation parameters can be reduced by determining the target deviation parameters from the deviation parameters, thereby reducing the amount of information transmitted between the client and the server, reducing the amount of computation on the client during the rendering process, and improving rendering efficiency.

[0021] In another aspect of the embodiment of the present application, a training method for a three-dimensional digital human model is provided, the method comprising:

[0022] Obtain sample videos corresponding to different personalized parameters;

[0023] Obtaining corresponding sample data from sample videos corresponding to each personalized parameter, the sample data including sample voice, sample user's emotion, and sample user's head parameters;

[0024] According to the sample data corresponding to each personalized parameter, the audio and video generation model is trained to obtain the personalized digital human generation model corresponding to each personalized parameter;

[0025] Different personalized digital human generation models are normalized to obtain a universal digital human generation model.

[0026] In this solution, multiple personalized digital human generation models are normalized, so that the obtained universal digital human generation model has high versatility and applicability, and the generation of corresponding three-dimensional digital humans can be achieved for different personalized parameters.

[0027] In a possible implementation, corresponding sample data is obtained from sample videos corresponding to each personalized parameter, including: establishing a corresponding training digital human according to the sample video, and determining the head parameters of the sample user according to the head parameters of the training digital human; identifying the emotions of the sample user through the facial parameters of the training digital human; determining the sample speech corresponding to each personalized parameter, as well as the emotion of the sample user and the head parameters of the sample user corresponding to each speech frame of the sample speech.

[0028] In this solution, by establishing a method of training digital humans, the head parameters and emotions of sample users can be obtained more quickly and accurately, thereby obtaining sample data more quickly and accurately.

[0029] In a possible implementation, the emotions of the sample user are identified by training the facial parameters of the digital person, including: calculating the similarity between the facial parameters of the training digital person and the facial parameters of each pre-set emotional avatar model; determining the emotional label of the training digital person based on the similarity, the emotional label includes the emotion and weight of the target emotional avatar model, the target emotional avatar model is the emotional avatar model corresponding to the similarity greater than or equal to a threshold, and the emotion of the sample user includes the emotional label of the training digital person.

[0030] In a possible implementation, the sample video includes multiple video frames, the training digital human includes multiple frames, each frame of the training digital human corresponds one-to-one to a video frame, and the sample user's emotion includes an emotion label of the training digital human in each frame.

[0031] In this scheme, by determining the emotion label, the emotion of each training digital person can be determined in more detail. In addition to the emotion, the weight corresponding to the emotion can also be determined, so that more accurate and detailed emotion labels can be obtained.

[0032] In a possible implementation, after normalizing different personalized digital human generation models to obtain a universal digital human generation model, the method further includes: inputting each of P segments of sample speech into the personalized digital human generation model corresponding to M personalized parameters and the universal digital human generation model, to obtain P×M personalized animated digital humans and one universal animated digital human; wherein, one segment of speech corresponds to M personalized animated digital humans, and P and M are both positive integers greater than or equal to 2; respectively determining the deviations between the head parameters of each frame of each personalized animated digital human in the M personalized animated digital humans corresponding to each segment of speech in the P segments of speech and the head parameters of each frame of the universal animated digital human; performing weighted averaging on the deviations between the head parameters corresponding to each segment of speech to obtain adjustment parameters, and establishing a mapping relationship between the adjustment parameters and the personalized parameters.

[0033] In this solution, by establishing a mapping relationship, the adjustment parameters can be determined more quickly in the process of generating a three-dimensional digital human, so that the animation target digital human can be obtained more quickly.

[0034] In another aspect of an embodiment of the present application, a three-dimensional digital human generation device is provided, the device comprising: an acquisition module, a determination module and a generation module;

[0035] An acquisition module, used to acquire target personalized parameters; the target personalized parameters include at least one of target identity information and target emotion information;

[0036] A determination module, used to determine a target adjustment parameter based on a target personalized parameter and a mapping relationship; the mapping relationship includes a one-to-one correspondence between a plurality of personalized parameters and a plurality of adjustment parameters, and the adjustment parameter is used to indicate a deviation in head parameters between an animated digital human obtained by inputting the same speech segment into a personalized digital human generation model and a general digital human generation model;

[0037] The generation module is used to generate an animation target digital human based on the target adjustment parameters and the animation initial digital human, where the animation initial digital human is generated based on the first voice input to the universal digital human generation model.

[0038] In another aspect of the embodiment of the present application, a training device for a three-dimensional digital human model is provided, the device comprising: a video acquisition module, a data acquisition module, a training module and a normalization module;

[0039] A video acquisition module, used to acquire sample videos corresponding to different personalized parameters;

[0040] A data acquisition module, used to acquire corresponding sample data from sample videos corresponding to each personalized parameter, the sample data including sample voice, sample user's emotions and sample user's head parameters;

[0041] A training module, used to train the audio-visual generation model according to the sample data corresponding to each personalized parameter, and obtain a personalized digital human generation model corresponding to each personalized parameter;

[0042] The normalization module is used to normalize different personalized digital human generation models to obtain a universal digital human generation model.

[0043] The computing device provided in the embodiment of the present application includes a memory and a processor. The memory stores a computer program that can be run on the processor. When the processor executes the program, the method of the embodiment of the present application is implemented.

[0044] The computer-readable storage medium provided in the embodiment of the present application has a computer program stored thereon, and when the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0045] In the three-dimensional digital human generation method and the three-dimensional digital human model training method provided in the embodiment of the present application, the target personalized parameters can be obtained, and the target adjustment parameters can be determined based on the target personalized parameters and the mapping relationship, wherein the mapping relationship includes a one-to-one correspondence between multiple personalized parameters and multiple adjustment parameters, and the adjustment parameters are used to indicate the deviation of the head parameters between the animated digital human obtained by inputting the same speech into the personalized digital human generation model and the general digital human generation model, and then the animated target digital human can be generated based on the target adjustment parameters and the animated initial digital human. Among them, since there is a mapping relationship between the target adjustment parameters and the personalized parameters, after adjusting the animated initial digital human according to the target adjustment parameters, an animated target digital human with personalized characteristics can be obtained, that is, different animated target digital humans can be generated by changing the target personalized parameters, which can further improve the diversity of the animated target digital humans generated by the server. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 This is a schematic diagram showing an interface of a method for generating a three-dimensional digital human provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of an application scenario of the method for generating a three-dimensional digital human provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of the process of generating a three-dimensional digital human provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of a head grid of a three-dimensional digital human provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of interface changes for rendering a three-dimensional digital human based on deviation parameters provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of a flow chart of a training method for a three-dimensional digital human model provided in an embodiment of the present application;

[0053] Figure 7 A schematic diagram of a process for determining sample data provided in an embodiment of the present application;

[0054] Figure 8 A schematic diagram of a process for identifying sample user emotions provided in an embodiment of the present application;

[0055] Fig. 9 A schematic diagram of an emotional avatar model provided in an embodiment of the present application;

[0056] Fig.10 A schematic diagram of a process for establishing a mapping relationship provided in an embodiment of the present application;

[0057] Fig.11 A schematic diagram of the head deviation between the personalized animated digital human and the general digital human provided in the embodiment of the present application;

[0058] Fig.12 A corresponding relationship diagram between the initial animation digital human and the target animation digital human provided in the embodiment of the present application;

[0059] Fig.13 A schematic diagram of the interaction between the client and the server provided in an embodiment of the present application;

[0060] Fig.14 A schematic diagram of the structure of a three-dimensional digital human generation device provided in an embodiment of the present application;

[0061] Fig.15 A schematic diagram of the structure of a training device for a three-dimensional digital human model provided in an embodiment of the present application;

[0062] Fig.16 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0065] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0066] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are used to distinguish similar or different objects, and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0067] The following is an explanation of the three-dimensional digital human provided in the embodiments of the present application and its practical application scenarios.

[0068] First of all, the three-dimensional digital human can be a virtual image displayed on the client, such as a virtual anchor, a virtual teacher, etc.

[0069] Optionally, the three-dimensional digital human is a virtual character image developed based on computer technology and artificial intelligence technology, with the characteristics of high-quality character image and exquisite visual expression. During use, the user can input sound or text, and make the three-dimensional digital human make corresponding actions, expressions, etc. according to the sound or the sound converted from text, so that the audience mistakenly believes that the sound is made by the three-dimensional digital human.

[0070] This technology can be used in different fields to meet corresponding needs. For example, three-dimensional digital humans can be used to simulate real anchors to conduct live streaming of goods, live games, or live chatting. Or, three-dimensional digital humans can be used to simulate real teachers to conduct online teaching.

[0071] The three-dimensional digital humans used in the above scenarios are mainly generated in real time. In actual use, they may also need to be pre-generated and then rendered in non-real time. For example, three-dimensional digital humans are used in movies and TV series to replace actors, and corresponding videos are produced based on three-dimensional digital humans. There is no specific restriction here. You can choose to obtain the three-dimensional digital human by real-time rendering or pre-rendering according to actual needs.

[0072] Among them, in the real-time rendering scene, taking the above-mentioned virtual anchor as an example, the three-dimensional digital human can be displayed in the live broadcast screen, and according to the input voice, the three-dimensional digital human can show different actions and expressions, such as happy actions and expressions, angry actions and expressions, etc., without specific restrictions here.

[0073] In the application process of three-dimensional digital humans, in order to reduce the pressure on the client, the server can generate the data needed for the rendering process and send it to the client, which then performs the rendering.

[0074] For the client, it can display the corresponding three-dimensional digital human in the corresponding display interface. The following explains the scene of the three-dimensional digital human displayed in the client interface provided in the embodiment of the present application.

[0075] Figure 1 This is a schematic diagram of the interface display of the three-dimensional digital human generation method provided in the embodiment of this application. Please refer to Figure 1 , Figure 1 The interface shown is a live broadcast interface, in which a three-dimensional digital human 110 can be displayed, wherein the image of the three-dimensional digital human can be customized by the user, for example: a simulated real person image, a cartoon image, etc., which is not specifically limited here.

[0076] In this live broadcast scene, relevant parameters can be given to the three-dimensional digital human by inputting voice or text, so that the three-dimensional digital human can perform corresponding actions.

[0077] For example, a user inputs a voice and sets a corresponding emotion, such as an emotion of surprise. Then the corresponding three-dimensional digital human can show an expression of surprise according to the set emotion, and the lips can also make a mouth shape of surprise.

[0078] In the application of three-dimensional digital humans, in addition to the client needing to render and display the corresponding three-dimensional digital humans, the process of calculating the relevant parameters of each action of the three-dimensional digital humans can be completed by the server.

[0079] In this scenario, the server may be, for example, a server, one or more servers, or a server cluster. The client may be various terminal devices, mobile phones, personal computers, tablet computers, etc., without specific limitation. The client and the server are connected in communication.

[0080] The following is an explanation of the actual application scenarios of the three-dimensional digital human generation method including the client and the server provided in the embodiments of the present application.

[0081] Figure 2 This is a schematic diagram of an application scenario of the three-dimensional digital human generation method provided in the embodiment of the present application. Please refer to Figure 2 , the scenario may include a server 210 and multiple clients 220 corresponding to the server 210 .

[0082] Among them, the server end 210 can be an end running on a server, for example: a cloud server, an AI (Artificial Intelligence) server, a rack server, etc. that can realize the relevant parameter calculations of a three-dimensional digital human. Among them, the AI ​​server can be a server for performing artificial intelligence calculations, and the rack server can be, for example, a server installed in a standardized network equipment rack, and no specific restrictions are made here.

[0083] The client 220 can be a terminal device used by the user, for example, including but not limited to mobile phones, wearable devices (such as smart watches, smart bracelets, smart glasses, etc.), tablet computers, laptops, car terminals, PCs (Personal Computers), etc., without specific limitations here.

[0084] It should be noted that in an application scenario, the server 210 can synchronously implement data interaction for multiple clients 220. For example, the client sends the relevant parameters of the three-dimensional digital human to the server, and the server calculates the relevant parameters based on the relevant parameters to obtain the relevant data required for rendering, and then sends these data to the corresponding client, allowing the client to complete the rendering work.

[0085] The embodiment of the present application provides a method for generating a three-dimensional digital human. The specific implementation process of the method for generating a three-dimensional digital human provided in the embodiment of the present application is explained below. It should be noted that the execution subject of the method can be the above-mentioned server.

[0086] Figure 3 This is a flow chart of the method for generating a three-dimensional digital human provided in the embodiment of the present application. Please refer to Figure 3 , the method comprising:

[0087] S310: Obtain target personalized parameters.

[0088] It should be noted that the target personalized parameter includes at least one of target identity information and target emotion information.

[0089] Optionally, the personalized parameters may be parameters corresponding to the selected presets after the user selects multiple presets in the client, or may be parameters determined by the user through input. There is no specific limitation here and the parameters may be set according to actual needs.

[0090] The target personalized parameter may be a personalized parameter sent by the client to the server, or a default personalized parameter in the server, which is not specifically limited here and can be set accordingly according to the actual scenario.

[0091] For example: if you need to use the preset expressions on the server to generate the corresponding three-dimensional digital human, you can use the default personalized parameters in the server; if you need to generate three-dimensional digital humans corresponding to different expressions according to the needs of the client, you can use the personalized parameters sent by the receiving client.

[0092] It should be noted that if the personalized parameters sent by the client are used, multiple personalized parameters that can be pre-selected, generated in advance or generated in real time can be set for each client. The personalized parameters corresponding to the three-dimensional digital human actively selected by the user or displayed by default on the client can be used as the above-mentioned target personalized parameters.

[0093] Among them, in the target personalized parameters, the target identity information can refer to the identity of the three-dimensional digital person, such as age, gender, race, etc., and the target emotional information can refer to the emotions that the three-dimensional digital person can express, such as happiness, anger, sadness, etc. There is no specific restriction here, and corresponding settings can be made based on the emotions possessed by humans.

[0094] It should be noted that the target personalized parameter may be a personalized parameter determined by the client and sent to the server after the user performs a selection operation or an input operation through the client.

[0095] In one embodiment, the target personalization parameters may include one or more parameters.

[0096] For example, the target personalization parameters can be represented by Table 1 and Table 2:

[0097] Table 1

[0098] gender age mood male / /

[0099] Table 1 shows the target personalized parameters with only one parameter, for example: gender male, and other parameters are not limited. Table 1 shows only one example. In the actual implementation process, there may also be only one other parameter such as age, emotion, etc.

[0100] Table 2

[0101] gender age mood female 20 Happy

[0102] Table 2 shows the target personalized parameters with multiple parameters, for example: gender is female, age is 20 years old, and emotion is happy. Table 2 takes four different parameters as an example to form the target personalized parameters. In the actual implementation process, there may also be two, three, five or more parameters, which are not specifically limited here.

[0103] It should be noted that the emotions in Table 1 and Table 2 can be a fixed emotion, a combination of multiple emotions, or a changing emotion, which can be set accordingly according to actual needs, for example: an emotion in which happiness accounts for 0.5 and sadness accounts for 0.5, or a changing emotion of first being sad and then being happy, etc., and the duration of sadness and happiness can be set accordingly, without specific restrictions here, and the emotion parameter can be set accordingly according to actual needs.

[0104] When characterizing the target personalized parameters, each type of parameter (including parameters without specific types, such as age in Table 1, etc.) can be represented as target personalized parameters in the manner of Tables 1 and 2 above. For example, the target personalized parameters corresponding to Table 1 can be characterized as: male, unlimited age, unlimited race, and unlimited emotion; or, only parameters with specific types can be characterized as target personalized parameters, for example, the target personalized parameters corresponding to Table 1 are characterized as: male.

[0105] S320: Determine a target adjustment parameter based on the target personalized parameter and the mapping relationship.

[0106] The mapping relationship may include a one-to-one correspondence between multiple personalized parameters and multiple adjustment parameters.

[0107] The adjustment parameters are used to indicate the deviation of the head parameters between the animated digital human obtained by inputting the same speech into the personalized digital human generation model and the general digital human generation model.

[0108] Among them, the personalized digital human generation model can correspond to at least one personalized parameter. It should be noted that, since the target personalized parameter can be one parameter or multiple parameters, and the target personalized parameters can be different, multiple personalized digital human generation models can be set on the server side, and each set of target personalized parameters can correspond to a personalized digital human generation model.

[0109] The personalized digital human generation model can be used to generate a corresponding three-dimensional digital human's sound and picture model according to at least one personalized parameter corresponding to it. For example, a speech can be input into the personalized digital human generation model to obtain a three-dimensional digital human with the characteristics of the personalized parameter.

[0110] The universal digital human generation model can be a model used to generate a three-dimensional digital human without personalization. For example, it can be a sound and picture model used to generate a gender-independent, age-independent, race-independent, and emotion-independent sound and picture model. Through this model, a universal three-dimensional digital human can be obtained. The universal three-dimensional digital human has no gender tendency, no age tendency, no race tendency, and no emotional tendency.

[0111] The same speech can be input into a personalized digital human generation model and a universal digital human generation model to obtain a three-dimensional digital human with the characteristics of the personalized parameters and a universal three-dimensional digital human respectively. The head parameters of the three-dimensional digital human with the characteristics of the personalized parameters and the head parameters of the universal three-dimensional digital human in the three-dimensional space can be determined, and the deviation of the two head parameters can be calculated.

[0112] Among them, the head parameters can be the position coordinates of the key points of the head in the three-dimensional space. For example, Q points can be selected from the head of a three-dimensional digital human with the characteristics of the personalized parameters, and Q points can be selected from the corresponding positions of the head of a universal three-dimensional digital human. The differences in the positions of these Q points in the spatial coordinate systems where the two three-dimensional digital humans are located are compared, so as to obtain the deviation of the head parameters. The deviation of the position of each key point of the head can be used as one of the sub-adjustment parameters. If there are differences in the above N points, there can be N adjustment parameters corresponding to the at least one personalized parameter.

[0113] The mapping relationship may include a one-to-one correspondence between multiple personalized parameters and multiple adjustment parameters. The server may determine the target adjustment parameter corresponding to the target personalized parameter by searching the mapping relationship.

[0114] The mapping relationship provided in the embodiment of the present application is explained more clearly below through Table 3:

[0115] Table 3

[0116]

[0117]

[0118] Among them, any adjustment parameter δ i Each includes N frame adjustment parameters. The personalized parameters may include at least one of gender, age, and emotion.

[0119] In the actual implementation process, the target adjustment parameter corresponding to the target personalized parameter can be determined from the mapping relationship through the above mapping relationship and the target personalized parameter.

[0120] S330: Generate an animation target digital human based on the target adjustment parameters and the animation initial digital human.

[0121] The initial digital human being of the animation is obtained based on the first voice input into the general digital human generation model.

[0122] Optionally, the first voice may be a voice input by a user through a client, or a voice obtained by converting input text into voice using a preset text-to-speech conversion tool, wherein the text-to-speech conversion tool may be run on a client or on a server. If it is run on a client, the client may obtain the text and convert the text into voice and send it to the server; if it is run on a server, the client may send the text to the server, which converts the text into voice.

[0123] After the server obtains the first voice, the first voice can be input into the general digital human generation model to obtain an animated initial digital human.

[0124] After the server determines the initial digital human of the animation, if the client receives the personalized parameters input by the user, the server can determine the target adjustment parameters corresponding to the personalized parameters through a mapping relationship (for example, Table 3, or Table 5-1), and then adjust the initial digital human of the animation through the target adjustment parameters to obtain the target digital human of the animation. Among them, the target adjustment parameters may record the position deviation of some key points of the head. The head parameters of the initial digital human of the animation can be adjusted according to the position deviation of these key points of the head, that is, the position of the corresponding key points of the head can be adjusted. After all adjustments are completed, the target digital human of the animation can be obtained.

[0125] In a three-dimensional digital human generation method provided in an embodiment of the present application, a target personalized parameter can be obtained, and a target adjustment parameter can be determined based on the target personalized parameter and a mapping relationship, wherein the mapping relationship includes a one-to-one correspondence between multiple personalized parameters and multiple adjustment parameters, and the adjustment parameter is used to indicate the deviation of the head parameters between the animated digital human obtained by inputting the same speech into the personalized digital human generation model and the general digital human generation model, and then the animated target digital human can be generated based on the target adjustment parameter and the animated initial digital human. Among them, since there is a mapping relationship between the target adjustment parameter and the personalized parameter, after adjusting the animated initial digital human according to the target adjustment parameter, an animated target digital human with personalized characteristics can be obtained, that is, different animated target digital humans can be generated by changing the target personalized parameter, which can further improve the diversity of the animated target digital humans generated by the server.

[0126] A feasible implementation process of generating an animated target digital human provided in an embodiment of the present application is explained below.

[0127] Optionally, the first speech includes multiple speech frames, the target adjustment parameter includes multiple frame adjustment parameters, and each frame adjustment parameter corresponds to one speech frame.

[0128] Among them, the first voice can be composed of multiple voice frames, the initial digital human of the animation can also be composed of multiple frames, each frame of the initial digital human of the animation can be adjusted by a frame adjustment parameter, and each frame adjustment parameter can correspond to a voice frame, that is, each frame of the initial digital human of the animation can also correspond to a voice frame.

[0129] In one embodiment, generating an animated target digital human based on the target adjustment parameter and the animated initial digital human specifically includes: based on a frame adjustment parameter corresponding to each speech frame, adjusting each frame of the animated initial digital human corresponding to the speech frame to generate the animated target digital human.

[0130] It should be noted that, in the process of adjusting the initial animation digital human, each frame of the initial animation digital human can be adjusted based on a frame adjustment parameter corresponding to each speech frame. For example, the first speech can be divided into 100 speech frames, and each speech frame can correspond to a frame adjustment parameter. At this time, it can be understood that any one of the δ in Table 3 i Each includes 100 frame adjustment parameters. Based on these 100 frame adjustment parameters, the 100 frames of the initial animation digital human can be adjusted. After the adjustment is completed, the animation target digital human can be obtained.

[0131] Among them, the frame adjustment parameters can be used to adjust some head parameters of the initial animation digital human, for example: adjusting the positions of some selected head key points, so as to achieve adjustment of a certain frame. After the initial animation digital human of the corresponding frame is adjusted based on the frame adjustment parameters corresponding to each voice frame, the target animation digital human can be obtained.

[0132] In a method for generating a three-dimensional digital human provided by an embodiment of the present application, a mapping relationship table can be queried based on personalized parameters to determine a target adjustment parameter, and each frame of the initial animation digital human corresponding to the speech frame is adjusted based on a frame adjustment parameter corresponding to each speech frame included in the target adjustment parameter to generate an animated target digital human. Among them, by adjusting each frame of the initial animation digital human, the adjustment of the three-dimensional digital human can be more comprehensively achieved, and since each frame adjustment parameter corresponds to a speech frame, adjusting each frame of the initial animation digital human based on the frame adjustment parameter corresponding to each speech frame can also obtain a more realistic three-dimensional digital human that better matches the content of the first speech.

[0133] A feasible implementation process of adjusting each frame of the initial digital human animation provided in the embodiment of the present application is explained below.

[0134] In one embodiment, each frame of the digital human includes multiple grid units in a three-dimensional space, and each frame adjustment parameter includes: an identifier of a grid unit to be adjusted among the multiple grid units and an offset value of each grid unit to be adjusted, that is, δ in Table 3. i .

[0135] It should be noted that both the initial animation digital human and the target animation digital human may include multiple frames, and each frame may include multiple grid units in the three-dimensional space.

[0136] Among them, the grid unit can be a grid point in a grid structure that constitutes a triangular network or a quadrangular network. These grid structures are spread over the head of the three-dimensional digital human, thereby forming a head skeleton of the three-dimensional digital human. The head parameters of the three-dimensional digital human can be represented on the head skeleton in the form of these grid units. The head parameters may include, for example: head contour parameters, facial parameters, lip parameters, etc.

[0137] Figure 4 This is a schematic diagram of the head grid of the three-dimensional digital human provided in the embodiment of this application. Please refer to Figure 4 , Figure 4 The three-dimensional digital human shown in the figure may be the above-mentioned initial animation digital human, for example, it may be a digital human output by a general digital human generation model, or it may be a digital human with personalized characteristics output by a personalized digital human generation model, and no specific limitation is made here.

[0138] It should be noted that the grid unit to be adjusted can be any one or more key grid units selected from the grid units of the three-dimensional digital human's head. There is no specific limitation here and it can be selected according to actual needs. For example, if facial movements need to be adjusted, the grid units of the three-dimensional digital human's face can be selected; if the lip shape needs to be adjusted, the grid units of the three-dimensional digital human's lips can be selected.

[0139] Among them, the identifier of the grid unit to be adjusted can be the label corresponding to the grid unit, for example: it can be represented by a preset digital serial number, letter serial number, etc., and specific restrictions are made here. Each grid unit to be adjusted can be represented by the label.

[0140] The offset value of the grid unit to be adjusted may be used to indicate the offset amount of the grid unit, for example, the amount of change of the three-dimensional coordinates of the grid unit in the xyz axis of the three-dimensional space.

[0141] In one embodiment, based on each frame adjustment parameter, each frame of the animated initial digital human corresponding to the voice frame is adjusted to generate an animated target digital human, specifically including: based on the identifier of the grid unit to be adjusted in each frame adjustment parameter and the offset value of each grid unit to be adjusted, the three-dimensional space coordinates of the grid unit to be adjusted in the animated initial digital human corresponding to the voice frame are adjusted to generate an animated target digital human corresponding to the voice frame.

[0142] It should be noted that during the adjustment process, the coordinates of the grid unit to be adjusted in the animated initial digital human corresponding to the frame can be adjusted in three-dimensional space based on the identification of the grid unit to be adjusted in each frame adjustment parameter and the offset value of the grid unit to be adjusted.

[0143] For example: In the actual implementation process, the grid unit to be adjusted can be first determined from multiple grid units according to the identification of the grid unit to be adjusted in the adjustment parameters of each frame, and then the coordinates of the corresponding grid unit to be adjusted in the initial animation digital human can be adjusted based on the offset value of the grid unit to be adjusted. The positions of each grid unit to be adjusted in the initial animation digital human can be adjusted based on the adjustment method indicated by the offset value, so as to obtain the animation target digital human corresponding to the voice frame. The offset value is the target adjustment parameter δ in Table 3 i The frame adjustment parameter included in the frame adjustment parameter is used to represent the change amount of the three-dimensional coordinates of the cell to be adjusted.

[0144] Based on the above method, the initial animation digital human corresponding to each speech frame can be adjusted respectively to obtain the target animation digital human.

[0145] In a three-dimensional digital human generation method provided in an embodiment of the present application, the three-dimensional spatial coordinates of the grid units to be adjusted in the animation initial digital human corresponding to the voice frame can be adjusted based on the identifiers of the grid units to be adjusted in each frame adjustment parameter and the offset values ​​of each grid unit to be adjusted, so as to generate an animation target digital human corresponding to the voice frame. In this method, the grid units to be adjusted and the values ​​to be adjusted of each grid unit to be adjusted can be determined more accurately and quickly according to the identifiers of the grid units to be adjusted and the offset values, so that the adjustment of the grid units can be achieved more quickly and accurately, and a more accurate animation target digital human can be obtained.

[0146] The following is an explanation of the implementation steps that can be performed after obtaining the animation target digital human in the three-dimensional digital human generation method provided in the embodiment of the present application.

[0147] The method also includes: sending the animated target digital human to the client, so that the client renders the animated target digital human.

[0148] It should be noted that after the server adjusts the initial animation digital human in the above manner and obtains the animation target digital human, the animation target digital human can be sent to the client, wherein the client can be a client that provides target personalized parameters to the server, or a client that provides the first voice, and no specific restrictions are made here.

[0149] After the client obtains the animation target digital human, it can render the animation target digital human.

[0150] Optionally, rendering may include rendering in a variety of different ways, for example, the skeleton and texture files sent by the server may be received in the form of a data stream, and then rendering may be performed according to the configuration file of the digital human, wherein the configuration file of the digital human indicates the coordinates of the skeleton of the digital human in three-dimensional space, and may indicate the specific positions of each texture on the skeleton of the digital human, and may also indicate the type of texture to be used to render the digital human.

[0151] In addition, during the rendering process, you can also use lighting and shadows to simulate real effects, and use the rendering engine to generate a three-dimensional digital human displayed on the client's monitor, which is the rendered three-dimensional digital human.

[0152] In the three-dimensional digital human generation method provided in the embodiment of the present application, the animated target digital human can be sent to the client so that the client can render the animated target digital human. By transmitting the target animated digital human, the client can more accurately and quickly render the animated target digital human, and then the client can display the rendered three-dimensional digital human.

[0153] The following is an explanation of the implementation process of sending the animated target digital human to the client in the three-dimensional digital human generation method provided in the embodiment of the present application.

[0154] Sending the animated target digital human to the client specifically includes: sending the target digital human and the target deviation parameter corresponding to the first speech frame in the first speech to the client.

[0155] The target deviation parameter is at least part of a plurality of deviation parameters; each deviation parameter includes a deviation of head parameters of the animated target digital human corresponding to adjacent speech frames in the animated target digital human.

[0156] It should be noted that the target digital human can be the digital human corresponding to the first voice frame in the first voice, the target digital human can be the first frame in the animated target digital human, or it can be another frame. There is no specific limitation here, and it can be set according to the correspondence between the frame of the animated target digital human and the voice frame of the first voice.

[0157] Optionally, the deviation parameter may be the deviation of the head parameters of the animated target digital human in two adjacent frames corresponding to every two adjacent speech frames in the animated target digital human. The target deviation parameter may be a part of the head deviation parameters, for example, if the deviation parameter includes the position deviation of 100 key points of the head, the target deviation parameter may be the position deviation of 50 key points of the head.

[0158] For example, if the first voice includes 200 frames, there may be a deviation of the head parameters of a corresponding animated target digital human in each two adjacent frames, that is, there are deviations of the head parameters of 199 animated target digital humans. The target digital human corresponding to the first voice frame and the remaining 199 target deviation parameters may be sent to the client.

[0159] It should be noted that the deviation parameters can be recorded in the form of position differences of key points of the head. For example, if there is a position difference of 20 grid units between two adjacent animated target digital humans, the deviation parameters can record the numbers of these 20 grid units and the deviation values ​​of these 20 grid units.

[0160] Optionally, based on the target digital human, the client can start from the second voice frame and render each voice frame according to the target deviation parameters of the animated target digital human corresponding to the voice frame and the next voice frame.

[0161] In the method for generating a three-dimensional digital human provided in the embodiment of the present application, the target digital human and the target deviation parameter corresponding to the first voice frame in the first voice can be sent to the client. The target deviation parameter is at least part of a plurality of deviation parameters; each deviation parameter includes the deviation of the head parameters of the animated target digital human corresponding to adjacent voice frames in the animated target digital human. Among them, by determining the target digital human and the target deviation parameter, the client can more accurately render the three-dimensional digital human.

[0162] Optionally, the deviation parameters of different frames may not be sent to the client at one time, but new deviation parameters may be obtained in real time and sent to the client in sequence according to the order corresponding to the voice frames.

[0163] In one embodiment, each deviation parameter includes: an identifier of a grid unit to be adjusted among multiple grid units of a target digital person corresponding to adjacent speech frames and an offset value of each grid unit to be adjusted; sending the target digital person and the target deviation parameter corresponding to the first speech frame in the first speech to the client, including: sending multiple target deviation parameters to the client in sequence according to the time sequence of each speech frame in the first speech.

[0164] Optionally, the target deviation parameters in the deviation parameters corresponding to each voice frame may be sent to the client in sequence according to the order of the voice frames in the first voice. The deviation parameters corresponding to each voice frame may be the deviation of the head parameters of the animated target digital human corresponding to the voice frame and the next frame of the voice frame.

[0165] Correspondingly, the client can also obtain the corresponding target deviation parameters in sequence, and then perform rendering.

[0166] For the client, in each rendering process, the position of the corresponding grid unit in the three-dimensional digital human rendered in the previous frame can be adjusted according to the position difference of the grid unit in the target deviation parameter.

[0167] In the method for generating a three-dimensional digital human provided in the embodiment of the present application, multiple target deviation parameters can be sent to the client in sequence according to the time sequence of each voice frame in the first voice. Among them, the transmission of multiple target deviation parameters according to the order of the voice frames in the first voice can more accurately realize the rendering of the three-dimensional digital human by the client, thereby improving the real-time performance and accuracy of the rendering of the three-dimensional digital human by the client.

[0168] In the process of sending the deviation parameters, all the deviation parameters may be sent to the client, or only a part of the deviation parameters may be sent to the client. For example, if the deviation parameters record the difference values ​​of 20 grid cells, of which the difference of 5 grid cells has a lower impact on the overall three-dimensional digital human image, the difference of the remaining 15 grid cells may be sent to the client as the target deviation parameter, and the difference of these 5 grid cells will not be transmitted accordingly.

[0169] In one embodiment, sending a target deviation parameter to a client includes: determining the target deviation parameter from a plurality of deviation parameters according to preset conditions, the preset conditions including at least one of bandwidth configuration information of the client, display requirements of the client, and a weight of a facial area corresponding to the deviation parameter; and sending the target deviation parameter to the client.

[0170] It should be noted that bandwidth configuration information refers to the size of the transmission bandwidth between the client and the server, which can be used to characterize the amount of information that can be transmitted during a communication process. If the bandwidth is large, more information can be transmitted, and if the bandwidth is small, less information can be transmitted. In this case, if the bandwidth is small, the difference of the grid cells in the deviation parameter that have a low impact on the three-dimensional digital human image can be deleted and not transmitted, that is, only the difference of some of the grid cells is transmitted.

[0171] Among them, the grid units with lower influence may be grid units at certain positions, such as grid units at the forehead and ears of the digital human. The specific grid units with lower influence may be actively set by the user, and no specific limitation is made here.

[0172] The display requirement of the client may be a personalized configuration on the client. For example, due to reasons such as resolution, the number of grid units that can be displayed by the three-dimensional digital human is reduced. In this case, only some of the grid units can be displayed on the client. In this case, only the differences in the grid units that can be displayed on the client may be transmitted. For the grid units that cannot be displayed, the difference values ​​will not be transmitted.

[0173] The weight of the facial area corresponding to the deviation parameter may refer to the weight of the facial area position where the head key point corresponding to the deviation parameter is located. For example, the weights of the cheek and mouth positions in the facial area of ​​the three-dimensional digital human may be set to be higher, while the weights of other positions may be lower. If the head key point corresponding to a certain deviation parameter is at a position with a higher weight, it may be determined that the deviation parameter is transmitted; if the head key point corresponding to a certain deviation parameter is at a position with a lower weight, it may be determined that the deviation parameter is not transmitted. It should be noted that the weight may be set according to actual needs and is not a fixed weight.

[0174] In the actual transmission process, the target deviation parameter can be determined from the deviation parameter according to any one or more of the above three conditions, and the determined target deviation parameter can be transmitted to the client.

[0175] In the method for generating a three-dimensional digital human provided in the embodiment of the present application, a target deviation parameter can be determined from a plurality of deviation parameters according to a preset condition, and the target deviation parameter can be sent to the client. The preset condition includes at least one of the bandwidth configuration information of the client, the display requirements of the client, and the weight of the facial area corresponding to the deviation parameter. The number of grid cells with differences in the deviation parameters can be reduced by determining the target deviation parameter from the deviation parameters, thereby reducing the amount of information transmitted between the client and the server, reducing the amount of computation of the client in the rendering process, and improving the rendering efficiency.

[0176] In order to more clearly demonstrate the process of rendering a three-dimensional digital human using the deviation parameters provided in the embodiment of the present application, the rendering process is explained below through the changes of two frames of three-dimensional digital humans.

[0177] Figure 5 This is a schematic diagram of the interface changes of rendering a three-dimensional digital human based on deviation parameters provided in the embodiment of the present application. Please refer to Figure 5 , Figure 5 The display interface on the left is the display interface corresponding to the Nth frame of the three-dimensional digital human, and the display interface on the right is the display interface corresponding to the N+1th frame of the three-dimensional digital human. N can be a positive integer greater than or equal to 1.

[0178] according to Figure 5It can be seen from the three-dimensional digital human shown in that there is a certain difference between the movements of the three-dimensional digital human on the left and the movements of the three-dimensional digital human on the right. This difference can be obtained by rendering the three-dimensional digital human corresponding to the Nth frame according to the target deviation parameter corresponding to the N+1th frame.

[0179] For example, if N is 1, then Figure 5 The three-dimensional digital human in the left display interface is the target digital human corresponding to the first voice frame transmitted by the server, and the three-dimensional digital human in the right interface is the three-dimensional digital human obtained by rendering the target digital human according to the target deviation parameter of the animated target digital human corresponding to the first voice frame and the second voice frame.

[0180] For example, Figure 5 The emotion of the three-dimensional digital person on the left side of the middle is sadness, and the emotion of the three-dimensional digital person on the right side is fear.

[0181] It should be noted that in addition to executing the above rendering process, the client can also pre-load the skeleton, textures and configuration files of the three-dimensional digital human during the initialization process, so that it can render more quickly after receiving the three-dimensional digital human to be displayed sent by the server.

[0182] The above process is an explanation of the steps performed by the server and the client during the actual application of the three-dimensional digital human. In the actual implementation process, the general digital human generation model stored in the server can be obtained after training. The training method of the three-dimensional digital human model provided in the embodiment of the present application is explained below.

[0183] Figure 6 This is a flow chart of the training method of the three-dimensional digital human model provided in the embodiment of the present application. Please refer to Figure 6 , the method comprising:

[0184] S610: Obtain sample videos corresponding to different personalized parameters.

[0185] Optionally, the sample videos may be pre-acquired 2D videos (two-dimensional videos), and these sample videos may be classified in advance to obtain sample videos corresponding to each personalized parameter.

[0186] It should be noted that the corresponding sample videos can be configured according to the required personalized parameters, or the personalized parameters in these sample videos can be determined through the collected sample videos, and no specific limitation is made here.

[0187] For example, if 100 different sets of personalized parameters are configured, a corresponding sample video may be set for each set of personalized parameters. Alternatively, if 20 sample videos are acquired in advance, the personalized parameters existing in the 20 sample videos may be extracted.

[0188] It should be noted that the two-dimensional video may be a video with a real person speaking, for example, a video of a teacher giving a lecture. In this sample video, there needs to be a head image of a real person, preferably a face image.

[0189] For any two-dimensional video, in the actual implementation process, the real person may show different emotions, for example, changing from happy to sad, or the real person in the video may change, for example, the first half of the video is shot with a male, and the second half changes to a female, etc.

[0190] Before obtaining sample videos, these two-dimensional videos can be segmented. For example, segments with different emotions or identities in the video can be identified by clustering, and multiple segments can be obtained, and each segment can be used as a sample video.

[0191] Optionally, multiple sample videos may be obtained by segmenting the two-dimensional video, and each sample video may correspond to a set of personalized parameters. For example, the personalized parameters corresponding to a sample video are: male, 50 years old, happy; and the personalized parameters corresponding to another sample video are: sad.

[0192] It should be noted that the correspondence between the sample videos and the personalized parameters can be presented in the form of Table 4.

[0193] Table 4

[0194] Sample Video Personalized parameters Sample Video 1 male Sample Video 2 female Sample Video 3 Male, 20 years old … … Sample Video N-1 Male, happy Sample Video N Male, 20 years old, happy

[0195] Please refer to Table 4. Each sample video may correspond to a set of different personalized parameters. The personalized parameters may be one or more, and there is no specific limitation here.

[0196] S620: Obtain corresponding sample data from the sample videos corresponding to each personalized parameter.

[0197] The sample data includes sample voice, sample user's emotions and sample user's head parameters.

[0198] Optionally, after obtaining the sample video, corresponding sample data can be determined from the sample video. For the same sample video, multiple groups of sample data can be obtained, wherein each group of sample data can include sample voice, sample user's emotions, and sample user's head parameters.

[0199] The sample voice may be a part of the voice corresponding to the sample video, the emotion of the sample user may be the emotion of the real person in the sample video, and the head parameters of the sample user may be the head parameters of a three-dimensional model (that is, the sample user) established in three-dimensional space based on the head image of the real person in the sample video.

[0200] For each sample video corresponding to a personalized parameter, the above-mentioned multiple groups of sample data can be obtained.

[0201] It should be noted that the head parameters of the sample user may specifically include the head contour parameters, facial parameters, lip parameters, etc. of the sample user. These parameters may specifically be the positions of the grid units at the corresponding positions. For example, the head contour parameters may be the positions of the grid units that constitute the head contour of the entire three-dimensional digital human; the facial parameters may be the positions of the grid units that constitute the face of the three-dimensional digital human; and the lip parameters may be the positions of the grid units that constitute the lips of the three-dimensional digital human.

[0202] Optionally, the head parameters and the emotions of the sample users may correspond to the sample voices, and each voice frame may correspond to the head parameters and the emotions of the sample users.

[0203] For example: if a sample speech in a sample data has 100 speech frames, there may be 100 emotions of the sample user and 100 head parameters of the sample user, and the emotion and head parameter of each frame of the sample user corresponds to a speech frame of the sample speech.

[0204] S630: Training the audio-visual generation model according to the sample data corresponding to each personalized parameter to obtain a personalized digital human generation model corresponding to each personalized parameter.

[0205] Optionally, sample data corresponding to each personalized parameter may be obtained in the above manner, and then the audio-visual generation model may be trained based on the sample data.

[0206] Among them, the audio and video generation model can be an audio and video encoder, or a neural network structure, whose input can be sample sounds and the emotions of sample users, and whose output can be a three-dimensional digital human. The head parameters of the output three-dimensional digital human can be compared with the head parameters of the sample user. After a large amount of training, if the audio and video generation model converges, for example: when the degree of match between the head parameters of the output three-dimensional digital human and the head parameters of the sample user is greater than a preset similarity threshold, it can be determined that the audio and video generation model converges, thereby obtaining a personalized digital human generation model corresponding to the personalized parameters.

[0207] It should be noted that the above-mentioned training process can be performed for a set of personalized parameters corresponding to each sample video, so as to obtain a personalized digital human generation model corresponding to each set of personalized parameters. That is to say, multiple personalized digital human generation models are obtained by training the same audio and video generation model based on different sample data.

[0208] S640: Normalize different personalized digital human generation models to obtain a universal digital human generation model.

[0209] It should be noted that after obtaining multiple personalized digital human generation models, these personalized digital human generation models can be normalized.

[0210] It should be noted that there are many methods for normalization processing. For example, the normalization processing of input features can be used, that is, before training these personalized digital human generation models, the input data is normalized to ensure that different sample data have similar scales. Alternatively, inter-layer normalization processing can be used. Since each personalized digital human generation model is trained based on the same audio and video generation model, these personalized digital human generation models have a certain degree of similarity. For example, the number of neural network layers is the same, and normalization can be applied after each neural network layer is generated, especially before the activation function. Alternatively, the weight normalization method can be used to re-parameterize the weights of each layer in the neural network, decouple the length of the weight vector and its direction, and normalize these personalized digital human generation models.

[0211] In the actual implementation process, any one of the above-mentioned normalization methods can be used to implement the normalization processing of multiple personalized digital human generation models, so as to obtain a universal digital human generation model.

[0212] The general digital human generation model can be obtained by normalizing all personalized digital human generation models.

[0213] In actual application, the input parameter of the universal digital human generation model may be a first voice, and the output may be a universal three-dimensional digital human.

[0214] In the training method of the three-dimensional digital human model provided in the embodiment of the present application, sample videos corresponding to different personalized parameters can be obtained; corresponding sample data can be obtained from the sample videos corresponding to each personalized parameter; the audio and video generation model is trained according to the sample data corresponding to each personalized parameter to obtain a personalized digital human generation model corresponding to each personalized parameter; different personalized digital human generation models are normalized to obtain a universal digital human generation model. Among them, normalizing multiple personalized digital human generation models can make the obtained universal digital human generation model have high versatility and applicability, and can realize the generation of corresponding three-dimensional digital humans for different personalized parameters.

[0215] The following is an explanation of the implementation process of determining sample data based on sample videos provided in the embodiments of the present application.

[0216] Figure 7 Please refer to the flowchart of determining sample data provided in the embodiment of this application. Figure 7 For the sample video, the corresponding sample speech in each frame of the sample video can be determined, and the three-dimensional model established based on the video image of the real person can be used to determine the emotion and head parameters of the three-dimensional model established in the three-dimensional space, and the head parameters of the model are used as the head parameters of the sample user, and the facial emotions of the model are used as the emotions of the sample user.

[0217] Among them, a three-dimensional model can be first established based on the two-dimensional video, that is, a digital human is trained, and the head parameters of the sample user are determined based on the head parameters of the trained digital human, and then the emotions of the trained digital human are identified to obtain the emotions of the sample user.

[0218] In one embodiment, corresponding sample data is obtained from sample videos corresponding to each personalized parameter, including: establishing a corresponding training digital human according to the sample video, and determining the head parameters of the sample user according to the head parameters of the training digital human; identifying the emotions of the sample user through the facial parameters of the training digital human; determining the sample speech corresponding to each personalized parameter, as well as the emotion of the sample user and the head parameters of the sample user corresponding to each speech frame of the sample speech.

[0219] It should be noted that the speech in the sample video can be extracted and used as the sample speech in the sample data.

[0220] Optionally, the head area of ​​the real person in the video can be determined from the sample video, and then the head parameters of the sample user in the two-dimensional image can be determined from the head area. A conversion method from two-dimensional video to three-dimensional modeling is adopted to establish a training digital human and obtain the head parameters of the training digital human. The head parameters of the training digital human can be used as the head parameters of the sample user in the sample data.

[0221] After obtaining the trained digital human, the sample user's emotions can be identified based on the facial parameters of the trained digital human. For example, the sample user's emotions can be determined by comparing the differences between the facial parameters of the three-dimensional digital human and the facial parameters of a preset three-dimensional digital human with emotional tendencies.

[0222] For each speech frame, there can be a corresponding training digital human. The head parameters of the sample user corresponding to the training digital human are the head parameters of the sample user corresponding to the speech frame. Correspondingly, the emotions of the sample user corresponding to the training digital human are the emotions of the sample user corresponding to the speech frame.

[0223] Through the above steps, the head parameters of the sample user and the emotion of the sample user corresponding to each speech frame of the sample speech in the sample data can be obtained respectively.

[0224] It should be noted that the training digital human can be a digital human obtained by converting a two-dimensional graphic into a three-dimensional graphic. For example, projection modeling can be used to obtain the depth information of the two-dimensional image through different angles of the real person's head in multiple frames of two-dimensional images, and then a training digital human can be established based on the depth information of the two-dimensional image and the real person's facial image displayed in the two-dimensional image.

[0225] In a training method for a three-dimensional digital human model provided in an embodiment of the present application, a corresponding training digital human can be established based on a sample video, and the head parameters of a sample user can be determined based on the head parameters of the training digital human; the emotions of the sample user can be identified by the facial parameters of the training digital human; the sample voice corresponding to each personalized parameter, and the sample user's emotions and the head parameters of the sample user corresponding to each voice frame of the sample voice can be determined. Among them, by establishing a training digital human, the head parameters and emotions of the sample user can be obtained more quickly and accurately, so that the sample data can be obtained more quickly and accurately.

[0226] It should be noted that after obtaining the trained digital human, the emotion of the trained digital human can be determined, and the determination can be performed in the following manner.

[0227] Figure 8 Please refer to the flowchart of identifying the sample user's emotions provided in the embodiment of this application. Figure 8 , by training the facial parameters of the digital human, the emotions of the sample user are recognized, including:

[0228] S810: Calculating the similarity between the facial parameters of the training digital human and the facial parameters of each preset emotional avatar model.

[0229] Optionally, multiple emotional avatar models can be pre-stored on the server side, each of which can be a three-dimensional digital human with an expression tendency. Each three-dimensional digital human with an expression tendency can represent an emotion, such as happiness, sadness, anger, etc., and the facial expression of each emotional model is matched with the corresponding emotion, such as the expression corresponding to the happy emotion, the expression corresponding to the sad emotion, the expression corresponding to the angry emotion, etc., without specific limitation here.

[0230] It should be noted that the facial parameters of each emotion model may be parameters corresponding to the emotion represented by the model.

[0231] For example, different emotional avatar models can be set for different emotions. For example, human emotions can be divided into 52 different emotions, and an emotional avatar model can be set for each emotion.

[0232] It should be noted that the number and distribution of grid units in all emotional avatar models may be the same as the number and distribution of grid units in training digital humans.

[0233] After obtaining the trained digital human, the facial parameters of the trained digital human can be compared with the facial parameters of the emotional avatar model, for example, similarity can be calculated, wherein the similarity can be determined by comparing the relative positions of each grid unit corresponding to the facial parameters in the trained digital human and the emotional avatar model.

[0234] The relative position may be the relative position between multiple grid units in the training digital human and the emotional avatar model, for example, the position deviation between any two grid units.

[0235] It should be noted that the number and distribution of grid cells in the emotional avatar model can be the same as the number and distribution of grid cells in the training digital human. In the actual implementation process, the similarity of facial parameters between the training digital human and each pre-set emotional avatar model can be determined based on the degree of matching of the relative positions.

[0236] S820: Determine the emotion label of the training digital human according to the similarity.

[0237] The emotion label includes the emotion and weight of the target emotion avatar model, and the target emotion avatar model is the emotion avatar model corresponding to the similarity greater than or equal to the threshold. The emotion of the sample user includes the emotion label of the training digital human.

[0238] It should be noted that the similarity can be expressed in the form of weights. For example, if the similarity between the training digital human and the emotional avatar model corresponding to the emotion of happiness is compared, the weight of the emotion of happiness in the training digital human can be obtained. For example, if the similarity is 60%, the weight of the emotion of happiness can be obtained as 0.6.

[0239] Optionally, the similarity between the training digital human and each emotional avatar model can be determined in the above manner, thereby obtaining the emotional label of the three-dimensional digital human.

[0240] Among them, emotional labels can be recorded in the form of "emotion + weight", for example: "happy 0.6, sad 0.1, angry 0.1" can mean that in the expression corresponding to the emotional label, the weight of the emotion of happiness is 0.6, the weight of the emotion of sadness is 0.1, and the weight of the emotion of anger is 0.1.

[0241] It should be noted that the weight may represent the degree of similarity between the training digital human and the emotional avatar model of the corresponding emotion. The above-mentioned emotional label may record that the expression has a similarity of 0.6 to the emotion of happiness, a similarity of 0.1 to the emotion of sadness, and a similarity of 0.1 to the emotion of anger.

[0242] In addition, it is also possible to record only multiple weights in the emotion tag in a certain preset order, for example, an order such as "0.6, 0.1, 0.1, 0.5, ..., 0.2", where each position in the order can correspond to a similarity. Taking setting 52 emotion avatar models as an example, the weights corresponding to the 52 emotions can be recorded in the emotion tag.

[0243] It should be noted that in the actual process of determining emotion labels, the cases with lower similarity can be discarded and only the results with higher similarity can be retained. For example, for the label "happy 0.6, sad 0.1, angry 0.1" in the aforementioned example, if the similarity threshold is 0.5, the emotion label can be changed to the result "happy 0.6".

[0244] In the actual process of expressing emotional labels, if there is only one label and its weight is 100%, the emotion corresponding to the label can be used as the label to express it, for example: "happy", "sad", etc.; if there is only one label but its weight is not 100%, the emotion corresponding to the label and its weight can also be used as the label to express it, for example: "happy 0.6", "sad 0.5", etc.; if there are multiple labels and each label has a corresponding weight, they can be expressed in a combined manner, for example: "happy 0.6, sad 0.2", "angry 0.6, sad 0.5, happy 0.1".

[0245] That is to say, after determining the similarity, a target emotional avatar model, that is, an emotional avatar model with a similarity greater than or equal to a threshold, can be determined from the emotional avatar models, thereby obtaining a corresponding emotional label.

[0246] Optionally, the sample user's emotions may include emotion labels of the trained digital human, and the emotion labels may be used to represent the sample user's emotions. Alternatively, the emotion labels and other relevant data may be combined to represent the sample user's emotions, which is not specifically limited here.

[0247] It should be noted that the emotions of sample users can be expressed in the form of emotion labels, that is, the emotions of sample users in the same frame are not unique, and may be mixed with both "happy" and "angry". In this case, the emotions of sample users in a certain frame can be expressed in the form of emotion labels with weights. For the emotions of sample users in multiple frames, multiple emotion labels can be used to express them.

[0248] In the training method of the three-dimensional digital human model provided in the embodiment of the present application, the similarity between the facial parameters of the training digital human and each pre-set emotional avatar model can be calculated; and the emotional label of the training digital human can be determined according to the similarity. Among them, by determining the emotional label, the emotion of each training digital human can be determined in more detail, and in addition to the emotion, the weight corresponding to the emotion can also be determined, so that a more accurate and detailed emotional label can be obtained.

[0249] In one embodiment, the sample video includes multiple video frames, the training digital human includes multiple frames, each frame of the training digital human corresponds one-to-one to a video frame, and the emotion of the sample user includes an emotion label of the training digital human in each frame.

[0250] Optionally, the sample speech in the sample video may include multiple speech frames, each speech frame may correspond to a video frame in the sample video, and accordingly, may correspond to a frame of training digital human, and the emotions of the training digital human corresponding to each video frame may be different. For example, the expression of the training digital human in the first video frame is a happy expression, and the expression of the training digital human in the tenth video frame becomes a sad expression. In this case, the emotion label of each video frame can be recorded, and these multiple emotion labels together constitute the emotion of the sample user, that is, the emotion of the above-mentioned sample user.

[0251] It should be noted that in the process of training the model, the emotions of the sample users can be trained with one or more specific emotions, or, the weight of the emotions can be added to the specific emotions for training. For example, the emotions of the sample users in each frame can be input into the audio and video generation model in the form of an emotion label for training. No specific restrictions are made here.

[0252] Through the above steps, we can obtain sample speech, sample user's emotions and sample user's head parameters respectively, so as to obtain more accurate and detailed sample data. After training the audio and video generation model according to these sample data, we can obtain a personalized digital human generation model.

[0253] It should be noted that a general digital human generation model can be obtained after normalization processing based on the personalized digital human generation model.

[0254] Correspondingly, the emotional avatar model explained above can also be a three-dimensional digital human with preset expressions, and the three-dimensional digital human can be displayed in the following manner.

[0255] Fig. 9 This is a schematic diagram of the emotional avatar model provided in the embodiment of the present application, please refer to Fig. 9 , Fig. 9 The 3D digital human on the left can be the emotional avatar model corresponding to the emotion of surprise. Fig. 9 The 3D digital human on the right can be the emotional avatar model corresponding to the emotion of calmness. Fig. 9 Only the emotional avatar models corresponding to these two emotions are taken as examples. In the actual implementation process, different emotional avatar models can be set for different emotions.

[0256] A feasible implementation process of establishing a mapping relationship between adjustment parameters and personalized parameters provided in an embodiment of the present application is explained below.

[0257] Fig.10 Please refer to the flowchart of establishing the mapping relationship provided in the embodiment of this application. Fig.10 After normalizing different personalized digital human generation models to obtain a universal digital human generation model, the method further includes:

[0258] S1010: Input each of the P segments of sample speech into the personalized digital human generation model and the general digital human generation model corresponding to the M personalized parameters, and obtain P×M personalized animated digital humans and one general animated digital human.

[0259] It should be noted that both the personalized digital human generation model and the general digital human generation model can be digital human generation models that are pre-trained and stored in the server. In the actual implementation process, each of the P segments of sample speech can be input into the personalized digital human generation model and the general digital human generation model corresponding to the M personalized parameters respectively. Through the personalized digital human generation model, P×M personalized animated digital humans can be obtained, and through the general digital human generation model, a general animated digital human can be obtained.

[0260] Among them, the personalized animated digital human can be a digital human with personalized characteristics, and its personalized characteristics can be the characteristics of personalized parameters corresponding to the personalized digital human generation model, such as: gender, age, race, and emotions, etc., which are not specifically limited here.

[0261] S1020: Determine the deviation between the head parameters of each frame of each personalized animated digital human in the M personalized animated digital humans corresponding to each speech segment in the P speech segments and the head parameters of each frame of the universal animated digital human.

[0262] It should be noted that both the personalized animated digital human and the universal animated digital human are generated based on each speech frame of the same sample speech. The two digital humans have the same number of frames and each frame corresponds one to one. The deviation of the head parameters between the personalized animated digital human and the universal animated digital human in each frame can be compared separately, specifically, the position deviation of multiple head key points in three-dimensional space.

[0263] For example, the head parameters of each frame of the P×M personalized animated digital humans can be compared with the head parameters of each frame of the general digital human to obtain the position deviations of these head key points in the three-dimensional space.

[0264] Fig.11 This is a schematic diagram of the head deviation between the personalized animated digital human and the general digital human provided in the embodiment of the present application. Please refer to Fig.11 , Fig.11 In the content shown in , the digital person on the upper side is any one of P×M personalized animated digital people, the digital person on the lower side is a universal animated digital person, and the horizontal axis is the order of voice frames.

[0265] It should be noted that a speech segment may include N speech frames, which are continuous in time. After the same speech frame is input into a personalized digital human generation model, a personalized animated digital human corresponding to the personalized parameters is generated, wherein each speech frame may correspond to a personalized digital human, see Fig.11 , N voice frames can correspond to N personalized digital people. Similarly, these N voice frames also correspond to N universal digital people. These N personalized digital people that are continuous in time constitute a personalized animation digital person corresponding to the personalized parameters. These N universal digital people that are continuous in time constitute a universal animation digital person. The Q points on the head mentioned above are all included in Fig.11 In the head parameters of each personalized digital person 1-N and each universal digital person an.

[0266] S1030: Perform weighted averaging processing on the deviations between the head parameters corresponding to each speech segment to obtain adjustment parameters, and establish a mapping relationship between the adjustment parameters and the personalized parameters.

[0267] Continue to refer Fig.11 , Fig.11 The head deviation between the personalized animated digital human and the universal animated digital human actually includes the head deviation between the personalized digital human corresponding to N speech frames respectively, that is to say, for the same speech, the head deviation between the personalized animated digital human and the universal animated digital human obtained after inputting into a certain personalized digital human generation model and a universal digital human generation model includes the head deviation between the personalized digital human and the universal digital human corresponding to each speech frame, and the head deviation between the personalized digital human and the universal digital human corresponding to each speech frame includes the difference in the three-dimensional spatial coordinates of the Q key points mentioned above included in the personalized digital human and the universal digital human corresponding to each speech frame.

[0268] For example, see Fig.11 , the head parameters of digital human 1 and digital human a corresponding to the first speech frame both include Q key grid units. The coordinates of the Q key grid units on digital human 1 are: A1(X1, Y1, Z1), B1(X2, Y2, Z2), ... N1(XQ, YQ, ZQ); the coordinates of the Q key grid units on digital human a are: Aa(x1, y1, z1), Ba(x2, y2, z2), ... Na(xQ, yQ, zQ), and each key grid unit on the personalized digital human and the universal digital human is one-to-one corresponding. Therefore, the head deviation between digital human 1 and digital human a is the difference between the three-dimensional coordinates of the grid unit A1-N1 on the personalized digital human 1 and the grid unit Aa-Na on the digital human a, denoted as δ 111 ; Among them, any grid unit on the digital human can be understood as a point in the three-dimensional space coordinate system.

[0269] Similarly, the calculation method of the head parameter deviation between the personalized digital human 2-N and the universal digital human bn can refer to the calculation method between the personalized digital human 1 and the universal digital human a, which will not be repeated here. Thus, N groups of head parameter deviations (δ 111 , δ 112 ...δ 11N ).

[0270] The head parameter deviation between each personalized digital human and each universal digital human can be regarded as a frame adjustment parameter. In this way, the head parameter deviation between a personalized animated digital human and a universal animated digital human can be obtained. The adjustment parameters to be obtained include the N frame adjustment parameters corresponding to the N speech frames. Under the same speech segment, for a personalized animated digital human and a universal animated digital human corresponding to a personalized parameter, the obtained adjustment parameter is (δ 111 , δ 112 ...δ 11N ), where δ 111 is the frame adjustment parameter corresponding to the first speech frame, δ 112 is the frame adjustment parameter corresponding to the second speech frame, δ 11N is the frame adjustment parameter corresponding to the Nth speech frame; and so on, for the M personalized animated digital humans corresponding to the M personalized parameters, the N frame adjustment parameters between the M personalized animated digital humans and the universal animated digital human are calculated respectively, and the first mapping relationship shown in Table 5-1 can be obtained:

[0271] Table 5-1

[0272] Personalized parameters (M) Adjusting parameters male <![CDATA[(δ 111 ,d 112 ……d 11N )]]> Female, 20 years old, yellow race, happy <![CDATA[(δ 121 ,d 122 ……d 12N )]]> Female, happy <![CDATA[(δ 131 ,d 132 ……d 13N )]]> …… …… Male, 40 years old, happy 0.5 + sad 0.5 <![CDATA[(δ 1N1 ,d 1N2 ……d 1NN )]]>

[0273] Table 5-1 shows only one example, where δ 111 , δ 112 ...δ 11N Each of them is a frame adjustment parameter; it should be noted that the first mapping relationship obtained in Table 5-1 is the corresponding relationship between the personalized parameters and the adjustment parameters under a certain speech segment, and if different speech segments are input into the personalized digital human generation model corresponding to the speech segments in Table 5-1 and the universal digital human generation model, another personalized animated digital human and another universal animated digital human obtained are different from the above examples (such as Fig.11 ) is different from the general animated digital human. That is to say, the above Table 5-1 is actually a mapping relationship corresponding to a speech segment, and the mapping relationships obtained for different speech segments are different.

[0274] Based on this, in one implementation, in order to improve the accuracy of the adjustment parameters, after any subsequent voice input by the user, the target adjustment parameters corresponding to the personalized parameters input by the user can be obtained based on the same mapping relationship table. When establishing the mapping relationship, P (P is an integer ≥ 1) different voice segments can be respectively input into the M personalized digital human generation models and the universal digital human generation model that are the same as the embodiment corresponding to Table 5-1 above, and P mapping relationships can be obtained, as follows: (For example, it can be Table 5-2, Table 5-3, etc., until Table 5-P.)

[0275] Table 5-2

[0276] Personalized parameters (M) Adjusting parameters male <![CDATA[(δ 211 ,d 212 ……d 21N )]]> Female, 20 years old, yellow race, happy <![CDATA[(δ 221 ,d 222 ……d 22N )]]> Female, happy <![CDATA[(δ 231 ,d 232 ……d 23N )]]> …… …… Male, 40 years old, happy 0.5 + sad 0.5 <![CDATA[(δ 2N1 ,d 2N2 ……d 2NN )]]>

[0277] Table 5-3

[0278] …

[0279] Table 5-P

[0280] Personalized parameters (M) Adjusting parameters male <![CDATA[(δ p11 ,d p12 ……d p1N )]]> Female, 20 years old, yellow race, happy <![CDATA[(δ p21 ,d p22 ……d p2N )]]> Female, happy <![CDATA[(δ p31 ,d p32 ……d p3N )]]> …… …… Male, 40 years old, happy 0.5 + sad 0.5 <![CDATA[(δ pN1 ,d pN2 ……d pNN )]]>

[0281] After obtaining the P mapping relationships as shown in Table 5-1 to Table 5-P, a weighted average calculation can be performed on the P adjustment parameters corresponding to each personalized parameter to finally obtain an averaged adjustment parameter. The calculation method is as follows:

[0282] For example, for the personalized parameter for males, the weighted average of the corresponding adjustment parameters in Table 5-1 to Table 5-P is:

[0283]

[0284] in, is the frame adjustment parameter corresponding to the first speech frame, is a frame adjustment parameter corresponding to the second speech frame, is the frame adjustment parameter corresponding to the Nth speech frame. The frame adjustment parameter corresponding to any speech frame is the difference between the three-dimensional coordinates of Q key points on the personalized digital human head and the general digital human head.

[0285] Similarly, we can obtain the adjustment parameter δi corresponding to each personalized parameter in Table 5-1 to Table 5-P, where 1≤i≤M;

[0286]

[0287] in, It can be expressed as δ i1 ;

[0288] It can be expressed as δ i2 ; It can be expressed as δ iN .

[0289] Through the above calculation method, the mapping relationship between the personalized parameters and the adjustment parameters can be expressed by the aforementioned Table 3. The adjustment parameter corresponding to any personalized parameter in Table 3 is calculated by formula (2).

[0290] For example, please refer to the aforementioned Table 3. A personalized parameter is: a 20-year-old happy girl. Through the above method, the adjustment parameter corresponding to the 15-year-old happy girl is δ2. Similarly, the adjustment parameters corresponding to other personalized parameters can also be obtained through the above method. In this way, multiple adjustment parameters corresponding to multiple personalized parameters can be obtained, so that a mapping relationship between the adjustment parameters and the personalized parameters can be established.

[0291] In the training method of the three-dimensional digital human model provided in the embodiment of the present application, a sample speech can be input into the personalized digital human generation model and the general digital human generation model corresponding to the personalized parameters to obtain a personalized animated digital human and a general animated digital human; determine the deviation between the head parameters of each frame of the personalized animated digital human and the head parameters of each frame of the general animated digital human; perform weighted average processing on the deviation between the head parameters corresponding to each speech segment to obtain an adjustment parameter, and establish a mapping relationship between the adjustment parameter and the personalized parameter. Among them, by establishing a mapping relationship, the adjustment parameter can be determined more quickly in the process of generating a three-dimensional digital human, so that the animated target digital human can be obtained more quickly.

[0292] The execution steps of the above method are mainly explained with the server as the execution subject. The following explains the three-dimensional digital human generation method provided in the embodiment of the present application with the client as the execution subject.

[0293] Optionally, the client may obtain target personalized parameters and the first voice.

[0294] Among them, the target personalized parameters include target identity information and target emotion information.

[0295] Optionally, for the client, the target personalized parameter may be actively input by the user, or may be selected by the user from a plurality of preset personalized parameters stored in the client, which is not specifically limited here.

[0296] The first voice may be a voice input by a user, or a voice converted from text input by a user, which is not specifically limited here.

[0297] Optionally, the client sends the target personalized parameter and the first voice to the server.

[0298] Optionally, the client may send the target personalized parameter and the first voice to the server, wherein the target personalized parameter may be sent by the client to the server after initialization, and the first voice may be sent to the server after the user inputs the voice.

[0299] The target personalized parameter and the first voice may be sent together, or may be sent sequentially according to the order in which they are acquired, and no specific limitation is made here.

[0300] Optionally, the client may receive the animated target digital human sent by the server.

[0301] It should be noted that after the client sends the target personalized parameters and the first voice to the server, the server can obtain the animated target digital human according to these two parameters and the general digital human generation model, and then send the animated target digital human to the client.

[0302] Optionally, in actual application scenarios, one server may correspond to multiple clients, and these clients may all execute the above steps, with the same server completing the generation of the corresponding three-dimensional digital human to be displayed, thereby sending the corresponding three-dimensional digital human to be displayed to the corresponding client according to the target personalized parameters and the source of the first voice.

[0303] Optionally, the client can render according to the animation target digital human.

[0304] In one embodiment, the first voice includes multiple voice frames; the method also includes: the client can obtain the target digital person and target deviation parameters sent by the server.

[0305] Optionally, the server may obtain the target digital human and multiple target deviation parameters through the animated target digital human, and may send the target digital human and the multiple target deviation parameters to the client, and the client renders the target digital human according to the target deviation parameters.

[0306] It should be noted that, during the process of displaying the three-dimensional digital human, the client can synchronously output the first voice, and the first voice can be output according to the frame correspondence between the first voice and the rendered three-dimensional digital human. Alternatively, the server can transmit the voice frame of the first voice corresponding to the frame while transmitting the three-dimensional digital human to be displayed and the target deviation parameters, for example: transmitting it to the client in the form of the WebSocket protocol, thereby realizing the output of the first voice, and no specific restrictions are made here.

[0307] It should be noted that the server may first send the target digital human to the client, and then send multiple target deviation parameters to the client in sequence according to the order of the voice frames.

[0308] For example: after receiving the target digital human, the client can display the target digital human in the display interface of the client. After receiving the target deviation parameter, the corresponding grid unit in the target digital human can be adjusted based on the target deviation parameter to obtain a rendered three-dimensional digital human. Each time the target deviation parameter is received, a rendering can be performed, and a rendered three-dimensional digital human can be obtained after each rendering. These rendered three-dimensional digital humans can be displayed in the display interface of the client in a corresponding order.

[0309] In one embodiment, the method further includes: sending target information to a server, the target information being used to indicate preset conditions, the preset conditions including at least one of the client's bandwidth configuration information, display requirements, and weights of facial areas, so that the server determines a target deviation parameter from a plurality of deviation parameters according to the preset conditions.

[0310] It should be noted that the target information may be actively sent by the client to the server, or may be sent by the client to the server after the server sends a request for obtaining the target information, and no specific limitation is made here.

[0311] After the client sends the target information indicating the preset condition to the server, the server may determine a target deviation parameter from the plurality of deviation parameters and send the target deviation parameter to the client.

[0312] It should be noted that the digital human sent by the server to the client can be an animation target digital human, and the server does not directly generate the animation target digital human, but rather obtains it after adjustment based on the animation initial digital human. The following explains an optional implementation process of converting the animation initial digital human into the animation target digital human provided in an embodiment of the present application.

[0313] Fig.12 Please refer to the corresponding relationship diagram between the initial animation digital human and the target animation digital human provided in the embodiment of this application. Fig.12 , Fig.12 The digital human on the upper side is the animation target digital human, and the digital human on the lower side is the animation initial digital human. Each frame of the animation initial digital human can be adjusted by the adjustment parameters corresponding to the frame to obtain its corresponding frame of animation target digital human; and the deviation of the head parameters between multiple frames of animation target digital humans is the aforementioned deviation parameter.

[0314] Fig.12 The animated initial digital human and the animated target digital human shown both have N frames.

[0315] Take the personalized parameter of male as an example, see Table 3. At this time, the target adjustment parameter is δ1, where δ1 includes N frame adjustment parameters. The value of each frame adjustment parameter has been calculated and stored as shown in formula (2) when the server establishes the mapping relationship. The value of each frame adjustment parameter is δ 11 To δ 1N .

[0316] The difference between two adjacent frames of the animated target digital human is a deviation parameter. The deviation parameter between two adjacent frames can be recorded as Ψ. The deviation parameter Ψ between two adjacent frames can include the positions of multiple head key points.

[0317] When the server sends the animated target digital human to the client, the target digital human of the first frame and the deviation parameter Ψ of each two adjacent frames may be sent. The deviation parameter Ψ may be sent directly, or the target deviation parameter may be determined from the deviation parameter Ψ of each two adjacent frames.

[0318] Next, another implementation process of the three-dimensional digital human generation method provided in the embodiment of the present application is explained from the perspective of data interaction between the client and the server.

[0319] Fig.13 For a schematic diagram of the interaction between the client and the server provided in the embodiment of the present application, please refer to Fig.13 , the method comprising:

[0320] S1310: The client obtains the target personalized parameters and the first voice.

[0321] S1320: The client sends the target personalized parameters and the first voice to the server.

[0322] S1330: The server determines a target adjustment parameter according to the target personalized parameter and the mapping relationship, and obtains an animated target digital human according to the target adjustment parameter and the animated initial digital human generated based on the first voice.

[0323] S1340: The server sends the animated target digital human to the client.

[0324] S1350: The client renders according to the animation target digital human.

[0325] It should be noted that the specific implementation process and principle of the above steps S1310-S1350 are the same as those in the previous embodiment and will not be repeated here.

[0326] For example, taking the live broadcast scene as an example, the client can be initialized, for example: load its own personalized configuration, determine the target personalized parameters according to the user's choice, etc. In addition, if the client has a three-dimensional digital human displayed last time, and the rendering scene this time is the same as the previous time, the three-dimensional digital human stored in the cache can be directly loaded, and the skeleton, map and configuration file can be loaded at the same time. Correspondingly, the server can also be initialized, and then establish a general digital human generation model based on the received first voice and target personalized parameters, and obtain the three-dimensional digital human to be displayed. Through the three-dimensional digital human of different frames, multiple deviation parameters can also be determined and sent to the client. The client can continue to render to complete the real-time reasoning of the digital human until the live broadcast of the digital human ends.

[0327] It should be understood that, although the steps in the above-mentioned flowcharts are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flowcharts may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0328] Based on the foregoing embodiments, the embodiments of the present application provide a three-dimensional digital human generation device, which includes the modules included and the units included in the modules, which can be implemented by a processor; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0329] Fig.14 This is a schematic diagram of the structure of the three-dimensional digital human generation device provided in the embodiment of the present application. Please refer to Fig.14 , a three-dimensional digital human generation device, the device comprising: an acquisition module 1410, a determination module 1420 and a generation module 1430;

[0330] The acquisition module 1410 is used to acquire target personalized parameters; the target personalized parameters include at least one of target identity information and target emotion information;

[0331] The determination module 1420 is used to determine the target adjustment parameter based on the target personalized parameter and the mapping relationship; the mapping relationship includes a one-to-one correspondence between multiple personalized parameters and multiple adjustment parameters, and the adjustment parameter is used to indicate the deviation of the head parameters between the animated digital human obtained by inputting the same speech into the personalized digital human generation model and the universal digital human generation model;

[0332] The generation module 1430 is used to generate an animation target digital human based on the target adjustment parameters and the animation initial digital human, where the animation initial digital human is generated based on the first voice input to the universal digital human generation model.

[0333] In one embodiment, the first speech includes multiple speech frames, the target adjustment parameters include multiple frame adjustment parameters, and each frame adjustment parameter corresponds to a speech frame; the generation module 1430 is specifically used to adjust each frame of the animated initial digital human corresponding to the speech frame based on a frame adjustment parameter corresponding to each speech frame, so as to generate an animated target digital human.

[0334] In one embodiment, each frame of the digital human includes multiple grid units in a three-dimensional space, and each frame adjustment parameter includes: an identifier of a grid unit to be adjusted among the multiple grid units and an offset value of each grid unit to be adjusted; a generation module 1430 is specifically used to adjust the three-dimensional space coordinates of the grid unit to be adjusted in the animated initial digital human corresponding to the voice frame based on the identifier of the grid unit to be adjusted and the offset value of each grid unit to be adjusted in each frame adjustment parameter, so as to generate an animated target digital human corresponding to the voice frame.

[0335] In one embodiment, the generation module 1430 is further used to send the animated target digital human to the client, so that the client renders the animated target digital human.

[0336] In one embodiment, the generation module 1430 is specifically used to send the target digital human and target deviation parameters corresponding to the first speech frame in the first speech to the client; wherein the target deviation parameters are at least part of multiple deviation parameters; each deviation parameter includes the deviation of the head parameters of the animated target digital human corresponding to adjacent speech frames in the animated target digital human.

[0337] In one embodiment, each deviation parameter includes: an identifier of a grid unit to be adjusted among multiple grid units of a target digital person corresponding to adjacent speech frames and an offset value of each grid unit to be adjusted; a generation module 1430 is specifically used to send multiple target deviation parameters to the client in sequence according to the time sequence of each speech frame in the first speech.

[0338] In one embodiment, the determination module 1420 is further used to determine a target deviation parameter from a plurality of deviation parameters according to preset conditions, wherein the preset conditions include at least one of the bandwidth configuration information of the client, the display requirements of the client, and the weight of the facial area corresponding to the deviation parameter; the generation module 1430 is further used to send the target deviation parameter to the client.

[0339] Fig.15 This is a structural diagram of the training device for the three-dimensional digital human model provided in the embodiment of the present application. Please refer to Fig.15 , a training device for a three-dimensional digital human model, the device comprising: a video acquisition module 1510, a data acquisition module 1520, a training module 1530 and a normalization module 1540;

[0340] The video acquisition module 1510 is used to acquire sample videos corresponding to different personalized parameters;

[0341] The data acquisition module 1520 is used to acquire corresponding sample data from the sample videos corresponding to each personalized parameter, where the sample data includes sample voice, sample user's emotion and sample user's head parameters;

[0342] The training module 1530 is used to train the audio-visual generation model according to the sample data corresponding to each personalized parameter, and obtain the personalized digital human generation model corresponding to each personalized parameter;

[0343] The normalization module 1540 is used to perform normalization processing on different personalized digital human generation models to obtain a universal digital human generation model.

[0344] In one embodiment, the data acquisition module 1520 is specifically used to establish a corresponding training digital human based on the sample video, and determine the head parameters of the sample user based on the head parameters of the training digital human; identify the emotions of the sample user through the facial parameters of the training digital human; determine the sample speech corresponding to each personalized parameter, as well as the sample user's emotions and the sample user's head parameters corresponding to each speech frame of the sample speech.

[0345] In one embodiment, the data acquisition module 1520 is also used to calculate the similarity between the facial parameters of the training digital human and the facial parameters of each pre-set emotional avatar model; based on the similarity, the emotional label of the training digital human is determined, and the emotional label includes the emotion and weight of the target emotional avatar model. The target emotional avatar model is the emotional avatar model corresponding to the similarity greater than or equal to the threshold, and the emotion of the sample user includes the emotional label of the training digital human.

[0346] In one embodiment, in the device, the sample video includes multiple video frames, the training digital human includes multiple frames, each frame of the training digital human corresponds one-to-one to a video frame, and the emotion of the sample user includes an emotion label of the training digital human in each frame.

[0347] In one embodiment, the normalization module 1540 is also used to input each of the P segments of sample speech into the personalized digital human generation model corresponding to M personalized parameters and the universal digital human generation model, to obtain P×M personalized animated digital humans and one universal animated digital human; wherein, one segment of speech corresponds to M personalized animated digital humans, and P and M are both positive integers greater than or equal to 2; respectively determine the deviations between the head parameters of each frame of each personalized animated digital human in the M personalized animated digital humans corresponding to each segment of the P segments of speech and the head parameters of each frame of the universal animated digital human; perform weighted averaging on the deviations between the head parameters corresponding to each segment of speech to obtain adjustment parameters, and establish a mapping relationship between the adjustment parameters and the personalized parameters.

[0348] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0349] It should be noted that in the embodiments of this application Fig.14 and Fig.15 The division of modules in the three-dimensional digital human generation device and the three-dimensional digital human training device shown is schematic, and is only a logical functional division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. It can also be implemented in the form of a combination of software and hardware.

[0350] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0351] Fig.16 For a schematic diagram of the structure of the computing device provided in the embodiment of the present application, please refer to Fig.16 , the present application embodiment provides a computing device, which may be the above-mentioned server, and is not specifically limited here. Its internal structure diagram may be as follows Fig.16 As shown. The computing device includes a processor 1620, a memory and a network interface 1640 connected via a system bus 1610. Among them, the processor 1620 of the computing device is used to provide computing and control capabilities. The memory of the computing device includes a non-volatile storage medium 1631 and an internal memory 1632. The non-volatile storage medium 1631 stores an operating system, a computer program and a database. The internal memory 1632 provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium 1631. The database of the computing device is used to store data. The network interface 1640 of the computing device is used to communicate with an external terminal through a network connection. For example, communication between a client and a server can be achieved through the network interface 1640. When the computer program is executed by the processor 1620, the above method is implemented.

[0352] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the above embodiment are implemented.

[0353] An embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps of the method provided in the above method embodiment.

[0354] Those skilled in the art will understand that Fig.16The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the computing device to which the scheme of the present application is applied. The specific computing device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0355] In one embodiment, the three-dimensional digital human generation device and the three-dimensional digital human model training device provided by the present application can be implemented in the form of a computer program. The computer program can be used in Fig.16 The computer program composed of each program module enables the processor to execute the steps of the method of each embodiment of the present application described in this specification.

[0356] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0357] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. For the sake of brevity, this article will not repeat them.

[0358] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0359] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0360] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0361] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0362] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0363] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0364] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0365] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0366] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0367] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0368] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for generating a three-dimensional digital human, characterized in that: The method comprises: Obtaining target personalized parameters; the target personalized parameters include at least one of target identity information and target emotion information; Determine a target adjustment parameter based on the target personalized parameter and the mapping relationship; the mapping relationship includes a one-to-one correspondence between a plurality of personalized parameters and a plurality of adjustment parameters, and the adjustment parameter is used to indicate the deviation of the head parameters between the animated digital human obtained by inputting the same speech into the personalized digital human generation model and the universal digital human generation model; Based on the target adjustment parameters and the animation initial digital human, an animation target digital human is generated, wherein the animation initial digital human is generated based on a first voice input to the universal digital human generation model.

2. The method according to claim 1, characterized in that The first speech includes a plurality of speech frames, the target adjustment parameters include a plurality of frame adjustment parameters, and each frame adjustment parameter corresponds to a speech frame; and generating an animated target digital human based on the target adjustment parameters and the animated initial digital human specifically includes: Based on a frame adjustment parameter corresponding to each speech frame, each frame of the initial animation digital human corresponding to the speech frame is adjusted to generate the target animation digital human.

3. The method according to claim 2, characterized in that The digital human of each frame includes a plurality of grid units in a three-dimensional space, and each of the frame adjustment parameters includes: an identifier of a grid unit to be adjusted among the plurality of grid units and an offset value of each of the grid units to be adjusted; The adjusting each frame of the initial animation digital human corresponding to the speech frame based on the frame adjustment parameter to generate the target animation digital human specifically includes: Based on the identifier of the grid unit to be adjusted in each frame adjustment parameter and the offset value of each grid unit to be adjusted, the three-dimensional space coordinates of the grid unit to be adjusted in the animated initial digital human corresponding to the voice frame are adjusted to generate the animated target digital human corresponding to the voice frame.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: The animated target digital person is sent to a client, so that the client renders the animated target digital person.

5. The method according to claim 4, characterized in that: Sending the animation target digital human to the client specifically includes: The target digital human and target deviation parameters corresponding to the first speech frame in the first speech are sent to the client; wherein the target deviation parameters are at least part of the multiple deviation parameters; each deviation parameter includes the deviation of the head parameters of the animated target digital human corresponding to adjacent speech frames in the animated target digital human.

6. The method according to claim 5, characterized in that Each of the deviation parameters includes: an identifier of a grid unit to be adjusted among the multiple grid units of the target digital person corresponding to adjacent speech frames and an offset value of each of the grid units to be adjusted; The step of sending the target digital person and the target deviation parameter corresponding to the first speech frame in the first speech to the client includes: The multiple target deviation parameters are sent to the client in sequence according to the time sequence of each speech frame in the first speech.

7. The method according to claim 5 or 6, characterized in that: The sending the target deviation parameter to the client includes: Determining a target deviation parameter from the plurality of deviation parameters according to a preset condition, wherein the preset condition includes at least one of bandwidth configuration information of the client, display requirements of the client, and a weight of a facial area corresponding to the deviation parameter; The target deviation parameter is sent to the client.

8. A training method for a three-dimensional digital human model, characterized in that: The method comprises: Obtain sample videos corresponding to different personalized parameters; Acquire corresponding sample data from the sample videos corresponding to the respective personalized parameters, wherein the sample data includes sample voice, sample user's emotion and sample user's head parameters; According to the sample data corresponding to each personalized parameter, the audio and video generation model is trained to obtain the personalized digital human generation model corresponding to each personalized parameter; The different personalized digital human generation models are normalized to obtain a universal digital human generation model.

9. The method according to claim 8, characterized in that The obtaining corresponding sample data from the sample videos corresponding to the individualized parameters includes: Establishing a corresponding training digital human according to the sample video, and determining the head parameters of the sample user according to the head parameters of the training digital human; Recognizing the emotion of the sample user through the facial parameters of the trained digital human; The sample speech corresponding to each personalized parameter, as well as the sample user's emotion and the sample user's head parameter corresponding to each speech frame of the sample speech are determined.

10. The method according to claim 9, characterized in that The step of identifying the emotion of the sample user by using the facial parameters of the trained digital human comprises: Calculating the similarity between the facial parameters of the training digital human and the facial parameters of each preset emotional avatar model; According to the similarity, the emotion label of the training digital human is determined, the emotion label includes the emotion and weight of the target emotion avatar model, the target emotion avatar model is the emotion avatar model corresponding to the similarity being greater than or equal to the threshold, and the emotion of the sample user includes the emotion label of the training digital human.

11. The method according to claim 10, characterized in that The sample video includes multiple video frames, the training digital human includes multiple frames, each frame of the training digital human corresponds one-to-one to the video frame, and the emotion of the sample user includes the emotion label of the training digital human in each frame.

12. The method according to claim 8, characterized in that After normalizing the different personalized digital human generation models to obtain a universal digital human generation model, the method further includes: Input each of the P segments of sample speech into the personalized digital human generation model corresponding to the M personalized parameters and the general digital human generation model, and obtain P×M personalized animated digital humans and one general animated digital human; wherein one segment of speech corresponds to M personalized animated digital humans, and P and M are both positive integers greater than or equal to 2; Determine the deviation between the head parameters of each frame of each personalized animated digital human in the M personalized animated digital humans corresponding to each speech segment in the P speech segments and the head parameters of each frame of the universal animated digital human; The deviations between the head parameters corresponding to each speech segment are processed by weighted average to obtain adjustment parameters, and a mapping relationship between the adjustment parameters and the personalized parameters is established.

Images