Digital human generation method and device, server and storage medium

Abstract

The invention relates to the field of artificial intelligence, and provides a digital person generation method and device, a server and a storage medium, and the method comprises the steps: obtaining a reference image of a target person of a to-be-generated digital person, and obtaining a preset posture parameter sequence; generating a plurality of target human body images according to the reference image and the plurality of preset posture parameters; generating a human body grid model of the target person according to the reference image and a preset human body model, and initializing parameters of a Gaussian body based on the human body grid model to obtain an initial digital person; and based on a three-dimensional Gaussian splash algorithm, optimizing a Gaussian body in the initial digital person by using a plurality of target human body images to obtain a target digital person. According to the method and the device, the digital person generation efficiency can be improved and the digital person generation cost can be reduced under the condition of ensuring the digital person quality, so that an insurance company can make the digital person for each insurance agent at relatively low cost and time.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, server and storage medium for generating a digital human. Background Art

[0002] With the development of new technology waves such as artificial intelligence and virtual reality, digital people represented by virtual anchors and virtual employees have successfully entered the public eye and shine in many fields such as film and television, games, media, culture and tourism, finance or insurance with diverse attitudes.

[0003] At present, the quality of digital humans generated based on 3D Gaussian Splatting (3DGS) is relatively good, but it requires multiple cameras to shoot high-quality multi-perspective human images of the characters and to build a three-dimensional human model of the characters. In addition, generating a digital human of a new character requires re-collecting high-quality multi-perspective human images of the new character and re-building the three-dimensional human model of the new character. The cost of generating digital humans is high and the efficiency is low.

[0004] For example, in the financial insurance field, if you want to create a digital human for each insurance agent to attract customers and maintain customer relationships, you need to use multiple cameras to shoot high-quality human images of each insurance agent from multiple perspectives and build a three-dimensional human model of each person, which is costly and takes a long time. Therefore, how to improve the efficiency of digital human generation and reduce the cost of digital human generation is an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present application provide a method, device, server and storage medium for generating a digital human, aiming to improve the efficiency of generating a digital human and reduce the cost of generating a digital human.

[0006] In a first aspect, an embodiment of the present application provides a method for generating a digital human, comprising:

[0007] Acquire a reference image of a target person of the digital human to be generated and acquire a preset posture parameter sequence, wherein the preset posture parameter sequence includes a plurality of different preset posture parameters, and different preset posture parameters correspond to different viewing angles;

[0008] Generate a plurality of target human images according to the reference image and the plurality of preset posture parameters, wherein the perspective and posture of the target person in each of the target human images are different;

[0009] Generate a human mesh model of the target person according to the reference image and the preset human body model, and initialize the parameters of the Gaussian body based on the human body mesh model to obtain an initial digital human;

[0010] Based on a three-dimensional Gaussian splashing algorithm, a plurality of target human images are used to optimize the Gaussian body in the initial digital human to obtain a target digital human.

[0011] In a second aspect, the embodiment of the present application further provides a digital human generation device, the digital human generation device comprising:

[0012] An acquisition module, used to acquire a reference image of a target person to be generated as a digital human and to acquire a preset posture parameter sequence, wherein the preset posture parameter sequence includes a plurality of different preset posture parameters, and different preset posture parameters correspond to different viewing angles;

[0013] A generating module, configured to generate a plurality of target human images according to the reference image and the plurality of preset posture parameters, wherein the viewing angle and posture of the target person in each of the target human images are different;

[0014] An initialization module, used to generate a human mesh model of the target person according to the reference image and a preset human body model, and initialize the parameters of the Gaussian body based on the human mesh model to obtain an initial digital human;

[0015] The digital human optimization module is used to optimize the Gaussian body in the initial digital human using a plurality of target human body images based on a three-dimensional Gaussian splash algorithm to obtain a target digital human.

[0016] In a third aspect, an embodiment of the present application further provides a server, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the digital human generation method as described in the first aspect are implemented.

[0017] In a fourth aspect, an embodiment of the present application further provides a storage medium for computer-readable storage, wherein a computer program is stored on the storage medium, wherein when the computer program is executed by a processor, the steps of the digital human generation method as described in the first aspect are implemented.

[0018] The embodiments of the present application provide a method, device, server and storage medium for generating a digital human. By using a reference image of a target person and a plurality of different preset posture parameters, human body images of the target person in different postures from multiple perspectives can be generated. There is no need to use multiple cameras to shoot human body images of the person from multiple perspectives, which reduces the data requirements for generating a digital human. By using the reference image and the preset human body model, a human body mesh model of the target person can be accurately and quickly generated without collecting additional data. Finally, the parameters of the Gaussian body are initialized based on the human body mesh model to obtain an initial digital human. Based on a three-dimensional Gaussian splash algorithm, the human body images of different postures from multiple perspectives are used to optimize the Gaussian body in the initial digital human, so that a digital human with good quality can be obtained. Therefore, while ensuring the quality of the digital human, the efficiency of generating the digital human can be improved and the cost of generating the digital human can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 It is a schematic diagram of the steps of a method for generating a digital human provided in an embodiment of the present application;

[0021] Figure 2 is an example diagram of the viewing angle corresponding to the preset posture parameters in the embodiment of the present application;

[0022] Figure 3 is a schematic block diagram of the structure of a digital human generation device provided in an embodiment of the present application;

[0023] Figure 4 yes Figure 3 A schematic block diagram of submodules of a digital human generation device;

[0024] Figure 5 It is a schematic block diagram of the structure of a server provided in an embodiment of the present application.

[0025] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0028] With the development of new technologies such as artificial intelligence and virtual reality, digital humans represented by virtual anchors and virtual employees have successfully entered the public eye and shined in many fields such as film and television, games, media, culture and tourism, finance or insurance. At present, the quality of digital humans generated based on 3D Gaussian Splatting (3DGS) is relatively good, but it requires multi-camera shooting of multi-view high-quality human images of characters and construction of 3D human models of characters. In addition, the generation of digital humans of new characters requires the re-collection of multi-view high-quality human images of new characters and the reconstruction of 3D human models of new characters. The cost of generating digital humans is high and the efficiency is low.

[0029] To solve the above problems, the embodiments of the present application provide a method, device, server and storage medium for generating a digital human. The method can generate human images of the target person in different postures from multiple perspectives through a reference image of the target person and a plurality of different preset posture parameters set in advance, and does not require multiple cameras to shoot human images of the person from multiple perspectives, thereby reducing the data requirements for generating a digital human. Moreover, through the reference image and the preset human model, the human body mesh model of the target person can be accurately and quickly generated without collecting additional data. Finally, the parameters of the Gaussian body are initialized based on the human body mesh model to obtain an initial digital person, and based on a three-dimensional Gaussian splash algorithm, the human body images of different postures from multiple perspectives are used to optimize the Gaussian body in the initial digital person, so that a digital person with good quality can be obtained, thereby ensuring the quality of the digital person and improving the efficiency of generating the digital person and reducing the cost of generating the digital person.

[0030] For example, in the field of finance and insurance, insurance agents can upload their own two-dimensional human images as reference images to the server through terminal devices. The server generates multiple target human images of the insurance agent based on the reference image and multiple preset posture parameters, and generates a human mesh model of the insurance agent based on the reference image and the preset human model. Then, the parameters of the Gaussian body are initialized based on the human mesh model to obtain the initial digital human of the insurance agent. Based on the three-dimensional Gaussian splash algorithm, the Gaussian body in the initial digital human is optimized using the multiple target human images of the insurance agent to obtain the target digital human of the insurance agent. This allows insurance companies to produce digital humans for each insurance agent at a lower cost and time.

[0031] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. The digital human generation method can be applied to a server, which can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0032] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0033] See also Figure 1 , Figure 1 It is a schematic diagram of the steps of a method for generating a digital human provided in an embodiment of the present application.

[0034] like Figure 1 As shown, the digital human generation method includes steps S101 to S104.

[0035] Step S101, obtaining a reference image of a target person of a digital human to be generated and obtaining a preset posture parameter sequence, wherein the preset posture parameter sequence includes a plurality of different preset posture parameters, and different preset posture parameters correspond to different viewing angles.

[0036] In this embodiment, the target person may be any real person for whom a digital person needs to be generated, for example, an insurance agent of an insurance company. The reference image may be a half-length front view or a full-length front view of the target person, which is not specifically limited in this embodiment. The preset posture parameter includes position information of multiple human joints, and the position information of multiple human joints can represent the posture of the human body.

[0037] In some embodiments, the angle difference between the viewing angles corresponding to any two adjacent preset posture parameters in the preset posture parameter sequence is the preset angle difference. The preset angle difference can be set based on actual conditions, and the embodiment of the present application does not specifically limit this. For example, the preset angle difference is 20°, 30°, 40°, or 45°, etc.

[0038] For example, if the preset angle difference is 45°, the preset posture parameter sequence includes 8 preset posture parameters, that is, the preset posture parameter sequence is [Pose1, Pose2, Pose3, Pose4, Pose5, Pose6, Pose7, Pose8]. Figure 2 As shown, the viewing angle corresponding to Pose1 is the due north direction of the photographed person 10. The viewing angle corresponding to Pose2 is the northeast direction of the photographed person 10. The viewing angle corresponding to Pose3 is the due east direction of the photographed person 10. The viewing angle corresponding to Pose4 is the southeast direction of the photographed person 10. The viewing angle corresponding to Pose5 is the due south direction of the photographed person 10. The viewing angle corresponding to Pose6 is the southwest direction of the photographed person 10. The viewing angle corresponding to Pose7 is the due west direction of the photographed person 10. The viewing angle corresponding to Pose8 is the northwest direction of the photographed person 10.

[0039] For another example, if the preset angle difference is 30°, the preset posture parameter sequence includes 12 preset posture parameters, that is, the preset posture parameter sequence is [Pose1, Pose2, Pose3, Pose4, Pose5, Pose6, Pose7, Pose8, Pose9, Pose 10 , Pose 11 , Pose 12], the viewing angle corresponding to Pose1 is the due north direction of the photographed person 10. The viewing angle corresponding to Pose2 is 30° north-east of the photographed person 10. The viewing angle corresponding to Pose3 is 60° north-east of the photographed person 10. The viewing angle corresponding to Pose4 is the due east direction of the photographed person 10. The viewing angle corresponding to Pose5 is 30° east-south of the photographed person 10. The viewing angle corresponding to Pose6 is 60° east-south of the photographed person 10. The viewing angle corresponding to Pose7 is the due south direction of the photographed person 10. The viewing angle corresponding to Pose8 is 30° south-west of the photographed person 10. The viewing angle corresponding to Pose9 is 60° south-west of the photographed person 10. Pose 10 The corresponding viewing angle is due west of the photographed person 10. 11 The corresponding viewing angle is 30° west of the photographed person 10 by north. 12 The corresponding viewing angle is 60° west of the photographed person 10 by north.

[0040] In some embodiments, a camera can be used to photograph the person being photographed at different viewing angles to obtain human body images at different viewing angles, and then a posture estimation model can be used to estimate the posture of the person being photographed in the human body images to obtain corresponding posture parameters, and then the obtained posture parameters are arranged in order of the size of the viewing angle to obtain a preset posture parameter sequence.

[0041] For example, the posture estimation model is used to estimate the posture of the person being photographed in the first human image, and the posture parameter Pose1 is obtained. The first human image is obtained by photographing the person being photographed by the camera in the due north direction of the person being photographed. The posture estimation model is used to estimate the posture of the person being photographed in the second human image, and the posture parameter Pose2 is obtained. The second human image is obtained by photographing the person being photographed by the camera in the northeast direction of the person being photographed. The posture estimation model is used to estimate the posture of the person being photographed in the third human image, and the posture parameter Pose3 is obtained. The third human image is obtained by photographing the person being photographed by the camera in the due east direction of the person being photographed. The posture estimation model is used to estimate the posture of the person being photographed in the fourth human image, and the posture parameter Pose4 is obtained. The fourth human image is obtained by photographing the person being photographed by the camera in the southeast direction of the person being photographed.

[0042] The pose estimation model is used to estimate the pose of the person in the fifth human image, and the pose parameter Pose5 is obtained. The fifth human image is obtained by photographing the person in the south of the person. The pose estimation model is used to estimate the pose of the person in the sixth human image, and the pose parameter Pose6 is obtained. The sixth human image is obtained by photographing the person in the southwest of the person. The pose estimation model is used to estimate the pose of the person in the seventh human image, and the pose parameter Pose7 is obtained. The seventh human image is obtained by photographing the person in the west of the person. The pose estimation model is used to estimate the pose of the person in the eighth human image, and the pose parameter Pose8 is obtained. The eighth human image is obtained by photographing the person in the northwest of the person.

[0043] Then, the obtained Pose1, Pose2, Pose3, Pose4, Pose5, Pose6, Pose7 and Pose8 are sorted in the order of the size of the viewing angle, and the preset posture parameter sequence [Pose1, Pose2, Pose3, Pose4, Pose5, Pose6, Pose7, Pose8] can be obtained.

[0044] Step S102: Generate multiple target human images according to the reference image and multiple preset posture parameters, wherein the target person in each target human image has a different perspective and posture.

[0045] In this embodiment, a plurality of target human images may be generated by using a reference image and a plurality of preset posture parameters through a preset latent diffusion model, wherein a target human image is obtained by performing diffusion processing on a reference image and a preset posture parameter through a preset latent diffusion model.

[0046] In some embodiments, generating multiple target human body images using a reference image and multiple preset posture parameters through a preset latent diffusion model may include: selecting a preset posture parameter from a preset posture parameter sequence; inputting the reference image and the selected preset posture parameter into the preset latent diffusion model for diffusion processing to obtain the target human body image; when the number of target human body images generated is less than the number of preset posture parameters in the preset posture parameter sequence, returning to execute the step of selecting a preset posture parameter from the preset posture parameter sequence, and each time the selected preset posture parameter is different; when the number of target human body images generated is equal to the number of preset posture parameters in the preset posture parameter sequence, stopping inputting data into the preset latent diffusion model.

[0047] In some embodiments, the preset latent diffusion model is the Animate Anyone model or the Stable Diffusion model containing the control network (ControlNet). This embodiment can accurately generate a target human body image corresponding to a perspective and a posture based on a reference image and preset posture parameters through the Animate Anyone model or the Stable Diffusion model containing the control network (ControlNet), and can ensure the three-dimensional consistency between multiple target human body images.

[0048] Step S103: Generate a human mesh model of the target person according to the reference image and the preset human body model, and initialize the parameters of the Gaussian body based on the human body mesh model to obtain an initial digital human.

[0049] In this embodiment, the preset human body model can be set based on actual conditions, and the embodiment of the present application does not specifically limit this. For example, the preset human body model is a SMPL (Skinned Multi-Person Linear) model.

[0050] In some embodiments, generating a human body mesh model of a target person according to a reference image and a preset human body model may include: determining the posture parameters and shape parameters of the target person according to the reference image; and generating the human body mesh model of the target person according to the posture parameters, shape parameters and the preset human body model. In this embodiment, the posture parameters and shape parameters of the target person can be determined through the reference image of the target person, so that the human body mesh model of the target person can be generated by querying the preset human body model through the posture parameters and shape parameters of the target person, without the need to collect the human body data of the target person, thereby improving the generation efficiency of the human body mesh model and reducing the cost.

[0051] In some embodiments, determining the posture parameters and shape parameters of the target person according to the reference image may include: estimating the posture of the target person in the reference image by a preset posture estimation model to obtain the posture parameters of the target person; estimating the shape of the target person in the reference image by a preset shape estimation model to obtain the shape parameters of the target person. The posture estimation model and the shape estimation model are pre-trained neural network models.

[0052] In some embodiments, the parameters of the Gaussian body include position, size, direction, covariance, color coefficient and opacity. Initializing the parameters of the Gaussian body based on the human body mesh model to obtain the initial digital human includes: uniformly sampling the surface of the human body mesh model to obtain multiple sampling positions, and the multiple sampling positions are uniformly distributed on the surface of the human body mesh model; using the sampling position as the position of the Gaussian body, and according to the human body mesh model, initializing the size and direction of the Gaussian body and randomly initializing the covariance, color coefficient and opacity of the Gaussian body to obtain the initial digital human.

[0053] Step S104: Based on the three-dimensional Gaussian splash algorithm, multiple target human body images are used to optimize the Gaussian body in the initial digital human to obtain the target digital human.

[0054] This embodiment uses a reference image of a target person and a plurality of different preset posture parameters to generate human images of the target person in different postures from multiple perspectives. It is not necessary to use multiple cameras to shoot human images of the person from multiple perspectives, which reduces the data requirements for digital human generation. In addition, a human mesh model of the target person can be accurately and quickly generated through the reference image and the preset human model, without the need to collect additional data. Finally, the parameters of the Gaussian body are initialized based on the human mesh model to obtain an initial digital person. Based on a three-dimensional Gaussian splash algorithm, the human body images of different postures from multiple perspectives are used to optimize the Gaussian body in the initial digital person, so that a digital person of good quality can be obtained, thereby ensuring the quality of the digital person while improving the efficiency of digital human generation and reducing the cost of digital human generation.

[0055] In some embodiments, based on the three-dimensional Gaussian splash algorithm, multiple target human images are used to optimize the Gaussian body in the initial digital human, and obtaining the target digital human may include: selecting a target human body image from multiple target human body images as a supervisory image; determining the deformation field of the three-dimensional standard space mapped to the two-dimensional view space according to the preset posture parameters corresponding to the supervisory image; projecting the Gaussian body in the initial digital human to the two-dimensional view space according to the deformation field to obtain a rendered image; determining the target loss value according to the difference between the supervisory image and the rendered image; when the target loss value is greater than the preset target loss value, optimizing the Gaussian body in the initial digital human according to the target loss value to update the initial digital human, and returning to execute the step of selecting a target human body image from multiple target human body images as a supervisory image; when the target loss value is less than or equal to the preset target loss value, stopping optimizing the Gaussian body in the initial digital human, and determining the latest initial digital human as the target digital human. This embodiment uses multi-view and multi-posture human body images of the target person to optimize the Gaussian body in the initial digital human, so as to improve the quality of the optimized target digital human. Optimizing the Gaussian body in the initial digital human may include optimizing the position, size, direction, covariance, color coefficient and opacity of the Gaussian body.

[0056] In some embodiments, determining a target loss value based on the difference between a supervised image and a rendered image may include: determining the grayscale difference between each pixel in the supervised image and the corresponding pixel in the rendered image; calculating the mean square error based on the grayscale difference between each pixel in the supervised image and the corresponding pixel in the rendered image; determining the D-SSIM (Depth-aware Structural Similarity) index between the supervised image and the rendered image; and determining the target loss value based on the mean square error and the D-SSIM index.

[0057] In some embodiments, when the target loss value is greater than a preset target loss value, the optimization times of the initial digital human are obtained; when the optimization times reach a preset first time, the optimization of the Gaussian body in the initial digital human is stopped, and the latest initial digital human is determined as the target digital human; when the optimization times are less than the first time, the Gaussian body in the initial digital human is optimized according to the target loss value to update the initial digital human, and the step of selecting a target human body image from multiple target human body images as a supervision image is returned to be executed. Among them, the first time number can be set based on actual conditions, and the embodiments of the present application do not specifically limit this. This embodiment can avoid optimizing the Gaussian body for a long time and save computing resources.

[0058] In some embodiments, when the target loss value is greater than a preset target loss value, the optimization times of the initial digital human and the selection times of each target human image are obtained; when the optimization times reach a first number and the selection times of each target human image reach a preset second number, the optimization of the Gaussian body in the initial digital human is stopped, and the latest initial digital human is determined as the target digital human; when the optimization times are less than the first number and / or there is at least one target human image whose selection times are less than the second number, the Gaussian body in the initial digital human is optimized according to the target loss value to update the initial digital human, and the step of selecting a target human image from multiple target human images as a supervision image is returned to be executed. This embodiment can avoid optimizing the Gaussian body for a long time and save computing resources.

[0059] In some embodiments, when the target loss value is greater than the preset target loss value, and the optimization times of the initial digital human reaches the preset first number and / or there is at least one target human image whose number of selections is less than the second number, the latest initial digital human is determined as the candidate digital human; a new preset posture parameter sequence is obtained, and a new plurality of target human images are generated according to the reference image and the new preset posture parameter sequence; based on the three-dimensional Gaussian splash algorithm, the Gaussian bodies in the candidate digital human are optimized using the new plurality of target human images to obtain the target digital human. This embodiment can further improve the quality of the digital human by regenerating a new plurality of target human images to continue the optimization when the initial digital human has not been optimized for a long time.

[0060] See also Figure 3 , Figure 3 It is a schematic block diagram of the structure of a digital human generation device provided in an embodiment of the present application.

[0061] like Figure 3 As shown, the digital human generation device 100 includes:

[0062] An acquisition module 110 is used to acquire a reference image of a target person to be generated as a digital human and to acquire a preset posture parameter sequence, wherein the preset posture parameter sequence includes a plurality of different preset posture parameters, and different preset posture parameters correspond to different viewing angles;

[0063] A generating module 120, configured to generate a plurality of target human images according to the reference image and the plurality of preset posture parameters, wherein the viewing angle and posture of the target person in each of the target human images are different;

[0064] An initialization module 130 is used to generate a human mesh model of the target person according to the reference image and the preset human body model, and initialize the parameters of the Gaussian body based on the human body mesh model to obtain an initial digital human;

[0065] The digital human optimization module 140 is used to optimize the Gaussian body in the initial digital human using a plurality of target human body images based on a three-dimensional Gaussian splash algorithm to obtain a target digital human.

[0066] In some embodiments, the generating module 120 is further configured to:

[0067] Selecting a preset posture parameter from the preset posture parameter sequence;

[0068] Inputting the reference image and the selected preset posture parameters into a preset potential diffusion model for diffusion processing to obtain a target human body image;

[0069] When the number of the generated target human body images is less than the number of preset posture parameters in the preset posture parameter sequence, returning to the step of selecting a preset posture parameter from the preset posture parameter sequence, and each selected preset posture parameter is different;

[0070] When the number of the generated target human body images is equal to the number of preset posture parameters in the preset posture parameter sequence, inputting data into the preset latent diffusion model is stopped.

[0071] In some embodiments, the preset latent diffusion model is an Animate Anyone model or a Stable Diffusion model containing a control network.

[0072] In some embodiments, the initialization module 130 is further configured to:

[0073] Determining the posture parameters and shape parameters of the target person according to the reference image;

[0074] A human body mesh model of the target person is generated according to the posture parameters, the shape parameters and a preset human body model.

[0075] In some embodiments, Figure 4 As shown, the digital human optimization module 140 includes:

[0076] A selection submodule 141, configured to select one of the target human images as a supervision image from the plurality of target human images;

[0077] A determination submodule 142, configured to determine a deformation field of a three-dimensional standard space mapped to a two-dimensional view space according to the preset posture parameters corresponding to the supervision image;

[0078] A rendering submodule 143 is used to project the Gaussian body in the initial digital human into a two-dimensional viewing space according to the deformation field to obtain a rendered image;

[0079] The determination submodule 142 is further used to determine a target loss value according to a difference between the supervision image and the rendering image;

[0080] An optimization submodule 144 is used for optimizing the Gaussian body in the initial digital human according to the target loss value when the target loss value is greater than a preset target loss value, so as to update the initial digital human, and returning to execute the step of selecting a target human image from the plurality of target human images as a supervision image;

[0081] The stopping module is used to stop optimizing the Gaussian body in the initial digital human when the target loss value is less than or equal to the preset target loss value, and determine the latest initial digital human as the target digital human.

[0082] In some embodiments, the acquisition module 110 is further used to acquire the optimization times of the initial digital human when the target loss value is greater than a preset target loss value;

[0083] The optimization submodule 145 is further used to stop optimizing the Gaussian body in the initial digital human when the optimization times reaches a preset first time, and determine the latest initial digital human as the target digital human; and when the optimization times are less than the first time, optimize the Gaussian body in the initial digital human according to the target loss value to update the initial digital human, and return to execute the step of selecting a target human image from the multiple target human images as a supervision image.

[0084] In some embodiments, the acquisition module 110 is further used to acquire the number of optimizations of the initial digital human and the number of selections of each of the target human images when the target loss value is greater than a preset target loss value;

[0085] The optimization submodule 145 is further configured to stop optimizing the Gaussian body in the initial digital human and determine the latest initial digital human as the target digital human when the optimization times reaches the first times and the selection times of each target human image reaches the preset second times; and to optimize the Gaussian body in the initial digital human according to the target loss value to update the initial digital human, and return to execute the step of selecting one target human image from the multiple target human images as a supervision image when the optimization times is less than the first times and / or there is at least one target human image that is selected less than the second times.

[0086] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned digital human generation method embodiment, and will not be repeated here.

[0087] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 5 Run on the server shown.

[0088] See also Figure 5 , Figure 5 It is a schematic block diagram of the structure of a server provided in an embodiment of the present application.

[0089] like Figure 5 As shown, the server includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a storage medium and an internal memory.

[0090] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any digital human generation method.

[0091] The processor is used to provide computing and control capabilities to support the operation of the entire server.

[0092] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the server to which the solution of the present application is applied. The specific server may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0093] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0094] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0095] Acquire a reference image of a target person of the digital human to be generated and acquire a preset posture parameter sequence, wherein the preset posture parameter sequence includes a plurality of different preset posture parameters, and different preset posture parameters correspond to different viewing angles;

[0096] Generate a plurality of target human images according to the reference image and the plurality of preset posture parameters, wherein the perspective and posture of the target person in each of the target human images are different;

[0097] Generate a human mesh model of the target person according to the reference image and the preset human body model, and initialize the parameters of the Gaussian body based on the human body mesh model to obtain an initial digital human;

[0098] Based on a three-dimensional Gaussian splashing algorithm, a plurality of target human images are used to optimize the Gaussian body in the initial digital human to obtain a target digital human.

[0099] In some embodiments, when the processor generates multiple target human body images according to the reference image and the multiple preset posture parameters, it is used to implement:

[0100] Selecting a preset posture parameter from the preset posture parameter sequence;

[0101] Inputting the reference image and the selected preset posture parameters into a preset potential diffusion model for diffusion processing to obtain a target human body image;

[0102] When the number of the generated target human body images is less than the number of preset posture parameters in the preset posture parameter sequence, returning to the step of selecting a preset posture parameter from the preset posture parameter sequence, and each selected preset posture parameter is different;

[0103] When the number of the generated target human body images is equal to the number of preset posture parameters in the preset posture parameter sequence, inputting data into the preset latent diffusion model is stopped.

[0104] In some embodiments, the preset latent diffusion model is an Animate Anyone model or a Stable Diffusion model containing a control network.

[0105] In some embodiments, when the processor generates the human body mesh model of the target person according to the reference image and the preset human body model, it is used to implement:

[0106] Determining the posture parameters and shape parameters of the target person according to the reference image;

[0107] A human body mesh model of the target person is generated according to the posture parameters, the shape parameters and a preset human body model.

[0108] In some embodiments, when the processor optimizes the Gaussian body in the initial digital human using a plurality of target human body images based on a three-dimensional Gaussian splash algorithm, the processor is used to implement:

[0109] Selecting a target human image from a plurality of target human images as a supervision image;

[0110] Determining a deformation field of a three-dimensional standard space mapped to a two-dimensional view space according to the preset posture parameters corresponding to the supervision image;

[0111] According to the deformation field, projecting the Gaussian body in the initial digital human into a two-dimensional view space to obtain a rendered image;

[0112] Determining a target loss value according to a difference between the supervision image and the rendered image;

[0113] When the target loss value is greater than a preset target loss value, optimizing the Gaussian body in the initial digital human according to the target loss value to update the initial digital human, and returning to execute the step of selecting one target human image from the plurality of target human images as a supervision image;

[0114] When the target loss value is less than or equal to the preset target loss value, the optimization of the Gaussian body in the initial digital human is stopped, and the latest initial digital human is determined as the target digital human.

[0115] In some embodiments, after determining the target loss value according to the difference between the supervision image and the rendered image, the processor is further configured to implement:

[0116] When the target loss value is greater than a preset target loss value, obtaining the number of optimization times of the initial digital human;

[0117] When the number of optimizations reaches a preset first number, the optimization of the Gaussian body in the initial digital human is stopped, and the latest initial digital human is determined as the target digital human;

[0118] When the optimization times are less than the first times, the Gaussian body in the initial digital human is optimized according to the target loss value to update the initial digital human, and the step of selecting a target human image from the multiple target human images as a supervision image is returned to be executed.

[0119] In some embodiments, after determining the target loss value according to the difference between the supervision image and the rendered image, the processor is further configured to implement:

[0120] When the target loss value is greater than a preset target loss value, obtaining the number of optimizations of the initial digital human and the number of selections of each target human image;

[0121] When the optimization times reaches the first times, and the selection times of each of the target human images reaches the preset second times, stop optimizing the Gaussian body in the initial digital human, and determine the latest initial digital human as the target digital human;

[0122] When the optimization times are less than the first times and / or there is at least one of the target human images whose selected times are less than the second times, the Gaussian body in the initial digital human is optimized according to the target loss value to update the initial digital human, and the step of selecting one target human image from the multiple target human images as a supervision image is returned to be executed.

[0123] It should be noted that those skilled in the art can clearly understand that, for the sake of convenience and brevity of description, the specific working process of the server described above can refer to the corresponding process in the aforementioned digital human generation method embodiment, and will not be repeated here.

[0124] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a server (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0125] An embodiment of the present application further provides a storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the digital human generation method of the present application.

[0126] The storage medium may be volatile or non-volatile. The storage medium may be an internal storage unit of the server described in the foregoing embodiment, such as a hard disk or memory of the server. The storage medium may also be an external storage device of the server, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the server.

[0127] Furthermore, the storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0128] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0129] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0130] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0131] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating a digital human, characterized in that: include: Acquire a reference image of a target person of the digital human to be generated and acquire a preset posture parameter sequence, wherein the preset posture parameter sequence includes a plurality of different preset posture parameters, and different preset posture parameters correspond to different viewing angles; Generate a plurality of target human images according to the reference image and the plurality of preset posture parameters, wherein the perspective and posture of the target person in each of the target human images are different; Generate a human mesh model of the target person according to the reference image and the preset human body model, and initialize the parameters of the Gaussian body based on the human body mesh model to obtain an initial digital human; Based on a three-dimensional Gaussian splashing algorithm, a plurality of target human images are used to optimize the Gaussian body in the initial digital human to obtain a target digital human.

2. The method for generating a digital human according to claim 1, characterized in that: The step of generating a plurality of target human body images according to the reference image and the plurality of preset posture parameters comprises: Selecting a preset posture parameter from the preset posture parameter sequence; Inputting the reference image and the selected preset posture parameters into a preset potential diffusion model for diffusion processing to obtain a target human body image; When the number of the generated target human body images is less than the number of preset posture parameters in the preset posture parameter sequence, returning to the step of selecting a preset posture parameter from the preset posture parameter sequence, and each selected preset posture parameter is different; When the number of the generated target human body images is equal to the number of preset posture parameters in the preset posture parameter sequence, inputting data into the preset latent diffusion model is stopped.

3. The method for generating a digital human according to claim 2, characterized in that: The preset potential diffusion model is the Animate Anyone model or the Stable Diffusion model containing a control network.

4. The method for generating a digital human according to claim 1, characterized in that: The step of generating a human body mesh model of the target person according to the reference image and the preset human body model comprises: Determining the posture parameters and shape parameters of the target person according to the reference image; A human body mesh model of the target person is generated according to the posture parameters, the shape parameters and a preset human body model.

5. The method for generating a digital human according to any one of claims 1 to 4, characterized in that: The method of optimizing the Gaussian body in the initial digital human using a plurality of target human body images based on a three-dimensional Gaussian splashing algorithm includes: Selecting a target human image from a plurality of target human images as a supervision image; Determining a deformation field of a three-dimensional standard space mapped to a two-dimensional view space according to the preset posture parameters corresponding to the supervision image; According to the deformation field, projecting the Gaussian body in the initial digital human into a two-dimensional view space to obtain a rendered image; Determining a target loss value according to a difference between the supervision image and the rendered image; When the target loss value is greater than a preset target loss value, optimizing the Gaussian body in the initial digital human according to the target loss value to update the initial digital human, and returning to execute the step of selecting one target human image from the plurality of target human images as a supervision image; When the target loss value is less than or equal to the preset target loss value, the optimization of the Gaussian body in the initial digital human is stopped, and the latest initial digital human is determined as the target digital human.

6. The method for generating a digital human according to claim 5, characterized in that: After determining the target loss value according to the difference between the supervision image and the rendering image, the method further includes: When the target loss value is greater than a preset target loss value, obtaining the number of optimization times of the initial digital human; When the number of optimizations reaches a preset first number, the optimization of the Gaussian body in the initial digital human is stopped, and the latest initial digital human is determined as the target digital human; When the optimization times are less than the first times, the Gaussian body in the initial digital human is optimized according to the target loss value to update the initial digital human, and the step of selecting a target human image from the multiple target human images as a supervision image is returned to be executed.

7. The method for generating a digital human according to claim 5, characterized in that: After determining the target loss value according to the difference between the supervision image and the rendering image, the method further includes: When the target loss value is greater than a preset target loss value, obtaining the number of optimizations of the initial digital human and the number of selections of each target human image; When the optimization times reaches the first times, and the selection times of each of the target human images reaches the preset second times, stop optimizing the Gaussian body in the initial digital human, and determine the latest initial digital human as the target digital human; When the optimization times are less than the first times and / or there is at least one of the target human images whose selected times are less than the second times, the Gaussian body in the initial digital human is optimized according to the target loss value to update the initial digital human, and the step of selecting one target human image from the multiple target human images as a supervision image is returned to be executed.

8. A digital human generation device, characterized in that: The digital human generation device comprises: An acquisition module, used to acquire a reference image of a target person to be generated as a digital human and to acquire a preset posture parameter sequence, wherein the preset posture parameter sequence includes a plurality of different preset posture parameters, and different preset posture parameters correspond to different viewing angles; A generating module, configured to generate a plurality of target human images according to the reference image and the plurality of preset posture parameters, wherein the viewing angle and posture of the target person in each of the target human images are different; An initialization module, used to generate a human mesh model of the target person according to the reference image and a preset human body model, and initialize the parameters of the Gaussian body based on the human mesh model to obtain an initial digital human; The digital human optimization module is used to optimize the Gaussian body in the initial digital human using a plurality of target human body images based on a three-dimensional Gaussian splash algorithm to obtain a target digital human.

9. A server, characterized in that: The server includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the digital human generation method according to any one of claims 1 to 7 are implemented.

10. A storage medium for computer-readable storage, characterized in that: The storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the digital human generation method according to any one of claims 1 to 7 are implemented.

Images