A single-image-based three-dimensional human model generation method and device

Abstract

This application relates to a method and apparatus for generating a 3D human body model based on a single image. The method includes: acquiring a single image containing a complete target person; generating a dance-driven video consistent with the target person's posture using a trained preset motion generation model based on the image and posture parameters extracted from it; extracting the target person's posture-driven information from the dance-driven video using a preset visual processing model; and using the posture-driven information as the basis for modeling posture-driven deformation, generating a 3D human body model based on the dance-driven video using a preset digital human modeling algorithm. This application cleverly utilizes the person's posture parameters extracted from a single image to generate a posture-consistent driving video. Based on this posture-consistent video, its posture-driven information is used to complete the construction of a high-quality dynamic model, solving the problem of how to perform high-quality dynamic reconstruction of a 3D human body based on a single image.

Description

Technical Field

[0001] This application relates to the field of computer graphics, and in particular to a method and apparatus for generating a three-dimensional human body model based on a single image. Background Technology

[0002] Traditional 3D digital human modeling methods typically rely on multi-view image acquisition, 3D scanning equipment (such as laser scanners and structured light devices), or complex motion capture systems, which are costly and cumbersome, making them difficult to popularize among ordinary users. To lower the modeling threshold, research has emerged on reconstructing 3D human bodies based on single images. Early solutions often used parametric human models (such as SMPL and SMPL-X) for fitting, but their geometric representation is limited by predefined templates, making it difficult to reproduce personalized details (such as clothing folds, hairstyles, and accessories). Subsequently, Neural Radiation Field (NeRF) and its variants were introduced into human body modeling. Although high-fidelity rendering can be achieved, training is time-consuming, inference speed is slow, and it is sensitive to pose changes, making it difficult to support real-time interaction.

[0003] In recent years, 3DGS, as an emerging explicit 3D representation technology, has significantly improved rendering efficiency while maintaining high-quality rendering and has been successfully applied to static scene reconstruction. However, applying it to dynamic and personalized digital human modeling still faces significant challenges: on the one hand, 3DGS requires a large number of multi-view images for optimization, while a single image lacks sufficient geometric and appearance constraints; on the other hand, the human body has complex deformations and semantic structures, and directly using 3DGS for single-image reconstruction can easily lead to geometric distortion, loss of identity features, or unreasonable poses.

[0004] Currently, no effective solution has been proposed for the problem of how to perform high-quality dynamic reconstruction of the 3D human body based on a single image in related technologies. Summary of the Invention

[0005] This application provides a method and apparatus for generating a three-dimensional human body model based on a single image, so as to at least solve the problem of how to perform high-quality dynamic reconstruction of a three-dimensional human body based on a single image in related technologies.

[0006] In a first aspect, embodiments of this application provide a method for generating a three-dimensional human body model based on a single image, the method comprising:

[0007] Obtain a single image of the target person, wherein the image contains the complete target person;

[0008] Based on the image of the person and the body posture parameters extracted from the image of the person, a dance-driven video with the same body posture as the target person is generated by a trained preset motion generation model;

[0009] The posture-driven information of the target person is extracted from the dance-driven video using a preset visual processing model.

[0010] The posture-driven information is used as the basis for modeling posture-driven deformation, and a three-dimensional human body model based on the dance-driven video is generated by a preset digital human modeling algorithm.

[0011] In some embodiments, generating a dance-driven video consistent with the target person's posture based on the person image and body posture parameters extracted from the image, using a trained preset motion generation model, includes:

[0012] Based on the image of the person, a dance-driven video that matches the appearance of the target person is generated by a trained preset motion generation model.

[0013] The body posture parameters of the target person are obtained by extracting parameters from the image of the person using a preset human body modeling tool.

[0014] Based on the posture parameters, the driving parameters of each joint of the target person are extracted from the dance driving video; according to the joint driving parameters, a dance movement sequence consistent with the posture of the target person is obtained.

[0015] Based on the dance sequence and the image of the person, a dance-driven video with the same posture as the target person is generated by a trained preset motion generation model.

[0016] In some embodiments, the body posture parameters of the target person are obtained by extracting parameters from the image of the person using a preset human body modeling tool, including:

[0017] The body posture parameters of the target person are obtained by extracting parameters from the image of the person using the SMPL-X tool.

[0018] In some embodiments, the method includes:

[0019] Based on the image of the person, a dance-driven video that matches the appearance of the target person is generated using the trained Wan2.2-Animat model;

[0020] Based on the dance sequence and the image of the person, a dance-driven video with the same posture as the target person is generated using the trained Wan2.2-Animat model.

[0021] In some embodiments, before generating a dance-driven video consistent with the target person's posture using a trained preset motion generation model based on the person image and posture parameters extracted from the image, the method further includes:

[0022] The training dataset is constructed using balanced sampling, wherein the training dataset contains images of people and motion reference videos. Balanced sampling refers to alternating frames between the images of people and the motion reference videos to form the training dataset.

[0023] The preset action generation model is trained using the training dataset to obtain the trained preset action generation model.

[0024] In some embodiments, extracting the posture-driven information of the target person from the dance-driven video using a preset visual processing model includes:

[0025] The pose-driven information of the target person is extracted from the dance-driven video using the Sapiens model, wherein the pose-driven information includes at least one of the following: geometric binary mask, depth map, normal map, and body part segmentation.

[0026] In some embodiments, the posture-driven information is used as the basis for modeling posture-driven deformation, and a 3D human body model based on the dance-driven video is generated using a preset digital human modeling algorithm, including:

[0027] The geometric binary mask, depth map, normal map, and body segmentation are used as the basis for modeling pose-driven deformation. A 3D human body model based on the dance-driven video is generated using the ExAvatar hybrid representation algorithm. The ExAvatar hybrid representation algorithm combines SMPL-X tools with 3DGS scene representation technology to treat 3D Gaussians as surface vertices and follow the mesh topology of SMPL-X. The 3D human body model is a dynamic and interactive model.

[0028] In some embodiments, the method further includes:

[0029] A lightweight attention-upsampling convolutional network SRnet is constructed for 3DGS rendered images in the ExAvatar hybrid representation algorithm. The attention-upsampling convolutional network SRnet is used to upsample the input low-resolution 3DGS rendered image and output a high-resolution super-resolution reconstructed image.

[0030] In some embodiments, the method further includes:

[0031] The attention upsampling convolutional network SRnet is trained using a composite loss function, which includes a color loss L1 to ensure the consistency of image color before and after super-resolution reconstruction.

[0032] In a second aspect, embodiments of this application provide a three-dimensional human body model generation device based on a single image. The device is used to perform the method described in the first aspect above. The device includes an image acquisition module, a video generation module, and a model generation module.

[0033] The image acquisition module is used to acquire a picture of the target person, wherein the picture contains the complete target person;

[0034] The video generation module is used to generate a dance-driven video that matches the target person's posture based on the person's image and the posture parameters extracted from the image, using a trained preset motion generation model.

[0035] The model generation module is used to extract the posture driving information of the target person from the dance-driven video through a preset visual processing model; and to use the posture driving information as the basis for modeling posture-driven deformation, and to generate a three-dimensional human body model based on the dance-driven video through a preset digital human modeling algorithm.

[0036] Compared to related technologies, this application provides a method and apparatus for generating a 3D human body model based on a single image. The method involves acquiring a single image containing a complete target person; generating a dance-driven video consistent with the target person's posture using a trained preset motion generation model based on the image and posture parameters extracted from it; extracting the target person's posture-driven information from the dance-driven video using a preset visual processing model; using this posture-driven information as the basis for modeling posture-driven deformation; and generating a 3D human body model based on the dance-driven video using a preset digital human modeling algorithm. This achieves 3D human body model generation based on a single image, cleverly utilizing the posture parameters extracted from the image to generate a posture-consistent driving video, avoiding inconsistencies in height and build. Based on this high-quality video, the posture-driven information is used to complete the construction of a high-quality dynamic model, solving the problem of how to perform high-quality dynamic reconstruction of a 3D human body based on a single image. Attached Figure Description

[0037] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0038] Figure 1 This is a schematic diagram of the system topology corresponding to the method according to the embodiments of this application;

[0039] Figure 2 This is a schematic diagram of the method executed in server 105 according to an embodiment of this application;

[0040] Figure 3 This is a flowchart of the steps of a method for generating a three-dimensional human body model based on a single image according to an embodiment of this application;

[0041] Figure 4 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0043] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0044] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0045] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0046] This application provides a method for generating a 3D human body model based on a single image. Figure 1 This is a schematic diagram of the system topology corresponding to the method according to the embodiments of this application, such as... Figure 1 As shown, the three-dimensional human body model generation method provided in this application embodiment can be executed in server 105. The generated three-dimensional human body model can be transmitted to cloud 104 via wired or wireless means, and then transmitted to client terminals 101, 102 and 103 for display and interaction. Figure 2 This is a schematic diagram illustrating the method executed in server 105 according to an embodiment of this application, as shown below. Figure 2 As shown, server 105 executes the three-dimensional human body model generation method provided in the application embodiment. Figure 3 This is a flowchart illustrating the steps of a single-image-based 3D human body model generation method according to an embodiment of this application, as follows: Figure 3 As shown, the method includes the following steps:

[0047] Step S102: Obtain a picture of the target person, wherein the picture contains the complete target person;

[0048] It should be noted that while single-image full-body digital human generation technology has significant advantages in terms of convenience and ease of use, a single image cannot provide a complete view of all parts of a person's body, especially areas that are occluded or not visible in the frame, such as the back, sides, or under the arms. This can lead to missing or inaccurate reconstruction of clothing, affecting the realism of the overall visual effect. In contrast, the embodiments of this application cleverly generate a high-quality driving video based on a single image of a person, and then generate a 3D human body model on this basis.

[0049] Step S104: Based on the person image and the body posture parameters extracted from the image, a dance-driven video consistent with the target person's body posture is generated through a trained preset motion generation model.

[0050] Step S104 specifically includes the following steps:

[0051] Step S1041: First, based on the image of the person, generate a dance-driven video that matches the appearance of the target person using a trained preset motion generation model;

[0052] Specifically, in step S1041, a dance-driven video matching the appearance of the target person is first generated based on the person's image using the trained Wan2.2-Animat model.

[0053] It should be noted that Wan2.2-Animate is an open-source AI video motion generation model developed by the Alibaba Cloud Tongyi Wanxiang team. It is a comprehensive upgrade of Animate Anyone, capable of accurately capturing the movements and expressions of characters in reference videos and then "transferring" them to custom static characters. In step S1041, the Wan2.2-Animate large model is used, leveraging its powerful i2v generalization capability to generate dance-driven videos from single images, complete clothing information, and maintain the consistency of the character's appearance across various movement changes.

[0054] Step S1042: Extract parameters from the human image using a preset human body modeling tool to obtain the body posture parameters of the target person;

[0055] Specifically, step S1042 involves extracting parameters from the image of the person using the SMPL-X tool to obtain the body posture parameters of the target person.

[0056] Step S1043: Based on body posture parameters, extract the driving parameters of each joint of the target person from the dance driving video, and then obtain a dance movement sequence that is consistent with the body posture of the target person.

[0057] It should be noted that SMPL-X (SMPL eXpressive) is an innovative 3D human body modeling tool. It is an extended version of the SMPL (Skinned Multi-Person Linear) model, which integrates the human face, hands, and body models into a unified parametric system. Through linear blending skinning technology, combined with learned correction blending shapes, it achieves accurate human body modeling. In steps S1042 and S1043, SMPL-X is used to fit and extract the body posture parameters of the target person from the image. The driving parameters of each joint point of SMPL-X are extracted from the driving video. The joint point driving parameters and body posture parameters are used to obtain a dance movement sequence consistent with the body posture of the target person (which can be understood as using transmission transformation to project SMPL-X into 2D space to extract a vitpose-like human skeleton, and each continuous movement results in a set of "stick figure" sequences).

[0058] Step S1044: Based on the dance movement sequence and the image of the person, a dance-driven video with the same posture as the target person is generated through the trained preset movement generation model.

[0059] Specifically, in step S1044, based on the dance movement sequence and the image of the person, a dance-driven video with the same posture as the target person is generated through the trained Wan2.2-Animat model.

[0060] It should be noted that Wan2.2-Animat uses the motion sequence obtained in step S1043 and the input image to finally generate a dance video with the same posture as the original image.

[0061] Before step S104, the method further includes step S103, which constructs a training dataset by means of balanced sampling. The training dataset contains images of people and motion reference videos. Balanced sampling refers to the alternation of frames between images of people and motion reference videos to form a training dataset. The training dataset is used to train a preset motion generation model (preferably the Wan2.2-Animat model) to obtain the trained preset motion generation model.

[0062] It should be noted that while the early image-based large-scale model Wan2.2-Animat possessed a certain ability to maintain consistency in human identity, it still could not guarantee that the identity of the person would deviate under dramatic movements and exaggerated facial expressions. Therefore, step S103 employs balanced sampling to train Wan2.2-Animat. During training, the input image and the generated video frames are alternated, effectively oversampling the input image, which helps prevent loss of realism. By using the original human image more frequently, identity consistency is maintained in the visible area, greatly improving Wan2.2-Animat's ability to maintain human identity consistency, thus enabling the final generated 3D human model to have a high fidelity to the original photograph.

[0063] Step S106: Extract the posture driving information of the target person from the dance-driven video using a preset visual processing model.

[0064] Specifically, step S106 involves extracting the pose-driven information of the target person from the dance-driven video using the Sapiens model. The pose-driven information includes a geometric binary mask, a depth map, a normal map, and body segmentation.

[0065] It should be noted that in the ExAvatar hybrid representation algorithm in subsequent step S108, SMPL-X processing only includes the human torso, limbs, and face, excluding the geometric structure and topological information of clothing. This results in a lack of reasonable geometric priors for complex clothing areas such as long skirts during the 3DGS initialization stage, typically relying only on sparse point clouds or random initialization. Consequently, the model reconstruction results are blurry, with unclear edges, and long skirts split into pants after the model is activated. Sapiens, on the other hand, is a next-generation human vision AI model designed to understand and simulate human behavior. It possesses four core functions specifically designed for human vision, including 2D geometric pose estimation, body part recognition and segmentation, depth prediction estimation, and surface normal prediction estimation. Therefore, step S106 utilizes the Sapiens model to extract pose-driven information such as geometric binary masks, depth maps, normal maps, and body part segmentation from the dance-driven video. This information serves as a reliable basis for the multilayer perceptron (MLP) modeling pose-driven deformation in ExAvatar, effectively ensuring the normal display of clothing (such as long skirts) in model reconstruction and dynamic interaction. In addition, during the network training phase, the loss of RGB pixels in MLP can be reduced, while the loss of normal maps, binary masks, etc. can be increased.

[0066] Step S108: Using the posture-driven information as the basis for modeling posture-driven deformation, a three-dimensional human body model based on dance-driven video is generated through a preset digital human modeling algorithm.

[0067] Specifically, step S108 uses the geometric binary mask, depth map, normal map, and body part segmentation as the basis for modeling pose-driven deformation. The ExAvatar hybrid representation algorithm is used to generate a 3D human body model based on dance-driven video. The ExAvatar hybrid representation algorithm combines SMPL-X tools with 3DGS scene representation technology to treat 3D Gaussians as surface vertices and follow the mesh topology of SMPL-X. The 3D human body model is a dynamic and interactive model.

[0068] It's worth noting that ExAvatar employs a combined representation of SMPL-X (the modeling tool) and 3D Gaussian sputtering (3DGS). Specifically, in ExAvatar, each 3DGS is treated as a vertex of a surface with predefined connectivity information (i.e., triangular faces), and these vertices follow the SMPL-X mesh topology. Due to this hybrid representation, ExAvatar can drive new facial animations using the SMPL-X facial expression space. This addresses the issue of insufficient facial expression and pose diversity in models, allowing the model to retain facial details while reducing artifacts during motion changes.

[0069] Through the steps described in this application embodiment, a three-dimensional human body model based on a single image is generated. It cleverly utilizes the human body posture parameters extracted from the human image to generate a driving video with consistent posture, avoiding inconsistencies in the height and weight of the human body. Based on this high-quality video, the posture driving information is used to complete the construction of a high-quality dynamic model, solving the problem of how to perform high-quality dynamic reconstruction of a three-dimensional human body based on a single image.

[0070] In some embodiments, for the ExAvatar hybrid representation algorithm in step S108 of the above embodiments, the method provided by this application further includes:

[0071] A lightweight attention-upsampling convolutional network SRnet is constructed for 3DGS rendered images in the ExAvatar hybrid representation algorithm. SRnet is used to upsample the input low-resolution 3DGS rendered image and output a high-resolution super-resolution reconstructed image.

[0072] In some embodiments, the method further includes:

[0073] The attention-upsampling convolutional network SRnet is trained using a composite loss function, which includes a color loss L1 to ensure the consistency of image color before and after super-resolution reconstruction.

[0074] It should be noted that, to improve the clarity of the rendering results, a lightweight attention-based upsampling convolutional network, SRnet, for 3DGS rendered images is further proposed. Preferably, a low-resolution image of 512×512×3 with Gaussian attributes is input, upsampled for super-resolution reconstruction, and outputs a high-resolution image of 1024×1024×3 or higher. The image is then insulated and upsampled. The original RGB image of the 3DGS rendered image is processed, extracting color and texture features, and the overall display effect is further improved after upsampling. Furthermore, SRnet employs a composite loss function during training, including L1 color loss to ensure color consistency before and after super-resolution reconstruction, and perceptual loss, etc.

[0075] It should be further noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0076] This application provides a single-image-based three-dimensional human body model generation device, which is used to execute the method provided in the above embodiments. The device includes an image acquisition module, a video generation module, and a model generation module.

[0077] The image acquisition module is used to acquire a picture of the target person, wherein the picture contains the complete target person;

[0078] The video generation module is used to generate a dance-driven video that matches the posture of the target person based on the person's image and the posture parameters extracted from the image, using a trained preset motion generation model.

[0079] The model generation module is used to extract the posture-driven information of the target person from the dance-driven video through a preset visual processing model; the posture-driven information is used as the basis for modeling posture-driven deformation, and a three-dimensional human body model based on the dance-driven video is generated through a preset digital human modeling algorithm.

[0080] Through the image acquisition module, video generation module, and model generation module in this application embodiment, a three-dimensional human body model based on a single image is generated. It cleverly utilizes the human body posture parameters extracted from the human image to generate a driving video with consistent posture, avoiding inconsistencies in the height and weight of the human body. Based on this high-quality video, the posture driving information is used to complete the construction of a high-quality dynamic model, solving the problem of how to perform high-quality dynamic reconstruction of a three-dimensional human body based on a single image.

[0081] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0082] This embodiment provides an electronic device including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0083] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0084] Optionally, the electronic device may further include a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for generating a three-dimensional human body model based on a single image. The display screen may be a liquid crystal display (LCD) or an e-ink display. The input device may be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0085] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0086] Furthermore, in conjunction with the single-image-based 3D human body model generation method in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the single-image-based 3D human body model generation methods in the above embodiments.

[0087] In one embodiment, Figure 4 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 4 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 4As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores the operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network, the internal memory provides an environment for the operating system and computer programs to run, the computer programs are executed by the processor to implement a single-image-based method for generating 3D human body models, and the database stores data.

[0088] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0089] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0090] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for generating a 3D human body model based on a single image, characterized in that, The method includes: Obtain a single image of the target person, wherein the image contains the complete target person; Based on the image of the person, a dance-driven video that matches the appearance of the target person is generated by a trained preset motion generation model. The body posture parameters of the target person are obtained by extracting parameters from the image of the person using a preset human body modeling tool. Based on the posture parameters, the driving parameters of each joint of the target person are extracted from the dance driving video; according to the joint driving parameters, a dance movement sequence consistent with the posture of the target person is obtained. Based on the dance sequence and the image of the person, a dance-driven video with the same posture as the target person is generated by a trained preset motion generation model. The posture driving information of the target person is extracted from the dance driving video that is consistent with the posture of the target person by using a preset visual processing model. The posture-driven information is used as the basis for modeling posture-driven deformation. A three-dimensional human body model based on the dance-driven video that is consistent with the body shape of the target human body is generated by a preset digital human modeling algorithm.

2. The method according to claim 1, characterized in that, The body posture parameters of the target person are obtained by extracting parameters from the image of the person using a preset human body modeling tool, including: The body posture parameters of the target person are obtained by extracting parameters from the image of the person using the SMPL-X tool.

3. The method according to claim 1, characterized in that, The method includes: Based on the image of the person, a dance-driven video that matches the appearance of the target person is generated using the trained Wan2.2-Animat model; Based on the dance sequence and the image of the person, a dance-driven video with the same posture as the target person is generated using the trained Wan2.2-Animat model.

4. The method according to claim 1, characterized in that, Before generating a dance-driven video consistent with the target person's posture using a trained preset motion generation model based on the person's image and the posture parameters extracted from the image, the method further includes: The training dataset is constructed using balanced sampling, wherein the training dataset contains images of people and motion reference videos. Balanced sampling refers to alternating frames between the images of people and the motion reference videos to form the training dataset. The preset action generation model is trained using the training dataset to obtain the trained preset action generation model.

5. The method according to claim 1, characterized in that, Extracting posture-driven information of the target person from the dance-driven video that matches the target person's posture using a preset visual processing model includes: The pose-driven information of the target person is extracted from the dance-driven video that is consistent with the pose of the target person using the Sapiens model. The pose-driven information includes at least one of the following: geometric binary mask, depth map, normal map and body part segmentation.

6. The method according to claim 5, characterized in that, Using the posture-driven information as the basis for modeling posture-driven deformation, a 3D human model based on the dance-driven video that matches the target human body's posture is generated through a preset digital human modeling algorithm, including: The geometric binary mask, depth map, normal map, and body segmentation are used as the basis for modeling pose-driven deformation. A 3D human body model based on the dance-driven video with the same body shape as the target human body is generated by the ExAvatar hybrid representation algorithm. The ExAvatar hybrid representation algorithm combines SMPL-X tools with 3DGS scene representation technology to treat 3D Gaussians as surface vertices and follow the mesh topology of SMPL-X. The 3D human body model is a dynamic and interactive model.

7. The method according to claim 6, characterized in that, The method further includes: A lightweight attention-upsampling convolutional network SRnet is constructed for 3DGS rendered images in the ExAvatar hybrid representation algorithm. The attention-upsampling convolutional network SRnet is used to upsample the input low-resolution 3DGS rendered image and output a high-resolution super-resolution reconstructed image.

8. The method according to claim 7, characterized in that, The method further includes: The attention upsampling convolutional network SRnet is trained using a composite loss function, which includes a color loss L1 to ensure the consistency of image color before and after super-resolution reconstruction.

9. A device for generating a three-dimensional human body model based on a single image, characterized in that, The apparatus is used to perform the method according to any one of claims 1 to 8, the apparatus comprising an image acquisition module, a video generation module, and a model generation module; The image acquisition module is used to acquire a picture of the target person, wherein the picture contains the complete target person; The video generation module is used to generate a dance-driven video that matches the target person's posture based on the person's image and the posture parameters extracted from the image, using a trained preset motion generation model. The model generation module is used to extract the posture driving information of the target person from the dance driving video that is consistent with the posture of the target person through a preset visual processing model; and to use the posture driving information as the basis for modeling posture driving deformation, and to generate a three-dimensional human body model based on the dance driving video that is consistent with the posture of the target person through a preset digital human modeling algorithm.

Images