2D digital human shadow generation method and device based on 3D model

By generating 2D digital human shadows based on 3D models, the problem of poor shadow effects in the existing technology is solved, efficient integration with 2D digital human videos is achieved, and the video picture quality is significantly improved.

CN118967916APending Publication Date: 2024-11-15INSPUR FINANCIAL INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410947736.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing 2D digital video generation technology lacks shadow generation function, or the shadow effect is affected by the direction of the main light source during acquisition, resulting in distortion of the picture.

Method used

Using a 3D model-based method, by selecting 2D digital people and setting background pictures, obtaining the closest 3D modeling library, initializing 3D scenes, adding light source information, creating animations of 3D digital people models based on action information, and generating shadow pictures frame by frame , and combine shadows with 2D digital human videos.

Benefits of technology

The generated shadow material is integrated with 2D digital human videos, which significantly improves the video image effect and achieves more realistic shadow performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967916A_ABST
    Figure CN118967916A_ABST
Patent Text Reader

Abstract

The invention discloses a 2D digital human shadow generation method and device based on a 3D model, and the method comprises the following steps: S1, selecting a 2D digital human, and obtaining a closest 3D modeling library; s2, initializing a 3D scene, placing the 3D digital human model at a specified position in the 3D scene, and enabling the 3D digital human model to be basically matched with the selected 2D digital human model; s3, calculating the normal direction of the projection plane according to the projection plane rectangle, and placing the projection plane at a position corresponding to a foothold of the 3D digital human model; s4, adding light source information in the 3D scene; s5, creating an action animation of the 3D digital human model according to the 2D digital human action information; s6, generating a shadow picture frame by frame according to the 3D scene, and obtaining a shadow video of the shadow or a picture sequence material of each frame; s7, synthesizing the 2D digital human video, the shadow video shadow obtained in the step S6 and the background picture in the step S1 into an output video; and S8, carrying out 2D digital human shadow calculation and outputting a shadow. The method has the advantage of remarkably improving the video picture effect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of digital imaging, and in particular to a method and device for generating a 2D digital human shadow based on a 3D model. Background Art

[0002] Currently common 2D digital human video generation technology uses voice-driven lip shape, video face swapping, video fusion and background replacement technologies to realize the face part of the original video recorded by real people. Most of them do not have the shadow generation function, or the shadow effect is obtained during recording, which is affected by the direction of the main light source during acquisition. After replacing the background, it is easy to produce conflicts and distort the picture.

[0003] In view of this, it is necessary to provide a method and device for generating 2D digital human shadows based on a 3D model. Summary of the invention

[0004] The method and device for generating 2D digital human shadow based on 3D model provided by the present invention effectively solve the problem of poor picture effect of existing 2D digital human video.

[0005] The technical solution adopted by the present invention is

[0006] A method for generating a 2D digital human shadow based on a 3D model comprises the following steps:

[0007] S1. Select a 2D digital human and set a background image, and obtain the closest 3D modeling library based on the height and weight of the 2D digital human model;

[0008] S2, obtaining the focal length, shooting distance and zoom information of the model, initializing a 3D scene, placing the 3D digital human model in the 3D modeling library at a specified position in the 3D scene, and fine-tuning the 3D digital human model so that the 3D digital human model basically matches the selected 2D digital human;

[0009] S3. Calculate the normal direction of the projection plane according to the projection plane rectangle, and place the projection plane at a position corresponding to the foothold of the 3D digital human model;

[0010] S4. Add light source information to the 3D scene;

[0011] S5, creating action animation of 3D digital human model according to the action information of 2D digital human;

[0012] S6. Generate shadow pictures frame by frame according to the 3D scene to obtain shadow videos or picture sequence materials of each frame;

[0013] S7, synthesizing the 2D digital human video, the shadow video shadow obtained in S6, and the background image in S1 into an output video;

[0014] S8. Perform 2D digital human shadow calculation and output shadow according to the output video obtained in S7.

[0015] Furthermore, the step S1 of selecting a 2D digital human specifically includes:

[0016] S11, select the 2D digital human, obtain the parameters of the 2D digital human, and the system automatically draws an editable foothold rectangle at the foothold position for the designer to adjust the projection plane and determine the normal direction;

[0017] S12. Select the background layer and add a background image;

[0018] S13, adjusting the projection plane rectangle and the perspective direction to meet the actual needs of generating shadows;

[0019] S14. Setting the main light source information to meet the actual needs of generating shadows.

[0020] Further, step S8 includes:

[0021] S81, obtain the shadow data of each frame: the shadow data of each frame = the illumination intensity of the main light source × the diffuse reflection coefficient of the projection plane × the maximum value function (0, the product function (normal direction, illumination direction));

[0022] S82, performing fuzzy processing on the shadow data of each frame;

[0023] S83, looping through each frame of the video to obtain a complete shadow video or a picture sequence material of each frame.

[0024] Furthermore: the 3D digital human models in the 3D modeling library in S1 are bound to common actions, the closest model can be retrieved according to different attributes, and the 3D model can be fine-tuned according to the input parameters.

[0025] Furthermore, in step S14, the attributes of the main light source information include but are not limited to: the distance of the light source from the model, the angle of the light source, and the light intensity.

[0026] Furthermore, the properties of the 2D digital human include but are not limited to: video for audio driving, 2D digital human acquisition focal length, shooting distance, landing point coordinates, model height, weight, and action information.

[0027] Furthermore, the method of step S7, synthesizing the 2D digital human video, shadow, and background image into an output video specifically includes:

[0028] S71, calculating the coordinates of the shadow material according to the coordinates of the 2D digital human video and the relative coordinates of the landing point;

[0029] S72, synthesize the output video by placing the 2D digital human video on the top layer, the shadow video on the middle layer, and the background image on the bottom layer.

[0030] A 2D digital human shadow generation system based on a 3D model, comprising:

[0031] 3D model library, used to store 3D digital human models;

[0032] The 2D digital human shadow generation module is used to generate the 2D digital human shadow.

[0033] A computer-readable storage medium having a computer program readable thereon, wherein the computer program, when executed by a processor, implements the steps of the method for generating a 2D digital human shadow based on a 3D model.

[0034] A computer device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein:

[0035] The memory is used to store computer programs;

[0036] The processor is used to execute the steps of the method for generating a 2D digital human shadow based on a 3D model by running the program stored in the memory.

[0037] Beneficial effects of the invention: generating more realistic shadow material based on the original video information of the 2D digital human and the required main light source direction, and fusing the generated shadow material with the 2D digital human video, can significantly improve the video picture effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic flow chart of a method for generating a 2D digital human shadow based on a 3D model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0040] The present application provides a first embodiment, that is, a method for generating a 2D digital human shadow based on a 3D model, comprising the following steps:

[0041] S1. Initialize the 2D digital human shadow generation system based on the 3D model, and obtain the closest 3D modeling library according to the height and weight of the 2D digital human model;

[0042] S2, obtaining the focal length, shooting distance and zoom information of the model, initializing a 3D scene, placing the 3D digital human model in the 3D modeling library at a specified position in the 3D scene, and fine-tuning the 3D digital human model so that the 3D digital human model basically matches the selected 2D digital human;

[0043] S3. Calculate the normal direction of the projection plane according to the projection plane rectangle, and place the projection plane at a position corresponding to the foothold of the 3D digital human model;

[0044] S4. Add light source information to the 3D scene;

[0045] S5, creating action animation of 3D digital human model according to the action information of 2D digital human;

[0046] S6. Generate shadow pictures frame by frame according to the 3D scene to obtain shadow videos or picture sequence materials of each frame;

[0047] S7, synthesizing the 2D digital human video, the shadow video shadow obtained in S6, and the background image in S1 into an output video;

[0048] S8. Perform 2D digital human shadow calculation and output shadow according to the output video obtained in S7.

[0049] It should be noted that the 2D digital human video in step S7 is generated using the existing technology.

[0050] In the above design, a more realistic shadow material is generated according to the original video information of the 2D digital human and the required main light source direction, and the generated shadow material is fused with the 2D digital human video, which can significantly improve the video picture effect.

[0051] Specifically: the step S1 of selecting a 2D digital human specifically includes:

[0052] S11, select the 2D digital human, obtain the parameters of the 2D digital human, and the system automatically draws an editable foothold rectangle at the foothold position for the designer to adjust the projection plane and determine the normal direction;

[0053] S12. Select the background layer and add a background image;

[0054] S13, adjusting the projection plane rectangle and the perspective direction to meet the actual needs of generating shadows;

[0055] S14. Setting the main light source information to meet the actual needs of generating shadows.

[0056] In the above design, the parameters of the 2D digital human include but are not limited to height and body shape (thin, relatively thin, medium, relatively fat, fat).

[0057] Specifically: step S8 includes:

[0058] S81. Obtain shadow data of each frame: shadow data of each frame = illumination intensity of main light source × diffuse reflection coefficient of projection plane × maximum value function (0, product function (normal direction, illumination direction)).

[0059] S82, performing fuzzy processing on the shadow data of each frame.

[0060] S83, looping through each frame of the video to obtain a complete shadow video or a picture sequence material of each frame.

[0061] Specifically: the 3D digital human model in the 3D modeling library in S1 is bound to common actions, the closest model can be retrieved according to different attributes, and the 3D model can be fine-tuned according to the input parameters.

[0062] Specifically: in step S14, the attributes of the main light source information include but are not limited to: the distance of the light source from the model, the angle of the light source, and the light intensity.

[0063] Specifically: the attributes of the 2D digital human include but are not limited to: video for audio driving, 2D digital human acquisition focal length, shooting distance, landing point coordinates, model height, weight, and action information.

[0064] Specifically: Step S7, the method of synthesizing the 2D digital human video, shadow, and background image into an output video specifically includes:

[0065] S71, calculating the coordinates of the shadow material according to the coordinates of the 2D digital human video and the relative coordinates of the landing point;

[0066] S72, synthesize the output video by placing the 2D digital human video on the top layer, the shadow video on the middle layer, and the background image on the bottom layer.

[0067] The present application also provides a second embodiment, that is, a 2D digital human shadow generation system for a 3D model, including: a 3D model library for storing a 3D digital human model;

[0068] The 2D digital human shadow generation module is used to generate the 2D digital human shadow.

[0069] The present application also provides a third embodiment, that is, a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for generating a 2D digital human shadow based on a 3D model are implemented.

[0070] In addition, the computer-readable storage medium of this embodiment may adopt any combination of one or more computer-readable storage media, wherein the computer-readable storage medium includes electrical, optical, electromagnetic, infrared or semiconductor systems, devices or components, or any combination thereof.

[0071] The present application also provides a fourth embodiment, that is, a computer device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; wherein:

[0072] The memory is used to store computer programs;

[0073] The processor is used to execute the steps of the above-mentioned method for generating 2D digital human shadow based on 3D model by running the program stored in the memory.

[0074] As an embodiment of the present invention, the communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0075] As an implementation mode of the present invention, the communication interface is used for communication between the above-mentioned terminal and other devices.

[0076] As an embodiment of the present invention, the memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0077] As an embodiment of the present invention, the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0078] The following are several preferred embodiments of the present application:

[0079] Embodiment 1:

[0080] 1. A method for generating a 2D digital human shadow based on a 3D model, comprising the following steps:

[0081] S1. Initialize the 2D digital human shadow generation system based on the 3D model, select the 2D digital human, and obtain the closest 3D modeling library according to the height and weight of the 2D digital human model; bind common actions to the 3D digital human model in the 3D modeling library, retrieve the closest model according to different attributes, and fine-tune the 3D model according to the input parameters.

[0082] 2. Specifically including: S11. Select the 2D digital human and obtain the parameters of the 2D digital human. The system automatically draws an editable landing point rectangle at the landing point position for the designer to adjust the projection plane and determine the normal direction; S12. Select the background layer and add a background image; S13. Adjust the projection plane rectangle and the perspective direction to meet the actual needs of generating shadows; S14. Set the main light source information. The attributes of the main light source information include but are not limited to: the distance of the light source from the model, the angle of the light source, and the light intensity, so as to meet the actual needs of generating shadows.

[0083] S2. Initialize a 3D scene based on the collected focal length, shooting distance and zoom information, place the 3D digital human model at a specified position in the 3D scene, and fine-tune the model based on the attributes so that it basically matches the selected 2D digital human; including

[0084] S3. Calculate the normal direction of the projection plane according to the projection plane rectangle, and place the projection plane at a position corresponding to the foothold of the 3D digital human model;

[0085] S4. Add light source information to the 3D scene;

[0086] S5. Create action animation of 3D digital human model according to the action information of 2D digital human; the attributes of 2D digital human include but are not limited to: video for audio drive, 2D digital human acquisition focal length, shooting distance, landing point coordinates, model height, weight, and action information.

[0087] S6. Generate shadow pictures frame by frame according to the 3D scene to obtain shadow videos or picture sequence materials of each frame;

[0088] S7, synthesizing the 2D digital human video, shadow, and background image into an output video; specifically including: S71, calculating the coordinates of the shadow material according to the coordinates of the 2D digital human video and the relative coordinates of the landing point; S72, synthesizing the output video in a manner of placing the 2D digital human video on the top layer, the shadow video on the middle layer, and the background image on the bottom layer.

[0089] S8. Perform 2D digital human shadow calculation, including:

[0090] S81, obtain the shadow data of each frame: the shadow data of each frame = the illumination intensity of the main light source × the diffuse reflection coefficient of the projection plane × the maximum value function (0, the product function (normal direction, illumination direction));

[0091] S82, performing fuzzy processing on the shadow data of each frame;

[0092] S83, looping through each frame of the video to obtain a complete shadow video or a picture sequence material of each frame.

[0093] Embodiment two:

[0094] A 2D digital human shadow generation system for a 3D model, comprising:

[0095] 3D model library, used to store 3D digital human models;

[0096] The 2D digital human shadow generation module is used to generate the 2D digital human shadow.

[0097] Embodiment three:

[0098] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for generating a 2D digital human shadow based on a 3D model.

[0099] Embodiment 4:

[0100] A computer device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein:

[0101] The memory is used to store computer programs;

[0102] The processor is used to execute the steps of the method for generating a 2D digital human shadow based on a 3D model by running the program stored in the memory.

[0103] To further explain in detail, it should be understood that the above is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating a 2D digital human shadow based on a 3D model, characterized in that: The steps include: S1. Select a 2D digital human and set a background image, and obtain the closest 3D modeling library based on the height and weight of the 2D digital human model; S2, obtaining the focal length, shooting distance and zoom information of the model, initializing a 3D scene, placing the 3D digital human model in the 3D modeling library at a specified position in the 3D scene, and fine-tuning the 3D digital human model so that the 3D digital human model basically matches the selected 2D digital human; S3. Calculate the normal direction of the projection plane according to the projection plane rectangle, and place the projection plane at a position corresponding to the foothold of the 3D digital human model; S4. Add light source information to the 3D scene; S5, creating action animation of 3D digital human model according to the action information of 2D digital human; S6. Generate shadow pictures frame by frame according to the 3D scene to obtain shadow videos or picture sequence materials of each frame; S7, synthesizing the existing 2D digital human video, the shadow video shadow obtained in S6, and the background image in S1 into an output video; S8. Perform 2D digital human shadow calculation and output shadow according to the output video obtained in S7.

2. The method for generating 2D digital human shadow based on 3D model according to claim 1, characterized in that: The step S1 of selecting a 2D digital human specifically includes: S11, select the 2D digital human, obtain the parameters of the 2D digital human, and the system automatically draws an editable foothold rectangle at the foothold position for the designer to adjust the projection plane and determine the normal direction; S12. Select the background layer and add a background image; S13, adjusting the projection plane rectangle and the perspective direction to meet the actual needs of generating shadows; S14. Setting the main light source information to meet the actual needs of generating shadows.

3. The method for generating 2D digital human shadow based on 3D model according to claim 2, characterized in that: Step S8 includes: S81, obtain the shadow data of each frame: the shadow data of each frame = the illumination intensity of the main light source × the diffuse reflection coefficient of the projection plane × the maximum value function (0, the product function (normal direction, illumination direction)); S82, performing fuzzy processing on the shadow data of each frame; S83, looping through each frame of the video to obtain a complete shadow video or a picture sequence material of each frame.

4. The method for generating 2D digital human shadow based on 3D model according to claim 1, characterized in that: The 3D digital human models in the 3D modeling library in S1 are bound to common actions, the closest model can be retrieved according to different attributes, and the 3D model can be fine-tuned according to the input parameters.

5. The method for generating 2D digital human shadow based on 3D model according to claim 2, characterized in that: In step S14, the attributes of the main light source information include but are not limited to: the distance of the light source from the model, the angle of the light source, and the light intensity.

6. The method for generating 2D digital human shadow based on 3D model according to claim 1, characterized in that: The properties of the 2D digital human include but are not limited to: video for audio driving, digital human acquisition focal length, shooting distance, landing point coordinates, model height, weight, and action information.

7. The method for generating 2D digital human shadow based on 3D model according to claim 1, characterized in that: Step S7, the method of synthesizing the 2D digital human video, shadow, and background image into an output video specifically includes: S71, calculating the coordinates of the shadow material according to the coordinates of the 2D digital human video and the relative coordinates of the landing point; S72, synthesize the output video by placing the 2D digital human video on the top layer, the shadow video on the middle layer, and the background image on the bottom layer.

8. A 2D digital human shadow generation system based on a 3D model, characterized by: include, 3D model library, used to store 3D digital human models; The 2D digital human shadow generation module is used to generate the 2D digital human shadow.

9. A computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for generating a 2D digital human shadow based on a 3D model are implemented.

10. A computer device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein, The memory is used to store computer programs; The processor is used to execute the steps of the method for generating a 2D digital human shadow based on a 3D model by running the program stored in the memory.