Light supplementing method and device based on three-dimensional human body, memory and electronic equipment

Abstract

The invention discloses a light supplementing method and device based on a three-dimensional human body, a memory and electronic equipment, and the method comprises the following steps: obtaining a portrait in an image; generating a portrait depth map, face 3D key points and a portrait grey-scale map based on the acquired portrait; generating a portrait 3D grid chart based on the portrait depth map and the face 3D key points; light supplementing light source information corresponding to the portrait is obtained, and the light supplementing light source information comprises light source intensity and the relative position of a light supplementing light source and the portrait; and generating a portrait three-dimensional light supplement graph based on the light supplement light source information, the portrait grey-scale graph and the portrait 3D grid graph. According to the light supplementing method and device based on the three-dimensional human body, the memory and the electronic equipment, portrait 3D information is added, and accurate face 3D information is fused, so that lighting is more stereoscopic, real and natural.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, memory, and electronic device for supplementary lighting based on a three-dimensional human body. Background Technology

[0002] In photography, film and television production, and video shooting, facial lighting plays a crucial role. This technology not only affects the aesthetics of the image but also directly impacts the emotional expression and visual impact of the work. The main function of facial lighting is to solve problems of insufficient or uneven lighting, making facial illumination more even and natural, thereby highlighting facial features and enhancing the overall texture and depth of the image. In complex lighting environments, proper lighting can significantly reduce the influence of shadows, enhance the three-dimensionality of the subject and the texture of the skin, allowing the viewer to focus more on the subject—the person themselves.

[0003] Traditional fill light methods mainly include two types: physical fill light and virtual fill light. Physical fill light primarily uses tools such as softboxes and reflectors to achieve the desired lighting effect by adjusting the angle and distance of the light source. Virtual fill light technology is a technique that uses post-processing software to add light. After the initial shooting, the brightness, contrast, shadows, and highlights of the image are adjusted to simulate the fill light effect in the actual scene, thereby improving the overall lighting effect of the image. It mainly includes the following two methods:

[0004] (1) Post-processing intelligent algorithm optimization: Virtual fill light technology, through the application of intelligent algorithms, can achieve automated image optimization and fill light processing. Modern software tools such as Adobe Photoshop and Lightroom have built-in various virtual fill light functions. These tools use advanced algorithms to analyze images and automatically adjust the brightness, contrast, and shadow / highlight parts of the image to optimize the lighting effect. However, these tools require more professional operation, and amateurs find it difficult to achieve good and natural results.

[0005] (2) Other Virtual Lighting Algorithms: A backlighting effect model is established, and the face is enhanced with lighting based on this model. The backlighting effect model is established based on the brightness of the face and background, simulating changes in light intensity. Specifically, a two-dimensional Gaussian distribution function is obtained, and the face's position is used as the center point to construct the maximum value of the Gaussian distribution function, thus building the backlighting effect model. This model can simulate realistic lighting effects under backlighting conditions, thereby enhancing the clarity and detail of the face in backlight environments. However, since this method does not utilize the 3D information of the face, it may lack a certain degree of stereoscopic depth and realism when enhancing the lighting effect. Summary of the Invention

[0006] One of the objectives of this invention is to provide a three-dimensional human body-based stereoscopic lighting method that can improve user-level operational convenience and integrate more accurate 3D facial information, making the lighting more three-dimensional, realistic, and natural.

[0007] To achieve the above objectives, the present invention provides a stereoscopic lighting method based on a three-dimensional human body, comprising the following steps:

[0008] Extract the human figure from the image;

[0009] Based on the acquired portrait, generate a portrait depth map, 3D facial key points, and a portrait grayscale image;

[0010] Generate a 3D mesh image of the human face based on the human face depth map and 3D key points of the human face;

[0011] Obtain the supplementary light source information corresponding to the portrait, wherein the supplementary light source information includes the light source intensity and the relative position of the supplementary light source and the portrait;

[0012] Based on the supplementary light source information, the grayscale image of the portrait, and the 3D mesh image of the portrait, a stereoscopic supplementary light image of the portrait is generated.

[0013] Furthermore, the step of generating a human portrait depth map includes the following sub-steps:

[0014] Obtain the depth map of the image;

[0015] Based on the acquired portrait, generate a portrait mask;

[0016] Generate a portrait depth map based on a portrait mask and the image's depth map.

[0017] Furthermore, the step of generating a 3D mesh map of a person based on the depth map and 3D facial key points includes the following sub-steps:

[0018] Based on 3D key points of the face, generate a 3D mesh map of the face and a face mask;

[0019] Calculate the (u, v) corresponding to each face pixel (x, y) in the face mask, where the (u, v) corresponding to each face pixel (x, y) refers to the intersection point P(x, v) of the ray passing through the face pixel (x, y) and the corresponding triangular mesh. p y p , z p The normalized value of the reciprocal of the distance between the two vertices of the triangular mesh;

[0020] Based on the (u, v) corresponding to each face pixel (x, y), calculate the vertex coordinates (x, y, z') corresponding to each face pixel (x, y) on the face mask, where z' represents the depth value of the face pixel (x, y) in the portrait depth map;

[0021] Based on the human face depth map, calculate the mean depth M of the face mask region relative to the human face depth map;

[0022] Add the depth mean M to the vertex coordinates (x, y, z') of each face pixel (x, y) to obtain the new vertex coordinates (x, y, z') of each face pixel (x, y) in the face mask, thus obtaining the fused face depth map after fusing the face mask and the face depth map.

[0023] Based on the fused portrait depth map, the depth value z' of all portrait pixels (x, y) is obtained, and then the portrait 3D mesh map is generated.

[0024] Furthermore, the step of calculating (u, v) corresponding to each face pixel (x, y) on the face mask specifically includes:

[0025] Obtain the intersection point P(x, y) of the ray passing through each face pixel (x, y) and the corresponding triangle mesh. p y p , z p );

[0026] Based on the intersection point P(x, y) of the triangular mesh corresponding to each face pixel (x, y) p y p , z p ), calculate (u, v) corresponding to each face pixel (x, y); where u represents the intersection point P(x, y). p y p , z p v is the normalized value of the reciprocal of the distance between the intersection point P(x) and the vertex P2 of the intersecting triangular mesh, where v represents the intersection point P(x). p y p , z p The normalized value of the reciprocal of the distance to vertex P3 of the intersecting triangular mesh.

[0027] Furthermore, in the step of calculating the vertex coordinates (x, y, z') corresponding to each face pixel (x, y) on the face mask based on (u, v) corresponding to each face pixel (x, y):

[0028] z' = (1-uv)*z1'+u*z2'+v*z3', where z1' represents the depth value of vertex P1 of the triangle, z2' represents the depth value of vertex P2 of the triangle, and z3' represents the depth value of vertex P3 of the triangle.

[0029] Furthermore, the step of obtaining the relative position of the supplementary light source with respect to the human figure includes the following sub-steps:

[0030] Construct a spherical coordinate system corresponding to the human figure, where the center coordinate of the spherical coordinate system is (x, y). c y c (x, y) represents the center of the face in the image, and the pixel coordinates of the fill light source are represented as (x, y).

[0031] Map the supplementary light source corresponding to the portrait onto a spherical coordinate system, and obtain the coordinates (x, y) of the supplementary light source from the center of the spherical coordinate system. c y c Given the distance L and the radius r of the sphere, the illumination angle θ of the light source is calculated.

[0032] Based on the illumination angle θ, The relative positions of the supplementary light source (x, y) and the human figure are obtained.

[0033] Furthermore, the distance from the supplementary light source (x, y) to the center coordinate (x, y) of the spherical coordinate system is obtained. c y c Given the distance L and the radius r of the sphere, the illumination angle θ of the light source is calculated. In the following steps, the distance L is obtained using the following formula (a):

[0034]

[0035] The illumination angle θ is obtained by the following formula (II):

[0036] sinθ=L / r (II)

[0037] The angle of illumination It can be obtained through the following formula (iii):

[0038]

[0039] Another objective of this invention is to provide a three-dimensional human body-based stereo lighting device that improves user-level ease of operation and integrates more accurate 3D facial information, making the lighting more three-dimensional, realistic, and natural.

[0040] To achieve the above objectives, the present invention also provides a stereoscopic lighting device based on a three-dimensional human body, comprising:

[0041] The portrait acquisition module is used to capture human portraits in images;

[0042] The portrait depth map generation module is used to generate portrait depth maps based on the acquired portrait.

[0043] The 3D face key point generation module is used to generate 3D face key points based on the acquired portrait.

[0044] The portrait grayscale image generation module is used to generate portrait grayscale images based on the acquired portrait.

[0045] The 3D human body mesh generation module is used to generate a 3D human body mesh based on the human body depth map and 3D facial key points.

[0046] The light source information acquisition module is used to acquire the supplementary light source information corresponding to the human figure, wherein the supplementary light source information includes the light source intensity and the relative position of the supplementary light source and the human figure;

[0047] The portrait stereo lighting generation module is used to generate a portrait stereo lighting image based on the lighting source information, the portrait grayscale image, and the portrait 3D mesh image.

[0048] Another objective of this invention is to provide a memory in which a computer program, when run by a processor, can provide supplementary lighting for real-time or static images. This program incorporates more accurate 3D facial information, making the lighting more three-dimensional, realistic, and natural.

[0049] To achieve the above objectives, the present invention also provides a memory that stores a computer program, which is read and executed by a processor to perform the methods described above.

[0050] Another object of the present invention is to provide an electronic device that stores a computer program capable of providing supplementary lighting for real-time or static images. This computer program integrates more accurate 3D facial information, making the lighting more three-dimensional, realistic, and natural.

[0051] To achieve the above objectives, the present invention also provides an electronic device, including: a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and the processor executes the computer program to implement the method described above.

[0052] In summary, the three-dimensional human body-based stereoscopic lighting method, device, memory, and electronic device of the present invention have the following beneficial effects: The present invention can automatically determine whether the human image information has dark areas and the location of these dark areas based on the detected human image information in real-time or static images, and automatically enter the lighting process. Alternatively, based on the adjustable area of ​​the light source, the adjustment range of the light source, and the image representing the light source shown in the real-time or static image, the dark areas of the face can be lit by manually dragging the position of the light source. During the lighting process, the illumination angle of the light source is calculated based on a spherical coordinate system relative to the face pre-established by the background program. The position of the light source relative to the face is obtained through the illumination angle, thereby obtaining the current light source lighting position based on the current light source position, thus lighting the dark areas. Compared with existing Photoshop software, the present invention simplifies the user-facing operation method. Users only need to drag the light source position to obtain the corresponding lighting position, thereby lighting the dark areas in the human image. Furthermore, this invention can automatically detect dark areas in a portrait and automatically adjust the position of the supplementary light source based on the location of the dark areas, thereby moving the supplementary light source to a position that can directly illuminate the dark areas, thus providing supplementary lighting for the darker areas of the portrait. Additionally, this invention proposes a new method for generating a 3D portrait mesh. Based on the (u, v) corresponding to each face pixel (x, y), the vertex coordinates (x, y, z') corresponding to each face pixel (x, y) are calculated, thus obtaining the depth value z' of each face pixel. A fused portrait depth map is obtained based on the depth value z', and a fused 3D portrait mesh is obtained based on the fused portrait depth map. This increases the 3D information of the portrait, especially by fusing more accurate 3D facial information, making the lighting more three-dimensional, realistic, and natural. Attached Figure Description

[0053] Figure 1 This is a block diagram of one embodiment of the electronic device of the present invention.

[0054] Figure 2 This is a schematic diagram of a real-time or static image according to an embodiment of the present invention.

[0055] Figure 3 This is a flowchart of one embodiment of the stereoscopic lighting method for a three-dimensional human body based on the present invention.

[0056] Figure 4 yes Figure 3 Generate 3D human portrait mesh Figure 1 Sub-flowcharts of the implementation method.

[0057] Figure 4a yes Figure 4 A 3D mesh image of a human face generated in the process.

[0058] Figure 4b yes Figure 4aA schematic diagram showing the coordinates of the ray after it intersects with one of the triangular meshes.

[0059] Figure 5 yes Figure 3 Generate 3D human portrait mesh Figure 1 Sub-flowcharts of the implementation method.

[0060] Figure 6 yes Figure 3 The flowchart for obtaining supplementary light source information.

[0061] Figure 6a yes Figure 6 A schematic diagram of the spherical coordinate system constructed in the diagram.

[0062] Figure 7 This is a block diagram of one embodiment of the stereoscopic lighting device based on a three-dimensional human body according to the present invention. Detailed Implementation

[0063] The following description, in conjunction with the accompanying drawings and specific embodiments, will further explain the supplementary lighting method, device, memory, and electronic device based on three-dimensional human body according to the present invention. However, this explanation does not constitute an improper limitation on the technical solution of the present invention.

[0064] Please see Figure 1 , Figure 1 A schematic diagram of the internal structure of an embodiment of an electronic device is shown. The electronic device 100 may include a mobile phone, camera, tablet computer, personal digital assistant (PDA), point of sales (POS), in-vehicle computer, desktop computer, laptop, server, etc. Internally, the electronic device 100 may include a processor 101, a memory 102 connected to the processor 101, a display screen 103, and a camera 104. The electronic device 100 may also include external devices such as a speaker 105 and a microphone 106. When the electronic device 100 communicates wirelessly with the outside world, it may also include a radio frequency (RF) circuit 107 and a connected antenna, a wireless Fidelity (WiFi) module 108 and a connected antenna, etc. Furthermore, the electronic device 100 also includes a power supply 109 for powering it. Those skilled in the art will understand that... Figure 1 The internal structure of the electronic device 100 shown does not constitute a limitation on the electronic device 100 and may include, but is not limited to, other components. Figure 1 Showing more or fewer components, or combining certain components, or different component arrangements.

[0065] The following text combines Figure 1The various components of the electronic device 100 are described in detail below. The memory 102 can be used to store programs and data. The processor 101 executes various functional applications and data processing of the electronic device 100 by running programs stored in the memory 102, such as executing the supplementary lighting method described below by running the corresponding computer program stored in the memory 102. The memory 102 may mainly include a program storage area and a data storage area. The program storage area may store the operating system (e.g., Android or iOS operating system), at least one application required for a function (e.g., sound playback function, image playback function, image supplementary lighting function, etc.); the data storage area may store data created based on the use of the electronic device 100 (e.g., audio data, phone book, etc.). Furthermore, the memory 102 may include a high-speed random access memory 102, and may also include non-volatile memory 102, such as at least one disk storage device 102, a flash memory device, or other volatile solid-state memory 102.

[0066] Processor 101 is the control and processing center of electronic device 100. It connects various parts of electronic device 100 via various interfaces and lines, and performs various functions and processes data of electronic device 100 by running or executing programs (or "modules") stored in memory 102 and calling data stored in memory 102, thereby providing overall monitoring of electronic device 100. Optionally, processor 101 may include at least one processing unit; alternatively, processor 101 may integrate application processor 101 and modem processor 101, wherein application processor 101 mainly handles the operating system, user interface, and applications, and modem processor 101 mainly handles wireless communication. It is understood that the modem processor 101 may also not be integrated into processor 101. Processor 101 executes the operating system stored in memory 102, calls applications, and completes the functions provided by the applications.

[0067] In this embodiment, the operating system in the electronic device 100 calls the program stored in the memory 102 to control the display screen 103 during or after shooting, and provides supplementary lighting to the subject through the display screen 103 to complete the supplementary lighting process provided by this embodiment of the invention.

[0068] The RF circuit 107 can be used to send and receive information or, during a call, to complete the reception and transmission of signals. Specifically, after receiving downlink information sent by the base station, it hands the downlink information over to the processor 101 for processing; additionally, it sends uplink data to the base station. Typically, the RF circuit 107 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the RF circuit 107 can also communicate wirelessly with networks and other devices. The wireless communication can use any communication standard or protocol, including but not limited to: Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0069] Figure 1 In this configuration, the audio circuit, speaker 105, and microphone 106 provide an audio interface between the user and the electronic device 100. The audio circuit converts the received audio data into electrical signals and transmits them to the speaker 105, where the speaker 105 converts them into sound signals for output. On the other hand, the microphone 106 converts the collected sound signals into electrical signals, which are then received by the audio circuit, converted into audio data, and output to the RF circuit 107 for transmission to other electronic devices 100, such as a mobile phone, or to the memory 102 for further processing.

[0070] WiFi is a short-range wireless transmission technology. Electronic device 100, through WiFi module 108, can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 1 WiFi module 108 is shown, but it is understood that it is not a necessary component of electronic device 100 and can be omitted as needed without changing the essence of the embodiments of the present invention.

[0071] The power supply 109 can be logically connected to the processor 101 through a power management system, thereby enabling the management of charging, discharging, and power consumption. The electronic device 100 may also include sensors 110 (e.g., light sensors, motion sensors, etc.). Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display unit according to the ambient light level, and the proximity sensor can turn off the display unit and / or backlight when the electronic device 100 is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and when stationary, it can detect the magnitude and direction of gravity. It can be used for applications that recognize the phone's posture (e.g., landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition functions (e.g., pedometer, tapping), etc.

[0072] Other sensors that may be configured on the electronic device 100, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here. Furthermore, the electronic device 100 may also include NFC (Near Field Communication) modules, Bluetooth modules, etc., which will also not be described in detail here.

[0073] The following describes the supplementary lighting method and apparatus based on a three-dimensional human body according to embodiments of the present invention.

[0074] The stereoscopic image supplementary lighting method based on a three-dimensional human body provided by this invention can be executed by the electronic device 100 as described above. The embodiments of this invention involve various techniques for image capture using the electronic device 100, which is used to implement various image capture functions. The image capture process of the electronic device 100, where the user captures an image, is described below through a specific scenario:

[0075] Scene 1:

[0076] 1. On the mobile phone, the user clicks the camera application icon, enters the shooting interface, and then turns on the camera to enter the shooting mode.

[0077] 2. The mobile phone determines whether the brightness of the portrait (e.g., face) area is significantly dark based on the brightness of the portrait displayed in the camera interface and the brightness relationship between the portrait and the surrounding environment. It can determine whether the area is a dark area by using a preset threshold.

[0078] 3. Use the virtual fill light source to compensate for the brightness of the darker area. This step can be done using the fill light method described below, thereby achieving real-time fill light during the image capture process.

[0079] Scene 2:

[0080] 1. On the mobile phone, the user clicks the camera application icon, enters the shooting interface, and then turns on the camera to enter the shooting mode.

[0081] 2. For example Figure 2 As shown, the shooting interface displays the real-time image 201 captured by the camera and an adjustable area of ​​the fill light source (the adjustable area is as follows). Figure 2 As shown in circle 202), the image of the light source represents the supplementary light source (e.g., Figure 2 (as shown in icon 203 of the solar symbol) and the intensity adjustment range of the supplementary light source (such as...) Figure 2 (As shown in the vertically arranged light source brightness bar 204 and the brightness adjustment component 205 on the light source brightness bar 204);

[0082] 3. Users can drag the sun icon 203 within the large circle 202 to adjust the position of the supplementary light source relative to the darker areas in the real-time image, thereby compensating for the brightness of those darker areas and increasing their overall brightness during image capture. The color temperature of the supplementary light source can be automatically adjusted based on the color temperature of the background in the real-time image. Users can drag the brightness adjustment component 205 vertically along the light source brightness bar 204 to adjust the brightness of the supplementary light source. When the user drags the brightness adjustment component 205 upwards along the light source brightness bar 204, it indicates an increase in the light source brightness; when the user drags the brightness adjustment component 205 downwards along the light source brightness bar 204, it indicates a decrease in the light source brightness.

[0083] Scene 3:

[0084] 1. On the mobile phone, the user clicks the camera application icon, enters the shooting interface, and then turns on the camera to enter the shooting mode.

[0085] 2. Generate an image (photo) based on the shooting command generated by the user pressing the shutter button;

[0086] 3. For example Figure 2As shown, the image display interface displays an adjustable area for the supplementary light source (the adjustable area is shown as the large circle 202 in the figure), a light source image representing the supplementary light source (shown as the sun icon 203 in the figure), and the intensity adjustment range of the supplementary light source (shown as the vertically set light source brightness bar 204 and the brightness adjustment piece 205 on the light source brightness bar 204 in the figure). Users can drag the sun icon 203 within the large circle 202 to adjust the position of the supplementary light source relative to darker areas in the image, thereby providing supplementary lighting to those darker areas. The color temperature of the supplementary light source can be automatically adjusted based on the color temperature of the background in the real-time image. Users can drag the brightness adjustment piece 205 vertically along the light source brightness bar 204 to adjust the brightness of the supplementary light source. When the user drags the brightness adjustment piece 205 upwards along the light source brightness bar 204, it indicates an increase in the light source brightness based on the current brightness; when the user drags the brightness adjustment piece 205 downwards along the light source brightness bar 204, it indicates a decrease in the light source brightness based on the current brightness.

[0087] 4. Based on the user-selected light source location and brightness, the phone generates an image with supplemental lighting.

[0088] Scene 4:

[0089] 1. On the mobile phone, the user clicks the camera application icon, enters the shooting interface, and then turns on the camera to enter the shooting mode.

[0090] 2. Generate an image (photo) based on the shooting command generated by the user pressing the shooting button;

[0091] 3. In the image display interface, automatically identify darker areas in the image, such as darker areas on a person's face, determine the direction of the fill light based on the darker area, and adjust the position and parameters of the fill light source (such as adjusting the color temperature of the light source based on the color temperature of the background, thereby adjusting the color and brightness of the fill light).

[0092] 4. Based on the adjusted position and parameters of the supplementary light source, supplementary light is applied to the darker areas of the image to generate a supplemented image.

[0093] Please see Figure 3 , Figure 3 This is a flowchart of a stereoscopic lighting method based on a 3D human body for the above scenario. The stereoscopic lighting method based on a 3D human body specifically includes the following steps:

[0094] S101. Obtain the human image from the image.

[0095] In real-time shooting applications, if the captured real-time image in the shooting interface contains a human figure, this step can be initiated directly. Alternatively, this step can be determined automatically by assessing whether the human figure area in the image requires supplemental lighting. For example, if the human figure area in the image is determined to require supplemental lighting, this step will proceed.

[0096] In applications where still images are generated after shooting, if a human figure is detected in the image, this step can be initiated directly. Alternatively, this step can be determined automatically by assessing whether the human figure area in the image requires additional lighting. For example, if the human figure area in the image is determined to require additional lighting, this step is initiated.

[0097] S102. Based on the acquired portrait, generate a portrait depth map, 3D key points of the face, and a portrait grayscale image.

[0098] The step of generating a human portrait depth map specifically includes the following sub-steps:

[0099] S1021. Obtain the depth map of the image; wherein, the depth map (PortraitDepth) can be generated using existing known techniques. The depth map is a grayscale image used to represent the distance information of each pixel in the image from the camera (or observer). The closer to the camera, the larger the grayscale value; conversely, the farther away, the smaller the grayscale value. Objects at infinite distance have a grayscale value of 0.

[0100] S1022. Based on the acquired portrait, generate a portrait mask; wherein, the portrait mask can be generated using existing known technologies, and some publicly available AI algorithms (such as Midas, DepthAnything, etc.) can generate good depth maps. The portrait mask is a grayscale image used to separate the person from the background, with the person's grayscale value close to 255 and the background's grayscale value close to 0.

[0101] S1023. Generate a portrait depth map based on the portrait mask and the image depth map; wherein, a depth map containing only the portrait is cropped from the image depth map, with a background depth value of 0.

[0102] Facial 3D landmarks are key points in a facial image that have specific meanings and locations, used to describe the position and shape of different facial features. In the steps of generating facial 3D landmarks and grayscale images, artificial intelligence technology is used to generate them quickly and accurately. For example, some publicly available AI algorithms (such as Google Face Landmarks) can generate relatively good facial 3D landmarks.

[0103] S103. Generate a 3D mesh image of the human face based on the human face depth map and 3D key points of the human face.

[0104] Please see Figure 4 , Figure 4 A sub-flowchart of generating a 3D mesh image of a human figure is shown in one embodiment. In this embodiment, step S103 includes the following sub-steps:

[0105] S1031. Based on the 3D key points of the face, generate a 3D mesh map of the face and a face mask.

[0106] S1032. Calculate (u, v) corresponding to each face pixel (x, y) in the face mask, wherein (u, v) corresponding to each face pixel (x, y) refers to the intersection point P(x, v) of the ray passing through the face pixel (x, y) and the corresponding triangular mesh. p y p , z p The normalized value of the reciprocal of the distance between the two vertices of the triangular mesh.

[0107] Please see Figure 4a and 4b For each face pixel (x, y), assume a ray with a starting point of (0, 0, 0) and an ending point of (x, y, 10000). Determine whether the given ray intersects a triangular mesh in the 3D face mesh, which has three vertices P1(x1, y1, z1), P2(x2, y2, z2), and P3(x3, y3, z3). The intersection point of the ray and the triangle can be calculated as P(x, y1, z2). p y p , z p The normalized values ​​u and v are the reciprocals of the distances from the intersection point on the triangle to vertices P2 and P3, where u and v range from 0 to 1, and u+v ranges from 0 to 1. Based on this, step S1032 includes the following sub-steps:

[0108] S1032a, Obtain the intersection point P(x, y) of the ray passing through each face pixel (x, y) when it passes through the corresponding triangular mesh. p y p , z p ).

[0109] S1032b, Based on the intersection point P(x, y) of the triangular mesh corresponding to each face pixel (x, y). p y p , z p ), calculate (u, v) corresponding to each face pixel (x, y); where u represents the intersection point P(x, y). py p , z p v is the normalized value of the reciprocal of the distance between the intersection point P(x) and the vertex P2 of the intersecting triangular mesh, where v represents the intersection point P(x). p y p , z p The normalized value of the reciprocal of the distance to vertex P3 of the intersecting triangular mesh; u and v both take values ​​in the range (0, 1), and u+v takes values ​​in the range (0, 1).

[0110] S1033. Based on the (u, v) corresponding to each face pixel (x, y), calculate the vertex coordinates (x, y, z') corresponding to each face pixel (x, y) on the face mask, where z' represents the depth value of the face pixel (x, y) in the portrait depth map, z' = (1-uv)*z1' + u*z2' + v*z3', where z1' represents the depth value of vertex P1 of the triangle, z2' represents the depth value of vertex P2 of the triangle, and z3' represents the depth value of vertex P3 of the triangle. The depth value z' represents the distance of each face pixel (x, y) from the camera.

[0111] S1034. Based on the portrait depth map, calculate the average depth M of the face mask region relative to the portrait depth map, that is, calculate the average depth M of the face mask region.

[0112] S1035. Add the depth mean M to the vertex coordinates (x, y, z') of each face pixel (x, y) to obtain the new vertex coordinates (x, y, z') of each face pixel (x, y) in the face mask, thus obtaining the fused face depth map after fusing the face mask and the face depth map.

[0113] S1036. Based on the fused portrait depth map, obtain the depth value z' of all portrait pixels (x, y), and then generate the portrait 3D mesh map.

[0114] Please see Figure 5 , Figure 5 A sub-flowchart of generating a 3D mesh image of a human figure is shown in another embodiment. In this embodiment, step S103 includes the following sub-steps:

[0115] S103a. Generate a 3D face mesh map based on 3D key points of the face.

[0116] S103b. Calculate (u, v) corresponding to each face pixel (x, y) in the face mask, wherein (u, v) corresponding to each face pixel (x, y) refers to the normalized value of the reciprocal of the distance between the intersection point P(xp, yp, zp) of the ray passing through the face pixel (x, y) and the corresponding triangular mesh and the two vertices of the triangular mesh when the ray passes through the triangular mesh.

[0117] S103c: Based on the (u, v) corresponding to each face pixel (x, y), obtain the (u, v) of each face pixel (x, y) in the face mask. In this step, when the ray has no intersection with the 3D mesh of the face region, the values ​​of u and v are both equal to 0, thus defining the face pixel (x, y) and its corresponding (u, v) in the face mask region. When the values ​​of u and v are both equal to 0, their corresponding z' are both 0.

[0118] S103d: Based on (u, v) corresponding to each face pixel (x, y) on the face mask, calculate the vertex coordinates (x, y, z') corresponding to each face pixel (x, y) on the face mask.

[0119] S103e. Based on the portrait depth map, calculate the mean depth M of the face mask area corresponding to the portrait depth map;

[0120] S103f: Add the depth mean M to the vertex coordinates (x, y, z') of each portrait pixel to obtain the new vertex coordinates (x, y, z') of each portrait pixel in the face mask, thus obtaining the fused portrait depth map after the face mask and the portrait depth map are fused.

[0121] S103g: Based on the fused portrait depth map, the depth value z' of all portrait pixels (x, y) is obtained, and then the portrait 3D mesh map is generated.

[0122] S104. Obtain the supplementary lighting source information corresponding to the portrait, wherein the supplementary lighting source information includes the light source intensity and the relative position of the supplementary lighting source and the portrait. Please refer to [link to relevant documentation]. Figure 2 and Figure 6 , Figure 6 This is a sub-flowchart for acquiring supplementary lighting source information. This implementation method combines... Figure 2 The illustrated portrait image serves as an example to illustrate the information of the supplementary lighting source. This step includes the following sub-steps:

[0123] S1041. Construct a spherical coordinate system corresponding to the human figure.

[0124] like Figure 6a As shown, Figure 6aThis is a schematic diagram of the constructed spherical coordinate system. In this system, the x and y axes form a plane parallel to the face, and the z-axis represents an axis perpendicular to the face. The center coordinate of the spherical coordinate system (x...) c y c (x, y) represents the center of the face in the image, and the pixel coordinates of the fill light source are represented as (x, y). The light shines along the z-axis towards the plane formed by the x and y axes. Figure 6a In the middle, the light ray has a θ, Directional illumination, the included angle ω of the light source coverage can be set to 10-50° (usually set to 15°), and the illumination angle θ, It can be calculated based on the distance L from the center of the light source and the radius r of the sphere.

[0125] S1042. Map the supplementary light source corresponding to the portrait onto a spherical coordinate system, and obtain the coordinates (x, y) of the supplementary light source from the center of the spherical coordinate system. c y c Given the distance L and the radius r of the sphere, the illumination angle θ of the light source is calculated. in:

[0126] The distance L is obtained by the following formula (I):

[0127]

[0128] The illumination angle θ is obtained by the following formula (II):

[0129] sinθ=L / r (II)

[0130] The angle of illumination It can be obtained through the following formula (iii):

[0131]

[0132] S1043, Based on illumination angle θ, The relative positions of the supplementary light source (x, y) and the human figure are obtained.

[0133] S105. Based on the supplementary light source information, the grayscale image of the portrait, and the 3D mesh image of the portrait, generate a stereoscopic supplementary lighting image of the portrait. Here, a spherical harmonic function can be used to simulate diffuse reflection lighting, representing a directional light illuminating the 3D portrait, thereby providing supplementary lighting for the portrait in the image. This is a well-known technique and will not be described in detail.

[0134] Please see Figure 7 , Figure 7A block diagram of one embodiment of the stereoscopic lighting device based on a three-dimensional human body according to the present invention is shown. In this embodiment, the stereoscopic lighting device based on a three-dimensional human body includes a human image acquisition module 310, a human image depth map generation module 320, a human face 3D key point generation module 330, a human image grayscale image generation module 340, a human image 3D mesh map generation module 350, a light source information acquisition module 360, and a human image stereoscopic lighting map generation module 370. The portrait acquisition module 310 is used to acquire a portrait in an image; the portrait depth map generation module 320 is used to generate a portrait depth map based on the acquired portrait; the face 3D key point generation module 330 is used to generate face 3D key points based on the acquired portrait; the portrait grayscale image generation module 340 is used to generate a portrait grayscale image based on the acquired portrait; the portrait 3D mesh image generation module 350 is used to generate a portrait 3D mesh image based on the portrait depth map and face 3D key points; the light source information acquisition module 360 ​​is used to acquire supplementary light source information corresponding to the portrait, wherein the supplementary light source information includes the light source intensity and the relative position of the supplementary light source and the portrait; the portrait stereo supplementary lighting image generation module 370 is used to generate a portrait stereo supplementary lighting image based on the supplementary light source information, the portrait grayscale image, and the portrait 3D mesh image.

[0135] In real-time shooting applications, when a person is captured in the real-time image captured by the shooting interface, the portrait acquisition module 310 can directly acquire the portrait in the real-time image. The portrait acquisition module 310 can also automatically determine whether to acquire the portrait in the real-time image by judging whether the portrait area in the image needs supplemental lighting. For example, if the portrait area in the image is determined to need supplemental lighting, then the portrait in the real-time image is acquired. In applications where a still image is generated after shooting, if a person is detected in the image, the portrait acquisition module 310 can directly acquire the portrait in the still image. The portrait acquisition module 310 can also automatically determine whether to acquire the portrait in the image by judging whether the portrait area in the image needs supplemental lighting. For example, if the portrait area in the image is determined to need supplemental lighting, then the portrait in the still image is acquired.

[0136] The portrait depth map generation module 320 is used to generate a portrait depth map based on the acquired portrait. The portrait depth map generation module 320 includes an image depth map acquisition submodule, a portrait mask generation submodule, and a portrait depth map generation submodule.

[0137] The image depth map acquisition submodule is used to acquire the depth map of the image. The depth map can be generated using existing known techniques. This depth map is a grayscale image representing the distance of each pixel in the image from the camera (or observer). The closer an object is to the camera, the higher its grayscale value; conversely, the farther away an object is, the lower its grayscale value. Objects at infinity have a grayscale value of 0.

[0138] The portrait mask generation submodule is used to generate a portrait mask based on the acquired portrait. The portrait mask can be generated using existing known technologies, and some publicly available AI algorithms (such as Midas and DepthAnything) can generate relatively good depth maps. The portrait mask is a grayscale image used to separate the person from the background; the person's grayscale value is close to 255, and the background's grayscale value is close to 0.

[0139] The portrait depth map generation submodule is used to generate a portrait depth map based on a portrait mask and the depth map of the image; wherein, a depth map containing only the portrait is cropped from the depth map of the image, with a background depth value of 0.

[0140] The portrait 3D mesh generation module 350 is used to generate a portrait 3D mesh based on the portrait depth map and 3D facial key points. The portrait 3D mesh generation module 350 includes the following sub-modules:

[0141] The Face 3D Mesh Generation Submodule is used to generate a face 3D mesh based on face 3D key points.

[0142] The face mask generation submodule is used to generate face masks based on 3D key points of the face.

[0143] The first calculation submodule is used to calculate the (u, v) corresponding to each face pixel (x, y) in the face mask, wherein the (u, v) corresponding to each face pixel (x, y) refers to the intersection point P(x, v) of the ray passing through the face pixel (x, y) and the corresponding triangular mesh. p y p , z p The normalized value is the reciprocal of the distance between the two vertices of the triangular mesh and the face pixel (x, y). The first calculation submodule can calculate (u, v) corresponding to each face pixel (x, y) using the method described above, which will not be repeated here.

[0144] The second calculation submodule calculates the vertex coordinates (x, y, z') of each face pixel (x, y) on the face mask based on the (u, v) corresponding to each face pixel (x, y). Here, z' represents the depth value of the face pixel (x, y) in the face depth map, and z' = (1-uv)*z1' + u*z2' + v*z3', where z1' represents the depth value of vertex P1 of the triangle, z2' represents the depth value of vertex P2 of the triangle, and z3' represents the depth value of vertex P3 of the triangle. The depth value z' represents the distance of each face pixel (x, y) from the camera.

[0145] The third calculation submodule is used to calculate the average depth M of the face mask region relative to the face depth map based on the face depth map, that is, to calculate the average depth M of the face mask region.

[0146] The fused portrait depth map generation submodule is used to add the depth mean M to the vertex coordinates (x, y, z') of each face pixel (x, y) to obtain the new vertex coordinates (x, y, z') of each face pixel (x, y) in the face mask, thereby obtaining the fused portrait depth map after fusing the face mask and the portrait depth map.

[0147] The portrait 3D mesh generation submodule is used to obtain the depth value z' of all portrait pixels (x, y) based on the fused portrait depth map, and then generate the portrait 3D mesh map.

[0148] The light source information acquisition module 360 ​​includes a spherical coordinate system construction submodule, a light source illumination angle calculation submodule, and a light source position acquisition submodule. Wherein:

[0149] The spherical coordinate system construction submodule is used to construct a spherical coordinate system corresponding to the human face. In this system, the x and y axes form a plane parallel to the face, and the z axis represents an axis perpendicular to the face. The center coordinate of the spherical coordinate system (x...) c y c (x, y) represents the center of the face in the image, and the pixel coordinates of the fill light source are represented as (x, y). The light ray shines along the z-axis towards the plane formed by the x and y axes. Figure 6a In the middle, the light ray has a θ, Directional illumination, the included angle ω of the light source coverage can be set to 10-50° (usually set to 15°), and the illumination angle θ, It can be calculated based on the distance L from the center of the light source and the radius r of the sphere.

[0150] The light source illumination angle calculation submodule is used to map the supplementary light source corresponding to the portrait onto a spherical coordinate system, and obtain the coordinates (x, y) of the supplementary light source from the center of the spherical coordinate system. c y c Given the distance L and the radius r of the sphere, the illumination angle θ of the light source is calculated. The illumination angle θ, It can be calculated using formulas (i), (ii) and (iii) shown above, and will not be repeated here.

[0151] The light source position acquisition submodule is used to obtain the position based on the illumination angle θ. The relative positions of the supplementary light source (x, y) and the human figure are obtained.

[0152] In summary, the three-dimensional human body-based stereoscopic lighting method, device, memory, and electronic device of the present invention have the following beneficial effects: The present invention can automatically determine whether the human image information has dark areas and the location of the dark areas based on the human image information detected in real-time or static images, and automatically enter the lighting process. Alternatively, based on the adjustable area of ​​the light source, the adjustment range of the light source, and the image representing the light source shown in the real-time or static image, the dark areas of the face can be lit by manually dragging the position of the light source. In the lighting process, the illumination angle of the light source is calculated based on the spherical coordinate system relative to the face pre-established by the background program. The position of the light source relative to the face is obtained through the illumination angle, thereby obtaining the current light source lighting position based on the current light source position, and thus lighting the dark areas. Compared with existing Photoshop software, the present invention simplifies the operation method for the user. The user only needs to drag the position of the light source to obtain the corresponding lighting position, and then light the dark areas in the human face. Furthermore, this invention can automatically detect dark areas in a portrait and automatically adjust the position of the supplementary light source based on the location of the dark areas, thereby moving the supplementary light source to a position that can directly illuminate the dark areas, thus providing supplementary lighting for the darker areas of the portrait. Additionally, this invention proposes a new method for generating a 3D portrait mesh. Based on the (u, v) corresponding to each face pixel (x, y), the vertex coordinates (x, y, z') corresponding to each face pixel (x, y) are calculated, thus obtaining the depth value z' of each face pixel. A fused portrait depth map is obtained based on the depth value z', and a fused 3D portrait mesh is obtained based on the fused portrait depth map. This increases the 3D information of the portrait, especially by fusing more accurate 3D facial information, making the lighting more three-dimensional, realistic, and natural.

[0153] It should be noted that the prior art portion of the protection scope of this invention is not limited to the embodiments given in this application. All prior art that does not contradict the solution of this invention, including but not limited to prior patent documents, prior publications, prior public uses, etc., can be included in the protection scope of this invention.

[0154] Furthermore, the combination of the technical features in this case is not limited to the combination methods described in the claims of this case or the combination methods described in the specific embodiments. All technical features described in this case can be freely combined or combined in any way, unless they contradict each other.

[0155] It should also be noted that the embodiments listed above are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments, and similar changes or modifications made thereto are those that can be directly derived or easily conceived by those skilled in the art from the content disclosed in the present invention, and should all fall within the protection scope of the present invention.

Claims

1. A three-dimensional human body based stereoscopic light compensation method, comprising the steps of: obtaining a human figure in an image; generating a human figure depth map, a human face 3D key point and a human figure gray scale map based on the obtained human figure; generating a human figure 3D mesh map based on the human figure depth map and the human face 3D key point; obtaining light compensation light source information corresponding to the human figure, wherein the light compensation light source information comprises light source intensity and relative position of the light compensation light source to the human figure; generating a human figure stereoscopic light compensation map based on the light compensation light source information, the human figure gray scale map and the human figure 3D mesh map.

2. The three-dimensional body based volumetric light supplementation method of claim 1, wherein, In the step of generating the human figure depth map, the following sub-steps are included: obtaining a depth map of the image; generating a human figure mask based on the obtained human figure; generating the human figure depth map based on the human figure mask and the depth map of the image.

3. The three-dimensional body based volumetric light compensation method of claim 1, wherein, In the step of generating the human figure 3D mesh map based on the human figure depth map and the human face 3D key point, the following sub-steps are included: generating a human face 3D mesh map and a human face mask based on the human face 3D key point; corresponding to each face pixel (x, y) in the face mask, wherein the (u, v) corresponding to each face pixel (x, y) refers to the normalized value of the reciprocal of the distance between the intersection point P(x p , y p , z p ) of the ray passing through the corresponding triangular mesh through the face pixel (x, y) and two vertices of the triangular mesh; calculating vertex coordinates (x, y, z') of each human face pixel (x, y) on the human face mask based on (u, v) corresponding to each human face pixel (x, y), wherein z' represents a depth value of the human face pixel (x, y) in the human figure depth map; calculating a depth mean value M of the human face mask region relative to the human figure depth map based on the human figure depth map; adding the depth mean value M to the vertex coordinates (x, y, z') of each human face pixel (x, y) to obtain new vertex coordinates (x, y, z') of each human face pixel (x, y) in the human face mask, thereby obtaining a fused human figure depth map after the human face mask and the human figure depth map are fused; obtaining depth values z' of all human figure pixels (x, y) based on the fused human figure depth map, and further generating the human figure 3D mesh map.

4. The three-dimensional body based volumetric light compensation method of claim 3, wherein, In the step of calculating (u, v) corresponding to each human face pixel (x, y) on the human face mask, the following steps are included: Obtaining the intersection point P(x, y, z) of the ray passing through each human face pixel (x, y) through the corresponding triangular mesh p p p );​​ Based on the intersection point P(x, y) of the triangular mesh corresponding to each face pixel (x, y) p y p , z p ), calculate (u, v) corresponding to each face pixel (x, y); where u represents the intersection point P(x, y). p y p , z p v is the normalized value of the reciprocal of the distance between the intersection point P(x) and the vertex P2 of the intersecting triangular mesh, where v represents the intersection point P(x). p y p , z p The normalized value of the reciprocal of the distance to vertex P3 of the intersecting triangular mesh.

5. The three-dimensional body based volumetric light compensation method of claim 4, wherein, In the step of calculating vertex coordinates (x, y, z') of each human face pixel (x, y) on the human face mask based on (u, v) corresponding to each human face pixel (x, y), the following formula is used: z' = (1-u-v)*z1'+u*z2'+v*z3', wherein z1' represents a depth value of a vertex P1 of a triangle, z2' represents a depth value of a vertex P2 of the triangle, and z3' represents a depth value of a vertex P3 of the triangle.

6. The three-dimensional body based volumetric light supplementation method of claim 1, wherein, In the step of obtaining the relative position of the light compensation light source relative to the human figure, the following sub-steps are included: A spherical coordinate system corresponding to the human image is constructed, wherein the center coordinates (x c , y c ) of the spherical coordinate system represent the center of the face in the image, and the pixel coordinates of the light supplementing light source are represented as (x, y); The light compensation light source corresponding to the portrait is mapped into a spherical coordinate system, a distance L of the light compensation light source (x, y) from a center coordinate (x c , y c ) of the spherical coordinate system and a spherical radius r are obtained, and an irradiation angle θ of the light source is calculated. Based on the illumination angle θ, The relative position of the light source (x, y) and the human figure is obtained.

7. The three-dimensional body based volumetric light compensation method of claim 6, wherein, To obtain the distance (x, y) of the supplementary light source from the center of the spherical coordinate system (x, y), c y c Given the distance L and the radius r of the sphere, the illumination angle θ of the light source is calculated. In the following steps, the distance L is obtained using the following formula (a): the irradiation angle θ is obtained by the following formula (II): sin θ = L / r (II) The illumination angle By the following equation (three):

8. A three-dimensional human body based stereoscopic light supplementing device, characterized by, which comprises: a human figure obtaining module, configured to obtain a human figure in an image; a human figure depth map generating module, configured to generate a human figure depth map based on the obtained human figure; a human face 3D key point generating module, configured to generate a human face 3D key point based on the obtained human figure; a human figure gray scale map generating module, configured to generate a human figure gray scale map based on the obtained human figure; a human figure 3D mesh map generating module, configured to generate a human figure 3D mesh map based on the human figure depth map and the human face 3D key point; The light source information acquisition module is configured to acquire light supplement light source information corresponding to the portrait, wherein the light supplement light source information comprises light source intensity and a relative position between the light supplement light source and the portrait. The portrait stereoscopic light supplement image generation module is configured to generate a portrait stereoscopic light supplement image based on the light supplement light source information, the portrait grayscale image, and the portrait 3D mesh image.

9. A memory having stored therein a computer program, characterized by: The computer program is read and run by a processor, and the computer program performs the method of any one of claims 1 to 7.

10. An electronic device, comprising: The computer program is read and run by a processor, and the computer program performs the method of any one of claims 1 to 7. The computer program is read and run by a processor, and the computer program performs the method of any one of claims 1 to 7. The computer program is read and run by a processor, and the computer program performs the method of any one of claims 1 to 7.

Images