Method and apparatus for performing object lighting manipulation on an image

Abstract

A method and a lighting manipulation device for performing an object lighting manipulation operation on a first image are provided. The method includes: receiving a first image and a light direction input; generating a second image with a first object detected from the first image; determining a second object lighting of the second image based on the light direction; and replacing the first object lighting of the first image with the second object lighting.

Description

Technical Field

[0001] The present disclosure relates to manipulation of illumination of objects in images. More particularly, the present disclosure relates to a method of performing object illumination manipulation on images in an image sequence and a user computing device implementing the method. Background Art

[0002] Figure 1 is a diagram showing a general operation of performing object lighting manipulation on an image according to the related art.

[0003] The method for manipulating the illumination of an object in an image generally includes: inputting an object image in operation S10, detecting object landmarks in operation S11, extracting object albedo and illumination in operation S12, changing the object illumination in operation S13, mixing the object albedo and the changed illumination in operation S14, and applying the mixed object albedo and the changed illumination to the object image in operation S15.

[0004] Mitsubishi Electric Laboratories, Inc., U.S. Pat. No. 7,324,688 B2, dated January 29, 2008, entitled "Face relighting for normalization of directional lighting," describes a method for performing lighting manipulation using a block-wise optimization algorithm to extract albedo and lighting maps for further face lighting normalization. The main drawback of the method disclosed in the '688 patent is that the unique facial features are altered by applying the lighting manipulation method to a facial image. As a result, a face on an image that has been subjected to such lighting manipulation method becomes similar to a reference face for such lighting manipulation.

[0005] Another U.S. patent, US 8,582,896 B2, dated November 12, 2013, entitled "Separating directional lighting variability in statistical face modeling based on texture space decomposition" by Digital Optics Europe Ltd. et al., describes a method for decomposing an image into albedo and lighting using a universal face model created by projecting a principal component analysis (PCA) onto a uniform light subspace trained on an arbitrary image database. According to the description of the '896 patent, this method can only eliminate lighting to a certain extent. In addition, the method of the '896 patent cannot eliminate colored lighting and cannot be designed to process facial images at high resolution.

[0006] Another US patent US 7,433,807 B2, dated October 7, 2008, by Microsoft Corporation, entitled "Facial image processing", describes a method of processing facial images using several flashes to create a 3D facial model and to separate albedo from illumination based on the 3D model normal. The method taught in the '807 patent requires a large amount of hardware that cannot be integrated into a user's mobile device. Furthermore, the algorithm disclosed in the '807 patent must analyze several different images, and therefore, it is computationally intensive and cannot be implemented in real time.

[0007] Another US patent US 9,858,654 B2 of January 2, 2018, entitled "Image manipulation" by ANTHROPICS Technologies Ltd. describes a method of processing an image that forms a 2D model based on PCA that is fused with an input facial image to produce an output image with a new lighting controlled based on the PCA coefficients. The method disclosed in the '654 patent is unable to remove harsh shadows and results in altering distinctive facial features in the image. Furthermore, the method proposed in the '654 patent is rather slow (due to the use of the corresponding PortraitPro software) and therefore cannot be used in real-time applications.

[0008] Other implementations are known in the art, but do not adequately address the above-mentioned issues.

[0009] The above information is presented as background information only to assist with understanding the present disclosure. No determination has been made and no assertion has been made as to whether any of the above content may be used as prior art with respect to the present disclosure. Summary of the invention

[0010] Solution to the problem

[0011] Aspects of the present disclosure will address at least the above problems and / or disadvantages and provide at least the advantages described below. Therefore, one aspect of the present disclosure is to provide a method for performing object lighting manipulation on images in an image sequence and a user computing device implementing the method.

[0012] Advantageous Effects of the Invention

[0013] It can address issues presented as background information. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0015] Figure 1 is a diagram showing a general operation of performing object lighting manipulation on an image according to the related art;

[0016] Figure 2 is a flow chart illustrating performing object lighting manipulation on an image in an image sequence according to an embodiment of the present disclosure;

[0017] Figure 3 is a flow chart illustrating generating adaptive object lighting according to an embodiment of the present disclosure;

[0018] Figure 4 is a flow chart for extracting actual object lighting of an image according to an embodiment of the present disclosure;

[0019] Figure 5 is a flow chart for replacing actual object lighting of an image with adaptive object lighting according to an embodiment of the present disclosure;

[0020] Figure 6 is a flow chart for generating object illumination using object landmarks to obtain rasterized object illumination in which the illumination is defined by a predetermined light direction according to an embodiment of the present disclosure;

[0021] Figure 7 is a schematic diagram of a user computing device implementing a method for performing object lighting manipulation on an image according to an embodiment of the present disclosure;

[0022] Figure 8 is an example of detecting a landmark L on an input image according to an embodiment of the present disclosure;

[0023] Fig. 9is an example of a landmark cropped image according to an embodiment of the present disclosure;

[0024] Fig.10 is an example of generating rasterized object lighting according to an embodiment of the present disclosure;

[0025] Fig.11 is an example of stretched rasterized object lighting according to an embodiment of the present disclosure;

[0026] Fig.12 is an example of adaptive object lighting with smooth regions defined by predetermined object spots generated according to an embodiment of the present disclosure;

[0027] Fig.13 is an example of detected object features such as object spots and object brightness transitions and a lambda map formed for an object cropped image according to an embodiment of the present disclosure;

[0028] Fig.14 is an example of a result of fast global smoothing (FGS) considering object features according to an embodiment of the present disclosure;

[0029] Fig.15 is a flow chart of performing a lighting manipulation operation according to an embodiment of the present disclosure;

[0030] Fig.16 is a structural diagram of a lighting manipulation device for performing a lighting manipulation operation according to an embodiment of the present disclosure; and

[0031] Fig.17 An example of a process of manipulating lighting when the object of an image is a face according to an embodiment of the present disclosure is shown.

[0032] Throughout the drawings, it should be noted that like reference numbers are used to depict the same or similar elements, features, and structures.

[0033] Best Mode for Carrying Out the Invention

[0034] Aspects of the present disclosure will address at least the above problems and / or disadvantages and provide at least the advantages described below. Therefore, one aspect of the present disclosure is to provide a method for performing object lighting manipulation on images in an image sequence and a user computing device implementing the method.

[0035] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0036] According to one aspect of the present disclosure, a method for performing an object lighting manipulation operation on a first image is provided. The method includes: receiving a first image and a light direction input; generating a second image having a first object detected from the first image; determining a second object lighting of the second image based on the light direction; and replacing the first object lighting of the first image with the second object lighting.

[0037] According to another aspect of the present disclosure, a lighting manipulation device for performing an object lighting manipulation operation on a first image is provided. The device includes: an input interface configured to receive a first image and a light direction input; and a processor configured to generate a second image with a first object detected from the first image, determine a second object lighting of the second image based on the light direction, and replace the first object lighting of the first image with the second object lighting.

[0038] Other aspects, advantages, and salient features of the present disclosure will become apparent to those skilled in the art from the following detailed description, which, in conjunction with the accompanying drawings, discloses various embodiments of the present disclosure. DETAILED DESCRIPTION

[0039] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. It includes various specific details to assist in understanding, but these specific details should be considered as exemplary only. Therefore, it will be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0040] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only and not for the purpose of limiting the present disclosure as defined by the appended claims and their equivalents.

[0041] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more of such surfaces.

[0042] Figure 2 is a flow chart illustrating performing object lighting manipulation on images in an image sequence according to an embodiment of the present disclosure.

[0043] refer to Figure 2, a method for real-time processing of an image to improve the illumination and color of an object is shown. The method begins with operation S100, in which an image is input and a predetermined light direction is received. The predetermined light direction may be a default light direction or a user-defined light direction. In an embodiment, the object on the input image may be, but is not limited to, a face, and the object surface may be facial skin. However, it should be understood that the disclosed illumination manipulation method may also be successfully applied to images containing any other object. To this end, a landmark detection algorithm using, for example, a support vector machine (SVM) or any other machine learning algorithm may be trained on object landmarks for which illumination manipulation by the method of the present disclosure is beneficial. After the image is input, a 3D mesh of the object may be prepared, and landmarks associated with object blobs may be marked. In the present disclosure, an object may refer to an object landmark.

[0044] The present disclosure is based on the Lambert diffuse reflection law R=IS, where R∈R N is the resulting image, I∈R N is lighting, S∈R N is the texture (albedo), and N is the number of image pixels. Therefore, the present disclosure provides a method for separating texture from illumination and replacing the original illumination map with a desired illumination map. In operation S105, object landmarks L = {l N}∈R2 and image features.

[0045] Figure 8 An example of detecting landmarks according to an embodiment of the present disclosure is shown.

[0046] refer to Figure 8 , the value n is the number of landmarks and can be set to 68, but is not limited to this particular value. In this context, object landmarks refer to points projected onto the object image plane that correspond to the centers of object reference regions that are predetermined based on a generic 3D view of the object reference region, such as the tip of the nose, the corners of the lips and eyes, the center of the pupil, and so on. Image features in this context refer to unique properties of the object texture that are not present in generic objects.

[0047] Return to reference Figure 2 In operation S110, the image R may be cropped based on the object landmarks. 1 ∈R N , to obtain an object cropped image 5 including the entire object and its landmarks.

[0048] Fig. 9 An example of a landmark cropped image 5 according to an embodiment of the present disclosure is shown.

[0049] refer to Fig. 9Operation S110 reduces the computational complexity of subsequent operations of the method by excluding areas that will not be processed from the input image.

[0050] Return to reference Figure 2 In operation S115, the object image 5 may be converted into a format having a luminance component and a color difference component, such as but not limited to YUV NV21. In operation S120, the object landmarks may be used to generate object illumination to obtain rasterized object illumination 25, wherein the illumination is defined by a predetermined light direction 10. In order to generate the rasterized object illumination, a generic object 3D mesh 15V is generated. g = {v g}∈R 3 and fit it to the detected object landmark L. The value m is the number of vertices of the 3D mesh and can be set to 3448, but is not limited to this specific value. 3 10 The lighting defined is applied to the fitted 3D mesh 15. Then, the generic object texture T∈R M 20 renders the adjusted object 3D mesh 15 and rasterizes it to form a rasterized object lighting 25. In other words, when processing the image sequence, V g fit to L, and perform rendering of each nth (may be 1, 2 or 4) image of the object 3D mesh 15 using L and T to obtain the object illumination M r 25. Therefore, in video processing, instead of performing rendering of a new 3D image for each frame, a certain number of frames are skipped by using the same image for the intermediate frames, thereby saving the computational resources required to implement the method.

[0051] Fig.10 An example of generating rasterized object lighting according to an embodiment of the present disclosure is shown.

[0052] refer to Fig.10 , the object on the image is a face, the object 3D mesh 15 is a 3D head mesh, the generic object texture 20 is a generic face texture, and the rasterized object lighting 25 is the rasterized face lighting.

[0053] Figure 6 is a flow chart of generating object illumination using object landmarks to obtain rasterized object illumination in which the illumination is defined by predetermined light directions according to an embodiment of the present disclosure.

[0054] refer to Figure 6, an example flow of operation S120 is shown. In operation S120.1, a generic object 3D mesh may be fitted to a landmark of a detected object represented on a current image in the image sequence. In operation S120.2, rendering of the 3D object may be performed using the object 3D mesh fitted to the object landmark, a predetermined light direction, and a generic object surface texture to obtain rasterized object lighting.

[0055] Return to reference Figure 2 , after obtaining the rasterized object lighting, the method generates adaptive object lighting in operation S125.

[0056] Figure 3 is a flow chart illustrating generating adaptive object lighting according to an embodiment of the present disclosure.

[0057] refer to Figure 3 , showing an example flow of operation S125. In operation S125, an adaptive object lighting M∈R may be generated. N 40. The goal of operations S120 and S125 is to render the target object lighting based on the generated object 3D mesh, the shape and color of which are fitted to the object input image. In operation S125.1, the boundary of the rasterized object lighting 25 can be stretched to the boundary of the object cropped image, that is, the stretched image M r To obtain the image M s That is, the size of the rasterized object lighting 25 can be adjusted to match the size of the object cropped image. To this end, in the embodiment of the present disclosure, for the image boundary M r Each pixel p b , in the representation image M r Find the first non-zero pixel p on a line orthogonal to the boundary of a row or column of pixels 0 If there is p 0 Pixels, then in / p b p 0 Remove a number of non-zero pixels (up to 9 pixels) in the direction to obtain the interior pixel p 1 , and pixel p 0 p 1 Interpolate to p b p 1 . This operation is performed to exclude the black area outside the object caused by rendering the object lighting 25 from the rendered image of the object lighting 25, and to make it easier to inpaint (i.e., recolor) the brightened object image caused by performing the method of the present disclosure into the corresponding image in the input image sequence. Operation S125.1 also enables real-time processing of high-resolution video streams by the method of the present disclosure, because the complexity of inpainting the brightened object image generated by the method into the corresponding image in the input image sequence is greatly reduced.

[0058] Fig.11 An example of stretching rasterized object lighting to form rasterized object lighting with stretched boundaries according to an embodiment of the present disclosure is shown.

[0059] Return to reference Figure 3 In operations S125.2 and S125.3, the Y, U, and V histograms (H YM , H UM , H VM ) and the object cropped image R 1 '5 Y, U and V histograms (H YR , H UR , H VR The calculation of the histogram may be performed by any method. In operation S125.4, the predetermined object surface mask A may be formed by removing the bright and dark areas of the object cropped image 5 defined by the corresponding brightness threshold from the object surface mask. r ∈R N Adjust to the object cropped image 5. To this end, in the embodiment of the present disclosure, adjust the object surface mask A r To obtain the adjusted object surface mask A adj ={a i , i=1..N}. Among them:

[0060]

[0061]

[0062] and

[0063]

[0064] The predetermined object surface mask is a binary object surface mask constructed by shading all regions located inside the convex polygon formed by the object landmarks. The predetermined object surface mask is used to find the image region containing the object surface. Removing the light and dark regions of the object cropped image 5 from the object surface mask is performed to exclude the object surface region containing invalid information about the illumination and color of the object. The corresponding brightness threshold for removing the light and dark regions from the object surface mask can be determined empirically.

[0065] In an embodiment of the present disclosure, the object face mask Af may be formed based on the object landmarks L by generating the object face mask Af using the object landmarks L except for the landmarks corresponding to the eyebrows, the mouth, and the nose. rThen, the color factor can be estimated based on the generated face mask Af and the object cropped image 5. This estimation may include: creating an object surface detection mask Af; p , which is a face mask with everything below the nose and above the eyebrows removed; an average box filter of size 5 (but not limited to this specific value) is applied to the face surface image R of the subject. p =F avg *R' I Perform smoothing and compute color factors:

[0066] and

[0067] c={Y,U,V},

[0068] where σ 0 It can be 1.1.

[0069] As a result, the target object surface mask A can be found r The surface area of ​​the object As:

[0070] In an embodiment of the present disclosure, when the object is an arbitrary object rather than a face, the object surface mask (i.e., a binary mask of the object surface) can be predetermined by shading all areas located inside a convex polygon formed by landmarks predetermined for any object of interest.

[0071] In operation S125.5, the values ​​falling into the removed bright and dark regions of the object cropped image 5 defined by the adjusted object surface mask Aadf may be removed from the obtained U and V histograms of the rasterized object illumination 30 with stretched boundaries and the U and V histograms of the object cropped image 5. In other words, in operation S125.5, zero pixels in Aadf are removed from the UV histogram, and the histogram H is made UM , H VM Equalization is performed to remove values ​​falling into the removed light and dark regions of the object cropped image 5 from the obtained Y, U and V histograms to exclude the object surface regions containing invalid information about the lighting and color of the object.

[0072] In operation S125.6, the brightness, contrast and color of the rasterized object lighting 30 with stretched boundaries can be adjusted by estimating the Y histogram of the object cropped image 5 and the rasterized object lighting 30 with stretched boundaries and applying the equalized U and V histograms of the same image (i.e., the image of the rasterized object lighting 30 with stretched boundaries and the object cropped image 5) to obtain the adaptive object lighting 40 that provides the recolored lighting. The aforementioned application of the equalized U and V histograms to form the adaptive object lighting 40 includes: recoloring the lighting image using their values. In other words, in one embodiment, in order to generate the adaptive object lighting 40. Adjust brightness And determine the equalized U and V histograms as follows:

[0073]

[0074]

[0075]

[0076]

[0077] and

[0078]

[0079] Operations S125.2 to S125.6 also ensure that the brightened object image generated by the method is more easily inpainted into a corresponding image in the input image sequence. In addition, operations S125.2 to S125.6 adapt the target lighting to the brightness and color of the specific object input image, thereby generating adaptive object lighting 40 and providing a realistic inpaint effect, so that unique object features will be preserved in the object image, which will be inpainted into the corresponding image in the image sequence. After forming the adaptive object lighting in operation S125.6, the method generates an adaptive object lighting having a smooth area defined by a predetermined object spot in operation S125.7.

[0080] Fig.12 is an example of adaptive object lighting with smooth regions defined by predetermined object spots generated according to an embodiment of the present disclosure.

[0081] refer to Fig.12 , in operation S125.7, the area defined by the predetermined object spots may be smoothed on the adaptive object lighting 40. In other words, in an embodiment of the present disclosure, the object spots may be smoothed in operation S125.7:

[0082]

[0083] where M is the adaptive object lighting 40 with smoothed object spots. and are average box filters of size 0.25w and 0.05w, respectively. In this case, object blobs refer to regions bounded by previously marked object landmarks that correspond to highly textured regions such as eyes, mouth, eyebrows, nose tip, etc. In embodiments where the object is a face, blobs include but are not limited to eyebrows, eyes, nose, and mouth. Smoothing is performed to prevent blobs from being removed in subsequent operations that redraw the brightened object image into the input image sequence.

[0084] Fig.12 is an example of adaptive object lighting with smooth regions defined by predetermined object spots generated according to an embodiment of the present disclosure.

[0085] refer to Fig.12 , shows an example of adaptive object lighting 40 with smooth areas defined by predetermined object spots.

[0086] Figure 4 is a flow chart of extracting actual object lighting of an image according to an embodiment of the present disclosure.

[0087] refer to Figure 4 , in operation S130, the actual object illumination 70 of the image can be extracted. The goal of operation S130 is to extract the actual object illumination using an edge-preserving fast global smoothing (FGS) filter that reveals low-frequency brightness transitions and preserves harsh shadows for removal in a subsequent operation of patching the brightened image into the input image sequence. In operation S130.1, object spots 50 can be detected by object landmarks L and object surface masks Ar, and object brightness transitions 55 can be detected on the object cropped image 5, and the detected object spots 50 and object brightness transitions 55 are object features 45. The object brightness transitions 55 define unique features of the object. In an embodiment where the object is a face, the spots 50 include, but are not limited to, eyebrows, eyes, nose, and mouth. In order to further improve the processing of the object spots 50, in operation S130.1, an object spot mask Ab can be additionally formed from the object spots 50. In an embodiment where an object spot mask is additionally formed, the resulting object spot mask can be similar to the object spots 50, including, but not limited to, eyebrows, eyes, nose, and mouth. The detection of the object brightness transition 55 may be performed in operation S130.1:

[0088] S=|(D x *F avg *Y′ I ) 2 +(D y , *F avg*Y′ I ) 2 -(D x *Y′ I ) 2 +(D y *Y′ I ) 2 |

[0089] in:

[0090] S is the object brightness transition map representing the unique characteristics of the object;

[0091] Y1' is the object cropped image R 1 '5's Y channel;

[0092] F avg Is the size of The average box filter, w f It is the width of the face;

[0093] D x =(D y ) T is the difference operator D x ={d ij},,and

[0094] The size is w f .

[0095] Then, the value S can saturate to And normalized to the interval [0; 1].

[0096] In operation S130.2, based on the detected object features 45, a lambda map 60 may be generated, which represents at least one region to be subjected to maximum smoothing, at least one region to be subjected to minimum smoothing, and zero or more regions to be subjected to smoothing ranging from maximum smoothing to minimum smoothing. The lambda map 60 is generated in operation S130.2:

[0097]

[0098] Fig.13 An example of detected object features such as object blobs 50 and object brightness transitions 55 and a lambda map 60 for an object cropped image 5 is shown in accordance with an embodiment of the present disclosure.

[0099] refer to Fig.13, in operation S130.3, FGS considering object features 45 may be performed by applying FGS filter 65 with lambda map 60 to object cropped image 5 to extract actual object illumination 70 on the image. Actual object illumination 70 may be represented as a separate actual object illumination image I. In operation S130.3, object feature-guided FGS may be performed by solving an optimization problem using a method having, but not limited to, an FGS filter (or the like) to obtain actual object illumination 70:

[0100] ∑ p ((I p -(R′ I ) p ) 2 +λ p ∑ q∈N(p) w p，q (R′ I )(I p -I q ) 2 )→min, Λ={λ p}.

[0101] For example, the implementation of this operation is described in the document "Fast global imagesmoothing based on weighted least squares" by Min et al., IEEE Transactions on Image Processing 23.12 (2014): 5638-5653 [1].

[0102] Fig.14 An example of the result of FGS considering object features according to an embodiment of the present disclosure is shown.

[0103] Figure 5 is a flow chart of replacing actual object lighting of an image with adaptive object lighting according to an embodiment of the present disclosure.

[0104] refer to Figure 5, showing the process of operation S135. In operation S135, the actual object illumination 70 on the image 5 can be replaced by the adaptive object illumination defined by the adaptive object illumination 40. The goal of operation S135 is to brighten the object on the image and use the object surface smoothing mask and consider the probability of landmark detection failure to patch the brightened object into the current image (frame) in the image sequence (video). This operation allows the brightened object image to be stably patched into the input video frame based on the generated object surface mask, the stable FGS filter and the error detector. Operation S135 realizes continuous real-time video processing without obvious flicker and artifacts related to the critical rotation angle of the object face surface on the image. In addition, due to the lightweight patching algorithm, operation S135 improves the performance of the lighting manipulation method, which allows the method to be used even in mobile devices with limited real-time video processing capabilities.

[0105] In operation S135.1, the generated object surface mask A defined by the object surface area As may be received / input. r In operation S135.2, a stable FGS filter may be applied to the object surface mask A by a regularized solution according to the following optimization (inverse) problem: r To perform stable FGS:

[0106]

[0107] To find A s , whose solution can be found in close matrix form (see Min et al.):

[0108]

[0109] Where A is the Laplacian matrix,

[0110] More specifically, a regularized solution to the optimization problem may include: adding a regularizer to the object surface mask can be equal to 0.8, but is not limited to this particular value; a stable FGS filter is applied by solving problem (1) to obtain A int ; Update difference map A Δ =αA Δ +(1-α)(A int -A prev ); Update the object surface mask A prev ←A int ; Mask A int Normalize to get a smooth mask A s =A int / max(A int ).

[0111] In operation S135.3, the probability of landmark detection failure can be estimated using features of an object represented in a currently processed image in the image sequence, the estimation comprising examining the detected object features by a support vector machine pre-trained on a sample comprising features of an arbitrary image and corresponding landmark detection failure probabilities. An operation of training a support vector machine (SVM) on a sample comprising features of an arbitrary image and corresponding landmark detection failure probabilities can be performed in advance. SVM training can be performed by any method. The object landmark detection failure probability pf can be estimated by applying the pre-trained SVM to some features X of the current image.

[0112] In operation S135.4, the adaptive object illumination defined by the adaptive object illumination may be patched into the current image in the image sequence by applying an object surface smoothing mask and taking into account the object landmark detection failure probability pf. In one embodiment, the patching may be performed as follows:

[0113]

[0114] in:

[0115] R 0 is to patch the adaptive object lighting image to the input image R 1 The results in;

[0116] A s is the surface area of ​​the object;

[0117] S is the object brightness transition map representing the unique characteristics of the object;

[0118] M is adaptive object lighting 40 with smooth object spots; and

[0119] I is the actual object illumination image.

[0120] Inpainting is performed to remove harsh shadows from objects while preserving their unique features. ) eliminates harsh shadows while retaining the unique features of the object by applying the object brightness transition map that must be saved in this operation. In operation S140, an image in which the actual object lighting in the image sequence has been replaced with the adaptive object lighting can be displayed.

[0121] Figure 7 A schematic diagram of a user computing device implementing a method for performing object lighting manipulation on an image according to an embodiment of the present disclosure is shown.

[0122] refer to Figure 7The user computing device 75 (e.g., a mobile device) in which the disclosed method may be implemented may include, but is not limited to, a processor 75.1, a camera 75.2, and a memory 75.3 storing processor-executable instructions that, when executed by the processor 75.1, cause the processor 75.1 to perform any of the operations of the disclosed object lighting manipulation method. The user computing device 75 may include Figure 7 Other components not shown in the figure, and are, for example, smart phones, tablet computers, virtual reality glasses, augmented reality goggles, PCs, laptop computers, smart watches, etc. In an alternative embodiment of the computing device / mobile device 75, the device may include a separated hardware unit. In this embodiment, each hardware unit may be responsible for executing the corresponding operation or sub-operation of the method, and is appropriately named, for example, the device hardware unit responsible for image input (see operation S100) may be referred to as an image input interface, etc. However, since those skilled in the art will envision other possible unit structures in the device, the present disclosure should not be limited to this unit structure, for example, a hardware unit may be responsible for several operations or sub-operations of the method, or several operations or sub-operations may be shared between two or more units. In addition, the disclosed method may be implemented by a processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or in some embodiments by a system on chip (SoC). In addition, the disclosed method may be implemented by a computer-readable medium storing computer executable instructions, so that the computer executes the method when the instruction is executed by a computer processor.

[0123] Fig.15 is a flowchart of performing a lighting manipulation operation according to an embodiment of the present disclosure.

[0124] refer to Fig.15 , the lighting control device 1600 can be based on Fig.15 The sequence shown performs object lighting manipulation on the first image.

[0125] In operation S1510, the lighting manipulation device 1600 may receive an input of a first image and a predetermined light direction. The predetermined light direction may be a default light direction or a user-defined light direction. The user may perform object lighting manipulation on the first image based on the light direction applied to the predetermined light direction.

[0126] For example, when capturing a person with respect to light, the user may determine the light direction as the front or side of the subject, and may adjust the lighting on the image to generate an image captured by receiving light along the determined light direction.

[0127] In operation S1520, the lighting manipulation apparatus 1600 may generate a second image having an object detected from the first image. The lighting manipulation apparatus 1600 may detect an object landmark in the first image, and may obtain the second image by cropping everything except the detected landmark from the first image. The object may be a main portion of an object focused on by a photographer who captured the first image.

[0128] For example, in an image of an upper body of a person, the object may be the person and the object landmark may be the person's face.

[0129] In this case, the lighting manipulation apparatus 1600 may obtain a second image in which only the object is enlarged.

[0130] In operation S1530, the lighting manipulation apparatus 1600 may adapt the object lighting of the second image to the predetermined light direction. The lighting manipulation apparatus 1600 may generate general object lighting by using the second image and the predetermined light direction.

[0131] Generic object lighting may represent lighting when a generic 3D mesh is fitted to object landmarks and then captured by receiving light along a determined light direction. Generic object lighting may be generated by applying a generic object texture, and therefore, its color or shape may be different from that of an object on an actual image.

[0132] The lighting manipulation device 1600 may generate adaptive object lighting based on the generic object lighting and the object features obtained from the second image.

[0133] Adaptive object lighting may be generated by stretching the boundary of the generic object lighting to the boundary of the second image, adjusting the stretched generic object lighting using the brightness and color of the second image to reflect the brightness and color of the actual image, and performing a speckle smoothing operation on the adjusted generic object lighting.

[0134] In operation S1540, the lighting manipulation apparatus 1600 may extract actual object lighting from the second image.

[0135] The lighting manipulation apparatus 1600 may replace the object lighting of the first image with the adaptive object lighting of the second image by the following operations.

[0136] First, the lighting manipulation apparatus 1600 may detect spots detected from the object surface mask based on object landmarks and shadows detected from the second image as object features.

[0137] Here, the object surface mask may be generated by estimating color factors from the second image to the object mask generated based on the object landmarks, and finding the object surface area using the estimated color factors.

[0138] Secondly, the lighting manipulation device 1600 can generate a lambda map based on the detected object features, the lambda map indicating at least one area to be subjected to maximum smoothing and at least one area to be subjected to minimum smoothing and zero or more areas to be smoothed in the range from maximum smoothing to minimum smoothing.

[0139] Next, the lighting manipulation apparatus 1600 may perform feature-guided FGS by applying the FGS filter with the λ map to the second image, thereby extracting the actual object lighting.

[0140] By using feature-guided FGS, the lighting manipulation apparatus 1600 can maintain object features while removing harsh shadows.

[0141] The lighting manipulation apparatus 1600 may replace the actual object lighting on the second image with the adaptive object lighting.

[0142] The lighting manipulation apparatus 1600 may perform stable FGS by applying a stable FGS filter to the object surface mask, thereby obtaining an object surface smooth mask.

[0143] The lighting manipulation device 1600 may restore the replaced second image to the original image sequence.

[0144] The lighting manipulation device 1600 can perform seamless video processing without noticeable flickering or extreme face angle artifacts by restoring the replaced second image to the original image sequence using the generated object surface smoothing mask. The lighting manipulation device 1600 can estimate the probability of object detection failure using the features of the object relative to the first image, and can restore adaptive object lighting to the first image using the object surface smoothing mask and the probability of object detection failure.

[0145] Additionally, the lighting manipulation device 1600 may perform lightweight patching with improved performance.

[0146] Fig.16 is a structural diagram of a lighting manipulation device for performing lighting manipulation according to an embodiment of the present disclosure.

[0147] refer to Fig.16 , the lighting manipulation device 1600 may include an input interface 1610 and a processor 1630 .

[0148] The input interface 1610 may receive an input of a first image in a sequence of images and an input of a predetermined light direction. The input may be automatically performed by the system or manually performed by a user.

[0149] The processor 1630 may execute Fig.15 Operations S1510 to S1540.

[0150] Fig.17 An example of a process of manipulating lighting when the object of an image is a face according to an embodiment of the present disclosure is shown.

[0151] refer to Fig.17 , an example of a first image 1710 . A plurality of object landmarks 1715 may be detected from the first image 1710 .

[0152] Lighting manipulation apparatus 1600 may generate second image 1720 by cropping the image to object landmark 1715 .

[0153] The object surface mask 1730 may be generated by estimating color factors from the second image 1720 and finding the object surface area using the estimated color factors.

[0154] Generic object lighting 1740 may be generated by fitting a generic 3D mesh to object landmarks 1715 and performing capture via receiving light along determined light directions.

[0155] The stretched illumination 1770 is generated by stretching the boundaries of the generic object illumination 1740 to the boundaries of the second image 1720 .

[0156] Based on the object landmarks 1715 and the shadows detected from the second image 1720 , object features 1750 , including spots, are detected from the object surface mask 1730 .

[0157] By adjusting the generic object illumination 1770 with the brightness and color of the second image 1720 and performing a speckle smoothing operation on the adjusted object illumination, an adaptive object 1780 is generated by reflecting the brightness and color of the actual image.

[0158] Adjustment of common object illumination 1770 of brightness and color with second image 1720 including stretched boundaries may include calculating Y, U, and V histograms of the common object illumination, calculating Y, U, and V histograms of second image 1720, equalizing the U and V histograms of the common object illumination with the calculated U and V histograms of second image 1720, adjusting the common object illumination using the second image and the Y histogram of the common object illumination, and applying the common object illumination and the equalized U and V histograms of the second image to the common object.

[0159] An object surface smoothing mask 1760 is generated by applying a stable FGS filter to the object surface mask 1730 .

[0160] The image sequence 1790 is restored by replacing the actual illumination with respect to the second image with the adaptive illumination 1780 and using the replaced image, the object features 1750 and the object surface smoothing mask 1760 .

[0161] As described above, according to one or more of the above embodiments, the method can use object feature-guided FGS to retain unique object features on the image and remove harsh shadows. In addition, due to the use of adaptive object lighting, the disclosed method can handle the colored lighting of the object and retain the unique color of the object. In addition, due to the use of a stable FGS filter, the disclosed method is suitable for real-time continuous video processing. Therefore, the disclosure improves the overall efficiency and quality of real-time processing of images in an image sequence, and the processing itself is feasible on a user computing device or mobile device.

[0162] The method according to the embodiment of the present disclosure can be implemented as a computer instruction that can be executed by various computer devices and recorded on a computer-readable recording medium. The computer-readable recording medium may include a program command, a data file, a data structure, or a combination thereof. The program command recorded on the computer-readable recording medium may be specially designed and constructed for the present disclosure, or may be known and available to a person of ordinary skill in the field of computer software. Examples of computer-readable media include storage media, such as magnetic media (e.g., hard disks, floppy disks, or tapes), optical media (e.g., compact disks-read-only memories (CD-ROMs) or digital versatile disks (DVDs)), magneto-optical media (e.g., optical floppy disks), and hardware devices (e.g., ROMs, RAMs, or flash memory) that are specially configured to store and execute program commands. Examples of program commands include high-level language codes that can be executed by a computer using an interpreter and machine language codes generated by a compiler.

[0163] It should be understood that the embodiments described herein should be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects in each embodiment should typically be considered as available for other similar features or aspects in other embodiments.

[0164] While the present disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. A method for performing an object lighting manipulation operation on a first image, the method comprising: receiving an input of the first image and a predetermined light direction, the predetermined light direction comprising one of a default light direction or a user-defined light direction; generating a second image having the first object detected from the first image; fitting a generic three-dimensional (3D) mesh to a second object in the second image, and applying illumination defined by the predetermined light direction to the generic 3D mesh fitted to the second object to determine second object illumination for the second image; generating a third object illumination based on the second object illumination and an object feature obtained from the second image; as well as The first object illumination of the first image is replaced with the third object illumination.

2. The method according to claim 1, wherein the method further comprises: adjusting the size of the second object illumination to correspond to the size of the second image; After adjusting the size of the second object lighting, adjusting the second object lighting based on the brightness and color of the second image; as well as After adjusting the second object illumination, the second object illumination is smoothed.

3. The method of claim 2, wherein adjusting the second object illumination comprises: Calculating a Y histogram, a U histogram, and a V histogram of the illumination of the second object; Calculating a Y histogram, a U histogram, and a V histogram of the second image; equalizing the U histogram and the V histogram of the second object illumination with the U histogram and the V histogram of the second image; adjusting the second object illumination using the second image and a Y histogram of the second object illumination; as well as The second object illumination and the equalized U histogram and V histogram of the second image are applied to the second object illumination.

4. The method of claim 1 , wherein replacing the first object lighting comprises: generating a lambda map based on the object features, the lambda map representing at least one region corresponding to maximum smoothing, at least one region corresponding to minimum smoothing, and zero or more regions corresponding to smoothing ranging from the maximum smoothing to the minimum smoothing; as well as The actual object illumination is extracted using the lambda map.

5. The method of claim 4, wherein extracting the actual object lighting comprises: A feature-guided fast global smoothing (FGS) filter with the lambda map is applied to the second image to generate the actual object illumination.

6. The method according to claim 4, further comprising: generating an object mask based on the first object; estimating color factors based on the object mask and the second image; as well as An object surface area is found based on the color factor.

7. The method of claim 6, wherein replacing the first object lighting comprises: A stable fast global smoothing (FGS) is performed on the object surface mask to obtain an object surface smoothed mask.

8. The method of claim 7, wherein replacing the first object lighting further comprises: estimating a probability of object detection failure based on features of the first object in the first image; as well as The first image is restored to the first object illumination based on the object surface smoothness mask and the probability of object detection failure.

9. The method according to claim 4, wherein the first object is a face, wherein the surface of the first object is facial skin, and The spots in the object features include at least one of eyebrows, eyes, nose or mouth.

10. An apparatus for performing an object lighting manipulation operation on a first image, the apparatus comprising: an input interface configured to receive an input of the first image and a predetermined light direction, wherein the predetermined light direction comprises one of a default light direction or a user-defined light direction; as well as The processor is configured to: generating a second image having the first object detected from the first image; fitting a generic three-dimensional 3D mesh to the second object in the second image, and applying illumination defined by the predetermined light direction to the generic 3D mesh fitted to the second object to determine a second object illumination for the second image; generating a third object illumination based on the second object illumination and an object feature obtained from the second image; as well as The first object illumination of the first image is replaced with the third object illumination.

11. The device according to claim 10, wherein: The processor is further configured to: adjusting the size of the second object illumination to correspond to the size of the second image, After adjusting the size of the second object lighting, adjusting the second object lighting based on the brightness and color of the second image, and After adjusting the second object illumination, the second object illumination is smoothed.

12. The device according to claim 11, wherein When the processor adjusts the second object illumination, the processor is further configured to: Calculating a Y histogram, a U histogram, and a V histogram of the illumination of the second object, Calculating a Y histogram, a U histogram and a V histogram of the second image, equalizing the U histogram and the V histogram of the second object illumination with the U histogram and the V histogram of the second image, adjusting the second object illumination using the second image and a Y histogram of the second object illumination, and The second object illumination and the equalized U histogram and V histogram of the second image are applied to the second object illumination.

13. The apparatus according to claim 10, wherein: The processor is further configured to: generating a lambda map based on the object features, the lambda map representing at least one region corresponding to maximum smoothing, at least one region corresponding to minimum smoothing, and zero or more regions corresponding to smoothing ranging from the maximum smoothing to the minimum smoothing; as well as The actual object illumination is extracted using the lambda map.

14. The device according to claim 13, wherein: The processor is further configured to: A feature-guided fast global smoothing (FGS) filter with the lambda map is applied to the second image to generate the actual object illumination.

15. The apparatus according to claim 13, wherein: The processor is further configured to: generating an object mask based on the first object; estimating color factors based on the object mask and the second image; and An object surface area is found based on the color factor.

16. The device according to claim 15, wherein: The processor is further configured to: A stable fast global smoothing (FGS) is performed on the object surface mask, thereby obtaining an object surface smoothing mask to replace the first object illumination.

17. The device according to claim 16, wherein: The processor is further configured to: estimating a probability of object detection failure based on features of the first object in the first image; and The first image is restored to the first object illumination based on the object surface smoothness mask and the probability of object detection failure.

18. The device according to claim 13, wherein the first object is a face, wherein the surface of the first object is facial skin, and The spots in the object features include at least one of eyebrows, eyes, nose or mouth.

19. A computer-readable recording medium having recorded thereon a program for executing the method according to claim 1 on a computer.

Images