Method for correcting facial deformation in facial depth images, imaging apparatus, and storage medium

Abstract

The present invention discloses a method for correcting facial deformation of a facial depth image, an imaging device, and a storage medium. The method for correcting facial deformation includes the steps of creating a generic facial deformation map including generic facial feature data, acquiring a user's facial image, and applying the generic facial deformation map to the user's facial image. The step of applying the generic facial deformation map to the user's facial image includes the steps of transforming the generic facial deformation map using the generic facial feature data and the user's facial feature data, and correcting depth information in the user's facial image using the transformed generic facial deformation map, thereby correcting the facial deformation of the user's facial image. According to the present invention, a facial depth image with higher accuracy can be obtained.

Description

[Technical Field]

[0001] [Cross-reference of related applications] This application claims priority to Chinese patent application No. 202110602233.9, filed with the State Intellectual Property Administration of the People's Republic of China on 31 May 2021, the entire contents of which are incorporated herein by reference.

[0002] This invention relates to the correction of facial deformation in facial images in computer vision. Specifically, this invention relates to a method for correcting facial deformation in facial images, an imaging apparatus, and a storage medium that can improve the depth accuracy of facial images. [Background technology]

[0003] Drawing technology is widely applied to many industry fields, including the internet, and is primarily used to uncover user characteristics, understand user preferences, and gain a deep and thorough understanding of the object being drawn. Similarly, large machinery and equipment can be drawn like people, and drawing technology can be used to complete feature extraction and comprehensive descriptive drawing. During the shipping, production, and maintenance processes of large machinery and equipment, a large amount of supervision and monitoring data is generated. While this data can directly reflect the distinctive features of the equipment, a large amount of data alone cannot intuitively uncover those features in depth. Therefore, by constructing drawings of the machinery and equipment, it is possible to accurately uncover and extract the features of each aspect of the machinery and equipment and present them comprehensively, making it easier to understand and grasp the equipment status and operating conditions in a timely manner.

[0004] Computer vision technology, as the study of how to enable machines to "see" and understand the world, is currently one of the most popular research areas in the field of deep learning. Specifically, by combining a camera and a computing unit, machine vision systems can perform tasks such as recognition, tracking, and measurement of targets in a given scene, in place of the human eye. Face recognition (face ID) is one of the most important branches of computer vision technology, and its relationship to computer vision is also the most closely related.

[0005] In recent years, with the spread of smartphones and mobile payments, facial recognition technology has been attracting increasing attention. In fact, the earliest research into facial recognition technology dates back to the 1950s, when scientists were already studying methods for extracting facial contours. However, this research was limited to a technical level and temporarily stagnated. In the 1980s, however, new advancements were made in facial recognition methods, and related knowledge from neurophysiology, neuroscience, and vision was introduced, leading to a new stage of development in facial recognition.

[0006] Currently, facial recognition can be used in many applications related to people's lives, such as unlocking mobile phones, opening automatic doors, personnel registration, employee attendance records, and facial recognition payments. For example, we are accustomed to using facial recognition technology every day to unlock our mobile phones or make payments, and it feels very natural, but behind this technology is a series of complex technical processing flows. Simply put, first, a device such as a camera head is used to replace the human eye and acquire an image, a computer is used to process the image information, and human cognitive modes are combined to detect, extract, and recognize facial features in the image.

[0007] To improve the accuracy of face detection and recognition, techniques have been proposed that utilize face depth images and point cloud information for face recognition. Among these, an indirect time-of-flight (iToF) system can be used to acquire face depth information. iToF depth measurement methods are widely applied in this field as indirect measurement means that measure the difference in distance or depth by measuring the phase shift, which is the phase difference between the transmitted wave and the reflected wave. These methods include the four-stage phase method, which calculates depth information using four phase control signals. [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] However, when using iToF to acquire depth information, the accuracy of the acquired depth data is affected by various factors. For example, due to the complex three-dimensional geometry of the face, the same emitted beam may reach the receiving sensor via multiple different paths, potentially causing signal attenuation and phase shift. Such multipath interference (MPI) results in poor accuracy of the acquired depth image, producing defects such as virtual images. Furthermore, human skin is not actually an ideal light-reflecting surface and is a surface that is highly likely to cause sub-surface scattering (SubSS). Therefore, sub-surface scattering due to the material properties of human skin also negatively affects the accuracy of the depth data. The existence of the above problems results in deformation of the acquired facial image.

[0009] In view of the above problems, the present invention provides a method, imaging apparatus, and storage medium capable of correcting facial deformation in facial images acquired by the iToF method, thereby enabling the acquisition of more accurate depth-of-sight images of the face. [Means for solving the problem]

[0010] A method for correcting facial deformation provided according to an embodiment of the present invention includes the steps of: creating a generic facial deformation map including generic facial feature data; acquiring a user's face image; and applying the generic facial deformation map to the user's face image. The step of applying the generic facial deformation map to the user's face image includes the steps of: deforming the generic facial deformation map using the generic facial feature data and the user's face feature data; and correcting the depth information in the user's face image using the deformed generic facial deformation map, thereby correcting the facial deformation in the user's face image. The generic facial deformation map may include data on the difference in depth information between sample data of random face samples and GT data.

[0011] In the method according to an embodiment of the present invention, the deformation map of the general-purpose face is created by acquiring iToF data of a face sample, acquiring GT data of the face sample, aligning the GT data and the iToF data to calculate a first depth difference image showing the difference in depth information between the GT data and the iToF data, detecting and extracting the feature data of the sample face of the face sample, matching the feature data of the sample face with the first depth difference image to obtain a second depth difference image having the feature data of the sample face as a marker, acquiring the feature data of the general-purpose face, and using the feature data of the sample face marked on the second depth difference image based on the feature data of the general-purpose face to deform the second depth difference image, thereby obtaining a third depth difference image that matches the feature data of the general-purpose face and has the feature data of the general-purpose face marked, as the deformation map of the general-purpose face.

[0012] In a method according to an embodiment of the present invention, the step of calculating the first depth difference image may include the steps of: converting the point cloud of the iToF data into a grid; aligning the GT data and the grid; projecting the aligned GT data and the grid to obtain a projected image of the GT data and a projected image of the iToF data; and calculating the first depth difference image based on the projected image of the GT data and the projected image of the iToF data.

[0013] In the method according to an embodiment of the present invention, the step of calculating the first depth difference image may include the steps of: using camera intrinsic parameters to directly project the GT data and obtain a projected image of the GT data; projecting the point cloud of the iToF data to obtain a projected image of the iToF data; and calculating the first depth difference image based on the projected image of the iToF data and the projected image of the GT data.

[0014] In a method according to an embodiment of the present invention, the step of applying the generic face deformation map may include the steps of: detecting and extracting user face feature data from the user's face image; deforming the generic face deformation map so as to match the generic face feature data marked on the generic face deformation map with the detected user face feature data, thereby obtaining a fourth depth difference image; and correcting the depth information in the user's face image based on the fourth depth difference image.

[0015] In the method according to an embodiment of the present invention, the step of creating the deformation map of the general face may further include: obtaining the shooting parameters of the iToF data of the face sample; repeating the step of changing the values of the shooting parameters to generate the third depth difference image; and generating a plurality of the third depth difference images that constitute the deformation map of the general face based on the values of a plurality of sets of the shooting parameters. Moreover, the step of applying the deformation map of the general face includes: obtaining the shooting parameters of the face image of the user, interpolating the shooting parameters to the deformation map of the general face to obtain a fifth depth difference image in which the feature data of the general face corresponding to the shooting parameters of the face image of the user is marked; detecting and extracting the feature data of the user's face from the face image of the user; obtaining a sixth depth difference image by deforming the fifth depth difference image so as to match the feature data of the general face marked on the fifth depth difference image and the detected feature data of the user's face; and further including the step of correcting the depth information in the face image of the user based on the sixth depth difference image.

[0016] In the method according to an embodiment of the present invention, the shooting parameters may include the angle between the angular direction of the face and the camera, and the distance between the camera and the face.

[0017] In the method according to an embodiment of the present invention, the angle includes a yaw angle and a pitch angle.

[0018] According to another embodiment of the present invention, the imaging device provided includes an imaging unit configured to acquire a face image of a user, a storage unit that stores a deformation map of a general face including feature data of the general face, and a detection unit that detects and extracts feature data of the user's face from the face image of the user, and uses the feature data of the general face and the feature data of the user's face to deform the deformation map of the general face, and uses the deformed deformation map of the general face to correct depth information in the face image of the user, thereby correcting face deformation in the face image of the user. And a processing unit configured to perform the operation.

[0019] According to still another embodiment of the present invention, a storage medium provided stores a computer-readable program that implements the above method when executed on a processor.

Advantages of the Invention

[0020] According to the present invention, in the method according to the present invention, a depth deformation map associated with face feature information is created to correct a depth error that occurs when an iToF device captures a known scene (here, a face). Such a depth deformation map is created from scenes recorded in different key structures (eg, [yaw angle, pitch angle, distance]) and then converted to suit a general face model. Also, by calculating the necessary deformations based on deformation interpolation in the key structure, the depth deformation map is applied to any other structure within a predetermined range, thereby fitting the general face model to a specific model and realizing correction for depth deformation of any specific face image. According to the present invention, a face depth image with higher accuracy can be obtained.

[0021] It should be understood that the beneficial effects of the present invention are not limited to the above effects, and may be any beneficial effects described in this specification.

Brief Description of the Drawings

[0022] The drawings of the specification, which constitute part of this application, are used to provide a further understanding of this specification, and the schematic embodiments and descriptions of the invention are used to interpret the invention and do not constitute an unreasonable limitation of this specification.

[0023] [Figure 1] Figure 1 is a schematic diagram illustrating an example of selecting facial feature points. [Figure 2] Figure 2 is a schematic diagram illustrating an example of the relative angle between the iToF camera and the subject's face. [Figure 3] Figure 3 is a flowchart illustrating the main steps of the face correction method according to the present invention. [Figure 4] Figure 4 is a schematic diagram illustrating the step-by-step processing and corresponding images for creating a general-purpose face deformation map in the face correction method according to the present invention. [Figure 5] Figure 5 is a schematic diagram illustrating the step-by-step processing and corresponding images for obtaining a first depth difference image by point cloud alignment according to the present invention. [Figure 6] Figure 6 is a schematic block diagram illustrating the features of a generic face obtained from a generic face. [Figure 7] Figure 7 is a schematic diagram illustrating an example of achieving morphining of depth difference images using facial feature detection and Delaunay triangulation according to the present invention. [Figure 8] Figure 8 illustrates an example of a shooting function according to an embodiment of the present invention. [Figure 9] Figure 9 illustrates an example of different distance imaging functions according to the present invention. [Figure 10] Figure 10 illustrates an example of different angle shooting functions at the same distance according to the present invention. [Figure 11] Figure 11 is a schematic diagram illustrating an example of a second depth difference image in a keypoint structure according to the present invention. [Figure 12] Figure 12 is a schematic diagram illustrating an example of a third depth difference image in the keypoint structure according to the present invention. [Figure 13] Figure 13 is a schematic diagram illustrating the interpolation calculation process performed based on a third depth difference image in the keypoint structure according to the present invention. [Figure 14] Figure 14 is a schematic diagram illustrating the simulation results of interpolation calculations performed based on a third depth difference image of keypoint structures at the same distance according to the present invention. [Figure 15] Figure 15 is a schematic diagram illustrating the interpolation calculation process performed based on a third depth difference image in the keypoint structure according to the present invention. [Figure 16] Figure 16 is a schematic diagram illustrating a modified example of the method for obtaining a first depth difference image according to the present invention. [Figure 17] Figure 17 is a flowchart illustrating an example of a process for correcting depth information by applying a general-purpose face deformation map according to the present invention. [Figure 18] Figure 18 illustrates an example of obtaining a specific fifth depth difference image DDI5 from a general-purpose facial deformation map GFDM based on specific imaging parameters. [Figure 19] Figure 19 is a block diagram illustrating an example of the structure of an imaging device according to an embodiment of the present invention. [Figure 20] Figure 20 is a block diagram illustrating an example of the structure of an electronic device according to an embodiment of the present invention. [Modes for carrying out the invention]

[0024] Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings. Please note that all dimensions, shapes, positional relationships, etc., in the drawings are merely approximate and are not necessarily shown in true proportion for ease of illustration, and therefore are not limiting. Furthermore, in the embodiments described below, the same parts, arrangements, steps, etc., are indicated by the same reference numerals, and repeated explanations of them will be omitted.

[0025] The present invention will be described in the following order. 1. Overview of the facial correction method of the present invention 2. Detailed explanation of the process for creating a generic face deformation map. 2.1. Explanation of each step in creating a generic face deformation map 2.2. Exemplary Explanation of Point Cloud Alignment 2.3. Exemplary Description of Facial Feature Extraction in Steps S14 and S16 2.4. Exemplary explanation of morphing in depth difference images 2.5. Exemplary Explanation of Interpolation Calculation Based on Imaging Parameters 2.6. Modified Method for Obtaining the First Depth Difference Image 3. Detailed explanation of the process of correcting depth information by applying a generic face deformation map. 4. Imaging apparatus according to the present invention 5. Electronic device according to the present invention

[0026] [1. Overview of the facial correction method according to the present invention] In the application of iToF technology for face recognition, the basic logical flow of currently existing mainstream methods is to identify similar or identical faces. In both cases, facial features are extracted from the acquired image, converted into an appropriate subspace, and then similarity is judged or classified within this subspace for learning, thereby performing face recognition.

[0027] In the process of extracting facial features from acquired images, many factors (including those mentioned above) can cause distortion in the facial image. For example, even with the same person, if their posture or facial orientation is different, humans may still be able to recognize them, but a computer may not. Methods to address this problem of facial distortion are known, and typically involve first detecting the feature points of the main facial features, and then performing alignment correction on the detected face based on these feature points. For example, in a method invented in 2014 by Vahid Kazemi and Josephine Sullivan, they selected 68 feature points (landmarks) as shown in Figure 1 for important parts of the face. Since the positions of these 68 points are fixed, it is possible to find these 68 points on any face simply by first building a learning system and then training the system. With these 68 points, the face can be corrected by using an affine transform to correct the original distorted face, minimizing errors.

[0028] Therefore, we envision that after acquiring depth image information using an iToF system, and then constructing a corresponding transformation relationship through feature extraction, a learning system can be created based on this relationship, and further, facial deformation caused by factors such as MPI and SubSS in different facial images captured using this system can be corrected. In this specification, when acquiring images using an iToF camera, facial deformation of facial images due to the various factors mentioned above is considered to depend mainly on the following factors.

[0029] (a) Angle The angle referred to here is the relative angle between the camera and the subject's face. For example, it can be specified as the angle between the camera's optical axis and the subject's line of sight. For instance, as shown in Figure 2, there is a yaw angle (yaw) θ between the camera's optical axis and the subject's line of sight.

[0030] (b) distance Here, this is the distance from the camera lens to the subject's face. As shown in Figure 2, the distance between the camera's optical axis and the subject's face is d.

[0031] Based on the above analysis, the facial deformation correction method for facial depth images according to the present invention can be configured to be divided into roughly the following three steps, as shown in Figure 3.

[0032] In S1, a generic face deformation map (GFDM) is constructed. The generic face deformation map (GFDM) is a set of maps or a library of maps that represent the facial deformation state of a typical face image. It can also be expressed as a function relating to influencing factors (e.g., angle and distance), and is therefore also called a generic face deformation map function. This step is similar to the "learning" step described above, which involves building and training the learning system. Therefore, this step only needs to be performed once, and once completed, a "correction system" is obtained that can later be used to correct facial deformation on specific user face images.

[0033] In S2, the user's face, which is the subject of the imaging, is photographed, and image data of the user's face is acquired. In addition to acquiring iToF data of the user's face, the imaging unit of the iToF system can also obtain other image data, such as IR image data and RGB image data included in the image data of the user's face, by adding infrared pixels and normal imaging pixels.

[0034] In S3, the generic face deformation map GFDM is applied to the user's face image to correct the facial deformation of the user's face image.

[0035] The following describes in detail each of the three steps of the method for correcting facial deformation in facial depth images according to the present invention.

[0036] [2. Detailed explanation of the steps for creating a generic face deformation map] (2.1. Explanation of each step in creating a generic face deformation map) Referring to Figure 4, the above-mentioned general-purpose face deformation map can be created, for example, by the following method.

[0037] First, in step S11, image data of a random face sample is acquired as sample data. The random face sample here may be a face sample of any randomly selected user or test subject, or it may be a face model of an advanced simulation. Such image data is collected on the face sample by an iToF system, and therein it may include at least iToF data and further include other types of image data such as IR image data and RGB image data. The iToF data includes depth information of the face sample. Here, the face sample can be collected using any known iToF system as needed. In the process of collecting the sample data, sample data of the entire head may be collected, or only sample data of the region of interest (i.e., face) may be collected. In the former case, the step of acquiring the sample data should further include the step of cutting out and extracting data of the face region of interest from the collected iToF data.

[0038] Next, in step S12, ground truth (GT) data is acquired to serve as a comparison standard in subsequent steps. The GT data here is obtained based on an accurate 3D face model of the same random face sample used in the previous step. For example, such accurate GT data of a face model can be obtained using 3D scanning technology or multi-angle imaging technology. GT data based on such an accurate face model possesses the accurate geometric and texture features of the random face sample. Therefore, in subsequent processing, a general-purpose face deformation map constructed based on such GT data can effectively eliminate or mitigate adverse effects on the face image caused by factors such as MPI and SubSS.

[0039] Next, in step S13, the sample data obtained in step S11 and the GT data obtained in step S12 are aligned, and a first depth difference image (DDI1) showing the difference between the GT data and the depth information of the face-focused region is calculated. Specifically, the above alignment can be achieved using point cloud alignment techniques known within this field.

[0040] For example, as shown in Figure 5, the point cloud of the obtained sample data of the region of interest may first be converted into a grid, and then the GT data and the grid may be aligned. Next, the aligned GT data and grid are projected to obtain projected images of the GT data and the sample data of the region of interest. Finally, a first depth difference image DDI1 is calculated based on the difference between the projected GT image and the projected image of the sample data of the region of interest. In this case, the output resolution of the first depth difference image DDI1 can be arbitrarily selected.

[0041] The following provides a more detailed explanation of an example of aligning sample data with GT data point clouds.

[0042] Next, in step S14, the feature SFL of the sample face of a random face sample is detected and extracted from the image data of the random face sample. In addition to the iToF data mentioned above, the image data of the random face sample further includes at least one of the following: IR image, RGB image, or laser image of the random face sample, which is used to extract the feature SFL of the sample face. For example, by adding infrared pixels and normal imaging pixels to the imaging unit of the iToF system, the image data of the random face sample can include iToF data, IR image data, and RGB image data. It should also be understood that since the iToF data of the random face sample and the above data used in this step both originate from the same random face sample image data, the processing described in this step can be completed all at once in step S11.

[0043] Using any facial feature detection technique applicable in this field, sample face landmarks (SFLs) of randomly acquired facial samples can be detected and extracted from the image data of those random facial samples. The extracted facial features may be, for example, a set of 68 feature points of a face, as shown in Figure 1. An example of the facial feature SFL extraction process related to this step will be described below.

[0044] Next, in step S15, the sample face features SFL are matched with the first depth difference image DDI1 to obtain a second depth difference image DDI2. Such matching can be understood as adding the detected and extracted sample face features SFL to the corresponding positions of the first depth difference image DDI1, which was obtained based on the same random face sample. In other words, the obtained second depth difference image DDI2 contains the sample face features SFL, which act as markers. Subsequently, if necessary, graphic filtering can be applied to the second depth difference image DDI2 using various filters to remove foreign objects from the map.

[0045] Subsequently, in step S16, generic face landmark (GFL) data is obtained. Generic face landmark (GFL) data is extracted based on a generic face image. As shown in Figure 6, for example, using the method described above, data of 68 feature points representing generic face features, as shown in Figure 1, can be extracted based on a generic face image (in this case, an IR image). The so-called "generic face image" here may be a face image that has common facial features of a specific group of people, defined based on factors such as a specific country, region, ethnicity, or gender. In addition to IR images, other appropriate image types such as RGB images or laser images may also be used. Thus, the generic face landmark (GFL) data corresponds to a generalized face feature structure or grain, which is used to modify the coordinate system of the face features in subsequent steps, so that the modified coordinate system can be applied to all face deformation corrections.

[0046] The process of extracting general-purpose facial features from general-purpose facial images is the same as the facial feature extraction process in step S14 described above, and any facial feature detection and extraction method known in the art can be used as needed. Examples of such facial feature detection and extraction processes can be found below. It is also possible to complete the acquisition of general-purpose facial feature data GFL in step S11 in one go and store it in a memory or other storage device so that it can be recalled.

[0047] Finally, in step S17, based on the general-purpose face feature data GFL, the second depth difference image DDI2 is morphed using the sample face feature data SFL marked on the second depth difference image DDI2 to obtain a third depth difference image DDI3 that matches the general-purpose face feature data GFL and is marked with the general-purpose face feature data GFL.

[0048] It should be understood that, in an ideal scenario, the resulting third depth difference image DDI3 is the general-purpose face deformation map GFDM. This ideal scenario involves, for example, the iToF system's camera directly facing the random face sample during the multiple steps described above, and maintaining a fixed distance between the camera and the random face sample. However, in most situations, the aforementioned shooting parameters (distance, angle) change. Therefore, to obtain the general-purpose face deformation map GFDM, the step of generating the third depth difference image DDI3 based on different shooting parameters must be repeated, thereby generating multiple third depth difference images DDI31, DDI32, DDI33, etc., corresponding to multiple sets of shooting parameters (distance, angle). Finally, the general-purpose face deformation map GFDM is obtained through interpolation calculations between these multiple third depth difference images DDI31, DDI32, DDI33, etc. In other words, the general-purpose face deformation map GFDM can be represented by interpolation calculation functions associated with the shooting parameters (distance, angle). The following describes in detail an example of interpolation calculation for obtaining a generic face deformation map (GFDM). Subsequently, if necessary, graphic filtering can be applied to the generic face deformation map using various filters to remove foreign objects from the generic face deformation map.

[0049] (2.2. Exemplary explanation of point cloud alignment in step S13) The point cloud alignment method described in step S13 above is known in this art. For example, by calling an algorithm in the Point Cloud Library (PCL) registered library, the correct point correspondence between multiple sets of point cloud datasets acquired based on different views is found, and they are aligned into a single point cloud model, thereby achieving point cloud alignment across different datasets. The algorithmic work in the PCL registered library is performed by finding the correct point correspondence between multiple given input datasets and estimating a stiffness transformation that can transform each individual dataset into a matching global coordinate frame through rotation and translation.

[0050] Typically, the output after pairing a pair of point cloud datasets is a single stiffness transformation matrix (4x4) that indicates the rotations and translations that must be applied to one of the datasets (which can be called the source set) and perfectly aligned with the other dataset (which can be called the target set or model). The steps for aligning point clouds in two datasets are roughly as follows. Note that any or all of the steps can be traversed as needed.

[0051] First, we determine the most representative point of interest (i.e., keypoint) in a scene (e.g., a face) from a set of points in two datasets. Here, a keypoint is a point of interest in the scene that has a "special attribute," such as a corner of a book or the letter "S" in "SONY" written on a book. There are many different keypoints available in PCL, such as NARF, SIFT, and FAST. Alternatively, each point or subset can be used as a keypoint. However, the problem with this direct input is that a very large number of correspondences arise because there are a large number of points in each frame.

[0052] Next, a feature descriptor is computed for each keypoint. The feature descriptor assembles the information of the keypoints to generate vectors that can be compared to each other. Similarly, many known feature options exist, such as NARF, FPFH, BRIEF, or SIFT, to be selected.

[0053] Next, a set of correspondences is estimated based on the similarity between features and locations, using a set of feature descriptors and their XYZ positions in the two datasets.

[0054] Next, since the data is generally considered noisy and not all correspondences are necessarily valid, unfavorable correspondences in the alignment process are rejected. For example, the rejection of unfavorable correspondences may be completed by using RANSAC or by reducing the number of correspondences found to use only those with a predetermined proportion.

[0055] Finally, from the remaining pair of good correspondences, the actual motion transformation is calculated and iterated over and over until several convergence criteria are met, thereby obtaining the transformation relationships of the alignment point cloud.

[0056] (2.3. Exemplary explanation of facial feature extraction in steps S14 and S16) In step S14 described above, it is necessary to detect and extract the sample face feature SFL of a random face sample from the image data of the random face sample. In step S16, it is necessary to extract general-purpose face feature data GFL based on general-purpose face images. A facial landmark is typically a facial structure used to locate and represent the salient region of the face. It is detected that the facial features are a subset of the shape prediction problem. Given an input image (usually an ROI specifying an area of ​​interest), the shape predictor attempts to locate keypoints of interest along the shape. Various methods exist in this field for detecting and extracting facial features. Below, only a few examples will be briefly described.

[0057] Typically, detecting facial features involves the following two steps: In step 1, the face is positioned in the image. In step 2, key facial structures are detected within the facial ROI.

[0058] Face detection (Step 1) can be achieved using various methods. For example, the built-in Haar cascade of OpenCV can be used. This is also applicable to HOG+linear SVM object detectors that have been pre-trained specifically for the face detection task. Alternatively, face positioning can be performed using algorithms based on deep learning. For this invention, the method by which faces are detected in the image is not that important; it is sufficient to obtain the face boundary frame (i.e., the (x,y) coordinates of the face in the image).

[0059] Given a facial region, the detection of key facial structures within that region can be applied as described in step 2. While various facial feature detectors and detection algorithms exist in this field, all methods essentially attempt to position and mark the facial regions of the mouth, right eyebrow, left eyebrow, right eye, left eye, nose, and chin.

[0060] For example, a facial feature predictor representing key facial structures can be used, utilizing the 68 feature points proposed by Kazemi and Sullivan, as shown in Figure 1, which is included in the dlib library. Of course, other types of facial recognition detectors also exist, including a 194-point model that can be trained on the HELEN dataset. These can be used to detect facial features in real time with high quality.

[0061] Regardless of the dataset used, the same dlib frames can be used to train the predictor with the input training data. This is particularly useful in situations where it is necessary to train a dedicated facial feature detector or customize a shape predictor.

[0062] It should be understood that the above brief explanation is merely an illustrative example of extracting facial features from a facial image. Any appropriate method for detecting and extracting facial features may be chosen as needed. To clarify the focus of the explanation, further details are omitted in this specification.

[0063] (2.4. Exemplary explanation of the deformation of the depth difference image in step S17) The following describes in detail the process of obtaining a third depth difference image DDI3 by morphing the second depth difference image DDI2 using the sample face feature data SFL marked on the second depth difference image DDI2, based on the general-purpose face feature data GFL described in step S17 above.

[0064] To convert image I to image J, we first need to establish a pixel correspondence between the two images. In other words, for each pixel (x) in the image... i ,y i Regarding ), in image J, the corresponding pixel (x j ,y j It is necessary to find the pixel positions (x) in the transformed image. If these correspondences are found, the image transformation can be performed in two stages. First, the pixel positions (x) in the transformed image are found. m ,y m It is necessary to calculate ). This is given by the following formula.

[0065]

number

[0066] Next, use the following formula (x m ,y m It is necessary to determine the pixel intensity in ).

[0067]

number

[0068] Clearly, according to equations (1) and (2) above, we can find corresponding points in image J for each pixel in image I, but this is time-consuming and unnecessary. Here, if we can calculate the correspondences of only some representative keypoints, the processing process will be greatly simplified. Therefore, the idea here is to simplify the deformation process by establishing correspondences using facial feature points that represent the key structure of the extracted face, and then interpolating the remaining pixels.

[0069] In the context of the present invention, for example, the 68 feature points described above can be used as corresponding points to establish a correspondence. Therefore, these corresponding points are obtained first. As shown in Figure 7, the feature data SFL of the sample face marked in the second depth difference image DDI2 is the data of the 68 feature points that are the corresponding points. Naturally, as can be understood, more points can be added or some points can be removed as needed. For example, feature points at locations such as both ears, the center of the skull, and the apex of the cheekbones can be added as corresponding points.

[0070] Next, as shown in Figure 7, the second depth difference image DDI2 is triangulated using Delaunay triangulation based on these points. The result of Delaunay triangulation is a list of triangles indicated by the point indices in the 68-point array. As shown in Figure 7, the triangulation divides the second depth difference image DDI2 into several triangles, each with three points from the 68 feature points as its vertices. The triangulation is stored as a 3-column array. For example, It may also take forms such as (38,40,37), (35,30,29), (38,37,20), (18,37,36), (33,32,30)...

[0071] As you might imagine, if we perform the above triangulation on two face images based on the same feature points, the corresponding triangles in the two images will indicate similar facial regions. Therefore, we can obtain a triangular (or regional) correspondence between the two face images.

[0072] When the second depth difference image DDI2 is triangulated, since the general face feature data GFL has already been acquired, it is equivalent to the positions and coordinates in the deformed image of each feature point in the second depth difference image DDI2 being known. Subsequently, by calculating an affine transform, each triangle in the second depth difference image DDI2 can be deformed, thereby matching the positions of each feature point in the second depth difference image DDI2 with the positions of each feature point in the general face feature data GFL. Through such triangle matching, the second depth difference image DDI2 is deformed into a third depth difference image DDI3 showing the general face deformation map GFDM.

[0073] The above-mentioned affine transform is the simplest way to transform one set of three points (i.e., one triangle) into any other set of three points (i.e., another triangle). This encodes translation (shift), scale, rotation, and shear. For example, using an affine transform, the shape of a square can be changed into a parallelogram in any direction and proportion. However, the flexibility of the affine transform is insufficient to transform a square into an arbitrary quadrilateral. In other words, when an affine transform is performed, parallel lines within the graphic continue to maintain their parallel relationship.

[0074] Currently, there are multiple software libraries that implement the calculation of affine transforms. For example, in OpenCV, an affine transform is a 2×3 matrix. The first two columns in the matrix encode rotation, scale, and shear, and the last column encodes translation.

[0075]

Number

[0076] Therefore, for the point (x, y), the point (x t , y t ) after affine transform can be obtained by the following formula.

[0077]

number

[0078] Therefore, by utilizing such affine transformations, it is possible to achieve matching deformations between each triangle in the second depth difference image DDI2 mentioned above and the general-purpose facial feature data GFL.

[0079] (2.5. Exemplary explanation of interpolation calculation based on imaging parameters in step S17) As described above, in most situations, the imaging parameters (distance, angle) of the iToF system change, which is also one of the causes of facial image deformation. Therefore, in most situations, in order to obtain a general-purpose facial deformation map GFDM, it is necessary to repeatedly perform the step of generating the third depth difference image DDI3 based on different imaging parameters, thereby generating multiple third depth difference images DDI31, DDI32, DDI33, etc., corresponding to multiple sets of imaging parameters (distance, angle). Finally, the general-purpose facial deformation map GFDM is obtained by interpolation calculations between these multiple third depth difference images DDI31, DDI32, DDI33, etc. The interpolation calculation function associated with imaging parameters (distance, angle) for representing the general-purpose facial deformation map GFDM according to the present invention will be described in detail below.

[0080] As mentioned above, the distance between the iToF camera and the subject's face, and the angle between the optical axis of the iToF camera and the field of view direction of the subject's face, can be used as the shooting parameters here. Figure 8 shows an example of distance and angle as the shooting function. As expected, a versatile face deformation map GFDM can be obtained simply by calculating a depth difference image, i.e., a third depth difference image DDI3, in a specific key point configuration using the method described above, and then interpolating the results for other configurations. As can be seen, other parameters may be added in addition to distance and angle. For example, if an RGB image is obtained, a color parameter may be added to account for skin color. However, when infrared light is used, the acquired skin data becomes more uniform (e.g., the level difference between white and black skin is about 15%) and much lower than in the RGB region. Therefore, the method of using only distance and angle as parameters is considered to be effective for all types of skin.

[0081] As shown in Figures 9 and 10, for example, at least three distances can be considered, and at least nine points at each distance can be used as keypoint structures. For example, the distances are 30 cm, 40 cm, and 50 cm. At each distance, by changing the pitch angle and yaw angle of the subject's face relative to the camera's optical axis, nine keypoint structures can be obtained as shown in Figure 10. In this way, a total of at least 27 keypoint structures can be obtained.

[0082] Therefore, by repeatedly changing the shooting parameters and performing the method described above to obtain the third depth difference image, a third depth difference image DDI3 for each keypoint structure can be obtained. In this way, the general-purpose face deformation map GFDM can be imagined as a set containing depth difference images at multiple keypoint structures for interpolation. Thus, such interpolation calculations can be defined in the following expression to show the state of the pixel (x,y).

[0083] Δ(x,y)=interpolate_generic_deformation(x,y,yaw,pitch,distance)

[0084] Here, `interpolate_generic_deformation` is an interpolation function for depth difference images based on all keypoint structures (e.g., 27), which allows us to obtain a generic face deformation map (GFDM) adapted to all possible positions.

[0085] The design and process based on interpolation of depth difference images in keypoint structures will be briefly explained below with reference to Figures 11 to 13.

[0086] Figure 11 shows filtered second depth difference images DDI2, marked with sample face feature data SFL, obtained, for example, from nine keypoint structures 50 cm away from the camera. Figure 12 shows third depth difference images DDI3, marked with general face feature data GFL, obtained after each of these nine second depth difference images DDI2 have been deformed.

[0087] Assuming the distance remains constant, interpolation calculations can be performed using these nine third depth difference images DDI3 to obtain the third depth difference image DDI3 at any position at that distance. Figure 13 illustrates the concept and process of this procedure. The "?" in the figure represents the third depth difference image at a certain position that needs to be obtained using interpolation. As envisioned, the coordinates of "?" on the coordinate axes can be determined by three possible methods. For example, the coordinates of the rectangle in Figure 13 are obtained by direct values ​​(y D ,p D ) indicates that the coordinates of the pentagon are linear values ​​(y L ,p L ) indicates the hexagon's coordinates, and the arc value (y C ,p CThis shows the interpolation of the deflection map at "?" (i.e., the y value) can be obtained by interpolating the deflection maps of the nine second depth difference images DDI2, which are the frames of the interpolation calculation. For example, as shown in Figure 13, the y value at "?" can be obtained by interpolating the y values ​​of the second depth difference images DDI2 at the fourth and fifth positions. Similarly, the interpolation of the pitch map at "?" (i.e., the p value) can be obtained by interpolation. For example, as shown in Figure 13, the p value at "?" can be obtained by interpolating the p values ​​of the second depth difference images DDI2 at the eighth and third positions.

[0088] Next, the blending value of the interpolated pitch map and the interpolated yaw map can be calculated using the following formula.

[0089] Diff = k*y + (1-k)*p (5)

[0090] Here, k is the ratio of the p-value to the y-value, and is given by the following formula.

[0091] k = y / (y + p) (6)

[0092] In equations (5) and (6) above, different interpolation types are used to set different k values ​​as needed, thereby directly determining the value (y D ,p D ), linear value (y L ,p L ), and / or arc value (y C ,p C This allows us to obtain ). Also, if both the y-value and p-value are 0, we set k = 0.5.

[0093] As can be understood, the interpolation process within the same distance described above also applies to situations with more yaw and pitch angle axes. In such extended situations, there are more third depth difference images, which are keypoint structures of the interpolated frames, and the interpolation calculation becomes finer.

[0094] Figures 14a-d show schematic simulations of the third depth difference image DDI3 at four random positions at 50cm, obtained by the interpolation calculation process shown in Figure 13. In Figures 14a-d, the two data bars at the bottom of the image show the pitch angle and yaw angle values ​​at that position, respectively. When the angle is 0, the color gamut in the data bar is in the center.

[0095] Below, we will examine the state of interpolation calculations that take into account the distances actually used.

[0096] First, as explained above, the interpolation of pitch and yaw angle values ​​at any position within the same distance can be calculated for multiple distances (for example, three distances of 50cm, 40cm, and 30cm, respectively) as described above. Then, as shown in Figure 15, the interpolation calculation of an arbitrary depth difference image at a given distance, pitch angle, and yaw angle (for example, "?" in the figure) may be performed in the following steps: (1) Of the three distance coordinate systems (for example, 30cm(d0), 40cm(d1), and 50cm(d2) shown in Figure 15), the corresponding pitch angle value p and y angle value y are interpolated in the nearest coordinate system to the given position (d). In the example shown in Figure 15, if the position "?" is between 30cm(d0) and 40cm(d1), first the corresponding y0 and p0 are obtained in the nearest 30cm coordinate system by the above interpolation calculation, and then i0 is obtained. (2) Next, the corresponding pitch angle value p and y angle value y are interpolated in the far-field coordinate system closest to the given position among the three distance coordinate systems. In the example shown in Figure 15, the corresponding y1 and p1 are obtained by the above interpolation calculation in the nearest far-field coordinate system of 40 cm, and then i1 is obtained. (3) Perform interpolation calculations for i0 and i1 based on the following distance coefficients, Diff = k * i0 + (1 - k) * i1 (7) In equation (7) above, k = (d1 + d) / (d1 + d0).

[0097] Thus, by repeatedly performing the interpolation calculation described above, an arbitrary depth difference image can be obtained by interpolating between the third depth difference images DDI3 of all available keypoint structures. Such an image set or function is the general-purpose face deformation map GFDM according to the present invention.

[0098] It should be understood that in the above interpolation calculation process, the corresponding pitch angle value p and y

[0099] (2.6. Modified version of the method for obtaining the first depth difference image) The above describes an example of obtaining the first depth difference image DDI1 by point cloud alignment, with reference to Figure 5. Alternatively, depending on the situation, the first depth difference image DDI1 can be obtained from point cloud data collected using other methods.

[0100] For example, as shown in Figure 16, if the camera intrinsics of the iToF camera used are known, the GT grid may be directly projected as a depth image onto the depth image domain using the camera intrinsics, and then the point cloud from the obtained sample data of the face-focused region may be projected to obtain a projected image of the face-focused region. Subsequently, a first depth difference image DDI1 may be calculated based on the projected GT image and the projected image of the face-focused region.

[0101] The camera intrinsic parameters mentioned here are a set of functions or models relating to the intrinsic imaging attributes of the iToF camera used, which can describe the positional transformation relationship between any point in the subject and its corresponding point in the image captured by the camera. For example, camera intrinsic attributes can include a model of the relationship between a point in iToF camera coordinates and its projection onto the image. Such a model is, for example, a pinhole model represented by the following camera matrix (homogeneous coordinates):

[0102]

number

[0103] In the above equation (8), f x and f y is the lens focal length in the x and y directions, and c x and c y is the camera principal point position, and s is the skew factor.

[0104] Furthermore, the camera's internal attributes may include strain parameters d = [k1k2k3p1p2], where k1, k2, and k3 are axial strain parameters, and p1 and p2 are tangential strain parameters.

[0105] By calling these known camera intrinsic parameters, the GT grid can be projected directly onto the depth image domain.

[0106] Furthermore, as shown in Figure 16, in this modified example, before calculating the first depth difference image DDI1, other types of image data (e.g., IR images, RGB images, or laser images) may be used to verify whether the GT projection image is aligned. It should be understood that this step is not mandatory.

[0107] [3. Detailed explanation of the process of correcting depth information by applying a generic face deformation map] First, consider the simplest scenario, namely that the captured user face image is exactly the same as the captured conditions of a previously obtained sample face image. That is, the user's face image has exactly the same capture parameters as the sample face image. In this scenario, first, the user's face feature data (client face landmark) CFL is detected and extracted from the user's face image. Next, the generic face deformation map GFDM is deformed so that the generic face feature data marked on the generic face deformation map GFDM matches the detected user's face feature data CFL. This deformation can be understood as the reverse process of the deformation described above. This yields a fourth depth difference image DDI4 (not shown). Finally, the depth information in the user's face image is corrected based on the fourth depth difference image DDI4.

[0108] Clearly, in many situations other than the simple scenario described above, the actual user's face image differs from the shooting conditions of the sample face images obtained during the learning process. In such situations, the depth information in the user's face image can be corrected by calculating a face deformation map corresponding to the user's face image using the general-purpose face deformation map GFDM. The following describes the processing of each step in an embodiment in which the general-purpose face deformation map is applied to correct depth information in such a situation, with reference to Figure 17. For clarity of illustration, Figure 17 also shows step S2, in which the user's face is photographed and image data of the user's face image is obtained.

[0109] In step S31, the user's facial feature data (CFL) is detected and extracted from the captured user's face image. The method for performing this step can be found by referring to the example of facial feature detection and extraction described above.

[0110] Next, in step S32, the shooting parameters (distance, angle) of the user's face image are obtained, and these shooting parameters are interpolated into a general-purpose face deformation map GFDM to obtain a specific fifth depth difference image DDI5 marked with general-purpose face feature data GFL corresponding to specific shooting parameters of the user's face image. Figure 18 illustrates an example of obtaining a fifth depth difference image DDI5 with two specific point structures from a general-purpose face deformation map GFDM based on two specific sets of shooting parameters.

[0111] Next, in step S33, the fifth depth difference image DDI5 is modified to match the general-purpose face feature data GFL marked on the fifth depth difference image DDI5 with the detected user face feature data CFL, thereby obtaining the sixth depth difference image DDI6.

[0112] Finally, in step S34, the iToF depth information in the user's face image captured by the iToF camera is corrected based on the sixth depth difference image DDI6, and the corrected user's face image is obtained.

[0113] The above describes a method for correcting facial deformation in facial images acquired by the iToF method according to the present invention. In the method according to the present invention, a depth deformation map associated with facial feature information is created to correct depth errors that occur when an iToF device captures a known scene (in this case, a face). Such a depth deformation map is created from scenes recorded with different key structures (e.g., [yaw angle, pitch angle, distance]), and then transformed to suit a general-purpose facial model. Furthermore, by calculating the necessary deformation based on deformation interpolation in the key structure, the depth deformation map is applied to any other structure within a predetermined range, thereby fitting the general-purpose facial model to a specific model and achieving correction for depth deformation of any specific facial image. According to this method, a facial depth image with higher accuracy can be obtained.

[0114] [4. Imaging apparatus according to the present invention] The present invention further provides an imaging device. This imaging device can capture images using an iToF imaging unit and can correct the depth image of a face captured by applying the face deformation correction method described above. Hereinafter, an example of the configuration of the imaging device 1900 according to the present invention will be described with reference to Figure 19. The imaging device 1900 includes, for example, an imaging unit 1901, a processing unit 1902, and a storage unit 1903.

[0115] The imaging unit 1901 may include an iToF sensor to acquire iToF data of the user's face. The imaging unit 1901 outputs the iToF data obtained by the iToF sensor to the processing unit 1902, which will be described later. The iToF data can be used to obtain distance information (depth) of the subject by calculating the phase difference between the illuminated light and the reflected light based on the sensing data. The imaging unit 1901 may also include other types of optical imaging units or sensor units to acquire other types of image information, such as IR images, RGB images, or laser images.

[0116] The processing unit 1902 includes a shooting parameter detection module, a face feature detection module, a face image deformation module, and a face image correction module. The processing unit 1902 can execute the face correction method of the present invention as described above using the corresponding modules.

[0117] For example, the processing unit 1902 can acquire the shooting parameters of the user's face image, such as the angle between the field of view direction of the face and the optical axis of the imaging device 1901 (e.g., yaw angle and pitch angle), and the distance between the imaging device 1901 and the user's face, using the shooting parameter detection module. The processing unit 1902 can detect and extract the user's face feature data CFL from the user's face image (e.g., IR or RGB image) obtained by the imaging unit 1901, using the face feature detection module. The processing unit 1902 can also interpolate the shooting parameters of the user's face image acquired by the shooting parameter detection module into the general-purpose face deformation map GFDM obtained from the storage unit 1903. The processing unit 1902 can also deform the general-purpose face deformation map GFDM obtained from the storage unit 1903 using the face image deformation module. Furthermore, the processing unit 1902 can correct the depth information in the user's face image using the deformed general-purpose face deformation map with the face image correction module, thereby correcting the face deformation of the user's face image. The general-purpose face deformation map GFDM stored in the memory unit 1903 may be one that has been pre-calculated and stored. Alternatively, the processing unit 1902 may further include a general-purpose face deformation map calculation module (not shown) that calculates the general-purpose face deformation map GFDM in the manner described above by calling data from the shooting parameter detection module and the face feature detection module, and stores it in the memory unit 1903 for use.

[0118] The imaging device according to the present invention can correct depth errors that occur when an iToF device captures a known scene (in this case, a face) by creating a depth deformation map associated with facial feature information. Such a depth deformation map is created from scenes recorded with different key structures (e.g., [yaw angle, pitch angle, distance]) and then transformed to suit a general-purpose face model. Furthermore, the imaging device according to the present invention applies the depth deformation map to any other structure within a predetermined range by calculating the necessary deformation based on deformation interpolation in the key structure, thereby fitting the general-purpose face model to a specific model and achieving correction for depth deformation of any specific face image. Therefore, the imaging device according to the present invention can obtain face depth images with higher accuracy.

[0119] [5. Electronic device according to the present invention] The correction method according to the present invention, as described above, can have various other applications besides the imaging apparatus described above. For example, it can be applied to other electronic devices and systems other than imaging apparatuses. For example, pedestrian recognition systems in in-vehicle autonomous driving systems, security systems, and mobile phone payment platforms.

[0120] The following describes an example configuration of an electronic device 2000 according to an application example of the present invention, with reference to Figure 20. The electronic device 2000 includes, for example, a CPU 2001, ROM 2002, RAM 2003, recording medium 2004, input / output interface 2005, communication interface 2006, operation input device 2008, and display device 2009. The electronic device 2000 also has, or is capable of communicating with, an iTOF sensor 2007. In the electronic device 2000, for example, a bus 2010, which is a data transmission path, is used to connect each component.

[0121] The CPU 2001 is composed of, for example, one or more processors, each consisting of an arithmetic circuit such as a CPU, and various processing circuits, and functions as a control unit that controls the entire electronic device 2000. Moreover, the CPU 2001 can execute a program based on the method of the present invention as described above.

[0122] ROM2002 stores control data such as programs or calculation parameters used by CPU2001. RAM2002 temporarily stores, for example, programs executed by CPU2001.

[0123] The recording medium 2004 stores various data, such as images used in the method according to the embodiment described above. Examples of the recording medium 2004 include non-volatile memory such as flash memory. Furthermore, the recording medium 2004 is detachable from the electronic device 2000.

[0124] Input / output interface 2005 can be connected to, for example, an input device 2008 or a display device 2009. Examples of input / output interfaces 2005 include USB (Universal Serial Bus) terminals, DVI (Digital Visual Interface) terminals, HDMI (High-Definition Multimedia Interface) (registered trademark) terminals, and various processing circuits.

[0125] The operation input device 2008 functions, for example, as an input unit that receives user input to the electronic device 2000, and is connected to the input / output interface 2008 inside the electronic device 2000.

[0126] The display device 2009 outputs the recognition result to the user, is installed in the electronic device 2000, and is connected to the input / output interface 2005 inside the electronic device 2000. Examples of the display device 2009 include liquid crystal displays or organic EL displays (organic electro-luminescence displays).

[0127] Furthermore, the input / output interface 2005 can also be connected to external devices such as external input devices (e.g., keyboards and mice) or external display devices. Moreover, the input / output interface 2005 may be connected to a drive (not shown). This drive is a reader / writer for removable recording media such as magnetic disks, optical disks, or semiconductor memory, and is either built into or external to the electronic device 2000. This drive reads information recorded on the equipped removable recording media and outputs it to the RAM 2003. Moreover, this drive can also write data to the equipped removable recording media.

[0128] The communication interface 2006 functions as a communication unit for communicating wirelessly or via wired connection with external devices of the electronic device 2000. The communication interface 2006 can acquire relevant data, such as general-purpose facial features, in the above-described method of the present invention via the Internet. Examples of the communication interface 2006 include a communication antenna and RF (radio frequency) circuit (wireless communication), an IEEE 802.15.1 port and transceiver circuit (wireless communication), an IEEE 802.11 port and transceiver circuit (wireless communication), or a LAN (local area network) terminal and transceiver circuit (wired communication).

[0129] The iTOF sensor 2007 functions as a device for acquiring iToF image data of a face according to the present invention.

[0130] The above describes an example of the hardware configuration of the electronic device 2000. Furthermore, the hardware configuration of the electronic device 2000 is not limited to the configuration shown in Figure 20. In detail, each of the above components may be constructed using general-purpose materials, or it may be constructed using hardware specialized to realize the function of each component. The above configuration can be appropriately modified depending on the technical level at the time of implementation.

[0131] The present invention can be configured or configured as follows, for example. (1) The steps include creating a deformation map of a generic face that includes generic face feature data, Steps to obtain the user's face image, The step of applying the generic face deformation map to the user's face image is included, and the step of applying the generic face deformation map to the user's face image is The steps include: transforming the deformation map of the generic face using the generic face feature data and the user face feature data; A method for correcting facial deformation, comprising the steps of correcting depth information in the user's facial image using the deformation map of the deformed general-purpose face, thereby correcting facial deformation in the user's facial image. (2) The aforementioned general-purpose face deformation map is, To obtain iToF data from a facial sample, To obtain GT data of the aforementioned face sample, Aligning the GT data and the iToF data, and calculating a first depth difference image that shows the difference in depth information between the GT data and the iToF data, To detect and extract the characteristic data of the sample face of the aforementioned face sample, The feature data of the sample face is matched with the first depth difference image, thereby obtaining a second depth difference image having the feature data of the sample face, which serves as a marker. To obtain the aforementioned general-purpose facial feature data, The method according to (1) above, wherein the sample face feature data marked on the second depth difference image based on the general-purpose face feature data is used to deform the second depth difference image, and a third depth difference image that matches the general-purpose face feature data and is marked with the general-purpose face feature data is obtained as a deformation map of the general-purpose face. (3) The step of calculating the first depth difference image is: The steps include converting the iToF data point cloud into a grid, A step of aligning the GT data and the grid, The steps include projecting the aligned GT data and the grid to obtain a projected image of the GT data and a projected image of the iToF data, The method according to (2) above, comprising the step of calculating the first depth difference image based on the projected image of the GT data and the projected image of the iToF data. (4) The step of calculating the first depth difference image is: The steps include: using the camera's internal parameters to directly project the GT data and obtain a projected image of the GT data; The steps include: projecting the point cloud of the iToF data to obtain a projected image of the iToF data; The method according to (2) above, comprising the step of calculating the first depth difference image based on the projected image of the iToF data and the projected image of the GT data. (5) The method according to (4) above, further comprising the step of determining whether the projected image of the GT data aligns with the image of the human sample after projecting the GT data. (6) The step of applying the aforementioned generic face deformation map is: The steps include detecting and extracting the user's facial feature data from the user's facial image, The steps include: deforming the generic face deformation map so as to match the generic face feature data marked on the generic face deformation map with the detected user face feature data, thereby obtaining a fourth depth difference image; The method according to any one of (2) to (5) above, further comprising the step of correcting the depth information in the user's face image based on the fourth depth difference image. (7) The step of creating the aforementioned general-purpose face deformation map is: The steps include: obtaining the shooting parameters of the iToF data of the face sample; The step of changing the value of the aforementioned shooting parameter and repeating the step of generating the third depth difference image, The method according to any one of (2) to (5) above, further comprising the step of generating a plurality of third depth difference images constituting the deformation map of the general-purpose face based on the values ​​of a plurality of sets of the aforementioned shooting parameters. (8) The step of applying the aforementioned generic face deformation map is: The steps include: obtaining the shooting parameters of the user's face image and interpolating the shooting parameters into the generic face deformation map to obtain a fifth depth difference image on which the generic face feature data corresponding to the shooting parameters of the user's face image is marked; The steps include detecting and extracting the user's facial feature data from the user's facial image, The steps include obtaining a sixth depth difference image by transforming the fifth depth difference image so as to match the general-purpose face feature data marked on the fifth depth difference image with the detected user face feature data, The method according to (7) above, further comprising the step of correcting the depth information in the user's face image based on the sixth depth difference image. (9) The method according to (7) above, wherein the shooting parameters include the angle between the angle of view of the face and the camera, and the distance between the camera and the face. (10) The angle is the method described in (9) above, including the yaw angle and the pitch angle. (11) The step of creating the aforementioned general-purpose face deformation map is: The steps include: obtaining the shooting parameters of the iToF data of the face sample; The step of changing the value of the aforementioned shooting parameter and repeating the step of generating the third depth difference image, The method further includes the step of generating a plurality of the third depth difference images as sample images based on the values ​​of a plurality of sets of the aforementioned shooting parameters, Among these, the face sample is a random face sample, and the step of creating the deformation map of the general-purpose face is, The steps include generating multiple sets of sample images for multiple face samples, The method according to any one of (2) to (5) above, further comprising the step of calculating the average value of multiple sets of sample images for each set of the shooting parameter values, thereby generating the general-purpose face deformation map consisting of multiple average third depth difference images. (12) The step of applying the aforementioned generic face deformation map is: The steps include: obtaining the shooting parameters of the user's face image and interpolating the shooting parameters into the generic face deformation map to obtain a fifth depth difference image on which the generic face feature data corresponding to the shooting parameters of the user's face image is marked; The steps include detecting and extracting the user's facial feature data from the user's facial image, The steps include obtaining a sixth depth difference image by transforming the fifth depth difference image so as to match the general-purpose face feature data marked on the fifth depth difference image with the detected user face feature data, The method according to (11) above, further comprising the step of correcting the depth information in the user's face image based on the sixth depth difference image. (13) An imaging unit configured to acquire a user's facial image, A storage unit for storing a deformation map of a general-purpose face that includes general-purpose face feature data, wherein the deformation map of the general-purpose face is generated by any one of the methods described in (7) and (9) to (11) above, A processing unit, The user's facial feature data is detected and extracted from the aforementioned user's facial image. The deformation map of the generic face is deformed using the generic face feature data and the user face feature data, and An imaging apparatus comprising a processing unit configured to correct depth information in the user's face image using the deformed deformation map of the general-purpose face, thereby correcting the facial deformation of the user's face image. (14) The aforementioned processing unit is The capture parameters of the user's face image are obtained and interpolated into the deformation map of the generic face to obtain a correction depth difference image marked with the feature data of the generic face. Based on the user's facial feature data, the general-purpose facial feature data marked on the correction depth difference image is used to deform the correction depth difference image, and The imaging apparatus according to (13) above, further configured to correct the depth information in the user's face image based on the deformed depth difference image for correction. (15) A storage medium that stores a computer-readable program that, when executed on a processor, performs the method described in any one of (1) to (12) above.

[0132] The correction method, imaging apparatus, and storage medium according to the present invention have been described above with reference to the drawings. However, the present invention is not limited thereto, and those skilled in the art will understand that various modifications, combinations, sub-combinations, and variations can be made without departing from the substance or scope limited to the appended claims of the present invention.

Claims

1. The steps include creating a deformation map of a generic face that includes generic face feature data, Steps to obtain the user's face image, The step includes applying the generic face deformation map to the user's face image, and the step of applying the generic face deformation map to the user's face image is The steps include: transforming the deformation map of the generic face using the generic face feature data and the user face feature data; A method for correcting facial deformation, comprising the steps of correcting depth information in the user's facial image using the deformation map of the deformed general-purpose face, thereby correcting facial deformation in the user's facial image.

2. The aforementioned general-purpose face deformation map is, To obtain iToF data from a facial sample, To obtain GT data of the aforementioned face sample, Aligning the GT data and the iToF data, and calculating a first depth difference image that shows the difference in depth information between the GT data and the iToF data, To detect and extract the characteristic data of the sample face of the aforementioned face sample, The feature data of the sample face is matched with the first depth difference image, thereby obtaining a second depth difference image having the feature data of the sample face, which serves as a marker. To obtain the aforementioned general-purpose facial feature data, The method according to claim 1, wherein the sample face feature data marked on the second depth difference image based on the general-purpose face feature data is used to deform the second depth difference image, and a third depth difference image is obtained as a deformation map of the general-purpose face, which is matched with the general-purpose face feature data and marked with the general-purpose face feature data.

3. The step of calculating the first depth difference image is: The steps include converting the iToF data point cloud into a grid, The steps include aligning the GT data with the grid, The steps include projecting the aligned GT data and the grid to obtain a projected image of the GT data and a projected image of the iToF data, The method according to claim 2, comprising the step of calculating the first depth difference image based on the projected image of the GT data and the projected image of the iToF data.

4. The step of calculating the first depth difference image is: The steps include: using the camera's internal parameters to directly project the GT data and obtain a projected image of the GT data; The steps include: projecting the point cloud of the iToF data to obtain a projected image of the iToF data; The method according to claim 2, comprising the step of calculating the first depth difference image based on the projected image of the iToF data and the projected image of the GT data.

5. The method according to claim 4, further comprising the step of determining whether the projected image of the GT data aligns with an image of a human sample after projecting the GT data.

6. The step of applying the aforementioned generic face deformation map is: The steps include detecting and extracting the user's facial feature data from the user's facial image, The steps include: deforming the generic face deformation map so as to match the generic face feature data marked on the generic face deformation map with the detected user face feature data, thereby obtaining a fourth depth difference image; The method according to any one of claims 2 to 5, further comprising the step of correcting the depth information in the user's face image based on the fourth depth difference image.

7. The step of creating the aforementioned general-purpose face deformation map is: The steps include obtaining the shooting parameters of the iToF data of the face sample, The step of changing the value of the aforementioned shooting parameter and repeating the step of generating the third depth difference image, The method according to any one of claims 2 to 5, further comprising the step of generating a plurality of third depth difference images constituting the deformation map of the general-purpose face based on the values ​​of a plurality of sets of the aforementioned shooting parameters.

8. The step of applying the aforementioned generic face deformation map is: The steps include: obtaining the shooting parameters of the user's face image and interpolating the shooting parameters into the generic face deformation map to obtain a fifth depth difference image on which the generic face feature data corresponding to the shooting parameters of the user's face image is marked; The steps include detecting and extracting the user's facial feature data from the user's facial image, The steps include obtaining a sixth depth difference image by transforming the fifth depth difference image so as to match the general-purpose face feature data marked on the fifth depth difference image with the detected user face feature data, The method according to claim 7, further comprising the step of correcting the depth information in the user's face image based on the sixth depth difference image.

9. The method according to claim 7, wherein the shooting parameters include the angle between the field of view direction of the face and the camera, and the distance between the camera and the face.

10. The method according to claim 9, wherein the angle includes the yaw angle and the pitch angle.

11. The step of creating the aforementioned general-purpose face deformation map is: The steps include obtaining the shooting parameters of the iToF data of the face sample, The step of changing the value of the aforementioned shooting parameter and repeating the step of generating the third depth difference image, The method further includes the step of generating a plurality of third depth difference images as sample images based on the values ​​of a plurality of sets of the aforementioned shooting parameters, Among these, the face sample is a random face sample, and the step of creating the deformation map of the general-purpose face is, The steps include generating multiple sets of sample images for multiple face samples, The method according to any one of claims 2 to 5, further comprising the step of calculating the average value of multiple sets of sample images for each set of the shooting parameter values, thereby generating a deformation map of the general-purpose face consisting of multiple average third depth difference images.

12. The step of applying the aforementioned generic face deformation map is: The steps include: obtaining the shooting parameters of the user's face image and interpolating the shooting parameters into the generic face deformation map to obtain a fifth depth difference image on which the generic face feature data corresponding to the shooting parameters of the user's face image is marked; The steps include detecting and extracting the user's facial feature data from the user's facial image, The steps include obtaining a sixth depth difference image by transforming the fifth depth difference image so as to match the general-purpose face feature data marked on the fifth depth difference image with the detected user face feature data, The method according to claim 11, further comprising the step of correcting the depth information in the user's face image based on the sixth depth difference image.

13. An imaging unit configured to acquire a user's facial image, A storage unit for storing a deformation map of a general-purpose face that includes general-purpose face feature data, wherein the deformation map of the general-purpose face is generated by the method described in any one of claims 7 and 9 to 11. A processing unit, The user's facial feature data is detected and extracted from the aforementioned user's facial image. The deformation map of the generic face is deformed using the generic face feature data and the user face feature data, and An imaging apparatus comprising a processing unit configured to correct depth information in the user's face image using the deformed deformation map of the general-purpose face, thereby correcting the facial deformation of the user's face image.

14. The aforementioned processing unit is The capture parameters of the user's face image are obtained and interpolated into the deformation map of the generic face to obtain a correction depth difference image marked with the feature data of the generic face. Based on the user's facial feature data, the general-purpose facial feature data marked on the correction depth difference image is used to deform the correction depth difference image, and The imaging apparatus according to claim 13, further configured to correct depth information in the user's face image based on the deformed depth difference image for correction.

15. A storage medium that stores a computer-readable program, when executed on a processor, which implements the method according to any one of claims 1 to 12.

Images