Face shape correction device, method for controlling face shape correction device, and program

Abstract

To provide a face shape correction device capable of appropriately correcting the estimated shape of a face, a method for controlling the face shape correction device, and a program.SOLUTION: An image processing device 100 functioning as a face shape correction device comprises acquisition means (control unit 11) for acquiring a face image, estimation means (control unit 11) for estimating the face shape of a subject using machine learning with the face image as input data and acquiring the estimated face shape as output data, and correction means (control unit 11) for correcting the estimated face shape. The correction means acquires a first feature quantity required for correcting the estimated face shape on the basis of information on the data used in machine learning, acquires a second feature quantity required for correcting the estimated face shape on the basis of information on the estimated face shape, and corrects the estimated face shape on the basis of the first and second feature quantities.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a facial shape correcting device, a control method for a facial shape correcting device, and a program. [Background technology]

[0002] There is known a technique for estimating the shape of a face from an image of the face of a subject using machine learning. For example, Non-Patent Document 1 discloses a technique that uses a neural network to learn the correspondence between a large number of face images and face shapes, thereby making it possible to estimate an unknown face shape, for example, regardless of the facial expression. Then, this estimated face shape can be used, for example, in CG production, image editing, etc. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Matan Sela, Elad Richardson, Ron Kimmel, “Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation”, [online], September 15, 2017, Cornell University, [September 26, 2023], Internet〈https: / / arxiv.org / abs / 1703.10131〉 Summary of the Invention [Problem to be solved by the invention]

[0004] In the estimation of face shape by machine learning, a face shape corresponding to an unknown face image is estimated based on a face image used in the machine learning and a face shape as learning data. Therefore, if a face image that is relatively far from the learning data is input, there is a risk that the face shape will not be accurately estimated. For example, if most of the learning data is Caucasian data with deep facial features, if a Mongoloid face shape is estimated, the estimation result will be a face shape with a higher nose and deeper facial features than a general Mongoloid face shape. Also, if most of the learning data is adult data, if a child's face shape is estimated, the estimation result will be a face shape with a lower chin and a higher nose than a general child's face shape. In this way, if there are factors between the learning data and the data estimated based on the learning data that affect the acquisition of an accurate estimation result, it is generally preferable to increase the learning data corresponding to the face shape to be estimated. In this case, it is very time-consuming to newly prepare a large amount of learning data corresponding to the face shape.

[0005] The present invention has been made in view of the above problems, and has an object to provide a face shape correcting device that, when a face shape is estimated, can appropriately correct the face shape, and a control method and program for the face shape correcting device. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the facial shape correction device of the present invention comprises an acquisition means for acquiring a facial image including the face of a subject, an estimation means for estimating the shape of the subject's face by using machine learning with the facial image as input data and acquiring the estimated facial shape as output data, and a correction means for correcting the estimated facial shape, wherein the correction means acquires a first feature amount required for correcting the estimated facial shape based on information on data from the machine learning, and acquires a second feature amount required for correcting the estimated facial shape based on information on the estimated facial shape, and corrects the estimated facial shape based on the first feature amount and the second feature amount. Effect of the Invention

[0007] According to the present invention, when the face shape is estimated, the face shape can be appropriately corrected. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a block diagram showing an example of a hardware configuration when the face shape correcting device according to the first embodiment is applied to an image processing device. [Diagram 2] 4 is a flowchart showing a process executed by the image processing device. [Diagram 3] FIG. 13 is a side view of an estimated face shape. [Figure 4] FIG. 1 illustrates an example of a configuration of an imaging device. [Diagram 5] 13A and 13B are a side view and a front view of a face in a second embodiment. [Figure 6] 4 is a flowchart showing a process executed by the image processing device. [Figure 7] 13A and 13B are a side view and a front view of a face in a third embodiment. [Figure 8] 4 is a flowchart showing a process executed by the image processing device. [Figure 9] FIG. 11 is a diagram for explaining an estimated face shape. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Each embodiment of the present invention will be described in detail below with reference to the drawings. However, the configurations described in each of the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in each embodiment. For example, each part constituting the present invention can be replaced with any configuration that can exert a similar function. In addition, any configuration may be added. In addition, any two or more configurations (features) of each embodiment can be combined.

[0010] First Embodiment Hereinafter, the first embodiment will be described with reference to Figs. 1 to 4. Fig. 1 is a block diagram showing an example of a hardware configuration when the face shape correction device according to the first embodiment is applied to an image processing device. The image processing device 100 shown in Fig. 1 is a desktop or notebook personal computer in this embodiment, but is not limited thereto and may be, for example, a tablet terminal, a smartphone, or the like. The image processing device 100 has an input unit 10, a control unit 11, and a storage unit 13, which are connected to each other so as to be able to communicate with each other. The input unit 10 has an operation unit that can be operated by a user who uses the image processing device 100, and an interface (I / F) for acquiring information necessary for face shape estimation by machine learning, which will be described later, and correction of the estimation result, etc. The operation unit is not particularly limited, and examples thereof include a mouse, a keyboard, a joystick, and the like.

[0011] The control unit 11 includes a calculation unit 12. The calculation unit 12 includes a CPU 121 and a GPU (Graphics Processing Unit) 122, and executes various programs stored in the storage unit 13. The programs include, for example, programs for causing a computer to execute each unit and each means (a control method for the face shape correction device) of the face shape correction device. The CPU 121 controls the entire image processing device 100. The GPU 122 is a calculator capable of performing processing specialized for computer graphic calculations. The GPU 122 can generally perform calculations required for neural networks, such as matrix calculations, in a shorter time than the CPU 121. The GPU 122 can also handle calculation processing together with the CPU 121. The image processing device 100 may also include, for example, a TPU (Tensor Processing Unit) instead of the GPU 122. The image processing device 100 may also include an NPU (Neural network Processing Unit), and the NPU may handle calculation processing together with the CPU 121.

[0012] The storage unit 13 is composed of a primary storage device such as a RAM, and a secondary storage device such as a HD or SSD. The storage unit 13 stores various information required for face shape estimation and correction, such as an image input via the input unit 10, a trained model (machine learning model), measurement values, statistics, etc. The storage unit 13 can also store information related to the face shape corrected by the calculation unit 12. The image processing device 100 is detachably connected to an imaging device (imaging means) 14. The imaging device 14 is not particularly limited, and may be, for example, a digital camera. When the imaging device 14 is connected to the image processing device 100, communication with the imaging device 14 is possible.

[0013] FIG. 2 is a flowchart showing a process executed by the image processing device. The process in each step of the flowchart shown in FIG. 2 is realized by the control unit 11 reading out a program stored in, for example, a non-volatile memory in the storage unit 13, and developing and executing the program in a volatile memory in the control unit 11. As shown in FIG. 2, in step S201, the control unit 11 acquires a face image including the face of the subject (acquisition step). This face image is stored in the storage unit 13. Note that the face image is not particularly limited, and for example, an image captured by the imaging device 14, an image stored in advance in the storage unit 13, etc. are used. In addition, in this embodiment, the control unit 11 functions as an acquisition means for acquiring a face image, but is not limited thereto. For example, a part functioning as an acquisition means may be provided separately from the control unit 11. After step S201 is executed, the process proceeds to step S202.

[0014] In step S202, the control unit 11 estimates the shape of the face of the subject by using machine learning with the face image acquired in step S201 as input data (estimation step). In this embodiment, a trained model having a neural network is used for machine learning. In the trained model, when various face images are input, a large number of learnings are performed to output a target estimated face shape described later. The estimated face shape includes a feature amount such as a vector that corresponds one-to-one with the shape, for example, as in a three-dimensional morphing model. This vector can be constructed, for example, by principal component analysis of a large amount of face shape data. In addition, as the machine learning algorithm, deep learning is preferable in this embodiment, but is not limited thereto, and may be, for example, a support vector machine, a logistics regression, a decision tree, or the like. Then, the control unit 11 acquires this estimated face shape (hereinafter referred to as "estimated face shape") as output data from the trained model. In this embodiment, the control unit 11 functions as an estimation means for estimating an estimated face shape, but is not limited thereto. For example, a part that functions as an estimation means may be provided separately from the control unit 11. After step S202 is executed, the process proceeds to step S203.

[0015] In step S203, the control unit 11 judges whether or not to execute a correction process for correcting the estimated face shape acquired in step S202. This judgment is made based on deviation information regarding the deviation between the face image to be estimated for the estimated face shape and the face image learned by the learned model. As the deviation information in this embodiment, information regarding race is used. In this case, if there is a deviation regarding race (attribute) between the face included in the face image to be estimated for the estimated face shape and the face included in the face image learned by the learned model, the correction process is executed. For example, when most of the face images that are learning data (data) in machine learning are Caucasian face images that tend to have deep facial features, if the face image to be estimated for the estimated face shape is not Caucasian (for example, Mongoloid, etc.), the execution of the correction process is judged. In addition, when the face image that is learning data in machine learning is not Caucasian face image, if the face image to be estimated for the estimated face shape is Caucasian, the execution of the correction process is judged. In addition, when the identity of the race in the training data is clear, it is treated as that race. In contrast, when the identity of the race in the training data is unknown, the race is estimated based on the training data. The method of estimating the race is not particularly limited, and examples include an estimation method using another machine learning model that has learned the correspondence between face images and races, and a rule-based estimation method based on the features of facial organs and skin color. In addition, when race estimation is performed for the entire training data and a specific race is biased as a result, the most common race is set as the race in the training data. The estimation of the race in the face image that is the target of the estimation of the estimated face shape can be performed in the same manner as the estimation of the race in the training data. In this embodiment, the control unit 11 functions as a judgment means for judging whether or not to execute the correction process, but is not limited to this. For example, a part that functions as a judgment means may be provided separately from the control unit 11. Then, if the control unit 11 judges that the correction process is to be executed as a result of the judgment in step S202, the process proceeds to step S204. On the other hand, if the control unit 11 judges that the correction process is not to be executed as a result of the judgment in step S202, the process ends.

[0016] In step S204, the control unit 11 acquires a first feature amount (first shape feature amount) required for the correction process. In this embodiment, the control unit 11 acquires the first feature amount based on information related to the learning data. In this case, the first feature amount may be the distance between the eyes and the tip of the nose of a face included in a face image that is the learning data. There are no particular limitations on the method of acquiring the first feature amount, and examples of the method include a method in which a user uses a measuring instrument such as a ruler, a vernier caliper, or a laser range finder to measure the distance between the eyes and the tip of the nose, and transmits the distance information to the control unit 11 via the input unit 10.

[0017] In step S205, the control unit 11 acquires a second feature amount (second shape feature amount) required for the correction process. In this embodiment, the control unit 11 acquires the second feature amount based on information related to the estimated face shape estimated in step S202. In this case, the second feature amount is the distance between the eyes and the apex of the nose of the estimated face shape. This distance will be described with reference to FIG. 3. FIG. 3 is a side view of the estimated face shape. In FIG. 3, the left-right direction of the face is the X-axis direction, the up-down direction of the face is the Y-axis direction, and the front-back direction of the face is the Z-axis direction. The distance 301 between the eyes and the apex of the nose along the Z-axis direction is the second feature amount.

[0018] In step S206, the control unit 11 executes a correction process based on the first feature amount acquired in step S204 and the second feature amount acquired in step S205 (correction step). In this embodiment, the control unit 11 functions as a correction means that executes the correction process, but is not limited to this. For example, a part that functions as a correction means may be provided separately from the control unit 11. In this embodiment, the control unit 11 executes the correction process based on the ratio between the first feature amount and the second feature amount. Specifically, the correction process is executed using the following formula (1).

number

[0019] A is the first feature amount, and B is the second feature amount. Z1 is the Z-axis coordinate of the estimated face shape estimated by the trained model. Therefore, the estimated face shape is corrected only in the Z-axis direction. Z2 is the Z-axis coordinate of the corrected estimated face shape. The control unit 11 stores the corrected estimated face shape in the memory unit 13.

[0020] Here, consider the case where the process executed by the image processing device ends at step S202, that is, the case where steps S203 and after are omitted. For example, when most of the face images that are learning data in machine learning are Caucasian face images that tend to have deep facial features, the face image that is the target of estimation of the estimated face shape is a Mongoloid face image. In this case, the estimation result is output data with a higher nose and deep facial features than the general Mongoloid face shape. Therefore, by executing a correction process, the estimated face shape is appropriately corrected so that the discrepancy in terms of race between the face image that is the target of estimation of the estimated face shape and the face image learned by the trained model is reduced (suppressed). As a result, the estimated face shape has a tendency to have a general Mongoloid face shape. Also, as described above, the first feature amount is acquired based on information about the training data. The information about the training data can be, but is not limited to, face images of each race that are actually input to the trained model, and can be, for example, statistical data (statistical values) that can identify each race or the nationals of each country. Also, step S203 may be omitted. In this case, after step S202 is executed, the process sequentially executes steps S204 to S206. This ensures that the correction process is always executed regardless of whether the correction process is required.

[0021] FIG. 4 is a diagram showing an example of the configuration of an imaging device. FIG. 4(A) is a side view of the imaging device. The imaging device 14 shown in FIG. 4(A) is a device capable of capturing a face image and measuring distances. The imaging device 14 has an optical system 401 and an imaging unit 402. The optical system 401 is composed of a lens having an optical axis 401a parallel to the z-axis direction. The imaging unit 402 is arranged on the optical axis 401a. FIG. 4(B) is a diagram showing an arrangement state of a plurality of pixels 410 included in the imaging unit 402. As shown in FIG. 4(B), the plurality of pixels 410 are arranged in a matrix along the x-axis direction and the y-axis direction. FIG. 4(C) is a cross-sectional view of the pixel. As shown in FIG. 4(C), each pixel 410 has a microlens 411, a color filter 412, a photoelectric conversion unit 413A, and a photoelectric conversion unit 413B, and is fixed on a base 414. Light from the optical system 401 passes through the microlens 411. In the imaging unit 402, each pixel 410 is given red, green, or blue spectral characteristics corresponding to the wavelength band of the color filter 412. The photoelectric conversion units 413A and 413B each have sensitivity corresponding to the wavelength band, and are electrically connected to a substrate 414.

[0022] FIG. 4(D) is a diagram of the exit pupil viewed from the intersection of the optical axis of the optical system and the imaging unit. A light beam that has passed through a first pupil region 420, which is a part of the exit pupil, is incident on the photoelectric conversion unit 413A. A light beam that has passed through a second pupil region 430, which is a part of the exit pupil and different from the first pupil region 420, is incident on the photoelectric conversion unit 413B. The photoelectric conversion unit 413A photoelectrically converts the light beam that has passed through the first pupil region 420 and entered, and the photoelectric conversion unit 413B photoelectrically converts the light beam that has passed through the second pupil region 430 and entered. In this way, an A image and a B image are generated. As shown in FIG. 4(D), a center of gravity position 421 of the first pupil region 420 is decentered (shifted) from the center of the exit pupil in the x-axis direction, and a center of gravity position 431 of the second pupil region 430 is decentered on the opposite side to the center of gravity position 421. The distance between the center of gravity position 421 and the center of gravity position 431 is the base length L432, and the direction of the line segment is called the pupil division direction. The positions of the A image and the B image change in the same direction as the pupil division direction. The relative position change amount between these images, that is, the parallax between the A image and the B image, is a value according to the defocus amount. By detecting and converting this parallax using a known method, it is possible to obtain distance information such as the defocus amount and the distance from the imaging device 14 to the subject. In addition, an image obtained by adding the A image and the B image can be used as image information.

[0023] The imaging device 14 configured as above may be used to acquire the first feature amount. This allows distance measurement using, for example, the face image of the subject, thereby saving the user the trouble of using a measuring device. In addition, since such a distance measurement method detects parallax and measures distance, the reliability of the distance measurement value is high in areas with a lot of texture where parallax can be easily calculated. On the other hand, the reliability of the distance measurement value is low in areas where parallax is difficult to calculate. Therefore, the reliability can be quantitatively calculated by determining the density and contrast of the texture in the distance measurement area. For example, the reliability C can be calculated by the following formula (2).

number

[0024] lm is the average pixel value in a minute region of the image when performing a correlation calculation to obtain the parallax. ls is the standard deviation of lm. When acquiring the first feature amount, it is preferable to use the distance measurement value of a region with high reliability C. This allows a more appropriate correction process to be performed. When the reliability C is low, distance measurement may be performed at multiple points in the region and the average value may be used. This reduces distance measurement errors due to noise. In addition, the correction amount obtained by the correction process may be corrected depending on the reliability. This allows the influence of excessive correction of the shape when the reliability is low to be reduced. The imaging device 14 may be capable of imaging by stereoscopic vision. In this case, the same effect is also obtained.

[0025] <Second embodiment> Hereinafter, the second embodiment will be described with reference to FIG. 5 and FIG. 6. The differences from the above-mentioned embodiment will be mainly described, and the description of the similar matters will be omitted. This embodiment is the same as the first embodiment except that the deviation information regarding the deviation between the face image to be estimated for the estimated face shape and the face image learned by the trained model is different. As the deviation information in this embodiment, information regarding age (attribute) is used. Then, when there is a deviation, that is, when the age of the person of the face included in the face image to be estimated for the estimated face shape is significantly different from that of the person of the face included in the face image learned by the trained model, the correction process is performed. For example, when many of the people (attributes) of the face image learned by the trained model are adults who have completed the physical changes in adolescence, while the person of the face image to be estimated for the estimated face shape is a child before the occurrence of the physical changes, the correction process is performed. Conversely, when the person of the face image learned by the trained model is a child, while the person of the face image to be estimated for the estimated face shape is an adult, the correction process is also performed. The correction process is performed based on the first feature amount and the second feature amount.

[0026] The first feature amount in this embodiment is the distance between the eyes and the apex of the nose of the face included in the face image learned by the learned model, and the distance between the lower lip and the chin included in the face image. The second feature amount in this embodiment is the distance between the eyes and the apex of the nose of the estimated face shape, and the distance between the lower lip and the chin of the estimated face shape. Each distance will be described with reference to FIG. 5. FIG. 5 is a side view and a front view of a face in the second embodiment. Note that FIG. 5(A) is a side view, and FIG. 5(B) is a front view. In FIG. 5(A) and FIG. 5(B), the left-right direction of the face is the X-axis direction, the up-down direction of the face is the Y-axis direction, and the front-back direction of the face is the Z-axis direction. The distance between the eyes and the apex of the nose in each feature amount is the distance 501 between the eyes and the apex of the nose along the Z-axis direction in FIG. 5(A), and the distance between the lower lip and the chin is the distance 503 between the lower lip and the chin along the Y-axis direction in FIG. 5(B).

[0027] FIG. 6 is a flowchart showing a process executed by the image processing device. FIG. 6(A) is a flowchart showing a process of correcting the distance between the eyes and the apex of the nose (nose height) of an estimated face shape. As shown in FIG. 6(A), in step S601, the control unit 11 detects the nose area. The method of detecting the nose area is not particularly limited, and for example, a method using a new machine learning model that detects the organ area of ​​a face image by semantic segmentation can be used. In addition, a method of detecting the coordinates of the nose using a cascade type classifier using Haar-Like features and extracting the contour around it to detect the area can also be used. For example, when the position of the image and the shape correspond to each other, the nose area corresponds to the estimated shape as it is. The same is true for a machine learning model that learns the correspondence between the image and the shape using a UV map. In addition, in the case of a machine learning model that outputs the feature amount of the shape, the change in the feature amount corresponds to the movement of the point cloud, that is, the change in the shape, so that the area can be detected by grasping the point cloud corresponding to the nose area in advance.

[0028] In step S602, the control unit 11 executes a correction process based on the first feature amount and the second feature amount. In this embodiment, the control unit 11 executes the correction process based on the ratio between the distance (first feature amount) between the eyes and the apex of the nose of the face included in the face image learned by the learned model and the distance (second feature amount) between the eyes and the apex of the nose of the estimated face shape. Specifically, the correction process is executed using the following formula (3).

number

[0029] A1 is the first feature amount, and B1 is the second feature amount. Z1' is the Z-axis coordinate of the estimated face shape estimated by the trained model. Therefore, in step S602, the estimated face shape is corrected only in the Z-axis direction. This correction causes the estimated face shape to be enlarged or reduced in the Z-axis direction. Z2' is the Z-axis coordinate of the corrected estimated face shape. Control unit 11 stores this corrected estimated face shape in memory unit 13.

[0030] Fig. 6(B) is a flowchart showing a process for correcting the distance between the lower lip and the chin of the estimated face shape. As shown in Fig. 6(B), in step S603, the control unit 11 detects the area (coordinates) of the lower lip. There is no particular limitation on the method for detecting the area of ​​the lower lip, and for example, the same method as in step S601 of the flowchart shown in Fig. 6(A) can be used.

[0031] In step S604, the control unit 11 executes a correction process based on the first feature amount and the second feature amount. In this embodiment, the control unit 11 executes the correction process based on the ratio between the distance between the lower lip and the chin (first feature amount) included in the face image learned by the learned model and the distance between the lower lip and the chin (second feature amount) of the estimated face shape. Specifically, the correction process is executed using the following formula (4).

number

[0032] A2 is the first feature amount, and B2 is the second feature amount. Y1 is the Y-axis coordinate of the estimated face shape estimated by the trained model. Therefore, in step S602, the estimated face shape is corrected only in the Y-axis direction. This correction causes the estimated face shape to be enlarged or reduced in the Y-axis direction. Y2 is the Y-axis coordinate of the corrected estimated face shape. Control unit 11 stores this corrected estimated face shape in memory unit 13.

[0033] Here, let us consider a case where the correction process is omitted. For example, when most of the face images that are learning data in machine learning are adult face images, the face image that is the target of the estimation of the estimated face shape is a child's face image. In this case, the estimation result is output as data with facial features similar to those of a general adult, such as a high nose and a long chin. Conversely, when most of the face images that are learning data in machine learning are children's face images, the face image that is the target of the estimation of the estimated face shape is an adult's face image. In this case, the estimation result is output as data with facial features similar to those of a general child, such as a low nose and a short chin. Therefore, by performing the correction process, the estimated face shape is appropriately corrected so as to reduce the age-related discrepancy between the face image that is the target of the estimation of the estimated face shape and the face image learned by the learned model.

[0034] <Third embodiment> Hereinafter, the third embodiment will be described with reference to FIG. 7 to FIG. 9. The differences from the above-mentioned embodiment will be mainly described, and the description of the similar matters will be omitted. This embodiment is the same as the first embodiment except that the deviation information regarding the deviation between the face image to be estimated for the estimated face shape and the face image learned by the trained model is different. As the deviation information in this embodiment, information regarding gender (attribute) is used. Then, when there is a deviation, that is, when the gender of the person of the face included in the face image to be estimated for the estimated face shape is different from that of the person of the face included in the face image learned by the trained model is different from each other, the correction process is executed. For example, when most of the people (attributes) of the face image learned by the trained model are male, and on the other hand, the person of the face image to be estimated for the estimated face shape is female, the correction process is executed. Conversely, when the person of the face image learned by the trained model is female, and on the other hand, the person of the face image to be estimated for the estimated face shape is male, the correction process is also executed. The correction process is executed based on the first feature amount and the second feature amount.

[0035] The first feature amount in this embodiment is the distance between the eyebrows and the eyebrow ridge of the face included in the face image learned by the learned model. The second feature amount in this embodiment is the distance between the eyebrows and the eyebrow ridge of the estimated face shape. This distance will be described with reference to FIG. 7. FIG. 7 is a side view and a front view of the face in the third embodiment. Note that FIG. 7(A) is a side view, and FIG. 7(B) is a front view. In FIG. 7(A) and FIG. 7(B), the left-right direction of the face is the X-axis direction, the up-down direction of the face is the Y-axis direction, and the front-back direction of the face is the Z-axis direction. The distance between the eyebrows and the eyebrow ridge in each feature amount is the distance 701 between the eyebrows and the eyebrow ridge along the Z-axis direction in FIG. 7(A). Note that, as shown in FIG. 7(B), the eyebrows and the eyebrow ridge have the same coordinate in the X-axis direction. In addition, it is preferable to adopt a value close to the root of the eyebrow for the coordinate of the eyebrow in the X-axis direction, but it is not particularly limited as long as it is on the eye. In addition, it is preferable to adopt the average value in the Y-axis direction of the eyebrow area for the coordinate of the eyebrow in the Y-axis direction. 7B, it is preferable to adopt a predefined fixed value or a value that maximizes the coordinate in the Z-axis direction in the approximate eyebrow ridge area above the eyebrow. If it is difficult to acquire the first and second feature amounts, an area on the face where each feature amount can be acquired may be set.

[0036] Fig. 8 is a flowchart showing the process executed by the image processing device. As shown in Fig. 8, in step S801, the control unit 11 calculates the difference between the first feature amount and the second feature amount using the following formula (5).

number

[0037] In step S802, the control unit 11 corrects unevenness in the eyebrow arch region. In this embodiment, the control unit 11 executes the correction process based on the difference between the first feature amount and the second feature amount. Specifically, the correction process is executed using the following formula (6).

number

[0038] The correction amount ΔZxy is a relational expression between the maximum correction amount ΔZmax, the coordinate value x in the X-axis direction of the estimated face shape, and the coordinate value y in the Y-axis direction. Cx(x) and Cy(y) are correction coefficients corresponding to the coordinate values ​​x and y, respectively. Cx and Cy will be described with reference to FIG. 9. FIG. 9 is a diagram for explaining the estimated face shape. A graph 901 shown in FIG. 9(A) is a convex function. This graph 901 shows the shape of the eyebrow arch at a specified X coordinate, with a peak near the center, i.e., the value of Cy is a maximum value of 1, and both sides of the peak gradually decrease. The maximum value of 1 ensures a large correction amount ΔZxy. A graph 902 shown in FIG. 9(B) is a convex function. This graph 902 shows the shape of the left eyebrow arch at a specified Y coordinate, with a peak on the nose side (negative side in the X-axis direction), i.e., the value of Cx is a maximum value of 1, and gradually decreases toward the temporal side (positive side in the X-axis direction). Therefore, the correction amount ΔZxy becomes larger toward the nose side and becomes smaller toward the temporal side. Note that men's eyebrow ridges tend to be higher than women's eyebrow ridges. The graph showing the shape of the eyebrow ridges on the right side is a graph obtained by flipping the graph 902 from side to side. The control unit 11 stores the corrected face shape in the memory unit 13.

[0039] Here, let us consider a case where the correction process is omitted. For example, when most of the face images that are learning data in machine learning are male face images, the face image that is the target of the estimation of the estimated face shape is a female face image. In this case, the estimation result is output as data with facial features similar to those of a general man, such as relatively excessively raised eyebrows. Conversely, when most of the face images that are learning data in machine learning are female face images, the face image that is the target of the estimation of the estimated face shape is a male face image. In this case, the estimation result is output as data with facial features similar to those of a general woman, such as flat eyebrows. Therefore, by performing the correction process, the estimated face shape is appropriately corrected so as to reduce the age-related discrepancy between the face image that is the target of the estimation of the estimated face shape and the face image learned by the learned model.

[0040] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. The present invention can also be realized by a process in which a program that realizes one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors of a computer in the system or device read and execute the program. The present invention can also be realized by a circuit (for example, an ASIC) that realizes one or more functions. The deviation information is information about race in the first embodiment, information about age in the second embodiment, and information about sex in the third embodiment, but it is sufficient that at least one of these pieces of information is included. In addition, when the deviation information includes information about race, information about age, and information about sex, it is preferable to perform the correction process for the information that requires the greatest degree of correction in order. Specifically, it is preferable to perform the correction process for the information about race, the correction process for the information about age, and the correction process for the information about sex in that order. In addition, when the statistical values ​​of the database are classified by domains such as sex and age, appropriate statistical values ​​may be used based on the classification. For example, if the subject whose face shape is to be estimated is a man in his twenties, statistical values ​​for men in their twenties may be used for the first and second feature amounts.

[0041] The disclosure of each embodiment includes the following configurations, methods, and programs. (Configuration 1) An acquisition means for acquiring a face image including a face of a subject; an estimation means for estimating a face shape of the subject by using machine learning with the face image as input data, and acquiring the estimated face shape as output data; a correction means for correcting the estimated face shape, The facial shape correction device is characterized in that the correction means acquires a first feature amount necessary for correcting the estimated facial shape based on information about the data in the machine learning, acquires a second feature amount necessary for correcting the estimated facial shape based on information about the estimated facial shape, and corrects the estimated facial shape based on the first feature amount and the second feature amount. (Configuration 2) When the data in the machine learning is a face image, the first feature amount is a distance between the eye and the tip of the nose of the face included in the face image, 2. The face shape correcting device according to configuration 1, wherein the second feature amount is a distance between the eyes and the apex of the nose of the estimated face shape. (Configuration 3) The facial shape correcting device according to configuration 2, wherein the correction means corrects the estimated facial shape based on a ratio between the first feature amount and the second feature amount. (Configuration 4) The estimation means uses a trained model having a neural network for the machine learning, The face shape correction device according to configuration 2 or 3, wherein the correction means corrects the estimated face shape when there is a discrepancy in terms of race between the face image and a face image learned by the trained model. (Configuration 5) When the data in the machine learning is a face image, the first feature amount is a distance between the eyes and the tip of the nose of the face included in the face image and a distance between the lower lip and the chin included in the face image, The face shape correction device according to configuration 1, wherein the second feature amount is a distance between the eyes and the tip of the nose of the estimated face shape, and a distance between the lower lip and the chin of the estimated face shape. (Configuration 6) The facial shape correction device described in Configuration 5, wherein the correction means corrects the estimated facial shape based on a ratio of the distance between the eyes and the tip of the nose of the face included in the facial image that is the data from the machine learning to the distance between the eyes and the tip of the nose of the estimated facial shape, and a ratio of the distance between the lower lip and the chin included in the facial image that is the data from the machine learning to the distance between the lower lip and the chin of the estimated facial shape. (Configuration 7) The estimation means uses a trained model having a neural network for the machine learning, The face shape correction device according to configuration 5 or 6, wherein the correction means corrects the estimated face shape when there is a discrepancy in age between the face image and a face image learned by the trained model. (Configuration 8) When the data in the machine learning is a face image, the first feature is a distance between an eyebrow and an eyebrow ridge of the face included in the face image, 2. The face shape correcting device according to configuration 1, wherein the second feature amount is a distance between the eyebrows and the brow arch of the estimated face shape. (Configuration 9) The facial shape correcting device according to configuration 8, wherein the correction means corrects the estimated facial shape based on a difference between the first feature amount and the second feature amount. (Configuration 10) The estimation means uses a trained model having a neural network for the machine learning, The face shape correction device according to configuration 8 or 9, wherein the correction means corrects the estimated face shape when there is a discrepancy in gender between the face image and a face image learned by the trained model. (Configuration 11) A determination means for determining whether or not to correct the estimated face shape is provided, The facial shape correcting device according to any one of configurations 1 to 10, wherein the correction means corrects the estimated facial shape when it is determined as a result of the judgment by the judgment means that correction of the estimated facial shape should be performed. (Configuration 12) The estimation means uses a trained model having a neural network for the machine learning, The facial shape correction device according to configuration 11, wherein the judgment means judges whether or not to perform correction of the estimated facial shape based on deviation information regarding a deviation between the facial image and a facial image learned by the trained model. (Configuration 13) The face shape correcting device according to configuration 12, wherein the deviation information includes at least one of information related to race, information related to age, and information related to sex. (Configuration 14) The face shape correcting device according to any one of configurations 1 to 13, characterized in that the face shape correcting device is detachably connected to an imaging means capable of capturing the face image and measuring the distance thereto. (Method 1) A method for controlling a face shape correction device that corrects a face shape, comprising: An acquisition step of acquiring a face image including a face of a subject; an estimation step of estimating a face shape of the subject by using machine learning with the face image as input data, and acquiring the estimated face shape as output data; and a correction step of correcting the estimated face shape, a control method for a facial shape correction device, characterized in that in the correction step, a first feature amount required for correcting the estimated facial shape is obtained based on information about the data in the machine learning, and a second feature amount required for correcting the estimated facial shape is obtained based on information about the estimated facial shape, and the estimated facial shape is corrected based on the first feature amount and the second feature amount. (Program 1) A program for causing a computer to execute each means of the face shape correcting device according to any one of configurations 1 to 14. [Explanation of symbols]

[0042] 11 Control section 14 Imaging device (imaging means) 100 Image processing device

Claims

1. An acquisition means for acquiring a face image including a face of a subject; an estimation means for estimating a face shape of the subject by using machine learning with the face image as input data, and acquiring the estimated face shape as output data; a correction means for correcting the estimated face shape, The facial shape correction device is characterized in that the correction means acquires a first feature amount necessary for correcting the estimated facial shape based on information about the machine learning data, and acquires a second feature amount necessary for correcting the estimated facial shape based on information about the estimated facial shape, and corrects the estimated facial shape based on the first feature amount and the second feature amount.

2. When the data in the machine learning is a face image, the first feature amount is a distance between the eyes and the tip of the nose of the face included in the face image, 2. The facial shape correction device according to claim 1, wherein the second feature amount is a distance between the eyes and the apex of the nose of the estimated facial shape.

3. 3. The facial shape correcting device according to claim 2, wherein the correcting means corrects the estimated facial shape based on a ratio between the first feature amount and the second feature amount.

4. The estimation means uses a trained model having a neural network for the machine learning, The facial shape correction device according to claim 2 , wherein the correction means corrects the estimated facial shape when there is a discrepancy in terms of race between the facial image and a facial image learned by the trained model.

5. When the data in the machine learning is a face image, the first feature amount is a distance between the eyes and the tip of the nose of the face included in the face image and a distance between the lower lip and the chin included in the face image, 2 . The facial shape correcting device according to claim 1 , wherein the second feature amount is a distance between the eyes and the apex of the nose of the estimated facial shape, and a distance between the lower lip and the chin of the estimated facial shape.

6. The facial shape correction device according to claim 5, characterized in that the correction means corrects the estimated facial shape based on a ratio of the distance between the eyes and the tip of the nose of the face contained in the facial image that is the data from the machine learning to the distance between the eyes and the tip of the nose of the estimated facial shape, and a ratio of the distance between the lower lip and the chin contained in the facial image that is the data from the machine learning to the distance between the lower lip and the chin of the estimated facial shape.

7. The estimation means uses a trained model having a neural network for the machine learning, The facial shape correction device according to claim 5 , wherein the correction means corrects the estimated facial shape when there is a discrepancy in age between the facial image and a facial image learned by the trained model.

8. When the data in the machine learning is a face image, the first feature amount is a distance between an eyebrow and an eyebrow ridge of the face included in the face image, 2. The facial shape correcting device according to claim 1, wherein the second feature amount is a distance between an eyebrow and an eyebrow arch of the estimated facial shape.

9. 9. The facial shape correcting device according to claim 8, wherein the correcting means corrects the estimated facial shape based on a difference between the first feature amount and the second feature amount.

10. The estimation means uses a trained model having a neural network for the machine learning, The facial shape correction device according to claim 8 , wherein the correction means corrects the estimated facial shape when there is a discrepancy in gender between the facial image and a facial image learned by the trained model.

11. a determination means for determining whether or not to correct the estimated face shape, 2. The facial shape correcting device according to claim 1, wherein the correcting means corrects the estimated facial shape when the determining means determines that correction of the estimated facial shape should be executed.

12. The estimation means uses a trained model having a neural network for the machine learning, The facial shape correction device according to claim 11, characterized in that the determination means determines whether or not to perform correction of the estimated facial shape based on deviation information regarding a deviation between the facial image and a facial image learned by the trained model.

13. 13. The facial shape correcting device according to claim 12, wherein the deviation information includes at least one of information related to race, information related to age, and information related to sex.

14. 2. The facial shape correction device according to claim 1, wherein the facial shape correction device is detachably connected to an imaging means capable of capturing the facial image and measuring the distance thereto.

15. A method for controlling a face shape correction device that corrects a face shape, comprising: An acquisition step of acquiring a face image including a face of a subject; an estimation step of estimating a face shape of the subject by using machine learning with the face image as input data, and acquiring the estimated face shape as output data; and a correction step of correcting the estimated face shape, A method for controlling a facial shape correction device, characterized in that, in the correction process, a first feature amount required for correcting the estimated facial shape is obtained based on information about the data in the machine learning, and a second feature amount required for correcting the estimated facial shape is obtained based on information about the estimated facial shape, and the estimated facial shape is corrected based on the first feature amount and the second feature amount.

16. A program for causing a computer to execute each of the means of the facial shape correcting device according to claim 1.

Images