Image processing device, image processing method, and program

Abstract

To provide an image processing device that estimates spectral reflectance, enabling more accurate reproduction of skin color than conventional methods. [Solution] An image processing apparatus comprising: a skin pigment estimation unit that estimates the amount of melanin and hemoglobin components in the skin region of a person in a target image; a spectral reflectance estimation unit that estimates the spectral reflectance of the skin region based on the amount of melanin and hemoglobin components estimated by the skin pigment estimation unit; and a spectral reflectance correction unit that corrects the spectral reflectance estimated by the spectral reflectance estimation unit based on a generated image generated using the spectral reflectance estimated by the spectral reflectance estimation unit and the target image.

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program. [Background technology]

[0002] By using the spectral reflectance of skin, it is possible to reproduce skin images in various environments with different light sources and imaging devices. Non-patent document 1 discloses a technique for estimating the spectral reflectance of skin by extracting melanin and hemoglobin components, using a model of skin consisting of two layers, the epidermis and dermis. In this technique, a model following the Lambert-Beer law is used as the optical model of the epidermis, and a model following the Kubelka-Munk model is used as the optical model of the dermis. Patent document 1 also discloses a technique for evaluating the skin color distribution across multiple regions obtained by dividing a facial image, in which the amount of melanin and hemoglobin of the skin is extracted from the facial image using a relationship formula between pre-set amounts of melanin and hemoglobin and tristimulus values ​​X, Y, and Z. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2009-169758 [Non-patent literature]

[0004] [Non-Patent Document 1] Alotaibi, S. and Smith, WAP, A biophysical 3Dmorphable model of face appearance, Proceedings of the IEEE International Conference on Computer Vision Workshops (2017) [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] However, there is a problem that in an image generated based on the spectral reflectance of the skin estimated by Non-Patent Document 1, the color of the skin may be significantly different from the actual color.

[0006] The present invention has been made in view of such circumstances, and provides an image processing apparatus, an image processing method, and a program for estimating a spectral reflectance capable of more accurately reproducing the color of the skin than before.

Means for Solving the Problems

[0007] This invention has been made to solve the above-described problems. One aspect of the present invention includes a skin pigment estimation unit that estimates the amount of melanin component and the amount of hemoglobin component in the skin region of a person in a target image, and based on the amount of melanin component and the amount of hemoglobin component estimated by the skin pigment estimation unit, a spectral reflectance estimation unit that estimates the spectral reflectance of the region, a generated image generated using the spectral reflectance estimated by the spectral reflectance estimation unit, and a spectral reflectance correction unit that corrects the spectral reflectance estimated by the spectral reflectance estimation unit based on the generated image and the target image.

[0008] Also, another aspect of the present invention is the above-described image processing apparatus, wherein the spectral reflectance correction unit corrects the spectral reflectance estimated by the spectral reflectance estimation unit based on the ratio of the pixel values of the generated image and the target image and the camera sensitivity corresponding to the target image.

[0009] Also, another aspect of the present invention is the above-described image processing apparatus, wherein the spectral reflectance correction unit repeatedly corrects the corrected spectral reflectance based on a corrected generated image generated using the corrected spectral reflectance corrected by the spectral reflectance correction unit and the target image.

[0010] Another aspect of the present invention is an image processing method having: a first step of estimating the amount of melanin component and the amount of hemoglobin component in the skin area of a person in a target image; a second step of estimating the spectral reflectance of the area based on the amount of melanin component and the amount of hemoglobin component estimated in the first step; and a third step of correcting the spectral reflectance estimated in the second step based on a generated image generated using the spectral reflectance estimated in the second step and the target image.

[0011] Another aspect of the present invention is a program for causing a computer to function as a melanin pigment estimator that estimates the amount of melanin component and the amount of hemoglobin component in the skin area of a person in a target image, a spectral reflectance estimator that estimates the spectral reflectance of the area based on the amount of melanin component and the amount of hemoglobin component estimated by the melanin pigment estimator, and a spectral reflectance corrector that corrects the spectral reflectance estimated by the spectral reflectance estimator based on a generated image generated using the spectral reflectance estimated by the spectral reflectance estimator and the target image.

Advantages of the Invention

[0012] According to this invention, an image processing apparatus, an image processing method, and a program estimate a spectral reflectance that can reproduce the color of the skin more accurately than before.

Brief Description of the Drawings

[0013] [Figure 1] It is a schematic block diagram showing the configuration of an image processing system 10 according to an embodiment of this invention. [Figure 2] It is a schematic block diagram showing the functional configuration of an image processing apparatus 100 in the same embodiment. [Figure 3] It is a flowchart for explaining an operation example of an image processing apparatus 100 in the same embodiment. [Figure 4] It is a graph showing an example of the repeated result of a spectral reflectance corrector 107 in the same embodiment. [Figure 5]This is an example of a target image in the same embodiment. [Figure 6] This is an example of an image generated in the same embodiment. [Figure 7] This is an example of a corrected image generated in the same embodiment. [Modes for carrying out the invention]

[0014] Embodiments of the present invention will be described below with reference to the drawings. Figure 1 is a schematic block diagram showing the configuration of an image processing system 10 according to one embodiment of the present invention. The image processing system 10 captures an image (target image) that includes a person's skin, estimates the spectral reflectance of the skin contained in the target image, and displays an image generated using the spectral reflectance. Note that the target image may not be a captured image, but rather computer graphics (CG) generated by a method such as ray tracing.

[0015] As shown in Figure 1, the image processing system 10 comprises an image processing device 100, a shooting device 200, and a display device 300. The image processing device 100 estimates the spectral reflectance of skin contained in a target image captured by the shooting device 200 and generates an image using this spectral reflectance. The generated image may be an image based on a different environment than the target image (for example, a different light source or shooting device). The image processing device 100 may be implemented by one or more computers reading and executing a program, or at least a part of it may be located on a so-called cloud. The shooting device 200 is a shooting device such as a digital camera that captures the target image. The display device 300 includes display means such as a liquid crystal display or an organic EL (Electro Luminescence) display and displays the image generated by the image processing device 100.

[0016] Figure 2 is a schematic block diagram showing the functional configuration of the image processing apparatus 100 in this embodiment. The image processing apparatus 100 includes a data input unit 101, a target image storage unit 102, a skin pigment estimation unit 103, a skin pigment information storage unit 104, a spectral reflectance estimation unit 105, a spectral reflectance storage unit 106, a spectral reflectance correction unit 107, and an output unit 108.

[0017] The data input unit 101 receives the target image captured by the imaging device 200 and stores it in the target image storage unit 102. The data input unit 101 may be connected to the imaging device 200 via an IP (Internet Protocol) network, a USB (Universal Serial Bus) cable, etc., and receive the target image from the imaging device 200, or it may acquire the target image from the imaging device 200 via a recording medium such as a USB memory or memory card. The data input unit 101 also receives the camera sensitivity corresponding to the target image and the spectral distribution of ambient light corresponding to the target image and stores them in the target image storage unit 102. The camera sensitivity corresponding to the target image may be the camera sensitivity of the imaging device 200 that captured the target image. The spectral distribution of ambient light corresponding to the target image may be the spectral distribution of ambient light at the time the target image was captured. The data input unit 101 may also receive or acquire the camera sensitivity and spectral distribution of ambient light from the imaging device 200 or other devices, similar to the target image.

[0018] Furthermore, the camera sensitivity corresponding to the target image may be a camera sensitivity specified by the user. The spectral distribution of ambient light corresponding to the target image may also be a spectral distribution of ambient light specified by the user. The data input unit 101 may be equipped with an input interface such as a keyboard, mouse, or touch panel, and may acquire these by reading the camera sensitivity and spectral distribution of ambient light stored in the image processing device 100 from those specified by the user using the input interface.

[0019] The target image storage unit 102 stores the target image, camera sensitivity, and spectral distribution of ambient light.

[0020] The melanin pigment estimation unit 103 estimates the amount of melanin component and the amount of hemoglobin component in the skin area of a person in the target image stored in the target image storage unit 102, and stores the estimated amount of melanin component and the amount of hemoglobin component in the melanin pigment information storage unit 104. Note that the melanin pigment estimation unit 103 may estimate the amount of melanin component and the amount of hemoglobin component for each pixel, or may estimate them for each region obtained by dividing the skin area. For the estimation of the amount of melanin component and the amount of hemoglobin component, the method described in Non-Patent Document 1 may be used, or other methods may be used. In the method described in Non-Patent Document 1, the shooting of the target image by the imaging device 200 is modeled as in Equation (1), and the spectral reflectance of the skin is modeled as in Equation (2).

[0021] [Number]

[0022] In Equation (1), i C is the RGB intensity (C ∈ {r, g, b}), that is, the pixel value of each of the RGB of the target image. λ is the wavelength of light. R(λ) is the spectral reflectance of the skin. S C (λ) is the camera sensitivity for each of R, G, and B. E(λ) is the spectral distribution of the ambient light. i C , S C (λ), and E(λ), the target image stored in the target image storage unit 102, the camera sensitivity, and the spectral distribution of the ambient light are used.

[0023] [Number]

[0024] Equation (2) defines the skin by a two-layer model of the first-layer epidermis and the second-layer dermis. The first-layer epidermis follows the Lambert-Beer law, and the second-layer dermis follows the Kubelka-Munk model. In Equation (2), f mel is the amount of melanin component, and f blood is the amount of hemoglobin component. R total (f mel , f bloodλ) is the spectral reflectance of the skin, and is R(λ) in equation (1). epidermal (f mel λ) is the permeability of the epidermis, and R dermal (f blood λ) is the reflectance of the dermis.

[0025] The skin pigment information storage unit 104 stores the amount of melanin and hemoglobin components estimated by the skin pigment estimation unit 103.

[0026] The spectral reflectance estimation unit 105 estimates the spectral reflectance of the skin region based on the melanin and hemoglobin component amounts estimated by the skin pigment estimation unit 103, and stores it in the spectral reflectance storage unit 106. The above-mentioned equation (2) may be used for this estimation. The spectral reflectance estimation unit 105 uses the melanin and hemoglobin component amounts stored in the skin pigment information storage unit 104 as the melanin and hemoglobin component amounts estimated by the skin pigment estimation unit 103. The spectral reflectance estimation unit 105 may estimate the spectral reflectance for each pixel, or for each region into which the skin region has been divided.

[0027] The spectral reflectance storage unit 106 stores the spectral reflectance of the skin region estimated by the spectral reflectance estimation unit 105, and the spectral reflectance of the skin region corrected by the spectral reflectance correction unit 107.

[0028] The spectral reflectance correction unit 107 corrects the spectral reflectance estimated by the spectral reflectance estimation unit 105 based on the generated image, which is generated using the spectral reflectance estimated by the spectral reflectance estimation unit 105, and the target image. The spectral reflectance correction unit 107 stores the corrected spectral reflectance in the spectral reflectance storage unit 106. The spectral reflectance correction unit 107 may also correct the spectral reflectance estimated by the spectral reflectance estimation unit 105 based on the ratio of the pixel values ​​of the generated image and the target image, and the camera sensitivity corresponding to the target image. Furthermore, the spectral reflectance correction unit 107 may repeatedly correct the corrected spectral reflectance based on the corrected generated image, which is generated using the corrected spectral reflectance corrected by the spectral reflectance correction unit 107, and the target image. The spectral reflectance estimation unit 105 uses the spectral reflectance stored in the spectral reflectance storage unit 106 as the spectral reflectance estimated by the spectral reflectance estimation unit 105. Furthermore, the spectral reflectance estimation unit 105 uses the target image stored in the target image storage unit 102 as the target image.

[0029] The spectral reflectance correction unit 107 may perform spectral reflectance correction using equations (3) to (9). First, the spectral reflectance correction unit 107 generates a generated image in which the skin region of the target image is replaced with pixel values ​​generated using the spectral reflectance estimated by the spectral reflectance estimation unit 105. The spectral reflectance correction unit 107 then sets the pixel values ​​of the target image to i gt-r i gt-g i gt-b The pixel values ​​of the generated image are i pre-r i pre-g i pre-b The ratio of the RGB pixel values ​​of the target image and the generated image is k. r , k g , k b This is calculated using equations (3) through (5).

[0030]

number

[0031] Furthermore, the spectral reflectance correction unit 107 adjusts the camera sensitivity S for each of the RGB channels. r (λ), S g (λ), S bUsing (λ), the weights ω for each RGB r (λ), ω g (λ), ω b (λ) is calculated from equation (6) to equation (8). Note that the spectral reflectance correction unit 107 is set to camera sensitivity S r (λ), S g (λ), S b As (λ), the camera sensitivity corresponding to the target image stored in the target image storage unit 102 is used.

[0032]

number

[0033] Then, the spectral reflectance correction unit 107 calculates the spectral reflectance R estimated by the spectral reflectance estimation unit 105 using equation (9). reconst(n-1) Correcting (λ), R reconst(n) (λ) is calculated. In this way, the weight (ω) corresponds to the camera sensitivity. r (λ), ω g (λ), ω b (λ) is the ratio (k) of the RGB pixel values ​​of the target image and the generated image. r , k g , k b By correcting the spectral reflectance using a coefficient obtained by multiplying it by (), the spectral reflectance can be corrected more significantly in the wavelength range where the camera sensitivity is high, enabling more efficient correction.

[0034]

number

[0035] Furthermore, when the spectral reflectance correction unit 107 repeatedly corrects the spectral reflectance, i pre-r i pre-g i pre-b Using the pixel values ​​of the corrected image generated with the spectral reflectance from the previous correction, kr, kg, and kb are calculated using equations (3) to (5), and R reconst(n) (λ) is the spectral reflectance corrected in the previous attempt, and the spectral reflectance R corrected in the current attempt. reconst(n+1)The calculation of (λ) using equation (9) is repeated. The spectral reflectance correction unit 107 may repeat this process until the PSNR (Peak Signal to Noise Ratio) value of the difference (or ratio) of pixel values ​​between the corrected generated image, which is generated using the spectral reflectance corrected this time, and the target image exceeds a threshold, or it may repeat a predetermined number of times, or it may repeat until other conditions are met.

[0036] The output unit 108 displays the corrected generated image on the display device 300, in which the skin area of ​​the target image is replaced with pixel values ​​generated using the corrected spectral reflectance stored in the spectral reflectance storage unit 106. The output unit 108 may also output the corrected spectral reflectance stored in the spectral reflectance storage unit 106, or it may output another image generated using the corrected spectral reflectance.

[0037] Figure 3 is a flowchart illustrating an example of the operation of the image processing device 100 in this embodiment. First, the data input unit 101 receives the target image as input (step S1). Camera sensitivity and the spectral distribution of ambient light may also be input along with the target image. Next, the skin pigment estimation unit 103 estimates the skin pigment (amount of melanin component, amount of hemoglobin component) (step S2). Next, the spectral reflectance estimation unit 105 estimates the spectral reflectance of the skin (step S3).

[0038] Next, the spectral reflectance correction unit 107 reconstructs the skin region of the target image using the spectral reflectance estimated in step S3 to generate a generated image (step S4). Next, the spectral reflectance correction unit 107 calculates a coefficient (for example, k) that indicates the ratio between the target image and the generated image. r , k g , k bStep S5 calculates the PSNR value of the difference between the target image and the generated image. Next, Step S6 calculates the PSNR value of the difference between the target image and the generated image. Next, Step S7 determines whether the PSNR value calculated in Step S6 is equal to or greater than a predetermined value (threshold). If it is equal to or greater than the threshold (Step S7-YES), Step S9 displays the generated image (or corrected generated image) generated in Step S4 on the display device 300.

[0039] On the other hand, if the value is not above the threshold (step S7-NO), the spectral reflectance correction unit 107 corrects the spectral reflectance using the coefficient calculated in step S5 (step S8), and returns to step S4. Thereafter, the spectral reflectance correction unit 107 uses the spectral reflectance corrected by the previous correction as the spectral reflectance, and uses the corrected generated image generated using the spectral reflectance corrected by the previous correction as the generated image, and repeats steps S4 to S8 until the PSNR value in step S7 is above the threshold.

[0040] Figure 4 is a graph showing an example of the iteration results of the spectral reflectance correction unit 107 in this embodiment. In the graph in Figure 4, the horizontal axis is the number of iterations, and the vertical axis is the PSNR value. The graph in Figure 4 shows an example of the change in the PSNR value when the spectral reflectance correction unit 107 repeatedly corrects the spectral reflectance using equations (3) to (9). As shown in Figure 4, the PSNR value increases with each iteration of the spectral reflectance correction by the spectral reflectance correction unit 107, and it can be seen that the corrected generated image approaches the target image.

[0041] Figures 5 to 7 show examples of the target image, generated image, and corrected generated image in this embodiment. The generated image in Figure 6 is an image using the spectral reflectance estimated by the spectral reflectance estimation unit 105. Comparing the generated image in Figure 6 with the target image in Figure 5, it can be seen that there is a difference in the color of the skin area. The corrected generated image in Figure 7 is an image using the corrected spectral reflectance obtained by repeating the correction by the spectral reflectance correction unit 107 until the PSNR value is above a threshold. Comparing the corrected generated image in Figure 7 with the target image in Figure 5, it can be seen that there is almost no difference in the color of the skin area.

[0042] The target image storage unit 102, skin pigment information storage unit 104, and spectral reflectance storage unit 106 shall be composed of non-volatile memory such as hard disk drives, SSDs (Solid State Drives), magneto-optical disk drives, and flash memory, or read-only storage media such as CR-ROMs, or volatile memory such as RAM (Random Access Memory), or a combination thereof.

[0043] In addition, in the image processing system 10 of the above embodiment, a CG generation device may be provided instead of the shooting device 200, and the target image may be a CG generated by the CG generation device. In that case, the camera sensitivity and the spectral distribution of ambient light may be the same as those corresponding to the CG. The camera sensitivity corresponding to the CG may be the sensitivity of the human eye. The spectral distribution of ambient light corresponding to the CG may be the spectral distribution of ambient light assumed when generating the CG. Furthermore, even in the case of a target image captured by the shooting device 200, the sensitivity of the human eye may be used as the camera sensitivity corresponding to the target image.

[0044] This disclosure may also be in the following embodiments. (1) One embodiment of the present disclosure is an image processing apparatus comprising: a skin pigment estimation unit that estimates the amount of melanin and hemoglobin in a region of a person's skin in a target image; a spectral reflectance estimation unit that estimates the spectral reflectance of the region based on the amount of melanin and hemoglobin estimated by the skin pigment estimation unit; and a spectral reflectance correction unit that corrects the spectral reflectance estimated by the spectral reflectance estimation unit based on a generated image generated using the spectral reflectance estimated by the spectral reflectance estimation unit and the target image.

[0045] This allows the image processing device to estimate the spectral reflectance, which can reproduce skin color more accurately than before.

[0046] (2) Another embodiment of the present disclosure is an image processing apparatus as described in (1), wherein the spectral reflectance correction unit corrects the spectral reflectance estimated by the spectral reflectance estimation unit based on the ratio of the pixel values ​​of the generated image and the target image and the camera sensitivity corresponding to the target image.

[0047] This allows the image processing device to correct the spectral reflectance according to the camera sensitivity in each wavelength range.

[0048] (3) Another embodiment of the present disclosure is an image processing apparatus as described in (1) or (2), wherein the spectral reflectance correction unit repeatedly corrects the corrected spectral reflectance based on the corrected generated image generated using the corrected spectral reflectance corrected by the spectral reflectance correction unit and the target image.

[0049] This allows the image processing device to correct the spectral reflectance so that it can reproduce the skin tone in the target image.

[0050] (4) Another embodiment of the present disclosure is an image processing method comprising: a first step of estimating the amount of melanin and hemoglobin in a region of a person's skin in a target image; a second step of estimating the spectral reflectance of the region based on the amount of melanin and hemoglobin estimated in the first step; and a third step of correcting the spectral reflectance estimated in the second step based on a generated image generated using the spectral reflectance estimated in the second step and the target image.

[0051] This allows the image processing method to estimate spectral reflectance, which can reproduce skin color more accurately than before.

[0052] (5) Another embodiment of the present disclosure is a program for causing a computer to function as: a skin pigment estimation unit that estimates the amount of melanin and hemoglobin components in a region of a person's skin in a target image; a spectral reflectance estimation unit that estimates the spectral reflectance of the region based on the amount of melanin and hemoglobin components estimated by the skin pigment estimation unit; and a spectral reflectance correction unit that corrects the spectral reflectance estimated by the spectral reflectance estimation unit based on a generated image generated using the spectral reflectance estimated by the spectral reflectance estimation unit and the target image.

[0053] This allows the computer that loads and runs the program to estimate the spectral reflectance, which can reproduce skin color more accurately than before.

[0054] Alternatively, the image processing device 100 may be realized by recording a program for realizing the functions of the image processing device 100 in Figure 1 onto a computer-readable recording medium, and then loading and executing the program recorded on this recording medium into a computer system. The term "computer system" here includes hardware such as the operating system and peripheral devices.

[0055] Furthermore, "computer system" shall also include the homepage provisioning environment (or display environment) if a WWW system is being used. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Moreover, "computer-readable recording media" also includes those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs over networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside computer systems that act as servers or clients in such cases. In addition, the above-mentioned programs may be for the purpose of realizing some of the functions described above, and may also be able to realize the above-mentioned functions in combination with programs already recorded in the computer system.

[0056] While embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and may include design modifications and the like that do not depart from the spirit of this invention. [Explanation of Symbols]

[0057] 10 Image Processing Systems 100 Image Processing Devices 101 Data Input Section 102 Target image storage unit 103 Skin pigment estimation section 104 Skin pigment information storage section 105 Spectral reflectance estimation section 106 Spectral reflectance storage section 107 Spectral reflectance correction section 108 Output section 200 imaging device 300 display device

Claims

1. A skin pigment estimation unit that estimates the amount of melanin and hemoglobin components in the skin area of ​​a person in the target image, A spectral reflectance estimation unit estimates the spectral reflectance of the region based on the amount of melanin and the amount of hemoglobin estimated by the skin pigment estimation unit, A spectral reflectance correction unit corrects the spectral reflectance estimated by the spectral reflectance estimation unit based on the generated image generated using the spectral reflectance estimated by the spectral reflectance estimation unit and the target image. An image processing device equipped with the following features.

2. The image processing apparatus according to claim 1, wherein the spectral reflectance correction unit corrects the spectral reflectance estimated by the spectral reflectance estimation unit based on the ratio of the pixel values ​​of the generated image and the target image and the camera sensitivity corresponding to the target image.

3. The image processing apparatus according to claim 1 or 2, wherein the spectral reflectance correction unit repeatedly corrects the corrected spectral reflectance based on the corrected generated image, which is generated using the corrected spectral reflectance corrected by the spectral reflectance correction unit, and the target image.

4. The first step is to estimate the amount of melanin and hemoglobin in the skin region of a person in the target image, A second step is to estimate the spectral reflectance of the region based on the amount of melanin component and the amount of hemoglobin component estimated in the first step, A third step in which the spectral reflectance estimated in the second step is corrected based on the generated image generated using the spectral reflectance estimated in the second step and the target image. An image processing method having the following characteristics.

5. Computers, A skin pigment estimation unit that estimates the amount of melanin and hemoglobin components in the skin area of ​​a person in the target image. A spectral reflectance estimation unit estimates the spectral reflectance of the region based on the amount of melanin and the amount of hemoglobin estimated by the skin pigment estimation unit. A spectral reflectance correction unit corrects the spectral reflectance estimated by the spectral reflectance estimation unit based on the generated image, which is generated using the spectral reflectance estimated by the spectral reflectance estimation unit, and the target image. A program designed to function as such.

Images