Image processing device and image processing method

Abstract

To enable separation of an input image into an object color image and a shadow image with high accuracy.SOLUTION: An image processing device includes an object color estimation section that estimates an object color image having a color component of an object included in an input image as a pixel value on the basis of a feature amount of the input image, and a shadow estimation section that estimates a shadow image having a shadow component of the input image as a pixel value on the basis of the feature amount of the input image. The shadow estimation section estimates the shadow image by limiting a color space that can be taken by the shadow component of the input image to a color space determined under a predetermined color condition. This technique can be applied to, for example, the image processing device or the like that separates an input image into an object color image and a shadow image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device and an image processing method, and more particularly to an image processing device and an image processing method that enable an input image to be separated into an object color image and a shade image with high accuracy. [Background technology]

[0002] There is an intrinsic image decomposition technique that separates an input image into an object color image (also called a reflectance image, etc.) and a shaded image. For example, Non-Patent Document 1 discloses a technique that estimates an object color image and a shaded image using a convolutional neural network (hereinafter referred to as CNN). Non-Patent Document 2 discloses a technique that separates an input image into an object color image and a shaded image by estimating the shaded image in grayscale.

[0003] Patent Document 1 discloses a method for improving image quality by estimating the perfect diffuse component of an image, performing further correction to obtain object color, and then applying the same estimated shadows and specular to the corrected object color. In Patent Document 1, since the perfect diffuse component includes a light source color component, spectral information of the subject is estimated based on the CIE daylight model or the like, and then the influence of the light source color is removed from the perfect diffuse component to restore the object color. Furthermore, as in Patent Document 2, there is also a method for improving the compression rate in video encoding by separating a video of object color images from a video of shadow images. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6004481 [Patent Document 2] U.S. Patent No. 10,659,787 [Non-patent literature]

[0005] [Non-Patent Document 1] Narihira et al., Direct Intrinsics: Learning Albedo-Shading Decomposition by Convolutional Regression, CVPR 2015 [Non-patent document 2] Fan et al., Revisiting Deep Intrinsic Image Decompositions, CVPR 2018 Summary of the Invention [Problem to be solved by the invention]

[0006] Eigenimage decomposition is essentially an ill-posed problem of finding two variables, object color and shading, from a single input image, and the solution is not uniquely determined. For this reason, Patent Document 1 assumes that the target object is a face and that the color of the target object is generally known, while Non-Patent Document 2 estimates a shading image in grayscale by assuming only white light. There is still room for improvement in the technology for separating an input image into an object color image and a shading image, and a technology for separating object color images and shading images with high accuracy is desired.

[0007] The present disclosure has been made in view of the above circumstances, and aims to enable an input image to be separated into an object color image and a shade image with high accuracy. [Means for solving the problem]

[0008] An image processing device according to one aspect of the present disclosure includes an object color estimation unit that estimates an object color image having color components of objects included in an input image as pixel values ​​based on feature amounts of the input image, and a shade estimation unit that estimates a shade image having shade components of the input image as pixel values ​​based on the feature amounts of the input image, and the shade estimation unit estimates the shade image by limiting the color space that can be taken by the shade components of the input image to a color space determined by predetermined color conditions.

[0009] An image processing method according to one aspect of the present disclosure includes an image processing device that estimates an object color image having pixel values ​​that represent color components of objects included in an input image based on features of the input image, and estimates a shading image having pixel values ​​that represent shading components of the input image based on the features of the input image, and the shading image is estimated by restricting the color space that the shading components of the input image can take to a color space determined by predetermined color conditions.

[0010] In one aspect of the present disclosure, an object color image having pixel values ​​representing color components of objects included in an input image is estimated based on feature amounts of the input image, and a shade image having pixel values ​​representing shade components of the input image is estimated based on the feature amounts of the input image. The shade image is estimated by restricting a color space that the shade components of the input image can take to a color space determined by predetermined color conditions.

[0011] The image processing device according to one aspect of the present disclosure can be realized by causing a computer to execute a program. The program to be executed by the computer can be provided by transmitting it via a transmission medium or by recording it on a recording medium.

[0012] The image processing device may be an independent device or an internal block constituting a single device. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram showing a configuration example of a first embodiment of an image processing device of the present disclosure. [Figure 2] FIG. 1 is a diagram showing the blackbody radiation locus on the xy chromaticity diagram. [Figure 3] FIG. 2 is a block diagram showing a first example of the configuration of the shadow estimation unit in FIG. 1. [Figure 4] FIG. 10 is a block diagram showing an example of the configuration of a shadow estimation unit according to a first example of the configuration when N=2. [Figure 5] FIG. 1 is a diagram illustrating the configuration of a CNN predictor. [Figure 6] FIG. 10 is a diagram illustrating a setting example when N=3. [Figure 7] 10A to 10C are diagrams illustrating another example of specifying a base color in the shade estimation unit according to the first configuration example. [Figure 8] 10A to 10C are diagrams illustrating another example of specifying a base color in the shade estimation unit according to the first configuration example. [Figure 9] FIG. 1 is a diagram illustrating a method for setting a color space expressed by N base colors. [Figure 10] FIG. 1 is a diagram illustrating the basis functions of the CIE daylight model. [Figure 11] FIG. 2 is a block diagram showing a second configuration example of the shadow estimation unit in FIG. 1. [Figure 12] FIG. 2 is a block diagram showing a third example configuration of the shadow estimation unit in FIG. 1. [Figure 13] FIG. 1 is a diagram illustrating a model of direct light and global light. [Figure 14] FIG. 2 is a diagram illustrating an example of the configuration of teacher data. [Figure 15] 10 is a flowchart illustrating an object color shade separation process performed by the image processing device according to the first embodiment. [Figure 16] FIG. 10 is a block diagram showing a configuration example of a second embodiment of an image processing device according to the present disclosure. [Figure 17] 10 is a flowchart illustrating an object color shade separation process performed by an image processing device according to a second embodiment. [Figure 18] FIG. 10 is a diagram illustrating an example of a termination condition. [Figure 19] FIG. 10 is a block diagram showing a configuration example of a third embodiment of an image processing device according to the present disclosure. [Figure 20] 11 is a flowchart illustrating an object color shade separation process performed by an image processing device according to a third embodiment. [Figure 21] FIG. 11 is a block diagram showing an example of the configuration of an image processing device according to a third embodiment with a cascade structure. [Figure 22] FIG. 1 is a block diagram showing an example of the configuration of a learning device that learns parameters of a CNN predictor. [Figure 23] FIG. 10 is a block diagram showing a configuration example of a fourth embodiment of an image processing device according to the present disclosure. [Figure 24] FIG. 24 is a block diagram showing a detailed configuration example of a shadow estimation unit in FIG. 23. [Figure 25] 24 is a diagram illustrating the processing of the shadow estimation unit in FIG. 23. FIG. [Figure 26] FIG. 1 is a block diagram illustrating an example configuration of an embodiment of a computer to which the technology of the present disclosure is applied. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, with reference to the accompanying drawings, a description will be given of an embodiment of the present technology. In this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted. The description will be given in the following order. 1. First embodiment of image processing device 2. First example of the shadow estimation unit 3. Second configuration example of the shadow estimation unit 4. Third example of the shadow estimation unit 5. Example of training data configuration 6. Object Color and Shade Separation Processing in the First Embodiment 7. Second embodiment of image processing device 8. Object Color and Shade Separation Processing in the Second Embodiment 9. Third embodiment of image processing device 10. Object color and shade separation processing in the third embodiment 11. Example of learning device configuration 12. Fourth embodiment of image processing device 13. Computer Configuration Examples

[0015] <1. First embodiment of image processing device> FIG. 1 is a block diagram showing an example configuration of a first embodiment of an image processing device of the present disclosure.

[0016] The image processing device 1 in Fig. 1 is a device that separates an input color image into an object color image and a shaded image and outputs them. The object color image is an image having the color components (reflectance components) of the object as pixel values, and the shaded image is an image having the shade components due to a light source (illumination) or the like as pixel values.

[0017] The image processing device 1 includes a feature extraction unit 11, an object color estimation unit 12, a shade estimation unit 13, and a color condition setting unit .

[0018] The feature extraction unit 11 extracts feature amounts of the input image and supplies them to the object color estimation unit 12 and the shade estimation unit 13. The object color estimation unit 12 estimates and outputs an object color image having pixel values ​​that represent color components (reflectance components) of objects included in the input image, based on the feature amounts of the input image supplied from the feature extraction unit 11. The shade estimation unit 13 estimates and outputs a shade image having pixel values ​​that represent shade components of the input image, based on the feature amounts of the input image supplied from the feature extraction unit 11. The color condition setting unit 14 sets color conditions for estimating the shade image and supplies them to the shade estimation unit 13. The color conditions are set, for example, according to color conditions specified by the user.

[0019] The image processing device 1 is characterized in that the color space, which is the solution space that the shadow component can take, is limited from all RGB spaces to a color space determined by predetermined color conditions in order to estimate a shadow image in the shadow estimation unit 13. For example, since the light source color in the natural world often follows blackbody radiation, the color condition setting unit 14 imposes a constraint that the estimated shadow color be close to the blackbody radiation color as a color condition.

[0020] FIG. 2 shows an xy chromaticity diagram in which the color of light is represented by plane coordinates (x, y).

[0021] The color of blackbody radiation is expressed as a curved blackbody radiation locus on an xy chromaticity diagram, as shown in Figure 2. According to the non-patent document "Design of advanced color: Temperature control system for HDTV applications, Journal of the Korean Physical Society," the blackbody radiation locus can be approximated by a cubic spline.

[0022] The image processing device 1 assumes that the illumination color of a scene observed in an input image is expressed as a linear sum of several base colors (hereinafter also referred to as basis colors), and estimates this sum as the final shaded image. If the number of bases (base number) N of a color space expressed by multiple base colors is 2, and the bases are color temperatures T1 = 3000 Kelvin and T2 = 8000 Kelvin, the color space that can be taken by a shaded image obtained by adding together the shaded images expressed by each base color is expressed by a line segment with endpoints at the two base colors of color temperatures T1 = 3000 and T2 = 8000, as shown in FIG. 2. That is, when the base number N = 2, the colors that appear in the shaded image always exist on a straight line connecting the points representing the two base colors on the xy chromaticity diagram. When the base number N > 2, the colors exist inside an N-gon on the xy chromaticity diagram. In this way, by making the color space expressed by multiple base colors a space that approximates the color space that represents the blackbody radiation color, the color of the shadow image can be stably estimated, and high-precision separation of the object color image and the shadow image from the input image can be achieved.

[0023] <2. First Configuration Example of Shadow Estimation Unit> FIG. 3 is a block diagram showing a first configuration example of the shadow estimation unit 13. As shown in FIG.

[0024] The shadow estimation unit 13 according to the first configuration example includes a first shadow image generation unit 31-1 to an N-th shadow image generation unit 31-N, and a shadow synthesis unit 32. N, which corresponds to the number of shadow image generation units 31, is an integer greater than 1 and corresponds to the base number that defines the color space of the shadow image.

[0025] Although the first shaded image generating unit 31-1 through the Nth shaded image generating unit 31-N each receive different base colors as color conditions from the color condition setting unit 14, they have the same configuration of the shaded image generating unit 31 and perform the same processing using the supplied base colors. The shaded image generating unit 31 will be described in detail later with reference to FIG. 4, but the shaded image generating unit 31 includes a shaded intensity estimating unit 41, a color parameter converting unit 42, and a multiplying unit 43, and generates a shaded image corresponding to a predetermined base color.

[0026] The first shaded image generating unit 31-1 generates a first shaded image (shaded image of a first color) corresponding to a first base color supplied from the color condition setting unit 14. The second shaded image generating unit 31-2 (not shown) generates a second shaded image (shaded image of a second color) corresponding to a second base color supplied from the color condition setting unit 14. Similarly, the Nth shaded image generating unit 31-N generates an Nth shaded image (shaded image of an Nth color) corresponding to the Nth base color supplied from the color condition setting unit 14. Each of the first color shaded image to the Nth color shaded image is a shaded image of three channels: R, G, and B.

[0027] The shade synthesis unit 32 generates (estimates) and outputs a shade image by synthesizing the shade image of the first color through the shade image of the Nth color supplied from the first shade image generation unit 31-1 through the Nth shade image generation unit 31-N, respectively. Specifically, the synthesis of the shade images is performed by linearly adding corresponding pixels of the shade image of the first color through the shade image of the Nth color for each of the R, G, and B channels. The output shade image is expressed in the colors of the three R, G, and B channels.

[0028] FIG. 4 shows an example of the configuration of the shadow estimation unit 13 according to the first example of the configuration in which the number of bases N is set to 2 (N=2) as a method of selecting the bases, and a color temperature T1=3000 Kelvin as a first base and a color temperature T2=8000 Kelvin as a second base are supplied from the color condition setting unit 14 as shown in FIG.

[0029] The first shade image generation unit 31-1 has a first shade intensity estimation unit 41-1, a first color parameter conversion unit 42-1, and a multiplication unit 43-1. The first shade intensity estimation unit 41-1 is supplied with the feature amount of the input image extracted by the feature amount extraction unit 11. The first color parameter conversion unit 42-1 is supplied with a color temperature T1=3000 Kelvin as a first base color condition from the color condition setting unit 14.

[0030] The first shade intensity estimation unit 41-1 estimates an intensity image of one channel of shade components (hereinafter referred to as a shade intensity image) corresponding to the base color supplied from the color condition setting unit 14, based on the feature amounts of the input image extracted by the feature extraction unit 11. The estimated shade intensity image is supplied to a multiplication unit 43-1.

[0031] The first color parameter conversion unit 42-1 converts the color temperature T1=3000 supplied from the color condition setting unit 14 into first color parameters. The first color parameters are color parameters [g R (T1),g G (T1),g B (T1)]. The three-channel color parameters [g R (T1),g G (T1),g B (T1)] is supplied to the multiplication unit 43-1.

[0032] The multiplication unit 43-1 multiplies the one-channel shading intensity image estimated by the first shading intensity estimation unit 41-1 by a color parameter [g R (T1),g G (T1),g B (T1)] to generate a first-color shaded image whose color is expressed by the first color parameter. The multiplication unit 43-1 supplies the generated first-color shaded image to the shade synthesis unit 32. The first-color shaded image becomes a shaded image of three channels, R, G, and B.

[0033] The second shade image generation unit 31-2 has a second shade intensity estimation unit 41-2, a second color parameter conversion unit 42-2, and a multiplication unit 43-2. ​​The second shade intensity estimation unit 41-2 is supplied with the feature amount of the input image extracted by the feature amount extraction unit 11. The second color parameter conversion unit 42-2 is supplied with a color temperature T2=8000 Kelvin as a second base color condition from the color condition setting unit 14.

[0034] The second shade intensity estimation unit 41-2 estimates a one-channel shade intensity image corresponding to the base color supplied from the color condition setting unit 14, based on the feature amounts of the input image extracted by the feature extraction unit 11. The estimated shade intensity image is supplied to a multiplication unit 43-2.

[0035] The second color parameter conversion unit 42-2 converts the color temperature T2=8000 supplied from the color condition setting unit 14 into second color parameters. The second color parameters are color parameters [g R (T2),g G (T2),g B (T2)]. The three-channel color parameters [g R (T2),g G (T2),g B (T2)] is supplied to the multiplication unit 43-2.

[0036] The multiplication unit 43-2 multiplies the one-channel shading intensity image estimated by the second shading intensity estimation unit 41-2 by a color parameter [g R (T2),g G (T2),g B (T2)] to generate a second-color shaded image whose color is expressed by the second color parameter. The multiplication unit 43-2 supplies the generated second-color shaded image to the shade synthesis unit 32. The second-color shaded image is a shaded image of three channels, R, G, and B.

[0037] The shadow synthesis unit 32 synthesizes the shadow image of the first color supplied from the first shadow image generation unit 31-1 and the shadow image of the second color supplied from the second shadow image generation unit 31-2, thereby generating (estimating) and outputting a shadow image.

[0038] The color temperature T n From the color parameter [g R (T n ),g G (T n ),g B (T n )]. R (T n ),g G (T n ),g B (T n )] is the so-called RGB value, which represents the ratio of the intensity of each color of R, G, and B. Color temperature T n From the color parameter [g R (T n ),g G (T n ),g B (T n ) can be obtained using, for example, Planck's formula or Wein's model. The following describes the conversion using Wein's model.

[0039] Wein's model has a color temperature of T n When the energy I(λ, T) emitted from a black body at wavelength λ is n ) and can be expressed by the following equation (1).

[0040]

number

[0041] In equation (1), h represents Planck's constant, e.g., h = 6.626 × 10 -34 k is the Boltzmann constant, for example, k = 1.3806 × 10 -23where c is a constant that represents the speed of light, for example, c = 2.9979 × 10 8 is.

[0042] When using a camera with RGB pixels to acquire an input image, the sensor spectral sensitivity of each of R, G, and B is approximately R ,λ G ,λ B It has narrowband wavelength sensitivity only in the color temperature T n The wavelength λ when R ,λ G ,λ B The energy at is I(λ R , T n ), I(λ G , T n ), I(λ B , T n ) where the color parameter g(T n )=[g R (T n ),g G (T n ),g B (T n )] is g G (T n ) is set to 1, that is, when the one-channel shadow intensity image obtained from the shadow intensity estimation unit 41 corresponds to the G channel among the three R, G, and B channels, g R (T n ) , g B (T n )teeth,

number

number

[0043] This color parameter g(T n ) = [g R (T n ), g G (T n ), g B (T n )] is multiplied by the one-channel shadow intensity image obtained by the shadow intensity estimation unit 41 to generate a shadow image of the nth color when the color temperature is T n .

[0044] <Configuration Example of CNN Predictor> As shown in FIG. 5, the image processing apparatus 1 can implement a feature amount extraction unit 11, an object color estimation unit 12, and first to Nth shadow intensity estimation units of a shadow estimation unit 13, i.e., 41-1 to 41-N, by a CNN predictor using a CNN. The CNN predictor is trained to output one object color image and N-channel shadow intensity images corresponding to the base number N when a predetermined input image is input. The N-channel (N sheets) of shadow intensity images estimated by the CNN predictor are supplied one channel at a time to the first to Nth shadow intensity estimation units 41-1 to 41-N.

[0045] In the learning process of the CNN predictor, the parameters of the CNN predictor are optimized so that the object color image obtained from the object color estimation unit 12 and the shadow image obtained from the shadow estimation unit 13 approach the teacher object color image and shadow image, respectively. Since all the operations of the first to Nth color parameter conversion units 42-1 to 42-N, multiplication units 43-1 to 43-N, and the shadow composition unit 32 in the subsequent stage of the CNN predictor are differentiable, it is possible to optimize the parameters by backpropagation of error. By the learning process, for example, the one-channel shadow intensity image output from the first shadow intensity estimation unit 41-1 is learned to correspond to the first base color, and the one-channel shadow intensity image output from the Nth shadow intensity estimation unit 41-N is learned to correspond to the Nth base color.

[0046] If the number of bases N is 2 (N=2), the first base is a color temperature T1=3000 Kelvin, and the second base is a color temperature T2=8000 Kelvin, the color space that the shading image can take will be a linear color connecting the point of the first base, color temperature T1=3000 Kelvin, and the point of the second base, color temperature T2=8000 Kelvin, as shown in Figure 2, resulting in a shading image that approximates the curved blackbody radiation locus.

[0047] For example, if the number of bases N is 3 (N=3), and in addition to the color temperature T1 of the first base = 3000 Kelvin and the color temperature T2 of the second base = 8000 Kelvin, and the achromatic color expressed by (x, y) = (0.33, 0.33) is used as the third base, the color space that the shadow image can take will be the colors inside the triangle whose vertices are the first base, second base, and third base, as shown in Figure 6, and the shadow image will be an approximation of the curved blackbody radiation locus.

[0048] <Other examples of specifying base colors> In the above example, the color condition setting unit 14 sets the color temperature T n (n=1, 2, . . . , N) is designated and supplied to the n-th color parameter conversion unit 42-n of the first shaded image generation unit 31-n.

[0049] However, the method of specifying the color conditions is not limited to this example, and other methods may be used. For example, as shown in FIG. 7, the base color may be specified by the x and y coordinate values ​​on the xy chromaticity diagram. Alternatively, as shown in FIG. 8, the color parameters [g R (T n ),g G (T n ),g B (T n )] can be directly specified as the base color. R (T n ),g G (T n ),g B (T n)] is directly specified as the base color, the n-th color parameter converter 42-n is omitted.

[0050] According to the shading estimation unit 13 of the first configuration example described above, the color space that the shading image can take is limited to a color space expressed by N base colors, so that the shading image can be stably estimated and highly accurate separation of the object color image and the shading image from the input image can be achieved.

[0051] In the above example, the solution space (color space) that the shadow component can take is restricted from all RGB spaces to a color space expressed by N base colors, and a shadow image is estimated, but this solution space may also be set from the xy chromaticity distribution of an actual image. Specifically, as shown in Fig. 9, the distribution of light source colors on the xy chromaticity diagram may be found from an image dataset 61, such as a data set of shadows and a data set of light source colors, and N bases may be set so that the convex hull that surrounds this distribution becomes the solution space.

[0052] 3. Second Configuration Example of Shadow Estimation Unit Next, a second configuration example of the shadow estimation unit 13 will be described.

[0053] In the first configuration example described above, the solution space that the shading component can take is restricted to a color space expressed by N base colors on the xy chromaticity diagram, but in the second configuration example, the restriction is set to follow the CIE daylight model.

[0054] According to the non-patent document "Deane B. Judd, David L. Macadam, and Gunter Wyszecki. Spectral distribution of typical daylight as a function of correlated color temperature. J. Opt. Soc. Am, 54(8):1031-1040, 1964," the CIE daylight model is a spectral distribution I of wavelength λ. CIE (λ) can be expressed as the following equation (3). I CIE(λ)=M1I1(λ)+M2I2(λ)+M3I3(λ) ········(3)

[0055] In equation (3), I1(λ), ​​I2(λ), and I3(λ) are basis functions for expressing daylight color based on actual measurements, and are expressed as shown in Figure 10. M1, M2, and M3 represent coefficients (parameters) for specifying the light source color.

[0056] FIG. 11 is a block diagram showing a second configuration example of the shadow estimation unit 13.

[0057] The shadow estimation unit 13 according to the second exemplary configuration includes a first shadow image generation unit 31-1 to a third shadow image generation unit 31-3, and a shadow synthesis unit 32. The first shadow image generation unit 31-1 to the third shadow image generation unit 31-3 each have the same configuration including a coefficient image estimation unit 81. Although the basis functions supplied as color conditions from the color condition setting unit 14 are different, the first shadow image generation unit 31-1 to the third shadow image generation unit 31-3 perform the same processing using the supplied basis functions. The first shadow image generation unit 31-1 to the third shadow image generation unit 31-3 correspond to basis functions I1(λ), ​​I2(λ), and I3(λ), respectively. The shadow synthesis unit 32 includes a spectrum synthesis unit 91 and a camera spectral sensitivity application unit 92.

[0058] The first shaded image generation unit 31-1 has a first coefficient image estimation unit 81-1, which is supplied with the feature of the input image extracted by the feature extraction unit 11 and the basis function I1(λ) as a color condition from the color condition setting unit 14.

[0059] The first shaded image generation unit 31-1 estimates a first coefficient image in which a coefficient M1 of the basis function I1(λ) is stored for each pixel from the feature amount of the input image and the basis function I1(λ), ​​and supplies the first coefficient image to the spectrum synthesis unit 91.

[0060] The second shaded image generation unit 31-2 has a second coefficient image estimation unit 81-2, which is supplied with the feature of the input image extracted by the feature extraction unit 11 and the basis function I2(λ) as a color condition from the color condition setting unit 14.

[0061] The second shaded image generation unit 31-2 estimates a second coefficient image in which a coefficient M2 of the basis function I2(λ) is stored for each pixel from the feature amount of the input image and the basis function I2(λ), and supplies the second coefficient image to the spectrum synthesis unit 91.

[0062] The third shaded image generation unit 31-3 has a third coefficient image estimation unit 81-3, which is supplied with the feature of the input image extracted by the feature extraction unit 11 and the basis function I3(λ) as a color condition from the color condition setting unit 14.

[0063] The third shaded image generation unit 31-3 estimates a third coefficient image in which a coefficient M3 of the basis function I3(λ) is stored for each pixel from the feature amount of the input image and the basis function I3(λ), and supplies the third coefficient image to the spectrum synthesis unit 91.

[0064] The spectral synthesis unit 91 calculates the spectral distribution I of the wavelength λ in the CIE daylight model by performing the calculation of equation (3) for each pixel of the first coefficient image to the third coefficient image. CIE The spectrum synthesis unit 91 calculates (λ) for each pixel. The spectrum synthesis unit 91 supplies the calculated result, which is a shadow spectral distribution image obtained by synthesizing the first coefficient image to the third coefficient image, to the camera spectral sensitivity application unit 92.

[0065] The camera spectral sensitivity application unit 92 converts the spectral distribution image of the shadow into a three-channel shadow image of R, G, and B by convolving it with the R, G, and B spectral sensitivity functions of the camera that captured the input image.

[0066] The camera spectral sensitivity application unit 92 outputs the shade images of the three channels of R, G, and B generated by the conversion as the shade images estimated by the shade synthesis unit 32.

[0067] According to the shading estimation unit 13 of the second configuration example described above, by imposing constraints so that the shading image conforms to the CIE daylight model, it is possible to stably estimate the shading image, thereby realizing highly accurate separation of the object color image and the shading image from the input image.

[0068] <4. Third Configuration Example of Shadow Estimation Unit> Next, a third configuration example of the shadow estimation unit 13 will be described.

[0069] In the first configuration example described above, the solution space that the shading components can take is restricted to a color space expressed by N base colors on the xy chromaticity diagram. In the third configuration example, assuming that the input image is an image taken outdoors, a configuration is adopted in which the shading components are estimated by setting a model according to the time the input image was taken (photography time).

[0070] Outdoors, light source color can be divided into two types: direct light, which is the light that is emitted directly from the sun onto an object, and global light, which is the light that is emitted onto an object after multiple reflections from clouds and the surrounding environment. Direct light is observed mainly in sunlight, while global light is observed mainly in the shade.

[0071] In the third configuration example, the number of the shade image generating units 31, which was N in the first configuration example, is limited to two, corresponding to the direct light and the global light. The color parameter g(T1)=[g R (T1),g G (T1),g B (T1)] and the color parameter g(T2) = [g R (T2),g G (T2),g B (T2)] and a second color shaded image are generated and combined to generate a three-channel shaded image of R, G, and B.

[0072] FIG. 12 is a block diagram showing a third configuration example of the shadow estimation unit 13. As shown in FIG.

[0073] The shadow estimation unit 13 according to the third configuration example includes a first shadow image generation unit 31-1, a second shadow image generation unit 31-2, and a shadow synthesis unit 32. The first shadow image generation unit 31-1 includes a first shadow intensity estimation unit 41-1, a first color parameter conversion unit 101-1, and a multiplication unit 43-1. The second shadow image generation unit 31-2 includes a second shadow intensity estimation unit 41-2, a second color parameter conversion unit 101-2, and a multiplication unit 43-2.

[0074] The shadow estimation unit 13 according to the third configuration example differs from the first configuration example shown in FIG. 3 in that the number of shadow image generation units 31 is limited to two and that the color parameter conversion unit 42 is changed to a color parameter conversion unit 101, but the other configurations are the same.

[0075] The first shade image generation unit 31-1 generates a shade image of a first color corresponding to direct light and supplies it to the shade synthesis unit 32. The second shade image generation unit 31-2 generates a shade image of a second color corresponding to global light and supplies it to the shade synthesis unit 32.

[0076] The first color parameter conversion unit 101-1 and the second color parameter conversion unit 101-2 are supplied with the shooting time as the color condition from the color condition setting unit 14.

[0077] The first color parameter conversion unit 101-1 converts the supplied photographing time into a color parameter g(T1)=[g R (T1),g G (T1),g B The second color parameter conversion unit 101-2 converts the supplied photographing time into a color parameter g(T2)=[g R (T2),g G (T2),g B (T2)].

[0078] For example, as disclosed in "Seo et al., Real-time adaptable and coherent rendering for outdoor augmented reality, EURASIP Journal on Image and Video Processing, 2018," the spectral distribution of direct sunlight changes over time, but its trajectory can be approximated by a unimodal function. Therefore, learning data is created by measuring the light source colors of direct and global light at each time in a scene where use is anticipated. Based on the learning data, a normal distribution or a quadratic or higher order function that describes the relationship between the shooting time and color temperature is generated for each of the direct and global light, as shown in FIG. 13. The generated functions are stored in the first color parameter conversion unit 101-1 and the second color parameter conversion unit 101-2. The relationship between the shooting time and color temperature described by a normal distribution or a quadratic or higher order function is a differentiable function, and therefore can be incorporated into a CNN for learning.

[0079] For example, if the range from 9:00 AM to 5:00 PM is divided into M equal parts, and the color temperature T of direct light at time x∈[0,M] is d is the normal distribution T d (x; A d , μ d , σ d 2 ) is modeled as

number

[0080] The series of light source colors measured at time x is {T x}, then T d (x; A d , μ d , σ d 2 ) parameter A d , μ d , σ d 2 is expressed in the logarithmic space as {T x} and T d (x; A d, μ d , σ d 2 ) to minimize the squared error J. d _min, μ d _min, σ d 2 It can be obtained as _min.

number

[0081] From the above, when the shooting time is provided, the color temperature of direct light T d can be determined, and thus, as in the first configuration example, the color temperature T d The color parameter g(T1)=[g R (T1),g G (T1),g B (T1)] can be generated. Similarly, for global light, the color temperature of the global light T i is a normal distribution T i (x; A i , μ i , σ i 2 ) and the global light color temperature T i The color parameter g(T2) = [g R (T2),g G (T2),g B (T2)] can be generated.

[0082] According to the shading estimation unit 13 of the third configuration example described above, by imposing constraints so that the shading image follows the direct light model and the global light model, it is possible to stably estimate the shading image, and to achieve highly accurate separation of the object color image and the shading image from the input image.

[0083] <5. Example of training data configuration> In the first to third configuration examples described above, the learning process for the CNN predictor involves comparing the shadow image finally output by the shadow estimation unit 13 with a teacher image to learn parameters for the CNN predictor that minimize the loss function. However, as shown in Fig. 14, the shadow image of the first color through the shadow image of the Nth color before being synthesized by the shadow synthesis unit 32 may also be compared with a teacher image to learn parameters for the CNN predictor that minimize the loss function. The teacher images for the shadow image of the first color through the shadow image of the Nth color can be generated, for example, by CG (Computer Graphics) or the like.

[0084] 6. Object Color and Shade Separation Processing of the First Embodiment Next, the object color and shadow separation processing performed by the image processing device 1 of the first embodiment will be described with reference to the flowchart in Fig. 15. In Fig. 15, the object color and shadow separation processing will be described when the shadow estimation unit 13 has the first configuration example shown in Fig. 3. This object color and shadow separation processing is started, for example, when an input image is supplied to the image processing device 1.

[0085] First, in step S1 , the feature extraction unit 11 extracts feature amounts of an input image and supplies them to the object color estimation unit 12 and the shade estimation unit 13 .

[0086] In step S2, the object color estimation unit 12 estimates and outputs an object color image having the color components (reflectance components) of the objects contained in the input image as pixel values ​​based on the features of the input image supplied from the feature extraction unit 11.

[0087] In step S3, the nth shade intensity estimation unit 41-n (n=1, 2, 3, . . . , N) estimates a one-channel shade intensity image corresponding to the nth base color supplied from the color condition setting unit 14, based on the feature amounts of the input image extracted by the feature extraction unit 11, and outputs the image to the multiplication unit 43-n. The first shade intensity estimation unit 41-1 to the Nth shade intensity estimation unit 41-N can simultaneously execute the process of estimating one-channel shade intensity images in parallel.

[0088] In step S4, the n-th color parameter conversion unit 42-n converts the color temperature T n is converted into the nth color parameter. The nth color parameter is the color parameter g(T n )=[g R (T n ),g G (T n ),g B (T n The three-channel color parameters [g R (T n ),g G (T n ),g B (T n )] is supplied to the multiplication unit 43-n. The first color parameter conversion unit 42-1 to the N-th color parameter conversion unit 42-N convert the color parameter g(T n ) conversion processes can be performed in parallel at the same time.

[0089] In step S5, the multiplication unit 43-n multiplies the one-channel shading intensity image estimated by the n-th shading intensity estimation unit 41-n by the color parameter [g R (T n ),g G (T n ),g B (T n )] to generate a shaded image of the nth color, the color of which is expressed by the nth color parameter. The multiplication units 43-1 to 43-N can simultaneously execute the process of generating a shaded image of the first color through a shaded image of the Nth color by multiplication in parallel. The generated shaded images of the first color through the Nth color are supplied to the shade synthesis unit 32.

[0090] In step S6, the shadow synthesis unit 32 synthesizes the shadow image of the first color through the shadow image of the Nth color to generate and output a shadow image.

[0091] The process of step S2 for estimating and outputting an object color image and the processes of steps S3 to S6 for generating and outputting a shaded image can be executed in parallel at the same time.

[0092] When the object color image and the shade image are generated and output from the image processing device 1, the object color and shade separation processing of FIG. 15 is completed.

[0093] The object color shadow separation process is executed as described above when the shadow estimation unit 13 has the first configuration example shown in Fig. 3. When the shadow estimation unit 13 has the second configuration example shown in Fig. 11, the processes of steps S3 to S5 are replaced by processes of the first coefficient image estimation unit 81-1 to the third coefficient image estimation unit 81-3, and the process of step S6 is replaced by processes of the spectrum synthesis unit 91 and the camera spectral sensitivity application unit 92. When the shadow estimation unit 13 has the third configuration example shown in Fig. 12, the process of step S4 is replaced by processes of the first color parameter conversion unit 101-1 and the second color parameter conversion unit 101-2.

[0094] The image processing device 1 according to the first embodiment can stably estimate a shaded image by limiting the solution space that the shaded component can take from all RGB spaces to a color space of a predetermined model. This allows for separation of an object color image and a shaded image with higher accuracy, and enables output of an object color image and a shaded image with better separation.

[0095] <7. Second embodiment of image processing device> FIG. 16 is a block diagram showing a configuration example of the second embodiment of the image processing device of the present disclosure.

[0096] The second embodiment of FIG. 16 differs from the first embodiment described above in that the object color image output by the object color estimation unit 12 is returned to the input of the feature extraction unit 11, but the other configurations are the same.

[0097] Even in the first embodiment described above, the problem of determining two variables, object color and shading, from a single input image remains ill-posed, and there may be cases where the separation of object color and shading is insufficient.

[0098] In the image processing device 1 according to the second embodiment, it is determined whether the object color image output by the object color estimation unit 12 satisfies a predetermined termination condition. If the predetermined termination condition is not satisfied, the object color image estimated by the object color estimation unit 12 is determined to be an image in which separation of the object color image and the shade image is insufficient and shades remain in the object color image, and is input again to the feature extraction unit 11. Then, if it is determined that the predetermined termination condition is satisfied, the object color image and the shade image are output from the image processing device 1 as the final object color image and the shade image. By repeatedly separating the object color image and the shade image until the predetermined termination condition is satisfied, it is possible to separate the object color image and the shade image with higher accuracy and to output better separated object color image and shade image.

[0099] As the predetermined termination condition, for example, a number of repetitions may be set in advance, and when the number of repetitions reaches the set number, it may be determined that the predetermined termination condition is met.

[0100] 8. Object Color and Shade Separation Processing of the Second Embodiment The object color and shade separation processing by the image processing device 1 of the second embodiment will be described with reference to the flowchart of Fig. 17. This object color and shade separation processing is started when an input image is supplied to the image processing device 1, for example.

[0101] First, in step S51, the feature extraction unit 11, the object color estimation unit 12, and the shade estimation unit 13 execute a process of separating an input image into an object color image and a shade image. The specific process of step S51 corresponds to a single separation process of separating an input image into an object color image and a shade image, and is the same as the object color and shade separation process of the first embodiment shown in FIG.

[0102] In step S52, the object color estimation unit 12 determines whether a preset termination condition is satisfied. If it is determined in step S52 that the preset termination condition is not satisfied, the process proceeds to step S53, where the object color estimation unit 12 inputs the estimated object color image as an input image to the feature extraction unit 11. After step S53, the process returns to step S51.

[0103] On the other hand, if it is determined in step S52 that the preset termination condition is satisfied, the process proceeds to step S54, where the object color estimation unit 12 outputs the estimated object color image. Also in step S54, the shade estimation unit 13 outputs the estimated shade image.

[0104] When the object color image and the shade image are output from the image processing device 1, the object color and shade separation processing of FIG. 17 ends.

[0105] FIG. 18 is a diagram illustrating another example of the termination condition.

[0106] 18A, the termination condition can be that the image error calculated by the root mean square error (RMSE) or the like between the input image input to the feature extraction unit 11 and the object color image estimated by the object color estimation unit 12 is smaller than a predetermined set value. When the image error becomes smaller than the predetermined set value, it is determined that the termination condition is met, and the object color estimation unit 12 outputs the estimated object color image, and the shade estimation unit 13 outputs the estimated shade image.

[0107] 18B, the termination condition can be that the object color estimation unit 12 has executed the process of estimating an object color image Q times (Q>1), and the difference between the image error between the (Q-1)th input image and the object color image output by the object color estimation unit 12 and the image error between the Qth input image and the object color image output by the object color estimation unit 12 is smaller than a predetermined set value. When the difference in image error becomes smaller than the predetermined set value, it is determined that the termination condition is met, and the object color estimation unit 12 outputs the estimated object color image, and the shade estimation unit 13 outputs the estimated shade image.

[0108] <9. Third embodiment of image processing device> FIG. 19 is a block diagram illustrating a configuration example of the third embodiment of the image processing device of the present disclosure.

[0109] 19 differs from the first embodiment in that a shadow processing unit 171 is newly added. The shadow processing unit 171 includes a shadow intensity adjustment unit 181 and an image synthesis unit 182.

[0110] The shadow intensity adjustment unit 181 is supplied with the shadow image estimated by the shadow estimation unit 13. The shadow intensity adjustment unit 181 performs a shadow intensity adjustment process to adjust the shadow intensity of the shadow image. Specifically, the shadow intensity adjustment unit 181 performs a power transformation on the luminance value of the supplied shadow image to perform a low-contrast process to reduce the contrast of the shadow. For example, the shadow intensity adjustment unit 181 may set the luminance value after the low-contrast process to [luminance value] with an exponent p=0.9 or the like. p The shadow intensity adjustment unit 181 supplies the generated low-contrast shadow image to the image synthesis unit 182.

[0111] The image synthesis unit 182 is supplied with the low-contrast shadow image from the shadow intensity adjustment unit 181, and also with the object color image estimated by the object color estimation unit 12. The image synthesis unit 182 multiplies corresponding pixels of the low-contrast shadow image and the object color image to generate an image in which the shadow effect is weakened compared to the input image (hereinafter referred to as a shadow-processed image). The generated shadow-processed image is again input to the feature extraction unit 11 as an input image. As in the second embodiment, the process of inputting the shadow-processed image, whose shadow intensity has been adjusted, as an input image to the feature extraction unit 11 is repeatedly executed until it is determined that a predetermined termination condition is satisfied.

[0112] The image processing device 1 according to the third embodiment reduces the amount of change between the input image and the object color image, and therefore can expect the same effects as those of the second embodiment, which performs recursive processing. That is, separation of the object color image and the shaded image can be performed with higher accuracy, and better separated object color image and shaded image can be output.

[0113] 10. Object Color and Shade Separation Processing of the Third Embodiment The object color and shade separation processing by the image processing device 1 of the third embodiment will be described with reference to the flowchart of Fig. 20. This object color and shade separation processing is started when an input image is supplied to the image processing device 1, for example.

[0114] First, in step S71, the feature extraction unit 11, object color estimation unit 12, and shade estimation unit 13 execute a process of separating an input image into an object color image and a shade image. The specific process of step S71 corresponds to a single separation process of separating into an object color image and a shade image, and is the same as the object color shade separation process of Fig. 15 in the first embodiment. However, it differs in that the shade image estimated by the shade estimation unit 13 is supplied to a shade intensity adjustment unit 181, and the object color image estimated by the object color estimation unit 12 is supplied to an image synthesis unit 182.

[0115] In step S72, the object color estimation unit 12 determines whether a preset termination condition is satisfied. If it is determined in step S72 that the preset termination condition is not satisfied, the process proceeds to step S73.

[0116] In step S73, the shading processing unit 171 generates a shading-processed image in which the shading effect is weakened compared to the input image from the estimated object color image and shading image, and inputs the generated image as an input image to the feature extraction unit 11. More specifically, the shading intensity adjustment unit 181 performs shading intensity adjustment processing on the shading image estimated by the shading estimation unit 13 to generate a low-contrast shading image, and supplies the generated image to the image synthesis unit 182. The image synthesis unit 182 generates the shading-processed image by multiplying corresponding pixels of the low-contrast shading image and the object color image. The shading-processed image generated by the image synthesis unit 182 is input to the feature extraction unit 11 as an input image, and the process returns to step S71.

[0117] On the other hand, if it is determined in step S72 that the preset termination condition is satisfied, the process proceeds to step S74, where the object color estimation unit 12 outputs the estimated object color image. Also in step S74, the shade estimation unit 13 outputs the estimated shade image.

[0118] When the object color image and the shade image are output from the image processing device 1, the object color and shade separation processing of FIG. 20 ends.

[0119] <Cascade structure configuration example> The image processing device 1 according to the third embodiment shown in FIG. 19 includes one feature extraction unit 11, one object color estimation unit 12, and one shade estimation unit 13, and is configured such that the object color image and shade image estimated by the object color estimation unit 12 and the shade estimation unit 13 are input again to the same feature extraction unit 11 and processed recursively.

[0120] However, as shown in Fig. 21, a configuration may also be adopted in which a feature extraction unit 11, an object color estimation unit 12, and a shade estimation unit 13 are provided in two stages, and the estimation results from the feature extraction unit 11A, the object color estimation unit 12A, and the shade estimation unit 13A in the front stage are input to the feature extraction unit 11B, the object color estimation unit 12B, and the shade estimation unit 13B in the rear stage, thereby repeatedly executing the object color and shade separation processing. The image processing device 1 in Fig. 21 is configured such that a shade processing unit 171 is provided between the feature extraction unit 11A, the object color estimation unit 12A, and the shade estimation unit 13A in the front stage and the feature extraction unit 11B, the object color estimation unit 12B, and the shade estimation unit 13B in the rear stage. While this image processing device 1 is an example of a two-stage cascade structure, a three- or more-stage cascade structure may of course be used.

[0121] 11. Example of learning device configuration FIG. 22 is a block diagram showing an example of the configuration of a learning device that learns parameters of a CNN predictor.

[0122] 22 includes a feature extraction unit 191, an object color estimation unit 192, a shade estimation unit 193, and a color condition setting unit 194. The detailed configurations of the feature extraction unit 191, the object color estimation unit 192, the shade estimation unit 193, and the color condition setting unit 194 are similar to those of the feature extraction unit 11, the object color estimation unit 12, the shade estimation unit 13, and the color condition setting unit 14, respectively, and therefore detailed description thereof will be omitted.

[0123] In the learning process by the learning device 190, an input image and a large number of object color images and shade images as teacher images corresponding to the input image are prepared as learning data, and parameters of the CNN predictor are calculated by error backpropagation.

[0124] When separating an object color image from a shaded image with high accuracy by recursion, as in the image processing device 1 according to the second embodiment shown in FIG. 16 and the image processing device 1 according to the third embodiment shown in FIG. 19, the above-mentioned shading processing unit 171 can also be used during learning, and a set of an input image obtained by processing a pre-prepared input image to reduce shading, an object color image with reduced reflectance, and a shaded image with reduced shading can be further added as learning data, thereby making it possible to calculate parameters for separating an object color image from a shaded image with higher accuracy.

[0125] <12. Fourth embodiment of image processing device> FIG. 23 is a block diagram illustrating a configuration example of the fourth embodiment of the image processing device of the present disclosure.

[0126] The image processing device 1 according to the fourth embodiment shown in FIG. 23 includes a feature extraction unit 11, an object color estimation unit 12, a shade estimation unit 201, a user setting unit 202, and an image synthesis unit 203.

[0127] Compared with the first embodiment described above, the image processing device 1 according to the fourth embodiment has the shadow estimation unit 13 and color condition setting unit 14 of the first embodiment replaced with a shadow estimation unit 201 and a user setting unit 202, and also has a new image synthesis unit 203 added.

[0128] Similar to the shade estimation unit 13 of the first embodiment, the shade estimation unit 201 generates, as intermediate products, shade images of first to Nth colors corresponding to first to Nth base colors supplied as color conditions from the user setting unit 202. The shade estimation unit 13 of the first embodiment generates a shade image by synthesizing all of the shade images of the first to Nth colors generated as intermediate products, but the shade estimation unit 201 selects at least one of the shade images of the first to Nth colors in accordance with a selection instruction from the user setting unit 202. The shade estimation unit 201 generates a shade image that reflects the user's intention using only the selected image from the shade images of the first to Nth colors generated as intermediate products, and supplies the generated shade image to the image synthesis unit 203.

[0129] As in the first embodiment, the user setting unit 202 sets color conditions for the shadow estimation unit 201. The user setting unit 202 also accepts a selection instruction from the user to select a part of the shadow images of the first to Nth colors generated as intermediate products in the shadow estimation unit 201, and supplies the instruction to the shadow estimation unit 201.

[0130] The image synthesis unit 203 generates an image in which the light source has been edited by multiplying the object color image supplied from the object color estimation unit 12 by corresponding pixels of the shade image reflecting the user's intention supplied from the shade estimation unit 201. The generated image is output to the outside of the image processing device 1.

[0131] Fig. 24 is a block diagram showing a detailed configuration example of the shadow estimation unit 201 in Fig. 23. However, for simplicity, the shadow estimation unit 201 in Fig. 24 is a configuration example in which the number of bases N is 2 (N=2).

[0132] The shadow estimation unit 201 is the same as the shadow estimation unit 13 in FIG. 4 in which the base number N is set to 2 in the first embodiment, except that the shadow synthesis unit 32 is replaced by a shadow synthesis unit 232.

[0133] The shadow synthesis unit 232 is supplied with a selection instruction from the user setting unit 202 indicating whether to select the shadow image of the first color supplied from the first shadow image generation unit 31-1 or the shadow image of the second color supplied from the second shadow image generation unit 31-2.

[0134] The shadow synthesis unit 232 selects one of the shadow image of the first color and the shadow image of the second color in accordance with a selection instruction from the user setting unit 202, and supplies it to the image synthesis unit 203 (FIG. 23).

[0135] FIG. 25 is a diagram for explaining the processing of the shadow estimation unit 201.

[0136] The input image includes shadows caused by light source 1 and light source 2. Based on the feature amount obtained from the input image, the first shade image generation unit 31-1 generates a shade image of a first color, which is an image including shadows caused by light source 1. Based on the feature amount obtained from the input image, the second shade image generation unit 31-2 generates a shade image of a second color, which is an image including shadows caused by light source 2.

[0137] When the user wishes to generate an image that eliminates the influence of the light source 2, the user selects the shadow image of the first color and issues a selection instruction to delete the shadow image of the second color in the user setting unit 202. This operation causes the shadow synthesis unit 232 to select and output only the shadow image of the first color as a shadow image that reflects the user's intention.

[0138] 24 and 25 show an example in which the base number N is 2, so one of the shading image of the first color and the shading image of the second color is selected, the other is not selected, and the selected one is output. However, if the base number N is 3 or more and shading images of multiple colors are selected, the shading synthesis unit 232 synthesizes and outputs the selected shading images of multiple colors, just like the shading synthesis unit 32 in the first embodiment.

[0139] The image synthesis unit 203 in Figure 23 synthesizes the object color image supplied from the object color estimation unit 12 with a shading image that eliminates the influence of the light source 2, so that the image output from the image processing device 1 is an image illuminated only with the first base color, excluding the influence of the light source 2.

[0140] According to the image processing device 1 of the fourth embodiment, the user can select and discard the shadow images of the first to Nth colors, which were intermediate products in the first embodiment described above, thereby enabling the user to separate light sources and edit the light sources.

[0141] <13. Computer configuration example> The series of processes performed by the image processing device 1 and learning device 190 described above can be executed by hardware or software. When the series of processes are executed by software, the programs constituting the software are installed on a computer. Here, the computer includes a microcomputer incorporated in dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0142] FIG. 26 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes using a program.

[0143] In the computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are interconnected by a bus 504.

[0144] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a storage unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0145] The input unit 506 includes a keyboard, mouse, microphone, touch panel, input terminal, etc. The output unit 507 includes a display, speaker, output terminal, etc. The storage unit 508 includes a hard disk, RAM disk, non-volatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0146] In the computer configured as above, the CPU 501 performs the above-described series of processes by, for example, loading a program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing the program. The RAM 503 also stores data necessary for the CPU 501 to execute various processes as needed.

[0147] The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0148] In a computer, the program can be installed in the storage unit 508 via the input / output interface 505 by inserting the removable recording medium 511 into the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the storage unit 508. Alternatively, the program can be installed in the ROM 502 or the storage unit 508 in advance.

[0149] In this specification, the steps described in the flowcharts may be performed in chronological order in the order described, but they do not necessarily have to be processed in chronological order, and may be performed in parallel or at any necessary timing, such as when a call is made.

[0150] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the technology of the present disclosure.

[0151] For example, it is possible to adopt a form in which all or part of the above-described embodiments are combined.

[0152] For example, the technology of the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0153] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0154] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0155] The effects described in this specification are merely examples and are not limiting, and there may be effects other than those described in this specification.

[0156] The technology of the present disclosure can have the following configurations. (1) an object color estimation unit that estimates an object color image having color components of objects included in the input image as pixel values ​​based on feature amounts of the input image; a shade estimation unit that estimates a shade image having a shade component of the input image as a pixel value based on the feature amount of the input image; Equipped with The shade estimation unit estimates the shade image by limiting a color space that can be taken by the shade component of the input image to a color space determined by predetermined color conditions. Image processing device. (2) N (N>1) base colors are given as the predetermined color conditions, The shade estimation unit estimates the shade image by limiting a color space that can be taken by the shade component of the input image to a color space expressed by the N base colors. The image processing device according to (1) above. (3) The shadow estimation unit N shaded image generators that generate shaded images corresponding to predetermined base colors; a shade synthesis unit that synthesizes the N shade images of the base colors generated by the N shade image generation units; have The image processing device according to (2) above. (4) The shadow image generation unit a shadow intensity image estimation unit that estimates a shadow intensity image based on the feature amount of the input image; a color parameter conversion unit that converts the predetermined base color into a color parameter; have The image processing device according to (3) above. (5) The base color is given by the color temperature The image processing device according to any one of (2) to (4). (6) The base colors are given by x and y coordinate values ​​on the xy chromaticity diagram. The image processing device according to any one of (2) to (4). (7) The base color is given by RGB color parameters The image processing device according to any one of (2) to (4). (8) The color space determined by the predetermined color conditions is a space according to the CIE daylight model. The image processing device according to (1) above. (9) The basis functions of the CIE daylight model are given as the predetermined color conditions. The image processing device according to (8) above. (10) The shadow estimation unit a first coefficient image estimation unit that estimates a first coefficient image that stores coefficients of a first basis function of the CIE daylight model; a second coefficient image estimation unit that estimates a second coefficient image that stores coefficients of a second basis function of the CIE daylight model; a third coefficient image estimation unit that estimates a third coefficient image that stores coefficients of a third basis function of the CIE daylight model; a synthesis unit that synthesizes the first coefficient image through the third coefficient image to generate a spectral distribution image of shadows; a spectral sensitivity application unit that converts the spectral distribution image of the shadow into the shadow image by convolving the spectral sensitivity functions of R, G, and B; have The image processing device according to (9) above. (11) The predetermined color condition is the photographing time of the input image. The image processing device according to (1) above. (12) The color parameters are converted into color parameters corresponding to direct light and color parameters corresponding to global light according to the photographing time, a first shaded image generating unit that generates a shaded image of a first color using color parameters corresponding to the direct light; a second shaded image generating unit that generates a shaded image of a second color using color parameters corresponding to the global light; a shade synthesis unit that synthesizes the shade image of the first color and the shade image of the second color; have The image processing device according to (11) above. (13) The input image further includes a feature extraction unit for extracting a feature of the input image. The image processing device according to any one of (1) to (12). (14) The process of inputting the object color image estimated by the object color estimation unit as the input image to the feature extraction unit is repeatedly executed until a predetermined termination condition is satisfied. The image processing device according to (13) above. (15) a shading processing unit that generates a shading-processed image in which shading intensity is adjusted, using the object color image estimated by the object color estimation unit and the shading image estimated by the shading estimation unit, The process of inputting the shaded image as the input image to the feature extraction unit is repeated until a predetermined termination condition is met. The image processing device according to (13) above. (16) The shadow estimation unit N (N>1) shaded image generating units that generate shaded images of colors corresponding to the predetermined color conditions; a shade synthesis unit that selects and synthesizes at least one of the N shade images of different colors generated by the N shade image generation units in accordance with a selection instruction from a user; and The image synthesized by the shadow synthesis unit and the object color image estimated by the object color estimation unit are synthesized. The image processing device according to (1) above. (17) The object color estimation unit and the shade estimation unit use a CNN predictor that uses parameters obtained by learning processing. The image processing device according to any one of (1) to (16). (18) The image processing device an object color image having color components of objects included in the input image as pixel values ​​based on the feature amount of the input image; estimating a shaded image having a shaded component of the input image as a pixel value based on the feature amount of the input image; The shaded image is estimated by restricting the color space that the shaded component of the input image can take to a color space determined by predetermined color conditions. Image processing methods. [Explanation of symbols]

[0157] 1 image processing device, 11 feature extraction unit, 12 object color estimation unit, 13 shading estimation unit, 14 color condition setting unit, 31 shading image generation unit, 31-1 to 31-N first to Nth shading image generation units, 32 shading synthesis unit, 41-1 to 41-N first to Nth shading intensity estimation units, 42-1 to 42-N first to Nth color parameter conversion units, 43-1 to 43-N multiplication unit, 61 image data set, 81-1 to 81-3 first to third coefficient image estimation units, 91 spectrum synthesis unit, 92 camera spectral sensitivity application unit, 101 color parameter conversion unit, 101-1 first color parameter conversion unit, 101-2 second color parameter conversion unit, 171 shading processing unit, 181 shading intensity adjustment unit, 182 Image synthesis unit, 190 learning device, 191 feature extraction unit, 192 object color estimation unit, 193 shading estimation unit, 194 color condition setting unit, 201 shading estimation unit, 202 user setting unit, 203 image synthesis unit, 232 shading synthesis unit, 501 CPU, 502 ROM, 503 RAM, 506 input unit, 507 output unit, 508 memory unit, 509 communication unit, 510 drive

Claims

1. an object color estimation unit that estimates an object color image having color components of objects included in the input image as pixel values ​​based on feature amounts of the input image; a shade estimation unit that estimates a shade image having a shade component of the input image as a pixel value based on the feature amount of the input image; Equipped with The shade estimation unit estimates the shade image by limiting a color space that can be taken by the shade component of the input image to a color space determined by predetermined color conditions. Image processing device.

2. N (N>1) base colors are given as the predetermined color conditions, The shade estimation unit estimates the shade image by limiting a color space that can be taken by the shade component of the input image to a color space expressed by the N base colors. The image processing device according to claim 1 .

3. The shadow estimation unit N shaded image generators that generate shaded images corresponding to predetermined base colors; a shade synthesis unit that synthesizes the N shade images of the base colors generated by the N shade image generation units; have The image processing device according to claim 2 .

4. The shadow image generation unit a shadow intensity image estimation unit that estimates a shadow intensity image based on the feature amount of the input image; a color parameter conversion unit that converts the predetermined base color into a color parameter; have The image processing device according to claim 3 .

5. The base color is given by the color temperature The image processing device according to claim 2 .

6. The base color is given by the x and y coordinate values ​​on the xy chromaticity diagram. The image processing device according to claim 2 .

7. The base color is given by RGB color parameters The image processing device according to claim 2 .

8. The color space determined by the predetermined color conditions is a space according to the CIE daylight model. The image processing device according to claim 1 .

9. The basis functions of the CIE daylight model are given as the predetermined color conditions. The image processing device according to claim 8 .

10. The shadow estimation unit a first coefficient image estimation unit that estimates a first coefficient image that stores coefficients of a first basis function of the CIE daylight model; a second coefficient image estimation unit that estimates a second coefficient image that stores coefficients of a second basis function of the CIE daylight model; a third coefficient image estimation unit that estimates a third coefficient image that stores coefficients of a third basis function of the CIE daylight model; a synthesis unit that synthesizes the first coefficient image through the third coefficient image to generate a spectral distribution image of shadows; a spectral sensitivity application unit that converts the spectral distribution image of the shade into the shade image by convolving the spectral sensitivity functions of R, G, and B; have The image processing device according to claim 9 .

11. The predetermined color condition is the photographing time of the input image. The image processing device according to claim 1 .

12. The color parameters are converted into color parameters corresponding to direct light and color parameters corresponding to global light according to the photographing time, a first shade image generating unit that generates a shade image of a first color using color parameters corresponding to the direct light; a second shade image generating unit that generates a shade image of a second color using color parameters corresponding to the global light; a shade synthesis unit that synthesizes the shade image of the first color and the shade image of the second color; have The image processing device according to claim 11 .

13. The input image further includes a feature extraction unit for extracting a feature of the input image. The image processing device according to claim 1 .

14. The process of inputting the object color image estimated by the object color estimation unit as the input image to the feature extraction unit is repeatedly executed until a predetermined termination condition is satisfied. The image processing device according to claim 13 .

15. a shading processing unit that generates a shading-processed image in which shading intensity is adjusted, using the object color image estimated by the object color estimation unit and the shading image estimated by the shading estimation unit, The process of inputting the shaded image as the input image to the feature extraction unit is repeated until a predetermined termination condition is met. The image processing device according to claim 13.

16. The shadow estimation unit N (N>1) shaded image generating units that generate shaded images of colors corresponding to the predetermined color conditions; a shade synthesis unit that selects and synthesizes at least one of the N shade images of different colors generated by the N shade image generation units in accordance with a selection instruction from a user; and The image synthesized by the shadow synthesis unit and the object color image estimated by the object color estimation unit are synthesized. The image processing device according to claim 1 .

17. The object color estimation unit and the shade estimation unit use a CNN predictor that uses parameters obtained by learning processing. The image processing device according to claim 1 .

18. The image processing device an object color image having color components of objects included in the input image as pixel values ​​based on the feature amount of the input image; estimating a shaded image having a shaded component of the input image as a pixel value based on the feature amount of the input image; The shaded image is estimated by restricting the color space that the shaded component of the input image can take to a color space determined by predetermined color conditions. Image processing methods.

Images