Light source estimation device, image processing system, and light source estimation method and program

Abstract

To provide a light source estimation device and a light source estimation method that can improve estimation accuracy for a light source.SOLUTION: A light source estimation device comprises: a reception part which receives scene constitution information including recognition information on an object including a background included in an image and image processed image information obtained by performing predetermined image processing on the image; and a light source estimation part which performs light source estimation based upon the scene constitution information and the image processed image information that the reception part receives.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a light source estimation device, an image processing system, a light source estimation method, and a program. [Background technology]

[0002] There are color correction technologies that can make images taken in different environments appear as if they were taken in the same environment. One example is WB correction (white balance correction), but this requires light source information at the time of shooting. However, light source information at the time of shooting is difficult to obtain in advance due to various conditions, such as whether it is indoors or outdoors, the time of shooting, and the weather. One solution is to measure the light source at the time of shooting, but measuring the light source of the shooting environment every time an image is taken is a cumbersome task. To address this, light source estimation methods are available that estimate the light source from the input image.

[0003] Regarding image processing technology, there is known a technology that reduces the influence of erroneous correction of white balance due to erroneous detection in subject recognition, and enables appropriate white balance control (see, for example, Patent Document 1). Furthermore, there is known a technology that can achieve photometry and colorimetry with weighting appropriate for the shape and position of a main subject such as a person, thereby enabling more appropriate exposure correction and white balance correction (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2021-182673 [Patent Document 2] Japanese Patent Application Laid-Open No. 2006-211416 Summary of the Invention [Problem to be solved by the invention]

[0005] Illuminant estimation techniques include methods for estimating the illuminant from an RGB image. In methods for estimating the illuminant from an RGB image, the estimated illuminant value may be affected by the color of an object that occupies a large proportion of the scene. If the estimated illuminant value is affected by the color of an object that occupies a large proportion of the scene, the accuracy of the illuminant estimation will be affected. Furthermore, depending on the light source estimation method, the hyperparameters that achieve good accuracy indoors and outdoors may differ, making it difficult to determine the hyperparameters for each scene.

[0006] The present invention has been made in view of the above circumstances, and has an object to provide a light source estimation device and a light source estimation method that can improve the accuracy of estimating a light source. [Means for solving the problem]

[0007] One aspect of the present invention is a light source estimation device comprising: a reception unit that receives scene composition information including recognition information of objects, including a background, included in an image, and image-processed image information obtained by performing a predetermined image processing on the image; and a light source estimation unit that performs light source estimation based on the scene composition information received by the reception unit and the image-processed image information.

[0008] Another aspect of the present invention is an image processing system comprising: a reception unit that receives scene composition information including recognition information of objects, including a background, included in an image, and image-processed image information obtained by performing a predetermined image processing on the image; and a light source estimation unit that performs light source estimation based on the scene composition information received by the reception unit and the image-processed image information.

[0009] Another aspect of the present invention is a light source estimation method executed by a computer, comprising the steps of: accepting scene composition information including recognition information of an object, including a background, included in an image; and image-processed image information obtained by performing predetermined image processing on the image; and performing light source estimation based on the scene composition information accepted in the accepting step and the image-processed image information.

[0010] Another aspect of the present invention is a program that causes a computer to execute the steps of accepting scene composition information including recognition information of objects, including a background, included in an image, and image-processed image information obtained by performing a predetermined image processing on the image, and estimating a light source based on the scene composition information accepted in the accepting step and the image-processed image information. [Effects of the Invention]

[0011] According to the present invention, it is possible to obtain an effect that the accuracy of estimating a light source can be improved. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram illustrating an example of the configuration of a light source estimation apparatus 100 according to a first embodiment of the present invention. [Figure 2A] 2 is a diagram showing an example of an image input to the light source estimation apparatus 100 according to the present embodiment. FIG. [Figure 2B] 3 is a diagram for explaining an example of processing by the light source estimation apparatus 100 according to the present embodiment. FIG. [Figure 2C] 3 is a diagram for explaining an example of processing by the light source estimation apparatus 100 according to the present embodiment. FIG. [Figure 3] 3 is a diagram for explaining an example of processing by the light source estimation apparatus 100 according to the present embodiment. FIG. [Figure 4] 3 is a diagram illustrating an example of the operation of the light source estimation apparatus 100 according to the present embodiment. FIG. [Figure 5] FIG. 2 is a diagram illustrating an example of the configuration of another light source estimation device 200 according to the present embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of the operation of the light source estimation apparatus 200 according to the present embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of the configuration of a light source estimation apparatus 300 according to a second embodiment of the present invention. [Figure 8] 3 is a diagram illustrating an example of the operation of the light source estimation apparatus 300 according to the present embodiment. FIG. [Figure 9] FIG. 10 is a diagram illustrating an example of the configuration of a light source estimation apparatus 400 according to a second embodiment of the present invention. [Figure 10] FIG. 10 is a diagram illustrating an example of the configuration of a light source estimation device 500 according to a third embodiment of the present invention. [Figure 11] FIG. 4 is a diagram illustrating an example of the operation of the light source estimation apparatus 500 according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Next, a light source estimation device, an image processing system, a light source estimation method, and a program according to the present embodiment will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiments to which the present invention is applied are not limited to the following embodiments. In all the drawings for explaining the embodiments, the same reference numerals are used for components having the same functions, and repeated explanations will be omitted. Furthermore, in this application, "based on XX" means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[0014] (First embodiment) The image processing system according to this embodiment performs color correction on images captured in different environments, making them appear as if they were captured in the same environment. The image processing system estimates the spectral distribution of a subject and converts it into an image captured under an arbitrary light source. The image processing system includes a color correction processing device and a light source estimation device. The color correction processing device performs color correction processing based on the input image, spectral sensitivity information of the camera used for capturing the image, and light source information at the time of capture acquired by the light source estimation device. The light source estimation device will be described below. [Light source estimation device] FIG. 1 is a diagram showing an example of the configuration of a light source estimation apparatus 100 according to a first embodiment of the present invention. The light source estimation apparatus 100 is realized as a smartphone, a mobile terminal, a personal computer, a tablet terminal, a wristwatch-type terminal, or other information processing equipment. The light source estimation apparatus 100 includes, for example, an input unit 102, an object recognition unit 103, an area detection unit 104, a scene structure information generation unit 105, an image processing unit 106, a light source estimation unit 108, an output unit 109, and a storage unit 110.

[0015] The input unit 102 inputs information. As an example, the input unit 102 may have an operation unit such as a keyboard and a mouse. In this case, the input unit 102 inputs information according to an operation performed by a user on the operation unit. As another example, the input unit 102 may input information from an external device. Image information of an image is input to the input unit 102. FIG. 2A is a diagram showing an example of an image input to the light source estimation apparatus 100 according to this embodiment. Examples of images include buildings, vehicles, people, etc. Returning to FIG. 1, the explanation will continue.

[0016] The output unit 109 is, for example, a display, and may be configured to include a touch panel. The storage unit 110 is realized by a hard disk drive (HDD), flash memory, random access memory (RAM), read only memory (ROM), or the like, and stores information.

[0017] The object recognition unit 103 acquires image information from the input unit 102. Based on the acquired image information, the object recognition unit 103 recognizes one or more objects (objects) included in the image. Here, the objects recognized by the object recognition unit 103 include the background. Specifically, the object recognition unit 103 labels pixels in the image. Based on the result of labeling the pixels in the image, the object recognition unit 103 recognizes one or more objects included in the image.

[0018] The object recognition unit 103 may also recognize the location where the image was taken based on the result of labeling the pixels in the image. The object recognition unit 103 may also determine the weather when the image was taken and the time when the image was taken based on the result of labeling the pixels in the image.

[0019] The area detection unit 104 acquires recognition results of one or more objects (information indicating one or more objects recognized by the object recognition unit 103) from the object recognition unit 103. The area detection unit 104 detects one or more detection areas from the image based on the acquired recognition results of the one or more objects. An example of information identifying one or more detection areas divided based on one or more objects included in the image is an image on which semantic segmentation has been performed. Specifically, the area detection unit 104 divides the image into one or more detection areas based on the recognition results of the one or more objects. The area of ​​the object may coincide with the detection area, or multiple areas of the object may coincide with the detection area. For example, the area detection unit 104 may perform semantic segmentation.

[0020] The scene construction information generation unit 105 acquires object recognition information, which is the recognition result of one or more objects included in the image, from the object recognition unit 103, and acquires information identifying one or more detection areas detected based on the one or more objects included in the image from the area detection unit 104. The scene construction information generation unit 105 creates scene construction information including the object recognition information, which is the recognition result of one or more objects included in the acquired image, and information identifying one or more detection areas detected based on the one or more objects included in the image. The scene structure information generating unit 105 inputs the created scene structure information to the light source estimating unit .

[0021] 2B is a diagram illustrating an example of processing by the light source estimation apparatus 100 according to this embodiment. Fig. 2B shows an example of an image divided into multiple detection regions based on multiple objects included in the image by the region detection unit 104. Examples of the image include detection regions for buildings, detection regions for vehicles, and detection regions for people. In this case, for example, the object recognition unit 103 acquires information indicating a building, information indicating a vehicle, information indicating a person, etc. as recognition information for one or more objects. The area detection unit 104 acquires a detection area for a building, a detection area for a vehicle, a detection area for a person, etc. as information for identifying one or more detection areas. Returning to FIG. 1, the explanation will be continued.

[0022] The image processing unit 106 acquires image information from the input unit 102. The image processing unit 106 processes the acquired image information to create image information for light source estimation (hereinafter referred to as "processed image information"). Here, the image processing includes identity transformation. A light source estimation method to be performed by the light source estimation unit 108 is set in the image processing unit 106, and the image information for light source estimation is created based on this light source estimation method. Here, as an example, the explanation will continue assuming that Shades of Gray is set as the light source estimation method in the image processing unit 106.

[0023] For example, when Shades of Gray is set as the light source estimation method, the image processing unit 106 creates image information (including the original image, which is an identity transformation) indicating an edge image. For example, the image processing unit 106 may create image information of the edge image by raising the pixel value to the pth power and then performing q-th derivative (p and q are integers such that p>0 and q>0). The image processing unit 106 inputs the created image-processed image information to the light source estimation unit 108. FIG. 2C is a diagram for explaining an example of processing by the light source estimation device 100 according to this embodiment. FIG. 2C shows an example of an edge image generated by the image processing unit 106 based on the image information. Returning to FIG. 1, the explanation will continue.

[0024] The light source estimation unit 108 acquires scene composition information from the scene composition information generation unit 105, and acquires image-processed image information from the image processing unit 106. The light source estimation unit 108 performs light source estimation based on the scene composition information and the image-processed image information using a method for estimating a light source from an RGB image.

[0025] The light source estimation unit 108 derives the area ratio of each of one or more detection regions included in the image based on the recognition information of one or more objects included in the scene configuration information and the information identifying one or more detection regions. FIG. 3 is a diagram for explaining an example of processing by the light source estimation device 100 according to this embodiment. With reference to FIG. 3, an example of deriving the area ratio of each of one or more detection regions included in an image based on the recognition information of one or more objects and the information identifying one or more detection regions will be described. As shown in FIG. 3, detection region (1) corresponds to a building and has an area ratio of 30%, detection region (2) corresponds to a car and has an area ratio of 10%, and detection region (3) corresponds to a person and has an area ratio of 5%.

[0026] The light source estimation unit 108 estimates the light source by a method of estimating the light source from an R (red), G (green), and B (blue) image based on the results of deriving the area ratio of each of one or more detection regions. The gray world algorithm assumes the statistical fact that averaging all colors in an image results in a color close to achromatic, and uses the average of all pixels in the image as the light source color (value). The light source estimation unit 108 derives the average pixel value of the image for each of multiple detection regions included in the image. The light source estimation unit 108 multiplies the average of the pixels of the image derived for each of the multiple detection regions included in the image by the inverse of the area ratio. This reduces the influence of the light source value on colors with large areas. Specifically, the light source estimation unit 108 derives the light source estimate value using equation (1).

[0027] Illuminant estimation value = (Illuminant value of detection area (1) / 30 [%] + Illuminant value of detection area (2) / 10 [%] + Illuminant value of detection area (3) / 5 [%]) / Total number of pixels (1) The light source estimation unit 108 outputs information indicating the derived light source estimation value to the output unit 109. The output unit 109 acquires information indicating the light source estimation value from the light source estimation unit 108 and outputs the information.

[0028] The object recognition unit 103, area detection unit 104, scene composition information generation unit 105, image processing unit 106 and light source estimation unit 108 are realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a computer program (software) stored in the memory unit 110. Furthermore, some or all of these functional units may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware.

[0029] The computer program may be stored in advance in a storage device such as a HDD or flash memory, or may be stored in a removable storage medium such as a DVD (Digital Versatile Disc) or CD-ROM, and installed by inserting the storage medium into a drive device.

[0030] (Operation of the light source estimation device 100) FIG. 4 is a diagram showing an example of the operation of the light source estimation apparatus 100 according to this embodiment. (Step S1-1) Image information is input to the input unit 102 . (Step S2-1) The object recognition unit 103 acquires image information from the input unit 102, and recognizes one or more target objects (objects) included in the image based on the acquired image information. (Step S3-1) The area detection unit 104 acquires recognition information (object recognition information) of one or more objects from the object recognition unit 103, and detects one or more detection areas from the image based on the acquired object recognition information. (Step S4-1) The scene construction information generation unit 105 acquires object recognition information from the object recognition unit 103, and acquires information identifying one or more detection areas (detection area information) from the area detection unit 104. The scene construction information generation unit 105 creates scene construction information including the acquired object recognition information and detection area information.

[0031] (Step S5-1) The image processing unit 106 acquires image information from the input unit 102, and performs image processing based on the acquired image information to create image-processed image information. (Step S6-1) The light source estimation unit 108 acquires scene composition information from the scene composition information generation unit 105, and acquires image-processed image information from the image processing unit 106. The light source estimation unit 108 performs light source estimation based on the scene composition information and the image-processed image information using a method for estimating a light source from an RGB image. (Step S7-1) The output unit 109 acquires the light source estimation value from the light source estimation unit 108 and outputs it.

[0032] In the above-described embodiment, the light source estimation unit 108 derives the light source estimation value using the Shades of Gray algorithm, but the present invention is not limited to this example. For example, the light source estimation unit 108 may be configured to derive the light source estimation value using a Mean Shifted Gray Pixel (MSGP) algorithm, a Grayness Index algorithm, or an algorithm similar to the above. Here, the Max RGB algorithm uses the maximum value of all pixels in the image as the light source color. The Gray Edge algorithm, Gray Pixel algorithm, MSGP algorithm, and Grayness Index algorithm extract pixels that appear to be achromatic in the image and use the average of these as the light source color.

[0033] In the above-described embodiment, as an example, a case where image information is input to the light source estimation apparatus 100 has been described, but the present invention is not limited to this example. For example, scene composition information and image-processed image information may be input to the light source estimation apparatus 100. 5 is a diagram showing an example of the configuration of another light source estimation apparatus 200 according to this embodiment. The light source estimation apparatus 200 is realized as a smartphone, a mobile terminal, a personal computer, a tablet terminal device, a wristwatch-type terminal device, or other information processing equipment.

[0034] Light source estimation apparatus 200 includes, for example, input unit 202, reception unit 203, light source estimation unit 208, output unit 209, and storage unit 210. Input unit 102, light source estimation unit 108, output unit 109, and storage unit 110 can be applied to input unit 202, light source estimation unit 208, output unit 209, and storage unit 210, respectively.

[0035] Scene configuration information and image-processed image information are input to the input unit 202. For example, the scene configuration information and image-processed image information may be generated by a device other than the light source estimation device 200. The receiving unit 203 acquires and receives the scene configuration information and the processed image information from the input unit 202. The light source estimation unit 208 acquires the scene composition information and the image-processed image information from the reception unit 203. The light source estimation unit 208 performs light source estimation based on the scene composition information and the image-processed image information using a method for estimating a light source from an RGB image. By configuring in this manner, light source estimation apparatus 200 can omit object recognition unit 103, area detection unit 104, scene configuration information generation unit 105, and image processing unit 106 compared to light source estimation apparatus 100. Therefore, light source estimation apparatus 200 can reduce the processing load compared to light source estimation apparatus 100.

[0036] The reception unit 203 and the light source estimation unit 208 are realized by, for example, a hardware processor such as a CPU executing a computer program (software) stored in the storage unit 210. Furthermore, some or all of these functional units may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware.

[0037] The computer program may be stored in advance in a storage device such as a HDD or flash memory, or may be stored on a removable storage medium such as a DVD or CD-ROM and installed by inserting the storage medium into a drive device.

[0038] (Operation of the light source estimation device 200) FIG. 6 is a diagram showing an example of the operation of the light source estimation apparatus 200 according to this embodiment. (Step S1-2) The input unit 202 receives scene configuration information and processed image information. (Step S2-2) The receiving unit 203 acquires and receives the scene configuration information and the processed image information from the input unit 202.

[0039] (Step S3-2) The light source estimation unit 208 acquires the scene composition information and the image-processed image information from the reception unit 203. The light source estimation unit 208 performs light source estimation based on the scene composition information and the image-processed image information using a method for estimating a light source from an RGB image. (Step S4-2) The output unit 209 acquires the light source estimation value from the light source estimation unit 208 and outputs it.

[0040] (Second embodiment) Color constancy can be generalized as the shades of gray equation (1) (see Reference 1).

number

[0041] The parameter p is the relative weight of multiple measurements used to estimate the light source. An example of a measurement is a pixel value. The higher the value of the parameter p, the more emphasized the measurement with a large value, and the lower the value, the more uniform the weighting between the measurements. The parameter p is 1 when the measurement values ​​are treated uniformly, and ∞ (infinity) when it takes the maximum value. The parameter σ is the scale of the local measurement. If the parameter n is 1 or greater, the parameter σ is combined with a derivative operation calculated by Gaussian differentiation, and if the parameter n is 0, the parameter σ is imposed by a Gaussian smoothing operation. By obtaining estimates of the parameters n, p, and σ, the parameters used in the illuminant estimation technique can be determined.

[0042] [Light source estimation device] 7 is a diagram showing an example of the configuration of a light source estimation apparatus 300 according to a second embodiment of the present invention. The light source estimation apparatus 300 is realized as a smartphone, a mobile terminal, a personal computer, a tablet terminal device, a wristwatch-type terminal device, or other information processing equipment. The light source estimation apparatus 300 includes, for example, an input unit 302, an object recognition unit 303, an area detection unit 304, a scene structure information generation unit 305, an image processing unit 306, a parameter determination unit 307, a light source estimation unit 308, an output unit 309, and a storage unit 310.

[0043] The input unit 302, object recognition unit 303, area detection unit 304, scene structure information generation unit 305, image processing unit 306, output unit 309, and memory unit 310 can respectively be implemented as the input unit 102, object recognition unit 103, area detection unit 104, scene structure information generation unit 105, image processing unit 106, output unit 109, and memory unit 110.

[0044] The parameter determination unit 307 acquires scene configuration information from the scene configuration information generation unit 305. Based on the acquired scene configuration information, the parameter determination unit 307 estimates parameters used in a method of estimating a light source from an RGB image, which is applied to the light source estimation unit 308. For example, the parameter determination unit 307 estimates the parameters n, p, and σ based on the scene configuration information. Specifically, the parameter determination unit 307 may store in advance a look-up table (LUT) of parameters linked to scene configuration information such as object recognition, and acquire specified parameters based on the scene configuration information. For example, if the parameter determination unit 307 can identify a tree, a building, and a lake through object recognition, it determines that the image is an outdoor image and acquires the parameters (n, p, σ). The parameter determination unit 307 inputs the parameter estimation results, including the estimated values ​​of the parameters n, p, and σ, to the image processing unit 306 and the light source estimation unit 308.

[0045] The image processing unit 306 acquires image information from the input unit 302 and acquires parameter estimation results from the parameter determination unit 307. The image processing unit 306 creates image information for light source estimation by performing image processing based on the acquired image information and parameter estimation results. For example, the image processing unit 306 may include a correspondence table that associates multiple parameters (n, p, σ) with an algorithm used to derive an illuminant estimate for each of the multiple parameters (n, p, σ). For example, the correspondence table associates the parameters (0, 1, 0) with the Gray World algorithm, the parameters (0, ∞, 0) with the Max RGB algorithm, and the parameters (1, p, σ) with the Gray Edge algorithm.

[0046] The image processing unit 306 acquires information indicating an algorithm used to derive a light source estimate from the correspondence table based on the estimated values ​​of parameters n, p, and σ included in the parameter estimation results. The image processing unit 306 creates image information for light source estimation based on the acquired algorithm used to derive the light source estimate. The image processing unit 306 inputs the created image-processed image information to the light source estimation unit 308.

[0047] The light source estimation unit 308 acquires scene construction information from the scene construction information generation unit 305, acquires parameter estimation results from the parameter determination unit 307, and acquires image-processed image information from the image processing unit 306. The light source estimation unit 308 performs light source estimation based on the scene construction information, parameter estimation results, and image-processed image information using a method for estimating a light source from an RGB image.

[0048] Specifically, the light source estimation unit 308 derives the area ratio of each of the multiple detection areas included in the image based on recognition information of one or more objects included in the scene composition information and information identifying one or more detection areas. The light source estimation unit 308 estimates the light source from the R (red), G (green), and B (blue) image using one of the gray world algorithm, max RGB algorithm, and gray edge algorithm, based on the derived results of the area ratio of each of the multiple detection regions.

[0049] For example, for each of the multiple detection areas included in the image, the light source estimation unit 308 derives either the average of the pixels in the image (average pixel value), the maximum value of all pixels in the image (maximum value of all pixel values), or the average of pixels in the image that are considered to be achromatic (average pixel value). The light source estimation unit 308 multiplies the reciprocal of the area ratio by either the average of the image pixels derived for each of the multiple detection regions included in the image, the maximum value of all the pixels in the image, or the average of the pixels that are considered to be achromatic in the image. This reduces the influence of colors with a large area on the light source value. The output unit 309 acquires information indicating the light source estimation value from the light source estimation unit 308 and outputs it.

[0050] The object recognition unit 303, area detection unit 304, scene configuration information generation unit 305, image processing unit 306, parameter determination unit 307, and light source estimation unit 308 are realized, for example, by a hardware processor such as a CPU executing a computer program (software) stored in the memory unit 310. Furthermore, some or all of these functional units may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware.

[0051] The computer program may be stored in advance in a storage device such as a HDD or flash memory, or may be stored on a removable storage medium such as a DVD or CD-ROM and installed by inserting the storage medium into a drive device.

[0052] (Operation of the light source estimation device 300) FIG. 8 is a diagram showing an example of the operation of the light source estimation apparatus 300 according to this embodiment. (Step S1-3) Image information is input to the input unit 302 . (Step S2-3) The object recognition unit 303 acquires image information from the input unit 302, and recognizes one or more objects included in the image based on the acquired image information.

[0053] (Step S3-3) The area detection unit 304 acquires the recognition results (object recognition information) of one or more objects from the object recognition unit 303, and detects one or more detection areas from the image based on the acquired object recognition information. (Step S4-3) The scene construction information generation unit 305 acquires object recognition information from the object recognition unit 303, and acquires information identifying one or more detection areas (detection area information) from the area detection unit 304. The scene construction information generation unit 305 creates scene construction information including the acquired object recognition information and detection area information.

[0054] (Step S5-3) The parameter determination unit 307 acquires scene configuration information from the scene configuration information generation unit 305. Based on the acquired scene configuration information, the parameter determination unit 307 estimates parameters used in a method of estimating a light source from an RGB image, which is applied to the light source estimation unit 308. (Step S6-3) The image processing unit 306 acquires image information from the input unit 302 and acquires parameter estimation results from the parameter determination unit 307. The image processing unit 306 performs image processing based on the acquired image information and image information to create image information for light source estimation (image-processed image information).

[0055] (Step S7-3) The light source estimation unit 308 acquires the parameter estimation results from the parameter determination unit 307, acquires scene structure information from the scene structure information generation unit 305, and acquires image-processed image information from the image processing unit 306. The light source estimation unit 308 performs light source estimation based on the parameter estimation results, scene structure information, and image-processed image information using a method for estimating a light source from an RGB image. (Step S8-3) The output unit 309 acquires the light source estimation value from the light source estimation unit 308 and outputs it.

[0056] (Modification of the second embodiment) MSGP is a statistical color constancy method that relies on a novel gray pixel detection and mean-shift clustering (see Reference 2). MSGP is based on the observation that true gray pixels are aligned toward one direction. The MSGP algorithm includes the following flow: (1) Calculate the "achromaticity" of all pixels (2) Extract the top N% of gray pixels based on the grayness value. (3) Cluster the extracted gray pixels with a clustering bandwidth h. (4) The center of gravity of the most important cluster is the light source estimate.

[0057] The MSGP algorithm depends on two parameters: the percentage N [%] of gray pixels selected from the grayness value, and the clustering bandwidth h. By estimating the pixel fraction N and the clustering bandwidth h, the parameters used in the illuminant estimation technique can be determined.

[0058] [Light source estimation device] 9 is a diagram showing an example of the configuration of a light source estimation apparatus 400 according to a modified example of the second embodiment of the present invention. The light source estimation apparatus 400 is realized as a smartphone, a mobile terminal, a personal computer, a tablet terminal device, a wristwatch-type terminal device, or other information processing equipment. The light source estimation apparatus 400 includes, for example, an input unit 402, an object recognition unit 403, an area detection unit 404, a scene structure information generation unit 405, an image processing unit 406, a parameter determination unit 407, a light source estimation unit 408, an output unit 409, and a storage unit 410.

[0059] The input unit 402, object recognition unit 403, area detection unit 404, scene structure information generation unit 405, image processing unit 406, output unit 409, and memory unit 410 can respectively be implemented as the input unit 102, object recognition unit 103, area detection unit 104, scene structure information generation unit 105, image processing unit 106, output unit 109, and memory unit 110. The parameter determination unit 407 acquires scene configuration information from the scene configuration information generation unit 405. Based on the acquired scene configuration information, the parameter determination unit 407 estimates parameters used in a method of estimating a light source from an RGB image, which is applied to the light source estimation unit 408.

[0060] For example, the parameter determination unit 407 estimates the pixel proportion N and the clustering bandwidth h based on the scene configuration information. Specifically, the parameter determination unit 407 may store in advance a lookup table of parameters associated with scene configuration information such as object recognition, and acquire specified parameters based on the scene configuration information. For example, if the result of object recognition shows that a certain object occupies more than half of the screen, the parameter determination unit 407 acquires parameters (N, h) associated with the certain object. The parameter determination unit 407 inputs the parameter estimation results, including the estimated values ​​of the pixel proportion N and the clustering bandwidth h, to the image processing unit 406 and the light source estimation unit 408.

[0061] The image processing unit 406 acquires image information from the input unit 402 and acquires parameter estimation results from the parameter determination unit 407. The image processing unit 406 creates image information for light source estimation by performing image processing based on the acquired image information and parameter estimation results. For example, the image processing unit 406 generates image information for MSGP based on the proportion N of pixels included in the parameter estimation result and the estimated value of the clustering bandwidth h. The image processing unit 406 inputs the generated image-processed image information to the light source estimation unit 408.

[0062] The light source estimation unit 408 acquires scene structure information from the scene structure information generation unit 405, acquires parameter estimation results from the parameter determination unit 407, and acquires image-processed image information from the image processing unit 406. The light source estimation unit 408 performs light source estimation from the RGB image using the MSGP algorithm based on the scene structure information, parameter estimation results, and image-processed image information.

[0063] Specifically, the light source estimation unit 408 derives the area ratio of each of the multiple detection areas included in the image based on recognition information of one or more objects included in the scene composition information and information identifying one or more detection areas. The light source estimation unit 408 estimates the light source from the R (red), G (green), and B (blue) image using the MSGP algorithm, based on the results of deriving the area ratio for each of the multiple detection regions. The light source estimation unit 408 derives the average of pixels that are considered to be achromatic in the image for each of the multiple detection regions included in the image. The light source estimation unit 408 multiplies the average of pixels that are considered to be achromatic in the image, derived for each of the multiple detection regions included in the image, by the reciprocal of the area ratio. This reduces the influence of large-area colors on the light source value. The output unit 409 acquires information indicating the light source estimation value from the light source estimation unit 408 and outputs it.

[0064] The object recognition unit 403, area detection unit 404, scene configuration information generation unit 405, image processing unit 406, parameter determination unit 407 and light source estimation unit 408 are realized, for example, by a hardware processor such as a CPU executing a computer program (software) stored in the memory unit 410. Furthermore, some or all of these functional units may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware.

[0065] The computer program may be stored in advance in a storage device such as a HDD or flash memory, or may be stored on a removable storage medium such as a DVD or CD-ROM and installed by inserting the storage medium into a drive device. The operation of light source estimation apparatus 400 can be explained using FIG. 8, and therefore a description thereof will be omitted.

[0066] (Third embodiment) In the third embodiment, pixels that can be used for light source estimation are detected by performing machine learning in advance using a training dataset. When an image is divided into small regions, it is determined whether the information in each region is highly reliable for estimating the light source, and weighting is performed based on the reliability determination results (see Reference 3).

[0067] [Light source estimation device] 10 is a diagram showing an example of the configuration of a light source estimation apparatus 500 according to the third embodiment of the present invention. The light source estimation apparatus 500 is realized as a smartphone, a mobile terminal, a personal computer, a tablet terminal device, a wristwatch-type terminal device, or other information processing equipment. The light source estimation apparatus 500 includes, for example, an input unit 502, an object recognition unit 503, a scene structure information generation unit 505, an image processing unit 506, a parameter determination unit 507, a light source estimation unit 508, an output unit 509, and a storage unit 510.

[0068] The input unit 502, object recognition unit 503, scene structure information generation unit 505, image processing unit 506, output unit 509, and memory unit 510 can be implemented as the input unit 102, object recognition unit 103, scene structure information generation unit 105, image processing unit 106, output unit 109, and memory unit 110, respectively. The scene construction information generation unit 505 acquires object recognition information (object recognition information) that is the recognition result of one or more objects included in the image from the object recognition unit 503. The scene construction information generation unit 505 creates scene construction information including the acquired object recognition information. The scene construction information generation unit 505 inputs the created scene construction information to the parameter determination unit 507 and the light source estimation unit 508. The parameter determination unit 507 acquires scene structure information from the scene structure information generation unit 505. Based on the acquired scene structure information, the parameter determination unit 507 determines a trained model to be used in a method of estimating a light source from an RGB image, which is applied to the light source estimation unit 508.

[0069] For example, the parameter determination unit 507 determines which of multiple trained models to use based on object recognition information included in the scene configuration information. For example, the multiple trained models may include a trained model created by machine learning a dataset of images taken indoors (hereinafter also referred to as an "indoor trained model") and a trained model created by machine learning a dataset of images taken outdoors (hereinafter also referred to as an "outdoor trained model"). The parameter determination unit 507 inputs the determination result of the trained model to the light source estimation unit 508. Here, an example of the determination result of the trained model is information indicating the trained model to be used.

[0070] The image processing unit 506 acquires image information from the input unit 502. The image processing unit 506 performs image processing based on the acquired image information to create image information for light source estimation (image processed image information). The image processing unit 506 inputs the created image processed image information to the light source estimation unit 508.

[0071] The light source estimation unit 508 acquires scene configuration information from the scene configuration information generation unit 505, acquires the determination result of the trained model from the parameter determination unit 507, and acquires processed image information from the image processing unit 506. The light source estimation unit 508 is configured to include multiple trained models. The multiple trained models include an indoor trained model and an outdoor trained model. The indoor trained model is created by machine learning the relationship between an image taken indoors as the objective variable and pixels contained in the image that are effective for estimating a light source, based on an indoor training dataset that includes images taken indoors as training data and pixels contained in the image that are effective for estimating a light source (pixels with a high reliability above a predetermined threshold) as training data.

[0072] The outdoor trained model is created by machine learning the relationship between an image taken outdoors as the objective variable and pixels contained in the image that are effective for estimating a light source (pixels with a high reliability above a predetermined threshold) based on an outdoor training dataset that includes images taken outdoors as training data and pixels contained in the image that are effective for estimating a light source (pixels with a high reliability above a predetermined threshold) as training data.

[0073] The light source estimation unit 508 selects a trained model to use from the multiple trained models based on the acquired trained model determination result. The light source estimation unit 508 inputs the acquired image processed image information into the selected trained model and acquires information indicating pixels effective for light source estimation output by the trained model. The light source estimation unit 508 performs weighting based on the acquired information indicating images effective for light source estimation. For example, the light source estimation unit 508 assigns a higher weight to pixels effective for light source estimation than to pixels other than pixels effective for light source estimation.

[0074] The light source estimation unit 508 derives the area ratio of each of the multiple detection regions included in the image based on the recognition information of one or more objects included in the scene configuration information and the information identifying one or more detection regions. The light source estimation unit 508 estimates the light source from the R (red), G (green), and B (blue) image based on the derived results of the area ratio for each of the multiple detection regions. The light source estimation unit 508 derives the results of weighting the pixel values ​​of the image pixels for each of the multiple detection regions included in the image. The light source estimation unit 508 multiplies the results of weighting the pixel values ​​of the image pixels derived for each of the multiple detection regions included in the image by the reciprocal of the area ratio. This makes it possible to reduce the influence of the light source value on colors with a large area. The output unit 509 acquires information indicating the light source estimation value from the light source estimation unit 508 and outputs it.

[0075] The object recognition unit 503, the scene configuration information generation unit 505, the image processing unit 506, the parameter determination unit 507, and the light source estimation unit 508 are realized, for example, by a hardware processor such as a CPU executing a computer program (software) stored in the memory unit 510. Furthermore, some or all of these functional units may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware.

[0076] The computer program may be stored in advance in a storage device such as a HDD or flash memory, or may be stored on a removable storage medium such as a DVD or CD-ROM and installed by inserting the storage medium into a drive device.

[0077] (Operation of the light source estimation device 500) FIG. 11 is a diagram showing an example of the operation of the light source estimation apparatus 500 according to this embodiment. (Step S1-4) Image information is input to the input unit 502 . (Step S2-4) The object recognition unit 503 acquires image information from the input unit 502, and recognizes one or more objects included in the image based on the acquired image information.

[0078] (Step S3-4) The scene construction information generation unit 505 acquires the object recognition information from the object recognition unit 503. The scene construction information generation unit 505 creates scene construction information including the acquired object recognition information. (Step S4-4) The parameter determination unit 507 acquires scene configuration information from the scene configuration information generation unit 505. Based on the object recognition information included in the acquired scene configuration information, the parameter determination unit 507 determines a trained model to be used in a method of estimating a light source from an RGB image, which is applied to the light source estimation unit 508. (Step S5-4) The image processing unit 506 acquires image information from the input unit 502. The image processing unit 506 processes the acquired image information to create image information for light source estimation.

[0079] (Step S6-4) The light source estimation unit 508 acquires the judgment result of the trained model from the parameter judgment unit 507, acquires scene composition information from the scene composition information generation unit 505, and acquires image-processed image information from the image processing unit 506. The light source estimation unit 508 performs light source estimation using a method for estimating a light source from an RGB image, based on the judgment result of the trained model, the scene composition information, and the image-processed image information. (Step S7-4) The output unit 509 acquires the light source estimation value from the light source estimation unit 508 and outputs it.

[0080] The light source estimation device according to this embodiment includes a reception unit that receives recognition information of an object including a background included in an image, scene composition information including the image, and image-processed image information obtained by performing predetermined image processing on the image, and a light source estimation unit that performs light source estimation based on the scene composition information received by the reception unit and the image-processed image information.

[0081] With this configuration, the light source estimation device can estimate the light source from the input image, eliminating the need to acquire light source information in advance. This makes it easier to apply color correction techniques compared to when light source information is acquired in advance. Furthermore, because scene composition information can be used for light source estimation, the accuracy of light source estimation can be improved compared to when scene composition information is not used.

[0082] The light source estimation device further includes an image processing unit that performs image processing on the image based on a light source estimation method performed by the light source estimation unit. The light source estimation unit performs light source estimation based on image-processed image information obtained by image processing performed by the image processing unit.

[0083] By configuring the light source estimation device in this manner, image processing can be performed based on the light source estimation method performed by the light source estimation unit, thereby reducing the effort required for image processing compared to performing predetermined image processing on an image externally.

[0084] The light source estimation device further includes an object recognition unit that recognizes an object included in the image from the image, and a scene composition information generation unit that generates scene composition information based on the recognition information of the object recognized by the object recognition unit. The light source estimation unit performs light source estimation based on the scene composition information generated by the scene composition information generation unit.

[0085] By configuring the light source estimation device in this manner, it is possible to recognize objects contained in an image from the image, generate scene composition information based on the recognized objects, and estimate the light source based on the generated scene composition information, thereby reducing the effort required to recognize objects contained in an image compared to recognizing objects contained in an image from an external image.

[0086] The light source estimation device further includes an object recognition unit that recognizes an object included in an image from the image, an area detection unit that detects a plurality of detection areas from the image based on recognition information of the object recognized by the object recognition unit, and a scene composition information generation unit that generates scene composition information based on the recognition information of the object recognized by the object recognition unit and the plurality of detection areas detected by the area detection unit. The light source estimation unit performs light source estimation based on the scene composition information generated by the scene composition information generation unit.

[0087] By configuring the light source estimation device in this manner, it is possible to recognize objects contained in an image from the image, detect multiple detection areas from the image based on the object recognition results, and generate scene composition information based on the object recognition information and the multiple detection areas. Therefore, compared to externally recognizing objects contained in an image from the image and detecting multiple detection areas based on the object recognition results, it is possible to save the effort of recognizing objects contained in an image from the image and dividing them into multiple detection areas based on the object recognition results.

[0088] The light source estimation device further includes a parameter determination unit that determines parameters to be used when light source estimation is performed by the light source estimation unit based on the scene configuration information. The light source estimation unit performs light source estimation based on the parameters determined by the parameter determination unit.

[0089] With this configuration, the light source estimation device can determine the parameters to be used when light source estimation is performed by the light source estimation unit, and therefore can estimate the light source using parameters that are appropriate for the light source estimation unit, compared to when the parameters are not determined, thereby improving the accuracy of light source estimation.

[0090] The image processing system according to this embodiment includes a reception unit that receives scene composition information including recognition information of objects including a background included in an image, and image-processed image information obtained by performing predetermined image processing on the image, and a light source estimation unit that performs light source estimation based on the scene composition information received by the reception unit and the image-processed image information.

[0091] With this configuration, the image processing system can estimate the light source from the input image, eliminating the need to acquire light source information in advance. This makes it easier to apply color correction techniques compared to when light source information is acquired in advance. Furthermore, because scene composition information can be used for light source estimation, the accuracy of light source estimation can be improved compared to when scene composition information is not used.

[0092] Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and design changes and the like are also included within the scope of the gist of the present invention. For example, the light source estimation devices 100 according to the first embodiment to the light source estimation device 500 according to the third embodiment may be combined as appropriate. Furthermore, for example, a computer program for realizing the functions of each of the above-described devices may be recorded on a computer-readable recording medium, and the computer program recorded on the recording medium may be read and executed by a computer system. Note that the "computer system" here may also include hardware such as an OS and peripheral devices.

[0093] In addition, "computer-readable recording medium" refers to writable non-volatile memory such as a flexible disk, optical magnetic disk, ROM, or flash memory, portable media such as a DVD (Digital Versatile Disc), or a storage device such as a hard disk built into a computer system. Furthermore, the term "computer-readable recording medium" also includes those that retain a program for a certain period of time, such as volatile memory (e.g., DRAM (Dynamic Random Access Memory)) within a computer system that serves as a server or client when a computer program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0094] The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be for realizing part of the above-mentioned functions. Furthermore, the above-mentioned functions may be realized in combination with a program already recorded in the computer system, that is, a so-called differential file (differential program).

[0095] Reference 1: Joost. van de Weijer, Theo. Gevers and Arjan. Gijsenij, “Edge-Based Color Constancy”, IEEE Transactions on Image Processing, vol. 16, no. 9, pp. 2207-2214, Sept. 2007, doi: 10.1109 / TIP.2007.901808. Reference 2: Yanlin Qian, Said Pertuz, Jarno Nikkanen, Joni-Kristian Kamarainen and Jiri Matas, “Revisiting Gray Pixel for Statistical Illumination Estimation,” arXiv:1803.08326 Document 3: Yuanming Hu, Baoyuan Wang and Stephen Lin, “FC^4: Fully Convolutional Color Constancy with Confidence-Weighted Pooling,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 330-339, doi: 10.1109 / CVPR.2017.43. [Explanation of symbols]

[0096] 100, 200, 300, 400, 500 Light source estimation device, 102, 202, 302, 402, 502 Input unit, 103, 303, 403, 503 Object recognition unit, 203 Reception unit, 104, 304, 404 Area detection unit, 105, 305, 405, 505 Scene configuration information generation unit, 106, 306, 406, 506 Image processing unit, 307, 407, 507 Parameter determination unit, 108, 208, 308, 408, 508 Light source estimation unit, 109, 209, 309, 409, 509 Output unit, 110, 210, 310, 410, 510 Storage unit

Claims

1. a receiving unit that receives scene configuration information including recognition information of an object including a background included in an image, and image-processed image information obtained by performing predetermined image processing on the image; a light source estimation unit that estimates a light source based on the scene configuration information received by the reception unit and the image-processed image information; A light source estimation device comprising:

2. an image processing unit that performs image processing on the image based on a light source estimation method performed by the light source estimation unit; Furthermore, The light source estimation device according to claim 1 , wherein the light source estimation unit estimates the light source based on image-processed image information obtained by image processing performed by the image processing unit.

3. an object recognition unit that recognizes an object included in the image from the image; a scene construction information generation unit that generates scene construction information based on recognition information of the object recognized by the object recognition unit; Furthermore, The light source estimation device according to claim 1 , wherein the light source estimation unit estimates a light source based on the scene configuration information generated by the scene configuration information generation unit.

4. an object recognition unit that recognizes an object included in the image from the image; an area detection unit that detects a plurality of detection areas from the image based on recognition information of the object recognized by the object recognition unit; a scene construction information generation unit that generates scene construction information based on recognition information of the object recognized by the object recognition unit and a plurality of detection areas detected by the area detection unit; Furthermore, The light source estimation device according to claim 1 , wherein the light source estimation unit estimates a light source based on the scene configuration information generated by the scene configuration information generation unit.

5. a parameter determination unit that determines parameters to be used when the light source estimation unit performs light source estimation based on the scene configuration information; Furthermore, The light source estimation device according to claim 1 , wherein the light source estimation unit estimates the light source based on the parameters determined by the parameter determination unit.

6. a receiving unit that receives scene configuration information including recognition information of an object including a background included in an image, and image-processed image information obtained by performing predetermined image processing on the image; a light source estimation unit that estimates a light source based on the scene configuration information received by the reception unit and the image-processed image information; An image processing system comprising:

7. 1. A computer-implemented method for illuminant estimation, comprising: receiving scene configuration information including recognition information of an object including a background included in an image, and image-processed image information obtained by performing predetermined image processing on the image; a step of performing light source estimation based on the scene configuration information received in the receiving step and the image-processed image information; A light source estimation method comprising:

8. On the computer, receiving scene configuration information including recognition information of an object including a background included in an image, and image-processed image information obtained by performing predetermined image processing on the image; a step of performing light source estimation based on the scene configuration information received in the receiving step and the image-processed image information; A program that executes.

Images