Light Source Estimation System

Abstract

To provide a light source estimation system for retrieving appropriate light source setting to a three-dimensional model by synchronizing a photographic parameter about the brightness of a rendering image with setting of a camera.SOLUTION: In a light source estimation system, a server 200 includes a first acquisition part 220 for acquiring a first picked-up image of an imaging area under a prescribed light source, an adjustment part 230 for performing light source setting corresponding to the prescribed light source in a three-dimensional model 211 obtained by modeling the imaging area, and adjusting a photographic parameter about the brightness of a first rendering image so as to reduce difference between the first rendering image generated from the three-dimensional model 211 and the first picked-up image, a second acquisition part 240 for acquiring a second picked-up image under an unknown light source, and a retrieval part 250 for retrieving light source setting corresponding to the unknown light source so as to reduce difference between a second rendering image generated from the three-dimensional model 211 and the second picked-up image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a light source estimation system. [Background technology]

[0002] Patent Document 1 discloses a method for accurately estimating a lighting environment using an image captured at a single viewpoint with a wide viewing angle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-149679 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present applicant has found the following problem. When setting a light source for a 3D model that models a real environment, there are cases where it is desirable to set the light source so as to reproduce an actual light source. However, since a captured image is affected not only by the light source but also by the shutter speed and gain of the camera, it has been difficult to find an optimal light source setting.

[0005] The present disclosure has been made in consideration of such problems, and realizes a light source estimation system that synchronizes shooting parameters related to the brightness of a rendered image with camera settings and searches for appropriate light source settings for a three-dimensional model. [Means for solving the problem]

[0006] A light source estimation system according to an embodiment of the present disclosure includes: a first acquisition unit that acquires a first captured image of an imaging area under a specified light source; an adjustment means for setting a light source corresponding to the specified light source in a three-dimensional model that models the imaging area, and adjusting a shooting parameter related to brightness of the first rendering image so that a difference between a first rendering image generated from the three-dimensional model and the first captured image becomes small; a second acquisition means for acquiring a second captured image of the imaging area under an unknown light source; a search means for searching for a light source setting corresponding to the unknown light source so that a difference between a second rendering image generated from the three-dimensional model and the second captured image becomes small after adjusting the camera parameters; Equipped with. Effect of the Invention

[0007] According to the present disclosure, it is possible to realize a light source estimation system that synchronizes shooting parameters related to the brightness of a rendered image with camera settings and searches for appropriate light source settings for a three-dimensional model. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of a light source estimation system according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing a configuration of a server according to the first embodiment. [Diagram 3] 13 is a flowchart showing the flow of a material setting method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, specific embodiments to which the present disclosure is applied will be described in detail with reference to the drawings. However, the present disclosure is not limited to the following embodiments. In addition, the following description and drawings are appropriately simplified for clarity of explanation.

[0010] EMBODIMENT 1 Hereinafter, a light source estimation system according to the first embodiment will be described with reference to the drawings. Fig. 1 is a block diagram showing a configuration of a light source estimation system 1000 according to the first embodiment. The light source estimation system 1000 includes a camera 100 and a server 200. The camera 100 and the server 200 are connected to each other via a network N so as to be able to communicate with each other.

[0011] The camera 100 is installed in a house 10 or the like. There may be more than one camera. The camera 100 may be placed, for example, in a corner of a room. The camera 100 is also called a home sensor. The camera 100 captures an image of an imaging area 20. The camera 100 outputs the captured image to a server 200 via a network N.

[0012] The imaging area 20 captured by the camera 100 is three-dimensionally modeled, and the server 200 stores the three-dimensional model. For example, a sink and furniture (e.g., a sofa, a shelf, and a desk) are arranged in the imaging area 20. A plurality of pieces of furniture may be arranged in the imaging area 20. The imaging area 20 may also include walls and a floor.

[0013] The camera 100 has a function for changing settings related to the brightness of a captured image. In other words, the camera 100 has a function for adjusting the shutter speed, gain, aperture, etc. It is assumed that the position and orientation of the camera 100 are known by camera calibration.

[0014] The imaging area 20 is illuminated by a light source 30. The light source 30 includes an artificial light source such as a fluorescent light, an LED (Light Emitting Diode) light source, or a lamp. The artificial light source 30 is configured to be switchable between a turned-on state and a turned-off state. The artificial light source 30 may be configured to be adjustable in brightness. The three-dimensional model of the imaging area 20 may include an object (referred to as a light source object) representing the artificial light source 30.

[0015] Additionally, the light source 30 may include light from natural sources such as the sun or the moon. The room including the imaging area 20 may be equipped with windows and curtains.

[0016] The server 200 searches for an appropriate light source setting for a three-dimensional model that models the imaging area 20. The server 200 includes a calculation unit such as a CPU (Central Processing Unit) and a storage unit such as a RAM (Random Access Memory) and a ROM (Read Only Memory) that store various control programs, data, etc. In other words, the server 200 has a function as a computer and performs processing based on the various control programs, etc.

[0017] The processing by the server 200 may be executed on the camera 100 side. It is assumed that the camera 100 includes a processor, a memory, etc. Therefore, the light source estimation system 1000 that does not include the server 200 may also be included in the first embodiment.

[0018] Next, the functions of the server 200 will be described in detail with reference to Fig. 2. The server 200 includes a storage unit 210, a first acquisition unit 220, an adjustment unit 230, a second acquisition unit 240, a search unit 250, a change unit 260, an update unit 270, and a setting unit 280.

[0019] The storage unit 210 stores a three-dimensional model 211 of the imaging area 20. The server 200 has a function of generating a rendering image from the three-dimensional model 211. The server 200 has a function of setting shooting parameters (e.g., aperture, exposure time, gain) related to the brightness of the rendering image.

[0020] The three-dimensional model 211 includes a sink, furniture, a floor, a wall, and the like as objects. The geometry of each object is assumed to be known. A material is set for each object in the three-dimensional model 211 by a setting unit 280, which will be described later. A material means a quality that is set for an object. By setting a material for an object, the optical characteristics of the furniture, floor, wall, and the like are expressed.

[0021] The server 200 also has a function for setting a light source in the three-dimensional model 211. The light source may be, for example, a point light source, a line light source, a surface light source, a parallel light source, a spotlight, or an ambient light. The ambient light may be generated using an environment map that controls the color and intensity of light from different directions. The server 200 has a function for setting the type, position, direction, brightness, and the like of the light source.

[0022] The three-dimensional model 211 may include a light source object that corresponds to the artificial light source 30. The light source object may be configurable to be on or off, and the brightness may be configurable.

[0023] Moreover, the storage unit 210 stores the above-mentioned control program (not shown).

[0024] The first acquisition unit 220 acquires a first captured image of the imaging area 20 under a specified light source. The first captured image is captured, for example, with only a predetermined artificial light source 30 (for example, a fluorescent light installed on the ceiling) turned on. In order to eliminate the influence of the natural light source 30, the first captured image may be captured at night. The first captured image may also be captured with all the artificial light sources 30 turned off.

[0025] The adjustment unit 230 performs light source setting corresponding to a specified light source in the three-dimensional model 211. In other words, the adjustment unit 230 performs light source setting that is preset so as to reproduce a specified light source. For example, the adjustment unit 230 places a light source (e.g., a point light source) of a specified brightness at a specified position in the three-dimensional model 211. The adjustment unit 230 may set a light source object corresponding to a specified artificial light source 30 to a turned-on state. When the first captured image is captured in a state in which all the artificial light sources 30 are turned off, the adjustment unit 230 may set an ambient light of a specified brightness.

[0026] Then, the adjustment unit 230 adjusts the shooting parameters related to the brightness of the first rendering image so that the difference between the first rendering image generated from the three-dimensional model 211 and the first captured image becomes small.

[0027] The photographing parameters relating to brightness are, specifically, the aperture, the exposure time, and the gain. The photographing parameters are also called image photographing parameters or camera parameters.

[0028] The difference described above may be a difference in average luminance, a difference in average brightness, a difference in luminance distribution, or a difference in pixel values ​​of each pixel. The pixel values ​​may be RGB (Red Green Blue) values ​​or gray values. The difference may be a square sum of the differences in each pixel value. The adjustment unit 230 may generate a binary image by binarizing each of the rendering image and the captured image with a predetermined threshold value, and adjust the shooting parameters so that the difference between the binary images is reduced. The adjustment unit 230 may adjust the camera parameters so that the similarity between the two images is increased by using a machine learning model that outputs the similarity between the two images.

[0029] The second acquisition unit 240 acquires a second captured image of the imaging area 20 under an unknown light source. The second captured image is captured, for example, in a state where a user living in the house 10 turns on any artificial light source 30 or in a state where the house 10 is illuminated by a natural light source 30.

[0030] After the adjustment unit 230 adjusts the shooting parameters, the search unit 250 searches for a light source setting corresponding to an unknown light source so that the difference between the second rendering image generated from the three-dimensional model 211 and the second captured image is small. That is, the search unit 250 sets the second captured image as a target image, sets the light source setting in the three-dimensional model 211 as a parameter (e.g., the position, direction, and brightness of the light source), and searches for parameters that minimize the above difference. The search unit 250 may search an environment map that controls the color and brightness of light from different directions. Like the adjustment unit 230, the search unit 250 can search for a light source setting when the difference is small based on the luminance average, luminance distribution, the difference in pixel value of each pixel, the sum of squares of the difference in each pixel value, a binary image, similarity, and the like.

[0031] Also, a technique called a differentiable renderer is known that searches for light source settings in a three-dimensional space from a two-dimensional image. The differentiable renderer is a technique that uses geometry, materials, light sources, and shooting parameters as variables to generate an image similar to a target image from a noise image (initial values). In the differentiable renderer, the least square error between the created image and the target image may be used as a loss function. The search unit 250 may use the differentiable renderer with an environment map and light sources as parameters to search for light source settings.

[0032] The search unit 250 may search for light source settings in response to a command from a robot or other user device placed in the house 10. For example, when the robot detects a change in brightness in the house 10, the search unit 250 may instruct the server 200 to search for light source settings.

[0033] After searching for the light source setting, the light source setting may be performed in the three-dimensional model 211 based on the search result. In this case, the rendered image generated from the three-dimensional model 211 reflects the actual state of the light source 30. Therefore, a machine learning model with high recognition accuracy can be generated by performing machine learning using the rendered image. The server 200 may add the rendered image to a training image.

[0034] Also, a technique called domain adaptation is known, which adapts a machine learning model to learning data of a different domain. In domain adaptation, knowledge obtained from a source domain with sufficient information is applied to a target domain with insufficient information to generate a machine learning model with high recognition accuracy in the target domain. For example, by performing domain adaptation using an image captured by the camera 100 as the source domain and a rendering image as the target domain, a machine learning model with high recognition accuracy under any light source can be generated.

[0035] The change unit 260 changes the brightness-related settings of the camera 100 (e.g., shutter speed, gain, aperture). For example, when the brightness of an image captured by the camera 100 has changed (e.g., when the number of whiteout pixels or blackout pixels is equal to or greater than a threshold), the change unit 260 sets the automatic exposure function (auto exposure) of the camera 100 to on. This sets an optimal shutter speed or gain for the camera 100. The aperture is assumed to be fixed. After the optimal shutter speed or gain has been set, the change unit 260 locks (stops) the automatic exposure function and fixes the above settings.

[0036] The brightness of the captured image may change when the user turns on or off the artificial light source 30, or when the brightness of the natural light source 30 changes over time. In this case, whiteout or blackout occurs in the captured image, so the change unit 260 changes the settings related to the brightness of the camera 100. The change unit 260 may change the settings related to the brightness of the camera 100, for example, when the number of pixels having a minimum brightness value (e.g., 0) becomes equal to or exceeds a threshold value, or when the number of pixels having a maximum brightness value (e.g., 255) becomes equal to or exceeds a threshold value.

[0037] When the change unit 260 changes the settings related to the brightness of the camera 100, the update unit 270 updates the shooting parameters (e.g., exposure time, gain) related to the brightness of the rendered image. The update unit 270, like the adjustment unit 230, adjusts the shooting parameters related to the brightness of the rendered image so that the difference between the rendered image generated from the three-dimensional model 211 and the captured image becomes small. Like the adjustment unit 230, the update unit 270 updates the shooting parameters based on the difference in average luminance, the difference in average brightness, the difference in luminance distribution, the difference in pixel value (e.g., RGB value, GRAY value) of each pixel, the binary image, the similarity, etc. The shooting parameters related to the brightness of the rendered image may be fixed after updating.

[0038] The setting unit 280 sets materials in advance for each object in the three-dimensional model 211. The flow of processing performed by the setting unit 280 will be described with reference to FIG.

[0039] First, the setting unit 280 generates a rendering image from the three-dimensional model 211 and performs segmentation of the rendering image (step S101). It is assumed that the three-dimensional model 211 has a light source setting according to the state of the light source 30. When the imaging area 20 is illuminated by a specified light source, a light source corresponding to the specified light source is set. When the imaging area 20 is illuminated by an unknown light source, a searched light source is set.

[0040] Next, the setting unit 280 estimates materials of segments with large areas (for example, walls, floors, and sinks) (step S102). The processes of steps S102 to S105 are performed for each segment.

[0041] Next, the setting unit 280 estimates the material of the segment using a machine learning model or the like (step S103). The estimation result is used as an initial value. The setting unit 280 may use, for example, the technique described in Abhimitra Meka, et.al., LIME: Live Intrinsic Material Estimation, CVPR, 2018. Next, the setting unit 280 sets the estimated material to the segment (step S103).

[0042] Next, the setting unit 280 searches for a material setting for the segment so that the difference between the actual image of the portion corresponding to the segment and the above-mentioned segment is small (step S104). The actual image is extracted from an image captured by the camera 100. The difference is expressed by a difference in average luminance, a difference in luminance distribution, a difference in average brightness, a difference in brightness distribution, a difference in pixel value of each pixel (e.g., RGB value, GRAY value), a binary image, etc. The setting unit 280 may set a material so that the similarity output by the machine learning model is large. The setting unit 280 may also search for a material for the segment using a differentiable renderer.

[0043] Next, the setting unit 280 sets the searched material to a segment, calculates the final difference between the actual image of the portion corresponding to the segment and the above-mentioned segment, and calculates the confidence according to the difference (step S105). The confidence conf is, for example, conf=λ*e -aΔ The accuracy is calculated by the formula: where λ and a are constants, and Δ represents the difference. The accuracy may also be calculated using a machine learning model.

[0044] Next, the setting unit 280 determines whether the processing for all segments with large areas among the segments extracted in step S101 has been completed (step S106). If the material has been set for all segments (YES in step S106), the processing ends, and if not (NO in step S106), the processing returns to step S102.

[0045] The light source estimation system according to the first embodiment uses a first captured image captured under a specified light source to synchronize the shooting parameters of a rendering image with the settings of the camera 100. This allows the light source estimation system 1000 to search for a light source in the three-dimensional model 211 with high accuracy. By reproducing the light source of the real environment in the three-dimensional model 211, it is possible to improve the accuracy of object recognition by domain adaptation and generate training images that reflect the actual light source environment.

[0046] The above-mentioned programs include instructions (or software codes) that, when loaded into a computer, cause the computer to perform one or more functions. The programs may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The programs may be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, transitory computer-readable medium or a communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0047] The present disclosure is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present disclosure. [Explanation of symbols]

[0048] 1000 Light Source Estimation System 100 Cameras 200 Servers 210 Storage section 211 3D Model 220 First acquisition part 230 Adjustment section 240 Second acquisition part 250 Search Department 260 Changes 270 Update Department 280 Setting section 10 houses 20 Imaging area 30 light source N Network

Claims

[Claim 1] A first acquisition means for acquiring a first captured image of an imaging area under a specified light source; an adjustment means for setting a light source corresponding to the specified light source in a three-dimensional model that models the imaging area, and adjusting a shooting parameter related to brightness of the first rendering image so that a difference between a first rendering image generated from the three-dimensional model and the first captured image becomes small; a second acquisition means for acquiring a second captured image of the imaging area under an unknown light source; a search means for searching for a light source setting corresponding to the unknown light source so that a difference between a second rendering image generated from the three-dimensional model and the second captured image becomes small after adjusting the shooting parameters; and A light source estimation system comprising:

Images