Information processing device, information processing method, and program

Abstract

An information processing device according to the present technology comprises a light source spectrum estimation unit that uses a light source model for expressing a light source spectrum, which is spectral information of an outdoor light source according to the weighted addition of a solar spectrum which is spectral information of emission light from the sun and a sky spectrum which is spectral information of emission light from the sky, to estimate the light source spectrum on the basis of an object spectrum which is spectral information of a subject captured by a spectroscopic camera.

Description

Information processing device, information processing method, and program

[0001] The present technology relates to an information processing device, an information processing method, and a program, and more particularly to a technology for estimating light source spectral components contained in a spectroscopic image obtained by a spectroscopic camera.

[0002] Spectroscopic sensors (multispectral sensors) are known for obtaining multiple narrowband images that are wavelength characteristic analysis images of light from a subject, in other words, analysis images of the subject's spectral information (spectral spectrum).In addition, applications have been developed that perform various analyses of a subject based on the spectral information obtained by the spectroscopic sensor, such as estimating the vegetation state of plants or the condition of human skin, based on these multiple narrowband images.

[0003] When obtaining spectral information of a subject, the spectral information of the light source illuminating the subject becomes a noise component, and it is desirable to remove this noise component. For this reason, in the field of spectral sensing, estimation of the light source spectrum, which is the spectral information of the light source, is performed.

[0004] For example, Patent Document 1 below discloses an image capturing device including an image sensor that captures a predetermined image, and a processor that acquires a basis based on the surrounding environment, estimates lighting information using the acquired basis, reflects the estimated lighting information, and performs color conversion related to the image.

[0005] JP 2023-048996 A

[0006] However, the technology described in Patent Document 1 employs a method for estimating illumination information (light source spectrum) by preparing a basis for each expected environment in advance and using the basis according to the results of environmental detection by a sensor. As a result, it is not possible to estimate the light source spectrum in response to environmental changes other than those in the expected environment.

[0007] The present technology has been developed in view of the above-mentioned problems, and aims to improve the robustness of light source spectrum estimation.

[0008] The information processing device according to the present technology includes a light source spectrum estimation unit that estimates the light source spectrum based on the object spectrum, which is spectral information of a subject, obtained by a spectroscopic camera, using a light source model that expresses the light source spectrum, which is spectral information of an outdoor light source, by weighted addition of a sunlight spectrum, which is spectral information of light irradiated from the sun, and a sky spectrum, which is spectral information of light irradiated from the sky. This makes it possible to estimate the light source spectrum by fitting a function as the light source model to the object spectrum detected by the spectroscopic camera.

[0009] FIG. 1 is a diagram showing an example of the configuration of a light source spectrum estimation system configured with an information processing device as an embodiment. FIG. 2 is a block diagram showing an example of the schematic configuration of a spectroscopic camera used in an embodiment. FIG. 3 is a diagram schematically showing an example of the configuration of a pixel array unit of a spectroscopic sensor. FIG. 4 is an explanatory diagram of band narrowing processing in an embodiment. FIG. 5 is a block diagram showing an example of the hardware configuration of an information processing device as an embodiment. FIG. 6 is a functional block diagram for explaining functions of an information processing device as a first embodiment. FIG. 7 is a diagram showing examples of a solar spectrum and a sky spectrum. FIG. 8 is a flowchart showing an example of a processing procedure for realizing a light source spectrum estimation method as a first embodiment. FIG. 9 is a block diagram showing an example of the configuration of a light source spectrum estimation system as a second embodiment. FIG. 10 is a functional block diagram for explaining functions of an information processing device as a modified example.

[0010] Hereinafter, with reference to the accompanying drawings, embodiments according to the present technology will be described in the following order: <1. First embodiment> [1-1. System configuration] [1-2. Configuration of spectroscopic camera] [1-3. Configuration of information processing device] [1-4. Functions of information processing device] <2. Second embodiment> <3. Modified example> <4. Summary of embodiments> <5. Present technology>

[0011] 1. First Embodiment [1-1. System Configuration] Fig. 1 is a diagram showing an example of the configuration of a light source spectrum estimation system including an information processing device according to an embodiment. As shown in the figure, the light source spectrum estimation system includes an information processing device 1, a spectroscopic camera 2, and an upward-facing camera 3. Here, the spectroscopic camera 2 refers to a camera equipped with a spectroscopic sensor as a light-receiving sensor. The "spectroscopic sensor" is a light-receiving sensor for obtaining multiple narrowband images that serve as wavelength characteristic analysis images of light from a subject.

[0012] The information processing device 1 is configured as a computer device, and performs processing to estimate a light source spectrum, which is spectral information of a light source, based on spectral information of a subject obtained by a spectroscopic camera 2, as will be described later.

[0013] Here, the spectral information means information indicating the light intensity for each wavelength.

[0014] The information processing device 1 in this embodiment is configured to estimate the light source spectrum of an outdoor light source (natural light source) as the light source spectrum, and estimates the light source spectrum using a light source model that expresses the light source spectrum, which is spectral information of an outdoor light source, by weighted addition of the sunlight spectrum, which is spectral information of light irradiated from the sun, and the sky spectrum, which is spectral information of light irradiated from the sky; details will be described later.

[0015] The upward-facing camera 3 is a camera that captures images of the sky, in other words, a camera whose imaging direction is directed upward, and is capable of obtaining images of the sky. The use of the upward-facing camera 3 will be described later.

[0016] 2 is a block diagram showing an example of the configuration of the spectroscopic camera 2. The spectroscopic camera 2 includes at least a spectroscopic sensor 4 and a spectroscopic image generating unit 5. As shown in the figure, the spectroscopic camera 2 of this embodiment includes, in addition to the spectroscopic sensor 4 and the spectroscopic image generating unit 5, a control unit 6, a timing unit 7, a GNSS (Global Navigation Satellite System) sensor 8, and a communication unit 9.

[0017] 3 schematically shows an example of the configuration of the pixel array section 4a of the spectroscopic sensor 4. As shown in the figure, the pixel array section 4a has spectroscopic pixel units Pu formed therein, each of which includes a plurality of pixels Px that receive light of different wavelength bands and are two-dimensionally arranged in a predetermined pattern. The pixel array section 4a is formed by two-dimensionally arranging a plurality of spectroscopic pixel units Pu.

[0018] In the example shown in the figure, each spectroscopic pixel unit Pu individually receives light in a total of eight wavelength bands, from λ1 to λ8, at each pixel Px, or in other words, the number of wavelength bands separately received within each spectroscopic pixel unit Pu (hereinafter referred to as the "number of received wavelength channels") is "8." However, this is merely an example for the purpose of explanation, and the number of received wavelength channels in a spectroscopic pixel unit Pu can be set arbitrarily as long as it is at least plural. Hereinafter, the number of received wavelength channels in a spectroscopic pixel unit Pu will be referred to as "N."

[0019] 2, the spectral image generating unit 5 generates M narrowband images based on a RAW image output from the spectral sensor 4. Here, M>N, and as an example, N=8 and M=41.

[0020] The spectroscopic image generation unit 5 has a demosaic unit 5a and a narrowband image generation unit 5b. The demosaic unit 5a performs demosaic processing on the RAW image from the spectroscopic sensor 4, and the narrowband image generation unit 5b performs narrowband processing (linear matrix processing) based on each wavelength band image for N channels obtained by the demosaic processing, thereby generating M narrowband images from N wavelength band images.

[0021] 4 is an explanatory diagram of the band narrowing process for obtaining M narrowband images. Narrowband images for M channels are obtained by performing a predetermined matrix operation for each pixel position based on waveband images for N channels obtained by the demosaic process performed by the demosaic unit 5a. In this way, in order to convert waveband images for N channels into narrowband images for M channels, pixel values ​​for N channels (in the figure, I ０ From I Ｎ－１ ) is used to calculate the pixel values ​​of M channels (I' ０From I' Ｍ－１ ) is the band narrowing process.

[0022] Here, if the pixel value after demosaic processing is R, the input wavelength channel is n (0 to N-1), the narrowband coefficient is C, the output pixel value by the narrowband processing is B, and the output wavelength channel is m (0 to M-1), the calculation formula for the narrowband processing can be expressed as the following [Equation 1].

[0023] That is, the pixel value B of the m=0th output wavelength channel ０ = R[0] × C ０ [0] + R[1] × C ０ [1] + R[2] × C ０ [2]+,...+R[N-1]×C ０ [N-1], and the pixel value B of the m=1st output wavelength channel １ = R[0] × C １ [0] + R[1] × C １ [1] + R[2] × C １ [2]+,...+R[N-1]×C １ [N-1]. The same applies to the following, and the pixel value B of the last m=M-1th output wavelength channel Ｍ－１ = R[0] × C Ｍ－１ [0] + R[1] × C Ｍ－１ [1] + R[2] × C Ｍ－１ [2]+,...+R[N-1]×C Ｍ－１ At this time, the band narrowing coefficient C is the pixel value B ０ C to find ０ [0] to C ０ [N-1], pixel value B １ C to find １ [0] to C １ [N-1], ..., pixel value B Ｍ－１ C to find Ｍ－１ [0] to C Ｍ－１ A total of N×M pieces of [N−1] are used.

[0024] In FIG. 2, the control unit 6 is configured with a microcomputer having, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory), and the CPU performs overall control of the spectroscopic camera 2 by executing processing based on, for example, a program stored in the ROM or a program loaded into the RAM.

[0025] The clock unit 7 measures the current time. The GNSS sensor 8 is an example of a position detection device that detects the current position of the spectroscopic camera 2. The uses of the current time information obtained by the clock unit 7 and the current position information of the spectroscopic camera 2 detected by the GNSS sensor 8 will be described later.

[0026] The communication unit 9 performs wired or wireless data communication with an external device. For example, the communication unit 9 may be configured to perform wired data communication with an external device according to a predetermined wired communication standard such as the USB (Universal Serial Bus) communication standard, wireless data communication with an external device according to a predetermined wireless communication standard such as the Bluetooth (registered trademark) communication standard, or wireless or wired data communication with an external device via a predetermined network such as the Internet. The control unit 6 can transmit and receive data to and from an external device (particularly the information processing device 1 in this example) via the communication unit 9.

[0027] 5 is a block diagram showing an example of the hardware configuration of the information processing device 1. As shown in FIG. 5, the information processing device 1 includes a CPU 11, a ROM 12, and a RAM 13. The CPU 11 functions as an arithmetic processing unit that performs various processes, and executes the various processes in accordance with a program stored in the ROM 12 or a program loaded from the storage unit 19 into the RAM 13. The RAM 13 also stores data and the like required for the CPU 11 to execute the various processes, as appropriate.

[0028] The CPU 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output interface (I / F) 15 is also connected to this bus 14.

[0029] An input unit 16 consisting of operators and operation devices is connected to the input / output interface 15. For example, the input unit 16 may be various operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, a remote controller, etc. In this example, the touch panel in the input unit 16 is formed on the display screen of the display unit 17, and the user can perform touch operations on the display screen. The input unit 16 detects user operations, and the CPU 11 interprets signals corresponding to the input operations.

[0030] A display unit 17, such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 18, such as a speaker, are connected integrally or separately to the input / output interface 15. The display unit 17 is used to display various types of information, and is configured as a display device provided in the housing of the information processing device 1, for example.

[0031] The display unit 17 displays images for various image processing, moving images to be processed, etc. on the display screen based on instructions from the CPU 11. The display unit 17 also displays various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the CPU 11.

[0032] A storage unit 19 and a communication unit 20 can be connected to the input / output interface 15. The storage unit 19 is configured by a hard disk drive (HDD) or a solid state drive (SSD), and stores various types of information.

[0033] The communication unit 20 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, etc. In particular, in the case of this embodiment, like the communication unit 9 in the spectroscopic camera 2 described above, the communication unit 20 is configured to perform wired data communication with an external device in accordance with a predetermined wired communication standard such as the USB communication standard, wireless data communication with an external device in accordance with a predetermined wireless communication standard such as the Bluetooth communication standard, or wireless or wired data communication with an external device via a predetermined network such as the Internet. This enables the CPU 11 to send and receive data to and from the spectroscopic camera 2 via the communication unit 20.

[0034] A drive 21 is also connected to the input / output interface 15 as required, and a removable recording medium 22 such as a memory card or optical disk is appropriately attached thereto.

[0035] The drive 21 makes it possible to read data files such as programs used in various processes from the removable recording medium 22. The read data files are stored in the storage unit 19, and images and sounds contained in the data files are output on the display unit 17 and the audio output unit 18. Furthermore, the computer programs and the like read from the removable recording medium 22 are installed in the storage unit 19 as necessary.

[0036] Here, the information processing device 1 is not limited to being configured as a single computer device as shown in Fig. 5, but may be configured as a system of multiple computer devices. The multiple computer devices may be systemized using a LAN (Local Area Network) or the like, or may be located in a remote location using a VPN (Virtual Private Network) or the like using the Internet or the like. The multiple computer devices may include computer devices as a server group (cloud) available through a cloud computing service.

[0037] [1-4. Functions of the Information Processing Device] As described above, in this embodiment, the information processing device 1 uses a predetermined light source model to estimate a light source spectrum based on the spectral information of the subject obtained by the spectroscopic camera 2. Various functions of the information processing device 1 according to the first embodiment, including the function of estimating the light source spectrum, will be described below. In the following description, the spectral information of the subject obtained by the spectroscopic camera 2 (the brightness values ​​of each narrowband image) will be referred to as the "object spectrum."

[0038] 6 is a functional block diagram for explaining various functions of the CPU 11 of the information processing device 1 according to the first embodiment. As shown in the figure, the CPU 11 functions as a light source spectrum estimation unit F1, a first environmental parameter estimation unit F2, a second environmental parameter estimation unit F3, an image processing unit F4, and a sunshine / shade area determination unit F5.

[0039] As described above, in this embodiment, a light source model that represents the light source spectrum by weighted addition of the sunlight spectrum and the sky spectrum is used to estimate the light source spectrum. Specifically, in this example, the Bird model is used as the light source model.

[0040] As is well known, the Bird model is an outdoor light source model that can calculate the solar spectrum and sky spectrum using the inputs of the earth-sun distance, solar zenith angle, aerosol turbidity in the air, precipitable water vapor (also called "precipitable water vapor"), and ozone mass. The solar altitude represents the distance between the earth and the sun, and the solar angle represents the angle of deviation of the sun's direction from the zenith direction of the observation position on the earth. The Bird model assumes that the target subject has sunny and shaded areas, and expresses the light source spectrum in terms of the proportion of the subject illuminated by sunlight and the proportion illuminated by the sky.

[0041] In the Bird model, by calculating the solar spectrum and sky spectrum, it is possible to approximately represent the light source spectrum from the sky in sunlight or shade regardless of weather (clear, cloudy, rainy) by weighted addition of these. Specifically, the light source spectrum corresponding to the weather and environmental type such as sunlight or shade can be represented by "light source spectrum = solar spectrum * b0 + sky spectrum * b1". Note that "b0" is the weighting coefficient for the solar spectrum, and "b1" is the weighting coefficient for the sky spectrum.

[0042] For reference, FIG. 7 shows an example of the solar spectrum and sky spectrum calculated by the Bird model.

[0043] In the first embodiment, the solar spectrum and sky spectrum corresponding to the actual environment are used to estimate the light source spectrum, so that the environmental parameters used to calculate the solar spectrum and sky spectrum are estimated based on images captured by the upward-facing camera 3, position information detected by the GNSS sensor 8, and current time information measured by the timing unit 7. Specifically, in this example, the solar altitude, solar angle, and ozone amount used to calculate the solar spectrum and sky spectrum are estimated based on position information detected by the GNSS sensor 8 and current time information measured by the timing unit 7. This estimation is performed by the first environmental parameter estimation unit F2. As is well known, the ozone amount can be estimated based on latitude, longitude, and time. Furthermore, the solar altitude can be estimated based on time, and the solar angle can be estimated based on latitude, longitude, and time.

[0044] Furthermore, the aerosol turbidity and precipitated water vapor amount used in calculating the solar spectrum and sky spectrum are estimated based on images captured by the upward-facing camera 3. This estimation is performed by the second environmental parameter estimation unit F3. A specific method for estimating the aerosol turbidity and precipitated water vapor amount based on images captured by the upward-facing camera 3 will be described later.

[0045] Here, even if a standard value for the ozone amount is used for the assumed environment, the effect on the calculated light source spectrum is minor, so it is possible to use a fixed value (a fixed value as a standard value) without estimating it based on the current location information and the current time information. This point will be explained later in the second embodiment.

[0046] Furthermore, the aerosol turbidity and precipitated water vapor have a relatively large effect on the calculated light source spectrum during the hours of sunrise and sunset, and therefore it is desirable to estimate them using the upward-facing camera 3. However, if the intended use is outside of those hours, standard values ​​may be used. This point will also be described later in the second embodiment.

[0047] A specific example of light source spectrum estimation by the light source spectrum estimation unit F1 will be described. First, for estimation, an error function Fd is used to calculate the error Ds as follows. Note that "^" means exponentiation. Error Ds = ∥Object spectrum - (light source spectrum × object spectral reflectance + specular reflection)∥^2 Here, "object spectral reflectance" means the spectral reflectance of the object as the subject. Spectral reflectance means information indicating the reflectance for each wavelength. In this example, the object spectral reflectance is modeled and handled using a polynomial of wavelength λ (especially a function of second order or lower) as shown below. Object spectral reflectance = m2λ^2 + m1λ + m0

[0048] Furthermore, "specular reflection" is defined as "c*solar spectrum."

[0049] In the first embodiment, the light source spectrum estimation unit F1 estimates the light source spectrum for each pixel of the spectroscopic image (each narrowband image) obtained by the spectroscopic camera 2 by deriving parameters (specifically, b0, b1, m0, m1, m2, and c) that minimize the above-mentioned error Ds while satisfying the constraints described below.

[0050] The constraints for illuminant spectrum estimation are as follows: 1) b1 is a common value for all pixels (because the sky spectrum is equivalent to ambient light); 2) b0, b1, and c are all 0 or greater; 3) For object spectral reflectance, v≦object spectral reflectance≦1.0 is satisfied (v is a user-specified value); 4) b0 / b1≧w (w is a user-specified value); 5) If b0 / b1<z, then c=0 (z is a user-specified value). Condition 3) above is a condition intended to improve the convergence of parameter derivation for spectral reflectance information modeled using a quadratic function. Condition 4) above corresponds to cases where, for example, there is an empirical rule that "b0 / b1" is always 0.1 or less even in deep shadow areas. This condition allows the user to specify a lower limit for the "b0 / b1" value, thereby improving the convergence of parameter derivation. Condition 5) above is intended to improve the convergence of parameter derivation by allowing the user to specify a condition for specular reflection.

[0051] A specific example of the parameter derivation process will be described with reference to the flowchart in Fig. 8. First, in step S101, the CPU 11 (light source spectrum estimation unit F1) sets initial values ​​for m0, m1, m2, and c, excluding b0 and b1. Note that the initial values ​​may be standard values ​​for each.

[0052] In step S102 following step S101, the CPU 11 substitutes initial values ​​m0, m1, m2, and c into the error function Fd to derive b0 and b1 that minimize the error Ds. That is, m0, m1, and m2, which are used to calculate the object spectral reflectance in the error function Fd, and c, which is used to calculate specular reflection, are fixed to their initial values, and b0 and b1 that minimize the error Ds are derived. Here, the calculation of the error Ds requires the calculation of the light source spectrum, which in turn requires the calculation of the solar spectrum and the sky spectrum. In the Bird model, the solar spectrum and the sky spectrum are expressed by functions with solar altitude, solar angle, ozone amount, aerosol turbidity, and precipitated water vapor amount as variables, respectively. Therefore, in the process of calculating the error Ds, the CPU 11 uses the solar altitude, solar angle, and ozone amount estimated by the first environmental parameter estimation unit F2, and the aerosol turbidity and precipitated water vapor amount estimated by the second environmental parameter estimation unit F3 as described below, to calculate the solar spectrum and sky spectrum.

[0053] In step S103 following step S102, the CPU 11 substitutes the derived b0 and b1 and the initial value c into the error function Fd to derive m0, m1, and m2 that minimize the error Ds. Furthermore, in the following step S104, the CPU 11 substitutes the derived b0, b1, m0, m1, and m2 into the error function Fd to derive c that minimizes the error Ds.

[0054] The processes from step S101 to step S104 are the first derivation process. After the first derivation process is completed, the CPU 11 performs the second and subsequent derivation processes from step S105 onwards.

[0055] Specifically, first, in step S105, the CPU 11 substitutes m0, m1, m2, and c derived in the immediately preceding derivation process into the error function Fd to derive b0 and b1 that minimize the error Ds. That is, for example, if the immediately preceding derivation process is the initial derivation process, then the CPU 11 substitutes m0, m1, m2, and c finally derived in the initial derivation process into the error function Fd to derive b0 and b1 that minimize the error Ds.

[0056] In step S106 following step S105, the CPU 11 substitutes the derived b0 and b1 and the c derived in the immediately preceding derivation process into the error function Fd to derive m0, m1, and m2 that minimize the error Ds. Further, in the following step S107, the CPU 11 substitutes the derived b0, b1, m0, m1, and m2 into the error function Fd to derive c that minimizes the error Ds.

[0057] In step S108 following step S107, the CPU 11 determines whether a derivation termination condition is met. Here, the derivation termination condition is a condition that indicates that the degree of parameter optimization (reduction of error Ds) has reached a desired level. For example, a condition based on the value of error Ds calculated using the finally derived b0, b1, m0, m1, m2, and c can be considered. As an example, a condition that the error Ds falls below a predetermined threshold or a condition that an inflection point (a point of inflection from a decrease to an increase) in the value of error Ds can be considered. If the latter condition is considered, the parameters derived in the derivation process immediately before the inflection point is detected are treated as the parameters of the final derivation. Alternatively, the derivation termination condition can be determined not based on the value of error Ds, but, for example, that the number of derivation processes has reached a predetermined number.

[0058] If it is determined in step S108 that the derivation termination condition is not satisfied, the CPU 11 returns to step S105, whereby the second and subsequent derivation processes are repeated until the derivation termination condition is satisfied.

[0059] On the other hand, if it is determined in step S108 that the derivation termination condition is met, the CPU 11 proceeds to step S109, where it calculates the light source spectrum using the finally derived b0 and b1, the sunlight spectrum, and the sky spectrum. That is, the light source spectrum is obtained by the above-mentioned calculation of "light source spectrum = sunlight spectrum * b0 + sky spectrum * b1." Note that the light source spectrum calculated by the above formula does not take into account the spectral sensitivity of the spectroscopic sensor 4. If the spectral sensitivity of the spectroscopic sensor 4 is taken into account, the light source spectrum (Nch) calculated by the above formula is convolved with the spectral sensitivity of the spectroscopic sensor 4 to obtain a sensor signal, and this is then narrowed using the above-mentioned [Formula 1] to obtain the light source spectrum for Mch.

[0060] The CPU 11 (light source spectrum estimation unit F1) executes the above-described derivation process for each pixel, thereby determining the light source spectrum for each pixel and the weighting coefficients b0 and b1 for each pixel.

[0061] Here, the above-described method for estimating a light source spectrum can be said to be the following estimation method: That is, the light source spectrum is estimated by estimating the parameters of the light source model and the object spectral reflectance, using the error between a trial object spectrum, which is a spectrum calculated by multiplying a trial light source spectrum, which is a light source spectrum calculated by setting candidate values ​​for the parameters of the light source model, by a candidate value for the object spectral reflectance indicating the spectral reflectance of the object as the subject, and the object spectrum as a reference value for estimation.

[0062] In the above, a method for estimating the light source spectrum was exemplified in which weighting factors b0 and b1 are derived using a parameter optimization method that minimizes the error Ds. However, it is also possible to estimate the light source spectrum using an AI model that has been trained by machine learning to input a spectral image captured by the spectroscopic camera 2 and output weighting factors b0 and b1. Specifically, b0 and b1 are inferred for each pixel of the image captured by the spectroscopic camera 2 using a deep neural network (DNN) capable of outputting data of the same size as the input image, such as a U-net used for segmentation. This DNN can be trained, for example, using the error Ds. During training, m0, m1, m2, and c are required to calculate the error Ds. These can be derived using b0 and b1 inferred by the DNN using values ​​derived, for example, using a method similar to that used in steps S103 and S104.

[0063] Next, the process of estimating the aerosol turbidity and the amount of precipitated water vapor by the second environmental parameter estimating unit F3 shown in FIG. 6 will be described.

[0064] As described above, the aerosol turbidity and the amount of precipitated water vapor are estimated based on images captured by the upward-looking camera 3. In this example, it is assumed that the upward-looking camera 3 is configured as a camera equipped with a spectroscopic sensor 4, similar to the spectroscopic camera 2, and that it obtains spectroscopic images (M narrowband images) as output images. For clarity, the solar spectrum and sky spectrum are expressed as functions with the solar altitude, solar angle, ozone amount, aerosol turbidity, and precipitated water vapor amount as variables, respectively. Hereinafter, for the sake of explanation, aerosol turbidity will be represented as τ, and precipitated water vapor amount as σ.

[0065] As with the process of deriving b0 and b1 described above, the aerosol turbidity τ and the amount of precipitated water vapor σ can be estimated by defining a predetermined error function and deriving parameters that minimize the error. Here, it can be said that the spectral information expressed by the following formula is detected on the imaging plane of upward-facing camera 3: "Sunlight spectrum * b0' + sky spectrum * b1'" Here, the weighting coefficient b0' for the sunlight spectrum and the weighting coefficient b1' for the sky spectrum in the above formula can be said to be coefficients that indicate the degree of sunlight at the location (imaging position) of upward-facing camera 3.

[0066] Based on the above assumptions, in this example, the following error function Fd' is used to estimate the aerosol turbidity τ and the amount of precipitated water vapor σ: Error Ds' = ∥Upward average spectral information − (sunlight spectrum * b0' + sky spectrum * b1')∥ ^2 Here, "upward average spectral information" refers to the average value of the entire image of the spectral information obtained by the upward camera 3. Specifically, it is M brightness values ​​obtained by calculating the average brightness value of the entire image for each of M narrowband images. As described above, the sunlight spectrum and the sky spectrum are expressed as functions with the solar altitude, solar angle, ozone amount, aerosol turbidity τ, and precipitated water vapor amount σ as variables, respectively. The solar altitude, solar angle, and ozone amount estimated by the first environmental parameter estimation unit F2, as well as the aerosol turbidity τ and precipitated water vapor amount σ, are substituted into the calculations of the sunlight spectrum and sky spectrum in the above equations, respectively.

[0067] The second environmental parameter estimation unit F3 estimates the aerosol turbidity τ and the amount of precipitated water vapor σ in the following procedure. First, initial values ​​of τ and σ are set. In this case, standard values ​​can also be used as the initial values.

[0068] Next, the second environmental parameter estimation unit F3 substitutes τ and σ as initial values ​​into the error function Fd' to derive b0' and b1' that minimize the error Ds'. Furthermore, the second environmental parameter estimation unit F3 substitutes the derived b0' and b1' and σ as the initial value into the error function Fd' to derive τ that minimizes the error Ds'. Next, the second environmental parameter estimation unit F3 substitutes the derived b0', b1', and τ into the error function Fd' to derive σ that minimizes the error Ds'.

[0069] In this manner, the initial derivation process of b0', b1', τ, and σ is performed. Thereafter, the second and subsequent derivation processes are performed as follows until a predetermined derivation termination condition is met. That is, first, τ and σ derived in the immediately preceding derivation process are substituted into an error function to derive b0' and b1' that minimize the error Ds'. Next, the derived b0 and b1 and σ derived in the immediately preceding derivation process are substituted into the error function Fd' to derive τ that minimizes the error Ds'. Furthermore, the derived b0', b1', and τ are substituted into the error function Fd' to derive σ that minimizes the error Ds'. In this case as well, the derivation termination condition can be set to the same condition as in the parameter derivation process of b0, b1, m0, m1, m2, and c described above.

[0070] By the above-described derivation process, it is possible to estimate the solar spectrum according to the environment, the aerosol turbidity τ for calculating the sky spectrum, and the amount of precipitated water vapor σ.

[0071] In the above example, the second environmental parameter estimation unit F3 estimates the environmental parameters by deriving them using a parameter optimization method that minimizes the error Ds'. However, the method for estimating the environmental parameters by the second environmental parameter estimation unit F3 is not limited to this. For example, it is possible to use an AI (artificial intelligence) model that has been trained to derive environmental parameters from images captured by the upward-facing camera 3. In this case, it is possible to perform the machine learning of the AI ​​model by deep learning, using images captured of the sky as learning input data and correct information on τ and σ as training data. In this case, it is possible to use the error function Fd' described above as the cost function.

[0072] Furthermore, with regard to the estimation of environmental parameters by the second environmental parameter estimation unit F3, the upward camera 3 is not limited to being a camera equipped with a spectroscopic sensor 4 like the spectroscopic camera 2, but any camera configured to be able to receive and separate light in at least multiple wavelength bands, such as an RGB camera, may be used.

[0073] 6, the image processing unit F4 performs predetermined image processing based on the light source spectrum estimated by the light source spectrum estimation unit F1 and the spectral image acquired by the spectroscopic camera 2. The image processing performed by the image processing unit F4 may be a light source cancellation process. Specifically, this process uses the light source spectrum estimated by the light source spectrum estimation unit F1 to remove the light source spectrum included in the object spectrum as the spectral image.

[0074] Alternatively, the image processing performed by the image processing unit F4 may include semantic segmentation processing based on the light source spectrum estimated by the light source spectrum estimation unit F1 and the spectroscopic image obtained by the spectroscopic camera 2. Specifically, it is conceivable to perform light source cancellation processing on the spectroscopic image using the light source spectrum estimated by the light source spectrum estimation unit F1, and then perform semantic segmentation processing on the canceled image using an AI model. In this case, the AI ​​model performing the semantic segmentation processing may be a model trained by machine learning using an image without light source removal and the light source spectrum as training input data, rather than a model trained by machine learning using an image with light source components removed as training input data. This makes it possible to achieve highly accurate semantic segmentation processing that eliminates the influence of light source components (specifically, the influence of sunlight and shade) without requiring light source cancellation processing.

[0075] 6, the sunshine / shade region determination unit F5 determines whether the subject captured by the spectroscopic camera 2 is a sunlit region or a shaded region using the sunlight spectrum weighting coefficient b0 and the sky spectrum weighting coefficient b1 estimated by the light source spectrum estimation unit F1. Basically, the sunlight spectrum weighting coefficient b0 tends to be larger in sunlit areas than in shaded areas, so pixels with a larger weighting coefficient b0 are determined to be in sunlit regions, and other pixels are determined to be in shaded regions. While various specific methods are conceivable, one possible method is to generate a histogram using b0, determine a threshold value for b0 for determining sunshine / shade based on the histogram, and then determine sunshine / shade based on the results of comparing the threshold value with b0.

[0076] In the above example, the image processing by the image processing unit F4 is realized by software processing by the CPU 11, but at least a part of the image processing may be realized by a method other than software processing by the CPU 11. For example, at least a part of the image processing by the image processing unit F4 may be realized by hardware processing by a signal processing unit such as a DPS (Digital Signal Processor) or GPU (Graphics Processing Unit) outside the CPU 11. In particular, in the case of semantic segmentation processing using an AI model, it is conceivable that the necessary convolution operation is realized by a hardware signal processing unit such as a GPU.

[0077] Although the above example shows the estimation of the light source spectrum on a pixel-by-pixel basis, in the first embodiment, the light source spectrum may be estimated for each divided region obtained by dividing the spectral image into regions in units of a predetermined number of pixels. For example, estimation may be performed for each divided region having a size of 2 x 2 = 4 pixels or 8 x 8 = 64 pixels. This allows the light source spectrum to be estimated at a finer granularity than for the entire image, thereby improving the accuracy of the estimation of the light source spectrum in the spatial direction.

[0078] 2. Second Embodiment Next, a second embodiment will be described. In the second embodiment, the light source spectrum is estimated not for each divided region as in the first embodiment, but by estimating an average light source spectrum for the entire image. Furthermore, in the second embodiment, while all environmental spectra are estimated in the first embodiment, fixed values ​​(standard values) are used for at least some environmental parameters instead of estimation. Specifically, in this example, of the environmental parameters solar altitude, solar angle, ozone amount, aerosol turbidity, and precipitated water vapor amount used to estimate the light source spectrum, standard values ​​are used for the ozone amount, aerosol turbidity, and precipitated water vapor amount, excluding solar altitude and solar angle.

[0079] 9 is a block diagram showing an example of the configuration of a light source spectrum estimation system according to the second embodiment. In the following description, parts that are the same as parts already described will be assigned the same reference numerals and will not be described again. The changes from the first embodiment are that the upward-facing camera 3 is omitted and that an information processing device 1A is used instead of the information processing device 1.

[0080] 10 is a functional block diagram for explaining functions of an information processing device 1A according to the second embodiment. Here, the CPU 11 of the information processing device 1A is referred to as CPU 11A. In FIG. 10, the functions of the CPU 11A are shown in blocks.

[0081] The differences from the CPU 11 in the first embodiment are that the second environmental parameter estimation unit F3 and the sunshine / shade area determination unit F5 are omitted, that a first environmental parameter estimation unit F2A is provided instead of the first environmental parameter estimation unit F2, and that a light source spectrum estimation unit F1A is provided instead of the light source spectrum estimation unit F1.

[0082] The first environmental parameter estimation unit F2A estimates the solar altitude and solar angle based on the current position information and current time information input from the spectroscopic camera 2.

[0083] The light source spectrum estimation unit F1A receives as input the standard value of the ozone amount (standard first environmental parameter in the figure), the standard values ​​of the aerosol turbidity and the amount of precipitated water vapor (standard second environmental parameter in the figure), and the solar altitude and solar angle estimated by the first environmental parameter estimation unit F2A, and estimates an average light source spectrum for the entire image based on the spectroscopic image input from the spectroscopic camera 2. Specifically, the average light source spectrum for the entire image is realized by using the average luminance value for the entire image of each of the M narrowband images as the "object spectrum" in the calculation of the error Ds described above. That is, in the calculation of the error Ds used in the processing of FIG. 8 described above, the average luminance value for the entire image of each of the M narrowband images input as the spectroscopic image can be used as the "object spectrum."

[0084] The image processing unit F4A performs predetermined image processing based on the spectral image input from the spectroscopic camera 2 and the average light source spectrum of the entire image obtained by the light source spectrum estimation unit F1A. For example, one possible image processing performed by the image processing unit F4A is to remove the average light source spectrum of the entire image from the spectral image.

[0085] Although the above example illustrates the use of standard values ​​for the ozone amount, aerosol turbidity, and precipitated water vapor amount, the combination of environmental parameters for which standard values ​​are used is not limited to this. For example, it is possible to use a standard value for only the ozone amount, or to use standard values ​​for only the aerosol turbidity and precipitated water vapor amount as second environmental parameters. Alternatively, it is possible to use a standard value for only either the aerosol turbidity or the precipitated water vapor amount. When estimating at least one of the aerosol turbidity and the precipitated water vapor amount, the system is configured to have an upward-facing camera 3, as in the first embodiment.

[0086] 3. Modifications Note that the embodiment is not limited to the specific example described above, and various modified configurations are possible. For example, the above describes an example of a method for estimating the weighting coefficient b0 of the solar spectrum and the weighting coefficient b1 of the sky spectrum. However, it is also possible to use standard values ​​instead of estimating these weighting coefficients b0 and b1. Figure 11 is a functional block diagram for explaining the functions of the CPU 11X corresponding to this case. This shows an example of a functional configuration assuming that standard values ​​are used for each environmental parameter other than the solar altitude and solar angle, namely, the amount of ozone, the aerosol turbidity, and the amount of precipitated water vapor, and that the average light source spectrum of the entire image is estimated, as in the second embodiment.

[0087] The difference from the CPU 11A in the second embodiment is that a light source spectrum estimation unit F1X is provided instead of the light source spectrum estimation unit F1A. The light source spectrum estimation unit F1X receives weighting coefficients b0 and b1 ("standard sunshine degree" in the figure) that are fixed values ​​as standard values, and estimates the average light source spectrum for the entire image using these weighting coefficients b0 and b1. Specifically, the light source spectrum and sky spectrum are calculated using the solar altitude and solar angle estimated by the first environmental parameter estimation unit F2A and standard values ​​for the amount of ozone, aerosol turbidity, and precipitated water vapor, and then the light source spectrum is estimated by substituting the standard values ​​for b0 and b1 in the aforementioned "sunlight spectrum * b0 + sky spectrum * b1."

[0088] When standard values ​​are used for the weighting coefficients b0 and b1, the ozone amount can be estimated based on location information and time information, as in the first embodiment. Furthermore, at least one of the aerosol turbidity and the amount of precipitated water vapor can be estimated based on images captured by the upward-facing camera 3, as in the first embodiment. In the configuration shown in Fig. 11, the light source spectrum can be estimated not using standard values ​​(standard degree of sunshine) for b0 and b1, but using b0' and b1' calculated by the second environmental parameter estimation unit F3 described in Fig. 6 based on images captured by the upward-facing camera 3.

[0089] In the above description, the Bird model has been cited as an example of a light source model used to estimate a light source spectrum. However, other light source models can be used as the light source model, such as the SMARTS2 (Simple Model of the Atmospheric Radiative Transfer of Sunshine 2) light source model or the MODTRAN6 (MODerate resolution atmospheric TRANsmission 6) light source model, as long as they represent the light source spectrum, which is spectral information of an outdoor light source, by weighted addition of the sunlight spectrum and the sky spectrum. The SMARTS2 light source model has a larger number of environmental parameters than the Bird model and can generate a more accurate spectrum. The MODTRAN6 light source model has an even larger number of environmental parameters than the SMARTS2 light source model and can generate an even more accurate spectrum. In addition, even when the number of environmental parameters increases, for example, when adopting a light source model such as SMARTS2 or MODTRAN6, the environmental parameters required for light source spectrum estimation can be derived in a similar manner by fixing parameters other than the target parameter and deriving parameters that minimize the error due to the error function.

[0090] Furthermore, in the explanation so far, an example has been given in which the information processing device according to the present technology is configured as a device separate from the spectroscopic camera 2, but the information processing device according to the present technology can also be configured as an integrated device with the spectroscopic camera 2.

[0091] 4. Summary of the Embodiments As described above, the information processing device (1 or 1A) according to the embodiment includes an illuminant spectrum estimation unit (F1, F1A) that estimates an illuminant spectrum based on an object spectrum, which is spectral information of a subject, acquired by a spectroscopic camera, using an illuminant model that represents an illuminant spectrum, which is spectral information of an outdoor light source, by weighted addition of a sunlight spectrum, which is spectral information of light irradiated from the sun, and a sky spectrum, which is spectral information of light irradiated from the sky. This makes it possible to estimate the illuminant spectrum by fitting a function serving as the illuminant model to the object spectrum detected by the spectroscopic camera. This eliminates the restriction that the environment in which the illuminant spectrum can be estimated is limited to the assumed environment, as in the prior art, thereby improving the robustness of the illuminant spectrum estimation. Furthermore, because the illuminant model represents an illuminant spectrum by weighted addition of a sunlight spectrum and a sky spectrum, it is possible to achieve appropriate illuminant spectrum estimation that takes into account whether the subject is in sunlight or shade.

[0092] Furthermore, in the information processing device according to the embodiment, the light source spectrum estimation unit estimates the light source spectrum by estimating the light source model parameters and the object spectral reflectance using an error (Ds) between a trial object spectrum, which is a spectrum calculated by multiplying a trial light source spectrum, which is a light source spectrum calculated by setting candidate values ​​for the light source model parameters, by a candidate value for the object spectral reflectance indicating the spectral reflectance of the object as the subject, as a reference value for estimation. This makes it possible to estimate, as correct values ​​for the light source model parameters and the object spectral reflectance, candidate values ​​that satisfy a condition that the error is small, such as minimizing the error between the trial object spectrum and the object spectrum detected by a spectroscopic camera, and thereby make it possible to appropriately estimate the light source spectrum according to the environment.

[0093] Furthermore, in the information processing device according to the embodiment, the light source spectrum estimation unit estimates a weighting coefficient (b0) for the sunlight spectrum and a weighting coefficient (b1) for the sky spectrum as parameters of the light source model based on the error, thereby making it possible to estimate an appropriate light source spectrum according to the environment, whether it is in sunlight or shade.

[0094] Furthermore, the information processing device (same as F1) according to the embodiment includes a sunshine / shade area determination unit (same as F5) that determines the sunshine area and the shade area of ​​the subject captured by the spectroscopic camera using the weighting coefficient of the sunlight spectrum and the weighting coefficient of the sky spectrum estimated by the light source spectrum estimation unit. This makes it possible to appropriately determine the sunshine area and the shade area of ​​the image captured by the spectroscopic camera.

[0095] In addition, in the information processing device according to the embodiment, the light source spectrum estimation unit includes a first environmental parameter estimation unit (F2 or F2A) that estimates at least some of the environmental parameters used to calculate the solar spectrum and sky spectrum based on at least one of the position information and current time information of the spectroscopic camera. This makes it possible to estimate appropriate values ​​for parameters that can be estimated based on the position information and current time information of the spectroscopic camera among the environmental parameters used to calculate the solar spectrum and sky spectrum, thereby making it possible to estimate appropriate spectra for the solar spectrum and sky spectrum according to the actual environment. This improves the accuracy of estimating the light source spectrum.

[0096] Furthermore, in an information processing device according to an embodiment, the light source spectrum estimation unit uses environmental parameters such as solar altitude, solar angle, and ozone amount to calculate the solar spectrum and sky spectrum, and the first environmental parameter estimation unit estimates at least one of the solar altitude, solar angle, and ozone amount based on at least one of the position information and current time information of the spectroscopic camera. The solar altitude, solar angle, and ozone amount are each environmental parameters that can be estimated based on the position information and the current time information. Therefore, with the above configuration, it is possible to estimate appropriate spectra for the solar spectrum and sky spectrum according to the actual environment, thereby improving the accuracy of estimating the light source spectrum.

[0097] Furthermore, in the information processing device according to the embodiment, the light source spectrum estimation unit includes a second environmental parameter estimation unit (F3) that estimates at least some of the environmental parameters used in calculating the solar spectrum and sky spectrum based on images captured by an upward-facing camera (F3) that captures images of the sky. This allows appropriate values ​​to be estimated for the environmental parameters used in calculating the solar spectrum and sky spectrum that can be estimated based on images captured by the upward-facing camera, making it possible to estimate appropriate solar spectrum and sky spectrum spectra that are appropriate for the actual environment. This improves the accuracy of estimating the light source spectrum.

[0098] In addition, in an information processing device according to an embodiment, the light source spectrum estimation unit uses environmental parameters such as aerosol turbidity and precipitated water vapor to calculate the solar spectrum and sky spectrum, and the second environmental parameter estimation unit estimates at least one of the aerosol turbidity and precipitated water vapor based on images captured by an upward-facing camera. This makes it possible to appropriately estimate these environmental parameters based on images captured of the sky when aerosol turbidity or precipitated water vapor is used to calculate the solar spectrum and sky spectrum, thereby enabling the estimation of appropriate solar spectrum and sky spectrum according to the actual environment. This improves the accuracy of the light source spectrum estimation.

[0099] Furthermore, in the information processing device according to the embodiment, the light source spectrum estimation unit (F1A) uses fixed values ​​for one of the environmental parameters used to calculate the solar spectrum and the sky spectrum. This makes it possible to use standard values ​​corresponding to a standard environment for at least one of the environmental parameters used to calculate the solar spectrum and the sky spectrum. Some environmental parameters have little effect on the estimated light source spectrum even when standard values ​​are used. Furthermore, using fixed values ​​for the environmental parameters eliminates the need to perform estimation processing for those environmental parameters. Therefore, with the above configuration, it is possible to reduce the processing load required for estimating the light source spectrum while suppressing a decrease in the estimation accuracy of the light source spectrum.

[0100] Furthermore, in the information processing device according to the embodiment, the light source spectrum estimation unit (F1) estimates the light source spectrum using the light source model for each divided region obtained by dividing the spectral image obtained by the spectroscopic camera into regions in units of a predetermined number of pixels. This allows the light source spectrum to be estimated at a finer granularity, such as pixel by pixel, rather than for the entire image. This improves the accuracy of estimating the light source spectrum in the spatial direction.

[0101] In addition, in the information processing device according to the embodiment, the light source spectrum estimation unit (F1A) estimates the average light source spectrum of the entire spectroscopic image obtained by the spectroscopic camera. This configuration is suitable for cases where information on the average light source spectrum of the entire subject is required.

[0102] Furthermore, the information processing device according to the embodiment includes an image processing unit (F4 or F4A) that performs light source cancellation processing on the spectral image obtained by the spectroscopic camera based on the light source spectrum estimated by the light source spectrum estimation unit. This allows the light source cancellation processing to be performed based on the appropriate light source spectrum estimated by the light source spectrum estimation unit. This improves the accuracy of the light source cancellation processing.

[0103] Furthermore, the information processing device according to the embodiment includes an image processing unit (F4) that performs semantic segmentation processing based on the light source spectrum estimated by the light source spectrum estimation unit and the spectroscopic image acquired by the spectroscopic camera. This allows the semantic segmentation processing of the subject to be performed based on the appropriate light source spectrum estimated by the light source spectrum estimation unit. This improves the accuracy of the semantic segmentation processing.

[0104] In an information processing method according to an embodiment, an information processing device estimates a light source spectrum based on an object spectrum, which is spectral information of a subject, obtained by a spectroscopic camera, using a light source model that represents a light source spectrum, which is spectral information of an outdoor light source, by weighted addition of a sunlight spectrum, which is spectral information of light irradiated from the sun, and a sky spectrum, which is spectral information of light irradiated from the sky. This information processing method can also achieve the same functions and effects as the information processing device according to the above-described embodiment.

[0105] Here, as an embodiment, a program that causes, for example, a CPU, a DSP, or a device including these, to realize the functions of the light source spectrum estimation unit F1 or F1A described with reference to Figure 8, etc., can be considered. That is, the program of the embodiment is a computer-readable program that causes a computer to realize a function of estimating a light source spectrum based on an object spectrum, which is spectral information of a subject obtained by a spectroscopic camera, using a light source model that represents a light source spectrum, which is spectral information of an outdoor light source, by weighted addition of a solar spectrum, which is spectral information of light irradiated from the sun, and a sky spectrum, which is spectral information of light irradiated from the sky. Such a program allows the functions of the light source spectrum estimation unit F1 or F1A described above to be realized in a device such as the information processing device 1 or 1A.

[0106] The above-described programs can be pre-recorded on a hard disk drive (HDD) or solid state drive (SSD) as a recording medium built into a computer or other device, or on a ROM within a microcomputer having a CPU. Alternatively, the programs can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), a Magneto Optical (MO) disc, a Digital Versatile Disc (DVD), a Blu-ray Disc (Blu-ray Disc (registered trademark)), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called packaged software. Furthermore, such programs can be installed on a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN or the Internet.

[0107] Furthermore, such a program is suitable for widely providing the light source spectrum estimation method according to the embodiment. For example, by downloading the program to a personal computer, a portable information processing device, a mobile phone, a game console, a video device, a PDA (Personal Digital Assistant), or the like, the personal computer or the like can function as a device that realizes the light source spectrum estimation method according to the present disclosure.

[0108] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0109] <5. Present Technology> The present technology may also have the following configuration. (1) An information processing device including a light source spectrum estimation unit that uses a light source model that expresses a light source spectrum that is spectral information of an outdoor light source by weighted addition of a sunlight spectrum that is spectral information of light irradiated from the sun and a sky spectrum that is spectral information of light irradiated from the sky to estimate the light source spectrum based on an object spectrum that is spectral information of a subject acquired by a spectroscopic camera. (2) The light source spectrum estimation unit estimates the light source spectrum by estimating parameters of the light source model and the object spectral reflectance using an error between a trial object spectrum that is a light source spectrum calculated by setting candidate values ​​for parameters of the light source model and the object spectral reflectance as a reference value for estimation. (3) The light source spectrum estimation unit estimates weighting coefficients for the sunlight spectrum and the sky spectrum as parameters of the light source model based on the error. (4) The information processing device according to (3), further comprising a sunshine / shade region determination unit that determines sunshine regions and shade regions of a subject imaged by the spectroscopic camera using the weighting coefficient of the solar spectrum and the weighting coefficient of the sky spectrum estimated by the light source spectrum estimation unit. (5) The information processing device according to any of (1) to (4), further comprising a first environmental parameter estimation unit that estimates at least some of the environmental parameters used by the light source spectrum estimation unit to calculate the solar spectrum and the sky spectrum, based on at least one of position information of the spectroscopic camera and current time information. (6) The information processing device according to (5), further comprising a sunshine / shade region determination unit that determines sunshine regions and shade regions of a subject imaged by the spectroscopic camera using the weighting coefficient of the solar spectrum and the sky spectrum estimated by the light source spectrum estimation unit.(7) The information processing device according to any one of (1) to (6), further comprising a second environmental parameter estimation unit that estimates at least some of the environmental parameters used by the light source spectrum estimation unit to calculate the solar spectrum and the sky spectrum based on an image captured by an upward-facing camera that captures an image of the sky. (8) The information processing device according to (7), wherein the light source spectrum estimation unit uses environmental parameters such as aerosol turbidity and precipitated water vapor amount to calculate the solar spectrum and the sky spectrum, and the second environmental parameter estimation unit estimates at least one of the aerosol turbidity and the precipitated water vapor amount based on the image captured by the upward-facing camera. (9) The information processing device according to any one of (1) to (4), wherein the light source spectrum estimation unit uses a fixed value for one of the environmental parameters used to calculate the solar spectrum and the sky spectrum. (10) The information processing device according to any one of (1) to (9), wherein the light source spectrum estimation unit estimates the light source spectrum using the light source model for each divided region obtained by dividing a spectroscopic image obtained by the spectroscopic camera into regions in units of a predetermined number of pixels. (11) The information processing device according to any of (1) to (9), wherein the light source spectrum estimation unit estimates an average light source spectrum for the entire spectral image obtained by the spectroscopic camera. (12) The information processing device according to any of (1) to (11), further comprising an image processing unit that performs light source cancellation processing on the spectral image obtained by the spectroscopic camera based on the light source spectrum estimated by the light source spectrum estimation unit. (13) The information processing device according to any of (1) to (11), further comprising an image processing unit that performs semantic segmentation processing based on the light source spectrum estimated by the light source spectrum estimation unit and the spectral image obtained by the spectroscopic camera. (14) An information processing method, wherein an information processing device estimates the light source spectrum based on an object spectrum that is spectral information of a subject obtained by a spectroscopic camera, using a light source model that expresses the light source spectrum that is spectral information of an outdoor light source by weighted addition of a sunlight spectrum that is spectral information of light irradiated from the sun and a sky spectrum that is spectral information of light irradiated from the sky.(15) A computer-readable program that causes the computer to realize a function of estimating a light source spectrum based on an object spectrum, which is spectral information of a subject obtained by a spectroscopic camera, using a light source model that expresses a light source spectrum, which is spectral information of an outdoor light source, by weighted addition of a sunlight spectrum, which is spectral information of light irradiated from the sun, and a sky spectrum, which is spectral information of light irradiated from the sky.

[0110] REFERENCE SIGNS LIST 1 Information processing device 2 Spectroscopic camera 3 Upward-facing camera 4 Spectroscopic sensor 4a Pixel array section 5 Spectroscopic image generation section 5a Demosaicing section 5b Narrowband image generation section 6 Control section 7 Timekeeping section 8 GNSS sensor 9 Communication section Px Pixel Pu Spectroscopic pixel unit 11, 11A CPU 12 ROM 13 RAM 14 Bus 15 Input / output interface 16 Input section 17 Display section 18 Audio output section 19 Storage section 20 Communication section 21 Drive 22 Removable recording medium F1, F1A Light source spectrum estimation section F2, F2A First environmental parameter estimation section F3, F3A Second environmental parameter estimation section F4, F4A Image processing section

Claims

1. An information processing device having a light source spectrum estimation unit that uses a light source model that represents a light source spectrum, which is spectral information of an outdoor light source, by weighted addition of a solar spectrum, which is spectral information of light irradiated from the sun, and a sky spectrum, which is spectral information of light irradiated from the sky, and estimates the light source spectrum based on an object spectrum, which is spectral information of a subject obtained by a spectroscopic camera.

2. The information processing device of claim 1, wherein the light source spectrum estimation unit estimates the light source spectrum by estimating the parameters of the light source model and the object spectral reflectance, using an error between a trial object spectrum, which is a spectrum calculated by multiplying a trial light source spectrum, which is a light source spectrum calculated by setting candidate values ​​for the parameters of the light source model, by a candidate value of object spectral reflectance indicating the spectral reflectance of the object as the subject, and the object spectrum as a reference value for estimation.

3. The information processing device according to claim 2, wherein the light source spectrum estimating section estimates a weighting coefficient of the sunlight spectrum and a weighting coefficient of the sky spectrum as parameters of the light source model based on the error.

4. The information processing device according to claim 3, further comprising a sun / shade area determination unit that determines sunshine areas and shade areas of a subject imaged by the spectroscopic camera using the weighting coefficient of the sunlight spectrum and the weighting coefficient of the sky spectrum estimated by the light source spectrum estimation unit.

5. The information processing device according to claim 1, further comprising a first environmental parameter estimation unit that estimates at least a portion of the environmental parameters used by the light source spectrum estimation unit to calculate the solar spectrum and the sky spectrum based on at least one of the position information and the time information of the spectroscopic camera.

6. The information processing device according to claim 5, wherein the light source spectrum estimation unit uses environmental parameters such as solar altitude, solar angle, and ozone amount to calculate the solar spectrum and the sky spectrum, and the first environmental parameter estimation unit estimates at least one of the solar altitude, the solar angle, and the ozone amount based on at least one of position information and current time information of the spectroscopic camera.

7. The information processing device according to claim 1, further comprising a second environmental parameter estimation unit that estimates at least a portion of the environmental parameters used by the light source spectrum estimation unit to calculate the solar spectrum and the sky spectrum based on an image captured by an upward-facing camera capturing an image of the sky.

8. The information processing device described in claim 7, wherein the light source spectrum estimation unit uses environmental parameters such as aerosol turbidity and precipitated water vapor amount to calculate the sunlight spectrum and the sky spectrum, and the second environmental parameter estimation unit estimates at least one of the aerosol turbidity and the precipitated water vapor amount based on an image captured by the upward-facing camera.

9. The information processing device according to claim 1, wherein the light source spectrum estimation unit uses a fixed value for any of the environmental parameters used in the calculation of the sunlight spectrum and the sky spectrum.

10. The information processing device according to claim 1, wherein the light source spectrum estimation unit estimates the light source spectrum using the light source model for each divided region obtained by dividing the spectroscopic image obtained by the spectroscopic camera into regions in units of a predetermined number of pixels.

11. The information processing device according to claim 1, wherein the light source spectrum estimation unit estimates an average light source spectrum for the entire spectroscopic image obtained by the spectroscopic camera.

12. The information processing device according to claim 1, further comprising an image processing unit that performs light source cancellation processing on the spectroscopic image obtained by the spectroscopic camera based on the light source spectrum estimated by the light source spectrum estimation unit.

13. The information processing device according to claim 1, further comprising an image processing unit that performs semantic segmentation processing based on the light source spectrum estimated by the light source spectrum estimation unit and a spectroscopic image obtained by the spectroscopic camera.

14. An information processing method in which an information processing device uses a light source model that represents a light source spectrum, which is spectral information of an outdoor light source, by weighted addition of a solar spectrum, which is spectral information of light irradiated from the sun, and a sky spectrum, which is spectral information of light irradiated from the sky, and estimates the light source spectrum based on an object spectrum, which is spectral information of a subject obtained by a spectroscopic camera.

15. A program readable by a computer device, which causes the computer device to realize a function of estimating a light source spectrum based on an object spectrum, which is the spectral information of a subject obtained by a spectroscopic camera, using a light source model that represents a light source spectrum, which is the spectral information of an outdoor light source, by weighted addition of a solar spectrum, which is the spectral information of light irradiated from the sun, and a sky spectrum, which is the spectral information of light irradiated from the sky.

Images