Information processing device, information processing method, recording medium

Abstract

We aim to improve the accuracy of subject analysis based on spectral images of subjects obtained using a spectroscopic sensor. [Solution] The information processing device receives a spectral image obtained by sensing subject light, which is obtained when light from a light source is reflected by the subject, via a polarizing filter using a spectral sensor, and includes a correction processing unit that performs a correction process to remove the light source component, which is a component of the light source, from the spectral image using the spectral information of the light source.

Description

[Technical Field]

[0001] This technology relates to an information processing device, an information processing method, and a recording medium, and more particularly to an image processing technology for spectral images of a subject obtained using a spectral sensor. [Background technology]

[0002] Spectroscopic sensors (multispectrum sensors) are known for obtaining multiple wavelength band images, which are wavelength characteristic analysis images of light from a subject, or in other words, analysis images of the spectral information (spectral spectrum) of the subject. Furthermore, applications have been developed that perform various analyses of subjects based on the spectral information obtained by these spectroscopic sensors, such as estimating the vegetation state of plants or the condition of human skin, based on these multiple wavelength band images.

[0003] When obtaining spectral information of a subject, the spectral information of the light source illuminating the subject becomes noise, and it is desirable to remove this noise. For example, Patent Document 1 below discloses a method of correcting the subject spectrum by removing the sunlight component based on the sunlight spectrum measured using a camera, and measuring the sunlight spectrum by separating it into direct light component and scattered light component using a light-shielding plate. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Patent No. 6988898 [Overview of the project] [Problems that the invention aims to solve]

[0005] However, the light sensed by the spectroscopic sensor may also include surface reflection components of the subject. This surface reflection component, like the light source component, can become noise when analyzing the subject based on its spectral information. For example, if we consider analyzing the vegetation conditions of a plant as the subject, the vegetation information is contained in the light that enters and diffuses inside the plant (called "internal diffused light"), while surface reflected light, which has not passed through the inside of the plant and does not contain the aforementioned internal diffused light, becomes a noise component in vegetation analysis.

[0006] This technology was developed in view of the above-mentioned problems, and aims to improve the accuracy of subject analysis based on spectral images of subjects obtained using a spectroscopic sensor. [Means for solving the problem]

[0007] The information processing device related to this technology receives a spectral image obtained by sensing subject light, which is obtained when light from a light source is reflected by an object, through a polarizing filter using a spectral sensor, and includes a correction processing unit that performs a correction process to remove the light source component, which is a component of the light source, from the spectral image using the spectral information of the light source. As described above, by using the spectral image obtained by sensing the subject light through a polarizing filter in the correction process, it becomes possible to use a spectral image in which the surface reflected light component of the subject has been suppressed for the correction process. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the configuration overview of the subject analysis system as the first embodiment. [Figure 2] This is a block diagram showing a schematic configuration example of a spectroscopic camera in the first embodiment. [Figure 3] This diagram schematically shows an example of the configuration of the pixel array section of a spectroscopic sensor. [Figure 4] This is an explanatory diagram of the bandwidth processing in the embodiment. [Figure 5] This is a block diagram showing an example of the hardware configuration of an information processing device as a first embodiment. [Figure 6]It is a functional block diagram for explaining various functions related to the subject analysis method as the first embodiment. [Figure 7] It is a flowchart of processing related to the manual adjustment mode in the first embodiment. [Figure 8] It is a flowchart of processing related to the automatic adjustment mode in the first embodiment. [Figure 9] Similarly, it is a flowchart of processing related to the automatic adjustment mode in the first embodiment. [Figure 10] It is a block diagram showing an outline configuration of a subject analysis system as a second embodiment. [Figure 11] It is a block diagram showing a schematic configuration example of a spectroscopic camera used in the first example of the second embodiment. [Figure 12] It is a functional block diagram for explaining various functions related to the subject analysis method as the first example of the second embodiment. [Figure 13] It is a flowchart showing an example of a processing procedure to be executed to implement the subject analysis method as the first example of the second embodiment. [Figure 14] It is a block diagram showing a schematic configuration example of a spectroscopic camera used in the second example of the second embodiment. [Figure 15] It is a diagram showing a configuration example of a pixel array unit included in the spectroscopic sensor according to the second example. [Figure 16] It is a diagram illustrating a schematic cross-sectional structure of each pixel included in the pixel array unit according to the second example. [Figure 17] It is an explanatory diagram of image organization processing and demosaicing processing according to the second example. Description of Embodiments

[0009] Hereinafter, embodiments of the present technology will be described in the following order with reference to the accompanying drawings. <1. First Embodiment> (1-1. System Overview) (1-2. Configuration of Spectroscopic Camera) (1-3. Configuration of Information Processing Apparatus) (1-4. Subject Analysis Method as the First Embodiment) (1-5. Processing Procedure) <2. Second Embodiment> (2-1. First example) (2-2. Second example) <3. Variant> <4. Summary of Embodiments> <5. This Technology>

[0010] <1. First Embodiment> (1-1. System Overview) Figure 1 is a diagram showing the configuration overview of a subject analysis system as a first embodiment, which is equipped with an information processing device 1 as a first embodiment of this technology. As shown in the figure, the subject analysis system of this embodiment includes a spectroscopic camera 3 and a spectroscopic camera 5 together with the information processing device 1.

[0011] Spectroscopic camera 3 is a camera for obtaining spectral information of the subject, and spectroscopic camera 5 is a camera for obtaining spectral information of the light source illuminating the subject. Here, "spectroscopic camera" refers to a camera equipped with a spectroscopic sensor as a light-receiving sensor. A "spectroscopic sensor" is a light-receiving sensor that obtains multiple wavelength band images, which are wavelength characteristic analysis images of light from a subject.

[0012] The subject analysis system in this example performs analysis on plants as the target subject outdoors. For this reason, spectroscopic camera 3 is set up to perform sensing on plants in outdoor locations such as fields, and spectroscopic camera 5 is set up to perform sensing with the sun as the target light source.

[0013] As will be described later, the spectral camera 3 has a spectral sensor in which multiple pixels are arranged in two dimensions, and is configured to obtain a spectral image. The spectral image of the subject obtained by the spectral camera 3 is input to the information processing device 1 as the subject spectral image.

[0014] In this example, the spectral camera 5 has a spectral sensor with multiple pixels arranged in two dimensions, similar to the spectral camera 3, and obtains light source spectral information, which is spectral information of the light source, based on the spectral image obtained by the spectral sensor. The light source spectral information only needs to be information indicating the wavelength analysis of light from the light source, that is, information indicating the amount of light received for each wavelength, and does not need to be image information. As a method for obtaining light source spectral information based on the spectral image, for example, one can consider averaging the band images for each wavelength (M band images described later) and combining them into a single value for each wavelength. The light source spectral information obtained by the spectral camera 5 is input to the information processing device 1.

[0015] The information processing device 1 is configured as a computer device and uses the light source spectral information input from the spectral camera 5 to perform correction processing to remove the light source component from the subject spectral image input from the spectral camera 3. Furthermore, the information processing device 1 calculates evaluation values ​​for analyzing the subject based on the corrected spectral image of the subject. Specifically, in this example, a vegetation evaluation value is calculated as the evaluation value for analyzing the subject as a plant. An example of a vegetation evaluation value will be explained later.

[0016] (1-2. Spectroscopic Camera Configuration) Figure 2 is a block diagram showing a schematic configuration example of the spectroscopic camera 3 used in the first embodiment. As shown in the figure, the spectroscopic camera 3 comprises an imaging optical system 30, a spectroscopic sensor 31, a spectroscopic image generation unit 32, a control unit 33, and a communication unit 34. The spectroscopic camera 3 also includes a rotary polarizing filter 40, an actuator 41, a GNSS (Global Navigation Satellite System) sensor 42, and a compass sensor 43, which will be explained in more detail later.

[0017] The imaging optical system 30 is comprised of various lenses for imaging, such as a cover lens and a focus lens, as well as an iris mechanism. This imaging optical system 30 collects light (incident light) from the subject and focuses it onto the light-receiving surface (imaging surface) of the spectral sensor 31.

[0018] Figure 3 is a schematic diagram showing an example of the configuration of the pixel array section 31a of the spectroscopic sensor 31. As shown in the figure, the pixel array section 31a is formed with a spectral pixel unit Pu, which consists of multiple pixels Px that receive light in different wavelength bands, arranged in a predetermined pattern in two dimensions. The pixel array section 31a is made up of multiple spectral pixel units Pu arranged in two dimensions.

[0019] In the example shown in the figure, each spectral pixel unit Pu receives light from a total of eight wavelength bands, from λ1 to λ8, individually at each pixel Px. In other words, the number of wavelength bands that each spectral pixel unit Pu receives (hereinafter referred to as the "number of receiving wavelength channels") is "8". However, this is merely an illustrative example, and the number of receiving wavelength channels in a spectral pixel unit Pu can be at least multiple and can be set arbitrarily. Hereafter, we will define the number of light-receiving wavelength channels in the spectral pixel unit Pu as "N".

[0020] In Figure 2, the spectral image generation unit 32 generates M wavelength band images based on the RAW image output from the spectral sensor 31. Here, in this example, "M>N", and for example, N=8 and M=41.

[0021] Furthermore, the number of M can also be the number of wavelength channels required for subject analysis. For example, as will be explained later, the calculation of vegetation evaluation values ​​used in this example requires spectral information from only two wavelengths: near-infrared (NIR) and red (RED). In this case, at least M=2 is sufficient. As can be seen from this point, the above-mentioned "M>N" condition is merely an example.

[0022] The spectral image generation unit 32 includes a demosaicing unit 35 and a bandwidth image generation unit 36. The demosaicing unit 35 performs demosaicing on the RAW image from the spectral sensor 31. This demosaicing process, similar to demosaicing on a Bayer array, involves interpolation based on the regularity of the two-dimensional array of color filters formed for each pixel Px to obtain the brightness value (on a pixel-by-pixel basis) for each wavelength band (λ1 to λ8 in this example) for each pixel Px.

[0023] The bandwidth image generation unit 36 ​​generates M wavelength bandwidth images from N wavelength band images by performing bandwidth processing (linear matrix processing) based on the images of each wavelength band for N channels obtained by demosaicing.

[0024] Figure 4 is an explanatory diagram of the banding process used to obtain M wavelength band images. Based on the N-channel wavelength band image obtained by demosaicing by the demosaicing unit 8, a predetermined matrix operation is performed for each pixel position to obtain an M-channel wavelength band image. In order to convert an N-channel wavelength band image into an M-channel wavelength band image, the pixel values ​​for N channels (in the figure, I0 to I0) are used for each pixel position. N-1 ) is used in matrix operations to obtain pixel values ​​for M channels (in the figure, I'0 to I' M-1 The process that obtains ) is called bandwidth optimization.

[0025] Here, if R is the pixel value after demosaicing, n is the input wavelength channel (from 0 to N-1), C is the narrowbanding coefficient, B is the output pixel value after bandwidthening, and m is the output wavelength channel (from 0 to M-1), then the calculation formula for bandwidthening can be expressed as shown in [Equation 1] below.

number

[0026] That is, the pixel value of the m=0-th output wavelength channel is B0 = R[0] × C0[0] + R[1] × C0[1] + R[2] × C0[2] + ··· + R[N-1] × C0[N-1], and the pixel value of the m=1-th output wavelength channel is B1 = R[0] × C1[0] + R[1] × C1[1] + R[2] × C1[2] + ··· + R[N-1] × C1[N-1]. The same applies hereinafter, and the pixel value B of the last m=M-1-th output wavelength channel M-1 = R[0] × C M-1 [0] + R[1] × C M-1 [1] + R[2] × C M-1 [2] + ··· + R[N-1] × C M-1 [N-1]. At this time, as the banding coefficients C, C0[0] to C0[N-1] for obtaining the pixel value B0, C1[0] to C1[N-1] for obtaining the pixel value B1, ..., C for obtaining the pixel value B M-1 for obtaining the pixel value B M-1 [0] to C M-1 [N-1], a total of N×M pieces are used.

[0027] In FIG. 2, the control unit 33 is configured to include, for example, a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and performs overall control of the spectroscopic camera 3 by causing the CPU to execute processing based on a program stored in, for example, the ROM or a program loaded into the RAM. For example, the control unit 33 controls the execution of operations of the spectroscopic sensor 31 and the spectral image generation unit 32. The control unit 33 also performs processing for transmitting and receiving various data to and from an external device via the communication unit 34. Particularly in this example, transmission and reception of data to and from the information processing device 1 is enabled.

[0028] The communication unit 34 performs wired or wireless data communication with an external device. For example, the communication unit 34 may be configured to perform wired data communication with an external device in accordance with a predetermined wired communication standard such as the USB (Universal Serial Bus) communication standard, wireless data communication with an external device in accordance with a predetermined wireless communication standard such as the Bluetooth (registered trademark) communication standard, or wireless or wired data communication with an external device via a predetermined communication network such as the Internet.

[0029] (1-3. Configuration of Information Processing Equipment) Figure 5 is a block diagram showing an example of the hardware configuration of the information processing device 1. As shown in the figure, the information processing device 1 includes a processor 11. The processor 11 is configured to have at least a CPU and executes various processes according to a program stored in the ROM 12 or a program loaded from the storage unit 19 into the RAM 13. Here, in order to efficiently perform various image signal processing, the processor 11 may also be configured to have a GPU (Graphics Processing Unit) in addition to the CPU.

[0030] RAM13 also stores data necessary for the processor 11 to perform various processes. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output interface (I / F) 15 is also connected to this bus 14.

[0031] An input unit 16, consisting of controls or operating devices, is connected to the input / output interface 15. For example, the input unit 16 could be various controls or operating devices such as a keyboard, mouse, keys, dial, touch panel, touchpad, or remote controller. The input unit 16 detects user operations, and the signal corresponding to the input operation is interpreted by the processor 11.

[0032] Furthermore, a display unit 17 consisting of an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 18 consisting of a speaker, are connected to the input / output interface 15, either as an integrated unit or as separate components. The display unit 17 is used for displaying various types of information and is composed of, for example, a display device provided on the casing of a computer device, or a separate display device connected to a computer device.

[0033] The display unit 17 displays images for various image processing tasks and videos to be processed on the display screen based on instructions from the processor 11. The display unit 17 also displays various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the processor 11.

[0034] The input / output interface 15 may also be connected to a storage unit 19 consisting of an HDD (Hard Disk Drive) or solid memory, or a communication unit 20 consisting of a modem or the like.

[0035] The communication unit 20 performs wired or wireless data communication with external devices, specifically the spectroscopic camera 3 and spectroscopic camera 5 in this example. Similar to the communication unit 34 described earlier, the communication unit 20 can be configured to perform wired data communication with external devices in accordance with a predetermined wired communication standard such as the USB communication standard, wireless data communication with external devices in accordance with a predetermined wireless communication standard such as the Bluetooth communication standard, or wireless or wired data communication with external devices via a predetermined communication network such as the Internet. The processor 11 can send and receive data with external devices via the communication unit 20.

[0036] The input / output interface 15 is also connected to a drive 21 as needed, and a removable recording medium 22 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted there.

[0037] Drive 21 allows data (including computer programs, etc.) used for various processes to be read from the removable recording medium 22. The read data is stored in the storage unit 19, or, if the read data is image data or audio data, the image or audio is output by the display unit 17 or the audio output unit 18. The computer program read from the removable recording medium 22 is installed in the storage unit 19 as needed.

[0038] In the information processing device 1 having the hardware configuration described above, for example, the software for processing in this embodiment can be installed via network communication by the communication unit 20 or via the removable recording medium 22. Alternatively, the software may be pre-stored in the ROM 12 or the storage unit 19, etc. The processor 11 performs processing operations based on various programs, thereby executing the necessary information processing and communication processing as the information processing device 1.

[0039] Furthermore, the information processing device 1 is not limited to being a single computer device as shown in Figure 5, but may be configured as a system of multiple computer devices. These multiple computer devices may be systematized via a LAN (Local Area Network), or they may be located remotely via a VPN (Virtual Private Network) using the Internet, etc. The multiple computer devices may also include computer devices that function as a group of servers (cloud) available through cloud computing services.

[0040] (1-4. Subject Analysis Method as the First Embodiment) As mentioned earlier, the light sensed by the spectral sensor may include surface-reflected light components of the subject, and these surface-reflected light components, like the light source components, can become noise components when analyzing the subject based on its spectral information.

[0041] Therefore, in this embodiment, in order to suppress the surface reflected light component in the spectral image obtained by the spectral camera 3, the subject light is sensed via a polarizing filter in the spectral camera 3. The subject light referred to here is obtained when light from a light source (the sun in this example) is reflected by the subject.

[0042] Specifically, in this example, the spectroscopic camera 3 is equipped with the rotary polarizing filter 40 shown in Figure 2. The rotary polarizing filter 40 is a rotating polarizing filter whose polarization axis angle changes as it is driven to rotate. In this example, the rotary polarizing filter 40 is mounted in the imaging optical system 30 at a position in front of the cover lens (on the subject side). Note that placing the rotary polarizing filter 40 at the front end of the imaging optical system is merely one example; the rotary polarizing filter 40 can be placed at any position in the optical path to the spectroscopic sensor 31.

[0043] In this example, the spectroscopic camera 3 is assumed to be mounted on an aircraft such as a drone, and is intended to perform imaging from a relatively high altitude. Therefore, the operation to rotate the rotary polarizing filter 40 is assumed to be performed remotely via the information processing device 1. For this reason, the spectroscopic camera 3 is provided with an actuator 41 that rotates the rotary polarizing filter 40. The actuator 41 is, for example, a motor or a solenoid, and is provided as a power source for rotating the polarizing filter 40. The power from this actuator 41 is transmitted to the rotary polarizing filter 40 via a drive mechanism (not shown), thereby driving the rotary polarizing filter 40 to rotate.

[0044] In this example, the information processing device 1 accepts an operation to rotate the rotary polarizing filter 40. This operation allows for the specification of the polarization axis angle. For example, there are controls for increasing the polarization axis angle (from 0 to 179 degrees) and controls for decreasing it, and the polarization axis angle can be specified by operating these controls. Note that the controls referred to here are not limited to physical controls, but may also include virtual controls displayed on the screen. The received operation information is transferred to the control unit 33 of the spectroscopic camera 3, and the control unit 33 controls the operation of the actuator 41 based on this operation information. As a result, the rotary polarizing filter 40 is rotated to a polarization axis angle corresponding to the user operation.

[0045] Furthermore, in this example, the information processing device 1 has an automatic polarization axis angle adjustment function. Specifically, the information processing device 1 in this example is provided with two polarization axis angle adjustment modes: a manual adjustment mode in which the polarization axis angle is adjusted according to the operation of specifying the polarization axis angle as described above, and an automatic adjustment mode in which the polarization axis angle is adjusted to the polarization axis angle calculated by the information processing device 1. When the automatic adjustment mode is instructed, the information processing device 1 calculates the polarization axis angle that suppresses the surface reflected light component using a predetermined calculation method, and instructs the control unit 33 of the spectroscopic camera 3 to the calculated polarization axis angle. In this case, the control unit 33 controls the actuator 41 so that the polarization axis angle of the rotary polarization filter 40 becomes the instructed polarization axis angle.

[0046] The GNSS sensor 42 detects the position information of the spectral camera 3 and outputs it to the control unit 33.

[0047] The orientation sensor 43 detects orientation information indicating the direction of the spectral camera 3, specifically orientation information indicating the direction of the optical axis (imaging optical axis) of the imaging optical system 30. In this example, this orientation information is considered to be three-dimensional orientation information, indicating not only the cardinal directions (east, west, north, south) but also the direction in the height direction. In other words, it is information indicating the three-dimensional direction in the real-world coordinate system.

[0048] The position information detected by the GNSS sensor 42 and the direction information detected by the direction sensor 43 are used to calculate the polarization axis angle in the automatic adjustment mode described above, but the details will be described later.

[0049] Figure 6 is a functional block diagram illustrating the various functions of the processor 11 in the information processing device 1 related to the subject analysis method as a first embodiment. As shown in the figure, the processor 11 has the functions of a correction processing unit F1, a vegetation evaluation value calculation unit F2, a polarization axis angle calculation unit F3, and a control unit F4.

[0050] The correction processing unit F1 receives a spectral image obtained by sensing subject light, which is obtained when light from a light source is reflected by the subject, via a polarizing filter using a spectral sensor 31, and performs a correction process to remove the light source component, which is a component of the light source, from the spectral image using the spectral information of the light source. Specifically, the correction processing unit F1 in this example receives a spectral image from the spectral camera 3, which is a spectral image of the subject, consisting of the M wavelength band images mentioned above, and also receives spectral information of the light source from the spectral camera 5. Then, the correction processing unit F1 performs a correction process to remove the light source component from the spectral image based on the input spectral information of the light source. This correction process is performed by dividing the brightness value of the spectral image by the brightness value of the corresponding wavelength band in the spectral information of the light source.

[0051] The vegetation evaluation value calculation unit F2 calculates the vegetation evaluation value based on the spectral image after correction processing by the correction processing unit F1. Here, the vegetation evaluation value is calculated on a pixel-by-pixel basis in the spectral image. An example of a specific evaluation value is NDVI (Normalized Difference Vegetation Index). NDVI is calculated using the luminance values ​​in the NIR (near-infrared) wavelength band and the luminance values ​​in the RED (red) wavelength band, using the formula (NIR-RED) / (NIR+RED).

[0052] It should be noted that NDVI is not the only vegetation evaluation value; other evaluation values ​​such as SAVI (Soil-Adjusted Vegetation Index) and EVI (Enhanced Vegetation Index) can also be calculated. Here, SAVI is calculated as {(1+L)(NIR-RED)} / (NIR+RED+L), and EVI is calculated as G×(NIR-RED) / (NIR+C1×RED-C2×BLUE+L).

[0053] The polarization axis angle calculation unit F3 calculates the polarization axis angle that suppresses the surface reflected light component of the subject, based on information indicating the optical axis orientation of the spectroscopic camera 3 and information indicating the normal orientation of the subject surface.

[0054] In this example, we take advantage of the fact that the subject is a plant to estimate the normal direction of the subject's surface, specifically the normal direction of the leaf surface, based on the position of the sun, which is the light source. Since plant leaves have a tendency to orient themselves as much as possible towards the sun (phototropism), it is possible to estimate the normal direction of the leaf surface if the position of the sun is known. To realize this estimation, this example assumes that information showing the correspondence between the position of the sun and the normal direction of the leaf surface (hereinafter referred to as "sun position-leaf normal direction correspondence information") has been experimentally created. For example, by utilizing the fact that both the position of the sun and the normal direction of the leaf point to a single point on the celestial sphere, it is conceivable to divide the celestial sphere into C×D sections and create information showing the correspondence such as "when the sun is at celestial coordinates (c,d), the normal direction of the leaf points to (i,j)" as sun position-normal direction correspondence information.

[0055] The position of the sun can be estimated based on camera position information, which is the position information detected by the GNSS sensor 42 in the spectroscopic camera 3, and the current date and time information. Here, the current date and time information can be obtained from, for example, an NTP server. Note that the method for estimating the position of the sun based on position information and current date and time information is publicly known, so a detailed explanation will be omitted. By estimating the position of the sun, information indicating the normal direction of the leaf surface can be obtained using the above-mentioned correspondence information between the sun's position and the leaf's normal direction.

[0056] If the normal orientation of the leaf surface is known, the polarization axis angle that suppresses the surface reflected light component can be determined in relation to the optical axis orientation of the spectroscopic camera 3. In this case, it should be noted that for pixels Px far from the optical axis, the direction of the line of sight to the subject captured by that pixel Px will be shifted from the direction of the optical axis. In particular, the wider the field of view, the greater the amount of this shift in direction.

[0057] In this example, the calculation method for the polarization axis angle takes into account the amount of deviation in the line of sight direction at pixels Px that are far from the optical axis. For this purpose, in this example, it is assumed that information indicating the "deviation pattern (deviation direction and amount of deviation)" of the line of sight relative to the optical axis for each pixel Px of the spectroscopic camera 3 is created as "camera field of view information". In this example, it is assumed that this camera field of view information is stored in the spectroscopic camera 3, and the polarization axis angle calculation unit F3 receives the camera field of view information from the control unit 33 of the spectroscopic camera 3.

[0058] Note that using camera field-of-view information for each pixel Px is just one example and is not limited to this. Camera field-of-view information can be used for each predetermined unit image portion. A unit image portion refers to an image portion divided into units of at least 1x1 pixels or larger. By setting the size of the unit image portion to 2x2 pixels or larger, it is possible to reduce the computational load required to obtain a spectral image with suppressed surface reflected light components.

[0059] The polarization axis angle calculation unit F3 receives optical axis orientation information as orientation information detected by the orientation sensor 43 of the spectroscopic camera 3, and calculates the line of sight orientation for each unit image portion (each pixel Px in this example) based on this optical axis orientation information and the camera field of view information. The line of sight orientation is not the amount of deviation relative to the optical axis for each unit image portion, but rather information indicating the direction in the real-world coordinate system.

[0060] Then, the polarization axis angle calculation unit F3 calculates the polarization axis angle for suppressing the surface reflected light component for each unit image portion, based on the line-of-sight orientation information for each unit image portion and the normal orientation information for the leaf surface. Here, the surface reflected light component that we want to block with the rotary polarizing filter 40 is mainly the s-polarized component of the s-polarized and p-polarized components emitted from the sun and illuminating the subject. The vibration direction s of the s-polarized light that we want to block with this rotary polarizing filter 40 can be determined by "s = o × v", where o is the normal direction of the leaf surface and v is the line of sight direction of the target unit image portion. The normal direction o and line-of-sight direction v of the leaf surface can be considered as three-dimensional vectors, and the direction of vibration s can be expressed as their cross product (o × v).

[0061] Then, the polarization axis angle calculation unit F3 calculates the polarization axis angle that suppresses the surface reflected light component based on the transmission axis vector p of the rotary polarization filter 40, which is obtained by the following formula. p = L × s Here, L is the optical axis orientation indicated by the optical axis orientation information. In other words, the transmission axis vector p can be obtained by the cross product of the optical axis orientation L and the vibration direction s.

[0062] In this case, to obtain a spectral image with suppressed surface reflection components, it is necessary to perform imaging for each unit image portion, adjusting the polarization axis angle according to the calculated transmission axis vector p. These multiple imaging sequences are achieved through the control of the control unit F4.

[0063] The control unit F4 controls the actuator 41 in the spectroscopic camera 3. Specifically, when manual adjustment mode is instructed, the control unit F4 controls the actuator 41 in accordance with the operation to rotate the rotary polarizing filter 40, and when automatic adjustment mode is instructed, it controls the actuator 41 so that the polarization axis angle of the rotary polarizing filter 40 matches the polarization axis angle calculated by the polarization axis angle calculation unit F3.

[0064] In this example, when the manual adjustment mode is activated, the control unit F4 performs a process to display a monitor image of the sensing image from the spectral sensor 31 on the display unit 17 when the rotary polarizing filter 40 is rotated. This allows users to perform adjustments while monitoring the effects of changing the polarization axis angle on a monitor image. Here, the monitor image may display each of the M wavelength band images as a spectral image, or it may display an RGB image or a grayscale image generated based on the spectral image.

[0065] Furthermore, in automatic adjustment mode, the control unit F4 controls the actuator 41 based on the polarization axis angle calculated by the polarization axis angle calculation unit F3 for each unit image portion. Specifically, if there are X unit image portions (where X is a natural number greater than or equal to 2) and X polarization axis angles have been calculated, the control unit F4 controls the actuator 41 to sequentially change the polarization axis angle of the rotary polarization filter 40 from the first polarization axis angle to the Xth polarization axis angle, causing the spectroscopic sensor 31 to perform X imaging cycles, thereby generating X types of spectral images with different polarization axis angles in the spectroscopic camera 3. In other words, X types of images with different polarization axis angles are generated as M wavelength band image groups. In the x-th (any number from 1 to X) M wavelength band image group, only the x-th unit image portion is captured with a polarization axis angle that suppresses the surface reflected light component.

[0066] In this case, the correction processing unit F1, when in automatic adjustment mode, performs correction processing using light source spectral information for each unit image portion based on the X types of spectral images (M wavelength band image groups) captured under the control of the control unit F4 as described above. That is, for the spectral image obtained in the xth imaging, correction processing using light source spectral information is performed on the xth unit image portion. By performing such correction processing for each unit image portion, it is possible to analyze the subject based on spectral images in which the surface reflected light component is suppressed and the light source component is removed.

[0067] (1-5. Processing Procedure) A specific example of the processing procedure for realizing the subject analysis method as the first embodiment described above will be explained with reference to the flowcharts in Figures 7 to 8. The processes shown in Figures 7 and 8 are executed by the processor 11 in the information processing device 1 based on a program stored in a predetermined storage device, such as the ROM 12 or the memory unit 19.

[0068] Figure 7 is a flowchart of the process related to the manual adjustment mode. First, in step S101, the processor 11 waits for a manual adjustment instruction, that is, an instruction for manual adjustment mode.

[0069] If it is determined in step S101 that a manual adjustment instruction has been given, the processor 11 proceeds to step S102 and executes the process of starting the display of the captured image. That is, the process of starting the display of the monitor image on the display unit 17 as described above. In this example, since the monitor image is generated based on the spectral image obtained by the spectral camera 3, the process in step S102 also involves starting the spectral image generation operation of the spectral camera 3 and inputting the generated spectral image from the spectral camera 3.

[0070] In step S103, following step S102, the processor 11 performs adjustment operation reception and adjustment control processing. Specifically, it receives an operation from the user to rotate the rotary polarizing filter 40 and instructs the control unit 33 of the spectroscopic camera 3 to adjust the polarization axis angle of the rotary polarizing filter 40 to the polarization axis angle corresponding to the operation.

[0071] In step S104, following step S103, the processor 11 determines whether a determination operation has been performed. That is, it determines whether an instruction operation has been performed to determine the polarization axis angle adjusted by the operation as the polarization axis angle for obtaining the target spectral image in the correction processing by the correction processing unit F1. If it is determined in step S104 that no decision operation has been performed, the processor 11 returns to step S103. This allows the user to adjust the polarization axis angle while checking the degree of suppression of surface reflected light with the adjusted polarization axis angle on the monitor image.

[0072] On the other hand, if it is determined in step S104 that a decision operation has been performed, the processor 11 proceeds to step S105 and performs correction and evaluation value calculation processing. Specifically, it performs correction processing using the light source spectral information input from the spectral camera 5 on the spectral image captured at the determined polarization axis angle, and calculates vegetation evaluation values ​​such as NDVI based on the spectral image after the correction processing.

[0073] The processor 11 completes the series of processes shown in Figure 7 after executing the process in step S105.

[0074] Figures 8 and 9 are flowcharts of the process related to the automatic adjustment mode. In step S111, the processor 11 is waiting for an automatic adjustment instruction, i.e., an instruction for automatic adjustment mode.

[0075] If it is determined in step S111 that a manual adjustment instruction has been given, the processor 11 proceeds to step S112 and calculates the position of the sun based on the camera position and date and time information. That is, it inputs the camera position information detected by the GNSS sensor 42 in the spectroscopic camera 3 and estimates the position of the sun based on the camera position information and the current date and time information.

[0076] In step S113, following step S112, the processor 11 estimates the leaf's normal azimuth vector based on the sun's position. That is, using the aforementioned sun position-leaf normal azimuth correspondence information, it obtains information indicating the normal azimuth of the leaf's surface based on the sun's position.

[0077] In step S114, following step S113, the processor 11 receives input for the optical axis orientation. That is, it receives input for the optical axis orientation information as orientation information detected by the orientation sensor 43 in the spectroscopic camera 3.

[0078] In step S115, following step S114, the processor 11 calculates the line-of-sight direction, which is the direction relative to the optical axis direction for each unit image portion, based on the camera field-of-view information. That is, using the camera field-of-view information, it obtains information indicating the directional deviation pattern (deviation direction and deviation amount) with respect to the optical axis direction for each unit image portion, and calculates the line-of-sight direction (three-dimensional direction information) for each unit image portion based on this directional deviation pattern information and the optical axis direction information (three-dimensional direction information) input in step S114.

[0079] In step S116, following step S115, the processor 11 calculates the polarization axis angle for each unit image portion based on the leaf normal direction vector, optical axis direction, and line of sight direction. That is, it calculates the polarization axis angle for suppressing the surface reflected light component for each unit image portion. Note that the specific method for calculating the polarization axis angle for suppressing the surface reflected light component for each unit image portion based on the leaf normal direction vector, optical axis direction, and line of sight direction has already been explained, so a redundant explanation will be avoided.

[0080] In response to having performed the process in step S116, the processor 11 proceeds to step S117 shown in Figure 9.

[0081] In step S117, the processor 11 sets the total number of polarization axis angles to be adjusted = X. This X can be rephrased as the total number of unit image portions in the spectral image.

[0082] In step S118, following step S117, the processor 11 sets the target polarization axis angle identifier x to 1.

[0083] Then, in the following step S119, the processor 11 instructs the spectral camera 3 to take an image at the x-th target polarization axis angle, and in step S120, it performs correction and evaluation value calculation processing for the unit image portion corresponding to the x-th target polarization axis angle. Specifically, it performs correction processing using light source spectral information on the x-th unit image portion of the spectral image obtained in response to the instruction in step S119, and calculates vegetation evaluation values ​​such as NDVI for the unit image portion after the correction processing.

[0084] In step S121, following step S120, the processor 11 determines whether the total number of target polarization axis angle identifiers x is greater than or equal to X. This is equivalent to determining whether the correction and evaluation value calculation process in step S120 has been completed for all unit image portions (viewing orientation).

[0085] In step S121, if the determination result is obtained that the target polarization axis angle identifier x is not greater than or equal to the total number X, the processor 11 proceeds to step S122, increments the target polarization axis angle identifier x by 1, and then returns to step S119. This makes it possible to perform the correction and evaluation value calculation process in step S120 for all unit image portions.

[0086] On the other hand, if in step S121 the determination result is obtained that the total number of target polarization axis angle identifiers x is X or more, the processor 11 completes the series of processes shown in Figures 8 and 9.

[0087] The above example illustrates a calculation method for determining the polarization axis angle that takes into account the amount of deviation in the line of sight direction at pixels Px that are far from the optical axis, but this is merely one example. For example, if the field of view of the spectroscopic camera 3 is telephoto, the amount of deviation in the line of sight direction at pixels Px far from the optical axis can be ignored. In such cases, it is sufficient to perform a correction process using light source spectral information on a single spectral image obtained by having the spectroscopic camera 3 perform imaging at the polarization axis angle determined for the optical axis orientation, and then calculate the vegetation evaluation value based on the corrected spectral image.

[0088] Furthermore, regarding the optical axis orientation, if it is assumed that the spectroscopic camera 3 will take images at a specific angle, it is not necessary to perform detection using the orientation sensor 43, and the orientation information indicating the specific angle can be used as the optical axis orientation information.

[0089] Furthermore, regarding the normal direction of the surface of a subject, if the direction of the surface normal can be considered to be fixed in a specific direction, such as in the case of the leaves of a plant that spreads along the ground, then instead of estimating it from information such as the position of the sun, it is possible to use azimuth information as a fixed value indicating that specific direction.

[0090] Furthermore, it is also possible to use user-inputted values ​​for the optical axis direction and the normal direction of the subject surface.

[0091] <2. Second Embodiment> (2-1. First example) Next, a second embodiment will be described. The second embodiment aims to suppress the surface reflected light component from each reflective surface when there are multiple reflective surfaces with different orientations within the frame, such as when leaves with different orientations are captured within the frame. Below, we will first describe a first example of the second embodiment.

[0092] Figure 10 is a block diagram showing the configuration overview of the subject analysis system as a first example in the second embodiment. In the following explanation, parts that are the same as those already explained will be denoted by the same reference numeral and their explanation will be omitted.

[0093] In this example of the subject analysis system, a spectroscopic camera 3A is used instead of the spectroscopic camera 3, and the system differs from the subject analysis system of the first embodiment in that multiple spectroscopic cameras 3A are provided, each with a different polarization axis angle. In this example, four types of spectroscopic cameras 3A are provided, with the polarization axis angles of the rotary polarization filter 40 set to 0 degrees, 45 degrees, 90 degrees, and 135 degrees, respectively. When distinguishing these spectroscopic cameras 3A with different polarization axis angles, as shown in the figure, the designation for polarization axis angle = 0 degrees is "3A_1", the designation for polarization axis angle = 45 degrees is "3A_2", the designation for polarization axis angle = 00 degrees is "3A_3", and the designation for polarization axis angle = 135 degrees is "3A_4".

[0094] Furthermore, in the subject analysis system of this example, an information processing device 1A is provided in place of information processing device 1. The hardware configuration of information processing device 1A is the same as that of information processing device 1, but as will be described later, the functions of the processor 11 differ from those of information processing device 1. For this reason, the processor 11 provided in information processing device 1A will be referred to as "processor 11A" below.

[0095] As shown in the diagram, the information processing device 1A receives spectral information from the spectral camera 5 for the light source, as well as spectral images of subjects from each spectral camera 3A.

[0096] Figure 11 is a block diagram showing a schematic configuration example of the spectroscopic camera 3A. The spectroscopic camera 3A differs from the spectroscopic camera 3 (Figure 2) in that the actuator 41, GNSS sensor 42, and orientation sensor 43 are omitted, and a grayscale image generation unit 45 is provided.

[0097] Since the polarization axis direction of the spectroscopic camera 3A is fixed at 0 degrees, 45 degrees, 90 degrees, or 135 degrees, the actuator 41 is omitted in this example. However, this is just one example, and a configuration with the actuator 41 is also conceivable. In the second embodiment, since the polarization axis angle can be fixed in each spectroscopic camera 3A, it is not essential to provide a rotary polarizing filter 40 as the polarizing filter 40.

[0098] Furthermore, in the second embodiment, the process of calculating the polarization axis angle to suppress the surface reflected light component based on the normal direction of the leaf surface and the optical axis direction, as in the first embodiment, is not performed, so the GNSS sensor 42 and the orientation sensor 43 are omitted.

[0099] The grayscale image generation unit 45 receives the RAW image (see Figure 3) output from the spectral sensor 31, calculates grayscale values ​​that serve as luminance index values ​​for each predetermined unit image portion, and obtains a grayscale image. Specifically, the grayscale image generation unit 45 calculates a grayscale value for each spectral pixel unit Pu by averaging the luminance values ​​of pixels Px for each spectral pixel unit Pu, and generates a grayscale image consisting of the calculated grayscale values. This grayscale image can be used as information for estimating the amount of surface reflected light received for each unit image portion. Here, the grayscale image generation unit 45 generating a grayscale image is an example of calculating the luminance index value for each unit image portion for a spectral image at a certain polarization axis angle.

[0100] Figure 12 is a functional block diagram illustrating the various functions of the processor 11A in the information processing device 1A related to the subject analysis method as a first example of the second embodiment. Compared to processor 11, processor 11A is similar in that it has a vegetation evaluation value calculation unit F2, but differs in that it has a correction processing unit F1A instead of a correction processing unit F1, and does not have a polarization axis angle calculation unit F3 and a control unit F4, but has an angle identification unit F5 and an image arrangement processing unit F6.

[0101] The angle determination unit F5 receives the grayscale images generated by the grayscale image generation unit 45 in each spectral camera 3A (3A_1 to 3A_4), i.e., the grayscale images with polarization axis angles of 0 degrees, 45 degrees, 90 degrees, and 135 degrees, as a group of grayscale images, and for each spectral pixel unit Pu, it identifies the polarization axis angle at which the surface reflected light component is estimated to be suppressed based on the grayscale value. Specifically, the angle determination unit F5 compares the grayscale values ​​for each spectral pixel unit Pu and identifies the grayscale image with the smallest grayscale value, i.e., the polarization axis angle. As a result, for each region of the spectral pixel unit Pu in the spectral image, the polarization axis angle at which the surface reflected light component is estimated to be suppressed is identified.

[0102] Here, the angle determination unit F5 processing described above assumes that the pixel positional relationships between images captured by each spectral camera 3A have been determined. These pixel positional relationships between images can be determined, for example, by a feature point matching method used in stereo cameras. Specifically, by performing feature point matching on the images obtained from each spectral camera 3A, the pixel positional relationships between each spectral camera 3A can be determined in terms of which pixels Px capture the same subject point as which other pixels Px. Furthermore, as the image captured by each spectral camera 3A used to determine such pixel positional relationships, for example, an image generated based on the RAW image obtained by the spectral sensor 31 of each spectral camera 3A, such as an RGB image, can be used. This RGB image can be generated by performing a linear matrix operation for RGB image generation on the N wavelength band images obtained by demosaicing the RAW image.

[0103] The image processing unit F6 receives subject spectral images for each polarization axis angle obtained by each spectral camera 3A as a group of subject spectral images, and based on the polarization axis angle information for each spectral pixel unit Pu obtained by the angle identification unit F5, it assembles a set of spectral images from the group of subject spectral images. Specifically, the image composition processing unit F6, for each of the M wavelength band images input to the subject spectral image group, i.e., M images for each polarization axis angle, determines the pixels of the image at the polarization axis angle specified by the angle specification unit F5 as the pixels to be used for each spectral pixel unit Pu, and then maps the determined pixels to one image for each of the M wavelength bands, thereby composing M wavelength band images, i.e., one set of spectral images.

[0104] Each of the M wavelength band images compiled by the image processing unit F6 as described above consists only of pixels whose surface reflected light component is estimated to be suppressed during processing by the angle determination unit F5. Therefore, even if there are multiple reflective surfaces with different orientations within the image frame, a spectral image can be obtained in which the reflected light component from each reflective surface is suppressed.

[0105] The correction processing unit F1A performs a light source component removal process (correction process) using light source spectral information on the spectral image compiled by the image compilation processing unit F6. In other words, for each pixel unit Pu, the correction process is performed on the spectral image of the polarization axis angle identified by the angle identification unit F5.

[0106] In this case, the vegetation evaluation value calculation unit F2 calculates the vegetation evaluation value based on the spectral image after correction processing by the correction processing unit F1A.

[0107] Figure 13 is a flowchart showing an example of the processing procedure that the processor 11A should perform in order to realize the subject analysis method as the first example in the second embodiment described above. The process shown in Figure 13 is executed by the processor 11A based on a program stored in a predetermined storage device, such as the ROM 12 or storage unit 19 of the information processing device 1A.

[0108] First, in step S151, the processor 11A performs a process to determine the pixel position relationship between cameras. That is, as illustrated earlier, it performs a process to determine the pixel position relationship by performing feature point matching on the image (for example, an RGB image) generated based on the RAW image obtained by the spectral sensor 31 of each spectral camera 3A.

[0109] It is not mandatory for the processor 11A to perform pixel position relationship determination processing. For example, it is possible to use pixel position relationship determination information obtained in advance by another device.

[0110] In step S152, following step S151, the processor 11A receives a group of grayscale images and performs a process to identify the grayscale image (polarization axis angle) with the smallest grayscale value for each spectral pixel unit Pu.

[0111] In step S153, following step S152, the processor 11A determines, for each spectral pixel unit Pu, each pixel in the image with the specified polarization axis angle to be used. That is, the processor 11A receives a group of subject spectral images as input, and for the input group of subject spectral images, i.e., M wavelength band images for each polarization axis angle, it determines, for each spectral pixel unit Pu, the pixels in the image with the polarization axis angle specified by the identification process in step S152 to be used.

[0112] Then, in step S154 following step S153, the processor 11A assembles a set of spectral images from the subject spectral images according to the determined information. That is, the processor 11A assembles M wavelength band images, i.e., a set of spectral images, by mapping the pixels determined in step S153 to one image for each of the M wavelength bands.

[0113] In step S155, following step S154, the processor 11A performs correction and evaluation value calculation processing on the compiled image. Specifically, it performs correction processing using light source spectral information on a set of spectral images compiled in step S154, and also performs vegetation evaluation value calculation processing based on the spectral image after correction processing.

[0114] Processor 11A completes the series of processes shown in Figure 13 after executing the process in step S155.

[0115] In the above example, we described a system that can simultaneously obtain multiple types of spectral images with different polarization axis angles as a subject spectral image group by using multiple spectral cameras 3A with different polarization axis angles. However, it is also possible to create a system that obtains multiple types of spectral images with different polarization axis angles in a time-resolved manner by using a spectral camera 3A equipped with a rotary polarization filter 40 and an actuator 41 and performing imaging while appropriately changing the polarization axis angle.

[0116] (2-2. Second example) The second example uses a spectral camera 3B in which the polarizing filter is formed as a pixel-level optical filter within the spectral sensor.

[0117] Figure 14 is a block diagram showing a schematic configuration example of the spectroscopic camera 3B used in the subject analysis system as a second example in the second embodiment. Compared to the spectroscopic camera 3A used in the first example, the spectroscopic camera 3B differs in that the rotary polarizing filter 40 is omitted, the spectroscopic sensor 31B having a polarizing filter layer 31p is provided instead of the spectroscopic sensor 31, an image processing unit 46 is added, and the spectroscopic image generation unit 32B is provided instead of the spectroscopic image generation unit 32.

[0118] Figure 15 shows an example of the configuration of the pixel array section 31aB of the spectroscopic sensor 31B. As shown in the figure, the pixel array section 31aB has a spectral pixel unit PC which is made up of multiple polarizing pixel units PP arranged in a two-dimensional array. Each polarization pixel unit PP consists of multiple types of pixels Px, each selectively receiving light with a different polarization angle. Specifically, in this example, the polarization pixel unit PP consists of a total of four types of pixels Px: a pixel Px that selectively receives light with a polarization angle of 0 degrees, a pixel Px that selectively receives light with a polarization angle of 45 degrees, a pixel Px that selectively receives light with a polarization angle of 90 degrees, and a pixel Px that selectively receives light with a polarization angle of 135 degrees. In each polarization pixel unit PP, these four types of pixels Px are arranged in a predetermined identical arrangement (i.e., a regular arrangement).

[0119] The spectral pixel unit PC is a pixel unit composed of multiple types of polarizing pixel units PP, each selectively receiving light in different wavelength bands, arranged in a predetermined two-dimensional pattern. Specifically, the spectral pixel unit PC in this example consists of four polarization pixel units PP arranged in a predetermined two-dimensional pattern: a polarization pixel unit PP that receives only light in the λ1 wavelength band (shown as a downward sloping diagonal line in the figure), a polarization pixel unit PP that receives only light in the λ2 wavelength band (shown as a textured surface in the figure), a polarization pixel unit PP that receives only light in the λ3 wavelength band (shown as a vertical line in the figure), and a polarization pixel unit PP that receives only light in the λ4 wavelength band (shown as a downward sloping diagonal line in the figure).

[0120] The pixel array section 31aB is comprised of a two-dimensional arrangement of the spectral pixel units PC described above. That is, multiple spectral pixel units PC are arranged in both the vertical (column) and horizontal (row) directions.

[0121] With the pixel array section 31aB configured as described above, it becomes possible to generate four types of polarization images ranging from 0 to 135 degrees, as four types of wavelength band images from λ1 to λ4. Note that while we have shown an example where N=4, this is merely an example for illustrative purposes, and the number of N is not particularly limited; for example, N=8 as in the first example.

[0122] Figure 16 illustrates the schematic cross-sectional structure of each pixel Px in the pixel array section 31aB. As shown in the figure, in this case, a polarizing filter 54 is inserted between the color filter 52 and the microlens 53 in the pixel Px. Below the color filter 52, a semiconductor layer 50 is formed on which a photodetector PD or the like as a light-receiving element is formed, and below the semiconductor layer 50, a wiring layer 51 is formed.

[0123] For clarification, the color filter 52 is an optical bandpass filter that selectively transmits light in a predetermined wavelength band. In this example, it is configured as a filter that transmits light in any of the wavelength bands λ1 to λ4 described above. Similarly, in this example, the polarizing filter 54 is configured as a filter that transmits light with a polarization angle of any of the above-mentioned 0 to 135 degrees. The formation layer of this polarizing filter 54 corresponds to the polarizing filter layer 31p shown in Figure 14. Note that the relative positions of the polarizing filter 54 and the color filter 52 may be reversed.

[0124] In Figure 14, the image arrangement unit 46 arranges images for each polarization axis angle based on the arrangement regularity of the polarization filter 54 from the RAM image obtained by the spectral sensor 31B.

[0125] The image processing process is shown as a transition from Figure 17A to Figure 17B. Figure 17A is the RAW image output by the spectroscopic sensor 31B, and Figure 17B is the polarization-separated image obtained through image processing. The image processing here involves extracting the luminance values ​​of pixels Px that receive light at the same polarization angle from the RAW image and compiling polarization-separated images for each polarization axis angle. In the polarization-separated image obtained through this image processing, as shown in Figure 17B, an image is obtained in which pixels that receive light in different wavelength bands are arranged in a predetermined pattern.

[0126] In Figure 14, the spectral image generation unit 32B generates a spectral image using polarization-separated images for each polarization axis angle obtained by the image organization process performed by the image organization unit 46 as described above as input. The spectral image generation unit 32B differs from the spectral image generation unit 32 in that it has a demosaicing unit 35B instead of a demosaicing unit 35, and a band image generation unit 36B instead of a band image generation unit 36.

[0127] The demosaicing unit 35B performs demosaicing on the polarization-separated images for each polarization axis angle obtained by the image assembly unit 46, as a pixel interpolation process according to the arrangement regularity of the color filter 52. This process yields N wavelength band images (four in this example, λ1 to λ4) for each polarization axis angle from 0 to 135 degrees, as shown in Figure 17C.

[0128] In Figure 14, the band image generation unit 36B receives N wavelength band images for each polarization axis angle obtained by the demosaicing unit 35B, and generates M wavelength band images, i.e., M channel spectral images, from the N wavelength band images for each polarization axis angle. With this band image generation unit 36B, the spectral image generation unit 32B outputs spectral images for each polarization axis angle, as shown in the figure.

[0129] Furthermore, in the spectroscopic camera 3B, the grayscale image generation unit 45 in this case takes the polarization-separated images for each polarization axis angle obtained by the image processing by the image processing unit 46 as input and generates grayscale images for each polarization axis angle. Specifically, in each polarization-separated image, the 2x2 pixel image portion where 4 pixels from λ1 to λ4 are arranged corresponds to the unit image portion for calculating the grayscale value. Therefore, grayscale images for each polarization axis angle are generated by averaging the brightness value of each pixel in each of these unit image portions.

[0130] In this second example, as explained in Figure 12 above, an information processing device 1A is used, which includes a processor 11A having a correction processing unit F1A, a vegetation evaluation value calculation unit F2, an angle identification unit F5, and an image organization processing unit F6. In this case, the processor 11A receives grayscale images for each polarization axis angle, which are generated by the grayscale image generation unit 45 using the polarization-separated images for each polarization axis angle obtained by the image arrangement unit 46 as input, as shown in Figure 12, and spectral images for each polarization axis angle generated by the band image generation unit 36B as a subject spectral image group. Except for this point, the processing content executed by the angle identification unit F5, image arrangement processing unit F6, correction processing unit F1A, and vegetation evaluation value calculation unit F2 is the same as that described in Figure 12. In this case, the correction processing unit F1A can be said to perform correction processing on the spectral image for at least one of the polarization axis angles among the spectral images for each polarization axis angle generated based on the images for each polarization axis angle compiled by the image compilation unit 46.

[0131] According to the subject analysis system as the second example in the second embodiment described above, in order to achieve subject analysis with a spectral image that suppresses surface reflected light components, it is not necessary to provide multiple spectral cameras 3A as in the subject analysis system as the first example. Therefore, by reducing the number of devices required to implement the system, it is possible to miniaturize the system size and reduce costs.

[0132] In the second embodiment, examples were given of generating spectral images with different polarization axis angles, namely 0 degrees, 45 degrees, 90 degrees, and 135 degrees; in other words, examples were given of generating spectral images with different polarization axis angles in 45-degree increments. However, the increments for the polarization axis angle can also be other than 45 degrees, such as every 10 degrees or every 5 degrees.

[0133] <3. Variant> Furthermore, the embodiments are not limited to the specific examples described above, and various other modified configurations can be adopted. For example, while the above example uses plants as the subject, the subject is not limited to plants. For instance, an example of a subject other than plants would be a transparent case, such as glass, containing an object inside. In this case, the analysis could be performed by analyzing the object based on its spectral information. Alternatively, the subject could be a pond or lake. In this case, the analysis could be performed by analyzing the organisms (such as algae or fish) below the water surface based on their spectral information.

[0134] Furthermore, the light source is not limited to the sun; this technology can be suitably applied even when the light source is something other than the sun.

[0135] Furthermore, while the above example cited obtaining light source spectral information by sensing with a spectral sensor, it is also conceivable to use information estimated from the current location and date / time information for light source spectral information related to the sun.

[0136] Furthermore, while the first embodiment mentions presenting a monitor image when the user adjusts the polarization axis angle, it is also conceivable to alternatively present the user with multiple spectral images captured at different polarization axis angles, and then have the user select one of these spectral images to be corrected by the correction processing unit F1.

[0137] Furthermore, while the above example shows a configuration in which the correction processing unit F1 (or F1A, hereafter the same) is located in a separate device from the spectroscopic camera 3 (or 3A, 3B, hereafter the same), it is also conceivable that the spectroscopic camera 3 has the correction processing unit F1. In other words, the correction processing is completed within the spectroscopic camera 3.

[0138] Furthermore, although the above examples show that the spectral image generation process (processing by the spectral image generation unit 32 in the first example of the first embodiment and the first example of the second embodiment, and processing by the spectral image generation unit 32B in the second example of the second embodiment) is performed on the spectral camera 3 side, the spectral image generation process can also be configured to be performed on the device (information processing device 1 or 1A) having the correction processing unit F1. Furthermore, in the second example of the second embodiment, the processing of the image composition unit 46 can also be performed on the device having the correction processing unit F1A (information processing device 1A). In this case, the processing of the grayscale image generation unit 45 can also be performed on the information processing device 1A.

[0139] <4. Summary of Embodiments> As described above, the information processing device (1,1A) as an embodiment includes a correction processing unit (F1,F1A) that receives a spectral image obtained by sensing subject light, which is obtained when light from a light source is reflected by the subject, through a polarizing filter using a spectral sensor, and performs a correction process to remove the light source component, which is a component of the light source, from the spectral image using spectral information of the light source. As described above, by using the spectral image obtained by sensing the subject light through a polarizing filter in the correction process, it becomes possible to use a spectral image in which the surface reflected light component of the subject has been suppressed for the correction process. Therefore, it becomes possible to use spectral images in which components unnecessary for the analysis of the subject are suppressed, thereby improving the accuracy of the subject analysis.

[0140] Furthermore, in the information processing device as an embodiment, the correction processing unit performs correction processing on a spectral image obtained by sensing the subject light with a spectral sensor via a polarizing filter, which is a rotary polarizing filter (rotary polarizing filter 40). The rotating polarizing filter is provided as part of the imaging optical system and is rotated to adjust the degree of suppression of surface reflected light components. By using a rotary polarizing filter as the polarizing filter, it becomes possible to use a standard spectroscopic sensor that does not have a polarizing filter, thus preventing limitations on the types of spectroscopic sensors that can be used.

[0141] Furthermore, in the information processing apparatus as an embodiment, the spectroscopic camera having a spectroscopic sensor has an actuator (41) that rotates a rotary polarizing filter, and is equipped with a control unit (F4) that controls the actuator. This eliminates the need for the user to directly touch and rotate the rotating polarizing filter when adjusting the degree of suppression of surface reflected light components. Therefore, even when the spectroscopic camera is located remotely from the information processing device, the degree of suppression of surface reflected light components can be adjusted by input operations to the information processing device. Alternatively, automatic adjustment of the degree of suppression of surface reflected light components without manual operation can be achieved.

[0142] Furthermore, in the information processing device as an embodiment, the control unit performs a process to display a monitor image of the sensing image from the spectroscopic sensor on the display unit when the operation to rotate the rotary polarizing filter is performed. This allows users to perform adjustments while monitoring the effects of changing the polarization axis angle on a monitor image. Therefore, the work efficiency can be improved when adjusting the degree of suppression of surface reflected light components.

[0143] Furthermore, in the information processing apparatus as an embodiment, a polarization axis angle calculation unit (F3) is provided that calculates a polarization axis angle for suppressing the surface reflected light component of a subject based on information indicating the position of the light source, information indicating the optical axis orientation of a spectroscopic camera having a spectroscopic sensor, and information indicating the normal orientation of the subject surface. The control unit controls the actuator so that the polarization axis angle of the rotary polarizing filter matches the polarization axis angle calculated by the polarization axis angle calculation unit. This allows the rotation of the rotary polarizing filter, which suppresses the surface reflection component of the subject, to be performed automatically. Therefore, it is possible to reduce the user's operational burden related to the analysis of the subject.

[0144] Furthermore, in the information processing device (1A) as an embodiment, for multiple types of spectral images obtained by sensing subject light through polarizing filters with different polarization axis angles, an angle determination unit (F5) is provided that inputs a luminance index value for each unit image portion in the multiple types of spectral images, and for each unit image portion, determines the polarization axis angle of the spectral image in which the surface reflected light component is estimated to be suppressed based on the luminance index value, and the correction processing unit (F1A) performs correction processing on the spectral image of the polarization axis angle determined by the angle determination unit for each unit image portion. As a result, even if there are multiple reflective surfaces with different orientations within the frame, the correction process will use, for each unit image portion, the image from among multiple spectral images in which the surface reflected light component is presumed to be suppressed. Therefore, even when multiple reflective surfaces with different orientations exist within the frame, it becomes possible to use a spectral image in which components unnecessary for the analysis of the subject are suppressed, thereby improving the accuracy of the subject analysis.

[0145] Furthermore, in the information processing apparatus as an embodiment, the polarizing filter is formed as a pixel-level optical filter within the spectral sensor. This eliminates the need to include a rotating polarizing filter in the imaging optical system, allowing for a smaller and lighter imaging optical system.

[0146] Furthermore, in the information processing device as an embodiment, the spectral sensor has multiple types of polarizing filters with different polarization axis angles arranged regularly in the planar direction, and the correction processing unit performs correction processing on the spectral image of at least one of the polarization axis angles from the spectral images for each polarization axis angle generated based on the images for each polarization axis angle organized based on the arrangement regularity of the polarizing filters. According to the above configuration, spectral images for each polarization axis angle can be obtained by sensing with a single spectral sensor. Therefore, it becomes unnecessary to use multiple spectroscopic cameras to obtain multiple types of spectroscopic images with different polarization axis angles, which reduces the number of devices required to realize the system, thereby enabling miniaturization of the system size and cost reduction.

[0147] Furthermore, in the information processing device as an embodiment, the subject is a plant, and it includes an evaluation value calculation unit (F2) that calculates a vegetation evaluation value based on the spectral image after correction processing by the correction processing unit. This allows for improved accuracy in analyzing subjects as plants using vegetation evaluation values.

[0148] An information processing method as an embodiment involves an information processing device receiving a spectral image obtained by sensing subject light, which is obtained when light from a light source is reflected by a subject, through a polarizing filter using a spectral sensor, and performing a correction process to remove the light source component, which is a component of the light source, from the spectral image using spectral information of the light source. This information processing method can also provide the same functions and effects as the information processing apparatus described in the above-described embodiment.

[0149] Here, as an embodiment, we can consider a program that causes a CPU, DSP (Digital Signal Processor), or a device including these to execute the processing described with reference to Figures 7 to 9 and Figure 13, etc. In other words, the program of the embodiment is a program readable by a computer device that takes a spectral image obtained by sensing subject light, which is obtained when light from a light source is reflected by a subject, through a polarizing filter using a spectral sensor, as input, and performs a correction process to remove the light source component, which is a component of the light source, from the spectral image using spectral information of the light source. Such a program allows the functions of the correction processing unit F1 or F1A described above to be realized in a device such as the information processing device 1 or 1A.

[0150] Such programs can be pre-recorded on a storage medium such as an HDD (Hard Disk Drive) built into a computer device, or on ROM within a microcomputer with a CPU. Alternatively, the data can be temporarily or permanently stored (recorded) on removable recording media such as flexible disks, CD-ROMs (Compact Disc Read Only Memory), MO (Magneto Optical) disks, DVDs (Digital Versatile Discs), Blu-ray Discs (registered trademark), magnetic disks, semiconductor memory, and memory cards. Such removable recording media can be provided as so-called packaged software. In addition to installing such programs from removable storage media to personal computers, they can also be downloaded from download sites via networks such as LANs (Local Area Networks) and the Internet.

[0151] Furthermore, such a program is suitable for providing a wide range of functions of the correction processing unit as an embodiment. For example, by downloading the program to a personal computer, portable information processing device, mobile phone, game console, video equipment, PDA (Personal Digital Assistant), etc., the personal computer, etc. can be used as a device to realize the functions of the correction processing unit of this disclosure.

[0152] Furthermore, the effects described herein are merely illustrative and not limited to those described herein, and other effects may also occur.

[0153] <5. This Technology> This technology can also be configured as follows: (1) The system includes a correction processing unit that receives a spectral image obtained by sensing subject light, which is obtained when light from a light source is reflected by the subject, via a polarizing filter using a spectral sensor, and performs a correction process to remove the light source component, which is a component of the light source, from the spectral image using the spectral information of the light source. Information processing device. (2) The correction processing unit, The correction process is performed on the spectral image obtained by sensing the subject light with the spectral sensor via the polarizing filter, which is a rotating polarizing filter. The information processing device described in (1) above. (3) The spectroscopic camera having the spectroscopic sensor has an actuator that rotates the rotary polarizing filter, The system includes a control unit for controlling the actuator. The information processing device described in (2) above. (4) The control unit, When the rotary polarizing filter is rotated, the monitor image of the sensing image from the spectroscopic sensor is displayed on the display unit. The information processing apparatus described in (2) or (3) above. (5) The system includes a polarization axis angle calculation unit that calculates a polarization axis angle to suppress the surface reflected light component of the subject, based on information indicating the position of the light source, information indicating the optical axis orientation of the spectroscopic camera having the spectroscopic sensor, and information indicating the normal orientation of the subject surface. The control unit, The actuator is controlled so that the polarization axis angle of the rotary polarizing filter matches the polarization axis angle calculated by the polarization axis angle calculation unit. The information processing device described in (3) above. (6) The system includes an angle determination unit that, for multiple types of spectral images obtained by sensing the subject light through polarizing filters with different polarization axis angles using a spectroscopic sensor, inputs a luminance index value for each unit image portion in the multiple types of spectral images, and determines the polarization axis angle of the spectral image in which the surface reflected light component is estimated to be suppressed based on the luminance index value for each unit image portion. The correction processing unit, For each unit image portion, the correction process is performed on the spectral image of the polarization axis angle identified by the angle identification unit. An information processing device as described in any of (1) to (3) above. (7) The polarizing filter is formed as a pixel-level optical filter within the spectroscopic sensor. The information processing device described in (1) above. (8) In the aforementioned spectroscopic sensor, multiple types of polarizing filters with different polarization axis angles are regularly arranged in the planar direction. The correction processing unit, Based on the images for each polarization axis angle, which are generated based on the arrangement regularity of the polarization filters, the correction process is performed on the spectral image for at least one of the polarization axis angles among the spectral images for each polarization axis angle. The information processing apparatus described in (1) or (6) above. (9) The subject of the above is a plant. The system includes an evaluation value calculation unit that calculates a vegetation evaluation value based on the spectral image after correction processing by the correction processing unit. An information processing device as described in any of (1) to (8) above. (10) Information processing device, A spectral image is obtained by sensing the subject light, which is obtained when light from a light source is reflected by the subject, through a polarizing filter using a spectral sensor. This spectral image is then input, and a correction process is performed to remove the light source component, which is a component of the light source, from the spectral image using the spectral information of the light source. Information processing methods. (11) A recording medium on which a program readable by a computer device is stored, A program is recorded in the computer device that implements a function to perform a correction process to remove the light source component, which is a component of the light source, from the spectral image obtained by sensing the subject light, which is obtained when light from a light source is reflected by the subject, via a polarizing filter using a spectroscopic sensor. Recording medium. [Explanation of Symbols]

[0154] 1,1A Information Processing Device 3,3A (3A_1 to 3A_4), 3B Spectroscopic camera (for subject) 5. Spectroscopic camera (for light source) 30 Imaging optical system 31,31B Spectroscopic Sensor 31a, 31aB Pixel array section 32,32B Spectral image generation section 33 Control Unit 34 Communications Department 35,35B Demosaicing Section 36,36B Bandwidth Image Generation Unit 40 Rotary Polarizing Filters 41 Actuator 42 GNSS sensors 43 Directional sensor Px pixels Pu Spectroscopic Pixel Unit 11,11A processor 16 Input section 17 Display section 19 Memory section 20 Communications Department F1, F1A Correction Processing Unit F2, F2A Vegetation Evaluation Value Calculation Unit F3 Polarization axis angle calculator F4 Control Unit F5 Pixel Usage Determination Section F6 Image Processing Unit 45 Image Compilation Section 46 Grayscale Image Generation Unit 31p polarizing filter layer PC Spectroscopic Pixel Unit PP polarizing pixel unit 50 Semiconductor Layers 51 Wiring layer 52 Color Filters 53 Microlenses 54 Polarizing filters

Claims

1. The system includes a correction processing unit that receives a spectral image obtained by sensing subject light, which is obtained when light from a light source is reflected by the subject, via a polarizing filter using a spectral sensor, and performs a correction process to remove the light source component, which is a component of the light source, from the spectral image using the spectral information of the light source. Information processing device.

2. The correction processing unit, The correction process is performed on the spectral image obtained by sensing the subject light with the spectral sensor via the polarizing filter, which is a rotating polarizing filter. The information processing apparatus according to claim 1.

3. The spectroscopic camera having the spectroscopic sensor has an actuator that rotates the rotary polarizing filter, The system includes a control unit for controlling the actuator. The information processing apparatus according to claim 2.

4. The control unit, When the rotary polarizing filter is rotated, the monitor image of the sensing image from the spectroscopic sensor is displayed on the display unit. The information processing apparatus according to claim 2.

5. The system includes a polarization axis angle calculation unit that calculates a polarization axis angle to suppress the surface reflected light component of the subject, based on information indicating the position of the light source, information indicating the optical axis orientation of the spectroscopic camera having the spectroscopic sensor, and information indicating the normal orientation of the subject surface. The control unit, The actuator is controlled so that the polarization axis angle of the rotary polarizing filter matches the polarization axis angle calculated by the polarization axis angle calculation unit. The information processing apparatus according to claim 3.

6. The system includes an angle determination unit that, for multiple types of spectral images obtained by sensing subject light through polarizing filters with different polarization axis angles using a spectroscopic sensor, inputs a luminance index value for each unit image portion in the multiple types of spectral images, and determines the polarization axis angle of the spectral image in which the surface reflected light component is estimated to be suppressed based on the luminance index value for each unit image portion. The correction processing unit, For each unit image portion, the correction process is performed on the spectral image of the polarization axis angle identified by the angle identification unit. The information processing apparatus according to claim 1.

7. The polarizing filter is formed as a pixel-level optical filter within the spectroscopic sensor. The information processing apparatus according to claim 1.

8. In the aforementioned spectroscopic sensor, multiple types of polarizing filters with different polarization axis angles are regularly arranged in the planar direction. The correction processing unit, Based on the images for each polarization axis angle, which are generated based on the arrangement regularity of the polarization filters, the correction process is performed on the spectral image for at least one of the polarization axis angles among the spectral images for each polarization axis angle. The information processing apparatus according to claim 1.

9. The subject of the above is a plant. The system includes an evaluation value calculation unit that calculates a vegetation evaluation value based on the spectral image after correction processing by the correction processing unit. The information processing apparatus according to claim 1.

10. Information processing device, A spectral image is obtained by sensing the subject light, which is obtained when light from a light source is reflected by the subject, through a polarizing filter using a spectral sensor. This spectral image is then input, and a correction process is performed to remove the light source component, which is a component of the light source, from the spectral image using the spectral information of the light source. Information processing methods.

11. A recording medium on which a program readable by a computer device is stored, A program is recorded in the computer device that implements a function to perform a correction process to remove the light source component, which is a component of the light source, from the spectral image obtained by sensing the subject light, which is obtained when light from a light source is reflected by the subject, via a polarizing filter using a spectroscopic sensor. Recording medium.

Images