Data processing device, data processing method and program

Abstract

To provide a data processing device, a data processing method and a program, capable of accurately estimating spectral reflectance of a subject even when a first temperature of an imaging environment in imaging a calibrating member is different from a second temperature of an imaging environment in imaging a subject.SOLUTION: A data processing device includes a processor. The processor acquires first imaging data obtained by imaging a calibrating member with a spectral imaging device in an imaging environment at a first temperature, and obtains second imaging data obtained by imaging a subject with the spectral imaging device in an imaging environment at a second temperature, and executes temperature-related correction processing for the second imaging data on the basis of the first imaging data and a correction value for the imaging data by the spectral imaging device.SELECTED DRAWING: Figure 19

Description

[Technical field]

[0001] The technology disclosed herein relates to a data processing device, a data processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses an image processing device that acquires a hyperspectral image of an object to be measured. The image processing device includes a spectroscopic means that spectroscopically separates reflected light from the object to be measured, a two-dimensional image sensor that receives the dispersed light and acquires imaging data, a memory that stores sensitivity characteristic data for a plurality of amounts of received light in each pixel of the image sensor, and a calculation means that corrects the imaging data with the stored sensitivity characteristic data and converts it into corrected imaging data.

[0003] Patent document 2 discloses a semiconductor device having a signal processing unit that performs signal processing to correct the spectral characteristics of an image signal obtained by imaging using a pixel array unit having a plurality of unit pixels, based on the temperature of the pixel array unit.

[0004] Patent Document 3 discloses an image processing device including a calibration data setting unit and an image correction unit. The calibration data setting unit sets, as calibration data, color chart image data at a temperature closer to the temperature at the time of photographing the subject among temperatures ranging from the temperature of the imaging device at the time of photographing the color chart before the subject is photographed to the temperature of the imaging device at the time of photographing the color chart after the subject is photographed, based on the temperature of the imaging device at the time of photographing the subject by the imaging device, the temperature of the imaging device at the time of photographing the color chart before and after the subject is photographed, and color chart image data acquired by photographing the color chart before and after the subject is photographed. The image correction unit corrects the image data of the subject acquired by photographing the subject using the calibration data set in the calibration data setting unit.

[0005] Patent Document 4 discloses a component analysis device including a housing, a light source unit, an incident optical system, a wavelength-tunable interference filter, an imaging unit, and a control unit. The light source unit is provided inside the housing and emits light to an object to be inspected. The incident optical system guides light reflected by the object to be inspected into the housing. The wavelength-tunable interference filter is provided inside the housing and extracts light of a predetermined wavelength from the light incident from the incident optical system. The imaging unit is provided inside the housing and receives the light extracted by the wavelength-tunable interference filter to capture a spectral image. The control unit is provided inside the housing and performs component analysis of the object to be inspected based on the spectral image. The wavelength-tunable interference filter is a Fabry-Perot etalon. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2010-256303 A [Patent Document 2] JP 2023-081054 A [Patent Document 3] JP 2010-081057 A [Patent Document 4] JP 2013-181912 A Summary of the Invention

[0007] One embodiment of the technology disclosed herein provides a data processing device, a data processing method, and a program that can accurately estimate the spectral reflectance of a subject even when a first temperature of an imaging environment when imaging a calibration member is different from a second temperature of the imaging environment when imaging the subject. [Means for solving the problem]

[0008] A first aspect of the technology of the present disclosure is a data processing device that includes a processor, which acquires first imaging data obtained by imaging a calibration member with a spectroscopic imaging device in an imaging environment where the temperature is a first temperature, acquires second imaging data obtained by imaging a subject with the spectroscopic imaging device in an imaging environment where the temperature is a second temperature, and performs temperature-related correction processing on the second imaging data based on the first imaging data and a correction value for the imaging data of the spectroscopic imaging device.

[0009] A second aspect of the technique of the present disclosure is the data processing device according to the first aspect, wherein the processor acquires the first temperature and the second temperature.

[0010] A third aspect of the technique of the present disclosure is the data processing device according to the first or second aspect, wherein the correction process is a process based on a difference between the first temperature and the second temperature.

[0011] A fourth aspect of the technology disclosed herein is a data processing device according to any one of the first to third aspects, wherein the processor generates a correction value to be used in the correction process based on temperature and imaging data.

[0012] A fifth aspect of the technique of the present disclosure is the data processing device according to the fourth aspect, wherein the correction value is a correction value selected from a group of correction values.

[0013] A sixth aspect of the technique of the present disclosure is the data processing device according to the fifth aspect, wherein the correction value group is stored in the memory in a table format that defines the relationship between the correction value and temperature.

[0014] A seventh aspect of the technology of the present disclosure is a data processing device according to the fifth or sixth aspect, wherein the correction process is a process of obtaining a first correction value corresponding to a first temperature from a group of correction values, obtaining a second correction value corresponding to a second temperature from the group of correction values, and deriving a second reflectance which is the spectral reflectance of the subject based on the first reflectance which is the spectral reflectance of the calibration member, the first imaging data, the second imaging data, the first correction value, and the second correction value.

[0015] An eighth aspect of the technology of the present disclosure is a data processing device according to the seventh aspect, in which, when the first reflectance is r1, the second reflectance is r2, the value of the first imaging data is w, the value of the second imaging data is u, the first correction value is c1, and the second correction value is c2, the second reflectance is calculated by equation (1). JPEG2025025139000002.jpg576

[0016] A ninth aspect of the technique of the present disclosure is a data processing device according to the seventh aspect, wherein the correction process includes a first correction process for correcting a value w of the first imaging data using equation (2) when the first reflectance is r1 and the first correction value is c1 to derive a corrected value w1 of the first imaging data, a second correction process for correcting a value u of the second imaging data using equation (3) when the second correction value is c2 to derive a corrected value u1 of the second imaging data, and a third correction process for deriving a second reflectance r2 using equation (4) based on the corrected value w1 of the first imaging data and the corrected value u1 of the second imaging data. JPEG2025025139000003.jpg657 JPEG2025025139000004.jpg651 JPEG2025025139000005.jpg655

[0017] A tenth aspect of the technology of the present disclosure is a data processing device according to any one of the first to ninth aspects, in which the spectroscopic imaging device is provided with an imaging element, and the processor is a data processing device that performs correction processing according to the imaging area of ​​the imaging element.

[0018] An eleventh aspect according to the technique of the present disclosure is the data processing device according to the tenth aspect, wherein the processor executes a correction process on a plurality of divided regions obtained by dividing an imaging region.

[0019] A twelfth aspect according to the technique of the present disclosure is a data processing device according to the eleventh aspect, wherein a correction value used in the correction process is set according to the divided area.

[0020] A thirteenth aspect of the technology of the present disclosure is a data processing device according to any one of the first to twelfth aspects, wherein the spectroscopic imaging device is provided with a pupil division filter having a plurality of spectral filters with different wavelength bands, and the plurality of divided regions have different division patterns depending on the wavelength band.

[0021] A fourteenth aspect of the technique of the present disclosure is a data processing device according to the thirteenth aspect, wherein the multiple divided regions are divided based on peak positions where the intensity of light in a wavelength band peaks on the imaging plane of an imaging element.

[0022] A fifteenth aspect of the technology disclosed herein is a data processing device according to the fourteenth aspect, wherein the imaging area includes a first area including a peak position and a second area farther from the peak position than the first area, and the first area has a larger area of ​​the divided area than the second area.

[0023] A 16th aspect of the technology disclosed herein is a data processing device relating to any one of the 11th to 15th aspects, in which the multiple divided areas are divided in at least one of the vertical and horizontal directions of the imaging area.

[0024] A seventeenth aspect of the technique of the present disclosure is the data processing device according to any one of the first to sixteenth aspects, wherein the processor derives an index related to the subject based on the second imaging data.

[0025] An eighteenth aspect of the technique of the present disclosure is the data processing device according to the seventeenth aspect, wherein the index includes a contrast in luminance of light in a plurality of wavelength bands.

[0026] A 19th aspect of the technique of the present disclosure is the data processing device according to the 17th or 18th aspect, wherein the index includes a normalized difference vegetation index.

[0027] A twentieth aspect of the technique of the present disclosure is the data processing device according to any one of the first to nineteenth aspects, wherein the spectroscopic imaging device is a multispectral camera.

[0028] A 21st aspect of the technique of the present disclosure is a data processing device according to the 20th aspect, in which the spectroscopic imaging device is a data processing device comprising an imaging element, a pupil division filter having a plurality of spectral filters with different wavelength bands and a plurality of first polarizers with different polarization directions, and a plurality of second polarizers with different polarization directions provided on the imaging element.

[0029] A 22nd aspect of the technique of the present disclosure is a data processing device according to the 21st aspect, wherein the processor acquires first spectral data, which is spectral data derived for each wavelength band from the first imaging data, based on spectral information about a plurality of spectral filters, first polarization information about a plurality of first polarizers, and second polarization information about a plurality of second polarizers; acquires second spectral data, which is spectral data derived for each wavelength band from the second imaging data, based on the spectral information, the first polarization information, and the second polarization information; and performs a correction process on the second spectral data based on the first spectral data and a correction value.

[0030] A 23rd aspect of the technology disclosed herein is a data processing device according to any one of the 1st to 22nd aspects, wherein a third reflectance, which is the spectral reflectance of a correction member used when obtaining a correction value used in the correction process, is a different spectral reflectance from the first reflectance, which is the spectral reflectance of a calibration member.

[0031] A 24th aspect of the technology of the present disclosure is a data processing method that includes obtaining first imaging data obtained by imaging a calibration member with a spectroscopic imaging device in an imaging environment where the temperature is a first temperature, obtaining second imaging data obtained by imaging a subject with the spectroscopic imaging device in an imaging environment where the temperature is a second temperature, and performing a temperature-related correction process on the second imaging data based on the first imaging data and a correction value for the imaging data of the spectroscopic imaging device.

[0032] A 25th aspect of the technology of the present disclosure is a program for causing a computer to execute processing including obtaining first imaging data obtained by imaging a calibration member with a spectroscopic imaging device in an imaging environment where the temperature is a first temperature, obtaining second imaging data obtained by imaging a subject with the spectroscopic imaging device in an imaging environment where the temperature is a second temperature, and performing temperature-related correction processing on the second imaging data based on the first imaging data and a correction value for the imaging data of the spectroscopic imaging device. [Brief description of the drawings]

[0033] [Figure 1] FIG. 1 is a block diagram illustrating an example of an inspection system. [Diagram 2] FIG. 1 is a perspective view showing an example of an imaging device. [Diagram 3] FIG. 2 is an exploded perspective view showing an example of a pupil division filter. [Figure 4] FIG. 2 is a block diagram showing an example of a hardware configuration of the imaging apparatus. [Diagram 5] FIG. 2 is an exploded perspective view showing an example of a portion of a photoelectric conversion element. [Figure 6] FIG. 2 is a block diagram showing an example of a functional configuration of an imaging apparatus. [Figure 7] 11 is a block diagram showing an example of the operation of an output value acquisition unit and an interference removal processing unit. FIG. [Figure 8] FIG. 4 is an explanatory diagram illustrating an example of a method for obtaining the spectral reflectance of a subject. [Figure 9]2 is a block diagram showing an example of a functional configuration for executing an inspection process in the processing device; FIG. [Figure 10] 11 is a block diagram showing an example of an operation of a first data acquisition unit. FIG. [Figure 11] 11 is a block diagram showing an example of an operation of a second data acquisition unit. FIG. [Figure 12] 11 is a block diagram showing an example of the operation of a wavelength band selection unit. FIG. [Figure 13] 11 is a block diagram showing an example of the operation of a region dividing unit and a temperature correcting unit. FIG. [Figure 14] FIG. 2 is a diagram showing an example of a division manner of a plurality of divided regions. [Figure 15] FIG. 13 is a diagram illustrating an example of a correction value group for each wavelength band. [Figure 16] 11A and 11B are diagrams illustrating an example of a manner in which a correction value group is acquired. [Figure 17] 4 is a block diagram showing an example of the operation of a temperature correction unit, an image data generation unit, and a display control unit. FIG. [Figure 18] 13 is a flowchart showing an example of the flow of a spectral image generation process. [Figure 19] 11 is a flowchart showing an example of the flow of an inspection process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] Hereinafter, an example of an embodiment of a data processing device, a data processing method, and a program according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0035] First, the terms used in the following description will be explained.

[0036] LED is an abbreviation for "light emitting diode". NDVI is an abbreviation for "Normalized Difference Vegetation Index". RGB is an abbreviation for "Red Green Blue". EL is an abbreviation for "Electro Luminescence". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". I / F is an abbreviation for "Interface". RAM is an abbreviation for "Random Access Memory". CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". HDD is an abbreviation for "Hard Disk Drive". DRAM is an abbreviation for "Dynamic Random Access Memory". SRAM is an abbreviation for "Static Random Access Memory". TPU is an abbreviation for "Tensor processing unit". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". ASIC is an abbreviation for "Application Specific Integrated Circuit". FPGA is an abbreviation for "Field-Programmable Gate Array". PLD is an abbreviation for "Programmable Logic Device". SoC is an abbreviation for "System-on-a-chip". IC is an abbreviation for "Integrated Circuit". IR is an abbreviation for "Infrared Rays".

[0037] In the description of this specification, "same" refers to sameness in the sense of including, in addition to complete sameness, an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and an error that does not go against the spirit of the technology of the present disclosure. In the description of this specification, "orthogonal" refers to orthogonal in the sense of including, in addition to complete orthogonal, an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and an error that does not go against the spirit of the technology of the present disclosure. In the description of this specification, "straight line" refers to a straight line in the sense of including, in addition to a perfect straight line, an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and an error that does not go against the spirit of the technology of the present disclosure.

[0038] Hereinafter, one embodiment of the technology of the present disclosure will be described.

[0039] 1, an inspection system 130 according to the first embodiment includes a plurality of light sources 132, a housing 134, a temperature sensor 136, an imaging device 10, and a processing device 90. The processing device 90 is an example of a "data processing device" according to the technology of the present disclosure.

[0040] The multiple light sources 132 are, for example, LED light sources, laser light sources, incandescent light bulbs, or the like. The light emitted from the multiple light sources 132 is unpolarized. As an example, the multiple light sources 132 are disposed at an upper part inside the housing 134. The number of the multiple light sources 132 may be any number, and instead of the multiple light sources 132, a single light source 132 may be used.

[0041] The housing 134 is configured to cover an imaging environment 138, which is an imaging space for imaging a subject or the like. In the imaging environment 138, a plurality of light sources 132, a temperature sensor 136, an entrance portion 10A of the imaging device 10, and a subject 2 are arranged. The imaging device 10 is provided on a ceiling portion 134A of the housing 134. The subject 2 is arranged on a bottom portion 134B of the housing 134. As an example, the subject 2 is a plant. In the example shown in FIG. 1, a leaf is shown as an example of a plant, but the plant may be a seed or a part other than the leaf or seed. The subject 2 may be anything other than a plant. In this embodiment, as an example, a case where the subject 2 is a plant will be described.

[0042] There are good and bad plants. The quality of a plant is evaluated, for example, using the NDVI (Natural Difference Index). A plant whose NDVI is equal to or greater than a preset threshold is determined to be good, and a plant whose NDVI is less than the preset threshold is determined to be bad. The preset threshold for determining whether a plant is good or bad can be set arbitrarily based on the plant variety, etc.

[0043] The temperature sensor 136 measures the temperature of the imaging environment 138 and outputs temperature data 74 indicative of the measured temperature. The temperature sensor 136 may be a temperature sensor 136 using various technologies such as a resistance temperature detector, a thermocouple, a radiation thermometer, an alcohol thermometer, a bimetallic thermometer, a pressure thermometer, or a thermistor thermometer.

[0044] The imaging device 10 is, for example, a multispectral camera. Here, an example is given in which the imaging device 10 is a multispectral camera, but the imaging device 10 may be a spectral camera such as a hyperspectral camera. The imaging device 10 may also be an RGB camera with a spectral filter. The imaging device 10 may also be a camera that captures images multiple times while switching between illuminations with different wavelengths. Below, an example will be described in which the imaging device 10 is a multispectral camera. The imaging device 10 is an example of a "spectral imaging device" and a "multispectral camera" according to the technology of the present disclosure.

[0045] The imaging device 10 includes an optical system 26 and an image sensor 28. The image sensor 28 is an example of an "imaging element" according to the technology of the present disclosure. The optical system 26 includes a first lens 30, a pupil division filter 16, and a second lens 32. The first lens 30, the pupil division filter 16, and the second lens 32 are arranged in this order along the optical axis OA from the subject 2 side to the image sensor 28 side.

[0046] The pupil division filter 16 has spectral filters 20λ1 to 20λ3. The spectral filters 20λ1 to 20λ3 are an example of the "multiple spectral filters" according to the technology of the present disclosure. Each of the spectral filters 20λ1 to 20λ3 is a bandpass filter that transmits light in a specific wavelength band. The spectral filters 20λ1 to 20λ3 have different wavelength bands from each other. Specifically, the spectral filter 20λ1 has a first wavelength band λ1, the spectral filter 20λ2 has a second wavelength band λ2, and the spectral filter 20λ3 has a third wavelength band λ3.

[0047] Hereinafter, when it is not necessary to distinguish between the spectral filters 20λ1 to 20λ3, each of the spectral filters 20λ1 to 20λ3 will be referred to as the "spectral filter 20." Furthermore, when it is not necessary to distinguish between the first wavelength band λ1, the second wavelength band λ2, and the third wavelength band λ3, each of the first wavelength band λ1, the second wavelength band λ2, and the third wavelength band λ3 will be referred to as the "wavelength band λ."

[0048] In the imaging device 10, as described in detail later, spectral image data 72λ1 to 72λ3 corresponding to each wavelength band λ are generated based on imaging data 70 obtained by imaging the subject 2. The spectral image data 72λ1 is the spectral image data corresponding to the first wavelength band λ1, the spectral image data 72λ2 is the spectral image data corresponding to the second wavelength band λ2, and the spectral image data 72λ3 is the spectral image data corresponding to the third wavelength band λ3. Hereinafter, when it is not necessary to distinguish between the spectral image data 72λ1 to 72λ3, each of the spectral image data 72λ1 to 72λ3 will be referred to as "spectral image data 72."

[0049] In the first embodiment, as an example, a case will be described in which three spectral image data 72 are generated based on light dispersed into three wavelength bands λ. However, three wavelength bands λ are merely an example, and two or more wavelength bands λ may be used.

[0050] The imaging device 10 has a zoom function. When the subject 2 is imaged by the imaging device 10, the angle of view of the imaging device 10 is adjusted by the zoom function. The angle of view of the imaging device 10 is set to an angle of view in which the imaging range of the imaging device 10 includes the subject 2.

[0051] The processing device 90 is communicatively connected to the imaging device 10 and the temperature sensor 136. The processing device 90 is, for example, an information processing device such as a personal computer or a server. The processing device 90 includes a display device 122 and a reception device 124. The display device 122 is, for example, a liquid crystal display or an EL display. The reception device 124 includes, for example, a keyboard, a mouse, a touch pad, and the like. As will be described in detail later, the processing device 90 performs an inspection on the subject 2 based on the information accepted by the reception device 124 and the imaging data 70 and temperature data 74 received from the imaging device 10, and displays the results of the inspection and / or images, etc. on the display device 122.

[0052] Next, the imaging device 10 according to the first embodiment will be described in detail.

[0053] 2, as an example, imaging device 10 includes lens device 12 and imaging device body 14. Lens device 12 has pupil division filter 16. As described above, imaging device 10 is a multispectral camera that captures light dispersed into multiple wavelength bands λ by pupil division filter 16 to generate and output multiple spectral image data 72λ1 to 72λ3.

[0054] As an example, as shown in FIG. 3, pupil division filter 16 has a frame 18, spectral filters 20λ1 to 20λ3, and polarizing filters 22A to 22C.

[0055] The frame 18 has openings 24A-24C. The openings 24A-24C are formed in a line around the optical axis OA. Hereinafter, when there is no need to distinguish between the openings 24A-24C, each of the openings 24A-24C will be referred to as "opening 24." The spectral filters 20λ1-20λ3 are provided in the openings 24A-24C, respectively, and are arranged in a line around the optical axis OA. As a result, the center of gravity of each of the spectral filters 20λ1-20λ3 is located at a position different from the optical axis OA.

[0056] Polarizing filters 22A-22C are provided corresponding to spectral filters 20λ1-20λ3, respectively. Specifically, polarizing filter 22A is provided in opening 24A and overlaps with spectral filter 20λ1. Polarizing filter 22B is provided in opening 24B and overlaps with spectral filter 20λ2. Polarizing filter 22C is provided in opening 24C and overlaps with spectral filter 20λ3.

[0057] Each of the polarizing filters 22A to 22C is an optical filter that transmits light that vibrates in a specific direction. The polarizing filters 22A to 22C are an example of a "first polarizer" according to the technology of the present disclosure. The polarizing filters 22A to 22C have different polarization directions. That is, the polarizing filters 22A to 22C have polarization axes with different polarization angles. Specifically, the polarizing filter 22A has a first polarization angle α1, the polarizing filter 22B has a second polarization angle α2, and the polarizing filter 22C has a third polarization angle α3. The polarization axis may be referred to as a transmission axis. As an example, the first polarization angle α1 is set to 0°, the second polarization angle α2 is set to 45°, and the third polarization angle α3 is set to 90°.

[0058] Hereinafter, when there is no need to distinguish between the polarizing filters 22A to 22C, each of the polarizing filters 22A to 22C will be referred to as "polarizing filter 22." Furthermore, when there is no need to distinguish between the first polarization angle α1, the second polarization angle α2, and the third polarization angle α3, each of the first polarization angle α1, the second polarization angle α2, and the third polarization angle α3 will be referred to as "polarization angle α."

[0059] In the example shown in Fig. 3, the number of the multiple openings 24 is three, corresponding to the number of the multiple wavelength bands λ, but the number of the multiple openings 24 may be greater than the number of the multiple wavelength bands λ (i.e., the number of the multiple spectral filters 20). Furthermore, the unused openings 24 among the multiple openings 24 may be blocked by a shielding member (not shown). Furthermore, in the example shown in Fig. 3, the multiple spectral filters 20 have different wavelength bands λ, but the multiple spectral filters 20 may include spectral filters 20 having the same wavelength band λ.

[0060] 4, the lens device 12 includes an optical system 26, and the imaging device body 14 includes an image sensor 28. The optical system 26 includes a pupil division filter 16, a first lens 30, and a second lens 32.

[0061] First lens 30 causes light reflected by subject 2 to be incident on pupil division filter 16. Second lens 32 causes light transmitted through pupil division filter 16 to form an image on imaging surface 34A of photoelectric conversion element 34 provided in image sensor 28.

[0062] The pupil division filter 16 is disposed at the pupil position of the optical system 26. The pupil position refers to the diaphragm surface that limits the brightness of the optical system 26. The pupil position here includes nearby positions, and the nearby positions refer to the range from the entrance pupil to the exit pupil. The configuration of the pupil division filter 16 is as described using Figure 3. For the sake of convenience, Figure 4 shows a state in which a plurality of spectral filters 20 and a plurality of polarizing filters 22 are linearly arranged along a direction perpendicular to the optical axis OA.

[0063] The image sensor 28 includes a photoelectric conversion element 34 and a signal processing circuit 36. The image sensor 28 is, for example, a CMOS image sensor. In the first embodiment, a CMOS image sensor is exemplified as the image sensor 28, but the technology of the present disclosure is not limited thereto, and the technology of the present disclosure is also applicable even if the image sensor 28 is another type of image sensor such as a CCD image sensor.

[0064] As an example, Fig. 4 shows a schematic configuration of the photoelectric conversion element 34. As an example, Fig. 5 specifically shows a portion of the configuration of the photoelectric conversion element 34. The photoelectric conversion element 34 has a pixel layer 38, a polarizing filter layer 40, and a spectral filter layer 42. Note that the configuration of the photoelectric conversion element 34 shown in Fig. 5 is only an example, and the technology of the present disclosure is applicable even if the photoelectric conversion element 34 does not have the spectral filter layer 42.

[0065] The pixel layer 38 has a plurality of pixels 44. The plurality of pixels 44 are arranged in a matrix and form an imaging surface 34A of the photoelectric conversion element 34. Each pixel 44 is a physical pixel having a photodiode (not shown), which photoelectrically converts received light and outputs an electrical signal according to the amount of received light.

[0066] Hereinafter, the pixels 44 provided in the photoelectric conversion element 34 will be referred to as "physical pixels 44" to distinguish them from the pixels that form the spectral image represented by the spectral image data 72. Also, the pixels that form the spectral image will be referred to as "image pixels."

[0067] The photoelectric conversion element 34 outputs the electrical signals output from the multiple physical pixels 44 to the signal processing circuit 36 ​​as imaging data 70. The signal processing circuit 36 ​​digitizes the analog imaging data 70 input from the photoelectric conversion element 34. The imaging data 70 is image data representing a captured image.

[0068] The plurality of physical pixels 44 form a plurality of pixel blocks 46. Each pixel block 46 is formed by two vertically and two horizontally, a total of four physical pixels 44. For convenience, in Fig. 4, the four physical pixels 44 forming each pixel block 46 are shown as being linearly arranged along a direction perpendicular to the optical axis OA, but the four physical pixels 44 are disposed adjacent to each other in the vertical and horizontal directions of the photoelectric conversion element 34 (see Fig. 5).

[0069] The polarizing filter layer 40 has a plurality of types of polarizers 48A to 48D. Each of the polarizers 48A to 48D is an optical filter that transmits light vibrating in a specific direction. The polarizers 48A to 48D are an example of a "second polarizer" according to the technology of the present disclosure. The polarizers 48A to 48D have different polarization directions. That is, the polarizers 48A to 48D have polarization axes with different polarization angles. Specifically, the polarizer 48A has a first polarization angle β1, the polarizer 48B has a second polarization angle β2, the polarizer 48C has a third polarization angle β3, and the polarizer 48D has a fourth polarization angle β4. As an example, the first polarization angle β1 is set to 0°, the second polarization angle β2 is set to 45°, the third polarization angle β3 is set to 90°, and the fourth polarization angle β4 is set to 135°.

[0070] Hereinafter, when it is not necessary to distinguish between the polarizers 48A to 48D, each of the polarizers 48A to 48D will be referred to as "polarizer 48." Furthermore, when it is not necessary to distinguish between the first polarization angle β1, the second polarization angle β2, the third polarization angle β3, and the fourth polarization angle β4, each of the first polarization angle β1, the second polarization angle β2, the third polarization angle β3, and the fourth polarization angle β4 will be referred to as "polarization angle β."

[0071] The spectral filter layer 42 has a B filter 50A, a G filter 50B, and an R filter 50C. The B filter 50A is a blue-range filter that transmits the most light in the blue wavelength band among the light in the multiple wavelength bands. The G filter 50B is a green-range filter that transmits the most light in the green wavelength band among the light in the multiple wavelength bands. The R filter 50C is a red-range filter that transmits the most light in the red wavelength band among the light in the multiple wavelength bands. The B filter 50A, the G filter 50B, and the R filter 50C are assigned to each pixel block 46. Note that the spectral filter layer 42 generally has an IR cut filter (not shown), but when the imaging device 10 captures near-infrared light, it is preferable to remove the IR cut filter.

[0072] In Fig. 4, for convenience, the B filter 50A, the G filter 50B, and the R filter 50C are shown linearly arranged along a direction perpendicular to the optical axis OA, but as an example, as shown in Fig. 5, the B filter 50A, the G filter 50B, and the R filter 50C are arranged in a matrix in a predefined pattern arrangement. In the example shown in Fig. 5, the B filter 50A, the G filter 50B, and the R filter 50C are arranged in a matrix in a Bayer array, which is an example of a predefined pattern arrangement. Note that the predefined pattern arrangement may be an RGB stripe array, an R / G checkerboard array, an X-Trans (registered trademark) array, a honeycomb array, or the like, other than the Bayer array.

[0073] Hereinafter, when there is no need to distinguish between B filter 50A, G filter 50B, and R filter 50C, they will each be referred to as "filter 50."

[0074] 4, the imaging device body 14 includes, in addition to the image sensor 28, a control driver 52, an input / output I / F 54, a computer 56, and a communication device 58. The signal processing circuit 36, the control driver 52, the computer 56, and the communication device 58 are connected to the input / output I / F 54.

[0075] The computer 56 has a processor 60, a storage 62, and a RAM 64. The processor 60 controls the entire imaging device 10. The processor 60 is, for example, a processing unit including a CPU and a GPU, and the GPU operates under the control of the CPU and is responsible for executing image-related processing. Here, a processing unit including a CPU and a GPU is given as an example of the processor 60, but this is merely one example, and the processor 60 may be one or more CPUs that integrate a GPU function, or one or more CPUs that do not integrate a GPU function.

[0076] The processor 60, the storage 62, and the RAM 64 are connected via a bus 66, which is connected to the input / output I / F 54. The storage 62 is a non-transitory storage medium, and stores various parameters and various programs. For example, the storage 62 is a flash memory (for example, an EEPROM). However, this is merely an example, and an HDD or the like may be used as the storage 62 together with the flash memory. The RAM 64 temporarily stores various information and is used as a work memory. For example, the RAM 64 may be a DRAM and / or an SRAM.

[0077] The processor 60 reads out a necessary program from the storage 62, and executes the read program on the RAM 64. The processor 60 controls the control driver 52 and the signal processing circuit 36 ​​in accordance with the program executed on the RAM 64. The control driver 52 controls the photoelectric conversion element 34 under the control of the processor 60.

[0078] The communication device 58 is connected to the processor 60 via the input / output I / F 54 and the bus 66. The communication device 58 is also connected to the processing device 90 so as to be able to communicate with it via a wired or wireless connection. The communication device 58 is responsible for sending and receiving information to and from the processing device 90. For example, the communication device 58 transmits data to the processing device 90 in response to a request from the processor 60. The communication device 58 also receives data transmitted from the processing device 90, and outputs the received data to the processor 60 via the bus 66.

[0079] 6 as an example, a spectral image generation program 80 is stored in the storage 62. The processor 60 reads out the spectral image generation program 80 from the storage 62, and executes the read spectral image generation program 80 on the RAM 64. The processor 60 executes a spectral image generation process for generating a plurality of spectral image data 72 in accordance with the spectral image generation program 80 executed on the RAM 64. The spectral image generation process is realized by the processor 60 operating as an output value acquisition unit 82 and an interference removal processing unit 84 in accordance with the spectral image generation program 80.

[0080] 7, when imaging data 70 output from the image sensor 28 is input to the processor 60, the output value acquisition unit 82 acquires an output value Y of each physical pixel 44 based on the imaging data 70. The output value Y of each physical pixel 44 corresponds to the luminance value of each image pixel included in the imaging data 70.

[0081] Here, the output value Y of each physical pixel 44 is a value including interference (i.e., crosstalk). That is, since light of each wavelength band λ of the first wavelength band λ1, the second wavelength band λ2, and the third wavelength band λ3 is incident on each physical pixel 44, the output value Y is a mixed value of a value corresponding to the amount of light of the first wavelength band λ1, a value corresponding to the amount of light of the second wavelength band λ2, and a value corresponding to the amount of light of the third wavelength band λ3.

[0082] In order to obtain the spectral image data 72, the processor 60 needs to perform a process of separating and extracting values ​​corresponding to each wavelength band λ from the output value Y for each physical pixel 44, that is, an interference removal process that removes interference, on the output value Y. Therefore, in the first embodiment, in order to obtain the spectral image data 72, the interference removal processing unit 84 performs the interference removal process on the output value Y of each physical pixel 44 acquired by the output value acquisition unit 82.

[0083] Here, the interference removal process will be described. The output value Y of each physical pixel 44 includes the luminance values ​​for red, green, and blue at each polarization angle β as components of the output value Y. The output value Y of each physical pixel 44 is expressed by equation (1).

number

[0084] However, Y β1_R is the luminance value of the red component of the output value Y whose polarization angle is the first polarization angle β1, and Y β2_R is the luminance value of the red component of the output value Y whose polarization angle is the second polarization angle β2, β3_R is the luminance value of the red component of the output value Y whose polarization angle is the third polarization angle β3, β4_R is the luminance value of the red component of the output value Y whose polarization angle is the fourth polarization angle β4.

[0085] Also, Y β1_G is the luminance value of the green component of the output value Y whose polarization angle is the first polarization angle β1, β2_Gis the luminance value of the green component of the output value Y whose polarization angle is the second polarization angle β2, β3_G is the luminance value of the green component of the output value Y whose polarization angle is the third polarization angle β3, β4_G is the luminance value of the green component of the output value Y whose polarization angle is the fourth polarization angle β4.

[0086] Also, Y β1_B is the luminance value of the blue component of the output value Y whose polarization angle is the first polarization angle β1, β2_B is the luminance value of the blue component of the output value Y whose polarization angle is the second polarization angle β2, β3_B is the luminance value of the blue component of the output value Y whose polarization angle is the third polarization angle β3, β4_B is the luminance value of the blue component of the output value Y whose polarization angle is the fourth polarization angle β4.

[0087] The pixel value X of each image pixel forming the spectral image data 72 is a luminance value X of polarized light in a first wavelength band λ1 having a first polarization angle α1 (hereinafter referred to as “first wavelength band polarized light”). λ1 and the luminance value X of polarized light in a second wavelength band λ2 having a second polarization angle α2 (hereinafter referred to as “second wavelength band polarized light”) λ2 and the luminance value X of polarized light in a third wavelength band λ3 having a third polarization angle α3 (hereinafter referred to as “third wavelength band polarized light”) λ3 and as components of the pixel value X. The pixel value X of each image pixel is expressed by equation (2).

number

[0088] The output value Y of each physical pixel 44 is expressed by equation (3).

number

[0089] In formula (3), A is an interference matrix. The interference matrix A (not shown) is a matrix that indicates the characteristics of interference. The interference matrix A is determined in advance based on a number of known values, such as the spectrum of the incident light, the spectral transmittance of the first lens 30, the spectral transmittance of the second lens 32, the spectral transmittance of the multiple spectral filters 20, the polarization angles of the multiple polarizing filters 22, the polarization angles of the multiple polarizers 48, and the spectral sensitivity of the image sensor 28.

[0090] The interference cancellation matrix A is the generalized inverse matrix of the interference matrix A. + In this case, the pixel value X of each image pixel is expressed by equation (4).

number

[0091] Interference cancellation matrix A + Similarly to the interference matrix A, the interference cancellation matrix A is a matrix defined based on the spectrum of the incident light, the spectral transmittance of the first lens 30, the spectral transmittance of the second lens 32, the spectral transmittance of the multiple spectral filters 20, the spectral sensitivity of the image sensor 28, and the like. + is stored in advance in the storage 62.

[0092] The interference cancellation processor 84 uses the interference cancellation matrix A stored in the storage 62. + and the output value Y of each physical pixel 44 acquired by the output value acquisition unit 82 are acquired, and the interference cancellation matrix A + and the output value Y of each physical pixel 44, the pixel value X of each image pixel is output according to equation (4).

[0093] Here, as described above, the pixel value X of each image pixel is the luminance value X of the first wavelength band polarized light. λ1 and the intensity value X of the second wavelength band polarized light. λ2 and the intensity value X of the third wavelength band polarized light. λ3 and are included as components of pixel value X.

[0094] The spectral image data 72λ1 of the imaging data 70 is a luminance value X λ1(i.e., the brightness values ​​X λ1 The spectral image data 72λ2 in the imaging data 70 is an image based on the luminance value X λ2 (i.e., the brightness values ​​X λ2 The spectral image data 72λ3 in the imaging data 70 is an image based on the luminance value X λ3 (i.e., the brightness value X λ3 (Image based on .)

[0095] In this way, the interference removal processing is performed by the interference removal processing unit 84, and the imaging data 70 is converted into the luminance value X λ1 and the intensity value X of the second wavelength band polarized light corresponding to the spectral image data 72λ1. λ2 and the intensity value X of the third wavelength band polarized light corresponding to the spectral image data 72λ2. λ3 That is, the imaging data 70 is separated into spectral image data 72 for each wavelength band λ of the plurality of spectral filters 20.

[0096] Next, the processing device 90 according to this embodiment will be described in detail.

[0097] The processing device 90 has a function of obtaining the NDVI of the subject 2 based on the imaging data 70 obtained by the imaging device 10, which is a multispectral camera. In order to obtain the NDVI of the subject 2, it is necessary to obtain the spectral reflectance of the subject 2. One example of a method for obtaining the spectral reflectance of the subject 2 is shown in FIG. 8. That is, as shown in FIG. 8(A), first, the calibration reference chart 1, whose spectral reflectance is known, is imaged by the imaging device 10, and the intensity of the incident light output from the light source is estimated from the value of the imaging data 70 obtained by imaging. For example, if the spectral reflectance of the calibration reference chart 1 is 0.8 and the value of the imaging data 70 is 64, the intensity of the incident light can be estimated to be 80.

[0098] 8(B), subject 2, whose spectral reflectance is unknown, is imaged by imaging device 10 under the same incident light intensity conditions as when calibration reference chart 1 is imaged, and the spectral reflectance of subject 2 is estimated based on the value of imaging data 70 obtained by imaging and the incident light intensity estimated using calibration reference chart 1. For example, if the incident light intensity estimated using calibration reference chart 1 is 80 and the value of imaging data 70 is 40, the spectral reflectance of subject 2 can be estimated to be 0.5.

[0099] However, even if the calibration reference chart 1 and the subject 2 are imaged in the same imaging environment 138 to set the same condition for the intensity of incident light, the temperature of the imaging environment 138 may differ between the timing when the calibration reference chart 1 is imaged and the timing when the subject 2 is imaged. Generally, the sensitivity of the image sensor 28 varies depending on the temperature. Therefore, if the temperature of the imaging environment 138 differs between the timing when the calibration reference chart 1 is imaged and the timing when the subject 2 is imaged, there is a risk that the spectral reflectance of the subject 2 will be estimated incorrectly, making it impossible to obtain a correct NDVI.

[0100] 8, if the sensitivity of the image sensor 28 changes by 1.2 times due to a difference in temperature of the imaging environment 138 between the timing when the calibration reference chart 1 is imaged and the timing when the subject 2 is imaged, the value of the imaging data 70 will be 48. In this case, although the spectral reflectance of the subject 2 is actually 0.5, the spectral reflectance of the subject 2 will be erroneously estimated to be 0.6.

[0101] Therefore, in this embodiment, even if the temperature of the imaging environment 138 differs between the timing when the calibration reference chart 1 is imaged and the timing when the subject 2 is imaged, it is possible to correctly estimate the spectral reflectance of the subject 2, thereby making it possible to obtain a correct NDVI. A specific description will be given below.

[0102] As an example, as shown in Fig. 9, the processing device 90 includes a computer 92. The computer 92 includes a processor 94, a storage 96, and a RAM 98. The processor 94, the storage 96, and the RAM 98 are realized by hardware similar to the above-mentioned processor 60, the storage 62, and the RAM 64 (see Fig. 4). The processor 94 is an example of a "processor" according to the technology of the present disclosure.

[0103] The storage 96 stores an inspection program 100. The inspection program 100 is an example of a "program" according to the technology of the present disclosure. The processor 94 reads out the inspection program 100 from the storage 96 and executes the read inspection program 100 on the RAM 98. The processor 94 executes an inspection process for evaluating the subject 2 based on NDVI according to the inspection program 100 executed on the RAM 98. The inspection process is realized by the processor 94 operating as a first data acquisition unit 102, a second data acquisition unit 104, a wavelength band selection unit 106, a region division unit 108, a temperature correction unit 110, an image data generation unit 112, and a display control unit 114 according to the inspection program 100.

[0104] As an example, as shown in FIG. 10, first, the calibration reference chart 1 is imaged. The calibration reference chart 1 is an example of a "calibration member" according to the technology of the present disclosure. The calibration reference chart 1 is placed on the bottom 134B of the housing 134. The calibration reference chart 1 is irradiated with light by a plurality of light sources 132. The imaging device 10 images the calibration reference chart 1 in a state in which the calibration reference chart 1 is irradiated with light by the plurality of light sources 132. Then, the imaging device 10 transmits imaging data 70 obtained by imaging the calibration reference chart 1 to the processing device 90. Hereinafter, the imaging data 70 obtained by imaging the calibration reference chart 1 is referred to as first imaging data 70A. The first imaging data 70A is an example of "first imaging data" according to the technology of the present disclosure. The first imaging data 70A is image data including a plurality of spectral image data 72. Hereinafter, the spectral image data 72 included in the first imaging data 70A is referred to as first spectral image data 72A. The first spectral image data 72A is an example of the "first spectral data" according to the technique of the present disclosure.

[0105] As described above, the first spectral image data 72A included in the first imaging data 70A is data derived for each wavelength band λ by performing interference removal processing based on each piece of information such as the spectrum of incident light, the spectral transmittance of the first lens 30, the spectral transmittance of the second lens 32, the spectral transmittance of the multiple spectral filters 20, the polarization angles of the multiple polarizing filters 22, the polarization angles of the multiple polarizers 48, and the spectral sensitivity of the image sensor 28. The spectral transmittance of the multiple spectral filters 20 is an example of “spectral information related to the multiple spectral filters” according to the technology of the present disclosure, the polarization angles of the multiple polarizing filters 22 is an example of “first polarization information related to the multiple first polarizers” according to the technology of the present disclosure, and the polarization angles of the multiple polarizers 48 is an example of “second polarization information related to the multiple second polarizers” according to the technology of the present disclosure.

[0106] The temperature sensor 136 measures the temperature of the imaging environment 138 when the calibration reference chart 1 is imaged, and outputs temperature data 74 indicating the measured temperature to the processing device. The imaging environment 138 is an example of an "imaging environment" according to the technology of the present disclosure. Hereinafter, the temperature data 74 obtained when the calibration reference chart 1 is imaged is referred to as first temperature data 74A, and the temperature indicated by the first temperature data 74A is referred to as first temperature t1. The first temperature t1 is an example of a "first temperature" according to the technology of the present disclosure.

[0107] The first data acquisition unit 102 acquires the first imaging data 70A and the first temperature data 74A received by the processing device 90. The first imaging data 70A and the first temperature data 74A are stored in the RAM 98. The first imaging data 70A and the first temperature data 74A may be stored in the storage 96. By acquiring the first temperature data 74A, a first temperature t1 is acquired.

[0108] As an example, as shown in FIG. 11, the subject 2 is then imaged. The subject 2 is irradiated with light by a plurality of light sources 132. The imaging device 10 images the subject 2 in a state in which the subject 2 is irradiated with light by the plurality of light sources 132. Then, the imaging device 10 transmits imaging data 70 obtained by imaging the subject 2 to the processing device 90. Hereinafter, the imaging data 70 obtained by imaging the subject 2 is referred to as second imaging data 70B. The second imaging data 70B is an example of "second imaging data" according to the technology of the present disclosure. The second imaging data 70B is image data including a plurality of spectral image data 72. Hereinafter, the spectral image data 72 included in the second imaging data 70B is referred to as second spectral image data 72B. The second spectral image data 72B is an example of "second spectral data" according to the technology of the present disclosure.

[0109] As described above, the spectral image data 72 included in the second imaging data 70B is data derived for each wavelength band λ by performing interference removal processing based on each piece of information such as the spectrum of the incident light, the spectral transmittance of the first lens 30, the spectral transmittance of the second lens 32, the spectral transmittance of the multiple spectral filters 20, the polarization angles of the multiple polarizing filters 22, the polarization angles of the multiple polarizers 48, and the spectral sensitivity of the image sensor 28.

[0110] The temperature sensor 136 measures the temperature of the imaging environment 138 when imaging the subject 2, and outputs temperature data 74 indicating the measured temperature to the processing device. Hereinafter, the temperature data 74 obtained when imaging the subject 2 is referred to as second temperature data 74B, and the temperature indicated by the second temperature data 74B is referred to as second temperature t2. The second temperature t2 is an example of the "second temperature" according to the technology of the present disclosure.

[0111] The second data acquisition unit 104 acquires the second imaging data 70B and the second temperature data 74B received by the processing device 90. The second imaging data 70B and the second temperature data 74B are stored in the RAM 98. The second imaging data 70B and the second temperature data 74B may be stored in the storage 96. By acquiring the second temperature data 74B, the second temperature t2 is acquired.

[0112] 12, the wavelength band selection unit 106 selects two spectral image data 72 from among the multiple spectral image data 72 included in the first imaging data 70A. In addition, the wavelength band selection unit 106 selects two spectral image data 72 from among the multiple spectral image data 72 included in the second imaging data 70B, which have the same wavelength bands as the two spectral image data 72 selected from the first imaging data 70A. In the example shown in FIG. 12, spectral image data 72λ1 corresponding to light in the first wavelength band λ1 and spectral image data 72λ2 corresponding to light in the second wavelength band λ2 are selected as the two spectral image data 72.

[0113] The two wavelength bands λ may be selected, for example, based on a wavelength band selection instruction from a user or the like received by the reception device 124 (see FIG. 1), or may be selected based on a result of image analysis processing performed on a plurality of spectral image data 72. In the example shown in FIG. 12, two wavelength bands λ are selected from the three wavelength bands λ corresponding to the three spectral filters 20 included in the pupil division filter 16, but when the pupil division filter 16 has four or more spectral filters 20, two wavelength bands λ may be selected from the four or more wavelength bands λ.

[0114] Furthermore, the two wavelength bands λ may be selected from, for example, two or more types of predefined combinations, or may be selected based on the attributes of the subject 2 (for example, the type of the subject 2, etc.). The two wavelength bands λ may be selected based on the imaging conditions of the subject 2 (for example, the illumination conditions by the multiple light sources 132). The illumination conditions may include the wavelength band λ of the light irradiated by the multiple light sources 132 and / or the light output.

[0115] For example, when the subject 2 is a plant, two wavelength bands λ may be selected from a first combination which is a combination of a near-infrared wavelength band having a central wavelength of 850 nm and a red wavelength band having a central wavelength of 650 nm, and a second combination which is a combination of a red wavelength band having a central wavelength of 717 nm and a green wavelength band having a central wavelength of 560 nm.

[0116] As an example, as shown in FIG. 13, the region dividing unit 108 divides the imaging surface 34A of the image sensor 28 into a plurality of divided regions 140 for each wavelength band λ. The plurality of divided regions 140 are an example of the "imaging region" according to the technology of the present disclosure. As an example, the plurality of divided regions 140 are divided in the vertical and horizontal directions of the imaging surface 34A. FIG. 13 shows a plurality of divided regions 140 divided in correspondence with a first wavelength band λ1 and a plurality of divided regions 140 divided in correspondence with a second wavelength band λ2. Here, the plurality of divided regions 140 are divided in different manners according to the wavelength band λ. A specific description will be given below.

[0117] In the imaging device 10, the center of gravity of each spectral filter 20 provided in the pupil division filter 16 is set to a position different from the optical axis OA (i.e., a position shifted from the optical axis OA) (see FIG. 3). Therefore, in response to the shift in the center of gravity of each spectral filter 20, a peak position P at which the intensity of light in the wavelength band λ reaches a peak on the imaging plane 34A also shifts from the center of the imaging plane 34A. For example, in the case of the first wavelength band λ1, the peak position P is located on the upper left side of the imaging plane 34A, and in the case of the second wavelength band λ2, the peak position P is located on the lower right side of the imaging plane 34A.

[0118] Therefore, the region dividing unit 108 divides the imaging surface 34A into a plurality of divided regions 140 based on a peak position P where the intensity of the light in the wavelength band λ reaches a peak on the imaging surface 34A for each wavelength band λ. Specifically, the region dividing unit 108 divides the imaging surface 34A for each wavelength band λ into a first region 142 indicated by diagonal hatching and a second region 144 indicated by dotted hatching. The first region 142 is a region that includes the peak position P, and the second region 144 is a region that is farther from the peak position P than the first region 142.

[0119] In the example shown in FIG. 13, for the first wavelength band λ1, when the imaging surface 34A is divided vertically and horizontally into nine equal parts, the first region 142 includes the central region, the left central region, the upper central region, and the upper left central region, and the second region 144 includes the right central region, the upper right central region, the lower right central region, the lower central region, and the lower left central region.

[0120] On the other hand, for the second wavelength band λ2, the first region 142 includes a central region, a central right region, a central lower region, and a central right lower region, and the second region 144 includes a central left region, a central lower left region, a central upper left region, a central upper region, and a central upper right region.

[0121] Then, the region dividing unit 108 sets the number of divisions of the divided regions 140 in the first region 142 to be smaller than the number of divisions of the divided regions 140 in the second region 144. In other words, the region dividing unit 108 sets the area of ​​the divided regions 140 in the first region 142 to be larger than the area of ​​the divided regions 140 in the second region 144.

[0122] 14, the divided regions 140 may be divided at equal intervals in the vertical and horizontal directions of the imaging surface 34A. The divided regions 140 may be divided only in the vertical direction of the imaging surface 34A, or may be divided only in the horizontal direction of the angle of view of the imaging device 10.

[0123] Returning to FIG. 13, the storage 96 stores a correction value group. The correction value group includes a plurality of correction values. Each correction value is a correction value that defines the relationship between the temperature and the value of the imaging data 70 for each wavelength band λ. FIG. 15 shows a correction value group when the imaging surface 34A is divided into 9 equal parts in the vertical and horizontal directions. The correction value group shown in FIG. 15(A) is a correction value group for the first wavelength band λ1, and the correction value group shown in FIG. 15(B) is a correction value group for the second wavelength band λ2. As shown in FIG. 15, the correction value is set according to the temperature. In addition, the correction value is set according to the divided region 140. The correction value group may be stored in the storage 96 in a table format that defines the relationship between the correction value, the temperature, and the divided region 140. The storage is an example of a "memory" according to the technology of the present disclosure.

[0124] FIG. 16 shows an example of a method for acquiring a group of correction values. When acquiring a group of correction values, a correction reference chart 3 is used. The correction reference chart 3 is an example of a "correction member" according to the technology of the present disclosure. The spectral reflectance of the correction reference chart 3 may be the same as or different from the spectral reflectance of the calibration reference chart 1. For example, a correction reference chart 3 having a spectral reflectance of 1.0 is used. The spectral reflectance of the calibration reference chart 1 is an example of a "second reflectance" according to the technology of the present disclosure, and the spectral reflectance of the correction reference chart 3 is a "third reflectance" according to the technology of the present disclosure.

[0125] Furthermore, the imaging conditions (that is, the conditions of the intensity of the incident light) when obtaining the correction value group may be the same as the imaging conditions when imaging the calibration reference chart 1, or may be different.

[0126] Then, the correction reference chart 3 is imaged by the imaging device 10 under conditions in which the temperature of the imaging environment 138 is varied for each wavelength band λ, and the values ​​of the imaging data 70 obtained by imaging are generated as correction values ​​in association with the temperatures indicated by the temperature data 74, and the generated correction values ​​are stored in the storage 96. The values ​​of the imaging data 70 are specifically values ​​of the spectral image data 72 corresponding to each wavelength band (i.e., the output values ​​of each wavelength band). The values ​​of the spectral image data 72 may be representative values ​​(i.e., the average value, the median value, the maximum value, or the minimum value) of the output values ​​of the physical pixels included in each divided region 140. In this way, a correction value group including a plurality of correction values ​​indicating the values ​​of the imaging data 70 corresponding to each temperature is obtained for each wavelength band λ.

[0127] Returning to FIG. 13, the temperature correction unit 110 performs a temperature correction process on the second imaging data 70B obtained by imaging the subject 2, based on the first imaging data 70A obtained by imaging the calibration reference chart 1 and the correction value for the imaging data 70 of the imaging device 10. The temperature correction unit 110 performs the correction process for each divided region 140. The temperature correction unit 110 also performs the correction process for each wavelength band λ. That is, the temperature correction unit 110 performs the correction process on the second spectral image data 72B included in the second imaging data 70B, based on the first spectral image data 72A included in the first imaging data 70A and the correction value, for each wavelength band λ and each divided region 140. The correction process is specifically a process based on the difference between the first temperature t1 indicated by the first temperature data 74A obtained when imaging the calibration reference chart 1 and the second temperature t2 indicated by the second temperature data 74B obtained when imaging the subject 2. The correction value is selected from a group of correction values ​​according to the first temperature t1 and the second temperature t2.

[0128] Specifically, in the correction process, first, a first correction value corresponding to the first temperature t1 and the divided region 140 is obtained from the correction value group, and a second correction value corresponding to the second temperature t2 and the divided region 140 is obtained from the correction value group. Then, a second reflectance, which is the spectral reflectance of the subject 2, is derived based on the first reflectance, which is the known spectral reflectance of the calibration reference chart 1, the value of the first spectral image data 72A, the value of the second spectral image data 72B, the first correction value, and the second correction value. The second reflectance is derived for each physical pixel. The calculation formula for deriving the second reflectance is as shown in Equation (5).

number

[0129] 8, if the sensitivity of the image sensor 28 changes by 1.2 times due to a difference in temperature of the imaging environment 138 between the timing when the calibration reference chart 1 is imaged and the timing when the subject 2 is imaged, the value of the imaging data 70 becomes 48 as described above. In this case, the spectral reflectance of the subject 2 is erroneously estimated as 0.6, even though the spectral reflectance of the subject 2 is actually 0.5. However, if a first correction value corresponding to the first temperature t1 and the divided region 140 is obtained from the correction value group, a second correction value corresponding to the second temperature t2 and the divided region 140 is obtained from the correction value group, and a second reflectance r2, which is the spectral reflectance of the subject 2, is calculated by equation (5), the spectral reflectance of the subject 2 can be correctly estimated as 0.5.

[0130] As an example, as shown in FIG. 17, the image data generating unit 112 acquires the second imaging data 70B (i.e., the second spectral image data 72B for each wavelength band λ) corrected by the temperature correcting unit 110, and generates image data based on the acquired second imaging data 70B. The image data is, for example, image data showing an index related to the subject 2. Specifically, the image data generating unit 112 derives the contrast C of the luminance of the light of the two wavelength bands λ for each image pixel included in the image shown by the second imaging data 70B by using the formula (6), and generates a contrast map that is an image showing the contrast C of each image pixel by a heat map. Here, L1 indicates the luminance of the light of one of the two wavelength bands λ in each image pixel, and L2 indicates the luminance of the light of the other of the two wavelength bands λ in each image pixel.

number

[0131] Note that equation (6) is the Michelson contrast equation and is derived from equation (7).

number

[0132] Furthermore, when the subject 2 is a plant and the near-infrared wavelength band and the red wavelength band are selected as the two wavelength bands λ, the vegetation normalized index NDVI may be derived by equation (8) as the contrast C described above. Then, an NDVI map, which is an image showing the vegetation normalized index NDVI of each image pixel, may be generated as the contrast map. Here, NIR indicates the luminance of light in the near-infrared wavelength band at each image pixel, and Red indicates the luminance of light in the red wavelength band at each image pixel.

number

[0133] Further, although an example of deriving contrast C is given here, an index related to determining whether subject 2 is good or bad may be derived using an equation other than equation (6). The index may also be an index related to a determination other than whether subject 2 is good or bad. The index may also be an index based on the brightness of light in two wavelength bands λ other than contrast C. The contrast map may also be represented by an isopleth map or the like other than a heat map. An example of the index may be the "brightness of light in wavelength band λ" and / or the "sum of the brightness of light in multiple wavelength bands λ".

[0134] The contrast C corresponding to each image pixel included in the contrast map is an example of an "index" and "contrast" according to the technology of the present disclosure. The vegetation normalized index NDVI corresponding to each image pixel included in the NDVI map is an example of a "normalized difference vegetation index" according to the technology of the present disclosure.

[0135] The display control unit 114 acquires the image data 150 generated by the image data generation unit 112. Then, the display control unit 114 performs control to display the image data 150 on the display device 122. As a result, the contrast map indicated by the image data 150 is displayed on the display device 122.

[0136] In this way, the processing device 90 can display a contrast map on the display device 122. This allows the quality of the subject 2 to be determined by visual observation based on the contrast map.

[0137] Next, a description will be given of the operation of the inspection system 130 according to this embodiment. First, a description will be given of the spectral image generation process executed by the imaging device 10. Fig. 18 shows an example of the flow of the spectral image generation process.

[0138] 18, first, in step ST10, the output value acquisition unit 82 acquires the output value Y of each physical pixel 44 based on the imaging data 70 output from the image sensor 28 (see FIG. 7). After the process of step ST10 is executed, the spectral image generation process proceeds to step ST12.

[0139] In step ST12, the interference cancellation processor 84 uses the interference cancellation matrix A + and the output value Y of each physical pixel 44 obtained in step ST10 are obtained, and the interference cancellation matrix A + and the output value Y of each physical pixel 44, the pixel value X of each image pixel is output (see FIG. 7). By performing the interference removal process in step ST12, the imaging data 70 is converted into the luminance value X λ1 and the intensity value X of the second wavelength band polarized light corresponding to the spectral image data 72λ1. λ2 and the intensity value X of the third wavelength band polarized light. λ3 After the process of step ST12 is executed, the spectral image generation process ends.

[0140] Next, a description will be given of the inspection process executed by the processing device 90. Fig. 19 shows an example of the flow of the inspection process.

[0141] 19, first, in step ST20, the first data acquisition unit 102 acquires first imaging data 70A obtained by imaging the calibration reference chart 1 and first temperature data 74A indicating the temperature of the imaging environment 138 when the calibration reference chart 1 is imaged (see FIG. 10). After the process of step ST20 is executed, the inspection process proceeds to step ST22.

[0142] In step ST22, the second data acquisition unit 104 acquires second imaging data 70B obtained by imaging the subject 2 and second temperature data 74B indicating the temperature of the imaging environment 138 when imaging the subject 2 (see FIG. 11). After the process of step ST22 is executed, the inspection process proceeds to step ST24.

[0143] In step ST24, the wavelength band selection unit 106 selects two spectral image data 72 from the plurality of spectral image data 72 included in the first imaging data 70A acquired in step ST20. In addition, the wavelength band selection unit 106 selects two spectral image data 72 having the same wavelength band as the two spectral image data 72 selected from the first imaging data 70A from the plurality of spectral image data 72 included in the second imaging data 70B acquired in step ST22 (see FIG. 12). After the process of step ST24 is executed, the inspection process proceeds to step ST26.

[0144] In step ST26, the region dividing section 108 divides the imaging surface 34A of the image sensor 28 into a plurality of divided regions 140 for each wavelength band λ (see FIG. 13). After the process of step ST26 is executed, the inspection process proceeds to step ST28.

[0145] In step ST28, the temperature correction unit 110 acquires, for each wavelength band λ and each divided region 140, a correction value corresponding to the first temperature t1 indicated by the first temperature data 74A acquired in step ST20 from the correction value group (see FIG. 13). In addition, the temperature correction unit 110 acquires, for each wavelength band λ and each divided region 140, a correction value corresponding to the second temperature t2 indicated by the second temperature data 74B acquired in step ST22 from the correction value group (see FIG. 13). Then, the temperature correction unit 110 performs a correction process on the second spectral image data 72B included in the second imaging data 70B based on the first spectral image data 72A included in the first imaging data 70A and the correction value for each wavelength band λ (see FIG. 13). Specifically, as shown in the above formula (5), the temperature correction unit 110 derives the second reflectance r2, which is the spectral reflectance of the subject 2, for each physical pixel based on the first reflectance r1, which is the known spectral reflectance of the calibration reference chart 1, the value w of the first spectral image data 72A, the value u of the second spectral image data 72B, the first correction value c1, which is the correction value corresponding to the first temperature, and the second correction value c2, which is the correction value corresponding to the second temperature. After the process of step ST28 is executed, the inspection process proceeds to step ST30.

[0146] In step ST30, the image data generating unit 112 generates image data expressing the contrast C of the light luminance of the two wavelength bands λ by a heat map based on the second imaging data 70B (i.e., the second spectral image data 72B for each wavelength band λ) corrected in step ST28 (see FIG. 17). After the process of step ST30 is performed, the inspection process proceeds to step ST32.

[0147] In step ST32, the display control unit 114 controls the display device 122 to display the image data generated in step ST30 (see FIG. 17). As a result, an image (i.e., a contrast map) showing the contrast C of the light luminance of the two wavelength bands λ by a heat map is displayed on the display device 122. After the processing of step ST32 is executed, the inspection processing ends. Note that the data processing method executed by the above-mentioned inspection processing is an example of a "data processing method" according to the technology of the present disclosure.

[0148] Next, the effects of this embodiment will be described.

[0149] As described above, in this embodiment, the first imaging data 70A is obtained by imaging the calibration reference chart 1 with the imaging device 10 in the imaging environment 138 where the temperature is the first temperature t1, and the second imaging data 70B is obtained by imaging the subject 2 with the imaging device 10 in the imaging environment 138 where the temperature is the second temperature t2. Then, based on the first imaging data 70A and a correction value for the imaging data 70 of the imaging device 10, a correction process for temperature is performed on the second imaging data 70B. Therefore, even if the first temperature t1 when imaging the calibration reference chart 1 is different from the second temperature t2 when imaging the subject 2, the spectral reflectance of the subject 2 can be estimated with high accuracy based on the second imaging data 70B. That is, the estimation accuracy can be improved compared to the case where the correction process is not performed.

[0150] Furthermore, in this embodiment, the correction process is performed according to a plurality of divided regions 140 of the image sensor 28. That is, as an example, the same correction value is used for the same divided region 140. Therefore, for example, even for the same divided region 140, the load on the CPU can be reduced compared to the case where the correction process is performed using different correction values.

[0151] Furthermore, the imaging device 10 includes a pupil division filter 16 having a plurality of spectral filters 20 with different wavelength bands λ, and the plurality of divided regions 140 are divided based on a peak position P where the intensity of light in the wavelength band λ reaches a peak on the imaging surface 34A of the image sensor 28. Here, on the imaging surface 34A, the position dependency of the intensity change increases the farther away from the peak position P. Therefore, for example, compared to a case where the plurality of divided regions 140 are divided regardless of the peak position P, the plurality of divided regions 140 can be divided in a manner that takes into account the position dependency of the intensity change.

[0152] Moreover, the division manner of the multiple divided regions 140 differs depending on the wavelength band λ. Here, on the imaging surface 34A, the peak position P differs depending on the wavelength band λ. Therefore, for example, compared to a case where the multiple divided regions 140 are divided regardless of the wavelength band, the multiple divided regions 140 can be divided in a manner that takes into account the peak position P according to the wavelength band λ.

[0153] Moreover, the imaging surface 34A includes a first region 142 including the peak position P and a second region 144 farther from the peak position P than the first region 142, and the area of ​​the divided region 140 in the first region 142 is larger than that of the second region 144. Here, as described above, in the imaging surface 34A, the position dependency of the intensity change increases with distance from the peak position P. Therefore, by making the area of ​​the divided region 140 in the first region 142 larger than that of the second region 144, the area of ​​each divided region 140 sharing the correction value in the first region 142 becomes larger than that of the second region 144, and therefore, for example, the load on the CPU can be reduced compared to the case where the area of ​​the divided region 140 is the same in the first region 142 and the second region 144. Furthermore, by making the area of ​​the divided regions 140 in the second region 144 smaller than that in the first region 142, the area of ​​each divided region 140 that shares a correction value in the second region 144 is smaller than that in the first region 142. Therefore, correction processing can be performed on each divided region 140 with greater precision than when, for example, the areas of the divided regions 140 in the first region 142 and the second region 144 are the same.

[0154] In addition, in this embodiment, first spectral image data 72A derived for each wavelength band λ by interference removal processing from the first imaging data 70A is obtained, and second spectral image data 72B derived for each wavelength band λ by interference removal processing from the second imaging data 70B is obtained. Then, correction processing is performed on the second spectral image data 72B based on the first spectral image data 72A and the correction value. Therefore, for example, compared to a case where correction processing is performed on the second imaging data 70B before interference removal processing based on the first imaging data 70A before interference removal processing and the correction value, correction processing can be performed for each wavelength band λ, so that the spectral reflectance of the subject 2 can be accurately estimated for each wavelength band λ.

[0155] Next, a modification of this embodiment will be described.

[0156] In the above embodiment, the imaging device 10 is a multispectral camera having a plurality of polarizing filters 22 and a plurality of polarizers 48, and acquiring the value of each wavelength band λ that is mixed in each physical pixel of the image sensor 28 by performing interference removal processing based on the polarization angles of the plurality of polarizing filters 22 and the polarization angles of the plurality of polarizers 48. However, the imaging device 10 may be a multispectral camera (hereinafter referred to as a multispectral camera of an existing type) having a spectral filter in each physical pixel of the image sensor 28, and acquiring the value of each wavelength band λ directly from each physical pixel without performing interference removal processing. In addition, in a multispectral camera of an existing type, a correction processing may be performed based on the first temperature t1 and the second temperature t2 in the same manner as described above.

[0157] In addition, in the above embodiment, the correction process is a process of deriving the second reflectance r2, which is the spectral reflectance of the subject 2, using the above formula (5) based on the first reflectance r1, which is the known spectral reflectance of the calibration reference chart 1, the value w of the first imaging data 70A, the value u of the second imaging data 70B, the first correction value c1, and the second correction value c2, but the following process may also be used.

[0158] In other words, the correction process related to the modified example may include a first correction process that corrects the value w of the first imaging data 70A using equation (9) based on a first reflectance r1, which is a known spectral reflectance of the calibration reference chart 1, and a first correction value c1, to derive a corrected value w1 of the first imaging data 70A; a second correction process that corrects the value u of the second imaging data 70B using equation (10) based on the second correction value c2, to derive a corrected value u1 of the second imaging data 70B; and a third correction process that derives a second reflectance r2, which is the spectral reflectance of the subject 2, using equation (11) based on the corrected value w1 of the first imaging data 70A and the corrected value u1 of the second imaging data 70B.

number

number

number

[0159] In the correction process according to this modification, the first correction process for deriving the value w1 of the corrected first imaging data 70A is performed separately from the second correction process and the third correction process, so that even when a plurality of different subjects 2 are imaged in the imaging environment 138 in which the light irradiated from the light source 132 and the state of the light source 132 are the same, it is not necessary to perform the first correction process each time the subject 2 is imaged. This makes it possible to improve the efficiency of the inspection process compared to the above embodiment in which it is required to image the calibration reference chart 1 each time the subject 2 is imaged.

[0160] In addition, in the above embodiment, interference removal processing is performed in the imaging device 10, and multiple spectral image data 72 are input from the imaging device 10 to the processing device 90, but the imaging data 70 before interference removal processing may be input from the imaging device 10 to the processing device 90, and interference removal processing may be performed in the processing device 90 to generate multiple spectral image data 72.

[0161] In addition, in the above embodiment, the processor 60 is exemplified for the imaging device 10, but instead of the processor 60 or together with the processor 60, at least one other CPU, at least one GPU, and / or at least one TPU may be used.

[0162] In addition, in the above embodiment, the processor 94 is exemplified as the processing device 90, but instead of the processor 94 or together with the processor 94, at least one other CPU, at least one GPU, and / or at least one TPU may be used.

[0163] In the above embodiment, the inspection program 100 is stored in the storage 96 of the processing device 90, but the technology of the present disclosure is not limited to this. For example, the inspection program 100 may be stored in the storage 62 of the imaging device 10, and the inspection program 100 may be executed by the processor 60 of the imaging device 10.

[0164] In the above embodiment, the imaging device 10 has been described with reference to an example in which the spectral image generation program 80 is stored in the storage 62, but the technology of the present disclosure is not limited to this. For example, the spectral image generation program 80 may be stored in a portable non-transient computer-readable storage medium (hereinafter simply referred to as a "non-transient storage medium") such as an SSD or a USB memory. The spectral image generation program 80 stored in the non-transient storage medium may be installed in the computer 56 of the imaging device 10.

[0165] In addition, the spectral image generation program 80 may be stored in a storage device such as another computer or server device connected to the imaging device 10 via a network, and the spectral image generation program 80 may be downloaded in response to a request from the imaging device 10 and installed on the computer 56 of the imaging device 10.

[0166] Furthermore, it is not necessary to store the entire spectral image generation program 80 in a storage device such as another computer or server device connected to the imaging device 10, or in the storage 62; only a portion of the spectral image generation program 80 may be stored therein.

[0167] In the above embodiment, the processing device 90 has been described with an example in which the inspection program 100 is stored in the storage 96, but the technology of the present disclosure is not limited to this. For example, the inspection program 100 may be stored in a non-transitory storage medium. The inspection program 100 stored in the non-transitory storage medium may be installed in the computer 92 of the processing device 90.

[0168] In addition, the inspection program 100 may be stored in a storage device such as another computer or server device connected to the processing device 90 via a network, and the inspection program 100 may be downloaded upon request from the processing device 90 and installed in the computer 92 of the processing device 90.

[0169] Furthermore, it is not necessary to store the entire inspection program 100 in a storage device of another computer or server device connected to the processing device 90, or in the storage 96; only a part of the inspection program 100 may be stored therein.

[0170] Furthermore, although the imaging device 10 includes a built-in computer 56, the technology of the present disclosure is not limited to this. For example, the computer 56 may be provided outside the imaging device 10.

[0171] Furthermore, although the processing device 90 has a built-in computer 92, the technology of the present disclosure is not limited to this. For example, the computer 92 may be provided outside the processing device 90.

[0172] In the above embodiment, the imaging device 10 is exemplified by the computer 56 including the processor 60, the storage 62, and the RAM 64, but the technology of the present disclosure is not limited to this, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the computer 56. Also, instead of the computer 56, a combination of a hardware configuration and a software configuration may be used.

[0173] In the above embodiment, the processing device 90 is exemplified by a computer 92 including a processor 94, a storage 96, and a RAM 98, but the technology of the present disclosure is not limited to this, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the computer 92. Also, instead of the computer 92, a combination of a hardware configuration and a software configuration may be used.

[0174] In addition, the following various processors can be used as hardware resources for executing the various processes described in the above embodiment. An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing various processes by executing software, i.e., a program. An example of a processor is a dedicated electronic circuit, which is a processor having a circuit configuration designed specifically for executing a specific process, such as an FPGA, a PLD, or an ASIC. Each processor has a built-in or connected memory, and each processor uses the memory to execute various processes.

[0175] The hardware resources that execute various processes may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resources that execute various processes may be a single processor.

[0176] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes various processes. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute various processes in a single IC chip, as typified by SoC. In this way, various processes are realized using one or more of the above-mentioned various processors as hardware resources.

[0177] Furthermore, more specifically, the hardware structure of these various processors can be an electronic circuit that combines circuit elements such as semiconductor elements. Also, the above-mentioned gaze detection process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be changed, without departing from the spirit of the invention.

[0178] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0179] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0180] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0181] 1 Calibration reference chart 2. Subject 3. Correction Reference Chart 10. Imaging device 10A input section 12 Lens device 14 Imaging device body 16 Pupil division filter 18 Frame 20 Spectral Filter 22 Polarizing Filter 24 Aperture 26 Optical system 28 Image Sensor 30 First lens 32 Second lens 34 Photoelectric conversion element 34A Image plane 36 Signal Processing Circuit 38 Pixel Layer 40 Polarizing filter layer 42 Spectral filter layer 44 pixels 46 pixel blocks 48 Polarizer 50 Filters 52 Control Driver 54 Input / Output Interface 56 Computers 58 Communication Equipment 60 processors 62 Storage 64 RAM 66 Bus 70 Imaging data 70A First Imaging Data 70B Second imaging data 72 Spectral Image Data 72A First Spectral Image Data 72B Second spectrum image data 74 Temperature Data 74A First Temperature Data 74B Second temperature data 80 Spectral Image Generator 82 Output value acquisition unit 84 Interference removal processing unit 90 Processing Equipment 92 Computer 94 Processors 96 Storage 98 RAM 100 Inspection Program 102 First Data Acquisition Unit 104 Second Data Acquisition Unit 106 Wavelength band selection unit 108 Area division part 110 Temperature correction section 112 Image data generation unit 114 Display control unit 122 Display device 124 Reception device 130 Inspection System 132 Light source 134 Case 134A Ceiling 134B Bottom 136 Temperature Sensor 138 Imaging Environment 140 divided area 142 First area 144 Second area 150 Image data

Claims

1. A processor is provided. The processor, acquiring first imaging data obtained by imaging the calibration member with a spectroscopic imaging device in an imaging environment where the temperature is a first temperature; acquiring second imaging data obtained by imaging the subject with the spectroscopic imaging device in the imaging environment where the temperature is a second temperature; A correction process for temperature is performed on the second imaging data based on the first imaging data and a correction value for the imaging data of the spectroscopic imaging device. Data processing device.

2. The processor obtains the first temperature and the second temperature.

2. A data processing apparatus according to claim 1.

3. The correction process is a process based on a difference between the first temperature and the second temperature.

2. A data processing apparatus according to claim 1.

4. The processor generates a correction value to be used in the correction process based on the temperature and the imaging data.

2. A data processing apparatus according to claim 1.

5. The correction value is a correction value selected from a group of correction values.

5. A data processing apparatus according to claim 4.

6. The correction value group is stored in a memory in a table format that defines the relationship between the correction value and temperature.

6. A data processing apparatus according to claim 5.

7. The correction process includes: obtaining a first correction value corresponding to the first temperature from the group of correction values; obtaining a second correction value corresponding to the second temperature from the group of correction values; A process of deriving a second reflectance which is a spectral reflectance of the subject based on a first reflectance which is a spectral reflectance of the calibration member, the first imaging data, the second imaging data, the first correction value, and the second correction value.

6. A data processing apparatus according to claim 5.

8. When the first reflectance is r1, the second reflectance is r2, the value of the first imaging data is w, the value of the second imaging data is u, the first correction value is c1, and the second correction value is c2, the second reflectance is calculated by the following formula (1):

8. A data processing apparatus according to claim 7.

9. The correction process includes: a first correction process for correcting a value w of the first imaging data by equation (2) to derive a value w1 of the first imaging data after correction, where r1 is the first reflectance and c1 is the first correction value; a second correction process for correcting a value u of the second imaging data by equation (3) when the second correction value is c2, to derive a value u1 of the second imaging data after correction; and a third correction process for deriving the second reflectance r2 by equation (4) based on the corrected value w1 of the first imaging data and the corrected value u1 of the second imaging data.

8. A data processing apparatus according to claim 7.

10. The spectroscopic imaging device includes an imaging element, The processor executes the correction process in accordance with an imaging area of ​​the imaging element.

2. A data processing apparatus according to claim 1.

11. The processor, The correction process is performed on a plurality of divided regions obtained by dividing the imaging region.

11. A data processing apparatus according to claim 10.

12. The correction value used in the correction process is set according to the divided area. A data processing apparatus according to claim 11.

13. The spectroscopic imaging device includes a pupil division filter having a plurality of spectroscopic filters with different wavelength bands, The divided regions have different division patterns according to the wavelength bands.

12. A data processing apparatus according to claim 11.

14. The divided regions are divided based on a peak position where the intensity of light in the wavelength band becomes a peak on the image forming surface of the image sensor.

14. A data processing apparatus according to claim 13.

15. the imaging region includes a first region including the peak position and a second region farther from the peak position than the first region, The first region has a larger area than the second region.

15. A data processing apparatus according to claim 14.

16. The divided regions are divided in at least one of the vertical and horizontal directions of the imaging region. A data processing apparatus according to claim 11.

17. The processor derives an index related to the subject based on the second imaging data.

2. A data processing apparatus according to claim 1.

18. The index includes a contrast in brightness of light in a plurality of wavelength bands.

18. A data processing apparatus according to claim 17.

19. The index includes the normalized difference vegetation index 18. A data processing apparatus according to claim 17.

20. The spectroscopic imaging device is a multispectral camera.

2. A data processing apparatus according to claim 1.

21. The spectroscopic imaging device is An imaging element; a pupil division filter including a plurality of spectral filters having different wavelength bands and a plurality of first polarizers having different polarization directions; A plurality of second polarizers having different polarization directions provided in the imaging element; Equipped 21. A data processing apparatus according to claim 20.

22. The processor, acquiring first spectral data, which is spectral data derived for each wavelength band from the first imaging data, based on spectral information about the plurality of spectral filters, first polarization information about the plurality of first polarizers, and second polarization information about the plurality of second polarizers; acquiring second spectral data, which is spectral data derived for each of the wavelength bands from the second imaging data, based on the spectral information, the first polarization information, and the second polarization information; The correction process is performed on the second spectral data based on the first spectral data and the correction value.

22. A data processing apparatus according to claim 21.

23. The third reflectance, which is the spectral reflectance of the correction member used when acquiring the correction value used in the correction process, is a spectral reflectance different from the first reflectance, which is the spectral reflectance of the calibration member.

2. A data processing apparatus according to claim 1.

24. acquiring first imaging data obtained by imaging the calibration member with a spectroscopic imaging device in an imaging environment where the temperature is a first temperature; acquiring second imaging data obtained by imaging the subject with the spectroscopic imaging device in the imaging environment where the temperature is a second temperature; and performing a temperature correction process on the second imaging data based on the first imaging data and a correction value for the imaging data of the spectroscopic imaging device; Data processing methods, including:

25. acquiring first imaging data obtained by imaging the calibration member with a spectroscopic imaging device in an imaging environment where the temperature is a first temperature; acquiring second imaging data obtained by imaging the subject with the spectroscopic imaging device in the imaging environment where the temperature is a second temperature; and performing a temperature correction process on the second imaging data based on the first imaging data and a correction value for the imaging data of the spectroscopic imaging device; A program for causing a computer to execute a process including the steps of:

Images