Imaging device, imaging device operation method, and program

Abstract

The present disclosure relates to an imaging device capable of visualizing and presenting an invisible imaging target, an imaging device operation method, and a program. Incident light from a to-be-measured object is segmented by a spectrum, original spectrum data is generated based on a spectrum result, the spectral reflectivity of the to-be-measured object is calculated according to the original spectrum data, a visual image is generated based on the spectral reflectivity, and the visual image is displayed in real time. The invention can be applied to spectrum cameras.

Description

Technical Field

[0001] The present disclosure relates to an imaging device, an operating method of the imaging device, and a program, and particularly to an imaging device, an operating method of the imaging device, and a program capable of visualizing and presenting an invisible imaging target. Background Art

[0002] Generally, in the agricultural field, in recent years, such a technology of visualizing an invisible target by applying image sensing using spectral measurement to allow observation or inspection of the target has become increasingly popular.

[0003] For example, farmland inspection employing the Normalized Difference Vegetation Index (NDVI) or Normalized Difference Vegetation Index using near infrared wavelength light or red wavelength light has been performed.

[0004] In addition, recent developments in plant physiology have enabled observation of the internal physical state of the plant itself. Therefore, a measurement method using images of the physical state of the plant, particularly images of photosynthesis and environmental stress responses (environmental stress responses), has been proposed (see Patent Document 1).

[0005] Reference List

[0006] Patent Literature

[0007] Patent document 1: PCT Patent Publication No. WO2020 / 013102. Summary of the invention

[0008] Technical issues

[0009] Incidentally, with respect to the measurement of the physical state of the plant described in Patent Document 1, in particular, the measurement of photosynthesis and environmental stress response, there already exists a mechanism that captures images required for measurement and then processes the captured images offline to perform the measurement of the physical state of the plant, in particular, the measurement of photosynthesis and environmental stress response.

[0010] However, information associated with photosynthesis, environmental stress response, etc. is not visible information unless offline processing is done.

[0011] Therefore, for a user who desires to capture an image of an imaging target in a desired state, it is difficult to realize recognition of an imaging direction and focusing on the imaging target. Therefore, an image of an imaging target in a desired state cannot be selectively captured in an appropriate state.

[0012] The present disclosure has been developed in view of the above circumstances, and particularly enables visualization and presentation of invisible imaging targets.

[0013] Solution to the problem

[0014] An imaging device and a program according to one aspect of the present disclosure relate to an imaging device and a program, comprising: a spectral unit that separates incident light from a measurement target; a spectral front end that generates a plurality of spectral raw data based on a spectral result obtained by the spectral unit; a spectral reflectance calculation unit that calculates the spectral reflectance of the measurement target based on the spectral raw data; a visualized image forming unit that forms a visualized image based on a specific value of the spectral reflectance; and a display unit that displays the visualized image in real time.

[0015] An operating method of an imaging device according to one aspect of the present disclosure relates to an operating method of an imaging device, the operating method comprising the following steps: separating incident light from a measurement target; generating spectral raw data based on a spectral result of the incident light; calculating a spectral reflectance of the measurement target based on the spectral raw data; forming a visualized image based on a specific value of the spectral reflectance; and displaying the visualized image in real time.

[0016] According to one aspect of the present disclosure, incident light from a measurement target is separated. Spectral raw data is generated based on the spectral result. Spectral reflectance of the measurement target is calculated based on the spectral raw data. A visualization image is formed based on a specific value of the spectral reflectance. The visualization image is displayed in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a diagram illustrating incident light and diffuse light.

[0018] Figure 2 is a diagram illustrating incident light, specularly reflected light, diffuse light, absorbed light, and transmitted light.

[0019] Figure 3 It is a diagram illustrating irradiance, radiant emissivity, emissivity, radiant intensity, and radiant flux.

[0020] Figure 4 It is a diagram illustrating spectral irradiance, spectral radiant emissivity, spectral radiant rate, spectral radiant intensity and spectral radiant flux.

[0021] Figure 5 is a diagram illustrating a method for measuring spectral reflectance.

[0022] Figure 6 is a diagram illustrating the spectral irradiance measurement method.

[0023] Figure 7 is an appearance configuration diagram of the imaging device of the present disclosure.

[0024] Figure 8 : is a functional block diagram illustrating functions implemented by the imaging device of the present disclosure.

[0025] Fig. 9 It is explained by Figure 8 Functional block diagram of the functions implemented by the functional measurement unit.

[0026] Fig.10 is a diagram illustrating a 3D data cube.

[0027] Fig.11 is a diagram illustrating an example of spectrum measurement using a multi-lens.

[0028] Fig.12 is a diagram illustrating a display example of an RGB image.

[0029] Fig.13 is a diagram illustrating a PRI color mapping image.

[0030] Fig.14 : is a diagram illustrating an example of a setting image for setting the maximum value and the minimum value of a color map image.

[0031] Fig.15 is a diagram illustrating a leaf surface light intensity PAR color mapping image.

[0032] Fig.16 is a diagram illustrating an NDVI color map image.

[0033] Fig.17 is a diagram illustrating an RGB image masked in an area where the NDVI is equal to or less than a predetermined value.

[0034] Fig.18 is a diagram illustrating an RGB image masked in an area where a predetermined vegetation index, photosynthesis rate, and environmental stress response are equal to or less than predetermined values.

[0035] Fig.19 is a diagram illustrating an example of a composite image formed by superimposing a PRI color map image on an RGB image and synthesizing these images.

[0036] Fig. 20 is a diagram illustrating the PAR-PRI scatter plot and regression analysis results.

[0037] Fig.21 is a diagram illustrating the PAR-PRI heat map and regression analysis results.

[0038] Fig. 22 is a diagram illustrating a PAR-PRI box-whisker plot.

[0039] Fig.23 This means that when the user Fig.19 An illustration of an example displayed when a ROI is set on a synthetic image in FIG.

[0040] Fig.24 : is a diagram illustrating an example of highlighting on a leaf surface light intensity PAR color map image.

[0041] Fig.25 : is a diagram illustrating an example of comparative display of individuals identified by the individual identifying section and arranged side by side in the left and right parts.

[0042] Fig.26 is a flow chart illustrating the imaging and display process.

[0043] Fig. 27 is a flow chart illustrating the light source spectrum calculation process.

[0044] Fig.28 is a flowchart illustrating the light source spectrum acquisition process.

[0045] Fig.29 is a flow chart illustrating the function measurement process.

[0046] Fig.30 A configuration example of a general-purpose computer is shown. DETAILED DESCRIPTION

[0047] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0048] It is to be noted that constituent elements having substantially the same functional configuration will be given the same reference numerals in the present specification and the drawings to avoid duplication of explanation.

[0049] A mode for carrying out the present technology will be described below. The description will be presented in the following order.

[0050] 1. Overview of the Disclosure

[0051] 2. Preferred Embodiments

[0052] 3. Examples executed by software

[0053] <<1. Overview of the present disclosure>>

[0054] <Diffuse reflectance spectroscopy>

[0055] The present disclosure particularly enables visualization and presentation of invisible imaging targets.

[0056] Therefore, first, diffuse reflectance spectroscopy as the principle of invisible imaging target measurement will be described, and associated terminology definitions will also be involved.

[0057] For example, invisible imaging targets are vegetation indices such as the Normalized Difference Vegetation Index (NDVI) and the Photochemical Reflectance Index (PRI), chlorophyll fluorescence generated from plants, and the like.

[0058] It should be noted that NDVI is defined as (ρNIR-ρR) / (ρNIR+ρR). In this definition, ρNIR and ρR are the values ​​of the spectral reflectance of near-infrared light (NIR) and the spectral reflectance of red light (R), respectively. In addition, PRI is defined as (ρ531-ρ570) / (ρ531+ρ570). In this definition, ρ531 and ρ570 are the values ​​of the spectral emissivity at a wavelength of 531 nm and the spectral reflectance at a wavelength of 570 nm, respectively. In addition, chlorophyll fluorescence is a value calculated based on the spectral radiance at a specific wavelength.

[0059] These vegetation indices corresponding to the invisible imaging targets are calculated from the spectral reflectance and the principle of the corresponding measurements follows the characteristics presented below.

[0060] Specifically, Figure 1 As shown, when incident light Li enters the sample surface Sf of a sample Sb constituting a leaf of a plant or the like, light having a specific wavelength is absorbed by material components inside the sample Sb. Therefore, when the incident light Li is re-released from the sample surface Sf as diffuse light Lo, the diffuse light Lo has spectral characteristics different from those of the incident light Li.

[0061] It should be noted that Figure 1 A state is shown in which incident light Li enters the sample surface Sf of the sample Sb having a thickness D, travels along the optical path indicated by the solid line and the dotted line, and is re-released as diffuse light Lo.

[0062] More specifically, if Figure 2 As shown, when incident light Li enters a sample Sb constituting a leaf of a plant, etc., a portion of the incident light Li is reflected as total reflection light Lrm by specular reflection. A different portion is reflected as diffuse light Lrd (= Lo), another different portion is absorbed by the sample Sb as absorption light Lab, and yet another different portion passes through the sample Sb as transmission light Lp.

[0063] It should be noted that when the incident light Li is red light, about 5% to 6% of the incident light Li becomes diffuse light Lrd (= Lo), about 84% of the incident light Li becomes absorbed light Lab, about 5% to 6% of the incident light Li becomes transmitted light Lp, and the rest of the incident light Li becomes totally reflected light Lrm.

[0064] Furthermore, the absorption light Lab is mostly absorbed by pigments such as chlorophyll and carotenoids.

[0065] In this case, the diffuse light Lo (=Lrd) is a part of the component of the incident light Li from which the absorbed light Lab and the transmitted light Lp are excluded, and therefore has spectral characteristics different from those of the incident light Li.

[0066] Based on the spectral characteristics of the scattered light Lo changed from the spectral characteristics of the incident light Li, the vegetation index and the like of the sample Sb which is an invisible measurement target and constitutes the internal material components of the plant is measured by diffuse reflection spectroscopy utilizing the above characteristics.

[0067] <Irradiance, radiant emissivity, radiant intensity, radiant intensity and radiant flux>

[0068] In order to measure the changes in the above-mentioned spectral characteristics, irradiance and radiation intensity are measured instead of light brightness and intensity measured by ordinary imaging devices.

[0069] The irradiance here refers to a physical quantity indicating the radiation flux released from a point-like radiation source in a predetermined direction.

[0070] In addition, radiation intensity refers to a physical quantity indicating radiation energy released from a point-like radiation source in a predetermined direction per unit time.

[0071] Irradiance (W / m 2 ) is the input per unit area Δs of the incident light Li from the sun as the light source on the ground S on the earth, and the radiation emissivity (W / m 2 ) corresponds to the reflection from the ground S generated according to the irradiance and is output. It should be noted that (W / m 2 ) is the unit of radiation, and illuminance (Iux) is the corresponding light measurement.

[0072] The radiance is the radiation emissivity (W / m 2 ) captures the brightness observed when light is reflected from the ground S.

[0073] Emissivity (W / sr / m 2 ) is expressed as the radiant emissivity (W / m 2 ).

[0074] In other words, the emissivity (W / sr / m 2 ) is calculated by dividing the radiant intensity (W / sr) by the area, while the radiant intensity (W / sr) is calculated by dividing the radiant flux (W) by the solid angle (sr).

[0075] Therefore, the radiation intensity (W / sr) is calculated by dividing the emissivity (W / sr / m 2 ) is calculated by integrating the radiation intensity (W / sr) with the solid angle, while the radiation flux (W) is calculated by integrating the radiation intensity (W / sr) with the solid angle. 2 ), radiant intensity (W / sr), and radiant flux (W) are each units of radiant quantity, and the corresponding light measurement quantity is expressed as brightness (cd / m2 ), luminous intensity (cd) and luminous flux (lm).

[0076] <Spectral irradiance, spectral radiant emissivity, spectral radiant intensity, spectral radiant intensity and spectral radiant flux>

[0077] As described above, the vegetation index corresponding to the invisible observation target is obtained by observing the spectral characteristics. Therefore, the above irradiance and radiation intensity are respectively expressed as spectral irradiance and spectral radiation intensity of a specific wavelength, and expressed as irradiance and radiation intensity per unit wavelength.

[0078] Specifically, each of the spectral irradiance and spectral radiant emissivity is expressed in (W / m 2 / nm) is obtained as a unit.

[0079] Therefore, the unit of spectral irradiance is (W / sr / m 2 / nm), the unit of spectral radiant intensity is (W / sr / nm), and the unit of spectral radiant flux is (W / nm).

[0080] Furthermore, in the case where the sample is considered to achieve perfect diffuse reflectance at the spectral irradiance observed by the imaging device C, the spectral radiant emissivity is calculated by multiplying the measured value by π.

[0081] Furthermore, in the case where the reflectance of the sample Sb is 100%, the spectral radiant emissivity and the spectral illuminance have the same value.

[0082] <Measurement of spectral reflectance>

[0083] refer to Figure 1 The spectral characteristics of the diffuse light Lo described include the spectral reflectance and spectral irradiance of the diffuse light Lo relative to the sample Sb. Depending on the type of vegetation index desired to be measured, etc., one or both of these may be required.

[0084] like Figure 5 As shown, the spectral reflectance R(λ) (which is the reflectance of the spectral component of the wavelength λ included in the incident light Li and corresponding to the leaf RR composed of the sample Sb) is a value calculated by dividing the spectral radiation emittance E(λ) (which indicates the scattered light Lo as the incident light Li reflected on the leaf RR and observed by the imaging device C) by the solar spectral irradiance I(λ) of the incident light Li (=E(λ) / I(λ)=E×I -1 ).

[0085] like Figure 6As shown, the sunlight spectral irradiance I(λ) in this case is obtained by using an imaging device C from the observed value obtained when observing the reflected light Lr generated by the reflection of the incident light Li on the standard diffuse reflection plate RB composed of a complete diffuse plate (Lambertian diffuser).

[0086] Here, the spectral irradiance Ic(λ) is obtained by calibrating the readout value Is(λ) read from the image sensor of the imaging device C. In addition, the sunlight spectral irradiance I(λ) is obtained by dividing the spectral irradiance Ic(λ) by the reflectivity of the standard diffuse reflection plate RB.

[0087] It should be noted that, in the case where only spectral reflectance is needed, if the ratio of input light to output light can be obtained, the spectral irradiance can be omitted from calculation. In addition, no calibration is required. Therefore, the processing and calibration for obtaining the spectral irradiance can be eliminated.

[0088] However, to measure fluorescence (chlorophyll fluorescence) and leaf surface light intensity, absolute values ​​are required. Therefore, the measurement of reflectance-calibrated spectral irradiance is necessary.

[0089] Meanwhile, since the measurement of vegetation index and the like obtained from spectral irradiance, spectral reflectance and the like using the above-mentioned diffuse reflectance spectroscopy needs to be applied to the offline processing of the captured image, the information obtained when imaging using the imaging device is invisible information that is invisible. Therefore, for example, it is not possible to selectively capture an image of an imaging target in a predetermined state based on the vegetation index in an appropriate state.

[0090] In other words, since the information for identifying the imaging target is invisible information, the desired imaging target cannot be visually identified. Therefore, the image of the desired imaging target cannot be captured with appropriate exposure and appropriate focus state.

[0091] Therefore, according to the present disclosure, information indicating the state of vegetation (which is invisible information such as a vegetation index) is allowed to be visualized in real time and presented as a live view display when observing vegetation using an imaging device, and images of imaging targets in specific states can be selectively captured under appropriate conditions.

[0092] <<2. Preferred Embodiment>>

[0093] Next reference Figure 7 and Figure 8 An imaging device of a preferred embodiment to which the technology of the present disclosure is applied is described. Note that, Figure 7 is a perspective view of the appearance of the imaging device 31 of the present disclosure, and Figure 8 3 is a functional block diagram illustrating functions implemented by the imaging device 31 .

[0094] like Figure 7As shown, the imaging device 31 has a typical interchangeable lens type camera shape, and includes a main body 40 , a lens unit 41 , a liquid crystal display (LCD) 42 , and keys 43 .

[0095] The lens unit 41 has a lens 121 ( Figure 8 ) and spectral unit 122 ( Figure 8 ) configuration, and separates incident light of each predetermined frequency band, and an image sensor (imaging element) 124 ( Figure 8 ) to collect and focus light on an imaging surface.

[0096] The LCD 42 is provided on the rear side of the imaging device 31 with respect to the incident direction of incident light, displays various types of information, and also has a touch panel 111 ( Figure 8 ) to receive various types of operation inputs. In addition, the LCD 42 converts invisible information associated with objects within the field of view of the lens unit 41, such as vegetation index, which would not be visible unless image processing is applied thereto, into visual information such as color map images, and presents the visual information in real time through what is generally called a live view display.

[0097] The key 43 functions as a shutter button that is operated when a still image is captured, and also functions as a button that is operated when instructions for recording start and recording end are issued during capturing of a moving image.

[0098] like Figure 8 As shown, the imaging device 31 includes an optical block 101, a spectral processing unit 102, a spectral application unit 103, a visualization unit 104, a statistical analysis unit 105, a recognition processing unit 106, a system control unit 107, a camera control unit 108, a touch panel 111, a recording device 112, a communication device 113, an LCD 42 and a key 43.

[0099] The optical block 101 includes a lens unit 41 , and generates a spectral imaging result including a pixel signal according to the amount of incident light separated and focused by the lens unit 41 , and outputs the spectral imaging result to the spectral processing unit 102 .

[0100] More specifically, the optical block 101 includes a lens 121 , a spectral unit 122 , a shutter 123 , and an image sensor 124 , and the lens 121 and the spectral unit 122 constitute the lens unit 41 .

[0101] The lens 121 configured to be driven in the incident direction of the incident light by the driver 195 controlled by the AF control section 194 of the camera control unit 108 transmits the incident light and also focuses the incident light on the imaging surface of the image sensor 124 .

[0102] The spectrum unit 122 is an optical unit that separates incident light. The spectrum unit 122 separates incident light of each predetermined wavelength band, and introduces the separated light into the image sensor 124 via the shutter 123. For example, the spectrum unit 122 is a spectrum type (CTIS (Computed Tomography Spectrometer) type) using a diffraction grating or a multi-lens bandpass filter type. For example, the spectrum unit 122 does not need to be these types of units, and may have any configuration as long as light separation can be achieved, such as a surface plasmon resonance type, a Fourier spectroscopy type, and a Fabry-Perot type.

[0103] In the case where the spectrum unit 122 is of the CTIS type using a diffraction grating, information including a spectrum direction and a resolution direction is input to the image sensor 124 .

[0104] In the case where the spectral unit 122 is a multi-lens bandpass filter type, a plurality of spectral images transmitted through the lens 121 are formed on the image sensor 124, and the lens 121 includes a plurality of lenses set for each of at least four types or more bands including visible light (RGB (red light, green light, blue light)) and NIR (near infrared light).

[0105] The shutter 123 is provided at the front stage of the image sensor 124 and is configured so as to be mechanically controlled to be opened or closed by a driver 193 controlled by an AE control section 192 of the camera control unit 108. In this way, the light amount is controlled by controlling the transmission or shielding of incident light entering through the lens unit 41.

[0106] The image sensor 124 includes a complementary metal oxide semiconductor (CMOS) image sensor, a charge coupled device (CCD) image sensor, etc. arranged in an array for each pixel, and when the shutter 123 is open, outputs a pixel signal corresponding to the amount of incident light focused and separated by the lens unit 41.

[0107] The spectrum processing unit 102 generates a 3D data cube described below based on the spectrum imaging result, and outputs the generated 3D data cube to the spectrum application unit 103 .

[0108] More specifically, the spectrum processing unit 102 includes a spectrum front end 131 and a spectrum radiance calculation unit 132 .

[0109] The spectral front end 131 divides the data into a plurality of wavelength images (spectral images), that is, divides the raw data as the readout values ​​obtained by the image sensor 124 into data for each wavelength, and outputs the divided raw data as spectral raw data to the spectral radiance calculation section 132 .

[0110] For example, in the case where the spectrum unit 122 is of CTIS type, information including the spectrum direction and the resolution direction read by the image sensor 124 is divided into a plurality of two-dimensional data for each wavelength and converted into a 3D data cube. Fig. 9 Describes the details of a 3D data cube.

[0111] Furthermore, in the case where the spectral unit 122 is a multi-lens bandpass filter type, images are cut out for each wavelength and then aligned with each other. Thereafter, the output form is roughly similar to information in a 3D data cube. It should be noted that the following reference Fig.10 Details of an example of a multi-lens bandpass filter are described.

[0112] The spectral radiance calculation section 132 calculates the spectral radiance of the measurement target from spectral raw data including readout values ​​obtained by the image sensor 124 and separated and provided by the spectral front end 131 in consideration of spectral characteristics of the optical system such as the image sensor 124 and the lens 121 .

[0113] More specifically, the spectral radiance calculation unit 132 calculates the spectral radiance based on, for example, the following formula (1).

[0114] I = Si / F ... (1)

[0115] In this formula, I represents spectral radiance, Si represents a readout value (spectral raw data) obtained by the image sensor 124 and separated, and F represents a spectral characteristic of the optical system.

[0116] The spectrum application unit 103 forms an RGB image as a visible image and invisible two-dimensional data including invisible information based on the spectral radiance provided in the form of a 3D data cube, and outputs these to the visualization unit 104 , the statistical analysis unit 105 , and the recognition processing unit 106 .

[0117] The spectrum application unit 103 includes a spectral reflectance calculation section 141 , a light source spectrum calculation section 142 , an RGB development section 143 , a vegetation index calculation section 144 , and a function measurement section 145 .

[0118] The spectral reflectance calculation unit 141 calculates the spectral reflectance by using the following formula (2) based on the spectral radiance of the measurement target provided from the spectral processing unit 102 and the spectral radiance provided from the light source spectrum calculation unit 142 and associated with the light source, and outputs the calculated spectral reflectance to the vegetation index calculation unit 144.

[0119] R = I / L ... (2)

[0120] In this formula, R represents spectral reflectance, I represents spectral radiance of the measurement target, and L represents spectral radiance of the light source.

[0121] It should be noted that, in addition to the spectral radiance provided from the spectral processing unit 102 and associated with the measurement target and the spectral radiance provided from the light source spectrum calculation unit 142 and associated with the light source, the spectral reflectance calculation section 141 can also calculate the spectral reflectance based on the spectral raw data, and output the calculated spectral reflectance to the vegetation index calculation section 144. At this time, in the case where the spectral radiance is not required (in the case where the functional measurement section 145 is not used), the spectral radiance calculation section 132 does not need to perform conversion to the spectral radiance. Specifically, in this case, the spectral reflectance calculation section 141 and the light source spectrum calculation section 142 operate according to the input of the readout value (spectral raw data) obtained and separated by the image sensor 124. In addition, the spectral reflectance calculation section 141 outputs the calculated spectral reflectance in the form of the above-mentioned 3D data cube.

[0122] The light source spectrum calculation unit 142 calculates the spectral radiance of the light source based on the readout value obtained by the image sensor 124 and associated with the area identified according to the user's specification of the position of the standard diffuse reflection plate in the image or the identification of the position of the standard diffuse reflection plate according to image recognition from the image, etc., and outputs the calculated spectral radiance to the spectral reflectance calculation unit 141.

[0123] Furthermore, in a case where the position of the above-mentioned standard diffuse reflection plate is not specified, identified, or detectable, the light source spectrum calculation section 142 may acquire information detected by a sensor that detects the spectral radiance of an external light source via communication.

[0124] Furthermore, the light source spectrum calculation section 142 may store the last detected spectral radiance of the light source and use the last detected spectral radiance when the above-mentioned standard diffuse reflection plate is not specified or cannot be detected.

[0125] The RGB developing section 143 forms an RGB image based on the spectral reflection luminance supplied from the spectral processing unit 102 and the spectral reflectance calculated by the spectral reflectance calculating section 141 , and outputs the formed RGB image to the recognition processing unit 106 .

[0126] It should be noted that the RGB image formed by the RGB developing section 143 may be an image including pixel values ​​based on the spectral radiance of a typical measurement target. However, the RGB image herein is an image including pixel values ​​based on the spectral reflectance obtained by dividing the spectral radiance of the measurement target by the spectral radiance of the light source. In other words, the RGB image herein is an image including pixel values ​​generated based on a value obtained by standardizing the spectral radiance of the measurement target based on the spectral radiance of the light source.

[0127] Specifically, for example, in the case of processing an image including pixel values ​​based on the spectral radiance of a typical measurement target as an RGB image, since sunlight as a light source is white light during the day, the image to be obtained during the day is an image of strong white as a whole. However, since sunlight as a light source turns red at night, the image to be obtained at night is an image of strong red as a whole. Therefore, an RGB image that is expected to have a uniform color may have different colors depending on the change of sunlight as a light source.

[0128] However, in the case where the RGB image includes pixel values ​​based on the spectral reflectance of the image formed by the RGB developing unit 143 of the present disclosure, the pixel values ​​are calculated based on values ​​standardized based on the spectral radiance of the sunlight as the light source. Therefore, the influence of the variation of the sunlight as the light source is eliminated. Therefore, the RGB image formed by the RGB developing unit 143 of the present disclosure can be turned into an image with appropriate colors regardless of the variation of the sunlight as the light source.

[0129] The vegetation index calculation section 144 calculates vegetation indices such as NDVI and PRI based on spectral reflectance, and outputs the calculated vegetation index as invisible two-dimensional data to the function measurement section 145 , the visualization unit 104 , the statistical analysis unit 105 , and the recognition processing unit 106 .

[0130] The functional measurement section 145 measures chlorophyll fluorescence and leaf light intensity based on the spectral radiance provided from the spectral processing unit 102 and associated with the measurement target and various types of vegetation indices provided from the vegetation index calculation section 144, calculates the photosynthesis rate (filtered SIF (solar induced chlorophyll fluorescence)) and the environmental stress response (filtered PRI) by applying processing using a specific algorithm, and outputs the calculated photosynthesis rate and environmental stress response as invisible two-dimensional data to the visualization unit 104, the statistical analysis unit 105, and the recognition processing unit 106. In addition, the functional measurement section 145 can obtain the photosynthesis rate (ETR (electron transfer rate): electron transfer rate of photosystem II) and the environmental stress response (NPQpri (non-photochemical quenching by PRI: quantitative estimate of the environmental stress response generated by PRI)) as more accurate values ​​compared with the filtered SIF and the filtered PRI. It is to be noted that reference will be made to the following. Fig. 9 The detailed configuration of the function measuring section 145 is described.

[0131] The visualization unit 104 including the color map 151 forms a color mapping image (visualization image) including an RGB image by applying color mapping such as RGB to two-dimensional invisible data, and outputs the formed color mapping image to the system control unit 107, wherein the two-dimensional invisible data includes various types of vegetation indices, chlorophyll fluorescence, leaf light intensity, photosynthesis rate, and environmental stress response provided from the vegetation index calculation unit 144 and the function measurement unit 145 of the spectral application unit 103.

[0132] The statistical analysis unit 105 performs statistical analysis on the corresponding numerical values ​​of two-dimensional invisible data including various types of vegetation indices, chlorophyll fluorescence, leaf light intensity, photosynthesis rate and environmental stress response provided from the vegetation index calculation unit 144 and the function measurement unit 145 of the spectral application unit 103, forms a graph image based on the analysis results, and then outputs the analysis results and the graph image to the system control unit 107.

[0133] More specifically, the statistical analysis unit 105 includes a statistical analysis section 161 and a graph generation section 162 .

[0134] The statistical analysis section 161 performs statistical analysis on the numerical values ​​of the two-dimensional invisible data including various types of vegetation indices, chlorophyll fluorescence, leaf surface light intensity, photosynthesis rate, and environmental stress response provided from the vegetation index calculation section 144 and the function measurement section 145 of the spectrum application unit 103, and outputs the analysis results as analysis values ​​to the graph generation section 162 and the system control unit 107. For example, the statistical analysis section 161 obtains the correlation between the leaf surface light intensity and the environmental stress response based on the statistical analysis.

[0135] When information for specifying a ROI (Region of Interest) area is input via the input section 185 from the touch panel 111 or the key 43 for inputting an operation input corresponding to operation details, the statistical analysis section 161 can extract only the ROI area to which statistical analysis is applied and perform statistical analysis on the extracted ROI area.

[0136] The graph generation unit 162 creates a scatter plot, a heat map, a box-and-whisker plot, or the like as a graph image based on the analysis result obtained by the statistical analysis unit 161, and outputs the created graph image to the system control unit 107. Specifically, for example, in the case where a correlation between leaf surface light intensity and environmental stress response is obtained through the statistical analysis performed by the statistical analysis unit 161, the graph generation unit 162 generates a graph representing the obtained correlation.

[0137] It is to be noted that each of the RGB image output from the RGB developing section 143, the color mapping image output from the visualization unit 104, and the graphic image provided from the graphic generating section 162 constitutes an RGB image group. The RGB image group is output to the image synthesizing section 182 and the recording section 183 of the system control unit 107. Figures 19 to 21 Describe the details of the statistical analysis and generated figures.

[0138] Recognition processing unit 106 performs recognition processing such as individual recognition, state recognition, and whiteboard recognition based on the RGB image and invisible two-dimensional data supplied from spectrum application unit 103 , and outputs recognition results to spectrum application unit 103 , system control unit 107 , and camera control unit 108 .

[0139] More specifically, the recognition processing unit 106 includes an individual recognition section 171 , a state recognition section 172 , and a whiteboard recognition section 173 .

[0140] The individual identification section 171 identifies the measurement target of each unit based on the RGB image and the invisible two-dimensional data supplied from the spectrum application unit 103. For example, the individual identification section 171 identifies individual plants one by one based on the RGB image and the invisible two-dimensional data supplied from the spectrum application unit 103.

[0141] It is to be noted that the individual recognition section 171 can realize recognition not only based on the RGB image and invisible two-dimensional data provided from the spectrum application unit 103, but also based on other information. For example, the individual recognition section 171 can recognize an individual by marking (attaching an identifier) ​​based on image recognition using an RGB image, marking using a two-dimensional barcode, marking each location information using geographic information system (GIS) information, marking input through operation of the touch panel 111, or by other methods.

[0142] The state recognition section 172 recognizes the state of the measurement target within the range of the measurement result (property or environmental response) based on the RGB image and the invisible two-dimensional data supplied from the spectrum application section 103 .

[0143] It is to be noted that the property is a term indicating a shape and a static characteristic. Therefore, in the case where the term "property" is adopted to indicate a shape, for example, "the property of a leaf of a plant" includes "the shape of the leaf of a plant" as an example. Meanwhile, in the case where the term "property" is adopted to indicate a static characteristic, for example, "the property of a leaf of a plant" includes "the chlorophyll concentration of the leaf of a plant" as an example.

[0144] In addition, environmental response is a term that indicates shape change and response characteristics. Therefore, in the case where the term "environmental response" is adopted to indicate shape change, for example, "environmental response of a plant" includes "changes in leaf traits according to plant adaptation" and the like as examples. At the same time, in the case where the term "environmental response" is adopted to indicate response characteristics, for example, "environmental response of a plant" includes "changes in photosynthesis speed according to changes in light intensity of the plant" as examples.

[0145] For example, the state identification section 172 identifies whether the plant is in a state showing an environmental stress response of a certain degree or more based on the environmental stress response included in the RGB image and the invisible two-dimensional data supplied from the spectrum application unit 103 .

[0146] White plate recognition section 173 recognizes the position of a white plate (standard diffuse reflection plate) based on the RGB image and invisible two-dimensional data supplied from spectrum application unit 103 , and outputs the recognition result to light source spectrum calculation section 142 of spectrum application unit 103 .

[0147] Note that each of the statistical analysis result obtained by the statistical analysis section 161, the individual identification result obtained by the individual identification section 171, and the state identification result obtained by the state identification section 172 constitutes an analysis value group. The analysis value group is output to the image synthesis section 182 and the recording section 183 of the system control unit 107.

[0148] The system control unit 107 forms an image based on the information including the RGB image group and the analysis value group provided from the spectrum application unit 103, the visualization unit 104, the statistical analysis unit 105 and the recognition processing unit 106, and displays the formed image on the LCD 42 or outputs the formed image to the recording device 112 and the communication device 113.

[0149] Furthermore, the system control unit 107 controls the camera control unit 108 based on operation input input from the key 43 and the touch panel 111 operated by the user, or information indicating an analysis value group.

[0150] More specifically, the system control unit 107 includes an image output section 181 , an image synthesis section 182 , a recording section (codec, compression, file management) 183 , an external sensor input section 184 , and an input section 185 .

[0151] The image synthesis section 182 synthesizes information indicating an RGB image group including an RGB image, a visualization image, and a graphic image, and information indicating an analysis value group (such as statistical analysis results and results of individual identification and state identification) into an image on one screen, outputs the synthesized image to the image output section 181, and displays the synthesized image on the LCD 43. Note that, the following reference is made to Figures 12 to 25 Details of a display example of a synthesized image formed by the image synthesizing section 182 are described.

[0152] The recording unit 183 encodes an RGB image group including an RGB image, a visualization image and a graphic image and an analysis value group (such as statistical analysis results and results of individual identification and state identification), compresses the encoded group, and enables a recording device 112 including an HDD, SSD, etc. to record the compressed group as file information, and also sends the file information to an external device not shown via the communication device 113.

[0153] At this time, the recording section 183 may divide the RGB image group and the analysis value group for each individual recognition result or state recognition result, and mark the RGB image group and the analysis value group with image additional data (metadata, etc.) such as Exif (Additional Identifier). Alternatively, the recording section 183 may switch the folder for recording the RGB image group and the analysis value group in the recording device 112, or cause the communication device 113 to send the RGB image group and the analysis value group to a different external device not shown.

[0154] The external sensor input section 184 receives input of measurement results from, for example, a sensor not shown provided externally (such as a sensor for measuring light source spectrum radiance of sunlight) and outputs the input to the light source spectrum calculation section 142 and the whiteboard recognition section 173 .

[0155] In addition, a sensor for measuring the spectral radiance of a light source associated with sunlight, etc. and provided to the external sensor input 184 may be attached to the imaging device 31 via an accessory, or may be of a type that introduces sunlight above the imaging device 31 via a dichroic mirror, etc.

[0156] The input section 185 receives various types of operation inputs from the keys 43 and the touch panel 111 , and supplies information associated with the received operation inputs to the spectrum application unit 103 and the camera control unit 108 .

[0157] More specifically, although not shown in detail by arrows etc. in the figure, when the spectral radiance of the light source is input, the input unit 185 provides information indicating the spectral radiance of the light source as received input to the light source spectrum calculation unit 142 and the camera control front end 191.

[0158] The camera control unit 108 controls the actions of the lens 121, shutter 123 and image sensor 124 of the optical block 101 based on the analysis value group (recognition result or statistical analysis result) obtained by the statistical analysis unit 105 and the recognition processing unit 106 or based on the operation input from the key 43 and the touch panel 111, based on the operation signal provided from the input unit 185.

[0159] More specifically, the camera control unit 108 includes a camera control front end 191 , an AE (Automatic Exposure) control section 192 , a driver 193 , an AF (Automatic Focus) control section 194 , and a driver 195 .

[0160] The camera control front end 191 receives input of an operation signal provided from the input unit 185 based on the analysis value obtained through the statistical analysis unit 105 and the recognition processing unit 106, or based on the operation input from the key 43 and the touch panel 111, and based on the received information, outputs a control signal for controlling the action of the lens 121, the shutter 123 and the image sensor 124 to at least any one of the AE control unit 192 and the AF control unit 194.

[0161] The AE control section 192 controls the action of a driver 193 for driving the shutter 123 to open and close based on a control signal received from the camera control front end 191 , and also adjusts the sensitivity of the image sensor 124 , for example, to control exposure associated with imaging.

[0162] The AF control unit 194 controls the operation of a driver 195 for driving the lens 121 based on a control signal received from the camera control front end 191 to control the focus position.

[0163] <Function Measurement Section>

[0164] Next, refer to Fig. 9 The functions implemented by the function measuring section 145 are described.

[0165] The function measuring section 145 includes a plant filter 211 , a leaf light intensity filter 212 , a leaf light intensity estimating section 213 , a chlorophyll fluorescence calculating section 214 , a plant filter 215 , and a leaf light intensity filter 216 .

[0166] The plant filter 211 acquires the NDVI and the PRI calculated by the vegetation index calculation section 144 , extracts the PRI corresponding to the NDVI having a value falling within a predetermined range through filtering, and outputs the extracted PRI to the foliage light intensity filter 212 .

[0167] Specifically, an area such as soil other than plants is mixed in the range of measurement (range of two-dimensional data). In this state, there is a possibility that the correct function (of the plant) cannot be evaluated.

[0168] Therefore, the plant filter 211 extracts only the area whose corresponding NDVI in the distribution indicating the PRI area falls within a predetermined range (eg, NDVI>0.5) by filtering to extract the PRI in the plant area. The function of the plant is thereby extracted as data allowing appropriate evaluation.

[0169] The leaf light intensity estimation section 213 estimates the leaf light intensity (PAR: Photosynthetically Active Radiation) based on the spectral radiance, and outputs the estimation result to the leaf light intensity filters 212 and 216 .

[0170] The foliage light intensity filter 212 performs filtering to extract a PRI corresponding to foliage light intensity (PAR) from the PRI extracted and provided from the plant filter 211 as an index indicating an environmental response in an area of ​​a predetermined range, and outputs the extracted PRI as a filtered PRI (environmental stress response).

[0171] For example, in the case of such environmental stress that has caused stomata to close due to soil drought, the degree of stress actually applied to the plant is greatly affected by the light intensity applied to the plant. Therefore, when there are a plurality of plants and leaves to which different degrees of leaf light intensity are applied within the measurement range (for example, leaf light intensity that varies depending on whether the leaves are facing the sun), only the PRI corresponding to the leaf light intensity within a predetermined range is extracted, thereby being able to output an environmental stress response from which the influence of the leaf light intensity is removed.

[0172] The chlorophyll fluorescence calculation unit 214 calculates chlorophyll fluorescence based on the spectral radiance, and outputs the calculated chlorophyll fluorescence to the plant filter 215 .

[0173] The plant filter 215 obtains the chlorophyll fluorescence calculation result obtained by the chlorophyll fluorescence calculation unit 214 and the NDVI calculated by the vegetation index calculation unit 144, and extracts the chlorophyll fluorescence corresponding to the NDVI having a value within a predetermined range through filtering, and outputs the extracted chlorophyll fluorescence to the leaf light intensity filter 216.

[0174] Specifically, similar to the plant filter 211, the plant filter 215 performs filtering to extract only the area of ​​NDVI within a predetermined range of the distribution of the area indicating chlorophyll fluorescence, thereby extracting chlorophyll fluorescence in the plant area. Chlorophyll fluorescence is thus extracted as data that allows appropriate evaluation.

[0175] Similar to the foliar light intensity filter 212, the foliar light intensity filter 216 performs filtering to extract chlorophyll fluorescence whose value indicating foliar light intensity (PAR) falls within a predetermined range from the chlorophyll fluorescence extracted and provided from the plant filter 215, and outputs the extracted chlorophyll fluorescence as filtered SIF (chlorophyll fluorescence).

[0176] <3D Data Cube>

[0177] Fig.10 An example of three-dimensional data, ie, a 3D data cube, generated by the spectral front end 131 and indicating the measurement target in the spatial direction (XY) and the wavelength direction (λ) is shown.

[0178] The 3D data cube is three-dimensional data of the measurement target in the spatial direction (XY) and the wavelength direction (λ). The coordinates of the corresponding points on the surface of the measurement target are represented by XY coordinates. The light intensity (λ) of the wavelength light at each coordinate position (x, y) is recorded in the data. The data cube shown in the figure includes 8×8×8 cube data. One cube represents data indicating the light intensity of a specific wavelength (λ) at a specific position (x, y). It should be noted that in the present disclosure, the spectral radiance calculation unit 132 is based on Fig.10 The spectral radiance is obtained by the light intensity (λ) of the 3D data cube shown and formed by the spectral front end 131, and the 3D data cube is provided to the spectrum application unit 103 in a form in which each light intensity (λ) is replaced by the spectral radiance. In addition, the spectral reflectance calculation unit 141, which has obtained the 3D data cube in a form replaced by the spectral radiance, calculates the spectral reflectance based on the spectral radiance, and provides the 3D data cube in a form replaced by the calculated spectral reflectance to the RGB development unit 143 and the vegetation index calculation unit 144.

[0179] It should be noted that the numbers 8×8×8 indicating the number of cubes are presented only by way of example. This number is variable depending on the spatial resolution or wavelength resolution of the spectrum measuring device.

[0180] <Example of using multiple lenses>

[0181] Fig.11 1 is a diagram schematically showing an example of imaging regions Ri formed for respective lenses among lenses provided on the image sensor 124 in the case of using a multi-lens.

[0182] Fig.11 An example of the imaging region Ri in the case where 9 multi-lenses are provided by way of example is shown. At the ends of the corresponding reference numerals, the imaging regions Ri formed for the respective lenses are given reference numerals 1 to 9 to distinguish the imaging regions Ri from each other.

[0183] Fig.11 The upper part of FIG. 1 schematically shows an arrangement example of imaging regions Ri1 to Ri9 on the image sensor 124, and Fig.11 The lower part of shows an example of the characteristics of the wavelength filter provided on the optical path for each imaging region Ri (ie, for each lens) and the details of the viewing angle.

[0184] according to Fig.11 In the example of FIG. 1 , only the imaging region Ri5 located at the center of the imaging regions Ri1 to Ri9 has a wide angle, and each of the other imaging regions Ri has a narrow angle. In the case where the measurement target is a tree, for example, Fig.11 As shown in the upper part of , images of a plurality of images of trees are included in the imaging region Ri5 with a wide angle. Meanwhile, for example, only images of trees, which are smaller in number than in the case of a wide angle, such as one image, are included in each imaging region Ri with a narrow angle.

[0185] In addition, according to Fig.11 In the example in FIG. 1 , the imaging region Ri1 belongs to the wavelength classification of “blue”, and, for example, a wavelength filter having CWL (center wavelength) = 460 nm and FWHM (full width at half maximum) = 30 nm is used. In addition, the imaging region Ri2 belongs to the wavelength classification of “red”, and, for example, a wavelength filter having CWL = 660 nm and FWHM = 20 nm is used.

[0186] In addition, the imaging region Ri3 belongs to the wavelength classification of "Red Edge 2", and for example, a wavelength filter with CWL = 715 nm and FWHM = 10 nm is used. The imaging region Ri4 belongs to the wavelength classification of "Green 1", and for example, a wavelength filter with CWL = 535 nm and FWHM = 20 nm is used.

[0187] The imaging region Ri5 having a wide angle belongs to the wavelength classification of “RGB”, and an RGB filter is used as a wavelength filter.

[0188] The RGB filter herein is a wavelength filter that transmits and separates light in R, light in G, and light in B for each pixel of the image sensor 124 . This wavelength filter used as an RGB filter is provided as a set of on-chip color filters provided for each pixel of the image sensor 124 .

[0189] According to this example, the wavelength filter of each imaging region Ri except the imaging region Ri5 among the imaging regions Ri1 to Ri9 is a wavelength filter that filters wavelengths of a predetermined wavelength band of all illumination light applied to the corresponding imaging region Ri as described above.

[0190] Imaging region Ri6 belongs to the wavelength category of "NIR1", and for example, a wavelength filter with CWL=755nm and FWHM=5nm is used. Imaging region Ri7 belongs to the wavelength category of "Green2", and for example, a wavelength filter with CWL=570nm and FWHM=20nm is used.

[0191] In addition, the imaging region Ri8 belongs to the wavelength classification of "Red Edge 1", and for example, a wavelength filter with CWL = 695 nm and FWHM = 10 nm is used. The imaging region Ri9 belongs to the wavelength classification of "NIR2", and for example, a wavelength filter with CWL = 763 nm and FWHM = 5 nm is used.

[0192] It is to be noted that a wavelength filter having CWL=757 nm and FWHM=1 nm can also be used for the above-mentioned “NIR1”, and a wavelength filter having CWL=761 nm and FWHM=1 nm can also be used for the above-mentioned “NIR2”.

[0193] Although Fig.11 An example in which the respective imaging regions Ri have the same size (corresponding to the image circle size) is shown herein, but the imaging region Ri of at least one lens may have a size different from that of the imaging regions Ri of the other lenses.

[0194] It is possible that not only the viewing angle but also the resolution or aspect ratio to be obtained is variable for each lens (each wavelength). In addition, the corresponding imaging area Ri on the image sensor 124 needs to be set at a geometrically appropriate position according to the shape of the image sensor 124.

[0195] As described above, in a configuration in which the imaging region Ri of at least one lens has a size different from that of the imaging region Ri of other lenses, an appropriate resolution or an appropriate aspect ratio of the imaging region Ri may be set for each lens. In addition, according to the shape of the image sensor 124, the imaging region Ri of the corresponding lens may be set at an appropriate position.

[0196] In addition, although Fig.11 An example in is a case where the lens has different viewing angles for different transmission wavelength bands provided by the wavelength filter, but the lens may have different viewing angles for the same transmission wavelength band provided by the wavelength filter.

[0197] It should be noted that Fig.11 The imaging regions Ri1 to Ri4 and Ri6 to Ri9 in the image processing unit 130 can be aligned and overlapped with each other to form a configuration substantially the same as the above-mentioned 3D cube data. Therefore, it is assumed in the description of the present disclosure that the spectral front end 131 creates a 3D data cube and the spectral radiance calculation unit 132 replaces the corresponding values ​​of the light intensity with the corresponding values ​​of the spectral radiance.

[0198] (PRI)

[0199] Next, PRI and PAR included in the vegetation index calculated by the function measuring section 145 will be described.

[0200] PRI (Photochemical Reflectance Index) is an indicator for measuring the stress response of plants to various types of stress factors and is generally calculated by the following formula (3).

[0201] PRI = (ρ531 - ρ570) / (ρ531 + ρ570) ... (3)

[0202] In this formula, ρ531 represents the spectral reflectance at a wavelength of about 531 nm, and ρ570 represents the spectral reflectance at a wavelength of about 570 nm.

[0203] It is considered that the degree of epoxidation / deepoxidation in the xanthophyll cycle is optically detected based on PRI. Therefore, considering the above-mentioned mechanism, it is expected that PRI can be used as an indicator for measuring the stress response of plants to each of various types of stress factors.

[0204] (Light intensity PAR on leaf surface)

[0205] The leaf surface light intensity PAR (Photosynthetically Active Radiation) is the light intensity of energy around the range from 400nm to 700nm, which is available for photosynthesis. As PAR increases, a larger amount of radiation is available for photosynthesis.

[0206] The reflectivity of the leaves can be estimated within a rough range. Therefore, by performing the following process based on the assumed reflectivity estimation, the measurement of the leaf surface light intensity PAR is achieved according to the measured radiance of the leaves. It should be noted that the leaf surface light intensity PAR varies according to the direction of each leaf relative to the sunlight, and is therefore a value including some variations.

[0207] First, when an image of corresponding values ​​of the intensity of PAR and NIR on a standard diffuse reflection plate is captured by the image sensor 124, the corresponding readout values ​​from the image sensor 124 are corrected using the calibration values, these corrected values ​​are obtained as measured values ​​A and B, and the ratio A / B is also calculated.

[0208] In addition, when capturing an image of the intensity of NIR on the leaf surface by the image sensor 124, the readout value from the image sensor 124 is corrected using the calibration value, and this calibration value is measured as the measurement value C.

[0209] Assume that the reflectance of NIR of the leaf is ρ in this text NIR , then the estimated light intensity of NIR on the leaf surface is C / ρ NIR . Using this value and the ratio of PAR to NIR (i.e., A / B), on the standard diffuse reflection plate, the leaf surface light intensity PAR is calculated as (C / ρ NIR )×(A / B).

[0210] <Display Example>

[0211] Next, with reference to Figures 12 to 25 a display example of a composite image will be described, in which the RGB image group and the analysis value group generated by the spectral application unit 103, the visualization unit 104, the statistical analysis unit 105, and the recognition processing unit 106 are combined by the image synthesis unit 182 and displayed on the LCD 42 by the image output unit 181.

[0212] <Display Example of RGB Image>

[0213] The RGB developing unit 143 of the spectral application unit 103 forms an RGB image by performing a conversion process using the standard relative spectral sensitivity curve defined by the CIE (International Commission on Illumination) based on the spectral radiance provided by the spectral radiance calculation unit 132 of the spectral processing unit 102.

[0214] It should be noted that the RGB developing unit 143 can adopt the RGB sensitivity curve of a typical image sensor to achieve a conversion compatible with the color reproducibility of existing cameras.

[0215] The RGB developing unit 143 forms, for example, an RGB image Prgb of the captured image of the leaf shown in Fig.12 based on the spectral reflectance calculated by the spectral reflectance calculation unit 141 according to the spectral radiance provided by the spectral radiance calculation unit 132 of the spectral processing unit 102. The image synthesis unit 182 displays the RGB image provided by the RGB developing unit 143 without change as needed.

[0216] In addition, when the whiteboard recognition unit 173 detects a whiteboard corresponding to the standard diffusion plate from the RGB image Prgb at this time, the frame SRB is synthesized and displayed at the position where the standard diffusion plate has been detected, as shown in the lower left part of Fig.12 .

[0217] <Display Example of Color Mapping Image>

[0218] (Color map image based on PRI)

[0219] The visualization unit 104 forms a visualized color map image by adding colors to the invisible two-dimensional data of various types of vegetation indices and foliage light intensity output from the vegetation index calculation section 144 and the function measurement section 145 of the spectrum application unit 103 using the color map 151 .

[0220] For example, in the case where the calculated PRI is provided, the visualization unit 104 maps the three primary colors of RGB by using the color mapping for visualization with naked eyes 151. Note that, needless to say, grayscale scaling may be employed.

[0221] Fig.13 is through Fig.12 A color mapping image Ppri is formed by applying color mapping based on the color mapping 151 to the invisible two-dimensional data including PRI at a viewing angle corresponding to the RGB image in the image.

[0222] In the lower part of the figure, a color mapping bar RBpri showing the mapped color in association with the numerical value of PRI is shown, showing a minimum value of -0.04 and a maximum value of 0.08.

[0223] For example, the correlation between the minimum and maximum values ​​of the value and the color map can be controlled by the user through Fig.14 The setting image SP shown in FIG.

[0224] The setting image SP includes setting columns corresponding to the types of invisible two-dimensional data. Fig.13 , the PRI setting column SPpri is surrounded by a thick frame, and the minimum value (min) is -0.040, while the maximum value (MAX) is 0.080. A display example of the color map bar RBpri corresponding to this state is presented.

[0225] In addition, setting columns are provided for PAR, NDVI, PRI, SIF, others, NPQpri, ..., ETR, ... in order from the top of the graph. It is also possible to set the maximum and minimum values ​​individually.

[0226] The range of each numerical value of the pixels constituting the invisible two-dimensional data is variable according to the measurement target. Therefore, these numerical values ​​can be set based on the calculation results obtained by calculating the maximum value and the minimum value of each numerical value in a specified screen, or can be set based on the calculation results obtained by calculating the maximum value and the minimum value in the image group when the image group including a plurality of images is specified. The visualization unit 104 forms these images in real time and outputs the formed images to the image synthesis section 182. The image synthesis section 182 causes the LCD 42 to display the color mapping image formed by the visualization unit 104 via the image output section 181 without change as needed.

[0227] (Color mapping image based on leaf light intensity PAR)

[0228] Fig.15 is through the corresponding Fig.12 A color mapping image Pls is formed by applying the color mapping based on the color mapping 151 to the leaf surface light intensity PAR at the viewing angle of the RGB image in.

[0229] Shown at the bottom of the figure is a color mapping bar RBls, which represents the mapping color associated with the numerical value of the leaf surface light intensity PAR, indicating a minimum value of 0 and a maximum value of 1000.

[0230] For example, the correlation between the minimum and maximum values ​​of the value and the color map can be controlled by the user through Fig.14 The setting image SP shown in FIG.

[0231] The visualization unit 104 forms these images in real time, and outputs the formed images to the image synthesis section 182. The image synthesis section 182 causes the LCD 42 to display the color mapping image formed by the visualization unit 104 via the image output section 181 without change as necessary.

[0232] <Filtering process>

[0233] (Plant Filter)

[0234] The RGB image as a visualization image based on the invisible two-dimensional data including the above-mentioned PRI also includes soil and the like other than plants. Therefore, preferably, in the case of visualization display or statistical analysis, the filter also excludes unnecessary areas. In this case, for example, NDVI can be used to exclude areas other than plants.

[0235] Fig.16 is in accordance with Fig.12 The visual perspective of the RGB image Prgb in the visualization image provides an example of display of the NDVI as a color mapping image Pndvi in ​​the case of invisible two-dimensional data.

[0236] Similar to the above, Fig.16 The lower portion of RBndvi shows a color map bar indicating the mapped colors associated with the numerical values ​​of NDVI. The minimum value is 0.6 and the maximum value is 1.0.

[0237] from Fig.16 It is obvious that Fig.12 In contrast, the range shown in black is an area where substances other than plants exist.

[0238] Specifically, Fig.16 The area where the NDVI is lower than the predetermined value is the range outside the area where plants exist. Fig.17 As shown in the RGB image Prgb-ndvi in ​​FIG. 1 , for example, the image synthesis unit 182 masks the image included in the image by arranging a predetermined color. Fig.12 The area Z2 in the RGB image Prgb in which the NDVI is less than a predetermined value is selected and filtering is applied so that Fig.12 The RGB image Prgb in is displayed only in the area Z1 except the area Z2 to form a composite image.

[0239] In this way, the area Z1 where only plants exist can be displayed as an RGB image. Similarly, the area Z1 where only plants exist can be extracted and used for statistical analysis.

[0240] (Other Filters)

[0241] Although the filtering process described above is based on NDVI, other types of invisible two-dimensional data may be used.

[0242] Fig.18 A display example of an RGB image Prgb-other formed by filtering based on invisible two-dimensional data including factors other than NDVI, such as leaf surface light intensity falling within a fixed range, is shown.

[0243] exist Fig.18 In the embodiment, a region Z12 showing a value lower or higher than a predetermined value in the predetermined invisible two-dimensional data is masked. Fig.12 The RGB image Prgb in is displayed in the area Z11 outside the area Z12.

[0244] In this way, only the region Z11 of the predetermined invisible two-dimensional data having a value lower or higher than the predetermined value can be displayed as an RGB image. In addition, only the region Z1 of the invisible two-dimensional data showing a value lower or higher than the predetermined value can be extracted and used for statistical analysis.

[0245] <Synthesis of visible and invisible data>

[0246] Although the above is an example of an RGB image as visible data and a color mapping image visualized for each invisible two-dimensional data, an image visualized based on the invisible two-dimensional data may be overlapped and synthesized on the visible data and displayed.

[0247] Fig.19 An example of display of an RGB image Prgb-pri including PRI extracted using a vegetation filter (NDVI: a predetermined value or lower) and PRI from Fig.13 The leaf surface intensity filter of the color map image Ppri in Fig.12 Convert the RGB image Prgb in to a color map.

[0248] like Fig.19 As shown, the image synthesis unit 182 Fig.13 The area Z21 in which the value of PRI in the color mapping image Ppri is larger than the predetermined value and Fig.12 The area Z22 composed of the RGB image Prgb is synthesized into the RGB image Prgb-pri.

[0249] Such display enables identification of only the environmental stress response of the leaf presenting a fixed leaf surface light intensity in the RGB image Prgb displayed on the LCD 42, and thus achieving appropriate imaging of the desired imaging target through appropriate exposure adjustment and appropriate focus adjustment.

[0250] It is to be noted that, in general, the environmental stress response includes a response caused by excessively high leaf surface light intensity in addition to the response caused by water stress caused by water shortage. Therefore, for example, if the region causing the environmental stress response is displayed without distinction between the region corresponding to high leaf surface light intensity and which can cause the environmental stress response caused by the leaf surface light intensity and other regions, it is difficult to distinguish between the region causing the environmental stress response due to water stress and the region causing the environmental stress response due to the leaf surface light intensity.

[0251] Therefore, if Fig.19 As shown in the RGB image Prgb-pri shown in FIG. 1 , after filtering the areas that are unlikely to cause stress generated by the light intensity on the leaf surface, the areas with induced stress are identified, so that the areas causing water stress can be clearly identified.

[0252] <Statistical Analysis Unit>

[0253] The statistical analysis unit 105 performs statistical analysis on the invisible two-dimensional data output from the spectrum application unit 103 and converts the analysis result into a graph.

[0254] More specifically, the statistical analysis unit 105 includes a statistical analysis section 161 and a graph generation section 162 .

[0255] The statistical analysis unit 161 performs statistical analysis on the invisible two-dimensional data and outputs the analysis result to the graph generation unit 162.

[0256] The graph generation unit 162 generates a graph based on the statistical analysis result of the invisible two-dimensional data provided from the statistical analysis unit 161, and outputs the generated graph.

[0257] For example, as Fig. 20 shown, based on the leaf surface light intensity PAR and PRI provided from the statistical analysis unit 161, the graph generation unit 162 generates a graph including a scatter plot having a horizontal axis representing the input (leaf surface light intensity PAR) and a vertical axis representing the output (PRI).

[0258] In addition, the statistical analysis unit 161 performs regression of an appropriate function (linear function, quadratic function, logistic function, or any function), and the graph generation unit 162 displays the parameters of the function on the graph image as a result of the regression.

[0259] In addition, for example, as Fig.21 shown, the graph generation unit 162 forms a graph image including a heat map based on similar data.

[0260] In addition, for example, the graph generation unit 162 forms a graph image including Fig. 22 the box plot shown in.

[0261] Note that these graph images can be displayed on an independent screen, or can be superimposed on an RGB image or other color mapping image displayed on the live view screen to provide a graph image as a sub-screen.

[0262] <Designation of ROI Region>

[0263] A predetermined region on the screen can be designated by operating the touch panel 111, the key 43, etc., to display the value (obtained by processing) of the designated region. In addition, the conversion of the graph based on the statistical analysis result can be updated according to the designation.

[0264] As Fig.23 shown, the average value of the predetermined invisible two-dimensional data (such as the average value of the PRI value) in the region Zroi designated by the user as the region of interest (ROI) can be calculated and displayed on the Fig.19 RGB image Prgb-pri shown.

[0265] Specifically, as Fig.23 shown, in the case where the region Zroi is designated as the ROI region by operating the touch panel 111, the statistical analysis unit 161 performs statistical analysis corresponding to the designated region Zroi.

[0266] The graphic generation unit 162 forms a graphic image based on the statistical analysis result corresponding to the area Zroi.

[0267] In addition, the position and size of the area of ​​the frame SRB corresponding to the position of the standard diffuse reflection plate can be specified by the user through a similar method.

[0268] <Display with highlight>

[0269] As described above, to designate an ROI region by operating the touch panel 111 or the key 43 , the user is required to hold the imaging device 31 while viewing the object and operate the touch panel 111 and the key 43 , and detailed designation of the position and size is limited.

[0270] Therefore, according to the present disclosure, the state recognition section 172 of the recognition processing unit 106 can highlight the area recognized as being above or below a predetermined threshold based on the invisible two-dimensional data output from the spectrum application unit 103. In this way, the user can easily recognize the measurement target.

[0271] More specifically, for example, Fig.24 As shown, the image synthesis unit 182 causes the state recognition unit 172 to highlight a region in the invisible two-dimensional data that is higher or lower than a predetermined threshold.

[0272] Presented as an example with highlighted display Fig.24 The image Pdm is an image in which areas presenting values ​​higher than a predetermined value and areas presenting values ​​lower than a predetermined value are superimposed on an RGB image, and a color mapped image is obtained by applying color mapping thereto according to a numerical value of leaf surface light intensity PAR indicating the intensity of stress associated with the plant.

[0273] exist Fig.24 In the image Pdm in FIG. 1 , each region showing a small value (low stress) in the leaf surface light intensity PAR (the value is smaller than a predetermined value determined as a normal region) is highlighted by frames Z51-1 to Z51-3 represented by a single-dot chain line having the symbol "well", and the region showing a large value (high stress) and determined as an abnormal region is highlighted by a frame Z52 corresponding to the stress detection frame represented by a dotted line having the symbol "drought stress."

[0274] Based on such a display method, the user can appropriately identify the high stress area and the low stress area by viewing the image Pdm. When it is desired to designate the low stress area as the imaging target, for example, the user can capture the image of the area highlighted by the boxes Z51-1 to Z51-3 represented by the single-point dashed line as the imaging target to appropriately capture the image of the imaging target.

[0275] In contrast, when it is desired to designate a high stress area as an imaging target, the user may capture an image of an area highlighted with a frame Z52 indicated by a dotted line as an imaging target to appropriately capture an image of the imaging target.

[0276] Furthermore, these highlighted regions may be designated as ROIs. In this case, the statistical analysis section 161 may regard the highlighted regions as ROI regions and perform statistical analysis based on the invisible two-dimensional data associated only with the highlighted regions.

[0277] In addition, in addition to the above-mentioned determination based on the numerical value of the leaf surface light intensity PAR, the state recognition unit 172 may also perform image recognition processing on the RGB image and the color mapping image. In this case, the state recognition unit 172 may search for characteristic areas through image recognition, recognize and cluster the characteristic areas, and perform state recognition associated only with the clustered characteristic areas.

[0278] For example, the state recognition unit 172 can recognize only leaves, a specific type of plant, or plants having a specific color and shape by performing image recognition processing, and achieve a state in which only leaves, a characteristic area where a specific type of plant exists, or a characteristic area where such a plant exists whose shape or color has been changed to a specific shape or color are recognized.

[0279] Furthermore, in the case where any diagnosis can be obtained from the statistical analysis results obtained from the statistical analysis section 161, the diagnostic information can also be highlighted. For example, in the case where a soil moisture sensor is included in the external sensor and indicates a small measurement value of soil moisture of a plant corresponding to the imaging target and located at the position of the soil moisture sensor, when the environmental stress response obtained by the functional measurement has a high value, an area showing high environmental stress can be highlighted to indicate the possibility that the environmental stress has been caused by water stress.

[0280] <Individual Identification Section>

[0281] The individual recognition section 171 may recognize the corresponding individual of the plant through image recognition using the RGB image and the color map image, and display the individual on the LCD screen with an identification ID given to the individual.

[0282] In this case, the individual recognition section 171 may further decompose the individual into units such as leaves and stems of a plant. In addition, the individual recognition section 171 may set a ROI according to the recognized area.

[0283] In addition, when the imaging device 31 includes GPS, IMU (accelerometer, orientation sensor, etc.), etc., the personal identification unit 171 can identify where the imaging target is located on the GIS based on the viewing angle information, and associate the GIS information with the personal information to attach a tag, for example.

[0284] Labeling can be done manually by the user, or an identification ID tag (e.g., an identification ID tag such as a two-dimensional barcode) serving as an identification index can be inserted into the screen during measurement (imaging) to achieve identification, with each identification ID tag being pre-arranged near the corresponding individual.

[0285] These types of tag information may be attached to output image data like Exif tags, or may be automatically classified for each folder to be stored in the recording device 112. Alternatively, when the communication device 113 performs data transmission, data may be transmitted while switching on the destination side according to the tag information.

[0286] Furthermore, the individual recognition section 171 can specify an individual and display the specified individual in a time lapse list as a function for classifying necessary data from stored data and displaying the classified data.

[0287] In addition, the individual identification section 171 can designate two individuals, display one of these individuals as a comparison reference sample, select the other as a sample of interest that has caused any phenomenon, and display the two individuals side by side on the left and right sides of one screen to display a comparison image that allows comparison.

[0288] Specifically, for example, the comparison image is as follows Figure 5 The image shown in .

[0289] Fig.25 The comparison image CP includes an image display column Pc in the left part for displaying individuals corresponding to the comparison reference sample side by side and an image display column Pt in the right part for displaying individuals corresponding to the sample of interest side by side, and has a configuration that allows visual comparison between the left part and the right part.

[0290] A display column IDc showing an identification ID for identifying an individual corresponding to a comparison reference sample is set above an image display column Pc showing an individual corresponding to a comparison reference sample, and a display column IDt showing an identification ID for identifying an individual corresponding to a sample of interest is set above an image display column Pt showing an individual corresponding to a sample of interest.

[0291] exist Fig.25 In the display column IDc and the display column IDt, #0001 and #0007 are respectively given to indicate that the corresponding identification IDs are #0001 and #0007.

[0292] On the left and right sides of the display column IDt, buttons Bu and Bd for switching the identification ID are provided, respectively. The button Bu is operated to decrease the value of the identification ID, and the button Bt is operated to increase the value of the identification ID. When the value of the identification ID is changed by operating the button Bu or Bd, the image of the person displayed in the image display column Pt is changed to the image corresponding to the identification ID and displayed in the image display column Pt.

[0293] The display column Cg provided on the right side of the image display column Pt is a column that displays graphs to indicate time series changes of invisible two-dimensional data, various types of statistical analysis results, etc. The numerical value of the current time is represented as a circle in the graph of the display column Cg.

[0294] The display columns Ec and Et provided on the right side below the image display columns Pc and Pt are columns that display various types of numerical values. Fig.25 , giving 0.012 and 0.020 in columns Ec and Et respectively.

[0295] The display column Ct provided in the lower right portion of the comparison image CP is a column that displays time. Fig.25 In the example, “2022 / 2 / 2 15:19” is used to indicate that the image currently displayed in the image display columns Pc and Pt was captured at 15:19 on February 2, 2022 on the Western calendar.

[0296] Buttons Br and Bp are provided on the left and right sides of the display column Ct, respectively. The button Br is operated to set the time forward, and the button Bp is operated to set the time backward. When the button Br or Bp is operated, the time in the display column Ct changes accordingly, and the images displayed in the image display columns Pc and Pt and the numerical values ​​displayed in the display columns Ec and Et are changed to images and numerical values ​​corresponding to the information related to the changed time and displayed.

[0297] According to such a display method of the comparison image CP, one individual is displayed as a comparison reference sample, another individual is selected as a sample of interest that may have caused any phenomenon, and the two individuals are displayed side by side in the left and right parts of one screen. Therefore, when switching individuals and causing time series changes, visual comparison can be achieved.

[0298] In addition, in the above example, the measurement results of the plant's vegetation indicators (NDVI and PRI) and photosynthesis rate, environmental stress response, etc. are synthesized with the RGB image to enable visual identification of status information representing the internal state of the plant, but, for example, measurement results used in non-destructive inspection within different objects based on spectral images can also be combined with the RGB image and displayed in this form.

[0299] For example, a specific position in the RGB image and the degree of the internal degradation state at the position can be visually recognized based on the spectrum measurement result in the calcium hydroxide absorption band of 1450 nm and the synthesis of the RGB image.

[0300] In addition, it is also possible to synthesize not only an RGB image that has the plant's vegetation index, photosynthesis rate, environmental stress response, and specific degradation status, but also to synthesize status information representing various measurement targets that can be measured based on spectral images to form a composite image and display it. It is also possible to form a graphic image and synthesize it by performing statistical processing.

[0301] As a result, the user can visually recognize various state information associated with the measurement target for each position based on the composite image formed and displayed as described above, can select an appropriate measurement target, and capture an image of the measurement target in an appropriate state.

[0302] <Imaging and Display Processing>

[0303] Next, refer to Fig.26 The flowchart in the description is given by Figure 8 The imaging and display processing is performed by the imaging device 31 in.

[0304] It should be noted that the reference Fig.26 The processing described in the flowchart in includes a series of processing from live view display of the subject to the operation of the shutter when imaging a plant or the like corresponding to the observation target using the imaging device 31.

[0305] In step S31 , the image sensor 124 captures an image in an optical light separation state by using the lens 121 , the spectrum unit 122 , and the shutter 123 of the optical block 101 , and outputs the captured image to the spectrum processing unit 102 .

[0306] In step S32, the spectrum front end 131 of the spectrum processing unit 102 generates a reference Fig.10 The described 3D data cube is generated and the generated 3D data cube is output to the spectral radiance calculation section 132.

[0307] In step S33 , the spectral radiance calculation section 132 calculates the spectral radiance based on the 3D data cube, and outputs the 3D data cube in which the normal spectrum pixel values ​​have been replaced with the values ​​of the spectral radiance to the spectrum application unit 103 .

[0308] In step S34 , the light source spectrum calculation section 142 performs light source spectrum calculation processing based on the 3D data cube including the spectral brightness to calculate the light source spectrum, and outputs the calculated light source spectrum to the spectral reflectance calculation section 141 .

[0309] Please note that the following reference Fig. 27 The flowchart in describes the details of the light source spectrum calculation process.

[0310] In step S35 , the spectral reflectance calculation section 141 calculates the spectral reflectance based on the 3D data cube including the spectral radiance and the light source spectrum provided from the light source spectrum calculation section 142 , and outputs the 3D data cube including the spectral reflectance to the RGB development section 143 and the vegetation index calculation section 144 .

[0311] In step S36 , the RGB development section 143 forms an RGB image based on the 3D data cube including the spectral reflectance, and outputs the formed RGB image to the recognition processing unit 106 and the image synthesis section 182 and the recording section 183 of the system control unit 107 .

[0312] In step S37, the vegetation index calculation unit 144 calculates various types of vegetation indices based on the spectral reflectance. For example, the vegetation index calculation unit 144 calculates the vegetation index including NDVI based on the reflectance of near-infrared light and the spectral reflectance of red light to form invisible two-dimensional data, and outputs the invisible two-dimensional data to the function measurement unit 145, the visualization unit 104, the statistical analysis unit 105, and the recognition processing unit 106.

[0313] In addition, for example, the vegetation index calculation unit 144 calculates the vegetation index including PRI based on the spectral reflectance at a wavelength of approximately 531 nm and the spectral reflectance at a wavelength of approximately 570 nm to constitute invisible two-dimensional data, and outputs the invisible two-dimensional data to the functional measurement unit 145, the visualization unit 104, the statistical analysis unit 105 and the recognition processing unit 106.

[0314] In step S38, the function measurement unit 145 performs function measurement processing to form invisible two-dimensional data by measuring leaf light intensity PAR, chlorophyll fluorescence, etc. based on spectral radiance and vegetation index and calculating photosynthesis rate and environmental stress response, and outputs the invisible two-dimensional data to the visualization unit 104, the statistical analysis unit 105, and the recognition processing unit 106. It should be noted that the following will refer to Fig.29 The flowchart in describes the details of the functional measurement processing.

[0315] In step S39 , the visualization unit 104 applies color mapping to various types of invisible two-dimensional data by using the color map 151 to form a color mapped image, and outputs the color mapped image to the image synthesis section 182 and the recording section 183 .

[0316] The visualization unit 104 forms a color map image by using the color map 151 based on the invisible two-dimensional data such as NDVI, PRI, photosynthesis rate, and environmental stress response.

[0317] In step S40, the statistical analysis section 161 of the statistical analysis unit 105 performs statistical processing on the invisible two-dimensional data, and outputs the processing result as an analysis value to the image synthesis section 182 and the recording section 183, and also outputs it to the graphic generation section 162. The graphic generation section 162 generates a graphic based on the statistical analysis result provided from the statistical analysis section 161, and outputs the generated graphic to the image synthesis section 182 and the recording section 183 as an RGB image group.

[0318] For example, the statistical analysis unit 161 performs statistical analysis on the leaf surface light intensity PAR-PRI, and the graph generation unit 162 generates a graph such as a reference graph based on the statistical analysis result. Figure 20 to Figure 22 The scatter plot, heat map, box-whisker plot, etc. of the leaf surface light intensity PAR-PRI.

[0319] It should be noted that at this time, the recognition processing unit 106 controls the individual recognition unit 171, the state recognition unit 172 and the whiteboard recognition unit 173 to perform individual recognition processing, state recognition processing and whiteboard recognition processing, and outputs the corresponding recognition results to the image synthesis unit 182 and the recording unit 183 of the system control unit 107 and the camera control front end 191 of the camera control unit 108.

[0320] In step S41 , the image synthesis section 182 synthesizes the RGB image, the color mapping image, the graph image, and the analysis value obtained by the statistical analysis section 161 to form a synthesized image, and outputs the synthesized image to the image output section 181 .

[0321] For example, the image synthesis section 182 forms a synthesized image by superimposing the color mapping image of PRI and the color mapping image of NDVI on the RGB image.

[0322] In addition, for example, the image synthesis unit 182 extracts only information associated with areas exhibiting values ​​higher or lower than a predetermined value based on the color mapping images of PRI and NDVI to extract only the image of the corresponding area from the RGB image, and forms a mask image with predetermined pixel values ​​for other areas.

[0323] Furthermore, for example, the image synthesis section 182 also synthesizes a synthesized image including an RGB image and a color mapping image with the graphic image and the analysis value.

[0324] In addition, the image synthesis unit 182 forms a synthesized image including highlighting by displaying a frame around an area showing a value higher or lower than a predetermined value on an RGB image or a color mapping image based on the values ​​of the vegetation index, photosynthesis rate, and environmental stress response.

[0325] In step S42, the image output section 181 displays the synthesized image on the LCD 42. Specifically, by performing the above-described series of processes, color mapping images based on various types of vegetation indices, photosynthesis speeds, and environmental stress responses, statistical analysis results, and the like in the image capture area in the direction of the optical block 101 of the imaging device 31 are displayed on the LCD 42 as a live view display.

[0326] In step S43, it is determined whether the shutter is operated by the operation of the touch panel 111, the key 43, or the like.

[0327] In a case where it is determined in step S43 that the shutter has been operated by the operation of the touch panel 111 , the key 43 , or the like, the process proceeds to step S44 .

[0328] In step S44, the input section 185 notifies the camera control front end 191 of the fact that the shutter has been operated. In response to this notification, the camera control front end 191 instructs the AE control section 192 and the AF control section 194 to capture an image.

[0329] Based on the instruction, the AF control section 194 controls the driver 195 to drive the lens 121 to achieve focus adjustment. At the same time, the AE control section 192 controls the driver 193 to control the opening or closing of the shutter 123 to capture an image by using the image sensor 124.

[0330] Thereafter, by executing the above-mentioned series of processing, the recording unit 183 causes the recording device 112 to record the analysis value groups, RGB image groups, etc. provided from each of the spectral application unit 103, the visualization unit 104, the statistical analysis unit 105 and the recognition processing unit 106, or sends these groups to an external device not shown via the communication device 113.

[0331] Note that, in the event that determination is made in step S43 that the touch panel 111 , the key 43 , or the like has not been operated and the shutter has not been operated, the processing in step S44 is skipped, and the processing proceeds to step S45 .

[0332] In step S45, it is determined whether an end instruction of an action such as power off has been issued. In the case where it is determined that an end instruction has not been issued, the process returns to step S31. In other words, the processes of steps S31 to S45 are repeated to continue the live view display until an end instruction is issued.

[0333] Thereafter, when an action end instruction such as shutoff of the power supply is issued in step S45, the process ends.

[0334] By repeating the above series of processes, a composite image including an RGB image, a color mapping image, a graphic image of statistical analysis results, and the like based on various types of vegetation indices, photosynthesis rates, and environmental stress responses in the image capture area within the field of view of the optical block 101 of the imaging device 31 is displayed on the LCD 42 as a live view display.

[0335] In this way, various types of vegetation indexes, photosynthesis rates, and environmental stress responses, which are generally considered to be invisible information, are converted into color mapping images as visible information. Therefore, the user can appropriately select an imaging target based on the vegetation index, photosynthesis rate, and environmental stress response, and capture an image of the imaging target through appropriate exposure adjustment and appropriate focus adjustment only by visually recognizing the image displayed on the LCD 42.

[0336] <Light source spectrum calculation processing>

[0337] Next, we will refer to Fig. 27 The flowchart in describes the light source spectrum calculation process.

[0338] In step S71 , the light source spectrum calculation section 142 performs a light source spectrum acquisition process to acquire a light source spectrum.

[0339] Please note that the following reference Fig.28 The flowchart in describes the details of the light source spectrum acquisition process.

[0340] In step S72 , the light source spectrum calculation section 142 determines whether the light source spectrum is acquired through the light source spectrum acquisition process.

[0341] In the event that it is determined in step S72 that the light source spectrum has been acquired, the process proceeds to step S73 .

[0342] In step S73 , the light source spectrum calculation section 142 outputs the acquired light source spectrum to the spectral reflectance calculation section 141 , and the process proceeds to step S79 .

[0343] In the case where it is determined in step S72 that the light source spectrum has not been acquired, the process proceeds to step S74.

[0344] In step S74 , the light source spectrum calculation section 142 determines whether the light source spectrum can be acquired as external sensor data via the external sensor input section 184 .

[0345] In a case where it is determined in step S74 that the light source spectrum can be acquired as external sensor data, the process proceeds to step S75 .

[0346] In step S75 , the light source spectrum calculation section 142 outputs the light source spectrum acquired as external sensor data to the spectral reflectance calculation section 141 , and the process proceeds to step S79 .

[0347] In a case where it is determined in step S74 that the light source spectrum cannot be acquired as external sensor data, the process proceeds to step S76 .

[0348] In step S76 , the light source spectrum calculation section 142 determines whether previously acquired data of the light source spectrum is stored.

[0349] In a case where it is determined in step S76 that previously acquired data of the light source spectrum has been stored, the process proceeds to step S77 .

[0350] In step S77 , the light source spectrum calculation section 142 outputs the previously acquired data of the light source spectrum to the spectral reflectance calculation section 141 as the acquired light source spectrum, and the process proceeds to step S79 .

[0351] In a case where it is determined in step S76 that previously acquired data of the light source spectrum is not stored, the process proceeds to step S78 .

[0352] In step S78 , the light source spectrum calculation section 142 outputs the typical light source spectrum as the acquired light source spectrum to the spectral reflectance calculation section 141 , and the process proceeds to step S79 .

[0353] In step S79 , the light source spectrum calculation section 142 stores the acquired light source spectrum as previously acquired data to be used in the future. Thereafter, the process ends.

[0354] Through the above-described processing, the light source spectrum is acquired and provided to the spectral reflectance calculation section 141 .

[0355] <Light Source Spectrum Acquisition Processing>

[0356] Next, we will refer to Fig.28 The flowchart in describes the light source spectrum acquisition process.

[0357] In step S91 , the light source spectrum calculation section 142 sets the spectral reflectance of the standard diffuse reflection plate.

[0358] For example, when the reflector has a fixed reflectivity regardless of wavelength, the light source spectrum calculation unit 142 sets the reflectivity. When the reflector has a reflectivity that is variable for each wavelength and has known reflectivity data, the light source spectrum calculation unit 142 reads the known reflectivity data.

[0359] In step S92 , the light source spectrum calculation section 142 estimates the light source spectrum, and estimates the spectral illumination brightness of the standard diffuse reflection plate.

[0360] In step S93 , the white plate recognition section 173 searches the RGB image for a region that exhibits a value close to the estimated spectral radiance of the standard diffuse reflection plate as the position of the standard diffuse reflection plate (white plate).

[0361] In step S94 , the light source spectrum calculation section 142 determines whether the white plate recognition section 173 has searched for the position of the standard diffuse reflection plate (white plate).

[0362] In the case where it is determined in step S94 that the white plate recognition section 173 has searched for the position of the standard diffuse reflection plate (white plate), the process proceeds to step S96 .

[0363] In step S96, the light source spectrum calculation unit 142 acquires information associated with the position of the standard diffuse reflection plate (whiteboard) searched by the whiteboard recognition unit 173, calculates the light source spectrum based on the spectral radiance with reference to the spectral reflectance of the set standard diffuse reflection plate, and outputs the calculated light source spectrum as the acquisition result.

[0364] Meanwhile, in the event that it is determined in step S94 that the white plate recognition section 173 has not yet searched for the position of the standard diffuse reflection plate (white plate), the process proceeds to step S95 .

[0365] In step S95 , the light source spectrum calculation section 142 determines whether the position of the standard diffuse reflection plate is specified as a search result by the user's operation on the touch panel 111 and provided via the input section 185 .

[0366] In a case where it is determined in step S95 that the position of the standard diffuse reflection plate has been specified as the search result and provided via the input section 185 by the user's operation on the touch panel 111 , the process proceeds to step S96 .

[0367] Specifically, in this case, the light source spectrum calculation unit 142 acquires information associated with the position of the standard diffuse reflection plate (whiteboard) input through the user's operation of the touch panel 111 as information associated with the position of the standard diffuse reflection plate (whiteboard) searched by the whiteboard recognition unit 173, calculates the light source spectrum based on the spectral radiance with reference to the spectral reflectivity of the set standard diffuse reflection plate, and outputs the calculated light source spectrum as the acquisition result.

[0368] In a case where it is determined in step S95 that the position of the standard diffuse reflection plate is not specified as the search result and is not provided via the input section 185 by the user's operation of the touch panel 111 , the process proceeds to step S97 .

[0369] In step S97 , the light source spectrum calculation section 142 outputs information indicating that the light source spectrum cannot be acquired.

[0370] The above process realizes the process of acquiring the spectrum of the light source. If the spectrum of the light source can be acquired, the calculation result is adopted. If the spectrum of the light source cannot be acquired, information indicating acquisition failure is output.

[0371] <Function measurement processing>

[0372] Next, refer to Fig.29 The flowchart in exemplifies the function measurement process performed by the function measurement section 145 .

[0373] In step S111 , the plant filter 211 acquires the NDVI and the PRI calculated by the vegetation index calculation section 144 , extracts the PRI corresponding to the NDVI having a value within a predetermined range through filtering, and outputs the extracted PRI to the foliage light intensity filter 212 .

[0374] In step S112 , the foliage light intensity estimation section 213 estimates the foliage light intensity based on the spectral radiance, and outputs the estimation result to the foliage light intensity filters 212 and 216 .

[0375] In step S113, the leaf light intensity filter 212 extracts the PRI corresponding to the predetermined leaf light intensity from the PRI extracted and provided by the plant filter 211 by filtering, and outputs the extracted PRI corresponding to the predetermined leaf light intensity as an environmental stress response (filtered PRI).

[0376] In step S114 , the chlorophyll fluorescence calculation section 214 calculates chlorophyll fluorescence (SIF) based on the spectral radiance, and outputs the calculated chlorophyll fluorescence to the plant filter 215 .

[0377] In step S115, the plant filter 215 obtains the NDVI calculated by the vegetation index calculation unit 144 and the chlorophyll fluorescence (SIF) provided from the chlorophyll fluorescence calculation part 214, extracts the chlorophyll fluorescence (SIF) corresponding to the NDVI having a value within a predetermined range through filtering, and outputs the extracted chlorophyll fluorescence (SIF) to the leaf light intensity filter 216.

[0378] In step S116, the foliage light intensity filter 216 extracts chlorophyll fluorescence (SIF) by filtering, and outputs the extracted chlorophyll fluorescence (SIF) as information associated with photosynthesis (filtered SIF), wherein the chlorophyll fluorescence (SIF) indicates a predetermined foliage light intensity in the chlorophyll fluorescence (SIF) provided from the plant filter 215 and corresponds to an NDVI having a value within a predetermined range.

[0379] Through the above-described processing, information related to photosynthesis (filtered SIF) and environmental stress response (filtered PRI) is generated and output based on spectral radiance, PRI, and NDVI.

[0380] <<3. Example of execution by software>>

[0381] Meanwhile, the above series of processing can be performed by hardware or can be performed by software. In the case of performing a series of processing by software, the program constituting the software is installed from a recording medium to a computer contained in dedicated hardware or a computer (such as a general-purpose computer) capable of performing various functions under various programs installed in the computer.

[0382] Fig.30 1001. An example of a configuration of a general-purpose computer is shown. The computer has a built-in central processing unit (CPU) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A read-only memory (ROM) 1002 and a random access memory (RAM) 1003 are connected to the bus 1004.

[0383] Connected to the input / output interface 1005 is an input section 1006 including a keyboard, a mouse or other input device through which a user inputs an operation command, an output section 1007 that outputs a processing operation screen and an image of a processing result to a display device, a storage section 1008 including a hard disk drive or the like for storing programs and various data, and a communication section 1009 that includes a local area network (LAN) adopter or the like and performs communication processing via a network represented by the Internet. Further connected to the input / output interface 1005 is a drive 1010 that reads data from and writes data to a removable storage medium 1011 such as a magnetic disk (including a floppy disk), an optical disk (including a CD-ROM (Compact Disc Read Only Memory) and a DVD (including a Digital Versatile Disk)), a magneto-optical disk (including an MD (Mini Disk), or a semiconductor memory.

[0384] The CPU 1001 executes various types of processing according to a program stored in the ROM 1002 or a program read from a removable storage medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, installed in the storage section 1008, and loaded from the storage section 1008 into the RAM 1003. Data and the like required for the CPU 1001 to execute various processing are also stored in the RAM 1003 as necessary.

[0385] According to the computer configured as above, for example, the CPU 1001 performs the above-described series of processes by loading a program stored in the storage section 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing the loaded program.

[0386] For example, the program executed by the computer (CPU 1001) may be recorded in the removable storage medium 1011 such as a package medium and provided in this form. Alternatively, the program may be provided via a wired or wireless transmission medium such as a local area network, the Internet, and digital satellite broadcasting.

[0387] By installing the removable storage medium 1011 to the drive 1010, the program of the computer can be installed in the storage unit 1008 via the input / output interface 1005. Alternatively, the program can be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. In contrast, the program can be pre-installed in the ROM 1002 or the storage unit 1008.

[0388] Note that the program executed by the computer may be a program that executes processing in time series in the order described in this specification, or a program that executes processing in parallel or at necessary timing such as when a call is made.

[0389] It should be noted that in Fig.30 CPU 1001 implementation in Figure 8 The functions of the spectrum processing unit 102, the spectrum application unit 103, the visualization unit 104, the statistical analysis unit 105, the recognition processing unit 106 and the system control unit 107 are implemented.

[0390] In addition, the system in this specification refers to a collection of multiple constituent elements (devices, modules (parts), etc.). A group of constituent elements is regarded as a system regardless of whether all the constituent elements are in the same housing. Therefore, multiple devices housed in different housings and connected to each other via a network and one device housing multiple modules in one housing are both regarded as systems.

[0391] It is to be noted that the embodiment according to the present disclosure is not limited to the above-described embodiment and may be modified in various ways without departing from the subject matter of the present disclosure.

[0392] For example, the present disclosure may have a configuration of cloud computing in which one function is shared by a plurality of devices and is executed by the devices cooperating with each other via a network.

[0393] Furthermore, the steps described with reference to the above flowcharts may all be performed by one device, or may be shared and performed by a plurality of devices.

[0394] Furthermore, in the case where one step includes a plurality of processes, the plurality of processes included in one step may be executed by one device, or may be shared and executed by a plurality of devices.

[0395] It is to be noted that the present disclosure may also be configured as follows.

[0396] <1>

[0397] An imaging device, comprising:

[0398] The spectral section separates the incident light from the measurement target;

[0399] A spectral front end, generating spectral raw data based on the spectral results obtained by the spectral unit;

[0400] A spectral reflectance calculation unit, which calculates the spectral reflectance of the measurement target based on the spectral raw data;

[0401] a visualized image forming unit that forms a visualized image based on a specific value of the spectral reflectance; and

[0402] The display unit displays the visual image in real time.

[0403] <2>

[0404] according to <1> The imaging device further comprises:

[0405] a state information calculation section that calculates state information indicating a measurement target based on a specific value of the spectral reflectance,

[0406] The visualized image forming unit forms a visualized image based on the state information.

[0407] <3>

[0408] according to <2> The imaging device further comprises:

[0409] The spectral radiance calculation unit calculates the spectral radiance based on the original spectral data.

[0410] Here, the state information calculation section calculates state information indicating the state of the measurement target based on the spectral radiance and the spectral reflectance.

[0411] <4>

[0412] according to <3> The imaging device

[0413] The spectral reflectance calculation unit calculates the spectral reflectance of the measurement target based on the spectral radiance.

[0414] <5>

[0415] according to <1> to <4> The imaging device of any one of the above,

[0416] The spectrum unit separates incident light into at least four types or more of wavelength bands including visible light and near infrared light.

[0417] <6>

[0418] according to <1> to <5> The imaging device of any one of the above,

[0419] Among them, the spectral unit is a spectroscopy type using a diffraction grating (CTIS), a surface plasmon resonance type, a Fourier spectroscopy type, a Fabry-Perot type, and a multi-lens bandpass filter type.

[0420] <7>

[0421] according to <2> An imaging device,

[0422] The state information includes a vegetation index of a plant corresponding to the measurement target and at least one of a plant trait and an environmental response.

[0423] <8>

[0424] according to <7> An imaging device,

[0425] The vegetation index includes at least one of a normalized difference vegetation index (NDVI) and a photochemical reflectance index (PRI) of plants corresponding to the measurement target.

[0426] <9>

[0427] according to <7> An imaging device,

[0428] The traits and environmental responses of the plant include at least any one of information associated with photosynthesis and environmental stress response of the plant corresponding to the measurement target.

[0429] <10>

[0430] according to <2> An imaging device,

[0431] Here, the state information is information indicating a deterioration state of concrete corresponding to a measurement target.

[0432] <11>

[0433] according to <3> An imaging device,

[0434] wherein the state information calculation unit generates state information including two-dimensional data, and

[0435] The visualized image forming section forms a color mapped image as a visualized image by applying color mapping according to a value of status information including two-dimensional data.

[0436] <12>

[0437] according to <11> The imaging device further comprises:

[0438] an RGB image forming unit that forms an RGB image based on the spectral reflectance; and

[0439] An image synthesis unit synthesizes the RGB image and the color mapping image to form a synthesized image,

[0440] The display unit displays the synthesized image as a visualized image.

[0441] <13>

[0442] according to <12> An imaging device,

[0443] The image synthesis unit superimposes a color mapping image of an area showing a value greater than a predetermined value or an area showing a value less than a predetermined value on the RGB image based on status information including two-dimensional data, and synthesizes the color mapping and the RGB image to form a synthesized image.

[0444] <14>

[0445] according to <12> An imaging device,

[0446] The image synthesis unit synthesizes the RGB image of an area showing a value greater than a predetermined value or an area showing a value less than a predetermined value and a mask image that masks areas other than the area based on state information including two-dimensional data to form a synthesized image.

[0447] <15>

[0448] according to <12> An imaging device,

[0449] The image synthesis unit forms a synthesized image based on the state information including the two-dimensional data, and the synthesized image highlights an area showing a value greater than a predetermined value or an area showing a value less than a predetermined value on the RGB image.

[0450] <16>

[0451] according to <15> The imaging device further comprises:

[0452] a state recognition unit that recognizes a state of a plant corresponding to a measurement target based on the state information including the two-dimensional data,

[0453] The image synthesis unit forms a synthesized image based on the state recognition result obtained by the state recognition unit, and the synthesized image highlights an area showing a value greater than a predetermined value or an area showing a value less than a predetermined value on the RGB image.

[0454] <17>

[0455] according to <16> The imaging device

[0456] The state recognition unit recognizes the state of the plant corresponding to the measurement target in a specific recognition area by performing image recognition processing based on an RGB image or a color mapping image included in the state information including two-dimensional data.

[0457] <18>

[0458] according to <12> The imaging device further comprises:

[0459] The individual recognition unit recognizes an individual of a plant corresponding to a measurement target included in the RGB image, the color mapping image, and the status information including the two-dimensional data.

[0460] <19>

[0461] according to <18> The imaging device

[0462] Among them, the individual identification unit identifies the individual plant corresponding to the measurement target by assigning an identifier to each individual plant based on any one of image recognition processing using RGB images, two-dimensional barcodes within RGB images, geographic information system (GIS) information, and manual input.

[0463] <20>

[0464] according to <19> The imaging device

[0465] Among them, RGB images, color mapping images, and status information including two-dimensional data are classified into different folders for each identifier and recorded in a recording device, or transmitted to different destinations for each identifier, and the folder includes an identifier for identifying an individual plant corresponding to a measurement target included in the RGB image, color mapping image, and status information as image additional data.

[0466] <21>

[0467] according to <12> The imaging device further comprises:

[0468] Statistical analysis department, statistical analysis includes status information of two-dimensional data,

[0469] The image synthesis unit forms a synthesized image including the analysis value as the statistical analysis result obtained by the statistical analysis unit.

[0470] <22>

[0471] according to <21> The imaging device further comprises:

[0472] a graph generating unit that generates a graph based on the statistical analysis result and outputs the graph as a graph image,

[0473] The image synthesis unit forms a synthesized image including the graphic image.

[0474] <23>

[0475] according to <21> The imaging device further comprises:

[0476] An input unit receives input of an RGB image, a color mapping image, and a region of interest (ROI) on a composite image;

[0477] The statistical analysis unit statistically analyzes the state information including the two-dimensional data in the ROI region input to the RGB image, the color mapping image, and the synthesized image.

[0478] <24>

[0479] An operating method of an imaging device, the operating method comprising the following steps:

[0480] Separating incident light from a measurement target;

[0481] Generate spectral raw data based on the spectral result of the incident light;

[0482] Calculate the spectral reflectance of the measurement target based on the spectral raw data;

[0483] forming a visualization image based on specific values ​​of spectral reflectance; and

[0484] Display visualization images in real time.

[0485] <25>

[0486] A program that causes a computer to:

[0487] The spectral section separates the incident light from the measurement target;

[0488] A spectral front end, generating spectral raw data based on the spectral results obtained by the spectral unit;

[0489] A spectral reflectance calculation unit calculates the spectral reflectance of the measurement target based on the spectral raw data;

[0490] a visualized image forming unit that forms a visualized image based on a specific value of the spectral reflectance; and

[0491] The display unit displays the visual image in real time.

[0492] Reference Mark List

[0493] 31 Imaging device

[0494] 41 lens units

[0495] 42LCD

[0496] 43 keys

[0497] 101 Optical Block

[0498] 102 Spectral Processing Unit

[0499] 103 Spectral Application Unit

[0500] 104 Visualization Unit

[0501] 105 Statistical Analysis Unit

[0502] 106 Identification processing unit

[0503] 107 System Control Unit

[0504] 108 Camera control unit

[0505] 111 Touch Panel

[0506] 112 Recording Device

[0507] 113 communication device

[0508] 121 lens

[0509] 122 Spectral Units

[0510] 123 Shutter

[0511] 124 Image Sensor

[0512] 131 Spectrum Front End

[0513] 132 Spectral radiance calculation unit

[0514] 141 Spectral reflectance calculation unit

[0515] 142 Light Source Spectrum Calculation Department

[0516] 143RGB Development Department

[0517] 144 Vegetation Index Calculation Department

[0518] 145 Functional Measurement Department

[0519] 151 Color Mapping

[0520] 161 Statistical Analysis Department

[0521] 162 Graphics Generation Department

[0522] 171 Individual Identification Department

[0523] 172 Status Identification Unit

[0524] 173 Whiteboard Recognition Department

[0525] 181 Image output unit

[0526] 182 Image Synthesis Department

[0527] 183 Recording unit (codec, compression, file management)

[0528] 184 External sensor input

[0529] 185 Input

[0530] 191 Camera Control Front End

[0531] 192AE Control Department

[0532] 193 Drive

[0533] 194AF Control Department

[0534] 195 Drive

[0535] 211 Plant Filter

[0536] 212 Leaf surface light intensity filter

[0537] 213 Leaf surface light intensity estimation unit

[0538] 214 Chlorophyll Fluorescence Calculation Department

[0539] 215 Plant Filter

[0540] 216 Leaf surface light intensity filter.

Claims

1. An imaging device, comprising: a spectral unit that separates incident light from a measurement target; A spectral front end, which generates spectral raw data based on the spectral result obtained by the spectral unit; A spectral reflectance calculation unit, which calculates the spectral reflectance of the measurement target based on the spectral raw data; a visualized image forming unit that forms a visualized image based on the specific value of the spectral reflectance; as well as The display unit displays the visualized image in real time.

2. The imaging device according to claim 1, further comprising: a state information calculation section that calculates state information indicating a state of the measurement target based on the specific value of the spectral reflectance, The visualized image forming unit forms the visualized image based on the state information.

3. The imaging device according to claim 2, further comprising: A spectral radiance calculation unit calculates the spectral radiance according to the original spectral data. The state information calculation section calculates the state information indicating the state of the measurement target based on the spectral radiance and the spectral reflectance.

4. The imaging device according to claim 3, in, The spectral reflectance calculation section calculates the spectral reflectance of the measurement target based on the spectral radiance.

5. The imaging device according to claim 1, in, The spectrum unit separates the incident light into at least four types or more of wavelength bands including visible light and near infrared light.

6. The imaging device according to claim 1, in, The spectroscopy unit is a spectroscopy type using a diffraction grating (CTIS: Computed Tomography Imaging Spectrometer), a surface plasmon resonance type, a Fourier spectroscopy type, a Fabry-Perot type, and a multi-lens bandpass filter type.

7. The imaging device according to claim 2, in, The state information includes a vegetation index of a plant corresponding to the measurement target and at least one of a property and an environmental response of the plant.

8. The imaging device according to claim 7, in, The vegetation index includes at least one of a Normalized Difference Vegetation Index (NDVI) and a Photochemical Reflectance Index (PRI) of the plant corresponding to the measurement target.

9. The imaging device according to claim 7, in, The traits and environmental responses of the plant include at least any one of information associated with photosynthesis and environmental stress response of the plant corresponding to the measurement target.

10. The imaging device according to claim 2, in, The state information is information indicating a deterioration state of concrete corresponding to the measurement target.

11. The imaging device according to claim 3, in, The state information calculation section generates the state information including two-dimensional data, and The visualized image forming section forms a color mapped image as the visualized image by applying color mapping according to a value of the status information including the two-dimensional data.

12. The imaging device according to claim 11, further comprising: an RGB image forming unit that forms an RGB image based on the spectral reflectance; as well as an image synthesis unit, synthesizing the RGB image and the color mapping image to form a synthesized image, The display unit displays the synthesized image as the visualized image.

13. The imaging device according to claim 12, in, The image synthesis unit superimposes the color mapping image of areas showing values ​​greater than a predetermined value or areas showing values ​​less than the predetermined value on the RGB image based on the status information including the two-dimensional data, and synthesizes the color mapping image and the RGB image to form the synthesized image.

14. The imaging device according to claim 12, in, The image synthesis section synthesizes the RGB image of an area showing a value greater than a predetermined value or an area showing a value less than the predetermined value and a mask image masking an area other than the area based on the state information including the two-dimensional data to form the synthesized image.

15. The imaging device according to claim 12, in, The image synthesis section forms the synthesized image that highlights a region showing a value greater than a predetermined value or a region showing a value less than the predetermined value on the RGB image based on the state information including the two-dimensional data.

16. The imaging device according to claim 15, further comprising: a state recognition section that recognizes a state of a plant corresponding to the measurement target based on the state information including the two-dimensional data, The image synthesis unit forms the synthesized image based on the state recognition result obtained by the state recognition unit, and the synthesized image highlights the area showing a value greater than the predetermined value or the area showing a value less than the predetermined value on the RGB image.

17. The imaging device according to claim 16, in, The state recognition section recognizes the state of the plant corresponding to the measurement target in a specific recognition area by performing image recognition processing based on the RGB image or the color map image included in the state information including the two-dimensional data.

18. The imaging device according to claim 12, further comprising: An individual identifying section identifies an individual of a plant corresponding to the measurement target included in the RGB image, the color mapping image, and the status information including the two-dimensional data.

19. The imaging device according to claim 18, in, The individual identification unit identifies the individual of the plant corresponding to the measurement target by assigning an identifier to each individual of the plant based on any one of image recognition processing using the RGB image, a two-dimensional barcode within the RGB image, geographic information system (GIS) information, and manual input.

20. The imaging device according to claim 19, in, The RGB image, the color mapping image, and the status information including the two-dimensional data are classified into different folders for each of the identifiers and recorded in a recording device, or are transmitted to different destinations for each of the identifiers, and the folders include the identifiers for identifying the individuals of the plants corresponding to the measurement targets included in the RGB image, the color mapping image, and the status information as image additional data.

21. The imaging device according to claim 12, further comprising: a statistical analysis unit for statistically analyzing the state information including the two-dimensional data, The image synthesis section forms the synthesized image including the analysis value as the statistical analysis result obtained by the statistical analysis section.

22. The imaging device according to claim 21, further comprising: a graph generating unit that generates a graph based on the statistical analysis result and outputs the graph as a graph image, The image synthesis unit forms the synthesized image including the graphic image.

23. The imaging device according to claim 21, further comprising: An input unit receives input of the RGB image, the color mapping image, and a region of interest (ROI) on the composite image; The statistical analysis unit statistically analyzes the state information including the two-dimensional data input into the ROI region on the RGB image, the color mapping image, and the synthesized image.

24. A method for operating an imaging device, the method comprising the following steps: Separating incident light from a measurement target; Generate spectral raw data based on the spectral result of the incident light; Calculating the spectral reflectance of the measurement target based on the spectral raw data; forming a visual image based on the specific value of the spectral reflectance; as well as The visualized image is displayed in real time.

25. A program that causes a computer to: A spectral unit that separates incident light from a measurement target; A spectral front end, which generates spectral raw data based on the spectral result obtained by the spectral unit; A spectral reflectance calculation unit, which calculates the spectral reflectance of the measurement target based on the spectral raw data; a visualized image forming unit that forms a visualized image based on the specific value of the spectral reflectance; as well as The display unit displays the visualized image in real time.

Images