Processing device, system, method, and computer program

Abstract

A processing device (10) is connected, via a network, to at least one terminal comprising a hyper-spectrum sensor which generates compressed image data in which hyper-spectrum information of a target object is compressed as two-dimensional image information. The information processing device (10) comprises: storage devices (110, 120) which store a plurality of sample data sets and a first restoration table for restoring hyper-spectrum data from the compressed image data, each sample data set including hyper-spectrum data for the sample and data indicating a characteristic value of the sample; and a processing circuit (140) that generates a statistical model for estimating the characteristic value from the hyper-spectrum data on the basis of the data sets of the plurality of samples, and that generates a second restoration table by editing the first restoration table in accordance with the generated statistical model.

Description

[Technical Field]

[0001] The present disclosure relates to a processing device, a system, a method, and a computer program for estimating properties of an object. [Background technology]

[0002] In recent years, efforts have been made to produce better-tasting agricultural products, such as fruits and vegetables with higher sugar content. The sugar content of agricultural products depends not only on the variety but also on the growing environment and process. Therefore, agricultural products are typically measured for their sugar content and shipped with a guaranteed sugar content. Sugar content can be measured using a saccharometer or non-destructive methods. For example, Patent Document 1 discloses a method for estimating characteristics, such as the Brix value representing the sugar content of fruit, based on the absorptance of light at multiple wavelengths, including infrared wavelengths. The system in Patent Document 1 estimates sample characteristics from scattered light generated when the sample is irradiated with light of a specific spectrum using a regression model that shows the relationship between sample characteristics and scattered light from the sample. Meanwhile, Patent Document 2 discloses a method for classifying samples, such as cells, using multivariate analysis. Patent Document 3 discloses a method for identifying cell types using principal component analysis.

[0003] Meanwhile, Patent Document 4 discloses an example of a hyperspectral imaging device that uses compressed sensing. The imaging device includes an encoding element, which is an array of multiple optical filters with different wavelength-dependent light transmittances, an image sensor that detects light transmitted through the encoding element, and a signal processing circuit. The encoding element is disposed on an optical path connecting the subject and the image sensor. The image sensor simultaneously detects light in which components of multiple wavelength bands are superimposed for each pixel to acquire a single wavelength-multiplexed image. The signal processing circuit reconstructs image data for each of the multiple wavelength bands by applying compressed sensing to the acquired wavelength-multiplexed image using information on the spatial distribution of the spectral transmittance of the encoding element. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] WO 01 / 069191 [Patent Document 2] WO 01 / 092859 [Patent Document 3] International Publication No. 2019 / 117177 [Patent Document 4] U.S. Patent No. 9,599,511 Summary of the Invention

[0005] The present disclosure provides a technique for estimating a characteristic of an object, such as the sugar content of an agricultural product.

[0006] A processing device according to one embodiment of the present disclosure is a processing device connected via a network to one or more terminals equipped with a hyperspectral sensor that generates compressed image data in which hyperspectral information of an object is compressed as two-dimensional image information. The processing device includes: a storage device that stores data sets of multiple samples and a first restoration table for restoring the hyperspectral data from the compressed image data, where each data set of a sample includes hyperspectral data of the sample and data indicating characteristic values ​​of the sample; and a processing circuit that generates a statistical model for estimating the characteristic values ​​from the hyperspectral data based on the data sets of the multiple samples, and edits the first restoration table in accordance with the generated statistical model to generate a second restoration table.

[0007] A processing device according to another aspect of the present disclosure is a processing device connected to one or more devices including a hyperspectral sensor that generates compressed image data in which hyperspectral information of an object is compressed as two-dimensional image information. The processing device includes: a storage device that stores data sets of multiple samples and a first restoration table for restoring the hyperspectral data from the compressed image data, where each data set of a sample includes hyperspectral data of the sample and data indicating characteristic values ​​of the sample; and a processing circuit that generates, based on the data sets of the multiple samples, a model for converting the hyperspectral data into spectral data associated with the characteristic values, and generates, based on the model and the first restoration table, a second restoration table for restoring the spectral data associated with the characteristic values ​​from the compressed image data.

[0008] A comprehensive or specific aspect of the present disclosure may be realized by a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable recording disk, or by any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. The computer-readable recording medium may include a volatile recording medium or a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory). An apparatus may be composed of one or more devices. When an apparatus is composed of two or more devices, the two or more devices may be located in a single device or may be located separately in two or more separate devices. In this specification and claims, the term "apparatus" may refer not only to a single device but also to a system consisting of multiple devices.

[0009] According to one aspect of the present disclosure, it is possible to improve the accuracy of estimating characteristics of an object, such as the sugar content of agricultural products.

[0010] According to another aspect of the present disclosure, even if data deviates from a statistical model due to changes in the attributes of the object, such as the variety or growing method of an agricultural product, it becomes possible to quickly respond to changes in the attributes of the object and estimate the characteristics of the object. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing an example of the results of principal component analysis of spectral data of agricultural products. [Figure 2] FIG. 2 shows that the results of principal component analysis of spectral data differ depending on the tomato variety. [Figure 3A] FIG. 3A is a diagram schematically illustrating an example of the configuration of a hyperspectral camera. [Figure 3B] FIG. 3B is a diagram schematically illustrating another configuration example of the hyperspectral camera. [Figure 3C] FIG. 3C is a diagram schematically illustrating yet another configuration example of the hyperspectral camera. [Figure 3D] FIG. 3D is a diagram schematically illustrating yet another configuration example of the hyperspectral camera. [Figure 4A] FIG. 4A is a diagram schematically illustrating an example of a filter array. [Figure 4B] FIG. 4B is a diagram showing an example of the spatial distribution of the light transmittance of each of a plurality of wavelength bands W1, W2, . . . , Wm included in the target wavelength range. [Figure 5] FIG. 5 is a block diagram showing the configuration of the sugar content estimation system according to the first embodiment. [Figure 6A] FIG. 6A is a diagram illustrating an example of learning data stored in the first storage device. [Figure 6B] FIG. 6B is a diagram showing an example of the spectral data transmitted from the sugar content estimation device and stored in the first storage device. [Figure 7] FIG. 7 is a diagram showing an example of a complete restoration table for each sugar content estimation device stored in the second storage device. [Figure 8]FIG. 8 is a diagram showing an example of a reduced restoration table for each sugar content estimation device stored in the third storage device. [Figure 9] FIG. 9 is a diagram illustrating an example of model data stored in the third storage device. [Figure 10A] FIG. 10A is a scatter plot showing an example of the relationship between the sugar content and the principal component score of a specific principal component that has been confirmed to have a correlation with sugar content. [Figure 10B] FIG. 10B shows an example of the correspondence table stored in the third storage device. [Figure 11A] FIG. 11A shows an example of the format of data transmitted by the statistical learning device. [Figure 11B] FIG. 11B shows another example of the format of data transmitted by the statistical learning device. [Figure 12] FIG. 12 is a diagram showing an example of a reduced restoration table stored in the fourth storage device. [Figure 13] FIG. 13 shows an example of data transmitted to the statistical learning device. [Figure 14] FIG. 14 is a diagram showing an overview of communication and processing between the statistical learning device and the sugar content estimating device. [Figure 15] FIG. 15 is a graph showing an example of principal component loadings for each wavelength of principal components that are correlated between sugar content and principal component scores. [Figure 16] FIG. 16 is a flowchart showing a specific example of the operations from step S1001 to step S1007 shown in FIG. [Figure 17] FIG. 17 is a flowchart showing the details of the process for generating the reduced restoration table. [Figure 18] FIG. 18 is a flowchart showing an example of the operation of the statistical learning device in a state where sugar content estimation is possible. [Figure 19] FIG. 19 is a flowchart showing an example of the sugar content estimation operation executed by the sugar content estimation device and the operation of communication with the statistical learning device. [Figure 20A] FIG. 20A is a diagram illustrating an example of information output from an output device. [Figure 20B]FIG. 20B is a diagram showing an example of an error display. [Figure 21] FIG. 21 is a flowchart showing an example of an operation for identifying an object region from a compressed image and extracting a pixel region for which spectral data is to be generated. [Figure 22] FIG. 22 is a flowchart showing a modified example of the operations of steps S2030 to S2080 shown in FIG. [Figure 23] FIG. 23 is a flowchart showing the detailed operation of step S2260 in FIG. [Figure 24] FIG. 24 is a block diagram showing the configuration of a sugar content estimation system according to the second embodiment. [Figure 25A] FIG. 25A is a diagram illustrating an example of model learning data stored in the first storage device. [Figure 25B] FIG. 25B is a diagram illustrating an example of data stored in the first storage device based on spectrum data transmitted from each terminal. [Figure 26A] FIG. 26A is a diagram showing an example of a reduced restoration table for each terminal and each category stored in the third storage device. [Figure 26B] FIG. 26B is a diagram illustrating an example of data of a principal component model stored in the third storage device. [Figure 26C] FIG. 26C is a diagram illustrating an example of data of an estimation model stored in the third storage device. [Figure 26D] FIG. 26B is a diagram illustrating another example of data of the estimation model stored in the third storage device. [Figure 27A] FIG. 27A is a diagram illustrating an example of a format of data transmitted by a statistical learning device. [Figure 27B] FIG. 27B is a diagram showing another example of the format of data transmitted by the statistical learning device. [Figure 28] FIG. 28 is a diagram showing an example of a reduced restoration table for each product type, which is recorded in the fourth storage device in the second embodiment. [Figure 29] FIG. 29 is a diagram illustrating an example of data transmitted to the statistical learning device. [Figure 30] FIG. 30 is a diagram showing an overview of communication and operation between the statistical learning device and sugar content estimating device in the second embodiment. [Figure 31] FIG. 31 is a flowchart showing a specific example of the operation at the time of initial setting of the statistical learning device in the second embodiment. [Figure 32] FIG. 32 is a flowchart showing an example of the operation of updating the model and the restoration table by the statistical learning device according to the second embodiment. [Figure 33] FIG. 33 is a flowchart showing an example of the operation of the sugar content estimation device in the second embodiment. [Figure 34] FIG. 34 is a block diagram showing the configuration of a sugar content estimation system according to the third embodiment. [Figure 35] FIG. 35 is a diagram showing an example of data transmitted by the sugar content estimating device. [Figure 36A] FIG. 36A is a diagram illustrating an example of data transmitted by the statistical learning device. [Figure 36B] FIG. 36B is a diagram showing another example of data transmitted by the statistical learning device. [Figure 37] FIG. 37 is a diagram showing an outline of communication and operations between the server and the terminal in the third embodiment. [Figure 38] FIG. 38 is a flowchart showing an example of the operation of the server in a state where sugar content estimation is possible. [Figure 39] FIG. 39 is a flowchart showing an example of the operation of the terminal according to the third embodiment. [Figure 40] FIG. 40 is a diagram showing an outline of communication and operations between the server and the terminal in the fourth embodiment. [Figure 41] FIG. 41 is a block diagram showing the configuration of a sugar content estimation system according to the fifth embodiment. [Figure 42] FIG. 42 is a diagram showing an outline of communication and operations between the server and the terminal in the fifth embodiment. [Figure 43] FIG. 43 is a flowchart showing an example of the operation of the terminal according to the fifth embodiment. [Figure 44]FIG. 44 is a diagram showing an outline of a system according to the sixth embodiment. [Figure 45] FIG. 45 is a block diagram showing an example of the configuration of a system according to the sixth embodiment. [Figure 46] FIG. 46 is a diagram showing an example of the distribution of characteristic values ​​in an object. [Figure 47] FIG. 47 is a diagram showing an example of data of a spectrum reduction model. [Figure 48] FIG. 48 is a diagram showing an outline of communication and processing between a processing device and an imaging device in the sixth embodiment. [Figure 49] FIG. 49 is a flowchart showing a specific example of the operations from steps S6001 to S6002 shown in FIG. [Figure 50] FIG. 50 is a flowchart showing a specific example of the operations from steps S6003 to S6004 shown in FIG. [Figure 51] FIG. 51 is a flowchart showing a specific example of the operations from steps S7001 to S7002 shown in FIG. [Figure 52] FIG. 52 is a flowchart showing a specific example of the operations in steps S7003 and S7004 shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0012] In this disclosure, all or part of a circuit, unit, device, component, or part, or all or part of a functional block in a block diagram, may be implemented by one or more electronic circuits, including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). An LSI or IC may be integrated on a single chip or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated on a single chip. While the terms LSI and IC are used here, the term may be changed depending on the degree of integration, and may be referred to as a system LSI, a VLSI (very large scale integration), or an ULSI (ultra large scale integration). A field programmable gate array (FPGA), which is programmable after LSI fabrication, or a reconfigurable logic device, which can reconfigure connections within an LSI or set up circuit partitions within an LSI, may also be used for the same purpose.

[0013] Furthermore, all or part of the functions or operations of a circuit, unit, device, component, or section can be implemented by software processing. In this case, the software is recorded on one or more non-transitory recording media such as ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified in the software are executed by the processor and peripheral devices. A system or device may include one or more non-transitory recording media on which software is recorded, a processor, and necessary hardware devices, such as interfaces.

[0014] [Findings that formed the basis of this disclosure] Before describing the embodiments of the present disclosure, the findings on which the present disclosure is based will be described.

[0015] Many objects, such as food, living organisms, soap, and candles, contain various chemicals. For objects that are not opaque, such as metals, and that exhibit even slight internal light scattering, the content or concentration of specific components can be estimated by detecting reflected or transmitted light from the object. Some materials have the property of absorbing light particularly strongly in one or more specific wavelength ranges. By utilizing this property, object properties can be estimated by illuminating the object with light and observing the spectrum of scattered or transmitted light from the object. For example, object properties can be estimated using a regression model that defines the relationship between the absorption or reflectance at each wavelength and the content or concentration of the target substance. Specifically, by illuminating an object with an unknown content or concentration of the target substance with light and extracting the necessary wavelength information from the scattered or transmitted light detected from the object and applying it to the regression model, the content or concentration of the target substance can be estimated. The regression model is generated based on the optical properties of the target substance. Therefore, although there should be little change depending on the object, such as food, that contains the target substance as an ingredient, in reality, the model changes depending on the object being observed. Therefore, methods have been proposed that statistically model the relationship between the spectrum of scattered or transmitted light from an actually observed object and the content or concentration of a target substance from actual data, regardless of the optical properties of the substance. These methods using regression models or statistical models may not fit an observed object that is different from the object observed when the model was generated, and the content or concentration of the target substance may not be accurately estimated. For example, when estimating the sugar content of fruit, the sugar content may be estimated for the variety used to create the statistical model, but may not be accurately estimated for a different variety.

[0016] The inventors performed principal component analysis (PCA) on data from numerous samples representing the spectra of scattered light from fruit surfaces and investigated a method for estimating sugar content based on a model that represents the relationship between the principal component scores of specific principal components obtained and the sugar content of the observed fruit. Figure 1 shows the results of an actual PCA. Figure 1 shows examples of the results of PCA analysis on the spectral data of the surfaces of several varieties of red tomatoes and the spectral data of a peach. The multiple PCs obtained as a result of the PCA analysis are numbered in order of their contribution. In Figure 1, PC1, PC2, PC3, and PC4 represent the first, second, third, and fourth principal components, respectively. The horizontal axis of each graph in Figure 1 represents wavelength, and the vertical axis represents the principal component loadings. In this example, the spectral data for each sample includes light intensity data for multiple wavelengths in the wavelength range from 450 nm to 950 nm. The spectral data for each sample was obtained by photographing the target fruit with a hyperspectral camera.

[0017] As shown in Figure 1, the loading patterns of each principal component for the tomato spectral data are significantly different from those for the peach spectral data. However, a correlation was confirmed between the principal component score of a certain principal component and sugar content in both tomatoes and peaches. A principal component analysis of the spectral data acquired by photographing tomatoes confirmed a correlation between the principal component score of the third principal component and sugar content (Brix value in this example). On the other hand, a principal component analysis of the spectral data acquired by photographing peaches confirmed a correlation between the principal component score of the second principal component and sugar content. However, the loading patterns of the third principal component for tomatoes are significantly different from those of the second principal component for peaches. This indicates that when estimating the amount or concentration of a substance based on its light absorption, the estimation results vary depending on the conditions of the object, such as the transparency of the peel or the state of the substance contained in the fruit. Thus, it is difficult to estimate properties such as sugar content based solely on the optical properties of the substance.

[0018] Furthermore, the inventors discovered that even for the same crop, the results of principal component analysis can vary significantly depending on the variety. Figure 2 illustrates how the results of principal component analysis of spectral data differ depending on the tomato variety. The graph in Figure 2 shows examples of principal component loadings for the first through fourth principal components obtained by principal component analysis for four varieties of tomatoes. In the example in Figure 2, principal component analysis was performed on the spectral data of multiple samples of red, yellow, purple, and green tomatoes. It was confirmed that, although there were differences in contribution rates, principal components with similar loading patterns could be obtained even for different varieties. However, during the principal component extraction process, the signs of the principal component loadings can sometimes be reversed depending on the variety. For tomato spectral data, the signs of the loadings of principal components correlated with sugar content reversed depending on the combination of varieties used in the analysis, which indicated the need to modify the model used to estimate sugar content from principal component scores.

[0019] New varieties of agricultural products are being actively developed, and new varieties that did not exist when the model was created may be released into the market. In such cases, methods based on conventional models cannot accurately estimate the sugar content of agricultural products. This problem is not limited to sugar content, but can also occur with any characteristic value that depends on the content or concentration of a specific substance contained in the target object.

[0020] Based on the above considerations, the present inventors developed a technology that detects differences between the spectral data of an object and the spectral data of multiple samples used in creating the model when the differences are significant, and prompts users to change the statistical model. This technology was disclosed in Japanese Patent Application No. 2020-095864. Such a technology allows users to quickly learn of the need to change the model, enabling them to quickly create a statistical model that fits a new object.

[0021] In this technology, hyperspectral data of an object is acquired by a device capable of acquiring luminance information in more wavelength bands than a typical camera that acquires luminance information in the three primary colors of red (R), green (G), and blue (B). In this specification, data containing luminance information in at least four wavelength bands is referred to as "hyperspectral data." Hyperspectral data can be generated not only by a hyperspectral camera, but also by a device capable of acquiring spectral information of a single point on an object. In this specification, a device capable of generating hyperspectral data, or a device that generates data such as an image for generating hyperspectral data, is referred to as a "hyperspectral sensor."

[0022] Hyperspectral data can be generated using, for example, the compressed sensing technology disclosed in Patent Document 4. This technology uses an imaging device that generates hyperspectral images. The imaging device includes an image sensor and a filter array called an encoding element, which is arranged on the optical path of light incident on the image sensor. The transmission spectra of the multiple filters in the filter array vary depending on the filter. An imaging device equipped with such a filter array generates two-dimensional image data in which hyperspectral information is compressed. Hyperspectral data can be generated for each pixel by performing an operation using the generated two-dimensional image data and matrix-format data (hereinafter also referred to as a "restoration table") that reflects the spatial distribution of the transmission spectra of the filter array.

[0023] Hereinafter, with reference to FIGS. 3A to 4B, a configuration example of an imaging device (hereinafter also referred to as a "hyperspectral camera") that generates hyperspectral data using compressed sensing technology will be described.

[0024] 3A is a diagram schematically illustrating an example configuration of a hyperspectral camera 500. The hyperspectral camera 500 includes an image sensor 502, a filter array 504, and an optical system 506. The filter array 504 has a structure and function similar to that of the "encoding element" disclosed in Patent Document 4. The optical system 506 and the filter array 504 are disposed on the optical path of light incident from the object 70. The filter array 504 is disposed between the optical system 506 and the image sensor 502.

[0025] FIG. 3A illustrates an apple as an example of the object 70. The object 70 is not limited to an apple, and may be other agricultural products. The image sensor 502 generates data for a compressed image 510 in which information of a plurality of wavelength bands is compressed as a two-dimensional monochrome image. The processing circuit 530 generates image data for each of a plurality of wavelength bands included in a predetermined target wavelength range based on the data of the compressed image 510 generated by the image sensor 502. This generated image data of a plurality of wavelength bands is referred to as "hyperspectral image data." Here, the number of wavelength bands included in the target wavelength range is m (m is an integer equal to or greater than 4). In the following description, the generated image data of a plurality of wavelength bands is referred to as images 520W1, 520W2, ..., 520W m These are collectively referred to as a hyperspectral image 520. In this specification, a signal representing an image, i.e., a set of signals representing the pixel values ​​of each pixel, may be simply referred to as an "image." The multiple wavelength bands may be W1, W2, ..., Wm. Wavelength band W1 corresponds to image 520W1, wavelength band W2 corresponds to image 520W2, ..., wavelength band Wm corresponds to image 520W1. m may correspond.

[0026] The filter array 504 is an array of multiple light-transmitting filters arranged in rows and columns. The multiple filters include multiple types of filters with different spectral transmittances, i.e., wavelength-dependence of light transmittance. The filter array 504 modulates the intensity of incident light for each wavelength and outputs the modulated light. This process performed by the filter array 504 is referred to as "encoding" in this specification.

[0027] 3A, filter array 504 is disposed near or directly above image sensor 502. Here, "near" means close enough that a reasonably clear image of light from optical system 506 is formed on the surface of filter array 504. "Directly above" means that the two are so close that there is almost no gap between them. Filter array 504 and image sensor 502 may be integrated.

[0028] Optical system 506 includes at least one lens. Although optical system 506 is shown as a single lens in Figure 3A, optical system 506 may be a combination of multiple lenses. Optical system 506 forms an image on the imaging surface of image sensor 502 through filter array 504.

[0029] The filter array 504 may be located away from the image sensor 502. Figures 3B to 3D are diagrams showing configuration examples of a hyperspectral camera 500 in which the filter array 504 is located away from the image sensor 502. In the example of Figure 3B, the filter array 504 is located between the optical system 506 and the image sensor 502 and at a position away from the image sensor 502. In the example of Figure 3C, the filter array 504 is located between the object 70 and the optical system 506. In the example of Figure 3D, the hyperspectral camera 500 includes two optical systems 506A and 506B, with the filter array 504 located between them. As in these examples, an optical system including one or more lenses may be located between the filter array 504 and the image sensor 502.

[0030] The image sensor 502 is a monochrome photodetector having a plurality of photodetecting elements (also referred to as "pixels" in this specification) arranged two-dimensionally. The image sensor 502 may be, for example, a charge-coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or an infrared array sensor. The photodetecting elements include, for example, photodiodes. The image sensor 502 does not necessarily have to be a monochrome sensor. For example, a color sensor having R / G / B, R / G / B / IR, or R / G / B / W filters may be used. Using a color sensor can increase the amount of wavelength-related information and improve the accuracy of reconstructing the hyperspectral image 520. The wavelength range to be acquired may be determined arbitrarily and may be not limited to the visible wavelength range but may also be ultraviolet, near-infrared, mid-infrared, far-infrared, or other wavelength ranges.

[0031] The processing circuit 530 is an integrated circuit including a processor such as a CPU. The processing circuit 530 generates a plurality of images 520W1, 520W2, . . . 520W3, each including information of a plurality of wavelength bands, based on the compressed image 510 acquired by the image sensor 502 and a restoration table created in advance. m , i.e., hyperspectral image 520 data is generated.

[0032] 4A is a diagram schematically illustrating an example of a filter array 504. The filter array 504 has a plurality of regions arranged two-dimensionally. An optical filter having an individually set spectral transmittance is arranged in each region. The spectral transmittance is expressed by a function T(λ), where λ is the wavelength of incident light. The spectral transmittance T(λ) can take a value between 0 and 1.

[0033] 4A, filter array 504 has 48 rectangular regions arranged in 6 rows and 8 columns. This is merely an example, and in actual applications, more regions may be provided. The number of regions may be approximately the same as the number of pixels in image sensor 502, for example. The number of filters included in filter array 504 is determined depending on the application and may range from several tens to several tens of millions, for example.

[0034] FIG. 4B shows multiple wavelength bands W1, W2, . . . , W included in the target wavelength range. m 4B is a diagram showing an example of the spatial distribution of light transmittance for each wavelength band. In the example shown in FIG. 4B, the difference in shading in each region represents the difference in transmittance. The lighter the region, the higher the transmittance, and the darker the region, the lower the transmittance. As shown in FIG. 4B, the spatial distribution of light transmittance differs depending on the wavelength band.

[0035] In the example shown in FIG. 4B, a grayscale transmittance distribution is assumed, in which the transmittance of each region can take any value between 0 and 1. However, a grayscale transmittance distribution is not necessarily required. For example, a binary scale transmittance distribution may be adopted, in which the transmittance of each region can take a value of either approximately 0 or approximately 1. In a binary scale transmittance distribution, each region transmits most of the light in at least two wavelength ranges among the multiple wavelength ranges included in the target wavelength range, and does not transmit most of the light in the remaining wavelength ranges. Here, "most of the region" means approximately 80% or more. A portion of all cells, for example, half of the cells, may be replaced with transparent regions.

[0036] The processing circuit 530 reconstructs a hyperspectral image 520 containing information on multiple wavelength bands based on the compressed image 510 output from the image sensor 502 and a reconstruction table indicating the spatial distribution characteristics of the transmittance for each wavelength band of the filter array 504. A method for reconstructing the hyperspectral image 520 will be described below.

[0037] The data to be obtained is the data of the hyperspectral image 520, and this data is referred to as hyperspectral data f. If the number of wavelength bands is m, the hyperspectral data f is composed of image data f1, f2, . . . , f m The image 520W1 of the first wavelength band corresponds to image data f1, the image 520W2 of the second wavelength band corresponds to image data f2, and so on. The image 520W m is the image data f m Here, as shown in FIG. 3A, the horizontal direction of the image is the x direction, and the vertical direction of the image is the y direction. The number of pixels in the x direction of the image data to be obtained is n x Let the number of pixels in the y direction be n y Then, the image data f1, f2, . . ., f m Each of the x ×n y It is two-dimensional data of pixels. Therefore, data f has n elements. x ×n y ×m three-dimensional data. This three-dimensional data is sometimes called "hyperspectral image data" or "hyperspectral data cube." On the other hand, the number of elements of data g of compressed image 510 obtained by encoding and multiplexing by filter array 504 is n x ×n y The data g can be expressed by the following equation (1).

[0038]

number

[0039] where f1, f2, . . . , f m Each of the x ×n y Therefore, the vector on the right side is strictly n x ×n y ×m rows and 1 column. Vector g is a one-dimensional vector of n x ×n yThe matrix H is calculated by converting it into a one-dimensional vector with one row and one column. The matrix H is calculated by converting each component f1, f2, . . . , f m is encoded and intensity-modulated with different encoding information (also called "mask information") for each wavelength band, and then added together. x ×n y row n x ×n y ×m columns matrix. The data representing this matrix H corresponds to the aforementioned "restoration table."

[0040] If vector g and matrix H are given, it seems possible to calculate data f by solving the inverse problem of equation (1). However, the number of elements n of the data f to be calculated is x ×n y ×m is the number of elements of the acquired data g x ×n y Since the number of pixels is larger than the number of pixels in the image, this problem is an ill-posed problem and cannot be solved as is. Therefore, the processing circuit 530 utilizes the image redundancy contained in the data f to find a solution using a compressed sensing technique. Specifically, the desired data f is estimated by solving the following equation (2).

[0041]

number

[0042] Here, f' represents the estimated data for f. The first term in the parentheses in the above equation represents the amount of deviation between the estimated result Hf and the acquired data g, the so-called residual term. Here, the sum of squares is used as the residual term, but the absolute value or the square root of the sum of squares, etc., may also be used as the residual term. The second term in the parentheses is a regularization term or stabilization term. Equation (2) means to find f that minimizes the sum of the first and second terms. The processing circuit 530 can converge the solution through recursive iterative calculations and calculate the final solution f'.

[0043] The first term in the parentheses in equation (2) represents the sum of squares of the difference between the acquired data g and Hf, which is the result of transforming the estimation process f by the matrix H. The second term, Φ(f), is a constraint for regularizing f and is a function that reflects the sparsity information of the estimation data. This function has the effect of smoothing or stabilizing the estimation data. The regularization term can be expressed, for example, by the discrete cosine transform (DCT) of f, the wavelet transform, the Fourier transform, or the total variation (TV) of f. For example, using the total variation transform can obtain stable estimation data that suppresses the influence of noise in the observation data g. The sparsity of the object 70 in the space of each regularization term varies depending on the texture of the object 70. A regularization term that makes the texture of the object 70 sparser in the regularization term space can be selected. Alternatively, multiple regularization terms can be included in the calculation. τ is a weighting coefficient. The larger the weighting coefficient τ, the greater the amount of redundant data reduction and the higher the compression rate. The smaller the weighting factor τ, the weaker the convergence to a solution. The weighting factor τ is set to an appropriate value that allows f to converge to a certain extent but does not result in over-compression.

[0044] In the configurations of FIGS. 3B and 3C , the image encoded by the filter array 504 is acquired in a blurred state on the imaging plane of the image sensor 502. Therefore, by storing this blur information in advance and reflecting the blur information in the aforementioned matrix H, the hyperspectral image 520 can be reconstructed. Here, the blur information is expressed by a point spread function (PSF). The PSF is a function that defines the degree of spread of a point image to surrounding pixels. For example, if a point image corresponding to a pixel on an image spreads due to blurring to a k×k pixel region around the pixel, the PSF can be defined as a group of coefficients, i.e., a matrix, that indicates the effect on the brightness of each pixel within that region. The hyperspectral image 520 can be reconstructed by reflecting the effect of blurring of the encoding pattern by the PSF in the matrix H. The filter array 504 may be positioned at any position, but a position that does not cause the encoding pattern of the filter array 504 to be lost due to excessive diffusion can be selected.

[0045] Through the above processing, a hyperspectral image 520 can be constructed from the compressed image 510 acquired by the image sensor 502. The processing circuit 530 may extract an area including an image of the object 70 from the image of each wavelength band in the hyperspectral image 520, and determine one representative value for each wavelength band by processing such as averaging the values ​​of multiple pixels within that area. Data in which the representative values ​​determined in this manner are arranged in order of wavelength band may be used as the above-mentioned hyperspectral data.

[0046] The hyperspectral data generated by the above method includes information on the light intensity or brightness values ​​of each of multiple wavelength bands included in a predetermined target wavelength range. Each of the multiple wavelength bands has a relatively narrow bandwidth. Each wavelength band may have a width of, for example, approximately 1 nm to 20 nm. As an example, if the target wavelength range ranges from 451 nm to 950 nm and each wavelength band has a width of 5 nm, the target wavelength range includes 100 wavelength bands. Calculating the brightness values ​​of each of such a large number of wavelength bands using the above calculations requires a large amount of calculation, and generating the hyperspectral data may take a long time.

[0047] On the other hand, when estimating a characteristic value such as sugar content of an object, information of all wavelength bands included in the target wavelength range is not equally important, and the important wavelength bands may vary depending on the optical characteristics of the object. The inventors conceived the idea of ​​compressing the size of the restoration table and reducing the amount of calculation required for restoration calculation by aggregating information of several wavelength bands of relatively low importance among the wavelength bands included in the target wavelength range, and came up with the configuration of the embodiment of the present disclosure described below. An overview of the embodiment of the present disclosure will be described below.

[0048] A processing device according to an exemplary embodiment of the present disclosure is connected via a network to one or more terminals equipped with a hyperspectral sensor that generates compressed image data in which hyperspectral information of an object is compressed as two-dimensional image information. The processing device includes a storage device and a processing circuit. The storage device stores a dataset of multiple samples and a first restoration table for restoring the hyperspectral data from the compressed image data. Each sample dataset includes hyperspectral data of the sample and data indicating characteristic values ​​of the sample. The processing circuit generates a statistical model for estimating the characteristic values ​​from the hyperspectral data based on the dataset of the multiple samples, and edits the first restoration table in accordance with the generated statistical model to generate a second restoration table.

[0049] The hyperspectral data may include luminance information for each of a plurality of wavelength bands included in a target wavelength range. The first restoration table may be data for restoring the luminance information for each of the plurality of wavelength bands from the compressed image data. In this specification, the first restoration table may be referred to as a "full restoration table." The second restoration table may be data for restoring the luminance information for each of a plurality of new wavelength bands obtained by integrating parts of the plurality of wavelength bands from the compressed image data. In this case, the data size of the second restoration table is smaller than the data size of the first restoration table. In this specification, the second restoration table may be referred to as a "reduced restoration table."

[0050] According to the above configuration, a smaller second restoration table can be generated from the first restoration table in accordance with a statistical model generated based on a data set of multiple samples. The second restoration table stores detailed information on wavelength bands that are important for estimating characteristic values ​​of an object, and aggregates and compresses information on several wavelength bands that are relatively less important. By using the second restoration table, the amount of calculation required to restore hyperspectral data can be reduced compared to when the first restoration table is used.

[0051] The processing circuit can calculate principal component loadings of specific principal components associated with the characteristic values ​​by principal component analysis based on the hyperspectral data of the multiple samples, and generate the second restoration table from the first restoration table based on the principal component loadings.

[0052] The processing circuit determines principal component loadings of specific principal components associated with the characteristic values ​​by principal component analysis based on the hyperspectral data of the plurality of samples, determines some wavelength bands from the plurality of wavelength bands that have a relatively low contribution to the principal components based on the principal component loadings, and generates the second restoration table by integrating information about the some wavelength bands in the first restoration table.

[0053] The hyperspectral sensor may include a filter array including a plurality of filters each having a different transmission spectrum. The first restoration table may be data reflecting the spatial distribution of the transmission spectra of the filter array. The first restoration table may be, for example, the aforementioned matrix-format data.

[0054] The processing circuit may transmit the statistical model and the second restoration table to the terminal. In this case, the processor included in the terminal can restore hyperspectral image data from the compressed image data generated by the hyperspectral sensor using the transmitted second restoration table. This hyperspectral image data has a smaller data size than hyperspectral image data restored using the first restoration table. The processor of the terminal can generate hyperspectral data by extracting an object region from the generated hyperspectral image data and performing processing such as averaging the values ​​of multiple pixels in the region for each of multiple wavelength bands corresponding to the second restoration table. The processor can estimate characteristic values ​​of the object from the hyperspectral data using the statistical model transmitted from the processing device.

[0055] The processing circuit may generate the statistical model, the first restoration table, and the second restoration table for each predetermined classification. For example, the statistical model, the first restoration table, and the second restoration table may be generated for each classification, such as crop species, crop variety, production area, cultivation method, or fertilizer used. In this case, the processing circuit may acquire data indicating a specific classification for which the estimation process is to be performed from a terminal, and generate the statistical model, the first restoration table, and the second restoration table based on the acquired data.

[0056] The processing circuit may acquire partial compressed image data generated by the terminal by extracting data of a partial region from the compressed image data, generate hyperspectral data corresponding to the partial region from the partial compressed image data using the second restoration table, estimate the characteristic value of the object from the hyperspectral data using the statistical model, and transmit data indicating the estimated characteristic value to the terminal. In this case, the terminal can acquire the estimated result of the characteristic value and output it, such as displaying it on a display device, without generating the hyperspectral data and estimating the characteristic value. Therefore, the performance required of the terminal's processor can be reduced compared to when the terminal generates the hyperspectral data and estimates the characteristic value.

[0057] The processing circuit may acquire the compressed image data from the terminal, generate hyperspectral data from the compressed image data using the second restoration table, estimate the characteristic value of the object from the hyperspectral data using the statistical model, and transmit data indicating the estimated characteristic value to the terminal. In this case, the terminal can transmit the compressed image data directly to the processing device and receive data indicating the characteristic value estimated by the processing device. This further reduces the performance required of the processor of the terminal.

[0058] The processing circuit may acquire hyperspectral data generated by the terminal from the compressed image data based on the second restoration table, estimate the characteristic value of the object from the hyperspectral data based on the statistical model, and transmit data indicating the characteristic value to the terminal. In this case, the terminal generates and transmits the hyperspectral data based on the second restoration table but does not estimate the characteristic value. The terminal can receive the data indicating the characteristic value estimated by the processing device and output the estimation result.

[0059] When the processing circuit determines that the statistical model and the second restoration table need to be changed based on the hyperspectral data generated from the compressed image data based on the second restoration table and the hyperspectral data in the dataset of the plurality of samples, the processing circuit may regenerate the statistical model based on the new dataset of the plurality of samples and update the second restoration table according to the regenerated statistical model. This makes it possible to regenerate a statistical model even for agricultural products of new varieties or cultivated methods that do not fit a previously created statistical model, and to optimize the second restoration table according to the newly generated statistical model.

[0060] A system according to an embodiment of the present disclosure includes a processing device according to any of the above aspects and one or more terminals according to any of the above aspects. When the system includes multiple terminals, the first restoration table and the second restoration table may be generated for each terminal.

[0061] The present disclosure also includes a signal processing method executed by a processor (or a processing circuit) included in the processing device or terminal according to any of the above aspects, and a computer program that defines the signal processing method.

[0062] A processing device according to another embodiment of the present disclosure is connected to one or more devices including a hyperspectral sensor that generates compressed image data in which hyperspectral information of an object is compressed as two-dimensional image information. The processing device includes a memory device and a processing circuit. The memory device stores a data set of multiple samples and a first restoration table for restoring the hyperspectral data from the compressed image data. Each data set of a sample includes hyperspectral data of the sample and data indicating a characteristic value of the sample. The processing circuit generates a model for converting the hyperspectral data into spectral data associated with the characteristic values ​​based on the data set of the multiple samples, and generates a second restoration table for restoring the spectral data associated with the characteristic values ​​from the compressed image data based on the model and the first restoration table.

[0063] According to the above configuration, a second restoration table can be generated to restore spectral data related to characteristic values ​​from compressed image data. The "spectral data related to characteristic values" can be, for example, luminance image data in which values ​​of one or more wavelength bands correlated with the characteristic values ​​are emphasized. The second restoration table is sent to a device (e.g., an imaging device) that generates compressed image data. The device can restore spectral data related to characteristic values ​​from the compressed image data using the second restoration table. For example, the device can generate luminance image data as spectral data, in which pixels with higher characteristic values, such as sugar content, have higher luminance. Generating such spectral data makes it easier for users to identify the distribution of characteristic values ​​in an object.

[0064] The hyperspectral data may include luminance information for each of a plurality of wavelength bands included in a target wavelength range. The first restoration table may be data for restoring the luminance information for each of the plurality of wavelength bands from the compressed image data. The second restoration table may be data for restoring luminance information obtained by weighting and adding the luminance information for each of the plurality of wavelength bands from the compressed image data.

[0065] In the above configuration, the second restoration table may be data for restoring, from the compressed image data, luminance image data in which each pixel has a value obtained by weighting and adding luminance values ​​for each of a plurality of wavelength bands. The weight may be set to a larger value for a wavelength band having a stronger correlation with the characteristic value.

[0066] The data size of the second restoration table may be smaller than the data size of the first restoration table. For example, the processing device can generate a second restoration table with a smaller data size by integrating information about some wavelength bands in the first restoration table. The integrated wavelength bands may be, for example, bands with a weight of zero or close to zero. Such data compression can reduce the load of the restoration process using the second restoration table.

[0067] The processing circuit may calculate weights corresponding to each of a plurality of wavelength bands associated with the characteristic values ​​by machine learning based on the hyperspectral data of the plurality of samples. The second restoration table may be generated from the first restoration table based on the weights. The machine learning may be based on statistical processing such as principal component analysis or independent component analysis as described above, or may be based on a neural network. By using such machine learning, it is possible to appropriately generate weights corresponding to each of a plurality of wavelength bands associated with the characteristic values.

[0068] The hyperspectral data may include luminance information for each of a plurality of wavelength bands included in the target wavelength range. The first restoration table may be data for restoring the luminance information for each of the plurality of wavelength bands from the compressed image data. The processing circuit may determine a weight corresponding to each of the plurality of wavelength bands by machine learning based on the hyperspectral data of the plurality of samples, determine some wavelength bands from the plurality of wavelength bands with relatively low weights based on the weights, and generate the second restoration table by integrating information about the some wavelength bands in the first restoration table. This process allows for efficient generation of a second restoration table with a small data size.

[0069] The hyperspectral sensor may include a filter array including a plurality of filters each having a different transmission spectrum, and the first restoration table may be data reflecting a spatial distribution of the transmission spectra of the filter array.

[0070] The processing circuit may transmit the second restoration table to the device, which can use the second restoration table to generate spectral data associated with the characteristic values ​​from the compressed image data.

[0071] The processing circuit may generate the model, the first restoration table, and the second restoration table for each predetermined classification, thereby making it possible to appropriately generate the model, the first restoration table, and the second restoration table according to the classification (e.g., variety) of the object.

[0072] The processing circuit may acquire partial compressed image data generated by the device by extracting data of a partial region from the compressed image data, generate spectral data corresponding to the partial region from the partial compressed image data using the second restoration table, and transmit the spectral data to the device. In this case, the device can acquire the spectral data from the processing device without generating the spectral data, and output, for example, an image based on the spectral data on a display device. Therefore, the performance required of the processor of the device can be reduced compared to when the device generates the spectral data itself.

[0073] The processing circuit may acquire the compressed image data from the device, generate spectral data associated with the characteristic value from the compressed image data using the second restoration table, and transmit the spectral data to the device. In this case, the device can transmit the compressed image data directly to the processing device and receive the spectral data generated by the processing device. This further reduces the performance required of the processor of the device.

[0074] When the processing circuit determines that the model and the second restoration table need to be changed based on the spectral data generated from the compressed image data based on the second restoration table and the hyperspectral data in the dataset of the plurality of samples, the processing circuit may regenerate the model based on the new dataset of the plurality of samples and update the second restoration table according to the regenerated model. This makes it possible to regenerate a model even for agricultural products of new varieties or cultivated methods that do not fit a previously created model, and to optimize the second restoration table according to the newly generated model.

[0075] A method according to yet another embodiment of the present disclosure is a computer-generated method including: obtaining a first restoration table for restoring hyperspectral data from compressed image data in which hyperspectral information of an object is compressed as two-dimensional image information; obtaining a model for converting the hyperspectral data into spectral data associated with the characteristic values, the model being generated based on a sample dataset including the hyperspectral data of the sample and data indicating characteristic values ​​of the sample; and generating a second restoration table for restoring the spectral data associated with the characteristic values ​​from the compressed image data based on the model and the first restoration table.

[0076] The hyperspectral data may include luminance information for each of a plurality of wavelength bands included in a target wavelength range. The first restoration table may be data for restoring the luminance information for each of the plurality of wavelength bands from the compressed image data. The second restoration table may be data for restoring luminance information obtained by weighting and adding the luminance information for each of the plurality of wavelength bands from the compressed image data.

[0077] The data size of the second restoration table may be smaller than the data size of the first restoration table.

[0078] The second restoration table can be generated by combining information on some of the wavelength bands included in the first restoration table.

[0079] According to yet another embodiment of the present disclosure, a computer program may be stored on a computer-readable non-transitory storage medium, causing a computer to execute the following steps: obtain a first restoration table for restoring hyperspectral data from compressed image data in which hyperspectral information of an object is compressed as two-dimensional image information; obtain a data set of a sample including hyperspectral data of the sample and data indicating characteristic values ​​of the sample; generate a model for converting the hyperspectral data into spectral data associated with the characteristic values ​​based on the data set; and generate a second restoration table for restoring the spectral data associated with the characteristic values ​​from the compressed image data based on the model and the first restoration table.

[0080] Exemplary embodiments of the present disclosure will be described in more detail below. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Among the components in the following embodiments, components that are not recited in the independent claims that represent the highest concepts will be described as optional components. Each figure is a schematic diagram and is not necessarily an exact illustration. Furthermore, in each figure, substantially identical components are assigned the same reference numerals, and duplicated descriptions may be omitted or simplified.

[0081] (Embodiment 1) A first exemplary embodiment of the present disclosure will be described.

[0082] The system according to this embodiment includes a processing device that generates a statistical model and one or more terminals that, based on the statistical model, estimate a characteristic value (e.g., sugar content) that depends on the content or concentration of a substance in the target object from spectral data of scattered or transmitted light from the target object. In the following description, the processing device that generates the statistical model is also referred to as a "statistical learning device," and the terminal that estimates the characteristic value is also referred to as an "estimation device." The statistical learning device and the estimation device are communicatively connected to each other via a network. The estimation device detects scattered or transmitted light from the target object and generates hyperspectral data containing information on light intensity or brightness for each of multiple wavelength bands included in a predetermined target wavelength range. In the following description, this hyperspectral data may be simply referred to as "spectral data." The estimation device acquires spectral data of scattered or transmitted light from the target object using a device equipped with a hyperspectral sensor. The estimation device acquires the statistical model from the statistical learning device and estimates a characteristic value, such as sugar content, of the target object from the spectral data based on the statistical model. The statistical learning device creates the statistical model by performing principal component analysis on multiple data sets stored in a storage device. Each data set includes spectral data and corresponding characteristic value data such as sugar content. Data sets are collected by a user (e.g., an administrator) using the system and stored in a storage device. The estimation device acquires spectral data of the target object and transmits the spectral data to the statistical learning device. The statistical learning device determines whether the statistical model needs to be changed based on the spectral data acquired from the estimation device and the spectral data of numerous samples used to previously create the statistical model. When the statistical learning device detects a new variety or a product produced using a new cultivation method that does not fit into the previously created statistical model, it causes the output device to output information requesting the spectral data and characteristic value data such as sugar content of the product of that variety or cultivation method. Based on this information, the user collects sets of spectral data and characteristic value data for agricultural products of the same variety as the target agricultural product and grown using the same cultivation method, and stores them in a storage medium.The statistical learning device adds the collected dataset to the existing dataset and performs principal component analysis to recreate a model, which it then sends to the estimation device. The estimation device then uses the newly created model to estimate characteristic values ​​such as sugar content from the spectral data. In this way, by successively switching to new models, the estimation device can estimate characteristic values ​​such as sugar content that depend on the content or concentration of component substances, even if the state of the observed object changes.

[0083] The estimation device in this embodiment generates hyperspectral data of an object using compressed sensing technology. More specifically, the estimation device includes a hyperspectral sensor that generates compressed image data representing a two-dimensional image in which hyperspectral information has been compressed. Here, "hyperspectral information" refers to brightness information for each of four or more wavelength bands included in a predetermined target wavelength range. The compressed image data may be acquired, for example, by imaging using a filter array including multiple types of optical filters with different transmission spectra. Information on multiple wavelength bands, for example, four or more, may be superimposed on data for each pixel of the compressed image data. Depending on the application, information on 10 or more or 100 or more wavelength bands may be superimposed on data for each pixel. The compressed image data may include information on multiple wavelength bands, each of which is spatially encoded.

[0084] The estimation device restores the hyperspectral data from the compressed image data using a restoration table pre-stored in a storage device. The restoration table may be data reflecting the spatial distribution of the transmission spectrum of the filter array. The statistical learning device updates the restoration table every time the statistical model is updated and distributes it to the estimation device.

[0085] The restoration tables used in this embodiment include a full restoration table and a reduced restoration table. The full restoration table is a restoration table that can restore all brightness values ​​of multiple narrow bands included in a predetermined target wavelength range from compressed image data. The full restoration table is created in advance for each terminal based on the transmission characteristics of the filter array of each terminal and recorded in the storage device of the statistical learning device. On the other hand, the reduced restoration table is a table in which two or more consecutive narrow bands included in the target wavelength range are integrated into one band, thereby reducing the overall size. The reduced restoration table is generated from the full restoration table by the statistical learning device and distributed to each estimation device. When creating the above statistical model, the statistical learning device identifies several bands included in the target wavelength range that contribute relatively little to the estimation of characteristic values ​​such as sugar content. Then, two or more consecutive bands among these bands are combined and processed as a single band. For example, the statistical learning device may integrate two or more consecutive bands in the full restoration table into a single element by averaging the values ​​of two or more elements corresponding to two or more consecutive bands in the full restoration table. Such processing generates a reduced-reconstruction table that is smaller in size than the full-reconstruction table. After generating the reduced-reconstruction table, the statistical learning device distributes the reduced-reconstruction table to the corresponding estimation device. When the statistical learning device updates the statistical model, it also updates the reduced-reconstruction table in accordance with the change in the statistical model and distributes the updated reduced-reconstruction table to the corresponding estimation device.

[0086] The statistical learning device further converts the full hyperspectral data included in the dataset of each learning sample into reduced hyperspectral data and performs principal component analysis again based on the reduced hyperspectral data of each sample. This creates a new statistical model for estimating characteristic values, such as sugar content, from the reduced hyperspectral data and distributes it to the estimation device. The statistical model includes a principal component model for calculating principal component scores of specific principal components correlated with the characteristic values ​​from the reduced hyperspectral data, and an estimation model for estimating the characteristic values ​​from the principal component scores. The principal component model is data indicating a vector of principal component loadings of specific principal components correlated with the characteristic values. The principal component scores can be calculated by taking the dot product of the principal component loading vector indicated by the principal component model and the reduced hyperspectral data. The estimation model is data specifying the correspondence between principal component scores and characteristic values.

[0087] The estimation device restores hyperspectral data from the compressed image data using the distributed reduced restoration table. The reduced restoration table has a smaller data size than the full restoration table, allowing the hyperspectral data to be restored in a shorter time. The hyperspectral data restored in this manner corresponds to the reduced hyperspectral data described above, and retains information at high density for bands that are important for estimating characteristic values, and retains information at low density for bands that are less important. The estimation device estimates characteristic values ​​of the object by applying the distributed statistical model to the restored reduced hyperspectral data.

[0088] The estimation device transmits the restored reduced hyperspectral data to the statistical learning device. The statistical learning device determines whether the statistical model needs to be changed based on the reduced hyperspectral data acquired from the estimation device and the spectral data of numerous samples used to create the previous statistical model. When the statistical learning device detects a new variety or agricultural product cultivated using a new cultivation method that does not fit into the previously created statistical model, it causes the output device to output information requesting the spectral data and characteristic value data, such as sugar content, of the agricultural product of the new variety or cultivation method. Based on this information, the user collects sets of spectral data and characteristic value data for agricultural products of the same variety and cultivated using the same cultivation method as the agricultural product in question, and stores them in a storage medium. The statistical learning device re-creates the model by adding the collected data set to the existing data set and performing principal component analysis. At this time, the combination of wavelength bands with high importance changes as the model is changed, so the reduced-restoration table is also updated. The statistical learning device transmits the updated model and reduced-restoration table to the estimation device. The estimation device uses the newly created model and reduced-restoration table to estimate characteristic values, such as sugar content, from the compressed image data.

[0089] The above configuration significantly reduces the time required from photographing an object to estimating its characteristic values, improving response. By successively replacing the model and reduced restoration table with new ones, it is possible to estimate characteristic values ​​such as sugar content even if the state of the observed object changes.

[0090] The configuration and operation of this embodiment will be described in more detail below. In the following description, the estimation device is assumed to estimate the sugar content of an object, and may be referred to as a "sugar content estimation device" or a "terminal."

[0091] [1-1.Configuration] 5 is a block diagram showing the configuration of a sugar content estimation system 10 in this embodiment. The sugar content estimation system 10 includes a statistical learning device 100 and multiple sugar content estimation devices 200. The statistical learning device 100 and each sugar content estimation device 200 are connected to each other via a communication network 50.

[0092] The statistical learning device 100 is, for example, a server computer managed by a business operator that operates the sugar content estimation system 10. The statistical learning device 100 includes a first storage device 110, a second storage device 120, a third storage device 130, a processing circuit 140, a communication circuit 170, an input device 180, and an output device 190. The communication circuit 170 includes a transmitter 172 and a receiver 174. Each sugar content estimation device 200 includes a hyperspectral (HS) camera 210, a fourth storage device 220, a restoration processing circuit 230, a fifth storage device 240, an estimation processing circuit 250, a communication circuit 270, an input device 280, and an output device 290. The communication circuit 270 includes a transmitter 272 and a receiver 274. Note that the input device 180 and the output device 190 may be external elements to the statistical learning device 100. Similarly, the input device 280 and the output device 290 may be elements external to the sugar content estimating device 200 .

[0093] [1-1-1. Statistical Learning Device] First, the configuration of statistical learning device 100 will be described.

[0094] Each of the first storage device 110, the second storage device 120, and the third storage device 130 is a device that stores data using any storage medium, such as a magnetic disk, flash memory, or optical disk. The first storage device 110 stores model training data for generating a statistical model used to estimate sugar content from hyperspectral data, and hyperspectral data received from each sugar content estimation device 200. The second storage device 120 stores a full restoration table according to the characteristics of the hyperspectral camera 210 of each sugar content estimation device 200. The third storage device 130 stores a reduced restoration table according to the characteristics of the hyperspectral camera 210 of each sugar content estimation device 200, and a statistical model for estimating sugar content from the hyperspectral data. The statistical model includes a model that generates a value that serves as a basis for estimating sugar content from hyperspectral data, and a model that estimates sugar content from the value generated by the model. In this embodiment, statistical learning device 100 includes three storage devices 110, 120, and 130, but these functions may be integrated into one or two storage devices, or the functions of these storage devices may be distributed among four or more storage devices.

[0095] FIG. 6A is a diagram illustrating an example of training data stored in the first storage device 110. The first storage device 110 stores spectral data of samples acquired by a hyperspectral camera in association with sugar content. The spectral data includes information on the brightness values ​​of multiple wavelength bands included in a predetermined target wavelength range (451 nm to 950 nm in the example of FIG. 6A). Here, the "brightness value" can be the pixel value of a single point that effectively represents the color of an object in an image acquired by photographing the object with a hyperspectral camera, or the average value of pixel values ​​of multiple points that effectively represent the color of the object. "Pixels that effectively represent the color of the object" refer to pixels that capture scattered or transmitted light from the object, such as pixels that do not receive light due to specular reflection. In the example of FIG. 6A, the wavelength range from 451 nm to 950 nm is divided into m (=100) wavelength bands, each 5 nm wide. The width of each wavelength band is not limited to 5 nm and can be set to any value. The target wavelength range is not limited to 451 nm to 950 nm, and can be set to various ranges depending on the application. The target wavelength range can be, for example, the visible light wavelength range of about 400 nm to about 700 nm, the near-infrared wavelength range of about 700 nm to about 2500 nm, or the near-ultraviolet wavelength range of about 10 nm to about 400 nm. For convenience, the term "light" used herein refers not only to visible light but also to non-visible light such as near-ultraviolet light and near-infrared light.

[0096] The sugar content value may be, for example, a Brix value measured by a sugar content meter. Learning data of a plurality of samples as shown in Fig. 6A is collected before the system starts operation and stored in the first storage device 110. In a system that estimates the content or concentration of a substance other than sugar (for example, a nutrient such as protein, lipid, or carotene), a value corresponding to the content or concentration of the substance is recorded in association with the spectral data instead of the sugar content.

[0097] FIG. 6B shows an example of spectral data stored in the first storage device 110 and transmitted from the sugar content estimation device 200. Unlike the model training data, the spectral data contains spectral information but not sugar content information of the target object. Furthermore, unlike the model training data, the widths of the multiple bands contained in the spectral data are not uniform. That is, the spectral data received from the sugar content estimation device 200 contains a mixture of portions with values ​​in relatively wide wavelength bands and portions with values ​​in relatively narrow wavelength bands. This is because the spectral data generated by the sugar content estimation device 200 is restored using a reduced restoration table. When the receiver 174 receives spectral data with a biased bandwidth from the sugar content estimation device 200, the data is recorded in the first storage device 110. Each time the statistical learning device 100 receives spectral data from the sugar content estimation device 200, it sequentially adds and records the spectral data in the first storage device 110.

[0098] FIG. 7 is a diagram showing an example of a complete restoration table stored in the second storage device 120 for each sugar content estimation device 200. The complete restoration table is data for restoring, for all pixels, information on all bands that can be restored from compressed image data acquired by imaging with the hyperspectral camera 210. The complete restoration table shown in FIG. 7 includes a terminal ID for identifying the sugar content estimation device 200 and information on the coefficients for each band of the filter arranged corresponding to each pixel. The bands in the complete restoration table correspond to the bands in the training data shown in FIG. 6A. The band division may be such that variety selection is possible using the brightness value for each wavelength band.

[0099] FIG. 8 is a diagram showing an example of a reduced-restoration table stored in the third storage device 130 for each sugar content estimation device 200. The reduced-restoration table includes information on the terminal ID for identifying the sugar content estimation device 200, the minimum and maximum wavelengths of each band to be restored, and the coefficients of each band recorded for each pixel. The reduced-restoration table differs not only for each sugar content estimation device 200 but also for each statistical model. Unlike the full restoration table stored in the second storage device 120, the bandwidths in the reduced-restoration table are not uniform. The reduced-restoration table restores the information of each band with a bandwidth corresponding to the statistical model. The bandwidths in the reduced-restoration table correspond to the bandwidths in the spectral data from each terminal shown in FIG. 6B. The reduced-restoration table is updated every time the statistical model is updated.

[0100] FIG. 9 is a diagram showing an example of model data stored in the third storage device 130. In this example, a statistical model is generated by principal component analysis. The data recorded includes the number k of principal components extracted by principal component analysis based on a data set of multiple samples, the number l of the principal component for which a significant correlation between the principal component score and sugar content was confirmed, and a loading vector indicating the loading for each wavelength band of the principal component. In FIG. 9, j indicates the loading corresponding to the j-th wavelength band among the m' wavelength bands shown in FIG. 6B.

[0101] FIG. 10A is a scatter plot showing an example of the relationship between sugar content and principal component scores of specific principal components that have been confirmed to be correlated with sugar content, obtained by principal component analysis of hyperspectral image data of a tomato as an example of an object. The straight line in the figure is a regression line obtained from the data points in the figure. Sugar content can be estimated based on this regression line. For example, sugar content can be estimated using the following equation:

[0102] Estimated sugar content = α × principal component score + β The processing circuit 140 determines the slope α and intercept β of the regression line from the principal component scores and sugar content data of each sample, and records the values ​​of α and β in the third storage device 130. The values ​​of α and β are transmitted to the sugar content estimation device 200 by the transmitter 172 and are used by the estimation processing circuit 250 of the sugar content estimation device 200 to estimate the sugar content.

[0103] As described above, the third storage device 130 stores the loading vector shown in Figure 9 and the slope α and intercept β of the regression line shown in Figure 10A. Each component of the loading vector is used as a model parameter for calculating the principal component scores from the spectral data. The slope α and intercept β of the regression line are used as model parameters for estimating the sugar content from the principal component scores.

[0104] Instead of the slope α and intercept β of the regression line, a correspondence table showing the relationship between principal component scores and sugar content may be stored in the third storage device 130. FIG. 10B shows an example of a correspondence table stored in the third storage device 130. This table defines the correspondence relationship between the range of principal component scores and sugar content. Instead of the data on the slope α and intercept β of the regression line, data from such a table may be transmitted from the transmitter 172 to the receiver 274 of the sugar content estimation device 200.

[0105] The processing circuit 140 is an integrated circuit including a processor such as a CPU. The processing circuit 140 statistically processes the learning data stored in the first storage device 110 to generate a statistical model for estimating sugar content from the spectral data. The processing circuit 140 generates and updates the statistical model and generates and updates a reduced restoration table corresponding to the statistical model. In this embodiment, the processing circuit 140 generates the statistical model by performing principal component analysis based on the spectral data of each sample stored in the first storage device 110, i.e., the brightness value data for each wavelength band. In the example of FIG. 6A, the spectral data of each sample is represented as an m-dimensional vector, and the spectral data of all samples (number of samples: n) is represented as an m-row, n-column matrix. The processing circuit 140 determines multiple principal components by performing principal component analysis on the spectral data of n samples and calculates principal component scores for each sample. The processing circuit 140 calculates the correlation coefficient between the calculated principal component scores and sugar content and extracts principal components that show a significant correlation. The processing circuit 140 generates a loading vector representing the principal component loadings for each wavelength band of the extracted principal components. The loading vector is used as a principal component model for calculating the principal component scores. The principal component scores are calculated by the dot product of the m-dimensional vector represented by the spectral data and the loading vector. As shown in FIG. 9, the processing circuit 140 records the number of principal components calculated during the principal component analysis, information indicating the ordinal number of the extracted principal component, and the loading vector data in the third storage device 130. The processing circuit 140 further generates an estimation model that defines the relationship between the principal component scores of the extracted principal components and sugar content. The processing circuit 140 generates an estimation model, for example, data indicating the slope α and intercept β of the regression line shown in FIG. 10A or the data in the table shown in FIG. 10B, and records it in the third storage device 130.

[0106] In this embodiment, when the processing circuit 140 acquires spectral data of an object from a sugar content estimation device 200, it adds the spectral data to the training data already stored in the first storage device 110 and performs principal component analysis to determine whether there is any change in the statistical model. Alternatively, when the spectral data transmitted from the sugar content estimation device 200 reaches a predetermined number sufficient for statistical processing, it performs principal component analysis on the spectral data transmitted from one or more sugar content estimation devices 200 to determine whether there is any difference with the statistical model generated based on the model training data. If the object is a different variety or was produced using a different cultivation method from the existing samples, it may be determined that there is a change in the model. If it determines that there is a change in the model, the processing circuit 140 causes the output device 190 to output information requesting the addition of sample data corresponding to the object. For example, if the output device 190 is a display, the output device 190 displays text or an image prompting the user to add sample data. Based on this output, a user (e.g., an operator of the system 10) collects samples of the same species as the object, acquires spectral data for each sample, and measures the sugar content. The spectral data can be acquired using, for example, a measuring device equivalent to the hyperspectral camera 210 included in the sugar content estimation device 200. The sugar content can be measured using, for example, a commercially available sugar content meter. The user records a set of spectral data and sugar content data of a newly collected sample in the first storage device 110. The processing circuit 140 performs principal component analysis, including the added sample data, as described above, modifies the principal component model for calculating principal component scores and the sugar content estimation model for estimating sugar content, and causes the transmitter 172 to transmit these models. These principal component models and sugar content estimation models are collectively referred to as "statistical models."

[0107] When the processing circuit 140 generates a statistical model, it generates a reduced restoration table for each sugar content estimation device 200 from the full restoration table corresponding to each sugar content estimation device 200 stored in the second storage device 120, according to the statistical model. The generated reduced restoration table is recorded in the third storage device 130. Furthermore, when the processing circuit 140 updates the statistical model, it updates the reduced restoration table recorded for each sugar content estimation device 200 to match the band combination required for the new statistical model. Details of the operations for generating and updating the reduced restoration table will be described later.

[0108] The communication circuit 170 transmits and receives signals to and from a communication circuit 270 of the sugar content estimation device 200 in response to commands from the processing circuit 140. The communication circuit 170 includes a transmitter 172 and a receiver 174.

[0109] The transmitter 172 transmits the parameters of the model for estimating the principal component scores and the parameters of the model for estimating the sugar content from the principal component scores or the data of the correspondence table stored in the third storage device 130. Furthermore, the transmitter 172 transmits the data of the reduced restoration table for each sugar content estimation device 200 stored in the third storage device 130.

[0110] 11A and 11B show examples of formats of data transmitted by transmitter 172. FIG.

[0111] FIG. 11A shows an example of transmitted data when the sugar content estimation device 200 estimates sugar content from principal component scores based on a regression model. In this example, the number of bands, band boundary wavelengths, data on the reduced-restoration table, and data on the statistical model are transmitted. The data on the reduced-restoration table includes data on coefficient sequences set for each pixel and each band. The coefficients are arranged in a predetermined pixel order and band order. The data on the statistical model includes data on principal component loadings for each band and data on regression line parameters, i.e., slope α and intercept β. In the example of FIG. 11A, the number of bands is transmitted in 8 bits, the band boundary wavelengths in 8 bits, each coefficient in the reduced-restoration table in 8 bits, the principal component loadings for each wavelength band in 16 bits, and each parameter, slope α and intercept β, of the regression line in 8 bits.

[0112] Fig. 11B shows an example of transmitted data when the sugar content estimation device 200 estimates sugar content from principal component scores by referring to a correspondence table. In this example, the number of bands, boundary wavelengths, coefficients of the reduced restoration table, and principal component loadings of each band are transmitted in the same format as the example shown in Fig. 11A. Next, data on the correspondence table between principal component scores and sugar content is transmitted. Data on the number of divisions, which indicates how many ranges the correspondence table divides the principal component scores into, is transmitted using 4 bits, and the boundary values ​​of the principal component score ranges are transmitted using 8 bits each, for the number of divisions. Furthermore, sugar content values ​​corresponding to the divided ranges of principal component scores are transmitted using 8 bits each, for the number of divisions.

[0113] The receiver 174 receives the spectral data transmitted by one or more sugar content estimating devices 200. The received spectral data is recorded in the first storage device 110.

[0114] Input device 180 is a device used to instruct processing circuit 140 to perform an operation. Input device 180 may include, for example, a keyboard, a pointing device, a touch panel, a microphone for voice input, or a spatial position sensor for motion input. Using input device 180, a user can record data and provide instructions to processing circuit 140 to start and end operations.

[0115] The output device 190 is a device used to output information requesting the addition of sample data when the processing circuit 140 determines that an update of the statistical model is necessary. The output device 190 may be, for example, a display that displays the request for sample data in text or the like. The output device 190 may also be a speaker that outputs the request for sample data in audio, or a printer that prints text.

[0116] [1-1-2.Sugar content estimation device] Next, the configuration of the sugar content estimation device 200 will be described.

[0117] The hyperspectral camera 210 includes a filter array containing multiple types of optical filters with different wavelength-dependent light transmittances, and an image sensor that captures an image using light transmitted through the filter array. The configuration of the hyperspectral camera 210 is similar to that of the hyperspectral camera 500 shown in any one of FIGS. 3A to 3D. However, the restoration processing circuit 230 in this embodiment restores reduced hyperspectral data from compressed image data using a reduced restoration table instead of a full restoration table. This reduces the amount of calculation compared to when a full restoration table is used.

[0118] FIG. 12 is a diagram showing an example of a reduced restoration table stored in fourth storage device 220. The reduced restoration table is a restoration model created according to the characteristics of the filter array of hyperspectral camera 210. Data indicating the characteristics of the filter array in each terminal is stored as a full restoration table in second storage device 120 of statistical learning device 100. The reduced restoration table is generated from the full restoration table by processing circuit 140 of statistical learning device 100 and transmitted to estimation device 200. The reduced restoration table can restore detailed spectral information only for wavelength bands of high importance identified based on the statistical model generated by statistical learning device 100. In the example of FIG. 12, boundary wavelengths of spectral bands and coefficients for each band for each pixel are stored.

[0119] The restoration processing circuit 230 refers to the reduced restoration table recorded in the fourth storage device 220, performs image processing on the compressed image generated by the hyperspectral camera 210, and restores the hyperspectral data. For example, it extracts a partial area representing an object from the compressed image, and restores a hyperspectral image of each band for the extracted area using the reduced restoration table.

[0120] Fifth storage device 240 stores the statistical models transmitted from statistical learning device 100, i.e., the principal component model for calculating the principal component scores and the estimation model for estimating sugar content from the principal component scores. The models recorded in fifth storage device 240 are the same as the models recorded in third storage device 130 of statistical learning device 100.

[0121] The estimation processing circuit 250 is an integrated circuit including a processor such as a CPU. The estimation processing circuit 250 estimates the sugar content of the object based on the hyperspectral data restored by the restoration processing circuit 230 and the statistical model stored in the fifth storage device 240. Details of the operation of estimating the sugar content will be described later.

[0122] The input device 280 is a device used to instruct the processor of the sugar content estimation device 200 to perform an operation. The input device 280 may include, for example, a keyboard, a pointing device, a touch panel, a microphone for voice input, or a spatial position sensor for input by motion. Using the input device 280, a user can record data and give instructions to start and end operations.

[0123] The output device 290 is a device that outputs information about the sugar content estimated by the estimation processing circuit 250. The output device 290 may include, for example, a display that displays the sugar content in text or the like. The output device 290 may also include a speaker that outputs the sugar content as audio or a printer that prints text. Furthermore, the output device 290 may be a device that transmits data to another device that operates based on the sugar content data. The other device that operates based on the sugar content data may be, for example, a fruit sorting device that sorts fruits (e.g., tomatoes) according to sugar content and stores them in different containers according to sugar content. Alternatively, the other device may be a display device that determines whether or not fruit can be thinned based on sugar content and indicates which fruits can be thinned. The other device may be a robot that determines whether or not fruit can be thinned and thins them. The other device may also be a device that manages the quality of fruit during the distribution process. For example, if fruit with a sugar content lower than the sugar content displayed on the container is detected in the container, the device may issue a warning using an image, light, or sound, or attach a reject label to the container. If fruit with insufficient sugar content is detected in a container, the quality control device may take action such as moving the fruit with insufficient sugar content to a storage location separate from containers containing only fruit with no quality issues.

[0124] Communication circuit 270 transmits and receives data in accordance with commands from estimation processing circuit 250. Communication circuit 270 is connected to communication circuit 170 of statistical learning device 100 via network 50. The connection to network 50 may be wired or wireless. Communication circuit 270 includes a transmitter 272 and a receiver 274.

[0125] Transmitter 272 is a data transmitter connected to a network. It acquires the spectrum data generated by estimation processing circuit 250 and transmits it to statistical learning device 100.

[0126] FIG. 13 shows an example of data transmitted by the transmitter 272 to the statistical learning device 100. The transmitted data shown in FIG. 13 includes a terminal ID for identifying the sugar content estimation device 200, an object ID, and brightness value data for each band restored using the reduced restoration table. The number of bands is set when the statistical learning device 100 generates a statistical model, and is therefore shared by the statistical learning device 100 and the sugar content estimation device 200. In the example of FIG. 13, the number of bands is a fixed value, so spectral data is transmitted consecutively for the number of objects for which the sugar content estimation device 200 estimated the sugar content. Note that the transmitted data may include information indicating the number of bands or the boundary wavelengths of the bands. The object ID may be omitted.

[0127] Receiver 274 is a data receiver connected to network 50. Receiver 274 receives the reduced-restoration table and statistical model data transmitted from statistical learning device 100. The statistical model data includes, for example, data of a principal component model that calculates principal component scores from spectral data and an estimation model that estimates sugar content from the principal component scores. The estimation model data may include, for example, parameters such as the slope α and intercept β of the regression line described above, or data of a table for estimating sugar content.

[0128] In this embodiment, the restoration processing circuit 230 and the estimation processing circuit 250 are separate circuits, but a single processing circuit may perform both the restoration processing and the estimation processing. The functions of the fourth storage device 220 and the fifth storage device 240 may be realized by a single storage device.

[0129] [1-2. Operation] Next, the operation of the sugar content estimation system 10 will be described.

[0130] [1-2-1. Overview of operation of sugar content estimation system 10] The statistical learning device 100 generates data on the reduced-restoration table and statistical model required for the hyperspectral data generation and sugar content estimation performed by the sugar content estimation device 200, and transmits these to the sugar content estimation device 200. The reduced-restoration table is generated from the full-restoration table based on principal component analysis using learning data stored in the first storage device 110. The statistical models generated by the statistical learning device 100 in this embodiment include a principal component model for calculating principal component scores of specific principal components from spectral data, and a sugar content estimation model for estimating sugar content from the principal component model. The principal component model can be defined, for example, by data on principal component loadings such as those shown in FIG. 9. The sugar content estimation model can be defined, for example, by regression line parameters α and β such as those shown in FIG. 10A, or by data in a correspondence table such as that shown in FIG. 10B.

[0131] When estimating the sugar content of an object, the sugar content estimation device 200 acquires compressed image data of the object using the hyperspectral camera 210 and restores hyperspectral data from the compressed image data using the reduced restoration table acquired from the statistical learning device 100. The sugar content estimation device 200 transmits the restored hyperspectral data to the statistical learning device 100. The sugar content estimation device 200 further estimates the sugar content of the object from the hyperspectral data based on the principal component model and sugar content estimation model received from the statistical learning device 100.

[0132] When the statistical learning device 100 acquires the latest spectral data from the sugar content estimation device 200, it updates the statistical model using new sample data as necessary. Along with updating the statistical model, it also updates the reduced-and-restored table corresponding to each sugar content estimation device 200. The statistical learning device 100 transmits data such as parameters or correspondence tables that define the updated statistical model and data of the updated reduced-and-restored table to the sugar content estimation device 200. The sugar content estimation device 200 receives data of the latest statistical model parameters or correspondence tables and data of the latest reduced-and-restored table from the statistical learning device 100, and always estimates the sugar content of the target object based on the latest statistical model and restoration table.

[0133] Fig. 14 is a diagram showing an overview of communication and processing between the statistical learning device 100 and the sugar content estimation device 200. Fig. 14 illustrates the process of generating a statistical model and a restoration table that is executed by the statistical learning device 100 as an initial setting, the process of restoring spectral data and the sugar content estimation process that are executed thereafter by the sugar content estimation device 200, and the process of updating the model and restoration table by the statistical learning device 100.

[0134] The initial setting includes the operations of steps S1001 to S1007 shown in FIG. 14. In step S1001, the processing circuitry 140 of the statistical learning device 100 acquires the spectral data and sugar content data (see FIG. 6A) of the latest model training sample from the first storage device 110. In step S1002, the processing circuitry 140 performs principal component analysis on the acquired spectral data to determine multiple principal components. Principal component analysis may be performed using, for example, a known principal component analysis algorithm. In step S1003, the processing circuitry 140 extracts, from the determined principal components, those principal components that have a significant correlation between the principal component score and sugar content. In step S1004, the processing circuitry 140 identifies wavelength bands whose absolute values ​​of the principal component loadings of the principal components extracted in S1003 are greater than a threshold, i.e., bands that are important to the principal components. For bands other than the identified important bands, the processing circuitry 140 reduces the number of spectral divisions by combining consecutive bands into one band.

[0135] A specific example of the process of reducing the number of spectral divisions will be described with reference to FIG. 15. FIG. 15 is a graph showing the principal component loadings for each wavelength of principal components correlated between the sugar content of tomatoes and their principal component scores, among multiple principal components obtained by principal component analysis based on tomato spectral data. The vertical axis represents the principal component loadings, and the horizontal axis represents the wavelength. For example, wavelengths whose absolute values ​​of the principal component loadings exceed a predetermined threshold (e.g., 0.1) are determined to be important wavelengths for the principal component. In the example of FIG. 15, the wavelength ranges a to b, c to d, and e to f are determined to be important wavelength ranges, and the bands included in these wavelength ranges are maintained as model training data. For example, if the model training data contains spectral data every 5 nm, the bands included in the wavelength ranges a to b, c to d, and e to f are processed with a 5-nm bandwidth, as in the model training data. In contrast, bands with wavelengths shorter than wavelength a are divided into bands every 50 nm, for example. The bands included in the wavelength ranges b to c and d to e are each combined into one band. For wavelength bands longer than wavelength f, the bands are divided into bands every 50 nm, for example.

[0136] In this way, the processing circuitry 140 changes the band division of the spectral data in the training data from a uniformly detailed division to a division in which detailed and coarse ranges are mixed. In the following step S1005, the processing circuitry 140 performs principal component analysis on the spectral data with the changed band division. This determines multiple principal components. Furthermore, in step S1006, the processing circuitry 140 extracts principal components correlated with sugar content from the principal components and generates a principal component model that calculates the principal component scores of the extracted principal components and a sugar content estimation model that estimates sugar content from the principal component scores. In the following step S1007, the processing circuitry 140 generates a reduced restoration table for each sugar content estimation device 200 from the full restoration table corresponding to the hyperspectral camera 210 of each sugar content estimation device 200. The method for generating the reduced restoration table will be described in detail later. The transmitter 172 transmits the generated reduced restoration table and statistical model (i.e., the principal component model and estimation model) to each sugar content estimation device 200. When each sugar content estimation device 200 receives the reduced restoration table and statistical model transmitted from the statistical learning device 100, it records them in the storage device 240. The above operations complete the initial setting of the sugar content estimation device 200. This enables the sugar content estimation device 200 to estimate sugar content.

[0137] Sugar content estimation by the sugar content estimation device 200 includes operations from steps S2001 to S2004. In step S2001, the sugar content estimation device 200 captures an image of an object with the hyperspectral camera 210. In step S2001, the hyperspectral camera 210 of the sugar content estimation device 200 performs hyperspectral imaging of the object in response to a user operation, thereby generating compressed image data. The restoration processing circuit 230 extracts one or more regions in which the object exists based on the generated compressed image data. In the following step S2002, the restoration processing circuit 230 references the reduced restoration table and generates hyperspectral image information for each band for each extracted region. In step S2003, the estimation processing circuit 250 generates hyperspectral data of the object from the hyperspectral image information for each region generated in step S2002. Details of the object extraction process and the process of generating hyperspectral data from the hyperspectral image information will be described later. Furthermore, in step S2004, estimation processing circuit 250 calculates principal component scores according to the generated hyperspectral data of the object and the principal component model, and estimates the sugar content from the obtained principal component scores according to the estimation model. The estimation results can be displayed via output device 290. Transmitter 272 transmits the hyperspectral data generated in step S2003 to statistical learning device 100.

[0138] When the sugar content estimation device 200 transmits the spectral data to the statistical learning device 100, the statistical learning device 100 executes processing related to updating the statistical model (steps S1009 to S1017).

[0139] In step S1009, the receiver 174 of the statistical learning device 100 receives the spectral data transmitted from the sugar content estimation device 200. The processing circuit 140 adds the received spectral data to the model training data. In step S1010, the processing circuit 140 performs principal component analysis on the received spectral data and spectral data obtained by converting the bandwidth of the spectral data in the training data to match the model. In the following step S1011, the processing circuit 140 determines whether the model needs to be updated. Specifically, the processing circuit 140 compares the principal component model obtained by the principal component analysis in step S1010 with the principal component model obtained in step S1005. If the difference between the new and old models exceeds a threshold, the processing circuit 140 determines that the model needs to be updated. The difference between the new and old models can be calculated, for example, by calculating the sum of squares of the difference between the components of the loading vector indicated by the principal component model previously created and the components of the loading vector indicated by the principal component model currently created. If an update is necessary, the statistical learning device 100 may notify the sugar content estimation device 200 of an alert. When the sugar content estimation device 200 receives this notification, it may display a message indicating that the result of sugar content estimation in step S2004 may not be accurate or that the statistical model needs to be changed.

[0140] If the model needs to be updated, the processing circuitry 140 may cause the output device 190 to output information requesting the addition of new training sample data. This notifies the user of the statistical learning device 100 that the model needs to be updated. The user collects as many samples as possible of the same type as the object from which the spectral data was acquired, and acquires spectral data and measures the sugar content of these samples. For example, the user collects new training data for samples equal to or greater than the number of spectral data received and recorded from the sugar content estimation device 200 from the time the previous principal component model was created until it was determined that an update is necessary. The user records the spectral data and sugar content data of each acquired sample in association with each other in the first storage device 110 of the statistical learning device 100.

[0141] Once the addition of the training data is complete, the processing circuit 140 acquires new model training data including the added data and performs principal component analysis in step S1012. The spectral data used in this principal component analysis has narrow, uniform band widths and contains detailed spectral information. In the following step S1013, the processing circuit 140 identifies, from among the multiple principal components obtained by the principal component analysis, those principal components that have a correlation between principal component scores and sugar content. In step S1014, the processing circuit 140 changes the band division configuration of the spectral data, as in step S1004, based on the band-by-band loadings of the principal components identified in step S1013. Furthermore, in step S1015, the processing circuit 140 performs principal component analysis using the spectral data whose band division configuration has been changed in S1014. In the following step S1016, the principal component model, which indicates the loading vectors of the principal components that have a correlation between the principal component scores and sugar content, and the sugar content estimation model, which indicates the parameters or table of a linear equation that indicates the relationship between the principal component scores and sugar content, are updated. Furthermore, in step S1017, the processing circuit 140 recreates and updates the reduced-restoration table in accordance with the band division configuration changed in step S1014. The transmitter 172 transmits the updated reduced-restoration table, principal component model, and estimation model to the sugar content estimation device 200.

[0142] In this manner, in this embodiment, the following operations are repeated. Generation or update of a statistical model by the statistical learning device 100 Update of the restoration table according to the bandwidth setting of the statistical model by the statistical learning device 100 Acquisition of new spectral data and sugar content estimation by the sugar content estimation device 200 Determining whether or not a model needs to be changed due to the addition of new spectral data by the statistical learning device 100 -Addition of new model training data Update of the statistical model and restoration table based on new model learning data by the statistical learning device 100 and distribution to the sugar content estimation device 200 By performing this operation, even if the variety or cultivation method of an object such as a fruit changes, a model corresponding to that object can be quickly generated, and a decrease in the accuracy of sugar content estimation due to changes in the object can be suppressed.

[0143] [1-2-2. Operation of statistical learning device 100] Next, a specific example of the operation of statistical learning device 100 in this embodiment will be described.

[0144] FIG. 16 is a flowchart showing a specific example of the operations from step S1001 to step S1007 shown in FIG. 14. In the initial state, the statistical learning device 100 has not yet generated a principal component model and an estimation model, and the storage device 240 of the sugar content estimation device 200 has not yet stored the principal component model and the estimation model. Therefore, the sugar content estimation device 200 cannot estimate the sugar content of an object such as fruit with high accuracy. Therefore, the statistical learning device 100 generates an initial principal component model and an estimation model and transmits these models to the sugar content estimation device 200. The operations shown in FIG. 16 are started by start input means such as the input device 180. The operations of each step are described below.

[0145] (Step S1110) The processing circuit 140 determines whether a sufficient amount of sample data necessary for learning is stored in the first storage device 110. The sufficient amount necessary for learning is a preset number, such as 10 times the number of spectral bands. If sufficient sample data is stored, the process proceeds to step S1140. If insufficient sample data is stored, the process proceeds to step S1120.

[0146] (Step S1120) If the sample data is insufficient, the processing circuit 140 causes the output device 190 to output a request for additional learning sample data. If the output device 190 includes a display, for example, the output content may be a character string or a warning image. If the output device 190 includes a speaker, the output content may be voice or a warning sound. After step S1120, the process proceeds to step S1130.

[0147] (Step S1130) The processing circuit 140 determines whether sample data has been added. The sample data is collected, for example, by an administrator of the estimation system 10. The administrator operates the input device 180 to issue a recording instruction to the processing circuit 140, whereby the sample data is recorded in the first storage device 110. Once the sample data has been added, the processing circuit 140 returns to step S1110 and determines again whether the amount of sample data is sufficient.

[0148] (Step S1140) If it is determined in step S1110 that there is a sufficient amount of sample data, the processing circuitry 140 performs principal component analysis based on the spectral data of the training sample data stored in the first storage device 110. The processing circuitry 140 determines multiple principal components through the principal component analysis and calculates the loading amount for each wavelength band for each principal component. The processing circuitry 140 calculates the principal component score for each principal component for each sample.

[0149] (Step S1150) The processing circuit 140 calculates the correlation between the principal component score of each principal component for each sample calculated in step S1140 and the sugar content in the learning sample data recorded in the first storage device 110, and determines whether there is a principal component for which the correlation between the principal component score and sugar content is significant. If there is a principal component for which the correlation between the principal component score and sugar content is significant, the process proceeds to step S1160. If there is no principal component for which the correlation between the principal component score and sugar content is significant, the process returns to step S1120 and requests the addition of data.

[0150] (Step S1160) Based on the band-by-band loadings of principal components determined to be correlated with sugar content, the processing circuit 140 determines which bands should retain detailed spectral information and which bands should compress information. Among the principal components identified in step S1150 that have a significant correlation between principal component scores and sugar content, the processing circuit 140 extracts the principal component with the highest correlation. The extracted principal component loadings are then arranged in correspondence with the corresponding spectral bands, and the bands are classified into bands whose absolute values ​​of the loadings exceed a predetermined threshold and bands whose absolute values ​​of the loadings are equal to or less than the threshold. For bands whose absolute values ​​of the loadings exceed the threshold, the data is used with the spectral bandwidth of the model training data intact in order to maintain detailed spectral information.

[0151] (Step S1170) The processing circuit 140 determines the band integration configuration for the bands whose information should be compressed, as identified in step S1160. For bands whose absolute load values ​​are equal to or less than a threshold, the spectral band width is widened to reduce the amount of information. For example, in the example of FIG. 15, if the threshold is set to 0.1, a small bandwidth, such as 5 nm, is maintained for each band included in the wavelength ranges a to b, c to d, and e to f, as in the example of FIG. 6A, and a value is recorded for each bandwidth. In contrast, for wavelengths shorter than a and longer than f, the bandwidth is widened, such as to 50 nm. The wavelength ranges between b and c and between d and e are each integrated into a single band.

[0152] (Step S1180) The processing circuit 140 converts the model training data stored in the first storage device 110 according to the bandwidth configuration determined in step S1170. For bands for which detailed information is determined to be maintained in step S1160, no conversion is performed, and the original data is maintained. For bands determined to be integrated into a single band in step S1170, the average value of the band values ​​for each integrated group is set as the value of the integrated band. For example, in the example of FIG. 15 , the bandwidth for wavelengths shorter than a and longer than f may be set to, for example, 50 nm. The average value of the pixel values ​​of the bands originally included in the integrated band is set as the pixel value of the integrated band with a bandwidth of 50 nm. The wavelength ranges from b to c and from d to e are each processed as a single band. The average value of the pixel values ​​of the multiple bands included in each of these wavelength ranges is set as the pixel value of the new band. As a result, spectral data with different widths is generated, similar to the example of FIG. 6B .

[0153] (Step S1190) The processing circuitry 140 performs principal component analysis on the spectral data converted in step S1180. The processing circuitry 140 determines multiple principal components through the principal component analysis, calculates the loadings for each band for each principal component, and calculates the principal component score for each principal component for each sample.

[0154] (Step S1200) The processing circuit 140 calculates the correlation between the principal component scores of the multiple principal components determined in step S1190 and the sugar content contained in the learning data stored in the first storage device 110, and identifies the principal component with the highest correlation between the principal component score and sugar content.

[0155] (Step S1210) Processing circuitry 140 generates a principal component model from the loading vectors of the principal components identified in step S1200. Processing circuitry 140 outputs the number of principal components determined in step S1190, the numbers of the identified principal components (i.e., the rank of the contribution level among the multiple principal components), and the principal component model to third storage device 130. Third storage device 130 stores the number of principal components, the numbers of the extracted principal components, and the principal component model.

[0156] (Step S1220) The processing circuit 140 generates an estimation model for estimating sugar content from the principal component scores based on the sugar content data stored in the first storage device 110 and the principal component scores of each sample for the principal components identified in step S1200, and transmits the generated estimation model to the third storage device 130. The third storage device 130 stores the estimation model. The estimation model is, for example, a regression model, and the third storage device 130 stores, for example, the parameters of the regression equation shown in FIG. 10A as the regression model. Alternatively, the estimation model may be data in the form of a correspondence table listing sugar contents corresponding to ranges of principal component scores, as shown in FIG. 10B.

[0157] (Step S1230) The processing circuit 140 generates a reduced restoration table corresponding to each sugar content estimation device 200 according to the band division configuration of the spectrum determined in step S1170. Details of the processing in step S1230 will be described later.

[0158] (Step S1240) The processing circuit 140 converts the generated principal component model, estimation model, and reduced restoration table into transmission data and causes the transmitter 172 to transmit the data. For example, data in the format shown in Fig. 11A or 11B is transmitted to each sugar content estimation device 200. Each sugar content estimation device 200 records the received data in a storage device.

[0159] Through the above operations, the sugar content estimation device 200 becomes able to estimate the sugar content of an object using the hyperspectral camera 210.

[0160] Next, the process of generating the reduced restoration table in step S1230 will be described in more detail.

[0161] Fig. 17 is a flowchart showing details of the process of generating a reduced restoration table in step S1230. Step S1230 includes steps S1231 to S1235 shown in Fig. 17. The process of each step will be described below.

[0162] (Step S1231) The processing circuit 140 determines whether or not reduced restoration tables have been generated corresponding to all sugar content estimation devices 200. If reduced restoration tables have been generated corresponding to all sugar content estimation devices 200, the process proceeds to step S1230. If there is a sugar content estimation device 200 for which reduced restoration tables have not been generated corresponding to the device 200, the process proceeds to step S1222.

[0163] (Step S1232) The processing circuit 140 selects one of the sugar content estimation devices that has not yet generated a reduced restoration table.

[0164] (Step S1233) The processing circuit 140 extracts a full restoration table corresponding to the sugar content estimation device 200 selected in step S1232 from the full restoration tables stored in the second storage device 120. Note that the hyperspectral camera 210 in the sugar content estimation device 200 is equipped with filters with different transmission characteristics for each pixel, and the characteristics of the filters corresponding to the same pixel position differ for each camera. Therefore, the restoration table for restoring the compressed image acquired by the hyperspectral camera 210 to a luminance image for each band also differs for each camera. The second storage device 120 stores a full restoration table for each sugar content estimation device 200. The reduced restoration table is generated based on the full restoration table.

[0165] (Step S1234) Processing circuit 140 groups the bands in the full restoration table according to the band division configuration determined in steps S1160 and S1170. In the new band division configuration, bands that maintain the same narrow bandwidth as the bands in the full restoration table are grouped together. In the new band division configuration, if one band includes two or more bands in the full restoration table, those two or more bands are grouped together.

[0166] (Step S1235) Processing circuit 140 adds the coefficients for each band grouped in step S1234. The value obtained by adding the coefficients for each band included in the group is used as the coefficient for the wider band that integrates the groups. Note that instead of adding the coefficients, the coefficients of the bands included in the group may be averaged.

[0167] After the operation in step S1235 is completed, the process returns to step S1231.

[0168] By repeating steps S1231 to S1235, reduced restoration tables corresponding to all sugar content estimation devices 200 can be generated.

[0169] Let H included in equation (1) be the full reconstruction table, the wavelength band obtained by integrating the first wavelength band and the second wavelength band be the first integrated wavelength band, and let us assume that H and Hr are the reduced reconstruction table Hr. Examples of H and Hr are shown below.

[0170] The data g shown in equation (1) is expressed as g=Hr[f 12 f3···f m ] T The image of the first integrated wavelength band is represented as image data f 12 Corresponds to.

[0171] n x ×n y = p, H is n x ×n y row n x ×n y×m matrix, that is, a matrix with p rows and p ×m columns, and Hr is a matrix with p rows and p ×(m-1) columns, and can be expressed as follows.

[0172]

number

[0173]

number

[0174] where b 11 =a 11 +a 1(p+1) , , b pp =a pp +a p(p+1) , or b 11 =(a 11 +a 1(p+1) ) / 2, , b pp =(a pp +a p(p+1) ) / 2. The explanation of step S1235 and step S1234 above may be interpreted with reference to the above example. The coefficients may be interpreted as elements of a matrix.

[0175] 18 is a flowchart showing an example of the operation of the statistical learning device 100 in a state in which the reduced-restoration table, principal component model, and estimation model are stored in the storage devices 220 and 240 of the sugar content estimation device 200, and sugar content estimation using the hyperspectral camera 210 is possible. Below, with reference to FIG. 18, we will explain the operation of the statistical learning device 100 to update the reduced-restoration table, principal component model, and estimation model, and to send these updated tables and models to the sugar content estimation device 200. The operation is started by a start input means such as the input device 180. The operation may also be started automatically based on a predetermined criterion, for example, at regular intervals.

[0176] (Step S1310) The processing circuit 140 determines whether the receiver 160 has received the data transmitted from the sugar content estimation device 200. If the data has been received, the process proceeds to step S1320. If the data has not been received, the process repeats step S1310.

[0177] (Step S1320) The processing circuit 140 outputs the data received in step S1310 to the first storage device 110. The first storage device 110 stores the received data. The received data is spectral data restored using the reduced restoration table. The band division configuration of this spectral data is the same as the band division configuration of the reduced restoration table and principal component model recorded in the third storage device 130. The spectral data does not include sugar content data.

[0178] (Step S1330) The processing circuit 140 converts the spectral information of the model training data recorded in the first storage device 110 to match the band division configuration in the reduced restoration table and statistical model recorded in the third storage device 130. This conversion process is the same as the process in step S1180.

[0179] (Step S1340) The processing circuit 140 performs principal component analysis using the model learning data whose spectral information has been converted in step S1330 and the spectral data transmitted from the sugar content estimation device 200.

[0180] (Step S1350) The processing circuitry 140 determines whether there is a change between the results of the principal component analysis performed in step S1340 and the results of a previously performed principal component analysis. First, the processing circuitry 140 extracts, from the principal components determined by the principal component analysis performed in step S1340, the designated principal component stored in the third storage device 130, i.e., the principal component with the same number as the principal component extracted as the principal component for which the model is to be generated. If the third storage device 130 stores the data shown in FIG. 9, for example, the loading vector of the l-th principal component is extracted. The processing circuitry 140 compares the loading vector of the l-th principal component stored in the third storage device 130 with the loading vector of the l-th principal component among the principal components determined in step S1340 and calculates the difference between them. The difference between these two loading vectors may be, for example, the sum of the squares or absolute values ​​of the differences between the loadings of corresponding components for all components (i.e., wavelength bands). If the difference between the loading vectors calculated in this way exceeds a predetermined value, the processing circuit 140 determines that there is a change in the results of the principal component analysis. If the difference between the new and old loading vectors exceeds the predetermined value, the processing circuit 140 proceeds to step S1360. If the difference between the new and old loading vectors does not exceed the predetermined value, the processing circuit 140 ends the processing.

[0181] (Step S1360) 16, the processing circuit 140 causes the output device 190 to output a request for new training sample data. If the output device 190 includes a display, the output content may be a character string or a warning image. If the output device 190 includes a speaker, the output content may be voice or a warning sound. The new training sample data may be, for example, a set of spectral data and sugar content data corresponding to the spectral data for a fruit variety that was not cultivated at the time the statistical model stored in the third storage device 130 was generated.

[0182] (Step S1370) The processing circuit 140 determines whether a sufficient amount of new model training data has been added to the first storage device 110. When a request for a new data set is output, an administrator of the statistical learning device 100 collects data sets for as many varieties as possible and stores them in the first storage device 110. The administrator performs an operation to record the data sets in the first storage device 110, for example, using the input device 180. When this operation is performed, the processing circuit 140 makes a determination in step S1370. The determination of whether the amount of model training data is sufficient is made, for example, based on whether the difference between the result of principal component analysis performed using the spectral data of all training data, including the new model training data, and the result of the principal component analysis performed in step S1340 is equal to or less than a predetermined value. More specifically, if the difference between the loading vector of the designated principal component calculated in step S1340 and the loading vector of the designated principal component calculated based on all training sample data, including the new training sample data, is equal to or less than a predetermined value, it can be determined that a sufficient amount of new training sample data has been added. If the amount of new training sample data is sufficient, the process proceeds to step S1140. If the amount of new training sample data is insufficient, the process returns to step S1360.

[0183] After this, the same operations as steps S1140 to S1240 shown in Fig. 16 are performed. As a result, the principal component model, estimation model, and reduced-restoration table are updated and transmitted to the sugar content estimation device 200. The sugar content estimation device 200 can use the updated principal component model, estimation model, and reduced-restoration table to more accurately estimate the sugar content of the object.

[0184] [1-2-3. Operation of the sugar content estimation device 200] Next, a specific example of the operation of the sugar content estimation device 200 will be described.

[0185] Figure 19 is a flowchart showing an example of the sugar content estimation operation performed by the sugar content estimation device 200 and communication with the statistical learning device 100. The operation shown in Figure 19 is performed when the fourth storage device 220 stores a reduced restoration table, the fifth storage device 240 stores a principal component model and a sugar content estimation model, and the sugar content of an object can be estimated. In response to instructions from the user, the sugar content estimation device 200 performs the operations of steps S2010 to S2120 shown in Figure 19. The operation of each step shown in Figure 19 will be explained below.

[0186] (Step S2010) Receiver 260 determines whether or not it has received a new model and a reduced restoration table from statistical learning device 100. If receiver 260 has received a new model and a reduced restoration table, it proceeds to step S2020. If receiver 260 has not received a new model and a reduced restoration table, it proceeds to step S2030.

[0187] (Step S2020) The receiver 274 receives the signal output from the statistical learning device 100 and acquires the reduced-and-restored table, the principal component model, and the estimated model. The receiver 274 sends the acquired data of the reduced-and-restored table to the fourth storage device 220 and updates the reduced-and-restored table already recorded in the fourth storage device 220 with the newly received reduced-and-restored table. Furthermore, the receiver 274 sends the acquired data of the principal component model and the estimated model to the fifth storage device 240 and updates the principal component model and the estimated model already recorded in the fifth storage device 240 with the newly received principal component model and the estimated model. After step S2020, the process proceeds to step S2030.

[0188] (Step S2030) The hyperspectral camera 210 captures an image of an object to obtain a compressed image. If a complete restoration process is performed on the compressed image using a complete restoration table, the image data will have spectral data with a fine band structure similar to that of the model training data, as shown in FIG. 6A.

[0189] (Step S2040) The hyperspectral camera 210 outputs the compressed image acquired by shooting to the restoration processing circuit 230. The restoration processing circuit 230 stores the compressed image.

[0190] (Step S2050) The restoration processing circuit 230 extracts from the compressed image a pixel region to be used for sugar content estimation, i.e., one or more small regions within the region in which the object is photographed. The compressed image is a black-and-white image in which the intensity of light transmitted through a filter array, which has different transmission characteristics depending on the filter, is recorded as a pixel value for each pixel. The restoration processing circuit 230 extracts from the compressed image a small region to be used for calculation from a pixel region in which the object is photographed and which does not receive light rays specularly reflected from the object. A specific example of the extraction operation will be described later.

[0191] (Step S2060) The restoration processing circuit 230 extracts, from the reduced restoration table stored in the fourth storage device 220, a set of coefficients corresponding to each region extracted from the compressed image in step S2050.

[0192] (Step S2070) The restoration processing circuit 230 applies the coefficient matrix extracted from the reduced restoration table for each region extracted from the compressed image to restore a band-by-band intensity image for each region, i.e., a hyperspectral image. The restored spectral bands correspond to the bands of each element of the loading vector indicated by the principal component model stored in the fifth storage device 240. The restoration processing is performed by calculations based on the above-mentioned equations (1) and (2).

[0193] (Step S2080) The restoration processing circuit 230 generates spectral data based on the hyperspectral image for each region of the object restored in step S2070. Data corresponding to each wavelength band is acquired for each pixel in the region. Specifically, the restoration processing circuit 230 acquires brightness value data for each wavelength band for multiple pixels in each region, and calculates a representative value (e.g., average or median) of the brightness values ​​of the multiple pixels for each band. For example, the restoration processing circuit 230 may execute the following process. The restoration processing circuit 230 calculates the brightness values ​​of the pixels included in the image 520W1 of the first wavelength band. 11 The brightness value of 11 , ~, pixels included in the image 520W1 of the first wavelength band n1 The brightness value of n1 , ···, image of m-th wavelength band 520W m Contains pixels 1m The brightness value of 1m , ~, mth wavelength band image 520W m Contains pixels nm The brightness value of nm The restoration processing circuit 230 obtains the representative value of the luminance values ​​of the first wavelength band as the luminance value 11 ~Brightness value n1 The representative value of the luminance value of the mth wavelength band is determined based on the luminance value 1m ~Brightness value nm The representative value of the luminance value of the first wavelength band is determined based on the luminance value 11 ~Brightness value n1 The average value of the luminance values ​​of the mth wavelength band is the luminance value 1m ~Brightness value nm Alternatively, the representative values ​​may be average values ​​of the wavelength bands. Data in which the representative values ​​thus determined are arranged in order of wavelength band can be generated as hyperspectral data. After performing the operation of step S2080, the process proceeds to steps S2090 and S2120.

[0194] (Step S2090) The estimation processing circuit 250 applies the principal component model stored in the fifth storage device 240 to the spectral data generated in step S2080 to obtain principal component scores. The principal component scores are obtained by calculating the dot product of the principal component loading vectors shown in Fig. 9 and a vector that is a data string of representative luminance values ​​within each wavelength band indicated by the spectral data.

[0195] (Step S2100) The estimation processing circuit 250 estimates the sugar content of the object from the principal component scores calculated in step S2090, based on the estimation model stored in the fifth storage device 240. As described above, the estimation model is data that defines, for example, the regression equation illustrated in Fig. 10A or the correspondence table illustrated in Fig. 10B.

[0196] (Step S2110) The estimation processing circuit 250 causes the output device 190 to output information indicating the sugar content estimated in step S2100. The output may take any form, such as text output to a display or printer, or audio output from a speaker. Alternatively, the output may take the form of a signal output to a sorting machine, thinning device, or quality control device for agricultural products such as fruits or vegetables. The output may also be a light or sound display or warning.

[0197] FIG. 20A is a diagram showing an example of information output from output device 190 in step S2110. In this example, output device 190 includes a display, and displays an image in which the sugar content value is superimposed on an image of a photographed object. FIG. 20B is a diagram showing an example of an error display output from output device 190. In this example, output device 190 displays an image that highlights, with a frame or the like, objects among the photographed objects whose sugar content could not be accurately estimated. This display allows the user to easily understand the sugar content estimation result or error state.

[0198] (Step S2120) The restoration processing circuit 230 instructs the transmitter 272 to transmit the spectral data of the object generated in step S2080 to the statistical learning device 100. The transmitted data may be, for example, a terminal ID that identifies the sugar content estimation device 200, an object ID, and spectral data for each object, as shown in FIG. 13 . The spectral data for each object may be a data string of representative brightness values ​​for each band of multiple pixels included in an area representing the object. The band division configuration of this spectral data is determined by a reduced restoration table transmitted in advance from the statistical learning device 100. The number of bands is shared by the statistical learning device 100 and the sugar content estimation device 200. The transmitter 272 transmits the spectral data of the object area as a data string in which representative brightness values ​​for a predetermined number of bands are described in a predetermined data amount (e.g., 16 bits).

[0199] The operation of step S2120 may be performed in parallel with the operations of steps S2090 to S2110, or may be performed before or after the operations of steps S2090 to S2110.

[0200] After the operations of steps S2110 and S2120, the sugar content estimation device 200 ends its operation.

[0201] Through the operations of steps S2010 to S2110 and S2120, the sugar content estimation device 200 can not only estimate the sugar content of an object based on the model, but also transmit spectral data of the object to the statistical learning device 100. By transmitting new spectral data from the sugar content estimation device 200 to the statistical learning device 100, the statistical learning device 100 can determine whether to update the model and reduced-restoration table based on the new spectral data. When the statistical learning device 100 updates the model and reduced-restoration table, the sugar content estimation device 200 receives the latest updated model and reduced-restoration table. This allows the sugar content estimation device 200 to always generate hyperspectral data based on the latest reduced-restoration table and estimate sugar content based on the latest model.

[0202] Next, a specific example of the region extraction operation in step S2050 will be described. Fig. 21 is a flowchart showing an example of the operation of identifying a region of an object from a compressed image captured by the hyperspectral camera 210 and extracting a pixel region from which spectral data should be generated. In the example of Fig. 21, step S2050 includes the operations of steps S2051 to S2054. The operation of each step will be described below.

[0203] (Step S2051) The restoration processing circuit 230 performs image processing on the compressed image acquired in step S2030 as a black and white image. The restoration processing circuit 230 performs processing to extract edges from the compressed image.

[0204] (Step S2052) The restoration processing circuit 230 performs a matching process on the extracted edges using the shape of the object as a template, and extracts an area showing the object.

[0205] (Step S2053) The restoration processing circuit 230 excludes from the extracted region, pixels whose pixel values ​​exceed a predetermined value close to the maximum value, as blown-out highlight pixels.

[0206] (Step S2054) The restoration processing circuit 230 extracts a certain number or more of consecutive pixels from the region extracted in step S2052, excluding the pixels excluded in step S2053, as a region for calculating spectral data. The consecutive pixels may be, for example, five or more pixels vertically and horizontally, but are not limited to this. The region of extracted pixels may be multiple.

[0207] In the above example, the restoration processing circuit 230 extracts a region representing the object from the compressed image and restores a hyperspectral image for only that region, but the present disclosure is not limited to such an operation. For example, the restoration processing circuit 230 may perform the processes shown in FIGS. 22 and 23.

[0208] FIG. 22 is a flowchart showing a modified example of the operations of steps S2030 to S2080 shown in FIG. 19. FIG. 23 is a flowchart showing the detailed operation of step S2260 in FIG. 22. In this example, the restoration processing circuit 230 restores a hyperspectral image using a reduced restoration table without extracting regions from the compressed image acquired in step S2030 (step S2250). The restoration processing is performed by calculation based on the above-mentioned equations (1) and (2). As a result, a hyperspectral image containing information about all pixels is restored. The restoration processing circuit 230 extracts regions showing one or more objects from the restored hyperspectral image containing information about all pixels in a manner similar to that of step S2050 (step S2260). For each extracted region, the restoration processing circuit 230 generates hyperspectral data by arranging representative brightness values ​​for each band (step S2080).

[0209] A specific example of the region extraction process in step S2260 will be described with reference to FIG. 23. In this example, the restoration processing circuit 230 first selects one or more images of bands with relatively narrow widths from the hyperspectral image containing information on all pixels restored in step S2250 (step S2261). A narrow band means a band that requires detailed spectral information and is considered to be a band in which an object can be easily detected. The restoration processing circuit 230 generates a composite image by adding the images of the one or more bands selected in step S2261 (step S2262). The estimation processing circuit 250 performs edge extraction on the composite image in step S2263 (step S2263). The edge extraction process is similar to the process performed in step S2051 described above. The restoration processing circuit 230 then matches the extracted edges with a template of the object and extracts one or more regions representing the object (step S2264). The restoration processing circuit 230 removes overexposed pixels from the extracted region (step S2265). Then, a pixel region for generating spectral data is extracted from the region representing the object (step S2266). The processing from steps S2263 to S2266 is the same as the processing from steps S2051 to S2054 described above.

[0210] [1-3. Effects, etc.] As described above, according to this embodiment, the statistical learning device 100 performs principal component analysis using model training data including hyperspectral data and sugar content data for each of multiple samples. Based on the analysis results, wavelength bands with relatively high importance and wavelength bands with relatively low importance for estimating the sugar content of an object are determined. The wavelength band division configuration is optimized by integrating multiple consecutive wavelength bands with relatively low importance. The statistical learning device 100 generates hyperspectral data for each sample with reduced data size according to the optimized wavelength band division configuration. A statistical model is generated by performing principal component analysis using the hyperspectral data. The statistical model includes a principal component model for determining principal component scores of specific principal components correlated with sugar content from the hyperspectral data of the object, and a sugar content estimation model for estimating sugar content from the principal component scores. Meanwhile, the statistical learning device 100 generates a reduced restoration table by reconfiguring a restoration table for restoring a hyperspectral image from a compressed image acquired by the hyperspectral camera 210 of the sugar content estimation device 200 in accordance with the optimized band division configuration. The statistical learning device 100 transmits the generated principal component model, sugar content estimation model, and reduced / restored table to the sugar content estimation device 200. The sugar content estimation device 200 can generate hyperspectral data of the object from the compressed image using the reduced / restored table, and estimate the sugar content from the hyperspectral data using the principal component model and sugar content estimation model.

[0211] Furthermore, the sugar content estimation device 200 transmits the hyperspectral data generated for estimating sugar content to the statistical learning device 100. The statistical learning device 100 determines whether the new hyperspectral data transmitted from the sugar content estimation device 200 deviates from the hyperspectral data of existing samples, in other words, whether a new model needs to be created. If the transmitted hyperspectral data deviates from the hyperspectral data of existing samples and the existing model cannot be applied, the statistical learning device 100 requests additional sample data for training in order to generate a new model. Once a sufficient amount of sample data necessary for re-training has been added, the statistical learning device 100 re-trains based on the added sample data and the existing sample data, and generates a new statistical model and a reduced restoration table according to a band division configuration corresponding to the new statistical model. Through this operation, even if a target whose sugar content cannot be estimated using previous models appears, for example, due to improved breeding or a new cultivation method, the statistical learning device 100 can quickly generate a new model and a reduced restoration table corresponding to the model. When the statistical learning device 100 generates a new model and a reduced-and-restored table, it transmits the data to the sugar content estimation device 200. This allows the estimation device 200 to always generate hyperspectral data based on a new reduced-and-restored table and estimate the sugar content of an object based on a new model. This makes it possible to more accurately estimate the sugar content of an object grown using a new variety or a new cultivation method.

[0212] In this embodiment, a reduced restoration table is used that restores detailed spectral data with a narrow bandwidth for bands that are important in calculating the principal component scores of principal components correlated with sugar content, and restores spectral data with a wide bandwidth for bands that are not important. This reduces the amount of calculation required for the restoration process by the sugar content estimation device 200, enabling sugar content to be estimated in a shorter time.

[0213] In this embodiment, sugar content is estimated as a characteristic that depends on the concentration of a specific substance contained in an object, but the technology of this embodiment may also be applied to a system that estimates a characteristic other than sugar content. For example, the technology of this embodiment may be applied to a system that estimates the concentration or content of a nutrient component other than sugar. This also applies to the following embodiments.

[0214] (Embodiment 2) Next, a second embodiment will be described. In this embodiment, the statistical learning device 100 classifies and stores model training data based on the variety or cultivation method, etc., and creates a principal component model, an estimated model, and a reduced restoration table for each classification. The sugar content estimation device 200 adds classification information to the spectral data of the target object and transmits it. The statistical learning device 100 determines the need for model retraining for each classification. By storing training data for each classification and generating a model and restoration table for each classification, the variance of the spectral data can be reduced, improving the accuracy of principal component score estimation. As a result, the accuracy of sugar content estimation can be improved. Classification can be performed based on factors that have a certain impact on the condition of the crop, such as the crop species, crop variety, production area, cultivation method, or fertilizer used. The following description will focus on differences from the first embodiment. Components that are the same or similar to those in the first embodiment are designated by the same reference numerals, and overlapping descriptions will be omitted.

[0215] [2-1.Configuration] Figure 24 is a block diagram showing the configuration of a sugar content estimation system 20 of this embodiment. The hardware configuration of the sugar content estimation system 20 is almost the same as the hardware configuration of the sugar content estimation system 10 of embodiment 1 shown in Figure 3. This embodiment differs from embodiment 1 in that a user of the sugar content estimation device 200 can input the variety of the object using the input device 280, and a statistical model (i.e., principal component model and estimation model) and reduced restoration table are generated for each variety. Note that instead of or in addition to variety, classifications such as crop species, cultivation area, cultivation method, fertilizer used, or cultivation season may be used.

[0216] FIG. 25A is a diagram showing an example of model training data stored in the first storage device 110 of the statistical learning device 100. FIG. 25B is a diagram showing an example of data stored in the first storage device 110 and based on spectral data transmitted from each terminal. In this embodiment, the processing circuit 140 of the statistical learning device 100 records, in the first storage device 110, model training data and data based on spectral data received from each sugar content estimation device 200 for one or more classifications, such as variety. In the example shown in FIG. 25A, the sugar contents of multiple samples are associated with the spectral data and recorded for each variety. As shown in FIG. 25B, the spectral data received from the sugar content estimation device 200 is also recorded for each variety. Although the bandwidths of the spectral data received from the sugar content estimation device 200 are not uniform, after receiving the spectral data, the processing circuit 140 converts the received spectral data into spectral data according to the same band division configuration as the model training data and records it.

[0217] The third storage device 130 in this embodiment stores a reduced restoration table for each sugar content estimation device 200 and for each category, and stores a principal component model and an estimation model for each category.

[0218] FIG. 26A is a diagram showing an example of a reduced-restoration table for each terminal and for each category, which is recorded in the third storage device 130 in this embodiment. The data in the reduced-restoration table in this example includes a terminal ID for identifying the sugar content estimation device 200 and an ID for specifying the category, such as variety. In this embodiment, a statistical model is generated for each category, so a reduced-restoration table is also generated for each category. For this reason, an ID for specifying the category is assigned to the reduced-restoration table and recorded in addition to the terminal ID. Unlike the full-restoration table stored in the second storage device 120, the bandwidth in the reduced-restoration table is not uniform. The reduced-restoration table restores the information of each band with a bandwidth corresponding to the statistical model. The reduced-restoration table is updated every time the statistical model is updated.

[0219] 26B is a diagram showing an example of principal component model data recorded in the third storage device 130. In this embodiment, a principal component model is generated and recorded for each classification, such as a variety. Therefore, the data indicating the principal component model includes an ID for identifying the classification, such as a variety ID. For each classification, a principal component number and a loading vector are recorded.

[0220] 26C is a diagram showing an example of data of an estimation model stored in the third storage device 130. In this example, parameters of a linear expression that expresses the relationship between principal component scores and sugar content are recorded for each category.

[0221] 26D is a diagram showing another example of data of an estimation model stored in the third storage device 130. In this example, the estimation model is recorded in the form of a table showing the correspondence between the range of principal component scores and sugar content. This table includes an ID for identifying a classification such as a variety, and is recorded for each classification.

[0222] In this embodiment, the processing circuit 140 generates a statistical model and a reduced reconstruction table for each classification and transmits the data to the transmitter 172 .

[0223] Figures 27A and 27B show examples of the format of data transmitted by the transmitter 172. Figure 27A shows an example of transmitted data when the sugar content estimation device 200 estimates sugar content from principal component scores based on a regression model. In this example, a variety ID is added in addition to the data shown in Figure 11A. Figure 27B shows an example of transmitted data when the sugar content estimation device 200 estimates sugar content from principal component scores by referring to a correspondence table. In this example, a variety ID is added in addition to the data shown in Figure 11B. If a classification other than variety is used, an ID identifying that classification is assigned to the transmitted data.

[0224] The sugar content estimation device 200 in this embodiment is configured to allow input of information indicating the classification of the variety or the like of an object. A user can input information indicating the classification of the variety or the like of an object using the input device 280. The estimation processing circuit 250 generates output data including the input data indicating the variety or the like and the compressed image data generated by the hyperspectral camera 210, and causes the transmitter 272 to transmit the output data.

[0225] 28 is a diagram showing an example of a reduced-and-restored table for each product type, which is recorded in the fourth storage device 220 in this embodiment. The data of the reduced-and-restored table stored in the fourth storage device 220 also includes an ID that identifies the product type.

[0226] Fifth storage device 240 stores the principal component models and estimation models generated for each variety, which are transmitted from statistical learning device 100. The contents of the principal component models and estimation models recorded in fifth storage device 240 are similar to the contents of the principal component models and estimation models recorded in third storage device 130 in statistical learning device 100.

[0227] The estimation processing circuit 250 selects a statistical model and an estimation model stored in the fifth storage device 240 according to the designated variety, and estimates the sugar content of the object in the hyperspectral image restored by the restoration processing circuit 230 based on the selected model.

[0228] Transmitter 272 transmits the spectrum data generated by restoration processing circuit 230 together with the variety information to statistical learning device 100.

[0229] FIG. 29 is a diagram showing an example of data transmitted by the transmitter 272 to the statistical learning device 100. The transmitted data shown in FIG. 29 includes a terminal ID for identifying the sugar content estimation device 200, a variety ID for identifying the variety, the number of data acquired for the same object, and brightness value data for each band of the spectral data reconstructed using the reduced reconstruction table. Because the statistical model differs for each variety, the number of bands also differs for each variety. However, by specifying the variety ID, information on the number of bands is shared between the sugar content estimation device 200 and the statistical learning device 100. Therefore, information on the number of bands does not need to be included in the transmitted data.

[0230] [2-2. Operation] The statistical learning device 100 generates a principal component model and an estimation model for each variety, and generates a reduced-restoration table corresponding to the principal component model for each variety for each sugar content estimation device 200. The statistical learning device 100 acquires variety information and the latest spectral data from the sugar content estimation device 200, and updates the principal component model and estimation model for each variety based on new model learning data as needed. As the principal component model is updated, the statistical learning device 100 also updates the reduced-restoration table corresponding to each sugar content estimation device 200. The statistical learning device 100 sequentially transmits the updated principal component model and estimation model data for each variety and the reduced-restoration table to the sugar content estimation device 200. The sugar content estimation device 200 receives the latest principal component model and estimation model data from the statistical learning device 100, and always performs estimation based on the latest statistical model.

[0231] FIG. 30 is a diagram showing an overview of the communication and operation between the statistical learning device 100 and sugar content estimation device 200 in embodiment 2. In this embodiment, initial setup is performed for each variety. That is, steps S1001 to S1007 in FIG. 14 are replaced with steps S1301 to S1307, which perform similar operations for each variety. The statistical learning device 100 generates a principal component model, an estimation model, and a reduction and restoration table for each variety and transmits them to the sugar content estimation device 200. The sugar content estimation device 200 records the principal component model, estimation model, and reduction and restoration table corresponding to each variety, completing the initial setup.

[0232] Once the initial settings for each variety are complete, the sugar content estimation device 200 becomes able to estimate the sugar content of the target object by capturing an image. The sugar content estimation device 200 accepts the user's selection of the variety via the input device 280 (step S2301). Once the variety is selected, the sugar content estimation device 200 captures the image of the target object (step S2001), restores the hyperspectral image (step S2002), generates spectral data (step S2003), and estimates the sugar content (step S2004) in a manner similar to the example shown in FIG. 14 . Here, the hyperspectral image restoration in step S2002 is performed using a reduced restoration table corresponding to the specified variety. The sugar content estimation in step S2003 is performed using a principal component model and estimation model corresponding to the specified variety. The transmitter 272 of the sugar content estimation device 200 transmits data indicating the specified variety to the statistical learning device 100, in addition to the spectral data generated in step S2003.

[0233] When the statistical learning device 100 receives the spectral data and variety data from the sugar content estimation device 200, it adds the received spectral data to the model training data for the specified variety (step S1309). Then, using the training data for that variety, it performs the same processing as steps S1010 to S1017 shown in FIG. 14. This makes it possible to determine whether the model corresponding to the specified variety needs to be updated, and to update the model and reduced-restoration table as necessary. The updated model and reduced-restoration table corresponding to the specified variety are transmitted to the sugar content estimation device 200 and recorded.

[0234] In the configuration of this embodiment, varieties for which sugar content can be estimated may be registered for each sugar content estimation device 200. In this case, the statistical learning device 100 generates a principal component model and an estimation model corresponding to the registered variety, as well as a reduced-restoration table corresponding to the principal component model and corresponding to the sugar content estimation device 200, and transmits these to the sugar content estimation device 200. For varieties not registered with the sugar content estimation device 200, a reduced-restoration table corresponding to the sugar content estimation device 200 is not generated. With this configuration, each sugar content estimation device 200 does not have a principal component model and an estimation model, nor a restoration table, for unregistered varieties, and therefore cannot accurately estimate the sugar content. By acquiring a model for each registered variety and a restoration table corresponding to that model, the sugar content estimation device 200 can estimate the sugar content of the registered variety with high accuracy.

[0235] FIG. 31 is a flowchart showing a specific example of the operation of the statistical learning device 100 during initial setup in the second embodiment. The flowchart shown in FIG. 31 differs from the flowchart shown in FIG. 16 in that steps S1410 and S1420 are added before step S1110. In the example shown in FIG. 31, during initial setup, the statistical learning device 100 generates an initial statistical model and reduced / restored table for each variety and transmits them to the sugar content estimation device 200. In step S1410, the processing circuit 140 determines whether principal component models, estimated models, and reduced / restored tables have been generated for all varieties for which learning data has been created, using the model training data for each variety stored in the first storage device 110. If models and restoration tables have been generated for all varieties, the operation ends. If there are any varieties for which models and restoration tables have not been generated, the process proceeds to step S1420. In step S1420, the processing circuit 140 selects one of the varieties for which model training data is stored in the first storage device 110 and for which a model and restoration table have not yet been generated. Thereafter, the operations of steps S1110 to S1240 are executed for the selected varieties, similar to the example shown in Fig. 16. As a result, a statistical model with an optimized band configuration and a reduced restoration table are generated for each selected variety, and transmitted to the sugar content estimation device 200. In step S1240, the transmitter 172 transmits data in the format shown in Fig. 27A or 27B, for example, to the sugar content estimation device 200. The operations of steps S1410 to S1240 are executed for all varieties.

[0236] FIG. 32 is a flowchart showing an example of the model and restoration table update operation performed by the statistical learning device 100 in embodiment 2. The operation shown in FIG. 32 is the same as the operation shown in FIG. 18 except that steps S1510 and S1550 are added after step S1320 in the operation shown in FIG. 18. In this embodiment, the processing circuit 140 records spectral data received from the sugar content estimation device 200 in the first storage device 110 according to the variety ID. The received data includes spectral data restored using the reduced restoration table corresponding to the variety. The spectral data has the same band division configuration as the reduced restoration table and statistical model for the variety stored in the third storage device 130.

[0237] In step S1510, the processing circuit 140 compares the variety ID recorded in the first storage device 110 with the variety ID received from the sugar content estimation device 200 to determine whether the received data corresponds to an existing variety. If the received variety ID matches the ID of an existing variety, the process proceeds to step S1520. If the received variety ID differs from the ID of an existing variety, the process proceeds to step S1330.

[0238] In step S1550, processing circuitry 140 sets a new variety ID for the new variety detected in step S1510, associates the ID with the received spectral data, and records them in first storage device 110. After step S1550, processing circuitry 140 proceeds to step S1360. Alternatively, processing circuitry 140 may cause output device 190 to output a request for setting a new variety, and acquire information about the new variety set by the operator via input device 180. Input device 180 may be, for example, a keyboard for character input or a microphone for voice input.

[0239] The operations from step S1330 onwards are the same as the operations of the corresponding steps shown in Fig. 18. The series of operations from step S1310 to step S1240 makes it possible to quickly respond to changes in the spectral data for each variety and update the statistical model, estimation model, and reduced-restoration tables corresponding to the statistical model, based on the spectral data and variety data transmitted from the sugar content estimation device 200. Based on the spectral data and new variety information transmitted from the sugar content estimation device 200, it is possible to quickly generate and transmit to the sugar content estimation device 200 a principal component model, estimation model, and reduced-restoration tables corresponding to the principal component model for a new variety.

[0240] Next, a specific example of the operation of the sugar content estimation device 200 in this embodiment will be described.

[0241] FIG. 33 is a flowchart showing an example of the operation of the sugar content estimation device 200 in this embodiment. In this embodiment, steps S2210, S2220, S2230, and S2240 are added to the operation shown in FIG. 19. In step S2010, the receiver 274 receives data including information on the variety, the reduced restoration table, and the statistical model from the statistical learning device 100, and the process proceeds to step S2210. In step S2210, a determination is made as to whether the acquired data is for a new variety. This determination is made based on whether the variety ID included in the received data is included in data already stored in the storage device 220 or 230. If the received data is for a new variety, the process proceeds to step S2220. If the received data is for an existing variety, the process proceeds to step S2020.

[0242] In step S2220, the receiver 274 sends the acquired reduced-and-restored table and variety ID data to the fourth storage device 220. The fourth storage device 220 stores the acquired reduced-and-restored table as a table corresponding to the new variety. Furthermore, the receiver 274 sends the acquired principal component model, estimated model, and variety ID data to the fifth storage device 240. The fifth storage device 240 stores the acquired principal component model and estimated model as models corresponding to the new variety. After step S2220, the process proceeds to step S2230.

[0243] The processing in step S2020 is the same as the example in Figure 19. In step S2020, the receiver 274 sends the acquired reduced-restoration table and variety ID data to the fourth storage device 220, and updates the reduced-restoration table corresponding to the variety ID already recorded in the fourth storage device 220 with the newly received reduced-restoration table. Furthermore, the receiver 274 sends the acquired principal component model, estimation model, and variety ID data to the fifth storage device 240, and updates the principal component model and estimation model corresponding to the variety ID already recorded in the fifth storage device 240 with the newly received principal component model and estimation model. After step S2020, the process proceeds to step S2030.

[0244] In step S2230, processing circuit 250 determines whether or not the operator has input a variety via input device 280. If an input has been made, processing proceeds to step S2240. If an input has not been made, step S2230 is repeated.

[0245] In step S2240, the processing circuitry 250 identifies the variety entered by the operator and records it in memory. The variety may be specified by, for example, the variety name or variety ID. Alternatively, the variety may be specified by selecting one from pre-prepared options. If an existing variety is specified, the processing circuitry 250 sets the variety using the existing variety ID. If a new or unknown variety is specified, the processing circuitry 250 may set the variety as unknown or other.

[0246] The operations from step S2030 to step S2120 are the same as the corresponding operations in the first embodiment shown in FIG.

[0247] [2-3. Effects, etc.] As described above, in this embodiment, the statistical learning device 100 performs principal component analysis using model learning data created for each classification, such as variety, and based on the results, generates a principal component model, sugar content estimation model, and reduced / restored table for each classification, and transmits these to the sugar content estimation device 200. In this way, in addition to the effects of embodiment 1, the sugar content estimation device 200 can more accurately estimate the sugar content of the target object using the reduced / restored table and model created for each classification.

[0248] (Embodiment 3) Next, a third embodiment will be described.

[0249] In the first and second embodiments, the sugar content estimation device 200 generates hyperspectral data of an object and estimates its sugar content. The statistical learning device 100 generates a reduced-restoration table necessary for generating the hyperspectral data and a statistical model necessary for estimating the sugar content, and transmits these to the sugar content estimation device 200. In contrast, the system of this embodiment includes one or more terminals that generate compressed image data of the object and a server that estimates characteristic values, such as the sugar content, of the object based on the compressed image data transmitted from the terminals. The server has the functions of the statistical learning device 100 described above and the functions of the restoration processing circuit 230 and the estimation processing circuit 250 in the sugar content estimation device 200. The terminal does not store the restoration table or statistical model. Instead, the terminal clips out a pixel region of the object from a compressed image acquired by a hyperspectral camera and transmits the clipped compressed image to the server. The server generates hyperspectral data based on the received compressed image and the reduced-restoration table it generated. The server further estimates the sugar content based on the generated hyperspectral data and the statistical model it generated. The server transmits the estimated sugar content data to the terminal. The terminal displays the received sugar content. As a result, the terminal does not need to be equipped with a storage device for storing the restoration table and statistical model, and the calculation load is reduced because restoration processing and estimation processing are not performed. The configuration and operation of this embodiment will be described below, focusing on the differences from embodiments 1 and 2.

[0250] [3-1.Configuration] FIG. 34 is a block diagram showing the configuration of a sugar content estimation system 30 according to the third embodiment. The sugar content estimation system 30 includes a server 300 and one or more terminals 400. The server 300 has the same hardware configuration as the statistical learning device 100 according to the second embodiment. However, the processing circuit 310 in the server 300 has the functions of the restoration processing circuit 230 and the estimation processing circuit 250 in the sugar content estimation device 200 in addition to the functions of the processing circuit 140 according to the second embodiment. The processing circuit 310 may be divided into a circuit for performing statistical learning, a circuit for performing restoration processing of hyperspectral information, and a circuit for performing sugar content estimation processing. The terminal 400, like the sugar content estimation device 200 according to the second embodiment, includes a hyperspectral camera 210, a communication circuit 270, an input device 280, and an output device 290. The terminal 400 does not include components corresponding to the storage devices 220 and 240, the restoration processing circuit 230, and the estimation processing circuit 250 in the sugar content estimation device 200, but instead includes a processing circuit 410. The processing circuit 410 performs image processing to extract regions representing one or more objects from the compressed image data generated by the hyperspectral camera 210 .

[0251] The processing circuit 410 extracts an area containing an object from the compressed image acquired by the hyperspectral camera 210, and cuts out an area, i.e., a group of pixels, suitable for generating spectral data of the object from that area. The processing circuit 410 performs processing such as superimposing the sugar content value for each area received by the receiver 274 onto the compressed image, and synthesizes or generates an image for display.

[0252] The transmitter 272 transmits to the server 300 the variety information input by the operator via the input device 280, one or more compressed images extracted by the processing circuit 410, and information indicating the area. The data transmitted by the transmitter 272 includes a terminal ID for identifying the terminal 400, a variety ID for identifying the variety of the object, an image ID for identifying which of the images was captured by the hyperspectral camera 210, a region ID for identifying the area from which spectral data of the object is to be extracted, information indicating the range of the region, and information on the compressed image corresponding to the region.

[0253] FIG. 35 is a diagram showing an example of data transmitted by the transmitter 272. In the example of FIG. 35, the transmitter 272 transmits information indicating the number of regions included in the image, followed by the terminal ID, product type ID, and image ID. Next, for each region, the transmitter 272 transmits the region ID and the range of the region. For example, when cutting out a rectangular region, the range of the region can be specified by four 8-bit numerical values ​​(total of 32 bits): the upper left coordinate, the upper right coordinate, the lower left coordinate, and the lower right coordinate. Next, the number of pixels in the region is transmitted, and the value of each pixel is transmitted for the number of pixels. Data grouped by region from the region ID to the pixel values ​​is transmitted in succession for the number of regions transmitted previously.

[0254] 36A and 36B are diagrams showing an example of data transmitted from the transmitter 172 to the receiver 274. In the example of FIG. 36A, the transmitted data includes information on the terminal ID, image ID, area ID, and sugar content. There is a one-to-one correspondence between area IDs and sugar content, and they are transmitted consecutively. In the example of FIG. 36B, the terminal ID and image ID are transmitted, followed by the number of cut-out areas contained in the image indicated by the image ID. Next, sugar content is transmitted for the number of areas mentioned above. If there is a regularity in the arrangement within the image, it is possible to identify areas based on the order of the data even without area IDs. When data for multiple images is transmitted, information on the number of areas cut out for each image is transmitted. The order of area arrangement is, for example, based on the top left pixel of the area as the reference, and the order proceeds from the top left to the bottom right starting from the reference pixel position.

[0255] [3-2. Operation] The server 300 generates a principal component model and an estimation model for each variety, and generates a reduced-restoration table corresponding to the principal component model for each variety for each terminal 400. Furthermore, the server 300 acquires variety information and data of the compressed image from which region extraction has been performed from the terminal 400, restores the hyperspectral image, generates hyperspectral data for each region, and estimates the sugar content. The server 300 transmits data indicating the estimated sugar content for each region to the terminal 400. Furthermore, the server 300 updates the principal component model and estimation model for each variety based on new model training data as needed. As the principal component model is updated, the server 300 also updates the reduced-restoration table corresponding to each terminal 400. This allows the server 300 to always estimate sugar content based on the latest model and restoration table.

[0256] Fig. 37 is a diagram showing an overview of the communication and operation between the server 300 and the terminal 400 in the third embodiment. In the example shown in Fig. 37, the operations from steps S1301 to S1307 are the same as the operations of the corresponding steps shown in Fig. 30. After generating a model and a reduced restoration table, the server 300 transmits information on the corresponding variety to the terminal 400. Upon receiving the variety information, the terminal 400 records the variety information in a storage device. This operation is called initial setting.

[0257] After completing the initial settings, the terminal 400 is able to acquire sugar content data of an object by capturing an image. The terminal 400 accepts the user's selection of a variety via the input device 280 (step S2301). Once the variety is selected, the hyperspectral camera 210 in the terminal 400 captures an image of the object and acquires a compressed image (step S2001). The processing circuit 410 extracts the object from the compressed image acquired in step S2001 and crops out an area for generating spectral data (step S4001). The terminal 400 transmits the cropped compressed image and data indicating the variety to the server 300 (step S4002).

[0258] The server 300 receives the extracted compressed image from the terminal 400. The processing circuit 310 restores the hyperspectral image based on the received compressed image and the reduction / restoration table corresponding to the specified variety and the terminal 400. The processing circuit 310 generates hyperspectral data from the obtained hyperspectral image (step S3002). The operations of the subsequent steps S1309 to S1017 are the same as those of the corresponding steps shown in FIG. 30. The processing circuit 310 updates the model and reduction / restoration table as necessary, and estimates the sugar content from the spectral data generated in S3002 based on the latest model (step S3003). The transmitter 172 transmits the estimated sugar content data for each region to the terminal 400. If the variety information has been updated, the transmitter 172 also transmits the latest variety information to the terminal 400.

[0259] In this embodiment, the terminal 400 can always obtain a highly accurate sugar content estimated based on the latest model. By limiting the restoration area, the server 300 reduces the calculation load and can more easily handle a concentration of data from the terminal 400. By limiting the area of ​​the compressed image and transmitting it, the load on the communication path can also be reduced.

[0260] Next, a specific example of the operation of the server 300 in this embodiment will be described.

[0261] The operation of the server 300 at the time of initial setup in this embodiment is basically the same as the operation shown in Fig. 31. However, after step S1230, instead of the operation of step S1240, the transmitter 172 transmits information on the product type corresponding to the generated model and reduced restoration table to the terminal 400.

[0262] Figure 38 is a flowchart showing an example of the operation of the server 300 when the terminal 400 has acquired variety information, the reduced restoration table, the principal component model, and the estimated model are stored in the storage device of the server 300, and sugar content estimation is possible. The operation from steps S1510 to S1230 shown in Figure 38 is the same as the corresponding operation shown in Figure 32. Below, differences from the operation shown in Figure 32 will be explained.

[0263] (Step S3100) Processing circuit 310 determines whether or not data of the compressed image extracted from terminal 400 has been received. If data has been received, processing proceeds to step S3110. If data has not been received, processing circuit 310 executes step S3100 again after a certain period of time.

[0264] (Step S3110) Processing circuit 310 acquires data of a compressed image in which one or more regions have been cut out, transmitted from terminal 400. Processing circuit 310 determines whether spectral data has been generated for all cut-out regions of the acquired compressed image. If spectral data generation has been completed for all regions, processing proceeds to step S1510. If there are any regions for which spectral data has not yet been generated, processing proceeds to step S3120.

[0265] (Step S3120) The processing circuit 310 selects one of the regions for which spectral data generation processing has not yet been performed.

[0266] (Step S3130) The processing circuit 310 selects a reduction / restoration table stored in the third storage device 130 based on information identifying the terminal 400 and information indicating the specified model included in the transmitted data. Then, based on information indicating the range of the cropped area included in the transmitted data, the processing circuit 310 extracts a necessary portion from the selected reduction / restoration table and restores the cropped area of ​​the compressed image. The restoration method is the same as in the first and second embodiments.

[0267] (Step S3140) The processing circuit 310 averages the spectral data of all pixels in the hyperspectral image restored in step S3130 to generate spectral data of the extracted region. Note that instead of calculating the average value of the spectral data of all pixels, the spectral data of the region may be determined by other processing methods, such as calculating the median.

[0268] (Step S3150) The processing circuit 310 records the spectrum data generated in step S3140 in the first storage device 110 in association with the variety information.

[0269] After the operation of step S3150 is completed, the process returns to step S3110. The operations of steps S3100 to S3150 are repeated until spectral data is generated for all the cut-out regions.

[0270] When spectral data has been generated for all the cut-out regions, the processing circuit 310 executes the operations of steps S1510 to S1230. These operations are the same as those shown in FIG.

[0271] (Step S3280) The processing circuit 140 estimates the sugar content for each region from the spectral data generated in step S3140 according to the principal component model and estimation model corresponding to the specified variety.

[0272] (Step S3290) The transmitter 172 transmits the sugar content estimated in step S3280 together with the region ID to the terminal 400. If a new variety is set in step S1550, the variety information may be included in the transmitted data.

[0273] Through the operations of steps S3100 to S3290, server 300 can estimate the sugar content of the object from the cropped compressed image sent from terminal 400, and send the estimated sugar content to terminal 400. Furthermore, it can update the statistical model and estimation model determined for each variety, and the reduced restoration table corresponding to the statistical model, based on the variety information and cropped compressed image received from terminal 400.

[0274] Next, a specific example of the operation of the terminal 400 in this embodiment will be described.

[0275] 39 is a flowchart showing an example of the operation of the terminal 400 in embodiment 3. The operation of each step will be described below.

[0276] (Step S2010) Processing circuit 410 determines whether or not data transmitted from server 300 has been received. If data has been received, the process proceeds to step S4110. If data has not been received, the process proceeds to step S2230.

[0277] The operations from step S2230 to step S2050 are the same as the operations of the corresponding steps shown in FIG.

[0278] (Step S4140) Transmitter 272 transmits data indicating the terminal ID that identifies terminal 400, the product type information input in step S2230, the area cut out in step S2050, and the cut-out compressed image. For example, data in the format shown in Fig. 36A or 36B is transmitted.

[0279] (Step S4110) When data is received from server 300 in step S2010, processing circuit 410 determines whether the received data is data related to a new variety. If the received data is data related to a new variety, the process proceeds to step S4120. If the received data is not data related to a new variety, the process proceeds to step S4130.

[0280] (Step S4120) The processing circuit 410 records the product type information acquired in step S2010 in the storage device. After step S4120, the process proceeds to step S2230.

[0281] (Step S4130) The output device 190 outputs the sugar content information received in step S2010. The output may be, for example, an image in which the sugar content is superimposed in text at a predetermined position on the compressed image acquired in step S2030. The output information is not limited to an image, and may be information such as text or audio.

[0282] [3-3. Effects, etc.] According to this embodiment, the terminal 400 transmits data indicating a region representing an object extracted from a compressed image acquired by photographing the object with the hyperspectral camera 210 to the server. The server 300 restores the hyperspectral image for each region according to a reduced restoration model set for each variety, estimates the sugar content for each region based on the principal component model and the estimation model, and transmits the estimation results to the terminal 400. The terminal 400 extracts only the region representing the object from the compressed image, reduces the image data, and transmits it to the server, simplifying the restoration calculations and enabling faster sugar content estimation with a reduced amount of calculation.

[0283] Furthermore, a restoration table for restoring spectral information is not recorded in the terminal 400, but is recorded only in the server 300. In this way, no device other than the server 300 can estimate sugar content from images acquired by the hyperspectral camera. Therefore, only registered terminals 400 can obtain sugar content estimation results. Because the restoration table is not recorded in the terminal 400, it is difficult to alter images and falsify spectral information with the intent of fraudulently changing the sugar content. Therefore, a system can be realized that makes it difficult to falsify sugar content.

[0284] (Embodiment 4) Next, a fourth embodiment will be described.

[0285] In the third embodiment, the terminal 400 extracts an object region from a compressed image acquired by the hyperspectral camera 210 and transmits the compressed image of the extracted portion of the region to the server 300. The server 300 generates hyperspectral data from the received compressed image of the portion of the region and estimates the sugar content from the generated spectral data. In contrast, in the present embodiment, the terminal 400 transmits the compressed image acquired by the hyperspectral camera 210 to the server 300 without processing it. The server 300 restores a hyperspectral image from the received compressed image and extracts an object region from the restored hyperspectral image. Then, it generates spectral data for each extracted region and estimates the sugar content of the object according to a model. The server 300 transmits the estimated sugar content data together with position information on the image to the terminal 400. This allows the terminal 400 to obtain an estimated sugar content of the object without performing high-load processing. The following mainly describes the differences from the third embodiment.

[0286] The configuration of the sugar content estimation system 40 of this embodiment is the same as that of the third embodiment shown in Fig. 34. However, it differs from the configuration of the third embodiment in that the compressed image data generated by the hyperspectral camera 210 is transmitted to the server 300 without any special processing by the processing circuit 410, and the processing circuit 410 of the server 300 extracts the area of ​​the target object. Therefore, the data transmitted and received between the communication circuits 170 and 270 also differs from that of the third embodiment.

[0287] The transmitter 272 transmits the variety information, the compressed image, and the terminal ID. The receiver 274 receives information indicating the position on the image of each extracted region and sugar content information. When a new variety is set in the server 300, the receiver 274 also receives the variety information. The position on the image of each region can be expressed, for example, as the position of a pixel expressed in a coordinate system with the upper left corner as the origin, the horizontal direction as x, and the vertical direction as y.

[0288] FIG. 40 is a diagram illustrating an overview of communication and operation between the server 300 and the terminal 400 in the fourth embodiment. The operation illustrated in FIG. 40 excludes step S4001 performed by the terminal 400 in FIG. 37 , and instead, steps S4301 and S4302 performed by the server 300 are added before step S3002. Except for these differences, the operation illustrated in FIG. 40 is similar to the operation illustrated in FIG. 37 . The terminal 400 transmits a compressed image acquired by imaging and data indicating a specified variety to the server 300. In step S4301, the processing circuitry 310 of the server 300 restores a hyperspectral image from the compressed image of all pixels received from the terminal 400 using a reduced restoration table corresponding to the specified variety and the terminal 400. In the subsequent step S4302, the processing circuitry 310 extracts a region corresponding to the object from the obtained hyperspectral image. The region corresponding to the object can be extracted, for example, by performing edge extraction on the image with the highest contrast among the restored images of each band and matching the obtained edge with the shape of the object. From the extracted object region, pixels with the upper limit pixel value in any band are excluded, and for example, 25 pixels in 5 rows and 5 columns may be extracted. In step S3002, processing circuitry 310 generates spectral data by determining a representative value from the data of the pixel region extracted in step S4302. The processing from step S3002 onwards is the same as the processing of the corresponding steps shown in Figure 37.

[0289] According to this embodiment, the terminal 400 transmits the compressed image acquired by imaging to the server 300 without any special processing. Therefore, the terminal 400 can always obtain a highly accurate sugar content estimated by the latest model without performing a high-load calculation.

[0290] In this embodiment, a reduced restoration table is generated for each terminal 400 in accordance with the statistical model for each variety and used when estimating sugar content, but converting the bandwidth depending on the variety is not an essential requirement. In other words, restoration may be performed using detailed spectral data across all bands. In this case, the operations of generating and recording the reduced restoration table may be omitted. The operation of recreating the principal component model and estimation model using the spectral data after band conversion may also be omitted.

[0291] (Embodiment 5) Next, a fifth embodiment will be described.

[0292] 41 is a block diagram showing the configuration of a sugar content estimation system 40 according to the fifth embodiment. The configuration of the sugar content estimation system 40 of this embodiment is similar to the configurations of the aforementioned embodiments. However, this embodiment differs from the aforementioned embodiments in that a terminal 401 performs the process of restoring a hyperspectral image from a compressed image and generating hyperspectral data, and a server 301 performs the process of estimating a sugar content from the hyperspectral data based on a statistical model.

[0293] In this embodiment, the server 301 transmits the reduced restoration table to the terminal 401, but does not transmit the statistical model to the terminal 401. The terminal 401 transmits to the server 301 hyperspectral data generated using the reduced restoration table received from the server 301. The server 301 estimates the sugar content by applying the statistical model to the hyperspectral data received from the terminal 401, and transmits the estimation result to the terminal 401. In this way, the terminal 401 can acquire and display the estimation result of the sugar content of the object without performing the estimation process itself.

[0294] FIG. 42 is a diagram illustrating an overview of the communication and operation between the server 301 and the terminal 401 in this embodiment. The initial setting operation from steps S1001 to S1007 performed by the server 301 is similar to the corresponding operation in embodiment 1. However, in this embodiment, after step S1007, the transmitter 172 of the server 301 transmits only the reduced-restoration table to the terminal 401. The terminal 401 stores the received reduced-restoration table in the storage device 220. The restoration processing circuit 230 of the terminal 401 generates a hyperspectral image from the compressed image generated by the hyperspectral camera 210 using the reduced-restoration table transmitted from the server 301, and generates hyperspectral data of the region of the object in the hyperspectral image. This operation is similar to the operation from steps S2001 to S2003 in embodiment 1. The terminal 401 in this embodiment does not perform the sugar content estimation operation in step S2004 in embodiment 1, but transmits the terminal ID, region information, and spectral data to the server 301.

[0295] The server 301, having received the spectral data, performs the update operations from step S1009 to step S1017, as in embodiment 1. The server 301 in this embodiment further applies a statistical model to the received spectral data to estimate the sugar content for each region. This operation is the same as step S3003 in the server 300 in embodiment 4. The server 301 transmits the sugar content for each region and the updated reduced restoration table to the terminal 401.

[0296] Server 301 in this embodiment executes operations similar to those shown in Figures 16 and 18. However, in step S1240 shown in Figures 16 and 18, only the reduced restoration table is sent to terminal 401. After step S1320 shown in Figure 18, sugar content estimation is performed, and data indicating the estimation result is sent to terminal 401. Except for these points, the operation of server 301 is similar to the operation of statistical learning device 100 in embodiment 1.

[0297] Meanwhile, the terminal 401 executes the operations shown in Fig. 43. Steps S2010 and S2030 to S2080 shown in Fig. 43 are the same as the operations of the corresponding steps in Fig. 19 (Embodiment 1). The operation of step S4130 is the same as step S4130 in Fig. 39 (Embodiment 4). In this embodiment, in step S2010, it is determined whether the data received by the receiver 274 is a reduced-restoration table. If the received data is not a reduced-restoration table, that is, if the received data is data indicating an estimation result of sugar content, the process proceeds to step S4130, where the estimation result of sugar content is output. If the received data is a reduced-restoration table, the process proceeds to step S2021, where the reduced-restoration table recorded in the storage device 220 is updated with the received reduced-restoration table. Thereafter, as in Embodiment 1, the restoration processing circuit 230 executes the operations of steps S2030 to S2080 to generate hyperspectral data. In the following step S2121, the transmitter 272 associates the hyperspectral data generated by the restoration processing circuit 230 with information that identifies the area in the image where the hyperspectral data was generated, and transmits the data to the server 301 together with the terminal ID.

[0298] As described above, in this embodiment, the processing circuitry 140 of the server 301 acquires hyperspectral data that the terminal 401 generates from the compressed image data based on the reduced-restoration table. Then, based on a statistical model, the processing circuitry 140 estimates characteristic values ​​such as the sugar content of the object from the hyperspectral data and transmits data indicating the characteristic values ​​to the terminal 401. This operation reduces the processing load of the terminal 401 compared to the first and second embodiments.

[0299] The above embodiments are merely examples, and the present disclosure is not limited to the above embodiments. For example, other embodiments may be configured by appropriately combining the configurations of the above embodiments. The configurations of the above embodiments may be applied to estimating characteristic values ​​other than sugar content of an object.

[0300] Other embodiments of the present disclosure will be exemplified below.

[0301] <Communication without going through a network> In the above first to fifth embodiments, the statistical learning device or server and the estimation device or terminal communicate via a network, but the present disclosure is not limited to such a configuration. The statistical learning device or server and the estimation device or terminal may be connected without a network. For example, the statistical learning device or server and the estimation device or terminal may be connected by wiring within a single system and configured to communicate with each other. The statistical learning device or server and the estimation device or terminal may be configured to communicate wirelessly via Wi-Fi (registered trademark) or Bluetooth (registered trademark), for example.

[0302] <Creating a reduction restoration table before shipping the camera> In the first to fifth embodiments, the reduced-restoration table is created and updated after the estimation device or terminal including the hyperspectral camera is shipped and the user starts using it. This is not limiting. For example, the reduced-restoration table may be created before shipping, and may not be updated after shipping. In this case, the reduced-restoration table is recorded in a storage device of the estimation device or terminal including the hyperspectral camera during the manufacturing process, and continues to be used without being edited after the user starts using it.

[0303] <Model training methods other than principal component analysis> In the first to fifth embodiments, principal component analysis is used to train a model used to estimate characteristic values ​​from hyperspectral data based on compressed image data generated by an imaging device. Based on a statistical model obtained by learning based on principal component analysis, principal component scores are calculated from the hyperspectral data, and a characteristic value (e.g., sugar content) of an object is estimated from the principal component scores. Instead of such a method based on principal component analysis, information from which characteristic values ​​can be estimated may be extracted using other learning methods. For example, other statistical methods such as independent component analysis or other machine learning methods such as neural networks may be used. As an example, a deep neural network (DNN) may be used to generate weights for each of multiple wavelength bands associated with a characteristic value as parameters for characteristic value estimation. A characteristic value estimation model based on the generated parameters may be generated, for example, by fitting a cubic curve. The learning method may be a machine learning method other than DNN. The order of the model for estimating the characteristic value may be other than cubic, and a method of fitting another model may be used.

[0304] <Creating a model to extract spectra related to characteristic values> In the first to fifth embodiments, principal component analysis is used to identify bands with high importance and bands with low importance for the spectral data output from the estimation device or terminal. This is a statistical information compression process for the spectral information. The method for compressing the spectral information may be a statistical method other than principal component analysis, such as factor analysis or independent component analysis, or a machine learning method such as a neural network. In the first to fifth embodiments, in addition to compressing the information on the spectral axis using principal component analysis, principal component scores are used to generate information weighted to bands with high importance (i.e., bands), and characteristic values ​​such as sugar content are estimated. In the first to fifth embodiments, processing the spectrum of a specific region in an image generates information about the characteristic value (e.g., sugar content) of an object appearing in that specific region. In other words, the processing result does not include image information, and the position information of the image and the compressed spectral information cannot be linked and displayed without recombining it with the original image. Therefore, instead of the methods in the first to fifth embodiments, a method may be used in which machine learning is used to compress the spectral information of a hyperspectral image and create a model for generating a smaller number of intensity images than the original hyperspectral image. An embodiment of a system for implementing such a method is described below.

[0305] (Embodiment 6) FIG. 44 illustrates an overview of a system 60 that uses machine learning to generate a small number of luminance images representing characteristic values ​​from a hyperspectral image and creates a reduced-restoration table corresponding to the model. In the following description, the small number of luminance images representing characteristic values ​​may be referred to as "characteristic value-representing images." The system 60 includes a processing device 600 and an imaging device 700. The processing device 600 uses machine learning to generate a reduced-restoration table for generating a small number of luminance images representing characteristic values ​​from a compressed image, using a hyperspectral image generated based on a compressed image output from a hyperspectral camera and a pre-prepared full restoration table. The created reduced-restoration table is stored in the storage device of the imaging device 700. A processing circuit in the imaging device 700 generates and outputs a small number of luminance images representing characteristic values ​​based on the compressed image and the reduced-restoration table. The characteristic value may be a value representing a characteristic such as the sugar content of an object, or a value representing a slight color difference or color spread of the object. The processing device 600 generates a reduced-restoration table suited to the characteristic values ​​and sends it to the imaging device 700, which then functions as a camera that uses the reduced-restoration table to generate a small number of brightness images representing the characteristic values ​​from the compressed image. By using such a camera, for example, when the characteristic is unevenly or locally distributed within the object, it is possible to capture the distribution of the characteristic within the object as an image.

[0306] FIG. 45 is a block diagram showing an example configuration of a system 60 according to this embodiment. The system 60 includes a processing device 600 and one or more imaging devices 700. The processing device 600 performs settings necessary for the imaging devices 700 to output images that express target characteristic values. The processing device 600 may be, for example, a computer such as a server managed by a business that operates the system 60 and sells or distributes the imaging devices 700 to users. The processing device 600 can be sequentially connected to multiple imaging devices 700 and perform settings for each imaging device 700, for example, during the manufacturing process of the imaging devices 700. The processing device 600 and the imaging devices 700 may be connected directly or via a network. The processing device 600 and the imaging devices 700 may be connected via a wired or wireless connection.

[0307] The processing device 600 includes a first storage device 110, a second storage device 120, a third storage device 130, a processing circuit 140, an input device 180, and an output device 190. Each imaging device 700 includes a hyperspectral camera 210, a fourth storage device 220, a restoration processing circuit 230, an input device 280, and an output device 290. The input device 180 and the output device 190 may be elements external to the processing device 600. Similarly, the input device 280 and the output device 290 may be elements external to the imaging device 700.

[0308] The configuration of the processing device 600 shown in FIG. 45 is the same as the configuration of the statistical learning device 100 in the first embodiment shown in FIG. 5, except that the communication circuit 170 is removed. Each of the first storage device 110, the second storage device 120, and the third storage device 130 is a device that stores data using any storage medium. The first storage device 110 stores learning data for generating a model used to generate a small number of intensity images representing the distribution of characteristic values ​​(e.g., the sugar content of an object) from hyperspectral data. The second storage device 120 stores a full restoration table corresponding to the characteristics of the hyperspectral camera 210 in each imaging device 700. The third storage device 130 stores a reduced restoration table corresponding to the characteristics of the hyperspectral camera 210 in each imaging device 700 and a model for compressing or reducing spectral information from the hyperspectral data. Hereinafter, this model may be referred to as a "spectral reduction model."

[0309] The training data stored in the first storage device 110 is similar to the training data in embodiment 1 shown in Fig. 6A. That is, the first storage device 110 stores the training data by associating spectral data with characteristic values ​​such as sugar content. The hyperspectral data in this embodiment is image data having information on the brightness values ​​of each of multiple unit bands for each pixel, and the training data is stored for each pixel.

[0310] FIG. 46 is a diagram showing an example of the distribution of characteristic values ​​in an object. The spatial distribution of characteristic values ​​(e.g., sugar content) can be expressed and recorded in the form of, for example, contour lines. In addition to the distribution of the magnitude of the characteristic values, the distribution of the types of substances that affect the characteristic values ​​(e.g., types of glucose, fructose, etc.) can also be recorded. Information on such distribution can be stored in the first storage device 110 in association with the spectral data of each pixel.

[0311] The full restoration table for each imaging device 700 stored in the second storage device 120 is the same as the full restoration table in the first embodiment shown in FIG.

[0312] FIG. 47 is a diagram illustrating an example of data of a spectrum reduction model stored in the third storage device 130. In this embodiment, the third storage device 130 stores a model trained by machine learning for generating a small number of luminance images in which spectral information is compressed or reduced. The machine learning may be learning using a statistical method as used in embodiments 1 to 5, or learning using a non-statistical method. In the example of FIG. 47, a model is recorded for each luminance image to be generated. The model corresponding to each luminance image includes information on weights corresponding to each band in the original hyperspectral data. The small number of luminance images representing characteristic values ​​can be generated, for example, by multiplying the luminance of each band by the corresponding weight shown in FIG. 47 for each pixel, and adding the result for all bands to obtain the luminance of that pixel. The stronger the correlation between a band and the characteristic value, the greater the weight it has.

[0313] The reduced-restoration table for each imaging device 700 stored in the third storage device 130 is similar to the reduced-restoration table in the first embodiment shown in Fig. 8. The reduced-restoration table is generated based on the spectrum reduction model shown in Fig. 47. For example, the reduced-restoration table can be generated by grouping bands by integrating multiple bands whose weights are smaller than a threshold value into one band in the spectrum reduction model, and determining a coefficient for each group.

[0314] The processing circuit 140 generates a model and a reduced restoration table corresponding to the model. The processing circuit 140 generates the model by learning the training data stored in the first storage device 110 using, for example, a deep neural network (DNN). The processing circuit 140 performs training so that, for example, pixels with higher characteristic values ​​have higher luminance. The processing circuit 140 uses the spectral data for each pixel as input data to the DNN and the corresponding characteristic values ​​as training data. The model may be, for example, a model that generates a luminance image for each of multiple characteristic values, or a model that generates different luminance images for multiple ranges of characteristic values. The training data does not need to include training data, i.e., information about the characteristic values. In this case, the processing device 600 may perform training based only on the distribution of the spectral data. For example, the processing device 600 may extract several spectral distribution features from the spectral data in multiple data sets and classify the spectral data into a small number of groups based on the extracted features. The processing device 600 may generate, as a model, conversion information for generating images in which the spectral features of each group are emphasized from the spectral information of the original image.

[0315] Fig. 48 is a diagram showing an overview of communication and processing between the processing device 600 and the imaging device 700. Fig. 48 illustrates an example of a learning model generation process executed by the processing device 600 as an initial setting, the subsequent initial setting of the imaging device 700, i.e., a process of generating a reduced / restored table and writing the reduced / restored table to the imaging device 700, and the subsequent imaging operation by the imaging device 700.

[0316] The initial setting of the processing device 600 includes the operations of steps S6001 and S6002 shown in Fig. 48. The processing circuit 140 first acquires learning data from the first storage device 110 (step S6001). Next, the processing circuit 140 performs learning using DNN based on the acquired learning data, generates a learning model that generates a small number of luminance images that represent characteristic values, and stores the generated learning model in the third storage device 130 (step S6002). This completes the initial setting of the processing device 600.

[0317] The initial setting of the imaging device 700 includes the operations of steps S7001, S6003, S6004, and S7002 shown in FIG. 48. First, the imaging device 700 connects to the processing device 600 and transmits its own device ID (step S7001). The processing device 600 acquires the device ID transmitted from the imaging device 700. The processing device 600 acquires a full restoration table corresponding to the connected imaging device 700 from the second storage device 120 based on the acquired device ID (step S6003). The processing circuit 140 generates a reduced restoration table corresponding to the imaging device 700 based on the previously created learning model and the full restoration table, and transmits the reduced restoration table to the imaging device 700 (step S6004). The imaging device 700 stores the reduced restoration table acquired from the processing device 600 in the fourth storage device 220 (step S7002). The series of operations in steps S7001, S6003, S6004, and S7002 completes the initial setting of the image capture device 700. This initial setting allows the image capture device 700 to function as a camera that outputs a small number of intensity images that represent characteristic values, i.e., characteristic value representation images, rather than as a hyperspectral camera that uniformly outputs detailed spectral data.

[0318] After the initial setting, the imaging device 700 disconnects from the processing device 600. Thereafter, the imaging device 700 performs imaging using compressed sensing with the hyperspectral camera 210 to generate a compressed image (step S7003). The restoration processing circuit 230 uses the reduced restoration table stored at the initial setting to generate and output a small number of luminance images representing characteristic values ​​from the compressed image (step S7004).

[0319] FIG. 49 is a flowchart showing a specific example of the operations of steps S6001 to S6002 shown in FIG. 48. In the initial state, the processing device 600 has not yet generated a learning model through machine learning, and the storage device 220 of the imaging device 700 has not yet stored a reduced-restoration table. Therefore, even if the imaging device 700 captures images using the hyperspectral camera 210, it is unable to restore image data for each band, nor is it able to output a small number of luminance images representing characteristic values. Therefore, the processing device 600 generates a model for compressing and reducing the spectral axis according to the target characteristic value. Based on the model for compressing and reducing the spectral axis, the processing device 600 generates a reduced-restoration table for the hyperspectral camera 210 included in each imaging device 700 and outputs it to the imaging device 700. The reduced-restoration table is data obtained by modifying the full-restoration table so that a small number of luminance images representing characteristic values ​​are output. The operations shown in FIG. 49 are initiated by a start input means such as the input device 180. The operations of each step are described below.

[0320] (Step S6110) The processing circuit 140 determines whether a sufficient amount of training data necessary for training is stored in the first storage device 110. The sufficient amount necessary for training is a preset number, such as 10 times the number of spectral bands. If sufficient sample data is stored, the process proceeds to step S6130. If insufficient sample data is stored, the process proceeds to step S6120.

[0321] (Step S6120) If the sample data is insufficient, processing circuit 140 causes output device 190 to output a request for additional training sample data. If output device 190 includes a display, for example, the output content may be a character string or a warning image. If output device 190 includes a speaker, the output content may be voice or a warning sound. After step S6120, processing returns to step S6110.

[0322] (Step S6130) The processing circuit 140 performs machine learning using the spectral data and characteristic values ​​for each pixel of the learning data stored in the first storage device 110 as input. Learning methods include, for example, statistical methods, methods using neural networks, and methods using SVM (Support Vector Machine). Through machine learning, the processing circuit 140 generates a spectrum reduction model that converts spectral information of all bands that can be captured by the hyperspectral camera 210 into a small amount of spectral information.

[0323] (Step S6140) The processing circuit 140 stores the spectral reduction model generated in step 6130 in the third storage device 130 .

[0324] 45 shows an example in which the first storage device 110 stores one type of training data and the third storage device 130 stores one spectrum reduction model, but this is merely an example. For example, the first storage device 110 may store multiple types of training data, and the third storage device 130 may store multiple spectrum reduction models corresponding to the multiple types of training data. When the first storage device 110 stores multiple types of training data, the operations of steps S6110 to S6140 may be repeated the number of times corresponding to the number of types of training data to generate respective spectrum reduction models for all types of training data. Alternatively, the operations of steps S6110 to S6140 may be performed each time a new type of training data is added.

[0325] Fig. 50 is a flowchart showing a specific example of the operations of steps S6003 to S6004 shown in Fig. 48. The processing device 600 extracts a full restoration table specific to the imaging device 700 based on the imaging device ID acquired from the imaging device 700, generates a reduced restoration table specific to the imaging device 700, and transmits it to the imaging device 700. The reduced restoration table is configured to generate several images from a compressed image in accordance with a function that expresses characteristic values ​​assigned to the imaging device 700. The operations shown in Fig. 50 are started by start input means such as the input device 180. The operations of each step will be described below.

[0326] (Step S6210) First, processing circuit 140 determines whether or not a device ID has been acquired from image capture device 700. If a device ID has been acquired from image capture device 700 in step S6210, processing circuit 140 proceeds to step S6220. If a device ID has not been acquired from image capture device 700 in step S6210, processing circuit 140 repeats step S6210.

[0327] (Step S6220) Based on the device ID of the imaging device 700 acquired in step S6210, the processing circuit 140 acquires the full restoration table corresponding to the device ID acquired in step S6210 from the full restoration tables for each imaging device 700 stored in the second storage device 120.

[0328] (Step S6230) Processing circuit 140 generates a reduced reconstruction table from the full reconstruction table extracted in step S6220 based on the spectrum reduction model stored in third storage device 130. The method for generating the reduced reconstruction table is the same as the method in step S1235 of FIG.

[0329] (Step S6240) The processing circuit 140 transmits the reduced restoration table generated in step S6230 to the image capture device 700.

[0330] Steps S6210 to S6240 are repeated each time a different imaging device 700 is connected to the processing device 600. This allows the processing device 600 to generate a reduced restoration table specific to each imaging device 700, enabling the imaging device 700 to output a small number of luminance images for expressing characteristic values.

[0331] 50, there is one type of characteristic value to be represented, and therefore one spectrum reduction model. This example is not limiting; there may be multiple types of characteristic values ​​to be represented, and multiple spectrum reduction models may be generated for each characteristic value and stored in the third storage device 130. In this case, the processing device 600 acquires information on the characteristic values ​​represented by the images generated by each of the image capture devices 700, along with the device ID. The information on the characteristic values ​​may be stored in advance in one of the storage devices together with the device ID. Alternatively, the processing device 600 may acquire the information on the characteristic values ​​together with the device ID from the image capture device 700, or the information on the characteristic values ​​may be input from the input device 180.

[0332] Next, the operation of the imaging device 700 will be described.

[0333] FIG. 51 is a flowchart showing a specific example of the operations of steps S7001 to S7002 shown in FIG. 48. In the initial state, the fourth storage device 220 included in the imaging device 700 does not store a restoration table, and image data cannot be restored and displayed even when images are captured using the hyperspectral camera 210. As an initial setting, the imaging device 700 acquires from the processing device 600 a reduced restoration table that matches the characteristic value representation function installed in the imaging device 700, and stores the reduced restoration table in the storage device 220. This enables the imaging device 700 to generate and output a small number of luminance images that represent characteristic values. The operation of each step will be described below.

[0334] (Step S7110) By operation of a user or an operator, the imaging device 700 is connected to the processing device 600, enabling transmission and reception of signals.

[0335] (Step S7120) The restoration processing circuit 230 outputs the device ID of the image capture device 700 to the processing device 600. Thereafter, the processing device 600 performs the operation shown in FIG.

[0336] (Step S7130) The restoration processing circuit 230 determines whether or not a reduced restoration table has been acquired from the processing device 600. If a reduced restoration table has been acquired in step S7130, the process proceeds to step S7140. If a reduced restoration table has not been acquired in step S7130, step S7130 is repeated.

[0337] (Step S7140) The image capturing apparatus 700 stores the reduced restoration table acquired in step S7130 in the storage device 220.

[0338] Through the operations from step S7110 to step S7140, the initial setting of the image capturing device 700 is completed.

[0339] Fig. 52 is a flowchart showing a specific example of the operations in steps S7003 and S7004 shown in Fig. 48. The operations in each step will be described below.

[0340] (Step S7210) The hyperspectral camera 210 captures an image of an object. The hyperspectral camera 210 acquires an image using compressed sensing. The captured image is output by compressing information from multiple spectra into the format of a single intensity image.

[0341] (Step S7220) The restoration processing circuit 230 restores image data of a small number of spectra representing the characteristic values ​​from the compressed image data generated in step S7210 using the reduction restoration table stored in the fourth storage device 220. The restoration in this embodiment does not completely restore information for all spectral bands that can be captured by the hyperspectral camera 210. The restoration processing circuit 230 performs reduction restoration to restore only a small number of spectral information pieces representing the characteristic values ​​and generate a luminance image for each spectral information piece. This reduction restoration generates a small number of luminance images representing the characteristic values.

[0342] (Step S7230) The output device 290 outputs the small number of luminance images generated in step S7220 as images representing the characteristic values. The output can be in the form of, for example, display on a display, printing, or output of image data.

[0343] By performing the operations from step S7210 to step S7230, a small number of brightness images representing characteristic values ​​can be output by capturing images using the hyperspectral camera 210, which visualize the distribution or boundary of a specific substance or state.

[0344] In the examples of FIGS. 51 and 52 , the imaging device 700 is initially configured as an imaging device that generates an image representing one type of characteristic value and performs an imaging operation. The imaging device 700 is not limited to this configuration, and may be configured to generate multiple types of luminance images representing multiple types of characteristic values. To this end, in the initial setting operation shown in FIG. 51 , the imaging device 700 acquires multiple reduced-restoration tables corresponding to multiple types of specific values ​​to be expressed and stores them in the storage device 220. To acquire multiple reduced-restoration tables corresponding to multiple types of characteristic values, the imaging device 700 may be configured to transmit information on multiple types of characteristic values ​​together with the device ID when outputting the device ID in step S7120. Alternatively, information indicating multiple types of characteristic values ​​corresponding to the device ID may be stored in one of the storage devices included in the processing device 600. Information specifying characteristic values ​​may be input from the input device 180 of the processing device 600 or the input device 280 of the imaging device 700, and the imaging device 700 may acquire the reduced-restoration table from the processing device 600 according to the input information on the characteristic values. 52 , in order to select a reduced-restoration table, the restoration processing circuit 230 may be configured to acquire information on the characteristic value to be expressed. The information on the characteristic value may be input, for example, from the input device 280. Based on the input information on the characteristic value, the restoration processing circuit 230 may extract a reduced-restoration table corresponding to the input characteristic value from the reduced-restoration tables stored in the storage device 220 and use the extracted table for restoration.

[0345] By performing the initial setting operation of the imaging device 700 shown in Fig. 51 and the imaging operation of the imaging device 700 shown in Fig. 52, the imaging device 700 can capture an image of an object and generate and output a small number of intensity images representing characteristic values. This makes it possible to generate a small number (e.g., about three to five) of images that emphasize spectra correlated with sugar content, for example, when the object is an agricultural product. By appropriately creating a model and a reduced restoration table, it is also possible to generate different intensity images for each type of sugar (e.g., glucose, fructose, etc.). In other words, it is possible to generate intensity images corresponding to each substance used to determine a certain characteristic (e.g., sugar content).

[0346] In this embodiment, one or more reduced / restored tables are generated during initial setup of the imaging device 700, i.e., before shipment, and stored in the storage device 220. This configuration is not limiting, and for example, a reduced / restored table may be added after the initial setup to add characteristic values ​​that can be expressed by the imaging device 700. When adding a function to express new characteristic values ​​to the imaging device 700, the operations of steps S7001, S6003, S6004, and S7002 in Fig. 48 are performed again. The connection between the processing device 600 and the imaging device 700 at this time may be a direct connection between the processing device 600 and the imaging device 700, as in the initial setup, or a connection via a network.

[0347] The processing device 600 in this embodiment generates a model for converting hyperspectral data into spectral data associated with characteristic values ​​by itself, but the model may be generated in advance by another device and stored in the storage device 130. In this case, the processing device 600 can read from the storage device 130 a model that has been generated in advance based on a data set of multiple samples, and generate a reduced restoration table for restoring spectral data associated with characteristic values ​​from compressed image data based on the model and the full restoration table.

[0348] (Other 1) The processing unit may be: The processing unit includes a memory and a circuit.

[0349] The memory stores a plurality of data sets (e.g., training data 110) and a first table (e.g., a complete restoration table 120 for each terminal). Each of the plurality of data sets includes first data indicating a value of a predetermined characteristic of a first sample and a luminance value in each of a plurality of first wavelength ranges (see FIG. 6A). The first data is generated based on second data.

[0350] The second data is generated using a plurality of first pixel values ​​of a plurality of first pixels included in a first image and the first table. A first camera captures an image of the first sample and outputs the first image.

[0351] The circuit determines a plurality of second wavelength ranges based on the plurality of data sets (see FIG. 15), and generates a second table based on the first table and the plurality of second wavelength ranges (see FIG. 17).

[0352] The first camera captures an image of a second sample different from the first sample and outputs a second image. Third data is generated using a plurality of second pixel values ​​of a plurality of second pixels included in the second image and the second table. A luminance value of the second sample in each of the plurality of second wavelength ranges is determined based on the third data. A value of the predetermined characteristic of the second sample is determined based on the luminance value of the second sample in each of the plurality of second wavelength ranges.

[0353] The first camera includes a filter array and an image sensor. The filter array is located between the first sample and the image sensor. The filter array is located between the second sample and the image sensor. The filter array includes a plurality of regions, each of which has a different transmittance characteristic for a wavelength of light.

[0354] If the vector indicating the plurality of first pixel values ​​is y1, the matrix indicating the first table is H, and the vector indicating the second data is x, then y1=Hx. If the vector indicating the plurality of second pixel values ​​is y2, and the matrix indicating the second table is H, thenr , where z is a vector indicating the third data, y2=H r z. The number of the first pixels is equal to the number of the second pixels. {(the number of components of H)-(the number of components of H) r (number of components)}=(number of the plurality of first pixels)×((number of the plurality of first wavelength ranges)−(number of the plurality of second wavelength ranges)).

[0355] A second camera different from the first camera may capture the first sample and output the first image. The second camera includes a second filter array and a second image sensor. The second filter array is located between the first sample and the second image sensor. The second filter array includes a plurality of second regions, each having a different transmittance characteristic for a wavelength of light.

[0356] (Other 2) As long as they do not deviate from the spirit of this disclosure, various modifications that would occur to a person skilled in the art to this embodiment, or forms constructed by combining components from different embodiments, may also be included within the scope of one or more aspects of this disclosure. [Industrial Applicability]

[0357] The technology disclosed herein can be widely used for estimating characteristics such as sugar content of agricultural products. For example, the technology disclosed herein can be applied to mobile devices such as smartphones, fruit thinning or sorting devices, or robots. [Explanation of symbols]

[0358] 10, 20 Sugar content estimation system 50 Network 70 Objects 100 Statistical Learning Device 110, 120, 130 storage devices 140 Processing Circuit 170 Communication Circuit 172 Transmitter 174 Receiver 180 Input Devices 190 Output Device 200 Sugar content estimation device 210 Hyperspectral Camera 220, 240 storage device 230 Restoration processing circuit 250 Estimation processing circuit 270 Communication Circuits 272 Transmitter 274 Receiver 280 Input Device 290 Output Device 300 servers 310 Processing Circuit 400 terminals 410 Processing Circuit 500 Hyperspectral Camera 502 Image Sensor 504 Filter Array 506 Optical system 510 Compressed Images 520 Reconstructed Hyperspectral Images 530 Processing Circuit 600 Processing Equipment 700 Imaging Device

Claims

1. A processing device connected to one or more devices including a hyperspectral sensor that generates compressed image data in which hyperspectral information of an object is compressed as two-dimensional image information, a storage device that stores a plurality of sample data sets and a first restoration table for restoring hyperspectral data from the compressed image data, wherein each sample data set includes hyperspectral data of the sample and data indicating characteristic values ​​of the sample; a processing circuit that generates a model for converting the hyperspectral data into spectral data associated with the characteristic values ​​based on the data set of the plurality of samples, and generates a second restoration table for restoring the spectral data associated with the characteristic values ​​from the compressed image data based on the model and the first restoration table; A processing device comprising:

2. the hyperspectral data includes brightness information for each of a plurality of wavelength bands included in a target wavelength range; the first restoration table is data for restoring the luminance information for each of the plurality of wavelength bands from the compressed image data, the second restoration table is data for restoring, from the compressed image data, luminance information obtained by weighting and adding the luminance information for each of the plurality of wavelength bands; The processing device of claim 1 .

3. The processing device according to claim 1 , wherein the data size of the second restoration table is smaller than the data size of the first restoration table.

4. The processing circuitry calculating weights corresponding to each of a plurality of wavelength bands associated with the characteristic values ​​by machine learning based on the hyperspectral data of the plurality of samples; generating the second restoration table from the first restoration table based on the weights; 、 The processing device according to any one of claims 1 to 3.

5. the hyperspectral data includes brightness information for each of a plurality of wavelength bands included in a target wavelength range; the first restoration table is data for restoring the luminance information for each of the plurality of wavelength bands from the compressed image data, The processing circuitry determining a weight corresponding to each of the plurality of wavelength bands by machine learning based on the hyperspectral data of the plurality of samples; determining, from among the plurality of wavelength bands, a portion of wavelength bands having a relatively low weight based on the weight; generating the second restoration table by integrating information about the part of wavelength bands in the first restoration table; The processing device of claim 1 .

6. The hyperspectral sensor includes a filter array including a plurality of filters each having a different transmission spectrum; the first restoration table is data reflecting a spatial distribution of the transmission spectrum of the filter array; The processing device according to any one of claims 1 to 5.

7. The processing device according to claim 1 , wherein the processing circuitry transmits the second restoration table to the device.

8. the processing circuit generates the model, the first restoration table, and the second restoration table for each predetermined classification; The processing device according to any one of claims 1 to 7.

9. The processing circuitry The device obtains partial compressed image data generated by extracting data of a partial region from the compressed image data, generating the spectral data corresponding to the partial region from the partial compressed image data using the second restoration table; transmitting the spectral data to the device; The processing device according to any one of claims 1 to 8.

10. The processing circuitry obtaining the compressed image data from the device; generating spectral data associated with the characteristic values ​​from the compressed image data using the second restoration table; transmitting the spectral data to the device; The processing device according to any one of claims 1 to 8.

11. The processing circuitry when it is determined that the model and the second restoration table need to be changed based on the spectral data generated from the compressed image data based on the second restoration table and the hyperspectral data in the data set of the plurality of samples; regenerating the model based on the dataset of new samples; updating the second restoration table in accordance with the regenerated model; The processing device according to any one of claims 1 to 8.

12. 1. A computer-generated method comprising: Obtaining a first restoration table for restoring hyperspectral data from compressed image data in which hyperspectral information of the object is compressed as two-dimensional image information; obtaining a model for converting the hyperspectral data into spectral data related to the characteristic values, the model being generated based on a dataset of the sample including hyperspectral data of the sample and data indicative of characteristic values ​​of the sample; generating a second restoration table for restoring spectral data associated with the characteristic values ​​from the compressed image data based on the model and the first restoration table; A method comprising:

13. the hyperspectral data includes brightness information for each of a plurality of wavelength bands included in a target wavelength range; the first restoration table is data for restoring the luminance information for each of the plurality of wavelength bands from the compressed image data, the second restoration table is data for restoring, from the compressed image data, luminance information obtained by weighting and adding the luminance information for each of the plurality of wavelength bands; The method of claim 12.

14. The method according to claim 12 or 13, wherein the data size of the second restoration table is smaller than the data size of the first restoration table.

15. the second restoration table is generated by combining information of some of the plurality of wavelength bands included in the first restoration table. The method of claim 13.

16. Obtaining a first restoration table for restoring hyperspectral data from compressed image data in which hyperspectral information of the object is compressed as two-dimensional image information; acquiring a dataset for the sample, the dataset including hyperspectral data for the sample and data indicative of a property value of the sample; generating a model based on the data set for converting the hyperspectral data into spectral data related to the characteristic values; generating a second restoration table for restoring spectral data associated with the characteristic values ​​from the compressed image data based on the model and the first restoration table; A computer program that causes a computer to execute the following.

Images