Information processing device, information processing method

Abstract

This method calculates a quantitative evaluation index value indicating the similarity between a spectral spectrum acquired using a spectroscopic sensor and the spectral spectrum of a hypothetical object, without using a spectral spectrum as a negative example, while reducing the computational load. [Solution] The information processing device according to this technology includes a calculation unit that calculates a deviation index value, which is an index value indicating the degree of deviation between the target spectrum, which is a spectral spectrum acquired using a spectroscopic sensor, and the assumed spectrum, based on the eigenvalues ​​and eigenvectors of feature quantities obtained by dimensionality reduction performed on the assumed spectrum, which is a spectral spectrum of a hypothetical object.

Description

Technical Field

[0001] The present technology relates to an information processing apparatus and an information processing method, and particularly relates to a technical field of performing processing on a spectral spectrum acquired using a spectral sensor.

Background Art

[0002] A spectral sensor (multi-spectrum sensor) for obtaining a wavelength characteristic analysis image of light from a subject, in other words, a plurality of wavelength band images that are analysis images of the spectral characteristics of the subject, is known. Also, based on those plurality of wavelength band images, for example, applications for performing various analyses of a subject based on the spectral spectrum acquired using a spectral sensor, such as evaluating the vegetation state of a plant or evaluating the state of human skin, have been developed.

[0003] Here, when performing analysis of a subject based on the spectral spectrum as described above, there are cases where it is required to determine whether the spectral spectrum acquired using the spectral sensor corresponds to the spectral spectrum of an assumed type of object (hereinafter referred to as an "assumed object"). For example, when the assumed object is a plant, the state of the plant is evaluated by calculating a vegetation evaluation value such as NDVI (Normalized Difference Vegetation Index) from the spectral spectrum. However, the calculation formula for the vegetation evaluation value is determined assuming the spectral spectrum of an object as a plant, and meaningless values are calculated for the spectral spectra of parts other than plants, such as soil and roads. That is, if a vegetation evaluation value is calculated for the entire image in which not only plants but also objects other than plants are shown as an image of the area to be evaluated, the evaluation values calculated for the parts other than plants become noise, and there is a risk that the vegetation evaluation of the target area cannot be correctly performed. For example, there are cases where we want to exclude the spectral spectra of objects other than the intended object from an image. To do this, it is necessary to determine whether the spectral spectrum acquired using a spectral sensor corresponds to the spectral spectrum of the intended object.

[0004] Traditionally, to exclude spectra of objects other than the intended object, a method was employed in which "negative examples"—spectral spectra of objects other than the intended object—were defined. In other words, this method involved excluding spectral spectra that corresponded to the "negative examples."

[0005] Regarding related prior art, Patent Document 1 below can be cited. Patent Document 1 below discloses a technique for determining the similarity between a known feature spectrum obtained when training data is input and a feature spectrum obtained when target data is input, with respect to a feature spectrum obtained in a specific layer of a neural network that performs classification processing. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2023-078766 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] However, in practice, it is difficult to correctly define negative examples for multidimensional information such as spectra using the methods described above for defining negative examples.

[0008] This technology was developed in view of the above-mentioned problems, and aims to calculate a quantitative evaluation index value that shows the similarity between the spectral spectrum acquired using a spectroscopic sensor and the spectral spectrum of a hypothetical object, without using spectral spectra as negative examples, while reducing the amount of computation. [Means for solving the problem]

[0009] The information processing device relating to this technology includes a calculation unit that calculates a deviation index value, which is an index value indicating the degree of deviation between the target spectrum, which is a spectral spectrum acquired using a spectroscopic sensor, and the assumed spectrum, based on the eigenvalues ​​and eigenvectors of feature quantities obtained by dimensionality reduction performed on the assumed spectrum, which is a spectral spectrum of a hypothetical object. The deviation index value described above makes it possible to quantitatively evaluate the similarity between the spectral spectrum obtained using a spectroscopic sensor and the spectral spectrum of a hypothetical object, without using a spectral spectrum as a negative example. Furthermore, since this deviation index value is calculated based on dimensionality-reduced information, the amount of computation is reduced. [Brief explanation of the drawing]

[0010] [Figure 1] This is an explanatory diagram illustrating the configuration of an information processing system equipped with an information processing device as the first embodiment. [Figure 2] This is a block diagram showing an example configuration of an information processing device as a first embodiment. [Figure 3] This is a block diagram showing an example of the configuration of a spectroscopic camera in the first embodiment. [Figure 4] This diagram schematically shows an example of the configuration of the pixel array section of a spectroscopic sensor. [Figure 5] This is an illustrative diagram of the spectral sensitivity of a spectroscopic sensor. [Figure 6] This is a functional block diagram illustrating the functions of an information processing device as a first embodiment. [Figure 7] This figure shows an example of the spectral distribution of a light source. [Figure 8] This figure shows an example of the reflectance spectrum of a hypothetical object. [Figure 9] This figure shows an example of a display screen provided by the display processing unit of the first embodiment. [Figure 10]It is a flowchart showing an example of a specific processing procedure to be executed to realize an information processing method as the first embodiment. [Figure 11] It is a diagram for explaining an example of the configuration of an information processing apparatus as the second embodiment. [Figure 12] It is a block diagram showing an example of the configuration of a spectroscopic camera in the second embodiment. [Figure 13] It is an explanatory diagram of an example of noise generated in a spectroscopic image. [Figure 14] It is a diagram for explaining an example of the configuration of an information processing apparatus as the third embodiment. [Figure 15] It is a diagram showing an example of a presentation screen by a presentation processing unit in the third embodiment. [Figure 16] It is a flowchart showing an example of a specific processing procedure to be executed to realize an information processing method as the third embodiment. [Figure 17] It is a diagram for explaining an example of the configuration of an information processing apparatus as another example of the third embodiment. [Figure 18] It is a flowchart showing an example of a specific processing procedure to be executed by an information processing apparatus as another example of the third embodiment. [Figure 19] It is a diagram for explaining an example of the configuration of an information processing apparatus as a first alternative example. [Figure 20] It is a diagram showing an example of a map image of a divergence evaluation value calculated for each dimension. [Figure 21] It is a diagram for explaining an example of the configuration of an information processing apparatus as a second alternative example. [Figure 22] It is a diagram showing an example of a bounding box detected by an object detection process. [Figure 23] It is a diagram for explaining an example of the configuration of an information processing apparatus as a modification example.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, referring to the accompanying drawings, embodiments of the present technology will be described in the following order. <1. First Embodiment> (1-1. System Overview) (1-2. Example of an information processing device configuration) (1-3. Example of a spectroscopic camera configuration) (1-4. Information Processing Method as the First Embodiment) (1-5. Processing Procedure) <2. Second Embodiment> <3. Third Embodiment> <4. Another example related to deviation index values> <5. Variation> <6. Summary of Embodiments> <7. This Technology>

[0012] <1. First Embodiment> (1-1. System Overview) Figure 1 is a diagram illustrating the configuration overview of an information processing system equipped with an information processing device 1 as a first embodiment. As shown in the figure, the information processing system of this embodiment includes a spectroscopic camera 2 together with the information processing device 1. Here, "spectroscopic camera" refers to a camera equipped with a spectroscopic sensor as a light-receiving sensor. A "spectroscopic sensor" is a light-receiving sensor that obtains multiple wavelength band images, which are wavelength characteristic analysis images of light from a subject. As will be described later, the spectroscopic camera 2 in this example is configured to obtain images of four or more wavelength bands, more than the three wavelength bands of R (red), G (green), and B (blue), as multiple wavelength band images. In this case, the wavelength band of each image is narrower than that of an RGB image. In other words, the wavelength resolution is increased. By increasing the wavelength resolution, it becomes possible to perform more detailed analysis of the subject.

[0013] In the following explanation, we will use the terms "spectral spectrum" and "spectral spectrum image," but "spectral spectrum" refers to information that shows the light intensity for each wavelength band. Furthermore, a "spectral image" refers to a collection of spectral data for each pixel. In other words, it refers to the "multiple wavelength band image" mentioned above.

[0014] In this example, the spectroscopic camera 2 generates a spectral image and a confirmation image as images related to the analysis of the subject. Here, "verification image" refers to an image used to verify the content of the image. When performing various analyses on a subject based on spectral images, it is conceivable to present the user with an image to confirm what kind of subject the spectral camera 2 is capturing. However, spectral images, i.e., individual images of multiple wavelength bands, represent the reflected light intensity of a specific narrow wavelength band and do not necessarily reproduce the shape of the subject as perceived by a person. Therefore, an image that can represent the shape of the subject as perceived by a person is generated as a confirmation image. Specifically, in this example, an RGB image is generated as a verification image. It is not mandatory to use an RGB image as a verification image; for example, a monochrome image could also be used.

[0015] The spectral image obtained by the spectroscopic camera 2, as well as the confirmation image, are transmitted to the information processing device 1.

[0016] The information processing device 1 is configured as a computer device equipped with a microcomputer having a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory). The information processing device 1 performs various processes related to the analysis of the subject based on the spectral image input from the spectroscopic camera 2. In addition, the information processing device 1 in this example uses the confirmation image input from the spectroscopic camera 2 to present various analysis-related information to the user.

[0017] (1-2. Example of an information processing device configuration) Figure 2 is a block diagram showing an example configuration of the information processing device 1. As shown in the figure, the information processing device 1 is equipped with a CPU 11. The CPU 11 executes various processes according to the program stored in the ROM 12 or the program loaded into the RAM 13 from the storage unit 19. The RAM 13 also appropriately stores data necessary for the CPU 11 to execute various processes.

[0018] The CPU 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output interface (I / F) 15 is also connected to this bus 14.

[0019] An input unit 16, consisting of controls or operating devices, is connected to the input / output interface 15. For example, the input unit 16 could be various controls or operating devices such as a keyboard, mouse, keys, dial, touch panel, touchpad, or remote controller. The input unit 16 detects user operations, and the CPU 11 interprets the signals corresponding to the input operations.

[0020] Furthermore, a display unit 17 consisting of an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 18 consisting of a speaker, are connected to the input / output interface 15, either as an integrated unit or as separate components. The display unit 17 is used for displaying various types of information and is composed of, for example, a display device provided on the casing of a computer device, or a separate display device connected to a computer device.

[0021] The display unit 17 displays images for various image processing tasks and videos to be processed on the display screen based on instructions from the CPU 11. The display unit 17 also displays various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the CPU 11.

[0022] The input / output interface 15 may also be connected to a storage unit 19 consisting of an HDD or solid memory, or a communication unit 20 consisting of a modem or the like.

[0023] The communications unit 20 performs communication processing via transmission lines such as the Internet, and communicates with various devices via wired / wireless communication, bus communication, etc.

[0024] The input / output interface 15 is also connected to a drive 21 as needed, and a removable recording medium 22 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted there.

[0025] Drive 21 allows data files, such as programs used for various processes, to be read from the removable recording medium 22. The read data files are stored in the storage unit 19, or images and sounds contained in the data files are output by the display unit 17 and the audio output unit 18. Computer programs and other data read from the removable recording medium 22 are installed in the storage unit 19 as needed.

[0026] In the information processing device 1 having the hardware configuration described above, for example, the software for processing in this embodiment can be installed via network communication by the communication unit 20 or via the removable recording medium 22. Alternatively, the software may be pre-stored in the ROM 12 or the storage unit 19, etc. The CPU 11 performs processing operations based on various programs, thereby executing the necessary information processing and communication processing as the information processing device 1.

[0027] Furthermore, the information processing device 1 is not limited to being a single computer device as shown in Figure 2, but may be configured as a system of multiple computer devices. These multiple computer devices may be systematized via a LAN (Local Area Network), or they may be located remotely via a VPN (Virtual Private Network) using the Internet, etc. The multiple computer devices may also include computer devices that function as a group of servers (cloud) available through cloud computing services.

[0028] (1-3. Example of a spectroscopic camera configuration) Figure 3 is a block diagram showing an example configuration of the spectroscopic camera 2. As shown in the figure, the spectral camera 2 includes a spectral sensor 25, a demosaicing processing unit 26, a spectral sensitivity correction processing unit 27, a wavelength conversion processing unit 28, a communication unit 29, and a control unit 30.

[0029] Figure 4 is a schematic diagram showing an example of the configuration of the pixel array section 25a of the spectroscopic sensor 25. As shown in the figure, the pixel array section 25a is formed with a spectral pixel unit Pu, which consists of multiple pixels Px that receive light in different wavelength bands, arranged in a predetermined pattern in two dimensions. The pixel array section 25a is composed of multiple spectral pixel units Pu arranged in two dimensions.

[0030] In the example shown in the figure, each spectral pixel unit Pu receives light from a total of eight wavelength bands, from λ1 to λ8, individually at each pixel Px. In other words, the number of wavelength bands that each spectral pixel unit Pu receives (hereinafter referred to as the "number of receiving wavelength channels") is "8". However, this is merely an illustrative example, and the number of receiving wavelength channels in a spectral pixel unit Pu can be at least multiple and can be set arbitrarily. Hereafter, we will define the number of light-receiving wavelength channels in the spectral pixel unit Pu as "Q".

[0031] In Figure 3, the demosaicing unit 26 performs demosaicing on the RAW image from the spectral sensor 4. As can be seen by referring to Figure 4, in this case, the RAW image shows the received light value for only one of the Q wavelength bands (in this example, one of the wavelength bands from λ1 to λ8) for each pixel. Therefore, demosaicing is performed so that information on the received light value for each of the Q wavelength bands can be obtained for each pixel. This demosaicing process yields Q wavelength band images.

[0032] The spectral sensitivity correction processing unit 27 performs spectral sensitivity correction on Q wavelength band images obtained by demosaicing processing by the demosaicing processing unit 26. Spectral sensitivity correction is a correction to remove the influence of the spectral sensitivity (sensitivity for each wavelength) of the spectral sensor 25 on the spectral image. Here, the spectral spectrum detected by the spectral sensor 25 can be said to be obtained by multiplying (convolving) the spectral spectrum of the subject light incident on the spectral sensor 25 by the spectral sensitivity of the spectral sensor 25. Therefore, in order to accurately reproduce the spectral spectrum of the subject, it is necessary to remove the influence of the spectral sensitivity of the spectral sensor 25 from the spectral spectrum detected by the spectral sensor 25.

[0033] Figure 5 is an illustrative diagram of the spectral sensitivity of the spectral sensor 25. As the spectral sensor 25 exhibits variations in sensitivity across different wavelength bands, a process to correct (i.e., remove) these variations is performed as spectral sensitivity correction processing. Figure 5 illustrates the spectral sensitivity when Q=6.

[0034] The wavelength conversion processing unit 28 performs wavelength conversion processing to generate a new image of a wavelength band different from any of the Q wavelength bands, based on the Q wavelength band images that have undergone spectral sensitivity correction processing by the spectral sensitivity correction processing unit 27. This wavelength conversion process is performed for each pixel using the light intensity values ​​of each of the Q wavelength bands. Specifically, the wavelength conversion processing unit 28 has predetermined conversion coefficients for each of the Q wavelength bands for calculating the light intensity of the target wavelength band to be generated. For each pixel of the input Q wavelength band image, the wavelength conversion processing unit 28 multiplies the light intensity value of each of the Q wavelength bands by the corresponding conversion coefficient and calculates the sum of these values. This allows the light intensity value of the target wavelength band to be generated to be obtained for each pixel, and enables the generation of a new wavelength band image that is different from any of the Q wavelength bands.

[0035] In this case, very high wavelength resolution is sometimes required for wavelength analysis of the subject. However, by performing the wavelength conversion process described above, it is possible to generate wavelength band images with a narrower bandwidth than each of the Q wavelength bands, thereby meeting the requirements for such high wavelength resolution. When generating wavelength band images that are narrower than each of the Q wavelength bands, it becomes possible to generate more than Q wavelength band images.

[0036] Furthermore, depending on the wavelength conversion process, it is possible to generate wavelength band images with a wider bandwidth than each of the Q wavelength bands. For example, it is possible to generate an RGB image from Q wavelength band images. In this example, this point is utilized, and the wavelength conversion processing unit 28 generates an RGB image, which is then used as the aforementioned verification image.

[0037] The control unit 30 is configured with, for example, a microcomputer having a CPU, ROM, and RAM, and the CPU performs overall control of the spectroscopic camera 2 by executing processing based on programs stored in ROM or loaded into RAM. For example, the control unit 30 issues commands to the spectral sensor 25, demosaicing processing unit 26, spectral sensitivity correction processing unit 27, and wavelength conversion processing unit 28 to execute or stop operations, etc. Furthermore, the control unit 30 instructs the wavelength conversion processing unit 28 to provide information indicating the target wavelength band for generation, causing the wavelength conversion processing unit 28 to generate a wavelength band image showing the light intensity of the specified wavelength band. In this example, the control unit 30 instructs the wavelength conversion processing unit 28 to provide information indicating the R, G, and B wavelength bands, causing the wavelength conversion processing unit 28 to generate an RGB image.

[0038] The communication unit 29 performs wired or wireless data communication with an external device. For example, the communication unit 29 may be configured to perform wired data communication with an external device in accordance with a predetermined wired communication standard such as the USB (Universal Serial Bus) communication standard, wireless data communication with an external device in accordance with a predetermined wireless communication standard such as the Bluetooth (registered trademark) communication standard, or wireless or wired data communication with an external device via a predetermined network such as the Internet. In this example, the communication unit 29 is capable of performing data communication using the communication method supported by the communication unit 20 of the information processing device 1, thereby enabling data communication between the spectroscopic camera 2 and the information processing device 1.

[0039] The control unit 30 can send and receive data with an external device via the communication unit 29. In this example, the control unit 30 transmits Q wavelength band images, which have undergone spectral sensitivity correction processing by the spectral sensitivity correction processing unit 27, to the information processing device 1 via the communication unit 29. The control unit 30 also transmits the RGB image generated by the wavelength conversion processing unit 28 to the information processing device 1 via the communication unit 29 as a confirmation image.

[0040] Although the above examples show spectral sensitivity correction processing and wavelength conversion processing being performed in the spectral camera 2, at least one of these spectral sensitivity correction processing and wavelength conversion processing can also be performed in the information processing device 1. In this case, the spectral camera 2 transmits the demosaiced spectral image to the information processing device 1. The information processing device 1 then performs spectral sensitivity correction processing and wavelength conversion processing on the spectral image transmitted in this manner.

[0041] In this case, it is also possible to use images captured by a camera separate from the spectral camera 2 for verification purposes.

[0042] (1-4. Information Processing Method as the First Embodiment) Figure 6 is a functional block diagram illustrating the functions of the CPU 11 of the information processing device 1 in this embodiment. As shown in the figure, the CPU 11 has the functions of an arithmetic unit F1, a presentation processing unit F2, an analysis processing unit F3, and an evaluation processing unit F4.

[0043] The calculation unit F1, assuming that the object to be analyzed using the spectral camera 2 (hereinafter referred to as the "assumed object") has been determined, calculates a deviation index value, which is an index value indicating the degree of discrepancy between the target spectrum, which is the spectral spectrum acquired using the spectral sensor 25, and the assumed spectrum, which is the spectral spectrum of the assumed object.

[0044] Here, the spectral spectrum of the reflected light from the subject incident on the spectral sensor 25 can be said to be the product of the reflection spectrum of the subject itself and the spectral spectrum of the light emitted by the light source that illuminates the subject. To give a specific example, if the assumed object is a plant and the light source is assumed to be the sun, the spectrum of the light incident on the spectral sensor 25 will be represented by the product of the reflection spectrum of the plant itself and the spectral spectrum of sunlight.

[0045] Figure 7 shows an example of the spectral spectrum of a light source, and Figure 8 shows an example of the reflectance spectrum of a hypothetical object. Specifically, Figure 7 shows examples of multiple light source spectra under different environmental conditions, such as the spectral spectrum of sunlight on a clear day and the spectral spectrum of sunlight on an overcast day. Figure 8 also illustrates several possible reflection spectra for a hypothetical object. For example, if the leaves of multiple plant species are defined as the hypothetical object, the reflection spectra of each leaf of those multiple plant species would correspond. Alternatively, if one plant species such as tomato, cabbage, or lettuce is the target, it is conceivable to define a leaf in a healthy state as the hypothetical object. In that case, the reflection spectra shown in Figure 8 could correspond to the reflection spectra of leaves in a healthy state from that particular plant species.

[0046] In this embodiment, as described above, multiple spectra are assumed for the light source spectrum depending on the environmental conditions, and the assumed object is assumed to have a certain range of characteristics (i.e., it has a certain range of variation in its spectral spectrum). Based on this idea, the spectral spectrum of the assumed object (assumed spectrum) is defined. Specifically, in this embodiment, when the number of spectra on the assumed object side is "e" and the number of spectra on the light source side is "f", the assumed spectrum is defined as a group of e × f spectra obtained by multiplying each of the e spectra on the object side by the f spectra on the light source side. For example, if e=100 and f=10, the expected spectrum is defined as a total of 1000 multiplicative spectra (the multiplicative spectra of the object and the light source).

[0047] In this embodiment, a deviation index value is calculated for the target spectrum (spectral spectrum acquired using the spectroscopic sensor 25) relative to the assumed spectrum. In this case, since the assumed spectrum is considered to be information consisting of a very large number of dimensions as described above, in this embodiment, the assumed spectrum is treated by reducing its dimensionality.

[0048] In this embodiment, the calculation unit F1 calculates a deviation index value, which is an index value indicating the degree of deviation between the target spectrum and the assumed spectrum, based on the eigenvalues ​​and eigenvectors of the feature quantities obtained by dimensionality reduction performed on the assumed spectrum.

[0049] In this example, we will illustrate the case where Singular Value Decomposition (SVD) is used as a method of dimensionality reduction.

[0050] For the assumed spectrum, when performing singular value decomposition as a dimensionality reduction method, the e × f spectra representing the assumed spectrum are replaced with a matrix representation. That is, a matrix is ​​generated in which these e × f spectra are arranged in the row direction. This matrix can be represented as a g x h matrix, where e × f = h, and g is the number of wavelength channels in each of the h assumed spectra. In this example, since the number of wavelength channels in the target spectrum used for comparison is "Q", g = Q.

[0051] When singular value decomposition is performed on such a hypothetical spectrum matrix,

number

[0052] For clarification, in the above [Equation 1], "U" is the left singular matrix, "V T (T represents the transpose matrix) is a right singular matrix, and these "U" and "V T The terms between " and " are singular value vectors. In singular value vectors, from σ1 to σ K This is a singular value. Here, "1" through "K" in the singular value σ represent the order of the principal component.

[0053] In this example, the calculation unit F1 calculates the deviation index value based on the eigenvalues ​​and eigenvectors obtained through dimensionality reduction using the singular value decomposition described above. Specifically, in the information processing device 1 of this example, the singular values ​​obtained by the singular value decomposition of [Equation 1] and the left singular matrix are stored in a memory device accessible by the CPU 11, such as the memory unit 19, and the arithmetic unit F1 calculates the deviation index value based on the singular values ​​and the left singular matrix stored in these memory devices.

[0054] As a result, the information processing device 1 no longer needs to perform calculations for singular value decomposition when calculating the deviation index value. Therefore, it is possible to reduce the processing load and speed up the calculation of deviation index values.

[0055] This section will explain a specific example of calculating the deviation index value. First, the calculation unit F1 in this example calculates the coefficient C according to the following [Equation 2].

number

[0056] The left and right singular matrices obtained by singular value decomposition represent the principal components in the spectral direction and the principal components in the data direction of the assumed spectrum group, respectively. Each spectrum can be understood as being represented by the sum of the products of these principal component vectors and the singular values. Therefore, if the target spectrum falls within the spectrum group that is assumed to be the target spectrum, then by multiplying the target spectrum from left to right by the transpose (inverse) of the left singular matrix, as shown in [Equation 2], and then multiplying by the reciprocal of the corresponding singular value, the weight coefficients of the principal components necessary to represent the target spectrum can be obtained. If a similar calculation is performed on an unexpected spectral spectrum, it becomes difficult to express it straightforwardly using the sum of the products of the principal components, resulting in the output of coefficients with extremely large absolute values. In other words, by performing the calculation in [Equation 2] and evaluating the magnitude of the obtained coefficient C, it becomes possible to estimate how much an arbitrary spectral spectrum deviates from (or, in other words, how similar) an assumed spectrum is to an assumed spectrum, without having to prepare the negative examples mentioned earlier.

[0057] Here, the order of principal components to be used in calculating the deviation index should be determined appropriately based on the actual implementation. Here, the upper limit of the order of the principal components referenced in calculating the deviation index is defined as "n". If an upper limit n is set, the coefficient C described above is calculated for each principal component from the first to the nth order (i.e., k ranges from 1 to n).

[0058] As described above, the coefficients C calculated for each dimension from the first to the nth order can be used as deviation index values ​​as they are, but for the first-order coefficient C, it is also possible to calculate it as the first-order component deviation degree D1 as shown in [Equation 3] below.

number

[0059] Then, the calculation unit F1 in this example calculates the linear component deviation D1 and the coefficients C2 calculated for each dimension from the second to the nth dimension. n Using these, the overall deviation degree Da is calculated by the following [Equation 4].

number

[0060] Here, the calculation unit F1 calculates each deviation index value, such as the coefficient C, the first-order component deviation degree D1, and the overall deviation degree Da, for each pixel of the spectral image transmitted from the spectral camera 2. This allows us to calculate an index value for each pixel in the spectral image that indicates the degree of deviation from the expected spectrum.

[0061] In Figure 6, the presentation processing unit F2 performs the process of presenting the deviation index values ​​calculated for each pixel by the calculation unit F1 to the user using a two-dimensional map. In this example, the display processing unit F2 performs the process of displaying the aforementioned confirmation screen along with the two-dimensional map on the same screen.

[0062] Figure 9 shows an example of a display screen generated by the display processing unit F2. In this example, this presentation screen is displayed on the display unit 17. As shown in the figure, the display screen is provided with a deviation map display area Md for displaying a two-dimensional map of deviation index values ​​and a confirmation image display area Im for displaying confirmation images. The display processing unit F2 performs the processing of displaying a deviation map, which is a two-dimensional map of the total deviation degree Da calculated for each pixel, in the deviation map display area Md, and displaying confirmation images transmitted from the spectral camera 2 in the confirmation image display area Im.

[0063] The illustrated example shows a deviation map for a hypothetical object, where the overall deviation Da for each pixel is represented in grayscale. Specifically, white indicates the lowest deviation, and black indicates the highest deviation. As can be seen from the confirmation image in the figure, in this case, the imaging range contains a mixture of parts with plant leaves and other parts such as soil. Generally, the overall deviation Da value is low for the plant parts, while the overall deviation Da value is high for the non-plant parts.

[0064] By presenting the deviation index values ​​for each pixel as map information in this way, users can intuitively grasp the two-dimensional distribution pattern of values ​​within the image.

[0065] In Figure 9, the verification image and the deviation map are shown in separate areas on the screen; however, it is also possible to superimpose the deviation map onto the verification image.

[0066] In Figure 6, the analysis processing unit F3 performs analysis processing on the subject based on the deviation index value calculated by the calculation unit F1. Specifically, in this example, the analysis processing unit F3 determines whether the target object is a hypothetical object based on the relationship between the deviation index value calculated for the target spectrum and the threshold value. In this example, the analysis processing unit F3 determines whether the target object is a hypothetical object for each pixel by determining whether the overall deviation degree Da calculated for each pixel by the calculation unit F1 is less than or equal to a predetermined threshold value.

[0067] The evaluation processing unit F4 calculates an evaluation value different from the deviation index value based on the target spectrum, focusing only on the target spectrum determined to be that of a hypothetical object by the analysis processing unit F3. Specifically, in this example, the evaluation processing unit F4 calculates an evaluation value different from the deviation index value based only on the spectral spectrum of the pixels determined to be that of a hypothetical object, using the information from the pixel-by-pixel determination results performed by the analysis processing unit F3. In this example, assuming plants as the hypothetical object, we calculate the NDVI (Normalized Difference Vegetation Index) as an evaluation value different from the deviation index value. The NDVI is a standardized measure of healthy vegetation and is calculated as ((IR-R) / (IR+R)). Here, IR represents the pixel value in the near-infrared wavelength band, and R represents the pixel value in the red wavelength band. The NDVI is a value calculated within the range of -1 to 1, and a value closer to 1 indicates better health.

[0068] In this example, on the screen displaying the deviation map illustrated in Figure 9, an execution button B1 is placed to instruct the detailed analysis as the calculation of NDVI described above. The evaluation processing unit F4 performs the calculation of NDVI when this execution button B1 is operated.

[0069] As described above, by calculating evaluation values ​​only for the target spectrum determined to be that of the assumed object, it becomes possible to prevent evaluation values ​​from being calculated for objects other than the assumed object. Therefore, it is possible to prevent noise from occurring in the evaluation values ​​and improve the accuracy of the analysis of the target object.

[0070] Furthermore, the NDVI is merely an illustrative example of the evaluation value calculated by the evaluation processing unit F4 based on the target spectrum, and is not the only limiting factor. Although not shown in the diagram, the evaluation processing unit F4 can also present the user with a two-dimensional map of the calculated evaluation values.

[0071] As can be understood from the above explanation, pixels with low deviation index values ​​such as the overall deviation index Da can be determined to be pixels of objects of the assumed object class. Therefore, by determining whether the deviation index value is below a threshold for each pixel, semantic segmentation (the process of associating information that identifies whether or not each pixel in an image belongs to the corresponding class) can be achieved.

[0072] Furthermore, the nature of deviation index values ​​such as the overall deviation degree Da changes depending on what kind of object is assumed as the assumed object for the assumed spectrum. For example, if the assumed object is a "healthy plant," the deviation index value can also be used as an index value indicating the degree of health of the plant as the target object.

[0073] (1-5. Processing Procedure) Figure 10 is a flowchart showing an example of a specific processing procedure that the CPU 11 should execute in order to realize the information processing method as the first embodiment described above. First, in step S101, the CPU 11 calculates the overall deviation Da for each pixel. Specifically, the CPU 11 calculates the overall deviation Da based on the singular values ​​and left singular matrices (from 1st to nth order in this example) stored in a predetermined storage device such as the memory unit 19, as described above, and the spectral image transmitted from the spectral camera 2, using the calculation method described above with reference to [Equation 2] to [Equation 4].

[0074] In step S102, following step S101, the CPU 11 performs the process of presenting the deviation map. That is, it performs the process of displaying a presentation screen on the display unit 17, which includes a deviation map that is a two-dimensional map of the total deviation Da calculated for each pixel, as exemplified in Figure 9.

[0075] In step S103, following step S102, the CPU 11 determines whether or not an instruction for detailed analysis has been given. That is, it determines whether or not the execution button B1 located on the presentation screen shown in Figure 9 has been operated.

[0076] In step S103, if it is determined that the execute button B1 has been pressed and a detailed analysis has been ordered, the CPU 11 proceeds to step S104 and executes the deviation degree determination process for each pixel. In other words, in this example, it is determined whether the overall deviation degree Da for each pixel is below a threshold.

[0077] In step S105, following step S104, the CPU 11 performs an evaluation value calculation process targeting only pixels with a low degree of deviation. That is, the CPU 11 calculates an evaluation value based on the spectral spectrum, specifically the NDVI in this example, targeting only pixels whose overall deviation Da is determined to be below a threshold by the deviation degree determination process in step S104.

[0078] In step S106, following step S105, the CPU 11 performs the process of presenting evaluation result information. This process of presenting evaluation result information could include, for example, presenting the calculated evaluation values ​​to the user as numerical information, or presenting a two-dimensional map of evaluation values ​​to the user. However, the specific presentation method is not particularly limited; at the very least, the CPU 11 should perform the process of presenting information indicating the evaluation values ​​to the user.

[0079] The CPU 11 completes the series of processes shown in Figure 10 after executing the process in step S106.

[0080] Furthermore, if the CPU 11 determines in step S103 that the execute button B1 has been operated and that a detailed analysis has been instructed, it also completes the series of processes shown in Figure 10.

[0081] <2. Second Embodiment> In the first embodiment, the assumed spectrum was defined as the spectral spectrum at the stage of incidence to the spectral sensor 25. Therefore, for the target spectrum, the spectral spectrum after correction of the spectral sensitivity by the spectral sensitivity correction processing unit 27 was input to the information processing device 1. However, in this case, calculating the deviation index value requires performing spectral sensitivity correction processing on the output of the spectral sensor 25.

[0082] Therefore, in the second embodiment, the assumed spectrum is defined not at the incidence stage to the spectral sensor 25, but at the output stage from the spectral sensor 25 (however, after demosaicing), thereby making it possible to omit the spectral sensitivity correction process when calculating the deviation index value.

[0083] Figure 11 is a diagram illustrating an example configuration of the information processing device 1A as a second embodiment. Note that the information processing device 1A differs from the information processing device 1 only in some functions of the CPU 11; other components are the same as those of the information processing device 1, and are therefore not shown here.

[0084] In the following explanation, parts that are the same as those already explained will be denoted by the same reference numeral and their explanation will be omitted.

[0085] As shown in the figure, the information processing device 1A is equipped with a CPU 11A instead of a CPU 11. The CPU 11A differs from the CPU 11 in that it has an arithmetic unit F1A instead of an arithmetic unit F1. The calculation unit F1A calculates the deviation index value based on the eigenvalues ​​and eigenvectors of the feature quantities obtained by dimensionality reduction performed on the assumed spectrum into which the spectral sensitivity characteristics of the spectral sensor 25 are convolved.

[0086] In the second embodiment, the assumed spectrum is defined as a spectrum obtained by multiplying the reflection spectrum of the object, the spectrum of the light source, and the spectral sensitivity characteristics of the spectral sensor 25. The calculation unit F1A calculates deviation index values ​​such as coefficient C, dimensional deviation degree D, and overall deviation degree Da using a method similar to the method described in the first embodiment, based on the singular values ​​and left singular matrix obtained by singular value decomposition for the assumed spectrum defined in this way, and the target spectrum.

[0087] In the second embodiment, the spectral camera 2A, as illustrated in Figure 12, receives a spectral image output from the demosaicing processing unit 26, rather than the output from the spectral sensitivity correction processing unit 27, as input to the control unit 30. The control unit 30 transmits the spectral image (demosaicized) from the stage before the spectral sensitivity correction processing is performed to the information processing device 1A as an image of the target spectrum to be compared with the assumed spectrum.

[0088] Furthermore, the spectral sensitivity correction processing unit 27 in the spectral camera 2A is provided so that the wavelength conversion processing unit 28 can generate an RGB image for verification purposes. In the spectral camera 2A, if only the calculation of the deviation index value is considered, the spectral sensitivity correction processing unit 27 can be omitted.

[0089] Furthermore, in the above, the assumed spectrum was the spectrum of Qch (channel) before wavelength conversion by the wavelength conversion processing unit 28, and therefore the target spectrum for calculating the deviation index value was also the spectrum of Qch (channel) before wavelength conversion by the wavelength conversion processing unit 28. However, it is also possible to assume the spectrum based on the number of channels after wavelength conversion by the wavelength conversion processing unit 28 as the assumed spectrum, and to adopt a configuration in which the target spectrum for calculating the deviation index value is also the spectrum based on the number of channels after wavelength conversion by the wavelength conversion processing unit 28. In this configuration, where the target spectrum is input as the spectrum obtained by wavelength conversion by the wavelength conversion processing unit 28, the spectral sensitivity correction process can be omitted when calculating the deviation index value by using a spectrum obtained by multiplying the incident spectrum by the spectral sensitivity characteristics of the sensor and the wavelength conversion characteristics of the wavelength conversion processing unit 28 as the assumed spectrum.

[0090] <3. Third Embodiment> The third embodiment assumes that machine learning of an AI (artificial intelligence) model is performed using spectral images as input data for inference processing, and uses the deviation index value of the embodiment as an evaluation index for spectral images used as input images for machine learning.

[0091] Specifically, in this case, the AI ​​model could be one that uses the spectral data of each pixel as input to perform object recognition processing as semantic segmentation. For example, one could be an AI model that performs inference tasks such as class identification of leaves, soil, and other obstacles on a pixel-by-pixel basis.

[0092] When preparing a training dataset, annotation work is performed on the training input images. This annotation work involves operations such as specifying the regions in the target image where objects of the target class to be identified exist, and specifying the corresponding class information for each specified region, on the display screen.

[0093] When using images actually sensed by a spectral sensor as training input images, the training input images may contain noisy areas due to optical factors, etc. For example, it is known that noise called false color can occur at the edges of the subject (see, for example, the "X" part in Figure 13). Furthermore, in addition to these noise areas, training input images may also contain parts where the subject is in a state different from what was expected. For example, this could include parts of a plant that are withered or discolored due to some factor. In the following, the noise caused by factors not related to the subject, such as the false colors mentioned above, and the unexpected parts, such as the withered state mentioned above, will be collectively referred to as "abnormal parts."

[0094] In the third embodiment, we propose a method in which, for spectral images prepared as candidate input images for training, a deviation index value is calculated for the image used as the work target during the annotation process, and the presence or absence of the above-mentioned abnormal parts is determined based on the calculated deviation index value.

[0095] Figure 14 is a diagram illustrating an example configuration of the information processing device 1B as a third embodiment. Note that the information processing device 1B differs from the information processing device 1 only in some functions of the CPU 11; other components are the same as those of the information processing device 1, and are therefore not shown here.

[0096] Information processing device 1B differs from information processing device 1 in that it has CPU 11B instead of CPU 11, and CPU 11B differs from CPU 11 in that it has presentation processing device F2B instead of presentation processing device F2, and the analysis processing device F3 and evaluation processing device F4 are omitted. In the third embodiment, if a spectrum is defined as the assumed spectrum by multiplying the spectral sensitivity characteristics of the spectral sensor 25, similar to the second embodiment, the configuration will have a calculation unit F1A instead of a calculation unit F1.

[0097] In the third embodiment, the calculation unit F1 inputs a spectral image prepared as a candidate for training input images as the spectral image used to calculate the deviation index value. Specifically, the spectral image that was the target of the annotation work described above is used to calculate the deviation index value.

[0098] The presentation processing unit F2B performs a process to present to the user information indicating pixels whose deviation index value is greater than or equal to a threshold value, based on the deviation index value calculated for each pixel by the calculation unit F1.

[0099] Figure 15 shows an example of a display screen generated by the display processing unit F2B. In this example, this presentation screen is displayed on the display unit 17. As shown in the diagram, the display screen in this case shows the spectral image that was the target of the annotation work. If there is a pixel in this target image whose deviation index value is above a threshold, the display processing unit F2B performs a process to present information indicating the corresponding pixel to the user (see the "S" part in the diagram).

[0100] Furthermore, if there are pixels in the image being worked on whose deviation index value is above a threshold, the presentation processing unit F2B will display exclusion suggestion information on the presentation screen, such as "A region with a high degree of deviation has been detected. Do you want to exclude it from the training dataset?", and will also display "Yes" button B2 and "No" button B3 to allow the user to choose whether or not to exclude the image.

[0101] If the "Yes" button is pressed, CPU11B performs an exclusion process to remove the image from the training dataset. This exclusion process could involve assigning an exclusion flag to the image, or, if there is a memory area where images for the training dataset should be stored, preventing the image from being stored in that memory area. However, the specific processing method is not limited as long as it prevents the image from being managed as part of the training dataset.

[0102] Figure 16 is a flowchart showing an example of a specific processing procedure that the CPU 11B should execute in order to realize the information processing method as the third embodiment described above. First, in step S201, CPU 11B calculates the total deviation Da for each pixel of the spectral image being worked on.

[0103] In the following step S202, the CPU 11B determines whether the total deviation Da for each pixel is above a threshold. This determination is equivalent to determining whether the target pixel is an abnormal pixel corresponding to an abnormal area.

[0104] In the subsequent step S203, the CPU 11B determines whether or not there are pixels whose total deviation Da is greater than or equal to a threshold. If there are any pixels where the overall deviation Da is greater than or equal to a threshold, the CPU 11B proceeds to step S204 and performs the process of presenting the relevant region and exclusion suggestion information. As information to indicate the relevant region (abnormal region), it is conceivable to present information indicating the relevant region using a predetermined color, as exemplified in the "S" part of Figure 15.

[0105] In step S205, following step S204, the CPU 11B determines whether or not an exclusion instruction was given. Specifically in this example, it determines whether or not either the "Yes" button B2 or the "No" button B3, as illustrated in Figure 15, was pressed. If the "Yes" button B2 was pressed, the CPU 11B determines that an exclusion instruction was given, and if the "No" button B3 was pressed, the CPU 11B determines that no exclusion instruction was given.

[0106] If the "Yes" button B2 is pressed in step S205 and it is determined that an exclusion instruction has been given, the CPU 11B proceeds to step S206 to execute the exclusion process for the spectral image of the work target, and completes the series of processes shown in Figure 16.

[0107] Furthermore, if in step S203 it is determined that there were no pixels where the total deviation Da was greater than or equal to the threshold, the CPU 11B completes the series of processes shown in Figure 16. In addition, if in step S205 it is determined that there was no exclusion instruction (i.e., the "No" button B2 was pressed), the CPU 11B also completes the series of processes shown in Figure 16.

[0108] In the example above, we showed how to have the user specify whether or not to exclude images in which abnormal parts were detected based on the deviation index value, and then exclude the relevant images if an exclusion instruction was given. However, instead, it is also possible to adopt a method that automatically excludes images in which abnormal parts were detected.

[0109] Figure 17 is a diagram illustrating an example configuration of an information processing device 1C as another example of a third embodiment that performs such automatic exclusion processing. The information processing device 1C differs from the information processing device 1B in that it has a CPU 11C instead of a CPU 11B, and the CPU 11C differs from the CPU 11B in that it has an exclusion processing device F5 instead of a presentation processing device F2B.

[0110] The exclusion processing unit F5 performs a process to exclude spectral images containing pixels with deviation index values ​​above a threshold from the AI ​​model's training dataset. In other words, it automatically performs the aforementioned exclusion process for spectral images containing pixels with deviation index values ​​above a threshold.

[0111] Figure 18 is a flowchart showing an example of a specific processing procedure that CPU 11C should perform. CPU 11C performs the processing shown in Figure 18 for each spectral image prepared as a candidate input image for training the AI ​​model.

[0112] In step S301, the CPU 11C calculates the total deviation Da for each pixel of the spectral image to be processed, and in the following step S302, it determines whether the total deviation Da for each pixel is above a threshold. Furthermore, in the following step S303, the CPU 11C determines whether there are any pixels where the total deviation Da is above the threshold, and if there are any pixels where the total deviation Da is above the threshold, it proceeds to step S304, performs the process of excluding the spectral image to be processed, and then completes the series of processes shown in Figure 18.

[0113] Furthermore, if CPU11C determines in step S303 that there are no pixels where the total deviation Da is greater than or equal to the threshold, it completes the series of processes shown in Figure 18.

[0114] In the third embodiment, it is also conceivable that the threshold value for deviation index values ​​to be excluded from the training dataset can be variably set according to user operations or other factors.

[0115] Furthermore, while the above example shows how spectral images containing pixels with deviation index values ​​above a threshold are excluded from the AI ​​model's training dataset, exclusion based on deviation index values ​​is not limited to image-level exclusion; it can also be performed at the pixel level. Specifically, in training an AI model that performs inference tasks as semantic segmentation, a filtering process is performed to prevent pixels from being used for training, such as by masking pixels in the spectral image used as training input where the deviation index value is above a threshold. This method can also improve training accuracy.

[0116] When the above method is adopted, the information processing device can be described as having the following configuration: The arithmetic unit is equipped with a filtering unit that calculates a deviation index value for each pixel of the spectral image used as the learning input image for the AI ​​model, and processes the data so that pixels with a deviation index value above a threshold are not used for learning the AI ​​model.

[0117] In this case, as in the third embodiment, when calculating deviation index values ​​for spectral images that are the target images for annotation work for machine learning, it is also possible to switch the calculation method for deviation index values ​​for each class in the annotation. As a concrete example, annotation work sometimes involves specifying in advance which class the label to be assigned belongs to, and then applying the label to the image. For instance, if the label information to be assigned is "leaf," the user would declare to the annotation tool that they are going to perform the annotation of "leaf," and then follow the procedure of enclosing the area in the image that they consider to be a "leaf" with a bounding box or similar. In such cases, it is conceivable to switch the calculation method for the deviation index for each class. For example, when assigning the label "leaf," the deviation index is calculated based on the eigenvalues ​​and eigenvectors of the features obtained for subjects belonging to the "leaf" class. Alternatively, when assigning the label "soil," the deviation index is calculated based on the eigenvalues ​​and eigenvectors of the features obtained for subjects belonging to the "soil" class. This makes it possible to improve the accuracy of detecting abnormal parts.

[0118] Furthermore, depending on the type of class, it may be possible to omit the detection of anomalies using deviation index values. For example, for classes that represent unspecified objects, such as the "Other" class, rather than classes that represent specific objects such as "Leaf" or "Soil," it may be possible to omit the detection of anomalies using deviation index values.

[0119] <4. Another example related to deviation index values> Here, as explained above using coefficient C and the first-order component deviation degree D1, we have given examples of calculating deviation index values ​​for each dimension. It is also possible to display such dimensional deviation index values ​​on the analysis screen. Figure 19 is a diagram illustrating an example configuration of information processing device 1D, which presents dimensional deviation index values ​​in this way as a first alternative example. The information processing device 1D differs from the information processing device 1 in that it has a CPU 11D instead of a CPU 11, and the CPU 11D differs from the CPU 11 in that it has a presentation processing device F2D instead of a presentation processing device F2. The presentation processing unit F2D performs the process of presenting the dimensional deviation index values, such as coefficients C and linear component deviation degree D1 calculated by the calculation unit F1 for each dimension, to the user via the display unit 17. In this example, the presentation processing unit F2D performs the process of presenting the dimensional deviation index values ​​calculated by the calculation unit F1 for each pixel to the user using a two-dimensional map.

[0120] Figure 20 shows an example of a map image of the deviation evaluation value calculated for each dimension. Specifically, the deviation index value for each dimension is denoted as coefficient C. Figure 20A shows a map image of coefficient C calculated for the first principal component, while Figures 20B, 20C, and 20D show examples of map images of coefficient C calculated for the second principal component, the third principal component, and the fourth principal component, respectively. The spectral image used to calculate the coefficient C is the same image used to calculate the overall deviation Da in Figure 9.

[0121] Furthermore, when presenting deviation index values ​​for each dimension, it is conceivable that deviation index values ​​up to a specified order could be presented in response to an operation to specify the order (an operation to specify the order to be included in the deviation degree), such as by manipulating a slider bar.

[0122] In the explanation so far, we have given an example of calculating an overall deviation index value by summing the deviation index values ​​obtained for each dimension from low-order to high-order, as explained as the overall deviation index Da. However, it is also possible to calculate independent deviation index values ​​for low-order and high-order dimensions.

[0123] The deviation index values ​​used as coefficient C in the first embodiment tend to indicate the general characteristics of the object when calculated for the lower-order principal components, and the more specific characteristics of the object when calculated for the higher-order principal components. Therefore, by referring only to the lower-order deviation index values, it is possible to evaluate differences in the class of objects, such as the evaluation of plant-likeness, and by referring only to the higher-order deviation index values, it is possible to evaluate the specific state of the object, such as the health and growth status of a plant.

[0124] Figure 21 is a diagram illustrating a configuration example of an information processing device 1E as a second alternative example in which deviation index values ​​for low-order only and deviation index values ​​for high-order only are calculated separately. The information processing device 1E differs from the information processing device 1 in that it has a CPU 11E instead of a CPU 11, and the CPU 11E differs from the CPU 11 in that it has an arithmetic unit F1E instead of an arithmetic unit F1, and an analysis processing unit F3E instead of an analysis processing unit F3.

[0125] The calculation unit F1E calculates the deviation index value (e.g., coefficient C) for only the low-order principal components. Here, "lower order" refers, for example, up to the second order. Alternatively, it may include up to the third order. Or, it may refer only to the first order.

[0126] Furthermore, the calculation unit F1E calculates the deviation index value for only the higher-order principal components. Here, "higher order" refers to dimensions of the third order or higher, for example. Alternatively, it could refer to dimensions of the fourth or fifth order or higher.

[0127] Higher-order deviation index values ​​can serve as indicators of the degree of deviation regarding specific conditions, such as the health status of the target object, provided that the classes of the assumed object and the target object match. Therefore, by calculating deviation index values ​​for higher-order factors only, it is possible to evaluate the specific state of the target object.

[0128] The analysis processing unit F3E determines whether the deviation index value calculated by the calculation unit F1E for only the low-order principal components is below a threshold. Specifically, in this example, the analysis processing unit F3E determines whether the deviation index value calculated for each pixel of the target spectral image, based only on the low-order principal components, is below a threshold.

[0129] By determining whether the deviation index calculated using only the low-order principal components is below a threshold, it is possible to determine whether the object belongs to a specific class. By not using higher-order deviation index values, the accuracy of determining whether an object belongs to a specific class can be improved.

[0130] Furthermore, in the information processing device 1E, as in the case of information processing device 1, it is conceivable to calculate evaluation values ​​such as NDVI only for pixels whose deviation index value is below a threshold.

[0131] In the explanation so far, singular value decomposition has been used as an example of a dimensionality reduction method. However, dimensionality reduction methods are not limited to this, and other methods such as principal component analysis (PCA), linear discriminant analysis (LDA), t-SNE (t-distributed stochastic neighbor embedding), and independent component analysis (ICA) can also be considered. In either case, the deviation index can be calculated based on the eigenvalues ​​and eigenvectors of the features obtained through dimensionality reduction.

[0132] <5. Variation> Furthermore, the embodiments are not limited to the specific examples described above, and various other modified configurations can be adopted. For example, in the first embodiment, an example was given in which a deviation index value is calculated for each pixel of the spectral image. However, it is also conceivable that the deviation index value be calculated for pixels within the bounding box detected by object detection processing performed on an image captured of the same subject as the subject that the spectral sensor 25 is sensing.

[0133] Figure 22 shows an example of a bounding box BB detected by the object detection process described above. Here, the object detection process is shown as an example where object detection is performed with flower pots as the target class. Specifically, since two flower pots were captured in the image in this case, the object detection process detected bounding boxes BB for each flower pot.

[0134] Figure 23 is a diagram illustrating an example configuration of the information processing device 1F as a modified example for calculating deviation index values ​​for bounding box BB. The information processing device 1F differs from the information processing device 1 in that it has a CPU 11F instead of a CPU 11. As shown in the figure, the CPU 11F has an object detection processing unit F6, a calculation unit F1F, and an analysis processing unit F3F.

[0135] The object detection processing unit F6 performs object detection processing using an AI model, for example, on an RGB image transmitted from the spectroscopic camera 2 as a verification image. Furthermore, the captured image, which is of the same subject as the subject being sensed by the spectral sensor 25, is not limited to the RGB image described above. For example, an image captured by a camera other than the spectral camera 2 could also be used.

[0136] The calculation unit F1F calculates deviation index values, such as coefficient C, first-order component deviation D1, and overall deviation Da, for pixels within the bounding box BB detected by the object detection processing performed by the object detection processing unit F6.

[0137] As described above, by calculating the deviation index value only for pixels within the bounding box BB, when the object detected by the object detection process is a subject that includes the expected object, such as a flowerpot, it becomes unnecessary to calculate the deviation index value for the entire spectral image, thereby reducing the processing burden. Furthermore, since deviation index values ​​are calculated for the target subjects (subjects of the target class) detected by the object detection process, it becomes possible to analyze the state of those target subjects.

[0138] The analysis processing unit F3F performs at least one determination for each pixel within the bounding box BB: whether or not the deviation index value is below a threshold.

[0139] In this case, one possible analysis of the subject based on the deviation index value is to perform an analysis to determine the authenticity of the subject. For example, consider the two flowerpots shown in Figure 21. Suppose one contains a living plant, and the other contains a fake plant made of resin or similar material. In this case, if the deviation index value calculated for the bounding box BB is high, it can be estimated that the subject is a real plant. Conversely, if the deviation index value is low, it can be estimated that the subject is a fake plant.

[0140] Specifically, in this example, the analysis processing unit F3F performs the analysis to determine whether the proportion of pixels within the bounding box BB whose deviation index value is below a threshold is above a certain value. If the proportion is above a certain value, it determines that the result is true; otherwise, it determines that the result is false.

[0141] In the explanation so far, we have given an example where the first-order component deviation D1 and coefficient C are added without weight in the calculation of the overall deviation Da. However, depending on the application of the overall deviation Da, it is also possible to assign predetermined weights to these first-order component deviations D1 and coefficient C. For example, to improve the accuracy of class determination, one might consider increasing the weights of lower-order elements. Alternatively, to primarily evaluate differences in specific characteristics, one might consider increasing the weights of higher-order elements.

[0142] Furthermore, while the above explanation has illustrated a configuration in which a calculation unit for calculating deviation index values ​​is provided in a separate device from the spectroscopic camera 2, it is also conceivable that the spectroscopic camera 2 itself may be equipped with such a calculation unit.

[0143] <6. Summary of Embodiments> As described above, the information processing device as an embodiment (1, 1A, 1B, 1C, 1D, 1E, 1F) includes a calculation unit that calculates a deviation index value, which is an index value indicating the degree of deviation between the target spectrum, which is a spectral spectrum acquired using a spectroscopic sensor, and the assumed spectrum, based on the eigenvalues ​​and eigenvectors of feature quantities obtained by dimensionality reduction performed on the assumed spectrum, which is a spectral spectrum of a hypothetical object. The deviation index value described above makes it possible to quantitatively evaluate the similarity between the spectral spectrum obtained using a spectroscopic sensor and the spectral spectrum of a hypothetical object, without using a spectral spectrum as a negative example. Furthermore, since this deviation index value is calculated based on dimensionality-reduced information, the amount of computation is reduced. Therefore, it is possible to calculate a quantitative evaluation index value that shows the similarity between the spectral spectrum acquired using a spectroscopic sensor and the spectral spectrum of a hypothetical object, without using a spectral spectrum as a negative example, while reducing the amount of computation.

[0144] Furthermore, in the information processing apparatus as an embodiment, the spectral sensor is a two-dimensional sensor, the target spectrum is acquired for each pixel, and the calculation unit calculates the deviation index value for each pixel. This allows us to calculate an index value for each pixel in the spectral image that indicates the degree of deviation from the expected spectrum.

[0145] Furthermore, the information processing device as an embodiment includes a first presentation processing unit (presentation processing unit F2) that performs processing to present the deviation index value calculated for each pixel to the user using a two-dimensional map. This allows the user to see the degree of similarity between objects captured in a spectral image and a hypothetical object at a pixel-level granularity, and by presenting this as map information, the user can intuitively grasp the two-dimensional distribution of values ​​within the image.

[0146] Furthermore, the information processing device as an embodiment includes an analysis processing unit (F3, F3E, F3F) that performs analysis processing of the subject based on the deviation index value. Since the deviation index value indicates the degree of deviation from the assumed spectrum, it is possible to perform subject analysis from the perspective of whether or not the object in question corresponds to the assumed object. Furthermore, if the deviation index value is calculated using higher-order principal components, it will indicate the degree of deviation from a more specific perspective (for example, health status in the case of plants) rather than an abstract perspective such as the type of object. Therefore, when using higher-order deviation index values, it is also possible to perform state analysis of the subject. By using deviation index values ​​in this way, it is possible to perform analysis on the subject from various perspectives.

[0147] Furthermore, in the information processing apparatus as an embodiment, the analysis processing unit determines whether or not the target object is a hypothetical object based on the relationship between the magnitude of the deviation index value calculated for the target spectrum and the threshold value. The deviation index value can be used as likelihood information regarding whether or not the target object is the assumed object, and as described above, it is possible to determine whether or not the target object is the assumed object based on the relationship between the deviation index value and the threshold value. Since the determination of whether or not a target object is a target object can be made based on the spectral spectrum acquired using a spectroscopic camera, there is no need to perform separate object recognition processing using captured images, and the processing burden required to determine whether or not a target object is a target object can be reduced.

[0148] Furthermore, the information processing device as an embodiment includes an evaluation processing unit (F4) that calculates an evaluation value different from the deviation index value based on the target spectrum, targeting only the target spectrum determined to be a hypothetical object by the analysis processing unit. This makes it possible to prevent evaluation values ​​from being calculated for objects other than the intended object. Therefore, it is possible to prevent noise from occurring in the evaluation values ​​and improve the accuracy of the analysis of the target object.

[0149] Furthermore, in the information processing apparatus as an embodiment, the calculation unit calculates a deviation index value for pixels within the bounding box detected by object detection processing performed on an image captured of the same subject as the subject being sensed by the spectroscopic sensor. This eliminates the need to calculate the deviation index value for the entire spectral image when the object detected by the object detection process is a subject that includes the expected object, such as a flowerpot, thereby reducing the processing burden. Furthermore, a deviation index value is calculated for the target object (object of the target class) detected by the object detection process, allowing for analysis of the state of the target object. For example, if the detected target object is a flowerpot, the deviation index value can be used to determine whether the object planted in the pot is a real living plant or a fake made of resin, etc., thus enabling analysis of the state of the target object.

[0150] Furthermore, in the information processing apparatus as an embodiment, the arithmetic unit calculates the deviation index value based on the eigenvalues ​​and eigenvectors obtained by dimensionality reduction through singular value decomposition. Singular value decomposition is one of the appropriate methods for dimensionality reduction. Dimensionality reduction can be performed appropriately, improving the accuracy of the deviation index.

[0151] Furthermore, in the information processing apparatus as an embodiment, the calculation unit calculates a total deviation value, which comprehensively indicates the degree of deviation in multiple dimensions from the first order to a predetermined order, as a deviation index value. This allows for a comprehensive evaluation of the degree of discrepancy between the assumed spectrum and the target spectrum, independent of the dimensions of the principal components.

[0152] Furthermore, in the information processing apparatus as an embodiment, the singular values ​​and left singular matrix obtained by singular value decomposition are stored in a memory device, and the calculation unit calculates the deviation index value based on the singular values ​​and left singular matrix stored in the memory device. This eliminates the need for information processing devices that calculate deviation index values ​​to perform calculations for singular value decomposition when calculating deviation index values. Therefore, it is possible to reduce the processing load and speed up the calculation of deviation index values.

[0153] Furthermore, in the information processing apparatus as an embodiment, the calculation unit calculates the deviation index value for only the low-order principal components. The deviation index values, when calculated for lower-order principal components, function as indicators of characteristics at the class level of the subject, such as an index of plant-likeness. Therefore, by calculating the deviation index value for only the low-order principal components, it is possible to obtain an index value suitable for identifying the class of an object.

[0154] Furthermore, the information processing device as an embodiment includes an analysis processing unit (F3E) that performs analysis processing of the subject based on the deviation index value, and the analysis processing unit determines whether the deviation index value calculated for only the low-order principal components is below a threshold. This allows for the determination of whether an object belongs to a specific class. By not using higher-order deviation indices, the accuracy of determining whether an object belongs to a specific class can be improved.

[0155] Furthermore, in the information processing apparatus as an embodiment, the arithmetic unit calculates deviation index values ​​for only the higher-order principal components. Higher-order deviation index values ​​can serve as indicators of the degree of deviation regarding specific conditions, such as the health status of the target object, provided that the classes of the assumed object and the target object match. Therefore, by calculating deviation index values ​​for higher-order factors only, it is possible to evaluate the specific state of the target object.

[0156] Furthermore, in the information processing device as an embodiment, the assumed spectrum is a convolution of the spectral sensitivity characteristics of the spectral sensor. This eliminates the need to perform correction processing on the target spectrum used in calculating the deviation index value to remove the influence of the spectral sensitivity characteristics of the spectroscopic sensor. Therefore, the processing burden can be reduced.

[0157] Furthermore, the information processing device as an embodiment includes a second presentation processing unit (presentation processing unit F2B) that performs processing to present to the user information indicating pixels in which the deviation index value is equal to or greater than a threshold value, based on the deviation index value calculated for each pixel. This allows the system to present to the user pixels in the spectral image that deviate significantly from the intended object. For example, even in a spectral image that generally shows the intended object, anomalous pixels may occur where a spectrum different from the original spectral spectrum is detected due to optical factors, etc. In such cases, information indicating these anomalous pixels can be presented to the user.

[0158] Furthermore, in the information processing apparatus as an embodiment, the calculation unit includes an exclusion processing unit (F5) that calculates a deviation index value for each pixel of a spectral image prepared as a candidate input image for training the AI ​​model, and performs processing to exclude spectral images having pixels whose deviation index value is greater than or equal to a threshold from the training dataset of the AI ​​model. This allows spectral images containing even one pixel with a deviation index value above a threshold, i.e., a pixel estimated to be an abnormal pixel, to be excluded from the AI ​​model's training dataset. Therefore, it is possible to prevent AI model training from being performed using spectral images that contain anomalies, thereby improving the training accuracy and inference performance of the AI ​​model.

[0159] Furthermore, in the information processing apparatus as an embodiment, the calculation unit includes a filtering unit that calculates a deviation index value for each pixel of the spectral image used as the learning input image for the AI ​​model, and processes the data so that pixels whose deviation index value is above a threshold are not used for learning the AI ​​model. This prevents pixels with deviation index values ​​above a threshold, i.e., pixels estimated to be abnormal pixels, from being used in training the AI ​​model. Therefore, it is possible to improve the learning accuracy and inference performance of AI models.

[0160] An information processing method as an embodiment is an information processing method in which an information processing device calculates a deviation index value, which is an index value indicating the degree of deviation between the target spectrum, which is a spectral spectrum acquired using a spectroscopic sensor, and the assumed spectrum, based on the eigenvalues ​​and eigenvectors of feature quantities obtained by dimensionality reduction performed on the assumed spectrum, which is a spectral spectrum of a hypothetical object. This information processing method can also provide the same functions and effects as the information processing apparatus described in the above-described embodiment.

[0161] Furthermore, the effects described herein are merely illustrative and not limited to those described herein, and other effects may also occur.

[0162] <7. This Technology> This technology can also be configured as follows: (1) The system includes a calculation unit that calculates a deviation index value, which is an index value indicating the degree of deviation between the target spectrum (a spectral spectrum acquired using a spectroscopic sensor) and the assumed spectrum, based on the eigenvalues ​​and eigenvectors of feature quantities obtained by dimensionality reduction performed on the assumed spectrum (a spectral spectrum of a hypothetical object). Information processing device. (2) The aforementioned spectroscopic sensor is a two-dimensional sensor, and the target spectrum is acquired for each pixel. The aforementioned arithmetic unit, The aforementioned deviation index value is calculated for each pixel. The information processing device described in (1) above. (3) The system includes a first presentation processing unit that performs processing to present the deviation index value calculated for each pixel to the user using a two-dimensional map. The information processing device described in (2) above. (4) The system includes an analysis processing unit that performs analysis of the subject based on the aforementioned deviation index value. An information processing device as described in any of (1) to (3) above. (5) The aforementioned analytical processing unit Based on the relationship between the magnitude of the deviation index value calculated for the target spectrum and the threshold value, it is determined whether the target object is the assumed object. The information processing device described in (4) above. (6) The analysis processing unit includes an evaluation processing unit that calculates an evaluation value different from the deviation index value based on the target spectrum, using only the target spectrum determined to be the assumed object by the analysis processing unit. The information processing device described in (5) above. (7) The aforementioned arithmetic unit, The deviation index value is calculated using the pixels within the bounding box detected by object detection processing performed on an image captured of the same subject as the subject being sensed by the spectroscopic sensor. An information processing device as described in any of (2) to (6) above. (8) The aforementioned arithmetic unit, As part of the dimensionality reduction, the deviation index is calculated based on the eigenvalues ​​and eigenvectors obtained by performing dimensionality reduction using singular value decomposition. An information processing device as described in any of (1) to (7) above. (9) The aforementioned arithmetic unit, As the aforementioned deviation index value, a total deviation degree is calculated, which comprehensively represents the degree of deviation in multiple dimensions from the first order to a predetermined order. The information processing device described in (8) above. (10) The singular values ​​obtained by the singular value decomposition and the left singular matrix are stored in the memory device. The aforementioned arithmetic unit, The deviation index value is calculated based on the singular value and the left singular matrix stored in the memory device. The information processing apparatus described in (8) or (9) above. (11) The aforementioned arithmetic unit, The deviation index value is calculated for the principal components of the lower order only. An information processing device as described in any of (8) to (10) above. (12) The system includes an analysis processing unit that performs analysis of the subject based on the aforementioned deviation index value, The aforementioned analytical processing unit It is determined whether the deviation index value calculated for the principal components of the lower order only is below the threshold. The information processing device described in (11) above. (13) The aforementioned arithmetic unit, The aforementioned deviation index value is calculated for the principal components of the higher order only. An information processing device as described in any of (8) to (12) above. (14) The assumed spectrum is obtained by convolving the spectral sensitivity characteristics of the spectroscopic sensor. An information processing device as described in any of (1) to (13) above. (15) The system includes a second presentation processing unit that performs a process to present to the user information indicating the pixels for which the deviation index value is equal to or greater than a threshold value, based on the deviation index value calculated for each pixel. The information processing device described in (2) above. (16) The aforementioned arithmetic unit, The deviation index value is calculated for each pixel of the spectral image prepared as a candidate input image for training the AI ​​model. The system includes an exclusion processing unit that performs a process to exclude from the AI ​​model's training dataset any spectral image containing pixels whose deviation index value is greater than or equal to a threshold value. The information processing device described in (15) above. (17) The calculation unit is, The deviation index value is calculated for each pixel of the spectral image used as the input image for training the AI ​​model. The system includes a filtering unit that processes to prevent pixels whose deviation index value is above a threshold from being used in training the AI ​​model. The information processing device described in (2) above. (18) Information processing device, Based on the eigenvalues ​​and eigenvectors of the features obtained by dimensionality reduction performed on the assumed spectrum, which is the spectral spectrum of the assumed object, a deviation index value is calculated, which is an index value indicating the degree of deviation between the target spectrum, which is the spectral spectrum acquired using a spectroscopic sensor, and the assumed spectrum. Information processing methods. [Explanation of Symbols]

[0163] 1,1A,1B,1C,1D,1E,1F Information Processing Device 2.2A Spectroscopic Camera 11,11A,11B,11C,11D,11E,11F CPU 25 Spectroscopic Sensor 25a Pixel array section Px pixels Pu Spectroscopic Pixel Unit 26 Demosaicing Processing Section 27. Spectral sensitivity correction processing unit 28 Wavelength conversion processing unit 29 Communications Department 30 Control Unit F1, F1A, F1E, F1F Calculation Unit F2, F2B, F2D Display Processing Unit F3, F3E, F3F Analysis Processing Unit F4 Evaluation Processing Unit F5 Exclusion Processing Unit F6 Object Detection Processing Unit Im (Image display area for confirmation) Md deviation map presentation area B1 Execute button BB Bounding Box

Claims

1. The system includes a calculation unit that calculates a deviation index value, which is an index value indicating the degree of deviation between the target spectrum (a spectral spectrum acquired using a spectroscopic sensor) and the assumed spectrum, based on the eigenvalues ​​and eigenvectors of feature quantities obtained by dimensionality reduction performed on the assumed spectrum (a spectral spectrum of a hypothetical object). Information processing device.

2. The aforementioned spectroscopic sensor is a two-dimensional sensor, and the target spectrum is acquired for each pixel. The aforementioned arithmetic unit, The aforementioned deviation index value is calculated for each pixel. The information processing apparatus according to claim 1.

3. The system includes a first presentation processing unit that performs processing to present the deviation index value calculated for each pixel to the user using a two-dimensional map. The information processing apparatus according to claim 2.

4. The system includes an analysis processing unit that performs analysis of the subject based on the aforementioned deviation index value. The information processing apparatus according to claim 1.

5. The aforementioned analytical processing unit Based on the relationship between the magnitude of the deviation index value calculated for the target spectrum and the threshold value, it is determined whether the target object is the assumed object. The information processing apparatus according to claim 4.

6. The analysis processing unit includes an evaluation processing unit that calculates an evaluation value different from the deviation index value based on the target spectrum, using only the target spectrum determined to be the assumed object by the analysis processing unit. The information processing apparatus according to claim 5.

7. The aforementioned arithmetic unit, The deviation index value is calculated using the pixels within the bounding box detected by object detection processing performed on an image captured of the same subject as the subject being sensed by the spectroscopic sensor. The information processing apparatus according to claim 2.

8. The aforementioned arithmetic unit, As part of the dimensionality reduction, the deviation index is calculated based on the eigenvalues ​​and eigenvectors obtained by performing dimensionality reduction using singular value decomposition. The information processing apparatus according to claim 1.

9. The aforementioned arithmetic unit, As the aforementioned deviation index value, a total deviation degree is calculated, which comprehensively represents the degree of deviation in multiple dimensions from the first order to a predetermined order. The information processing apparatus according to claim 8.

10. The singular values ​​obtained by the singular value decomposition and the left singular matrix are stored in the memory device. The aforementioned arithmetic unit, The deviation index value is calculated based on the singular value and the left singular matrix stored in the memory device. The information processing apparatus according to claim 8.

11. The aforementioned arithmetic unit, The deviation index value is calculated for the principal components of the lower order only. The information processing apparatus according to claim 8.

12. The system includes an analysis processing unit that performs analysis of the subject based on the aforementioned deviation index value, The aforementioned analytical processing unit It is determined whether the deviation index value calculated for the principal components of the lower order only is below the threshold. The information processing apparatus according to claim 11.

13. The aforementioned arithmetic unit, The aforementioned deviation index value is calculated for the principal components of the higher order only. The information processing apparatus according to claim 8.

14. The assumed spectrum is obtained by convolving the spectral sensitivity characteristics of the spectroscopic sensor. The information processing apparatus according to claim 1.

15. The system includes a second presentation processing unit that performs a process to present to the user information indicating the pixels for which the deviation index value is equal to or greater than a threshold value, based on the deviation index value calculated for each pixel. The information processing apparatus according to claim 2.

16. The aforementioned arithmetic unit, The deviation index value is calculated for each pixel of the spectral image prepared as a candidate input image for training the AI ​​model. The system includes an exclusion processing unit that performs a process to exclude from the AI ​​model's training dataset any spectral image containing pixels whose deviation index value is greater than or equal to a threshold value. The information processing apparatus according to claim 15.

17. The calculation unit is, The deviation index value is calculated for each pixel of the spectral image used as the learning input image for the AI ​​model. The system includes a filtering unit that processes to prevent pixels whose deviation index value is above a threshold from being used in training the AI ​​model. The information processing apparatus according to claim 2.

18. Information processing device, Based on the eigenvalues ​​and eigenvectors of the features obtained by dimensionality reduction performed on the assumed spectrum, which is the spectral spectrum of the assumed object, a deviation index value is calculated, which is an index value indicating the degree of deviation between the target spectrum, which is the spectral spectrum acquired using a spectroscopic sensor, and the assumed spectrum. Information processing methods.

Images