Information processing method and imaging system

Abstract

An information processing method executed by a computer, wherein: a subject is imaged using a camera, and a first compressed image is acquired in a first period (S11), the first compressed image being a compressed image that includes a plurality of pixels, with the data of each of the plurality of pixels including at least four pieces of waveband information, and that is used for generating a hyperspectral image by means of a reconstruction process; the speed of movement of the camera is acquired (S12); a determination is made as to whether a condition indicating that the speed of movement is equal to or less than a predetermined threshold value is satisfied in the first period (S13); and hyperspectral image generation in a reconstruction process based on the first compressed image is not performed if it is determined that the condition is not satisfied in the first period.

Description

Information processing method and imaging system

[0001] The present disclosure relates to an information processing method and an imaging system.

[0002] Techniques have been proposed for capturing hyperspectral images having spectral information in more wavelength bands than red, green, and blue (four or more wavelength bands). For example, it is known that a hyperspectral image can be generated by applying compressed sensing to a camera and performing a reconstruction process on the compressed image obtained by the compressed sensing. Compressed sensing, as in the above example, is a technique for obtaining acquired data so that a larger amount of data can be generated from a small number of samples in the acquired data. For example, in Patent Documents 1 and 2, a compressed image is captured using a filter array including a plurality of filters having different wavelength dependencies, and four or more spectral images (hyperspectral images) corresponding one-to-one to the four or more wavelength bands are reconstructed based on the captured compressed image.

[0003] International Publication No. 2021 / 192891 Patent No. 7262003

[0004] “World’s first technology to combine metalens and AI with a conventional digital camera to capture hyperspectral images and videos – Combining optical technology and AI to transform a ‘normal camera’ into a ‘camera that can see the properties of objects’”, [online], October 24, 2022, Nippon Telegraph and Telephone Corporation, [Retrieved June 15, 2023], Internet <URL: https: / / group.ntt / jp / newsrelease / 2022 / 10 / 24 / 221024a.html> Ahasan Ahamed, et.al., “Reconstruction-based spectroscopy using CMOS image sensors with random photon-trapping nanostructure per sensor”, Proc. SPIE 11971, High-Speed ​​Biomedical Imaging and Spectroscopy VII, 1197106, 2 March 2022

[0005] However, obtaining a hyperspectral image requires more steps than obtaining a spectral image with less than four wavelength bands that corresponds one-to-one to a normal RGB or monochrome image, etc. Therefore, in the past, there were cases where it was not possible to provide a hyperspectral image appropriately.

[0006] Therefore, the present disclosure provides an information processing method and an imaging system that can appropriately provide hyperspectral images.

[0007] An information processing method according to one aspect of the present disclosure is an information processing method executed by a computer, which includes: capturing an image of a subject using a camera; acquiring, during a first period, a first compressed image that is a compressed image including a plurality of pixels, wherein data for each of the plurality of pixels includes information for four or more wavelength bands, and that is used to generate a hyperspectral image through a reconstruction process; acquiring a speed of movement of the camera; determining whether, during the first period, a condition indicating that the speed of movement is equal to or less than a predetermined threshold is satisfied; and if it is determined that the condition is not satisfied during the first period, not generating the hyperspectral image based on the first compressed image.

[0008] This comprehensive or specific aspect may be realized by a system, an apparatus, an integrated circuit, a computer program, or a computer-readable recording medium, or by any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. The computer-readable recording medium includes, for example, a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory).

[0009] According to the present disclosure, hyperspectral images can be provided appropriately.

[0010] FIG. 1 is a functional configuration diagram of an imaging system according to Embodiment 1. FIG. 2 is a schematic diagram of an imaging device according to Embodiment 1. FIG. 3A is a schematic diagram of a filter array according to Embodiment 1. FIG. 3B is a diagram showing an example of a transmission spectrum of a filter according to Embodiment 1. FIG. 3C is a diagram showing an example of a transmission spectrum of another filter according to Embodiment 1. FIG. 3D is a diagram showing an example of a transmittance of a first wavelength band of a filter array according to Embodiment 1. FIG. 3E is a diagram showing an example of a transmittance of a second wavelength band of a filter array according to Embodiment 1. FIG. 4 is a flowchart of an information processing method according to Embodiment 1. FIG. 5 is a flowchart of an information processing method according to Modification 1 of Embodiment 1. FIG. 6 is a flowchart of an information processing method according to Modification 2 of Embodiment 1. FIG. 7 is a diagram showing an example of a display of an image according to Modification 2 of Embodiment 1. FIG. 8 is a functional configuration diagram of an imaging system according to Embodiment 2. FIG. 9 is a flowchart of an information processing method according to Embodiment 2. FIG. 10 is a functional configuration diagram of an imaging system according to a modification of Embodiment 2. FIG. 11 is a flowchart of an information processing method according to a modification of Embodiment 2. Fig. 12 is a functional configuration diagram of an imaging system according to embodiment 3. Fig. 13 is a flowchart of an information processing method according to embodiment 4. Fig. 14 is a flowchart of an information processing method according to embodiment 5. Fig. 15 is a flowchart relating to a rough image generation process in the information processing method according to embodiment 5. Fig. 16 is a flowchart relating to a rough image generation process in the information processing method according to embodiment 5. Fig. 17 is a flowchart relating to a rough image generation process in the information processing method according to embodiment 5. Fig. 18 is a flowchart relating to a rough image generation process in the information processing method according to embodiment 5.

[0011] (Outline of the Present Disclosure) Before describing the embodiments, an outline of the present disclosure will be described.

[0012] As already explained, obtaining a hyperspectral image requires more steps than obtaining a spectral image with less than four wavelength bands that corresponds one-to-one to less than four wavelength bands, such as a normal RGB or monochrome image. Specifically, when generating a hyperspectral image by spatial scanning or wavelength scanning, a sufficient exposure time must be provided for each unit space or each unit wavelength in the scan. Since such an exposure time is required across the entire scan range, a long exposure time is required to obtain one hyperspectral image. Furthermore, as explained in the Background Art section above, when generating a hyperspectral image by compressed sensing, a long processing time is required for information processing to reconstruct the hyperspectral image after sensing.

[0013] For example, to provide hyperspectral images as moving images, it is necessary to maintain a frame rate that allows the images to appear as moving images. However, the large number of steps required to obtain a hyperspectral image makes it difficult to maintain the frame rate. For example, if it takes 100 ms to obtain one hyperspectral image, then theoretically, the hyperspectral image can only be provided at 10 fps (frames per second). This is far from a frame rate that allows the images to appear as moving images.

[0014] There is a demand for real-time two-dimensional spectroscopic analysis using hyperspectral images. To achieve this, a hyperspectral camera is moved by hand or other means until it reaches a desired angle of view, and then a hyperspectral image is generated at the desired angle of view. When the desired angle of view is viewed two-dimensionally, the emission characteristics (including reflection and absorption characteristics) of each spectrally dispersed wavelength band can be determined for each pixel.

[0015] To meet such demands, if a hyperspectral camera is moved while generating a hyperspectral image, the provided hyperspectral image cannot keep up with (be able to track) the moving angle of view, making it impossible to perform real-time two-dimensional spectroscopic analysis while checking the subject within the angle of view. In other words, the hyperspectral image cannot be provided appropriately.

[0016] Therefore, in the present disclosure, for a mobile hyperspectral camera, a predetermined condition is set for the movement speed (in other words, the speed of movement) of the hyperspectral camera, and if the predetermined condition is not met (i.e., the movement speed is greater than a predetermined threshold), hyperspectral images are not provided, thereby preventing a situation where the provision of hyperspectral images cannot keep up and allowing appropriate hyperspectral images to be provided.

[0017] To achieve the above, an information processing method according to a first aspect of the present disclosure is an information processing method executed by a computer, which includes: capturing an image of a subject using a camera; acquiring, during a first period, a first compressed image that is a compressed image including a plurality of pixels, wherein data for each of the plurality of pixels includes information for four or more wavelength bands, and that is used to generate a hyperspectral image through a reconstruction process; acquiring a speed of camera movement; determining whether, during the first period, a condition indicating that the speed of movement is equal to or less than a predetermined threshold is satisfied; and if it is determined that the condition is not satisfied during the first period, not generating a hyperspectral image based on the first compressed image.

[0018] According to this, if the camera movement speed is determined to be greater than or equal to a predetermined threshold, it is possible to prevent the generation of a hyperspectral image through the reconstruction process. Because the reconstruction process for generating a hyperspectral image requires a long processing time, if the movement speed is greater than or equal to the predetermined threshold, the provision of the hyperspectral image cannot keep up. In such a case, by not performing the reconstruction process for generating a hyperspectral image, the provision of the hyperspectral image can be prevented, and the hyperspectral image can be provided appropriately. Furthermore, not performing the reconstruction process for generating a hyperspectral image also has the advantage of saving processing resources.

[0019] An information processing method according to a second aspect of the present disclosure is the information processing method according to the first aspect, in which a second compressed image, which is a compressed image, is obtained in a second period prior to the first period, and if it is determined that a condition is satisfied in the first period, a hyperspectral image based on the first compressed image or a hyperspectral image based on the second compressed image is generated by a reconstruction process.

[0020] According to this, when the speed of motion is equal to or less than a predetermined threshold, a hyperspectral image can be generated and provided by a reconstruction process based on compressed images acquired in the first time period or the second time period.

[0021] An information processing method according to a third aspect of the present disclosure is the information processing method according to the first or second aspect, wherein, when it is determined that a condition is satisfied during a first period, a hyperspectral image is generated by a reconstruction process based on a first compressed image, and if the generation of the hyperspectral image in the reconstruction process based on the first compressed image is completed at the end of the first period, an image related to the generated hyperspectral image is output to be displayed on a display device, and if the generation of the hyperspectral image in the reconstruction process based on the first compressed image is not completed at the end of the first period, the first compressed image is output to be displayed on a display device.

[0022] According to this, if the speed of movement is equal to or less than a predetermined threshold, and if generation of the hyperspectral image in the reconstruction process has been completed, an image related to the hyperspectral image can be output and displayed, and if generation of the hyperspectral image in the reconstruction process has not been completed, a compressed image can be output and displayed. In other words, an appropriate hyperspectral image can be provided depending on not only the speed of camera movement but also the status of the reconstruction process for generating the hyperspectral image.

[0023] An information processing method according to a fourth aspect of the present disclosure is the information processing method according to the third aspect, wherein the image related to the hyperspectral image is the hyperspectral image itself or an image showing the analysis results of a subject analyzed based on the hyperspectral image.

[0024] According to this, once the reconstruction process for generating a hyperspectral image is completed, the hyperspectral image itself or an image showing the analysis results of the subject analyzed based on the hyperspectral image can be output and displayed.

[0025] An information processing method according to a fifth aspect of the present disclosure is an information processing method according to any one of the first to fourth aspects, and if it is determined that a condition is not satisfied during a first period, a first compressed image is output and displayed on a display device.

[0026] According to this, when the speed of the movement is not equal to or less than a predetermined threshold, the compressed image can be output and displayed.

[0027] An information processing method according to a sixth aspect of the present disclosure is an information processing method according to any one of the first to fifth aspects, in which the speed of movement is obtained as the speed of relative movement between the camera and the subject, derived based on a matching process between the subject included in the first compressed image and the subject included in the second compressed image.

[0028] This allows the speed of movement to be derived based on compressed images at two points in time. Since the speed of movement can be obtained using only the camera's image sensor, there is no need to provide an additional sensor such as a speed sensor, which is advantageous in terms of device cost.

[0029] An information processing method according to a seventh aspect of the present disclosure is an information processing method according to any one of the first to sixth aspects, in which the speed of movement is obtained as a detection result detected by a sensor that detects the speed of the camera.

[0030] This allows the speed of movement to be acquired from the detection result at a single point in time. Since the detection result can be obtained for each single point in time when acquiring the speed of movement, this has an advantage in terms of time resolution compared to speed acquisition means that require multiple points in time.

[0031] An information processing method according to an eighth aspect of the present disclosure is an information processing method executed by a computer, which acquires the speed of camera movement and determines whether a condition indicating that the speed of movement is equal to or less than a predetermined threshold is met, and if it is determined that the condition is met, uses the camera to capture an image of the subject with a first exposure time, thereby acquiring a hyperspectral image including multiple images corresponding to four or more wavelength bands, and if it is determined that the condition is not met, uses the camera to capture an image of the subject with a second exposure time shorter than the first exposure time, thereby acquiring a substitute image in place of the hyperspectral image.

[0032] According to this, when the camera movement speed is determined to be higher than a predetermined threshold, an alternative image can be acquired instead of acquiring a hyperspectral image. Because acquiring a hyperspectral image requires a long exposure time, providing a hyperspectral image cannot keep up if the movement speed is higher than a predetermined threshold. In such a case, by not acquiring a hyperspectral image, the provision of the hyperspectral image can be prevented, and the hyperspectral image can be provided appropriately. Another advantage of acquiring an alternative image instead of a hyperspectral image is that the alternative image can be used.

[0033] An information processing method according to a ninth aspect of the present disclosure is the information processing method according to the eighth aspect, wherein the alternative images include fewer images than the multiple images included in the hyperspectral image.

[0034] According to this, instead of the hyperspectral image, an alternative image containing fewer images than the number of images contained in the hyperspectral image can be used.

[0035] An information processing method according to a tenth aspect of the present disclosure is the information processing method according to the eighth or ninth aspect, wherein the camera detects light passing through a plurality of optical filters, each of which has a transmittance peak that corresponds one-to-one to one of four or more wavelength bands.

[0036] This makes it possible to obtain a hyperspectral image or an alternative image by detecting light that has passed through multiple optical filters that have transmittance peaks that correspond one-to-one to any of four or more wavelength bands.

[0037] An information processing method according to an eleventh aspect of the present disclosure is the information processing method according to either the eighth or ninth aspect, in which the camera generates a hyperspectral image by scanning in the wavelength direction or the spatial direction.

[0038] This allows for the acquisition of a hyperspectral image by scanning in the wavelength direction or the spatial direction, or alternatively, for the acquisition of an alternative image by scanning in part of the wavelength direction and the spatial direction.

[0039] An information processing method according to a twelfth aspect of the present disclosure is an information processing method according to any one of the first to eleventh aspects, and if it is determined that a condition is not satisfied in a first period, a display image including information of three or less wavelength bands is generated and output based on a first compressed image.

[0040] According to this, when the speed of movement is not equal to or less than a predetermined threshold, a display image including information of three or less wavelength bands can be generated, output, and displayed.

[0041] An information processing method according to a thirteenth aspect of the present disclosure is an information processing method according to the twelfth aspect, in which a second compressed image, which is a compressed image, is obtained in a second period prior to the first period, and if it is determined that the conditions are not satisfied in the first period, a display image based on the first compressed image or a display image based on the second compressed image is generated.

[0042] According to this, when the speed of movement is not below a predetermined threshold, a display image including information on wavelength bands of three or less based on the first compressed image, or a display image including information on wavelength bands of three or less based on the second compressed image, can be generated, output, and displayed.

[0043] An information processing method according to a fourteenth aspect of the present disclosure is the information processing method according to the twelfth or thirteenth aspect, in which the display image is generated without going through a reconstruction process.

[0044] This allows the display image to be generated and displayed by a process different from the reconstruction process.

[0045] An information processing method according to a fifteenth aspect of the present disclosure is an information processing method executed by a computer, which includes: capturing an image of a subject using a camera; acquiring, during a first period, a first compressed image including a plurality of pixels, wherein data for each of the plurality of pixels includes information for four or more wavelength bands, the first compressed image being used to generate a hyperspectral image through a reconstruction process; acquiring a speed of camera movement; determining, during the first period, whether a condition indicating that the speed of movement is equal to or less than a predetermined threshold is met; if it is determined that the condition is met during the first period, displaying a first screen based on the plurality of images corresponding to each of N wavelength bands; and if it is determined that the condition is not met during the first period, displaying (i) the compressed image or (ii) a second screen based on the plurality of images corresponding to each of M wavelength bands (M<N).

[0046] According to this method, when the camera motion speed is determined to be equal to or less than a predetermined threshold, a first screen is displayed using an image containing information on N wavelength bands from the compressed image. On the other hand, when the motion speed is greater than the predetermined threshold, a second screen is displayed using an image containing information on M wavelength bands (M<N). Because the reconstruction process for generating a hyperspectral image containing information on a relatively large number of N wavelength bands requires a long processing time, providing the hyperspectral image may be slow if the motion speed is greater than the predetermined threshold. In such cases, generating an image such as a hyperspectral image containing information on a relatively small number of M wavelength bands instead of a hyperspectral image containing information on N wavelength bands allows the hyperspectral image to be provided appropriately. Furthermore, reducing the number of wavelength bands included in the reconstruction process for generating a hyperspectral image also has advantages, such as saving processing resources.

[0047] An information processing method according to a sixteenth aspect of the present disclosure is the information processing method according to the fifteenth aspect, in which, if it is determined that the conditions are not satisfied in the first period, a plurality of images corresponding to each of the M wavelength bands are generated by a reconstruction process based on the compressed image, and a second screen is displayed.

[0048] This allows an image containing information from M wavelength bands to be generated by a reconstruction process and output for display.

[0049] An information processing method according to a seventeenth aspect of the present disclosure is an information processing method according to the fifteenth aspect, in which, if it is determined that the conditions are not satisfied in the first period, multiple images corresponding to each of the M wavelength bands are generated without going through a reconstruction process based on the compressed image, and a second screen is displayed.

[0050] According to this, the image to be displayed on the second screen can be generated and displayed by a process different from the reconstruction process.

[0051] An imaging system according to an eighteenth aspect of the present disclosure includes an image acquisition unit that uses a camera to capture an image of a subject and acquires, during a first period, a first compressed image that includes a plurality of pixels, wherein data for each of the plurality of pixels includes information on four or more wavelength bands, and that is used to generate a hyperspectral image through a reconstruction process; an information acquisition unit that acquires the speed of camera movement; a determination unit that determines whether, during the first period, a condition indicating that the speed of movement is equal to or less than a predetermined threshold is met; and a reconstruction unit that does not generate a hyperspectral image through the reconstruction process based on the first compressed image if it determines that the condition is not met during the first period.

[0052] This can achieve the same effects as the information processing method described above.

[0053] An imaging system according to a nineteenth aspect of the present disclosure includes an acquisition unit that acquires the speed of camera movement, a determination unit that determines whether the speed of movement satisfies a condition indicating that it is equal to or less than a predetermined threshold, and an image acquisition unit. If the image acquisition unit determines that the condition is satisfied, it uses the camera to image the subject with a first exposure time and acquires a hyperspectral image including a plurality of pixels, each of the plurality of pixels including information of four or more wavelength bands. If it determines that the condition is not satisfied, it uses the camera to image the subject with a second exposure time that is shorter than the first exposure time and acquires an alternative image to the hyperspectral image.

[0054] This can achieve the same effects as the information processing method described above.

[0055] (Embodiments) Hereinafter, embodiments will be specifically described with reference to the drawings.

[0056] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, the arrangement and connection of the components, steps, and the order of steps shown in the following embodiments are merely examples and are not intended to limit the technology of the present disclosure.

[0057] Furthermore, each figure is a schematic diagram and is not necessarily an exact illustration. Therefore, for example, the scales of the figures do not necessarily match. Furthermore, in each figure, substantially the same components are given the same reference numerals, and duplicated explanations may be omitted or simplified.

[0058] In the following, terms indicating the relationship between elements, such as parallel or perpendicular, terms indicating the shape of elements, such as rectangle or circle, and numerical ranges are not expressions that only express a strict meaning, but are expressions that also include a substantially equivalent range, for example, a difference of about a few percent.

[0059] In the following description, all or part of a circuit, unit, or device, or all or part of a functional block in a block diagram, may be implemented by one or more electronic circuits, including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). The LSI or IC may be integrated on a single chip or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated on a single chip. While the terms LSI and IC are used here, the term may be changed depending on the degree of integration, and may be referred to as a system LSI, a VLSI (very large scale integration), or an ULSI (ultra large scale integration). Field programmable gate arrays (FPGAs), which are programmed after the LSI is manufactured, or reconfigurable logic devices (RLDs), which can reconfigure connections within the LSI or set up circuit partitions within the LSI, can also be used for the same purpose.

[0060] Furthermore, all or part of a circuit, unit, or device, or all or part of a functional block in a block diagram, may be implemented by a software program. In this case, the software program is recorded on one or more non-transitory storage media such as a ROM, optical disk, hard disk drive, etc., and when the software program is executed by a processor, the functions specified in the software program are performed by the processor and peripheral devices. A system or device may include one or more non-transitory storage media in which a software program is stored, a processor, and necessary hardware devices (e.g., memory, interface, etc.).

[0061] [Functional Configuration of Imaging System 1000] First, the functional configuration of imaging system 1000 according to this embodiment will be specifically described with reference to Fig. 1. Fig. 1 is a functional configuration diagram of imaging system 1000 according to this embodiment. Note that Fig. 1 shows an exemplary functional configuration of imaging system 1000, and the functional configuration of imaging system 1000 is not limited to that shown in Fig. 1.

[0062] 1, the imaging system 1000 includes an imaging device 100, an information processing device 200, and a display device 300. The imaging device 100, the speed sensor 400, the information processing device 200, and the display device 300 included in the imaging system 1000 will be described in detail below.

[0063] The imaging device 100 is an example of a camera according to the first embodiment, has a configuration similar to that of the imaging device disclosed in Patent Document 1, and is capable of capturing a compressed image including a plurality of pixels. Each of the plurality of pixels included in the compressed image includes information of four or more wavelength bands. The imaging device 100 includes a control circuit 110 and an image sensor 120.

[0064] The control circuit 110 controls the image sensor 120 to cause the image sensor 120 to generate a compressed image.

[0065] The image sensor 120 is a monochrome photodetector having a plurality of photodetection elements arranged in a matrix. Examples of the image sensor 120 include a charge-coupled device (CCD) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, an infrared array image sensor, a terahertz array image sensor, and a millimeter-wave array image sensor. The image sensor 120 does not have to be a monochrome photodetector and may be a color photodetector. The wavelength range detectable by the image sensor 120 is not particularly limited and may be, for example, visible light, ultraviolet light, infrared light, terahertz waves, or any combination thereof.

[0066] 1, the imaging device 100 further includes a filter array 130 and an optical system 140. The configuration of the imaging device 100 including the filter array 130 and the optical system 140 will be described later with reference to FIG.

[0067] The speed sensor 400 is a sensor that detects the movement of the imaging device 100, i.e., the speed of the movement of the imaging device 100. The speed sensor 400 is realized by one or more sensors selected from sensors capable of detecting speed, such as an image sensor, a gyro sensor, an acceleration sensor, and a positioning sensor. The speed sensor 400 is built into the imaging device 100 and detects the speed of the movement of the imaging device 100 in which the speed sensor 400 is built. The speed sensor 400 may also be additionally attached to the imaging device 100 and detect the speed of the movement of the imaging device 100 to which the speed sensor 400 is attached. The speed sensor 400 may also be realized as an image sensor installed so as to be able to capture an image of the imaging device 100 and detect the speed of the movement of the imaging device 100 captured by the speed sensor 400. Note that the speed sensor 400 is not an essential component. An example of implementing the imaging system 1000 without the speed sensor 400 will be described later.

[0068] The information processing device 200 is communicably connected via wire and / or wireless to the imaging device 100, the speed sensor 400, and the display device 300. The information processing device 200 includes an image acquisition unit 210, an information acquisition unit 220, a parameter determination unit 230, a reconstruction calculation unit 240, a storage unit 250, and a determination unit 260.

[0069] The image acquisition unit 210 can acquire a compressed image from the imaging device 100 and store it in the storage unit 250. Note that the image acquisition unit 210 does not have to acquire the compressed image directly from the imaging device 100, and may acquire the compressed image via another device or the like.

[0070] The information acquisition unit 220 can acquire image quality adjustment information from the display device 300. Note that the information acquisition unit 220 does not have to acquire the image quality adjustment information directly from the display device 300, but may acquire the image quality adjustment information via another device or the like. The information acquisition unit 220 can also acquire the speed of movement of the imaging device 100 from the speed sensor 400. Note that the information acquisition unit 220 does not have to acquire the speed of movement directly from the imaging device 100, but may acquire the speed of movement via another device or the like.

[0071] The parameter determination unit 230 can determine the values ​​of the calculation parameters used in the restoration calculation based on the image quality adjustment information. That is, the parameter determination unit 230 can determine, based on the image quality adjustment information, one or more values ​​that correspond one-to-one to one of the calculation parameters used in the restoration calculation. The image quality adjustment information refers to information for adjusting the image quality of four or more spectral images obtained by the restoration calculation. For example, the image quality adjustment information may include a first priority value that indicates the priority of the image quality of the four or more spectral images, a second priority value that indicates the priority of the speed at which the four or more spectral images are generated from the compressed image, or a combination of the first priority value and the second priority value. The calculation parameters may be a first parameter (a weighting coefficient τ, described later) that corresponds to the influence of regularization in the restoration calculation and / or a second parameter that corresponds to the number of iterations of the iterative calculation included in the restoration calculation.

[0072] The reconstruction calculation unit 240 can generate four or more spectral images by performing a reconstruction calculation on the compressed image using the values ​​of the calculation parameters determined by the parameter determination unit 230. In other words, the reconstruction calculation unit 240 performs processing to execute a reconstruction process for generating a hyperspectral image consisting of four or more spectral images through calculation using the compressed image. The reconstruction calculation unit 240 outputs the generated four or more spectral images to the display device 300.

[0073] The restoration calculation performed in this embodiment may be the same as the restoration calculation described in Patent Document 1 or 2. Specifically, four or more spectroscopic images may be restored based on the following equation (1).

[0074]

[0075] Here, g is data representing a compressed image, and is expressed, for example, as a one-dimensional array (i.e., vector). If the compressed image is an image of n×m pixels, data g is expressed as a one-dimensional array having n×m elements. f is data representing w spectral images that correspond one-to-one to w wavelength bands, and is expressed, for example, as a one-dimensional array. f1 is the wavelength band W １ The spectral image data corresponding to the wavelength band W ２ f1, f2, ..., fw is the data of the spectral image corresponding to the wavelength band Ww. Each of f1, f2, ..., fw is expressed, for example, as a one-dimensional array. If each spectral image is an image of nxm pixels, each of f1, f2, ..., fw is expressed as a one-dimensional array having nxm elements, and data f is expressed as a one-dimensional array having nxmxw elements. H is a matrix with nxm rows and nxmxw columns, and may be referred to as a system matrix. H is the spectral image data corresponding to the wavelength band Ww of the filter array 130. １ Transmission spectrum of the filter array 130 wavelength band W ２ The transmission spectrum of the filter array 130 wavelength band W ｗ The data f satisfying the formula (1) can be estimated using a compressed sensing technique, and specifically, can be estimated by the formula (2).

[0076]

[0077] Equation (2) expresses finding f that minimizes the sum of the first and second terms in the parentheses. Data f as a final solution can be calculated by converging the solution through recursive iterative calculations.

[0078] The first term in the parentheses in equation (2) represents the sum of squares of the difference between Hf obtained by system transformation of data f in the estimation process using matrix H and data g, and is a so-called residual term. Here, the sum of squares is used, but instead of the sum of squares, the sum of absolute values ​​or the square root of the sum of squares may be used. The sum of squares of the difference between Hf and data g is (g １ -r１ ) × (g １ -r １ ) + ... + (g ｎ×ｍ -r ｎ×ｍ ) × (g ｎ×ｍ -r ｎ×ｍ ) where g = (g １ ...g ｎ×ｍ ) Ｔ , Hf = (r １ ...r ｎ×ｍ ) Ｔ is.

[0079] The second term in parentheses in Equation (2) is a regularization term, sometimes called a stabilization term. Φ(f) represents a constraint on the regularization of f and is a function reflecting the sparse information of the data f. This function has the effect of smoothing or stabilizing the data f. Φ(f) can be expressed, for example, by a discrete cosine transform (DCT), a wavelet transform, a Fourier transform, a total variation (TV), or any combination thereof. τ is a weighting coefficient for the regularization term and corresponds to the influence of regularization in the reconstruction calculation. The larger the value of τ, the greater the influence of regularization, the greater the amount of redundant data removed, and the stronger the convergence of the solution in the iterative calculation. Conversely, the smaller the value of τ, the less the influence of regularization, the less redundant data removed, and the weaker the convergence of the solution in the iterative calculation.

[0080] In Equation (2), τ may be used as a first parameter. Furthermore, the number of iterations of the recursive iterative calculation of Equation (2) may be used as a second parameter. The reconstruction calculation unit 240 can generate four or more spectroscopic images by performing the reconstruction calculation of Equation (2) using the values ​​of the first parameter and the second parameter.

[0081] The storage unit 250 can store a compressed image and / or four or more spectroscopic images (reconstructed images), etc. The storage unit 250 can be implemented using, for example, a hard disk drive and / or a solid state drive.

[0082] The determination unit 260 can determine whether or not the acquired speed of movement of the imaging device 100 satisfies a condition indicating that the speed is equal to or less than a predetermined threshold. Depending on the determination result of the determination unit 260 as to whether or not the condition is satisfied, the determination unit 260 generates a control signal for switching whether or not to generate four or more spectroscopic images, and outputs the control signal to the reconstruction calculation unit 240. The predetermined threshold will be described in detail later.

[0083] The display device 300 is a user interface and includes an input unit 310 and a display unit 320. The display device 300 can be implemented using, for example, a tablet computer, a smartphone, or a desktop computer.

[0084] The input unit 310 can receive image quality adjustment information from a user. The input unit 310 can be implemented using, for example, a touch screen, a touch pad, a mouse, a keyboard, or any combination thereof.

[0085] The display unit 320 can display a graphical user interface (GUI) for acquiring image quality adjustment information. The display unit 320 can be implemented using, for example, a liquid crystal display (LCD) and / or an organic light-emitting diode (OLED) display.

[0086] 1 illustrates an exemplary functional configuration of the imaging system 1000, and the functional configuration of the imaging system 1000 is not limited to that illustrated in FIG. 1. For example, part or all of the imaging device 100 may be included in the information processing device 200. Also, for example, part or all of the display device 300 may be included in the information processing device 200. Also, for example, the information processing device 200 may be divided into multiple devices, and may be realized by, for example, a cloud server.

[0087] [Configuration of Image Capturing Apparatus 100] Next, the configuration of the image capturing apparatus 100 will be described with reference to Fig. 2. Fig. 2 is a schematic diagram of the image capturing apparatus 100 according to the embodiment.

[0088] The imaging device 100 has a configuration similar to that of the imaging devices disclosed in Patent Documents 1 and 2. Specifically, the imaging device 100 includes a control circuit 110, an image sensor 120, a filter array 130, and an optical system 140. Note that the control circuit 110 is not shown in FIG. 2 .

[0089] The filter array 130 is disposed on the optical path of light incident from the object 70, which is the subject, and is disposed between the optical system 140 and the image sensor 120 in FIG. 2 . The filter array 130 functions as the encoding element of Patent Document 1. The filter array 130 may be integrated with the image sensor 120. Note that the arrangement of the filter array 130 is not limited to the arrangement shown in FIG. 2 . For example, the filter array 130 may be disposed between the optical system 140 and the image sensor 120, but away from the image sensor 120. Alternatively, for example, the filter array 130 may be disposed between the object 70 and the optical system 140. Alternatively, for example, the filter array 130 may be disposed within the optical system 140.

[0090] The optical system 140 is disposed on the optical path of light incident from the object 70, and in FIG. 2 is disposed between the object 70 and the filter array 130. The optical system 140 includes at least one lens and can form an image of the object 70 on the imaging surface of the image sensor 120 via the filter array 130. Note that the configuration and arrangement of the optical system 140 are not limited to the configuration and arrangement shown in FIG. 2. For example, the optical system 140 may be disposed between the filter array 130 and the image sensor 120. Alternatively, for example, the optical system 140 may include multiple lenses lined up on the optical path. In this case, the filter array 130 may be disposed between adjacent lenses of the multiple lenses.

[0091] [Configuration of Filter Array 130] The filter array 130 includes a plurality of filters. The number of the plurality of filters may be n×m. The n×m filters are １１ ,···,filter ｎｍ Contains filters １１ Wavelength band W １ ~Transmission spectrum S in wavelength band Ww １１, ..., wavelength band W of filter nm １ ~Transmission spectrum S in wavelength band Ww ｎｍ may all be different, or the transmission spectrum S １１ , ..., transmission spectrum S ｎｍ Some of the elements may be the same.

[0092] filter １１ Wavelength band W １ Transmittance S １１Ｗ１ ,···,filter １１ The transmittance S of the wavelength band Ww １１Ｗｗ ,···,filter ｎｍ Wavelength band W １ Transmittance S ｎｍＷ１ , ..., wavelength band W of filter nm ｗ Transmittance S ｎｍＷｗ may all be different, or the transmittance S １１Ｗ１ , ..., transmittance S １１Ｗｗ , ..., transmittance S ｎｍＷ１ , ..., transmittance S ｎｍＷｗ may be the same. w may be an integer of 4 or more. In the present disclosure, transmittance may mean light transmittance.

[0093] Wavelength band W α The transmittance of the filter β in the above formula may be expressed by the following formula (3):

[0094]

[0095] Wαmin is the wavelength band W α Wαmax is the minimum wavelength value of the wavelength band W α is the maximum wavelength value, h(λ) is a function indicating the transmission spectrum, and λ is the wavelength.

[0096] In addition, the wavelength band W α The transmittance of the filter β in the wavelength band W is not limited to the formula (3). α The transmittance of the filter β in the wavelength band W may be the transmittance obtained by dividing the formula (3) by (Wαmax-Wαmin). α The transmittance of the filter β in the wavelength band W αThe frequency λ that represents α０ Transmittance h (λ α０ ) may be used. α０ is Wαmin≦λ α０ ≦Wαmax. For example, the wavelength band W α The center frequency may be ((Wαmax−Wαmin) / 2).

[0097] The configuration of the filter array 130 will be described with reference to Figures 3A to 3E. Figure 3A is a schematic diagram of the filter array 130 according to an embodiment. The filter array 130 includes a filter 130a and a filter 130b. Figure 3B shows an example of the transmission spectrum of the filter 130a. Figure 3C' shows an example of the transmission spectrum of the filter 130b.

[0098] FIG. 3D shows the wavelength band W of the filter array 130 according to the embodiment. １ 3E is a diagram illustrating an example of the transmittance of the wavelength band W of the filter array 130 according to the embodiment. ２ 3D and 3E , the shading of each region represents the transmittance of the filter, with lighter regions representing higher transmittance and darker regions representing lower transmittance.

[0099] The filter array 130 includes a plurality of filters arranged in a matrix. In the example shown in FIG. 3A , the filter array 130 includes 48 filters arranged in 6 rows and 8 columns. Filter 130a is the filter arranged in the upper left of the 48 filters, and filter 130b is the filter arranged in the lower right of the 48 filters. Note that the number of filters included in the filter array 130 is not limited to 48. For example, the number of filters included in the filter array 130 may be approximately the same as the number of pixels of the image sensor 120, and may be determined depending on the application, for example, in a range from several tens to several tens of millions.

[0100] The wavelength dependence of the transmittance of the multiple filters included in the filter array 130 is different from one another. For example, in the filter 130a, １ The transmittance of the wavelength band W ２On the other hand, in the filter 130b, the transmittance of the wavelength band W １ The transmittance of the wavelength band W ２ In other words, the wavelength dependency of the transmittance of the filter 130a is different from the wavelength dependency of the transmittance of the filter 130b. １ and W ２ The transmittance of the other wavelength bands included in the four or more wavelength bands will be illustrated and described below, and the illustration and description of the transmittance of the other wavelength bands will be omitted.

[0101] [Information Processing Method] Next, an information processing method in the imaging system 1000 configured as above will be described with reference to Figures 4 to 7. Figure 4 is a flowchart of the information processing method according to the first embodiment.

[0102] First, the image acquisition unit 210 of the information processing device 200 acquires a compressed image from the imaging device 100 (S11). Furthermore, the information acquisition unit 220 of the information processing device 200 acquires the speed of movement of the imaging device 100 from the speed sensor 400 (S12). The display unit 320 of the display device 300 displays a GUI for acquiring image quality adjustment information. The input unit 310 of the display device 300 receives input of the image quality adjustment information via the GUI, and the information acquisition unit 220 of the information processing device 200 acquires the image quality adjustment information from the display device 300.

[0103] For example, the display unit 320 displays a GUI, and the input unit 310 receives input of image quality adjustment information. As the image quality adjustment information, values ​​can be set for the priority (first priority) of the image quality of four or more spectral images (i.e., hyperspectral images) and the priority (second priority) of the speed of the restoration calculation. In this embodiment, the first priority value increases as the first priority increases, but may also decrease as the first priority increases. Similarly, the second priority value increases as the second priority increases, but may also decrease as the second priority increases. Furthermore, the second priority value is dependent on the first priority value, and conversely, the first priority value is dependent on the second priority value. Therefore, it is not necessary to acquire both the first priority value and the second priority value; only one of the first priority value and the second priority value may be acquired.

[0104] Meanwhile, the determination unit 260 determines whether the acquired motion speed satisfies a condition indicating that it is equal to or less than a predetermined threshold (S13). If the determination unit 260 determines that the condition is satisfied (Yes in S13), it generates a control signal for switching to generate four or more spectral images and outputs it to the reconstruction calculation unit 240. As a result, the reconstruction calculation unit 240 performs restoration processing (S14). On the other hand, if the determination unit 260 determines that the condition is not satisfied (No in S13), it generates a control signal for switching to not generate four or more spectral images and outputs it to the reconstruction calculation unit 240. As a result, step S14 is skipped. In accordance with the control signal, the reconstruction calculation unit 240 outputs a hyperspectral image consisting of four or more spectral images generated based on the image quality adjustment information. The output hyperspectral image is displayed on the display unit 320 of the display device 300.

[0105] Here, the predetermined threshold will be described. The predetermined threshold is a boundary value between whether or not it is recognized that the provision of a hyperspectral image cannot keep up with the speed of movement of the imaging device 100. Therefore, the predetermined threshold may vary depending on factors such as the performance of the display unit 320 of the display device 300, which is involved in the user's visual perception. As an example, assume that the display unit 320 of the display device 300 is positioned and sized so that it can be viewed by the user at a viewing angle of 60 degrees. Note that the number of pixels of the display unit 320 in the horizontal direction is calculated as 1,920 pixels. From the user's perspective, the speed at which the movement of a subject displayed on the display unit 320 is recognized is considered to be approximately 1 to 10 minutes per second. Even at 10 minutes per second, this amounts to 3 pixels per 30 frames, so it can be said that the movement of the subject is recognized with a movement of 0.1 pixel per frame. Therefore, for example, under these conditions, the predetermined threshold is set to a speed at which the movement of the imaging device 100 is displayed on the display unit 320 as a movement of 1 pixel per frame. Such a predetermined threshold is one example, and may be set appropriately depending on conditions such as the performance of the display unit 320 of the display device 300, the arrangement and size of the display unit 320 that determine the viewing angle, and the like.

[0106] 5 is a flowchart of an information processing method according to Variation 1 of Embodiment 1. In this example, the method of acquiring the speed of movement of the imaging device 100 is different from the flowchart of the information processing method shown in FIG. 4. Also, in this example, more detailed processing is described than in the flowchart of the information processing method shown in FIG. 4. For example, such detailed processing may be performed in the above-described embodiment.

[0107] As shown in FIG. 5 , in the information processing method according to the first modification, first, the image acquisition unit 210 of the information processing device 200 acquires a compressed image from the imaging device 100 (S21). The image acquisition unit 210 of the information processing device 200 then determines whether or not there is an image of the previous frame (S22). If there is no image of the previous frame (No in S22), the process returns to step S21. On the other hand, if there is an image of the previous frame (Yes in S22), a motion vector is generated along with the image of the newly acquired frame (S23). The motion vector is information indicating the speed of movement of an object (i.e., a subject) in two frame images as the magnitude of a vector. The motion vector can be used to calculate the relative speed of the imaging device 100 with respect to the object. In other words, the speed of movement here is derived based on a matching process between the subject included in a first compressed image in a first period corresponding to a certain frame and the subject included in a second compressed image in a second period corresponding to a frame prior to the first period. Therefore, in this example, the speed of movement of the imaging device 100 can be obtained by deriving it from the images, even without the speed sensor 400.

[0108] As shown in FIG. 5 , the determination unit 260 determines whether a condition indicating that the speed of motion obtained from the motion vector is equal to or less than a predetermined threshold is satisfied (S24). If the determination unit 260 determines that the condition is satisfied (Yes in S24), it first determines whether a previous compressed image, such as a previous frame, is currently being restored (S25). Because restoration processing involves a relatively large processing load, parallel processing is difficult. Therefore, step S25 is provided to prevent parallel processing. If it is determined that a previous compressed image is not currently being restored (No in S25), restoration processing of a newly acquired compressed image is performed (S26). If it is determined that a previous compressed image is currently being restored (Yes in S25), step S26 is skipped.

[0109] Next, the restoration calculation unit 240 determines whether there is a hyperspectral image (HS image) for which restoration processing has been completed (S27). If it is determined that there is a hyperspectral image for which restoration processing has been completed (Yes in S27), the restoration calculation unit 240 sets the most recent hyperspectral image as the image for display (S28). If it is determined that there is no hyperspectral image for which restoration processing has been completed (No in S27), the process proceeds to step S30.

[0110] On the other hand, if the determination unit 260 determines that the condition is not satisfied (No in S24), it deletes the restored hyperspectral image (S29) and sets the most recent compressed image as the image for display (S30). The restoration calculation unit 240 then outputs the hyperspectral image or compressed image set as the image for display. The output hyperspectral image or compressed image is displayed on the display unit 320 of the display device 300.

[0111] FIG. 6 is a flowchart of an information processing method according to Modification 2 of Embodiment 1. FIG. 7 is a diagram illustrating an example of an image display according to Modification 2 of Embodiment 1. In this example, the image to be displayed is different from that of the flowchart of the information processing method illustrated in FIG. 5. Specifically, after step S27 returns Yes, the subject is analyzed based on the hyperspectral image, and an image showing the analysis results is generated. Here, based on the hyperspectral image, a density image is generated by converting the density magnitude of each pixel into a luminance value based on the concentration of a predetermined component contained in the subject (e.g., the lycopene concentration when the subject is a tomato) (S32). Then, instead of step S28, a step (S33) is performed in which the generated density image is set as the image to be displayed. Note that if step S25 returns Yes, it is determined whether a density image already exists (S34). If it is determined that a density image exists (Yes in S34), the process proceeds to step S33. If it is determined that a density image does not exist (No in S34), the process proceeds to step S27.

[0112] When a density image is displayed, for example, as shown in FIG. 7 , the "leaves" and "fruit" of a tomato, which is the subject of the image, can be distinguished based on lycopene concentration (in this case, dot-hatched areas indicate high concentrations). Also, within the "fruit," it is possible to distinguish between parts with high and low lycopene concentrations. Alternatively, it is possible to distinguish between fruits with high and low lycopene concentrations. Furthermore, a tomato is just one example of a subject. Any object can be used as a subject, and specific components contained in the object can be analyzed based on the hyperspectral image. As a result, a two-dimensional density image can be generated by converting the concentrations of specific components into brightness values. The density image can be generated by deriving the brightness value information for each pixel from one or more of the spectral images that make up the hyperspectral image. Note that algorithms, image filters, trained models, and the like for generating a density image can be used using the hyperspectral image as input. Information such as the algorithms, image filters, and trained models can be stored in the storage unit 250 or the like and retrieved as needed.

[0113] Next, a second embodiment will be described. FIG. 8 is a functional configuration diagram of an imaging system according to the second embodiment. The imaging system 1000a shown in FIG. 8 differs from the first embodiment in that the information processing device 200 includes only a determination unit 260, the display device 300 includes only a display unit 320, and an image is output directly from the imaging device 100 to the display device 300. The imaging device 100 of this embodiment captures an entire image space at all wavelengths using a snapshot method, using an image sensor that is two-dimensionally arranged, with a plurality of pixel groups each including a two-dimensional array of unit pixels, each of which has filters that transmit light of each wavelength. In other words, the imaging device 100 can capture multiple images corresponding to four or more wavelength bands in a single image capture.

[0114] The filters included in each unit pixel have optical characteristics that allow light of any wavelength between 460 and 650 nm to pass through. For example, in a 16-band wavelength band filter, of the filters included in each unit pixel, the first filter has a peak transmittance at a wavelength of 465 nm. The second filter has a peak transmittance at a wavelength of 474 nm. The second filter has a peak transmittance at a wavelength of 474 nm. The third filter has a peak transmittance at a wavelength of 485 nm. The fourth filter has a peak transmittance at a wavelength of 496 nm. The fifth filter has a peak transmittance at a wavelength of 546 nm. The sixth filter has a peak transmittance at a wavelength of 534 nm. The seventh filter has a peak transmittance at a wavelength of 522 nm. The eighth filter has a peak transmittance at a wavelength of 510 nm. The ninth filter has a peak transmittance at a wavelength of 586 nm. The tenth filter has a peak transmittance at a wavelength of 578 nm. The 11th filter has a peak transmittance at a wavelength of 562 nm. The 12th filter has a peak transmittance at a wavelength of 548 nm. The 13th filter has a peak transmittance at a wavelength of 630 nm. The 14th filter has a peak transmittance at a wavelength of 624 nm. The 15th filter has a peak transmittance at a wavelength of 608 nm. The 16th filter has a peak transmittance at a wavelength of 600 nm. In this way, the filters included in each unit pixel have peak light transmittances that correspond one-to-one to the wavelength bands. Alternatively, the imaging device 100 of this embodiment receives light dispersed into each wavelength in one direction, either vertically or horizontally, for a two-dimensional subject in image space, and accumulates charge for one pixel row. Then, the imaging device 100 moves on to the next pixel row and receives light of each wavelength for that pixel row. By repeating this operation, the imaging device 100 captures the entire image space at all wavelengths using a spatial scanning method.

[0115] Alternatively, the image capturing device 100 of this embodiment receives one of the light beams split into wavelengths for a two-dimensional subject in image space, and accumulates charge over the entire image space. Then, by repeating this operation of receiving the next light beam from the split wavelengths, the image capturing device 100 captures an image of the entire image space at all wavelengths by scanning in the wavelength direction.

[0116] Therefore, in this embodiment, although processes such as acquisition of compressed images and reconstruction processes for generating hyperspectral images from compressed images are not required, the device becomes large in scale relative to the number of pixels, and the operation procedures of the imaging device 100 tend to increase. Alternatively, the generation and acquisition of hyperspectral images that capture the entire image space at all wavelengths tends to require many scanning procedures. Therefore, as with the above embodiment, there is a problem in that it is difficult to provide hyperspectral images on a regular basis.

[0117] Therefore, in this embodiment, the speed of movement of the imaging device 100 is detected by a speed sensor, and depending on the speed of movement, a hyperspectral image is captured and output to the display device 300, or one or more alternative images in place of the hyperspectral image are captured and output to the display device 300.

[0118] Specifically, the imaging system 1000a operates as shown in Fig. 9 . Fig. 9 is a flowchart of an information processing method according to the second embodiment. As shown in Fig. 9 , first, the movement speed of the imaging device 100 is acquired from the speed sensor 400 (S41). The determination unit 260 determines whether the acquired movement speed satisfies a condition indicating that the movement speed is equal to or less than a predetermined threshold (S42). If the determination unit 260 determines that the condition is satisfied (Yes in S42), the determination unit 260 generates and outputs a control signal to the imaging device 100 so as to perform imaging with a first exposure time sufficient to generate a hyperspectral image.

[0119] As a result, the imaging device 100 captures (generates and acquires) a hyperspectral image using the first exposure time (S43). On the other hand, if the determination unit 260 determines that the condition is not satisfied (No in S42), it generates and outputs a control signal to the imaging device 100 to perform imaging using a second exposure time that is shorter than the first exposure time and insufficient for generating a hyperspectral image. As a result, the imaging device 100 captures (generates and acquires) a substitute image in place of the hyperspectral image using the second exposure time (S44). The imaging device 100 outputs the acquired image directly to the display device 300 and displays it on the display unit 320 (S45).

[0120] The substitute image is, for example, an image that includes information on a number of wavelength bands that is fewer than the number of wavelength bands of information included in the hyperspectral image. Specific examples of substitute images include a binary black-and-white image or an RGB image. A substitute image generated with a relatively short exposure time reduces the exposure time (reducing the first exposure time minus the second exposure time) and reduces the number of steps required to provide an image by, for example, combining light charges of multiple wavelengths into a single brightness value, skipping scanning of one or more wavelengths of the multiple wavelengths of information included in the hyperspectral image, or skipping scanning of one or more pixels (lines) in image space, thereby enabling the image to be provided at the same speed as the movement of the imaging device 100.

[0121] Specifically, when combining charges of light of multiple wavelengths into a single brightness value, any of the spatial scanning method, the wavelength scanning method, and the snapshot method can be applied. That is, in any of the methods, charges of light of multiple wavelengths can be combined into a single brightness value to capture a substitute image. Furthermore, when skipping scanning of one or more wavelengths, the wavelength scanning method can be applied. That is, a substitute image can be captured by skipping scanning of one or more wavelengths using the wavelength scanning method. Furthermore, when skipping scanning of one or more pixels (lines), the spatial scanning method can be applied. That is, a substitute image can be captured by skipping scanning of one or more pixels (lines) using the spatial scanning method.

[0122] As an example, when the imaging device 100 acquires a hyperspectral image using the snapshot method, the shortest first exposure time is Ts, and the number of wavelength bands of information contained in the hyperspectral image is B. When the wavelength bands of the hyperspectral image are combined to form a black-and-white image, the exposure time required for one wavelength band to accumulate charge is Ts / B seconds. For example, if Ts = 1 / 2 and B = 16, a binary black-and-white image can be obtained in 1 / 32 seconds (approximately 30 fps).

[0123] Similarly, when the imaging device 100 acquires a hyperspectral image by scanning in the spatial direction, when the charges of each wavelength band of the hyperspectrum are combined to obtain a charge for one wavelength, if, for example, Ts = 5 and B = 150, a binary black and white image can be obtained in 1 / 30 seconds (30 fps). Furthermore, by thinning out the number of lines corresponding to the number of scans in the spatial direction by half, a binary black and white image can be obtained in 1 / 60 seconds (60 fps).

[0124] Next, a modification of the second embodiment will be described. FIG. 10 is a functional configuration diagram of an imaging system according to the modification of the second embodiment. The imaging system 1000b shown in FIG. 10 differs from the first embodiment in that the information processing device 200 includes only an image acquisition unit 210, a storage unit 250, and a determination unit 260, and includes an output unit 270 that only outputs an image instead of the reconstruction calculation unit 240, and in that the display device 300 includes only a display unit 320. The imaging device 100 of this embodiment, like the second embodiment, captures images of the entire image space at all wavelengths using a snapshot method with respect to a two-dimensional subject in image space, using an image sensor that includes a plurality of pixel groups, each of which includes a two-dimensional array of unit pixels, each of which includes a filter that transmits light of each wavelength. Alternatively, the imaging device 100 may capture images of the entire image space at all wavelengths using a method of scanning in the spatial direction or the wavelength direction.

[0125] In this modification, instead of outputting a hyperspectral image, a density image based on the hyperspectral image is generated and output. Therefore, the information processing device 200 includes an image acquisition unit 210 that temporarily acquires an image from the imaging device 100, a storage unit 250 that stores an algorithm for generating the density image, and the like. The information processing device 200 also includes an output unit 270 that outputs the image from the information processing device 200 to the display device 300.

[0126] The imaging system 1000b according to this modification operates as shown in FIG. 11 . FIG. 11 is a flowchart of an information processing method according to a modification of the second embodiment. The flowchart shown in FIG. 11 differs from the flowchart shown in FIG. 9 in that it includes step S46. That is, in the imaging system 1000b according to this modification, a hyperspectral image is acquired using a first exposure time and sent to the information processing device 200. After that, the information processing device 200 generates a density image from the hyperspectral image (S46). Then, the process proceeds to step S45. Note that in this modification, the generated hyperspectral image or alternative image is acquired once by the information processing device 200. Therefore, the velocity sensor 400 is not provided, and a motion vector may be generated to acquire the velocity of the movement of the imaging device 100.

[0127] Next, a third embodiment will be described. Fig. 12 is a functional configuration diagram of an imaging system according to the third embodiment. The imaging system 1000c shown in Fig. 12 differs from the second embodiment in that an imaging drone 100c is used instead of the imaging device 100. Here, as in the second embodiment, a camera using a snapshot method or a method of scanning in the spatial direction or wavelength direction is assumed. However, as in the first embodiment, a configuration in which a compressed image is acquired and a reconstruction process is performed in the information processing device 200 to generate a hyperspectral image may also be used.

[0128] The connection between the imaging drone 100c and the information processing device 200 and the connection between the imaging drone 100c and the display device 300 are made by wireless communication. That is, a transceiver (not shown) is provided between the imaging drone 100c and the information processing device 200, and between the imaging drone 100c and the display device 300.

[0129] The imaging drone 100c of this embodiment also includes a control circuit 110c, instead of the control circuit 110, that also functions to control the autonomous movement of the imaging drone 100c, and also has a built-in speed sensor 400. In addition to the handheld imaging device 100 described above, a similar problem occurs with autonomously moving cameras such as the imaging drone 100c, in that the provision of hyperspectral images cannot keep up with the speed of the camera's movement. Therefore, as in the second embodiment, the determination unit 260 can switch between acquiring a hyperspectral image or acquiring a substitute image depending on the speed of the imaging drone 100c's movement. Also, as in the first embodiment, it may be possible to acquire a compressed image from the imaging drone 100c and switch between generating a hyperspectral image by performing a reconstruction process or using the compressed image as is. Alternatively, the determination unit 260 may be mounted on the imaging drone 100c, and the imaging system 1000c may be realized using only the imaging drone 100c and the display device 300.

[0130] Next, a fourth embodiment will be described. Fig. 13 is a flowchart of an information processing method according to the fourth embodiment. The fourth embodiment differs from the flowchart shown in Fig. 4 in that steps S51 and S52 are executed when step S13 is determined to be No.

[0131] The calculation time required to restore a hyperspectral image can be reduced by reducing the number of spectral bands to be restored. Therefore, when it is determined that there is not enough time to perform restoration processing for all spectral bands before the image is displayed, such as in situations where the speed of movement is fast, the number of spectral bands to be restored can be reduced to shorten the restoration processing time, and the restored spectral band image can be displayed in this shortened time.

[0132] If it is determined in step S13 that the condition is not satisfied (No in S13), it can be determined that the restoration process for all spectral bands will not be completed in time, as described above. In other words, if the restoration results for all spectral bands are waited for before displaying an image, the displayed image will differ significantly from the previous state of the imaging device 100. In such a case, in this embodiment, a specific spectral band to be restored is selected (S51).

[0133] The selection of the spectral band is either specified by a user at any timing such as when starting to capture an image, or is set in advance by a manufacturer, etc. In other words, in step S51, a set value is read out, and a spectral band corresponding to that set value is selected.

[0134] As described above, the number of selected spectral bands, in other words, the number of spectral bands to be restored, must be less than the total number of spectral bands. The number of spectral bands to be selected may be specified by a user or may be preset, such as by a manufacturer. Alternatively, the number of selected spectral bands may be variable according to the speed of the imaging device 100, with fewer spectral bands being selected as the speed increases and more spectral bands being selected as the speed decreases. In this case, a priority may be assigned to each spectral band in advance, and spectral bands with the highest priority may be selected and restored according to the speed, based on the number of bands. Alternatively, a set of spectral bands to be restored according to the speed may be determined in advance. The priority of the spectral bands and the set of spectral bands used here may be arbitrarily set by a user. The set of spectral bands may be, for example, bands corresponding to red (R), green (G), and blue (B), respectively.

[0135] Then, for the spectral band selected as described above, an image of the spectral band is generated by a reconstruction process (S52).

[0136] Next, a fifth embodiment will be described. Fig. 14 is a flowchart of an information processing method according to the fifth embodiment. The fifth embodiment differs from the flowchart shown in Fig. 4 in that step S61 is executed when step S13 is determined to be No.

[0137] Although it is less accurate, it is possible to generate simple images (hereinafter simply referred to as simple images) corresponding to multiple spectral bands from compressed images without performing computationally intensive processes such as iterative calculations or restoration processes using AI models. Therefore, when it is determined that there is not enough time to perform restoration processing for all spectral bands before displaying the image, such as in situations where the speed of movement is fast, a simple image with low accuracy is generated and displayed.

[0138] If it is determined in step S13 that the condition is not satisfied (No in S13), it can be determined that the restoration process for all spectral bands will not be completed in time, as described above. In other words, if an image is displayed after waiting for the restoration results for all spectral bands, the displayed image will differ significantly from the previous state of the imaging device 100. In such a case, in this embodiment, simple images corresponding to multiple spectral bands are generated (S61), although they may be less accurate. Any existing technology may be used to generate the simple images, as long as it satisfies the above-described objective of prioritizing a small amount of processing and a high processing speed over high accuracy.

[0139] Below, several examples of the process of generating a rough image will be described in detail. Figures 15 to 18 are flowcharts relating to the process of generating a rough image in the information processing method according to embodiment 5. Figure 15 is a flowchart that schematically shows an example of a processing method for generating a rough image.

[0140] In generating a simple image, first, a restoration table stored in a storage device such as the storage unit 250 is acquired (S101). The restoration table is table information expressed as a three-dimensional matrix in which the depth represents a wavelength band and the length and width represent pixel values ​​of each mask matrix, or as a two-dimensional matrix in which the width represents a wavelength band and the height represents pixel values ​​of multiple pixels included in each mask matrix.

[0141] In each mask matrix, the pixel values ​​may be normalized by, for example, the maximum gradation value according to the number of bits, and may be expressed in a range of 0 to 1. Alternatively, the pixel values ​​may be expressed in a range of gradations according to the number of bits. In the case of 8 bits, the pixel values ​​are 0 to 255, and the maximum gradation value is 255.

[0142] Each mask matrix is, for example, a two-dimensional matrix with n rows and m columns that represents a two-dimensional distribution of multiple numerical values. This data format makes it easier to understand how pixel values ​​are two-dimensionally distributed for a certain wavelength band.

[0143] Furthermore, each mask matrix may be, for example, a one-dimensional matrix of n x m rows and one column in which multiple numerical values ​​are arranged one-dimensionally, i.e., a vector. This data format allows multiple mask matrices corresponding to multiple wavelength bands to be expressed as a two-dimensional matrix.

[0144] The format of the restoration table is not limited to the above example. For example, the restoration table may be expressed as a two-dimensional matrix of n×N rows and m columns or an n row and m×N column matrix, in which N two-dimensional matrices are arranged vertically or horizontally. Alternatively, the restoration table may be expressed as a one-dimensional matrix of n×m×N rows and 1 column, in which N one-dimensional matrices are arranged vertically.

[0145] A plurality of simple images based on the compressed image are generated by calculation using the restoration table (S102). １ , W ２ , ..., W Ｍ The plurality of rough images generated in step S102 are also referred to as "plurality of rough images of restoration table type."

[0146] The plurality of simple images are generated by performing an operation on the compressed image based on the plurality of mask matrices included in the restoration table. Each simple image is generated using the compressed image and a corresponding mask matrix for the same wavelength band. More specifically, each simple image is generated by multiplying each of a plurality of pixels included in the compressed image by a corresponding matrix element among a plurality of matrix elements included in this one mask matrix.

[0147] Simple Image W ｋ The matrix element in row i and column j contained in ｋ (i, j), the matrix element of row i and column j contained in the compressed image is C(i, j), and the wavelength band W ｋ The matrix element in the i-th row and j-th column included in the mask matrix corresponding to S is defined as M(i, j, k). ｋ (i,j) = C(i,j) x M(i,j,k).

[0148] Instead of not generating a hyperspectral image through the reconstruction process when it is determined that the condition indicating that the speed of motion is equal to or less than a predetermined threshold is not satisfied, as in the present embodiment, it is possible to generate and display images that can be generated even under such circumstances, for example, images with a number of spectral bands fewer than the number of spectral bands contained in the hyperspectral image to be generated. For example, when the number of spectral bands contained in the compressed image is large, such as 20 or more, and the number of spectral bands contained in the hyperspectral image is correspondingly large, such as N, close to 20, if the above condition is not satisfied, the reconstruction process may be performed to generate and display images with only M spectral bands, which are fewer than N (i.e., M<N). Here, M is, for example, three (M=3), corresponding to red, green, and blue.

[0149] FIG. 16 is a flowchart schematically showing an example of another processing method for generating a rough image.

[0150] In generating the simple image here, first, weight matrices of three colors stored in a storage device such as the storage unit 250 are obtained (S201). Details of the method for generating these weight matrices will be described later. The weight matrices are the following information. In this example, a hyperspectral image is generated using a learning model that does not depend on the imaging device 100. As described above, the hyperspectral image includes multiple restored images corresponding to multiple wavelength bands. The multiple restored images corresponding to multiple wavelength bands are, more specifically, four or more restored images corresponding to four or more wavelength bands.

[0151] The weighting matrices are two or more weighting matrices generated based on a restoration table, and correspond to two or more specific wavelength bands. The two or more weighting matrices are generated based on the restoration table. Therefore, it can be said that the two or more weighting matrices contain coding information that reflects the optical transmission spectrum of the coding mask. The two or more weighting matrices are used to generate multiple simplified images. Each weighting matrix contains pixel values ​​of multiple pixels as multiple matrix elements. Each of the multiple matrix elements corresponds to one of the multiple pixels included in the compressed image.

[0152] The two or more specific wavelength bands are two or more wavelength bands included in the plurality of wavelength bands in the restoration table. Therefore, the number of the two or more specific wavelength bands is less than the number of the plurality of wavelength bands in the restoration table. The number of the plurality of wavelength bands may be interpreted as four or more wavelength bands. Furthermore, here, a parameter for reducing resolution used to generate a simplified image may also be used.

[0153] In the following, the two or more specific wavelength bands are referred to as three specific wavelength bands used in general color images: a red wavelength band, a green wavelength band, and a blue wavelength band. The central wavelength of the red wavelength band is in the range of 620 nm to 750 nm, the central wavelength of the green wavelength band is in the range of 495 nm to 570 nm, and the central wavelength of the blue wavelength band is in the range of 450 nm to 495 nm. The width of each of the multiple wavelength bands included in the target wavelength range W may be, for example, 10 mm or less. As an example, the red wavelength band is in the range of 681 nm to 690 nm, the green wavelength band is in the range of 538 nm to 547 nm, and the blue wavelength band is in the range of 468 nm to 467 nm. The widths of the multiple wavelength bands included in the target wavelength range W are all the same.

[0154] The weight matrix corresponding to the red wavelength band is also referred to as the “red weight matrix,” the weight matrix corresponding to the green wavelength band is also referred to as the “green weight matrix,” and the weight matrix corresponding to the blue wavelength band is also referred to as the “blue weight matrix.” The red, green, and blue weight matrices are also referred to as the “three-color weight matrix.”

[0155] The three-color weighting matrices are expressed as three-dimensional matrices in which the depth represents the wavelength band and the vertical and horizontal axes represent the pixel values ​​of the multiple pixels included in each weighting matrix. Alternatively, the three weighting matrices may be expressed as two-dimensional matrices in which the horizontal axes represent the wavelength band and the vertical axes represent the pixel values ​​of the multiple pixels included in each weighting matrix. After the weighting matrices are obtained (S201), multiple simple images based on the compressed image are generated by calculation using the three-color weighting matrices (S202). The multiple simple images generated in step S202 are also referred to as "multiple weighting matrix-type simple images."

[0156] 17 is a flowchart showing the details of the operation in step S202. In step S202, the following operations in steps S202a to S202c are executed.

[0157] <Step S202a> Three pseudo simple images corresponding to the three color wavelength bands are generated by weighting the compressed image by the three color weighting matrices. In this specification, the pseudo simple images are simply referred to as "pseudo images." Each pseudo image is generated using the compressed image and a corresponding weighting matrix for the same color wavelength band. More specifically, each pseudo image is generated by multiplying each of a plurality of pixels included in the compressed image by a corresponding matrix element from among a plurality of matrix elements included in this one weighting matrix.

[0158] <Step S202b> Next, the resolution of each pseudo image is reduced. More specifically, the pixel values ​​of multiple pixels included in each pseudo image are averaged in units of n x n images. n is a parameter for reducing the resolution and is also used when generating a three-color weighting matrix based on the restoration table. n can be, for example, between 2 and 10.

[0159] <Step S202c> Next, the dynamic range of the pixel values ​​of the multiple pixels included in each pseudo image with reduced resolution is adjusted, and the pseudo image is output as a simplified image. The dynamic range may be, for example, a range of 0 to 1 inclusive, in which the pixel values ​​are normalized by the maximum gradation value corresponding to the number of bits. Alternatively, the dynamic range may be, for example, a gradation range corresponding to the number of bits. For example, if the averaged pixel value falls outside the range of image luminance values, such as 0 to 255, the dynamic range may be adjusted by applying the minimum or maximum luminance value to adjust it to a certain range. If the luminance value of the averaged pixel is below the minimum value, the minimum value is applied, and if it exceeds the upper limit of the luminance value, the upper limit of the luminance value is applied.

[0160] 18 is a flowchart showing an example of a weighting matrix generation method that is executed before executing the processing method for generating a rough image. In generating the weighting matrix, the following steps S301 to S303 are executed.

[0161] <Step S301> The restoration table is obtained from the storage device.

[0162] <Step S302> Next, weight matrices for three colors are determined as follows: Each weight matrix is ​​determined so that, for the pixel values ​​of multiple pixels included in the mask matrix corresponding to the same color wavelength band in the restoration table, a high weight is assigned to a high pixel value and a low weight is assigned to a low pixel value.

[0163] Waveband W in the reconstruction table ｋ The matrix elements included in the matrix corresponding to are divided into n×n pixel units. The same applies to the matrix elements included in the three-color weight matrix. n is as described in step S202b shown in FIG. 17. The same n is used in steps S202b and S302. If n=4, one unit contains 16 pixels.

[0164] Waveband W in the reconstruction table ｋFor the matrix corresponding to the matrix, the matrix element for each unit of n × n pixels is M(u, v, k), and for the red weight matrix, the matrix element for each unit of n × n pixels is S Ｒ (u, v), and for the green weight matrix, the matrix elements in each unit of n × n pixels are S Ｇ (u, v), and for the blue weight matrix, the matrix elements in each unit of n × n pixels are S Ｂ Let (u, v).

[0165] S Ｒ (u, v) can be determined, for example, so that the function of the following equation (4) is minimized while satisfying the following equation (5). Ｒ represents the red wavelength band.

[0166]

[0167] S Ｇ (u, v) can be determined, for example, so that the function of the following equation (6) is minimized while satisfying the following equation (7). Ｇ represents the green wavelength band.

[0168]

[0169] S Ｂ (u, v) can be determined, for example, so that the function of the following equation (8) is minimized while satisfying the following equation (9). Ｂ represents the blue wavelength band.

[0170]

[0171] The S generated as above Ｒ (u, v), S Ｇ (u, v), and S Ｂ (u, v) is based on the assumption that the pixels in each n×n pixel unit have the same spectrum. Therefore, when generating the simple images, the operation of step S202b is executed to reduce the resolution of each pseudo image.

[0172] <Step S303> Next, the balance of the multiple matrix elements included in each weighting matrix is ​​adjusted as follows. Using a general color image as the sample image, the balance of the multiple matrix elements included in each weighting matrix can be adjusted to satisfy the following condition. This condition is that an image obtained by weighting the sample image with each weighting matrix is ​​similar to an image of the same color included in the sample image. The image obtained by weighting the sample image with each weighting matrix is ​​generated by multiplying each of the multiple pixels included in the sample image by a corresponding matrix element from the multiple matrix elements included in the weighting matrix.

[0173] Alternatively, the compressed image may be used as a sample image, and the balance of the matrix elements included in each weighting matrix may be adjusted to satisfy the following condition: In an image obtained by weighting the sample image with each weighting matrix, the average value of the pixel values ​​of the pixels, i.e., the average pixel value, is the same regardless of red, green, or blue. To satisfy this condition, at least one of the weighting matrices for the three colors is multiplied by a correction constant.

[0174] For example, the correction constant may be set so that the average pixel value of an image obtained by weighting a sample image by a weighting matrix for one color matches the average pixel value of an image obtained by weighting a sample image by a weighting matrix for another color, or the correction constant may be set so that the average pixel value of an image obtained by weighting a sample image by each weighting matrix matches a particular value, for example, half the maximum pixel value.

[0175] Instead of not generating a hyperspectral image through the reconstruction process when it is determined that the condition indicating that the speed of motion is equal to or less than a predetermined threshold is not satisfied, as in the present embodiment, a simplified image that can be generated even under such circumstances without going through the reconstruction process may be generated and displayed. For example, when the number of spectral bands included in the compressed image is large, such as 20 or more, and the number of spectral bands included in the hyperspectral image is correspondingly large, such as N, close to 20, the simplified image may include N spectral bands, which is the same as the number of spectral bands included in the hyperspectral image, or M spectral bands, which is less than N (i.e., M<N). Here, M is, for example, three (M=3), corresponding to red, green, and blue.

[0176] (Other Embodiments) While the information processing method has been described above based on the embodiments and modifications, the information processing method according to the present disclosure is not limited to the above embodiments and modifications. The present disclosure also includes other embodiments realized by combining any of the components in the above embodiments and modifications, and modifications obtained by applying various modifications that would occur to a person skilled in the art to the above embodiments and modifications without departing from the spirit of the present disclosure.

[0177] For example, in the above embodiment, a hyperspectral image was defined as an image including four or more spectroscopic images, but it may also be defined as an image including five or more, six or more, seven or more, eight or more, nine or more, or ten or more spectroscopic images.

[0178] Furthermore, in the imaging device 100 according to the above embodiment and modified example, the filter array 130 is used as an encoding element, but this is not limited to this. For example, a metalens described in Non-Patent Document 1 may be used instead of the filter array 130 according to the above embodiment and modified example. The metalens includes multiple regions with different transmission spectra. Furthermore, for example, an image sensor described in Non-Patent Document 2 may be used instead of the filter array 130 and image sensor 120 according to the above embodiment and modified example. The sensing region of the image sensor is processed so that light having a predetermined spectrum is obtained for each pixel.

[0179] The present disclosure is useful when utilizing hyperspectral imagery.

[0180] REFERENCE SIGNS LIST 100 Imaging device 100c Imaging drone 110, 110c Control circuit 120 Image sensor 130 Filter array 130a, 130b Filter 140 Optical system 200 Information processing device 210 Image acquisition unit 220 Information acquisition unit 230 Parameter determination unit 240 Reconstruction calculation unit 250 Storage unit 260 Determination unit 300 Display device 310 Input unit 320 Display unit 1000 Imaging system

Claims

1. An information processing method executed by a computer, comprising: capturing an image of a subject using a camera; acquiring, during a first period, a first compressed image which is a compressed image including a plurality of pixels, wherein data for each of the plurality of pixels includes information of four or more wavelength bands, and which is used to generate a hyperspectral image by a reconstruction process; acquiring a speed of movement of the camera; determining whether or not, during the first period, a condition indicating that the speed of movement is equal to or less than a predetermined threshold is satisfied; and if it is determined that the condition is not satisfied during the first period, not generating the hyperspectral image in the reconstruction process based on the first compressed image.

2. The information processing method of claim 1, further comprising obtaining a second compressed image, which is the compressed image, during a second period prior to the first period, and, if it is determined that the condition is satisfied during the first period, generating the hyperspectral image based on the first compressed image or the hyperspectral image based on the second compressed image by the reconstruction process.

3. The information processing method of claim 1, further comprising: generating the hyperspectral image by the reconstruction process based on the first compressed image when it is determined that the condition is satisfied during the first period; if the generation of the hyperspectral image in the reconstruction process based on the first compressed image is completed at the end of the first period, outputting an image related to the generated hyperspectral image for display on a display device; and if the generation of the hyperspectral image in the reconstruction process based on the first compressed image is not completed at the end of the first period, outputting the first compressed image for display on the display device.

4. The information processing method according to claim 3, wherein the image related to the hyperspectral image is the hyperspectral image itself or an image showing the analysis result of the subject analyzed based on the hyperspectral image.

5. The information processing method according to claim 1, further comprising the step of outputting the first compressed image and displaying it on a display device when it is determined that the condition is not satisfied during the first period.

6. The information processing method according to claim 2, wherein the speed of movement is obtained as the speed of relative movement between the camera and the subject, derived based on a matching process between the subject included in the first compressed image and the subject included in the second compressed image.

7. The information processing method according to claim 1, wherein the speed of the movement is obtained as a detection result detected by a sensor that detects the speed of the camera.

8. An information processing method executed by a computer, comprising: acquiring a speed of camera movement; determining whether the speed of movement satisfies a condition indicating that the speed of movement is equal to or less than a predetermined threshold; if it is determined that the condition is satisfied, using the camera to capture an image of a subject with a first exposure time, and acquiring a hyperspectral image including a plurality of images corresponding to four or more wavelength bands; if it is determined that the condition is not satisfied, using the camera to capture an image of a subject with a second exposure time shorter than the first exposure time, and acquiring a substitute image in place of the hyperspectral image.

9. The information processing method according to claim 8, wherein the alternative image includes a number of images that is fewer than the number of images included in the hyperspectral image.

10. The information processing method according to claim 8, wherein the camera detects light passing through a plurality of optical filters, each of the plurality of optical filters having a transmittance peak that corresponds one-to-one to one of four or more wavelength bands.

11. The information processing method according to claim 8, wherein the camera generates the hyperspectral image by scanning in a wavelength direction or a spatial direction.

12. The information processing method according to claim 1, further comprising the steps of: generating and outputting a display image including information of three or less wavelength bands based on the first compressed image when it is determined that the condition is not satisfied during the first period.

13. The information processing method according to claim 12, further comprising obtaining a second compressed image, which is the compressed image, during a second period prior to the first period, and generating the display image based on the first compressed image or the display image based on the second compressed image if it is determined that the condition is not satisfied during the first period.

14. The information processing method according to claim 12, wherein the display image is generated without involving the reconstruction process.

15. An information processing method executed by a computer, comprising: capturing an image of a subject using a camera; acquiring, during a first period, a first compressed image including a plurality of pixels, wherein data for each of the plurality of pixels includes information of four or more wavelength bands, the first compressed image being used to generate a hyperspectral image by a reconstruction process; acquiring a speed of movement of the camera; determining whether, during the first period, a condition indicating that the speed of movement is equal to or less than a predetermined threshold is satisfied; if it is determined that the condition is satisfied during the first period, displaying a first screen based on a plurality of images corresponding to each of N wavelength bands; and if it is determined that the condition is not satisfied during the first period, displaying (i) the compressed image or (ii) a second screen based on a plurality of images corresponding to each of M wavelength bands (M<N).

16. The information processing method according to claim 15, wherein if it is determined that the condition is not satisfied during the first period, the reconstruction process based on the compressed image generates the multiple images corresponding to each of the M wavelength bands, and displays the second screen.

17. The information processing method according to claim 15, wherein if it is determined that the condition is not satisfied during the first period, the plurality of images corresponding to each of the M wavelength bands are generated without going through the reconstruction process based on the compressed image, and the second screen is displayed.

18. An imaging system comprising: an image acquisition unit that uses a camera to capture an image of a subject, and acquires, during a first period, a first compressed image which is a compressed image including a plurality of pixels, wherein data for each of the plurality of pixels includes information of four or more wavelength bands, and is used to generate a hyperspectral image by a reconstruction process; an information acquisition unit that acquires a speed of movement of the camera; a determination unit that determines whether, during the first period, a condition indicating that the speed of movement is equal to or less than a predetermined threshold is satisfied; and a reconstruction unit that does not generate the hyperspectral image in the reconstruction process based on the first compressed image if it is determined that the condition is not satisfied during the first period.

19. An imaging system comprising: an acquisition unit that acquires the speed of camera movement; a judgment unit that judges whether the speed of the movement satisfies a condition indicating that it is equal to or less than a predetermined threshold; and an image acquisition unit, wherein the image acquisition unit, when it judges that the condition is satisfied, causes the camera to capture an image of the subject with a first exposure time, and acquires a hyperspectral image including a plurality of pixels, each of the plurality of pixels including information of four or more wavelength bands; and when it judges that the condition is not satisfied, causes the camera to capture an image of the subject with a second exposure time shorter than the first exposure time, and acquires a substitute image in place of the hyperspectral image.

Images