Information processing method and imaging system

Abstract

An information processing method executed by a computer includes: capturing a subject with a camera, acquiring a first compressed image in a first period, the first compressed image being a compressed image containing a plurality of pixels, and being a compressed image used for generating a hyperspectral image through a reconstruction process, the data of each of the plurality of pixels containing information of four or more wavebands (S11); acquiring a moving speed of the camera (S12); determining whether the moving speed is below a predetermined threshold in the first period (S13); and in a case where it is determined that the condition is not satisfied in the first period, not performing generation of the hyperspectral image through the reconstruction process based on the first compressed image.

Description

Technical Field

[0001] The present disclosure relates to an information processing method and an imaging system. Background Art

[0002] Techniques for capturing hyperspectral images have been proposed. Hyperspectral images have spectral information in more bands (4 or more bands (wavelength bands)) than the three colors of red, green, and blue. For example, it is known that by applying compressive sensing technology to a camera and performing a reconstruction process on the compressed image obtained by compressive sensing, a hyperspectral image can be generated. Compressive sensing, as shown in the above example, is a technique for acquiring the acquired data in such a way that more data can be generated from data with a smaller number of samples in the acquired data. For example, in Patent Documents 1 and 2, a compressed image is captured using a filter array including a plurality of filters having different wavelength dependences, and four or more spectral images (hyperspectral images) corresponding one-to-one to four or more bands are restored based on the captured compressed image.

[0003] Prior Art Documents

[0004] Patent Documents

[0005] Patent Document 1: International Publication No. 2021 / 192891

[0006] Patent Document 2: Japanese Patent No. 7262003 Gazette

[0007] Non-Patent Documents

[0008] Non-Patent Document 1: "The world's first technology to combine a metalens and artificial intelligence with an ordinary digital camera to obtain hyperspectral images and videos - Combining optical technology and artificial intelligence to turn an 'ordinary camera' into a 'camera that can see the properties of substances' (世界初、通常のデジタルカメラにメタレンズとAIを組み合わせてハイパースペクトル画像・動画の取得を実現する技術を確立~光技術とAIの融合で「普通のカメラ」を「モノの性質が見えるカメラ」に~)", [Online], October 24, 2022, Nippon Telegraph and Telephone Corporation, Retrieval Date: June 15, 2023, Internet <URL: https: / / group.ntt / jp / newsrelease / 2022 / 10 / 24 / 221024a.html>

[0009] Non-Patent Literature 2: Ahasan Ahamed et al., "Reconstruction-based spectroscopy using CMOS image sensors with random photon-trapping nanostructure per sensor", Proc. SPIE 11971, High-Speed ​​Biomedical Imaging and Spectroscopy VII, 1197106, March 2, 2022. Summary of the Invention

[0010] However, acquiring hyperspectral images requires more steps than acquiring typical RGB or monochrome images, which correspond one-to-one with fewer than four spectral bands. Therefore, in the past, it was sometimes impossible to provide adequate hyperspectral images.

[0011] Therefore, this disclosure provides an information processing method and an imaging system capable of appropriately providing hyperspectral images.

[0012] One aspect of this disclosure relates to an information processing method executed by a computer, comprising: taking a picture of a subject using a camera; acquiring a first compressed image during a first period, the first compressed image being a compressed image containing multiple pixels and being a compressed image for generating a hyperspectral image through a reconstruction process, wherein the data of each of the multiple pixels contains information of four or more bands; acquiring the movement speed of the camera; determining whether the condition that the movement speed is below a predetermined threshold is met during the first period; and, if it is determined that the condition is not met during the first period, not generating the hyperspectral image based on the first compressed image.

[0013] Furthermore, this general or specific technical solution can be implemented by a system, apparatus, integrated circuit, computer program, or computer-readable recording medium, or by any combination of a system, apparatus, method, integrated circuit, computer program, and recording medium. Computer-readable recording media include, for example, non-volatile recording media such as CD-ROM (Compact Disc-Read Only Memory).

[0014] According to this disclosure, it is possible to provide hyperspectral images appropriately. Attached Figure Description

[0015] Figure 1 This is a functional configuration (structure) diagram of the shooting system involved in Implementation Method 1.

[0016] Figure 2 This is a schematic diagram of the shooting device according to Embodiment 1.

[0017] Figure 3A This is a schematic diagram of the filter array involved in Implementation Method 1.

[0018] Figure 3B This is a diagram showing an example of the transmission spectrum of the filter involved in Embodiment 1.

[0019] Figure 3C This is a diagram showing an example of the transmission spectrum of another filter involved in Embodiment 1.

[0020] Figure 3D This is a diagram showing an example of the transmittance of the first band of the filter array involved in Embodiment 1.

[0021] Figure 3E This is a diagram showing an example of the transmittance of the second band of the filter array involved in Embodiment 1.

[0022] Figure 4 This is a flowchart of the information processing method involved in Implementation Method 1.

[0023] Figure 5 This is a flowchart of an information processing method involved in a variation of Implementation 1.

[0024] Figure 6 This is a flowchart of the information processing method involved in Variation 2 of Implementation 1.

[0025] Figure 7 This is a diagram showing an example of an image related to a variation 2 of embodiment 1.

[0026] Figure 8 This is a functional configuration diagram of the shooting system involved in Implementation Method 2.

[0027] Figure 9 This is a flowchart of the information processing method involved in Implementation Method 2.

[0028] Figure 10 This is a functional configuration diagram of the shooting system involved in a variation of implementation method 2.

[0029] Figure 11 This is a flowchart of an information processing method involved in a variation of Implementation 2.

[0030] Figure 12 This is a functional configuration diagram of the shooting system involved in Implementation Method 3.

[0031] Figure 13 This is a flowchart of the information processing method involved in Implementation Method 4.

[0032] Figure 14 This is a flowchart of the information processing method involved in Implementation Method 5.

[0033] Figure 15 This is a flowchart related to the simplified image generation process in the information processing method involved in Implementation Method 5.

[0034] Figure 16 This is a flowchart related to the simplified image generation process in the information processing method involved in Implementation Method 5.

[0035] Figure 17 This is a flowchart related to the simplified image generation process in the information processing method involved in Implementation Method 5.

[0036] Figure 18 This is a flowchart related to the simplified image generation process in the information processing method involved in Implementation Method 5. Detailed Implementation

[0037] (Summary of this disclosure)

[0038] Before describing the implementation methods, a summary of this disclosure will be given.

[0039] As mentioned earlier, acquiring hyperspectral images requires more steps than acquiring typical RGB or monochrome images, which correspond one-to-one with fewer than four spectral bands. Specifically, when generating hyperspectral images through spatial or wavelength scanning, sufficient exposure time needs to be set for each unit space or each unit wavelength in the scan. Moreover, since such exposure time is required across the entire scan range, a long exposure time is needed to acquire a single hyperspectral image. Furthermore, when generating hyperspectral images through compressed sensing as explained in the background section above, the processing of the sensed information for reconstructing the hyperspectral image requires a considerable processing time.

[0040] For example, to present hyperspectral images as moving images, a frame rate is needed to ensure they are rendered as moving images. However, the numerous steps involved in acquiring hyperspectral images make maintaining this frame rate difficult. For instance, assuming acquiring a hyperspectral image takes 100 ms, theoretically, the hyperspectral image can only be provided at a frame rate of 10 fps (frames per second). This is far from the frame rate required to render the image as a moving image.

[0041] There is a need for real-time two-dimensional spectral analysis using hyperspectral images. This requires moving the hyperspectral camera to the desired viewing angle using methods such as handheld operation, and then generating a hyperspectral image at that desired viewing angle. Moreover, when the desired viewing angle is observed in two dimensions, the emission characteristics (including reflection and absorption characteristics) of each band after spectral dispersion can be known pixel by pixel.

[0042] In such a scenario, if the hyperspectral camera is moved while generating a hyperspectral image, the provision of the hyperspectral image will not keep up with (cannot follow) the moving viewpoint, making it impossible to perform real-time two-dimensional spectral analysis while simultaneously confirming the subject within the viewpoint. In other words, it is impossible to provide a suitable hyperspectral image.

[0043] Therefore, in this disclosure, by setting a predetermined condition for the moving speed (in other words, the speed of activity) of the movable hyperspectral camera, hyperspectral images are not provided if the predetermined condition is not met (that is, the moving speed is greater than a predetermined threshold), thereby suppressing the situation where the provision of hyperspectral images cannot keep up and providing hyperspectral images appropriately.

[0044] To achieve the above objectives, the information processing method according to the first aspect of this disclosure is an information processing method executed by a computer, comprising: taking a picture of a subject using a camera; acquiring a first compressed image during a first period, the first compressed image being a compressed image containing multiple pixels and being a compressed image used to generate a hyperspectral image through a reconstruction process, wherein the data of each of the multiple pixels contains information of four or more bands; acquiring the movement speed of the camera; determining whether the condition that the movement speed is below a predetermined threshold is met during the first period; and not generating a hyperspectral image based on the first compressed image if it is determined that the condition is not met during the first period.

[0045] Therefore, depending on the camera's activity speed, if the activity speed is not below a predetermined threshold, the generation of a hyperspectral image through a reconstruction process can be omitted. The reconstruction process for generating a hyperspectral image requires a long processing time; therefore, when the activity speed is not below the predetermined threshold, the provision of hyperspectral images will be insufficient. In this case, by omitting the reconstruction process for generating the hyperspectral image, the hyperspectral image can be provided appropriately without performing the reconstruction process. Furthermore, omitting the reconstruction process for generating the hyperspectral image also has advantages such as saving processing resources.

[0046] The information processing method according to the second aspect of this disclosure is the same as the information processing method according to the first aspect. In the second period earlier than the first period, a second compressed image is obtained as a compressed image; if it is determined that the conditions are met in the first period, a hyperspectral image based on the first compressed image or a hyperspectral image based on the second compressed image is generated through a reconstruction process.

[0047] Accordingly, a hyperspectral image can be generated and provided based on a compressed image obtained in the first or second period through a reconstruction process when the activity speed is below a predetermined threshold.

[0048] The information processing method according to the third aspect of this disclosure is the same as the information processing method according to the first or second aspect. When it is determined that the conditions are met during the first period, a hyperspectral image is generated by a reconstruction process based on the first compressed image. If the generation of the hyperspectral image based on the first compressed image through the reconstruction process is completed at the end of the first period, an image related to the generated hyperspectral image is output and displayed on the display device. If the generation of the hyperspectral image based on the first compressed image through the reconstruction process is not completed at the end of the first period, the first compressed image is output and displayed on the display device.

[0049] Accordingly, if the activity speed is below a predetermined threshold, and the generation of a hyperspectral image during the reconstruction process has been completed, an image related to that hyperspectral image can be output and displayed; if the generation of a hyperspectral image during the reconstruction process has not been completed, a compressed image can be output and displayed. In other words, in addition to the camera's activity speed, a hyperspectral image can be appropriately provided based on the status of the reconstruction process used to generate the hyperspectral image.

[0050] The information processing method in the fourth aspect of this disclosure is the same as that in the third aspect, wherein the image associated with the hyperspectral image is the hyperspectral image itself or an image representing the analysis results of the subject obtained based on the analysis of the hyperspectral image.

[0051] Accordingly, when the reconstruction process used to generate the hyperspectral image is completed, the hyperspectral image itself or an image representing the analysis results of the subject obtained based on the hyperspectral image analysis can be output and displayed.

[0052] The information processing method according to the fifth aspect of this disclosure is the same as the information processing method according to any one of the first to fourth aspects, which outputs a first compressed image and causes the display device to display it when it is determined that the conditions are not met in the first period.

[0053] Therefore, it is possible to output and display a compressed image even when the activity speed is not below a predetermined threshold.

[0054] The information processing method involved in the sixth aspect of this disclosure is the information processing method involved in any one of the first to fifth aspects, wherein the activity speed is obtained as the relative activity speed between the camera and the subject derived from the matching processing of the subject contained in the first compressed image and the subject contained in the second compressed image.

[0055] Therefore, the motion speed can be derived from compressed images at two time points. Since only the camera's image sensor is needed to obtain the motion speed, no additional sensors such as a speed sensor are required, which is advantageous from the perspective of device cost.

[0056] The information processing method involved in the seventh aspect of this disclosure is the same as that involved in any one of the first to sixth aspects, wherein the activity speed is obtained as a detection result detected by a sensor of the speed of the detection camera.

[0057] Therefore, the activity velocity can be obtained from the detection results at a single time point. Since the activity velocity can be obtained by capturing the detection results at each individual time point, it has advantages from a temporal resolution perspective compared to methods that require multiple time points.

[0058] The information processing method disclosed in the eighth aspect is an information processing method executed by a computer, comprising: obtaining the movement speed of a camera; determining whether the condition that the movement speed is below a predetermined threshold is met; if the condition is met, taking a picture of a subject with the camera at a first exposure time to obtain a hyperspectral image containing multiple images corresponding to four or more bands respectively; if the condition is not met, taking a picture of the subject with the camera at a second exposure time shorter than the first exposure time to obtain a substitute image instead of the hyperspectral image.

[0059] Accordingly, depending on the camera's operating speed, if the operating speed is not below a predetermined threshold, a hyperspectral image can be obtained instead of a substitute image. Since acquiring a hyperspectral image requires a long exposure time, the provision of hyperspectral images will be insufficient when the operating speed is not below the predetermined threshold. In this case, by not acquiring a hyperspectral image, and thus not providing a hyperspectral image, it is possible to provide a hyperspectral image appropriately. Furthermore, obtaining a substitute image instead of a hyperspectral image also has the advantage of being able to utilize the substitute image.

[0060] The information processing method disclosed in the ninth aspect is the same as that disclosed in the eighth aspect, wherein the alternative image contains fewer images than the multiple images contained in the hyperspectral image.

[0061] Therefore, it is possible to replace a hyperspectral image with an alternative image containing fewer images than the number of images contained in a hyperspectral image.

[0062] The information processing method disclosed in the 10th aspect is the same as that disclosed in the 8th or 9th aspect, wherein a camera detects light passing through a plurality of optical filters, each of the plurality of optical filters having a peak transmittance corresponding one-to-one with any one of four or more bands.

[0063] Accordingly, hyperspectral images or alternative images can be obtained by detecting light passing through multiple optical filters that have peak transmittance values ​​corresponding one-to-one with any of four or more bands.

[0064] The information processing method disclosed in aspect 11 is the same as that disclosed in aspect 8 or 9, wherein the camera generates a hyperspectral image by scanning in the wavelength direction or the spatial direction.

[0065] Therefore, hyperspectral images can be obtained by scanning in the wavelength direction or the spatial direction. Alternatively, alternative images can be obtained by scanning a portion of the wavelength direction and the spatial direction.

[0066] The information processing method according to the 12th aspect of this disclosure is the same as the information processing method according to any one of the 1st to 11th aspects, wherein, if it is determined that the conditions are not met in the first period, a display image containing information of three or fewer bands is generated and output based on the first compressed image.

[0067] Accordingly, it is possible to generate, output, and display a display image containing information from three or fewer bands, even when the activity speed is not below a predetermined threshold.

[0068] The information processing method according to the 13th aspect of this disclosure is the same as the information processing method according to the 12th aspect, wherein a second compressed image is obtained as a compressed image in a second period earlier than the first period; and if it is determined that the conditions are not met in the first period, a display image based on the first compressed image or a display image based on the second compressed image is generated.

[0069] Accordingly, even when the activity speed is not below a predetermined threshold, it is possible to generate, output, and display a display image containing information of three or fewer bands based on a first compressed image or a display image containing information of three or fewer bands based on a second compressed image.

[0070] The information processing method relating to the 14th aspect of this disclosure is the same as the information processing method relating to the 12th or 13th aspect, wherein the display image is generated without a reconstruction process.

[0071] Accordingly, a display image can be generated and displayed through a process different from the reconstruction process.

[0072] The information processing method according to the 15th aspect of the present disclosure is an information processing method executed by a computer, including: photographing a subject using a camera; obtaining a first compressed image during a first period, the first compressed image being a compressed image including a plurality of pixels and being a compressed image for generating a hyperspectral image through a reconstruction process, and data of each of the plurality of pixels including information of four or more bands; obtaining the movement speed of the camera; determining whether the condition that the movement speed is below a predetermined threshold is satisfied during the first period; when it is determined that the condition is satisfied during the first period, displaying a first screen based on a plurality of images respectively corresponding to N bands; and when it is determined that the condition is not satisfied during the first period, displaying a second screen based on (i) the compressed image or (ii) a plurality of images respectively corresponding to M (M < N) bands.

[0073] Accordingly, according to the movement speed of the camera, when the movement speed is below a predetermined threshold, a first screen composed of images including information of N bands can be displayed based on the compressed image. On the other hand, when the movement speed is not below a predetermined threshold, a second screen composed of images including information of M (M < N) bands can be displayed. Since the reconstruction process for generating a hyperspectral image including information of relatively many N bands requires a long processing time, when the movement speed is not below a predetermined threshold, the provision of the hyperspectral image will not keep up. In this case, by generating an image such as a hyperspectral image including relatively few M bands of information to replace the hyperspectral image including information of N bands, the provision of the hyperspectral image can be appropriately performed. In addition, in the reconstruction process for generating a hyperspectral image, reducing the number of bands that are the objects of the process also has advantages such as saving processing resources.

[0074] The information processing method according to the 16th aspect of the present disclosure is the information processing method according to the 15th aspect. When it is determined that the condition is not satisfied during the first period, a plurality of images respectively corresponding to M bands are generated through a reconstruction process based on the compressed image, and the second screen is displayed.

[0075] Accordingly, an image including information of M bands can be generated, output, and displayed through the reconstruction process.

[0076] The information processing method according to the 17th aspect of the present disclosure is the information processing method according to the 15th aspect. When it is determined that the condition is not satisfied during the first period, a plurality of images respectively corresponding to M bands are generated without going through a reconstruction process based on the compressed image, and the second screen is displayed.

[0077] Therefore, an image can be generated and displayed on the second screen through a process different from the reconstruction process.

[0078] The 18th aspect of this disclosure relates to an imaging system comprising: an image acquisition unit that captures an image of a subject using a camera and acquires a first compressed image during a first period, the first compressed image being a compressed image containing multiple pixels and used to generate a hyperspectral image through a reconstruction process, wherein the data of each of the multiple pixels contains information of four or more bands; an information acquisition unit that acquires the movement speed of the camera; a determination unit that determines whether the condition that the movement speed is below a predetermined threshold is met during the first period; and a reconstruction unit that, if it is determined that the condition is not met during the first period, does not generate a hyperspectral image based on the first compressed image through a reconstruction process.

[0079] Therefore, it can achieve the same effect as the information processing methods mentioned above.

[0080] The 19th aspect of this disclosure relates to an imaging system comprising: an acquisition unit that acquires the movement speed of a camera; a determination unit that determines whether the condition that the movement speed is below a predetermined threshold is met; and an image acquisition unit that, when the condition is met, captures an image of a subject using the camera at a first exposure time to acquire a hyperspectral image containing multiple pixels, each of the multiple pixels containing information in four or more bands; and, when the condition is not met, captures an image of the subject using the camera at a second exposure time shorter than the first exposure time to acquire a substitute image for the hyperspectral image.

[0081] Therefore, it can achieve the same effect as the information processing methods mentioned above.

[0082] (Implementation Method)

[0083] The embodiments will now be described in detail with reference to the accompanying drawings.

[0084] Furthermore, the embodiments described below are either general or specific examples. The numerical values, shapes, materials, constituent elements, the arrangement and connection methods of constituent elements, steps, and the order of steps shown in the following embodiments are all examples and are not intended to limit the technology of this disclosure.

[0085] Furthermore, these figures are illustrative and not strictly precise diagrams. Therefore, for example, the scale may not be entirely consistent across different figures. Also, for substantially identical components, the same reference numerals are used, and sometimes redundant descriptions are omitted or simplified.

[0086] In addition, the following terms, such as parallel or perpendicular, which indicate the relationship between elements, and terms, such as rectangle or circle, which indicate the shape of elements, and numerical ranges, do not only represent strict expressions, but also include substantially equivalent ranges, such as expressions of a difference of a few percent.

[0087] Additionally, all or part of a circuit, unit (section), or device, or all or part of a functional block in a block diagram, can be implemented, for example, by one or more electronic circuits including semiconductor devices, integrated circuits (ICs), or large-scale integration (LSIs). An LSI or IC can be integrated on a single chip or composed of multiple chips. For example, functional blocks other than memory elements can also be integrated on a single chip. Here, although referred to as LSI or IC, the terminology varies depending on the degree of integration; it can also be called system LSI, VLSI (very large-scale integration), or ULSI (ultra-large-scale integration). FPGAs (Field Programmable Gate Arrays) programmed after LSI manufacturing, or RLDs (Reconfigurable Logic Devices) capable of reconfiguring the internal bonding relationships or circuit partitioning within an LSI, can also be used for the same purpose.

[0088] Furthermore, all or part of a circuit, unit (section), or device, or all or part of a functional block in a block diagram, can also be implemented by a software program. In this case, the software program is recorded on one or more non-transitory recording media such as ROM, optical disk, or hard disk drive. When the software program is executed by a processor, the processor and peripheral devices perform the functions determined by the software program. The system or device may also include one or more non-transitory recording media storing the software program, a processor, and the necessary hardware devices (e.g., memory, interfaces, etc.).

[0089] [Functional Composition of the Camera System 1000]

[0090] First, refer to Figure 1 The functional configuration of the shooting system 1000 involved in this embodiment will be described in detail. Figure 1 This is a functional configuration diagram of the shooting system 1000 according to this embodiment. Furthermore, Figure 1 This describes an exemplary functional configuration of the shooting system 1000; the functional configuration of the shooting system 1000 is not limited to... Figure 1 .

[0091] like Figure 1 As shown, the shooting system 1000 includes a shooting device 100, an information processing device 200, and a display device 300. Hereinafter, the shooting device 100, speed sensor 400, information processing device 200, and display device 300 included in the shooting system 1000 will be described in detail.

[0092] The imaging device 100 is an example of the camera according to Embodiment 1, and has the same configuration as the imaging device disclosed in Patent Document 1, and is capable of capturing compressed images containing multiple pixels. Each of the multiple pixels in the compressed image contains information in four or more wavelengths. The imaging device 100 includes a control circuit 110 and an image sensor 120.

[0093] The control circuit 110 controls the image sensor 120, enabling the image sensor 120 to generate compressed images.

[0094] Image sensor 120 is a monochrome-type photodetector having multiple photodetector elements arranged in a matrix. For example, image sensor 120 can be a CCD (Charge-Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, an infrared array image sensor, a terahertz array image sensor, or a millimeter-wave array image sensor. Furthermore, image sensor 120 may not be a monochrome-type photodetector; it may also be a color-type photodetector. The wavelength range detectable by image sensor 120 is not particularly limited; for example, it may be visible light, ultraviolet light, infrared light, terahertz waves, or any combination thereof.

[0095] also, Figure 1 Although not shown in the diagram, the imaging device 100 also includes a filter array 130 and an optical system 140. The configuration of the imaging device 100, including the filter array 130 and the optical system 140, will be discussed later. Figure 2 Please provide an explanation.

[0096] The speed sensor 400 is a sensor that detects the speed of movement of the imaging device 100, i.e., the speed of its activity. The speed sensor 400 can be implemented by one or more sensors selected from those capable of detecting speed, such as image sensors, gyroscopes, accelerometers, and positioning sensors. The speed sensor 400 is built into the imaging device 100 and detects the activity speed of the imaging device 100 with the built-in speed sensor 400. Alternatively, the speed sensor 400 can be additionally mounted to the imaging device 100 to detect the activity speed of the imaging device 100 with the speed sensor 400 mounted thereon. Furthermore, the speed sensor 400 can also be implemented as an image sensor configured to capture images of the imaging device 100, detecting the activity speed of the imaging device 100 captured by the speed sensor 400. However, the speed sensor 400 is not an essential component. An example of implementing the imaging system 1000 without the speed sensor 400 will be described later.

[0097] The information processing device 200 is communicatively connected to the imaging device 100, the speed sensor 400, and the display device 300 via wired and / or wireless means. The information processing device 200 includes an image acquisition unit 210, an information acquisition unit 220, a parameter determination unit 230, a restoration calculation unit 240, a storage unit 250, and a determination unit 260.

[0098] The image acquisition unit 210 can acquire compressed images from the imaging device 100 and store them in the storage unit 250. Alternatively, the image acquisition unit 210 may acquire compressed images not directly from the imaging device 100, but via other devices.

[0099] The information acquisition unit 220 can obtain image quality adjustment information from the display device 300. Alternatively, the information acquisition unit 220 may obtain image quality adjustment information via other devices, without directly obtaining it from the display device 300. Furthermore, the information acquisition unit 220 can obtain the movement speed of the shooting device 100 from the speed sensor 400. Again, the information acquisition unit 220 may obtain the movement speed via other devices, without directly obtaining it from the shooting device 100.

[0100] The parameter determination unit 230 is capable of determining the values ​​of the operational parameters used in the restoration operation based on the image quality adjustment information. In other words, the parameter determination unit 230 is capable of determining one or more values ​​corresponding one-to-one with one or more operational parameters used in the restoration operation based on the image quality adjustment information. The image quality adjustment information refers to information used to adjust the image quality of four or more spectroscopic images obtained through the restoration operation. For example, the image quality adjustment information may include the priority of the image quality of the four or more spectroscopic images (i.e., a first priority value), the priority of the speed at which the four or more spectroscopic images are generated from the compressed image (i.e., a second priority value), or a combination of the first priority value and the second priority value. The operational parameters may be a first parameter (the weighting coefficient τ described later) corresponding to the influence of regularization in the restoration operation and / or a second parameter corresponding to the number of iterations in the iterative operation included in the restoration operation.

[0101] The restoration calculation unit 240 can perform restoration operations on the compressed image using the values ​​of the calculation parameters determined by the parameter determination unit 230 to generate four or more spectroscopic images. In other words, the restoration calculation unit 240 performs the following process: by utilizing the calculations on the compressed image, it executes a reconstruction process to generate a hyperspectral image composed of four or more spectroscopic images. The restoration calculation unit 240 outputs the generated four or more spectroscopic images to the display device 300.

[0102] Furthermore, the restoration operation performed in this embodiment can be the same as the restoration operation described in Patent Document 1 or 2. Specifically, more than four spectroscopic images can be restored based on the following formula (1).

[0103]

[0104] Here, g represents the data of the compressed image, for example, represented by a one-dimensional array (i.e., a vector). If the compressed image is an n×m pixel image, then the data g is represented by a one-dimensional array with n×m elements. f represents the data of w spectroscopic images corresponding one-to-one with w bands, for example, represented by a one-dimensional array. f1 is the data of the spectroscopic image corresponding to band W1, f2 is the data of the spectroscopic image corresponding to band W2, ..., f w To be compatible with band W w The corresponding spectroscopic image data. f1, f2, ..., f w Each can be represented by a one-dimensional array, for example. If each spectroscopic image is an n×m pixel image, then f1, f2, ..., f wEach element is represented by a one-dimensional array with n×m elements, and the data f is represented by a one-dimensional array with n×m×w elements. H is an n×m row and n×m×w column matrix, sometimes called the system matrix. H can be based on the transmission spectrum of band W1 of filter array 130, the transmission spectrum of band W2 of filter array 130, ..., the transmission spectrum of band W of filter array 130. w The data f that satisfies Equation (1) can be estimated using compressed sensing, specifically, it can be estimated according to Equation (2).

[0105]

[0106] Equation (2) represents finding the value f that minimizes the sum of the first and second terms within the parentheses. The final solution f can be calculated by using recursive iterative operations to bring the solution to convergence.

[0107] The first term within the parentheses in equation (2) represents the sum of squares of the differences between Hf and data g obtained by systematically transforming the data f in the estimation process using matrix H, i.e., the so-called residual term. Here, the sum of squares is used, but the sum of absolute values ​​or the sum of squares and square roots can also be used instead. The sum of squares of the differences between Hf and data g is (g1-r1)×(g1-r1)+…+(g…r1-r1)×(g ... n×m -r n×m )×(g n×m -r n×m Furthermore, g = (g1…g) n×m ) T Hf = (r1…r) n×m ) T .

[0108] The second term within the parentheses in equation (2) is the regularization term, sometimes also called the stabilization term. Φ(f) represents the constraint condition in the regularization of f, and is a function reflecting the sparse information of the data f. This function brings about the effect of smoothing or stabilizing the data f. Φ(f) can be represented by, for example, Discrete Cosine Transform (DCT), Wavelet Transform, Fourier Transform, Total Variation (TV), or any combination thereof. τ is the weight coefficient of the regularization term, corresponding to the influence of regularization in the restoration operation. The larger the value of τ, the higher the influence of regularization, the more redundant data is deleted, and the stronger the convergence of the solution in the iterative operation. Conversely, the smaller the value of τ, the lower the influence of regularization, the less redundant data is deleted, and the weaker the convergence of the solution in the iterative operation.

[0109] In equation (2), τ can be used as the first parameter. Furthermore, the number of iterations of the recursive iterative operation of equation (2) can be used as the second parameter. The restoration operation unit 240 can perform the restoration operation of equation (2) by using the values ​​of the first and second parameters to generate more than four spectroscopic images.

[0110] The storage unit 250 is capable of storing compressed images and / or four or more spectroscopic images (reconstructed images), etc. The storage unit 250 can be implemented, for example, using a hard disk drive and / or a solid-state drive.

[0111] The determination unit 260 can determine whether the condition that the activity speed of the acquired imaging device 100 is below a predetermined threshold is met. Based on the determination result of whether the condition is met in the determination unit 260, a control signal for switching whether to generate more than four spectroscopic images is generated, and output to the restoration calculation unit 240. Furthermore, the predetermined threshold mentioned above will be explained in detail later.

[0112] The display device 300 serves as a user interface, comprising an input unit 310 and a display unit 320. The display device 300 can be implemented using, for example, a tablet computer, a smartphone, or a desktop computer.

[0113] The input unit 310 is capable of receiving image quality adjustment information input by the user. The input unit 310 can be implemented using, for example, a touch screen, touchpad, mouse, keyboard, or any combination thereof.

[0114] The display unit 320 can display a graphical user interface (GUI) for obtaining image quality adjustment information. The display unit 320 can be implemented, for example, using a liquid crystal display (LCD) and / or an organic light-emitting diode (OLED) display.

[0115] also, Figure 1 This describes an exemplary functional configuration of the shooting system 1000; the functional configuration of the shooting system 1000 is not limited to... Figure 1 For example, part or all of the shooting device 100 may be included in the information processing device 200. Additionally, for example, part or all of the display device 300 may also be included in the information processing device 200. Furthermore, for example, the information processing device 200 may be divided into multiple devices, for example, it may be implemented by a cloud server.

[0116] [Composition of the filming device 100]

[0117] Next, refer to Figure 2 The configuration of the shooting device 100 will be explained. Figure 2This is a schematic diagram of the imaging device 100 according to the embodiment.

[0118] The imaging device 100 has the same configuration as the imaging devices disclosed in Patent Documents 1 and 2. Specifically, the imaging device 100 includes a control circuit 110, an image sensor 120, a filter array 130, and an optical system 140. Furthermore, Figure 2 The diagram of control circuit 110 is omitted.

[0119] Filter array 130 is arranged in the optical path of light incident from object 70, which is the subject, in Figure 2 The filter array 130 is positioned between the optical system 140 and the image sensor 120. The filter array 130 functions as an encoding element as described in Patent Document 1. The filter array 130 can also be integrated with the image sensor 120. Furthermore, the configuration of the filter array 130 is not limited to... Figure 2 The filter array 130 can be configured, for example, between the optical system 140 and the image sensor 120, away from the image sensor 120. Alternatively, for example, the filter array 130 can be configured between the object 70 and the optical system 140. Alternatively, for example, the filter array 130 can be configured within the optical system 140.

[0120] Optical system 140 is positioned in the optical path of light incident from object 70. Figure 2 The optical system 140 is disposed between the object 70 and the filter array 130. The optical system 140 includes at least one lens capable of forming an image of the object 70 on the imaging surface of the image sensor 120 via the filter array 130. Furthermore, the configuration and arrangement of the optical system 140 are not limited to... Figure 2 The optical system 140 can be configured in various ways. For example, it can be positioned between the filter array 130 and the image sensor 120. Alternatively, the optical system 140 can include multiple lenses arranged in the optical path. In this case, the filter array 130 can also be positioned between adjacent lenses among the multiple lenses.

[0121] [Composition of filter array 130]

[0122] The filter array 130 includes multiple filters. The number of filters can be n×m. The n×m filters include... 11 ,……,filter nm .filter 11 In band W1 to band W w The transmission spectrum S in 11 ,……,filter nm In band W1 to band W w The transmission spectrum S in nmThey can all be different, or, the transmission spectrum S 11 ..., transmission spectrum S nm The elements contained therein can also be the same.

[0123] filter 11 The transmittance S of band W1 11W1 ,……,filter 11 band W w Transmittance S 11Ww ,……,filter nm The transmittance S of band W1 nmW1 ,……,filter nm band W w Transmittance S nmWw They can all be different, or, transmittance S 11W1 ..., transmittance S 11Ww ..., transmittance S nmW1 ..., transmittance S nmWw The components included can also be the same. w can be an integer greater than 4. Furthermore, in this disclosure, transmittance can mean light transmittance.

[0124] Band W α The transmittance of filter β in the filter can be expressed by the following formula (3).

[0125]

[0126] Wαmin is the band W α The minimum wavelength value, Wαmax is the band W α The maximum wavelength value, h(λ) is a function representing the transmission spectrum, where λ is the wavelength.

[0127] In addition, band W α The transmittance of filter β in the band is not limited to equation (3). For example, band W α The transmittance of filter β in the equation can also be obtained by dividing equation (3) by (Wαmax - Wαmin). Additionally, for example, in band W... α The transmittance of filter β in the W band can also represent the transmittance of the filter β in the W band. α frequency λ α0 Transmittance h(λ) at the location α0 ). λ α0 Is it satisfying Wαmin≤λ α0 Any frequency ≤ Wαmax is acceptable; for example, it could also be the band W. α The center frequency ((Wαmax-Wαmin) / 2).

[0128] Reference Figures 3A-3E The configuration of filter array 130 will be explained. Figure 3A This is a schematic diagram of the filter array 130 according to the embodiment. The filter array 130 includes filter 130a and filter 130b. Figure 3B This is an example of the transmission spectrum of filter 130a. Figure 3C This is an example of the transmission spectrum of filter 130b.

[0129] Figure 3D This is a diagram illustrating an example of the transmittance of band W1 of the filter array 130 involved in the embodiment. Figure 3E This is a graph illustrating an example of the transmittance of band W2 of the filter array 130 according to the embodiment. Figure 3D and Figure 3E In the diagram, the density of each region represents the transmittance of the filter; the lighter the region, the higher the transmittance, and the darker the region, the lower the transmittance.

[0130] The filter array 130 contains multiple filters arranged in a matrix. Figure 3A In the example shown, filter array 130 contains 48 filters arranged in 6 rows and 8 columns. Filter 130a is the filter located in the upper left corner of the 48 filters, and filter 130b is the filter located in the lower right corner of the 48 filters. Furthermore, the number of filters contained in filter array 130 is not limited to 48. For example, the number of filters contained in filter array 130 can be comparable to the number of pixels in image sensor 120, for example, it can be determined in the range of tens to tens of millions, depending on the application.

[0131] The wavelength dependence of the transmittance of the multiple filters included in filter array 130 is different for each other. For example, in filter 130a, the transmittance of band W1 is much lower than that of band W2. On the other hand, in filter 130b, the transmittance of band W1 is approximately the same as that of band W2. That is, the wavelength dependence of the transmittance of filter 130a is different from that of filter 130b. Furthermore, only the transmittance of two of the four or more bands, W1 and W2, is illustrated and explained here; the transmittance of other bands included in the four or more bands is omitted from the illustration and explanation.

[0132] [Information Processing Methods]

[0133] Next, refer to Figures 4-7 The information processing method in the shooting system 1000 configured as described above will be explained. Figure 4 This is a flowchart of the information processing method involved in Implementation Method 1.

[0134] First, the image acquisition unit 210 of the information processing device 200 acquires a compressed image from the imaging device 100 (S11). Next, the information acquisition unit 220 of the information processing device 200 acquires the movement speed of the imaging device 100 from the speed sensor 400 (S12). The display unit 320 of the display device 300 displays a GUI for acquiring image quality adjustment information. The input unit 310 of the display device 300 receives the input image quality adjustment information via the GUI, and the information acquisition unit 220 of the information processing device 200 acquires the image quality adjustment information from the display device 300.

[0135] For example, the display unit 320 displays a GUI, and the input unit 310 receives input of image quality adjustment information. As image quality adjustment information, values ​​can be set for the priority (first priority) of image quality and the priority (second priority) of restoration operation speed for four or more spectroscopic images (i.e., hyperspectral images). Furthermore, in this embodiment, a larger first priority value indicates a higher first priority, but a smaller first priority value can also indicate a higher first priority. Similarly, a larger second priority value indicates a higher second priority, but a smaller second priority value can also indicate a higher second priority. Additionally, the second priority value is subordinate to the first priority value, and vice versa. Therefore, it is possible to obtain neither the first nor the second priority value, or only one of the first and second priority values.

[0136] On the other hand, the determination unit 260 determines whether the condition that the obtained activity speed is below a predetermined threshold is met (S13). Furthermore, if the condition is met (S13: Yes), the determination unit 260 generates a control signal to switch to generating four or more spectroscopic images and outputs it to the restoration calculation unit 240. As a result, restoration processing is performed in the restoration calculation unit 240 (S14). On the other hand, if the condition is not met (S13: No), the determination unit 260 generates a control signal to switch to not generating four or more spectroscopic images and outputs it to the restoration calculation unit 240. As a result, step S14 is skipped. Based on the control signal, the restoration calculation unit 240 outputs a hyperspectral image composed of four or more spectroscopic images generated based on image quality adjustment information. The output hyperspectral image is displayed on the display unit 320 of the display device 300.

[0137] Here, the predetermined threshold will be explained. The predetermined threshold is a boundary value for determining whether the provision of hyperspectral images can keep up with the activity speed of the imaging device 100. Therefore, the predetermined threshold may vary depending on factors such as the performance of the display unit 320 of the display device 300, which is related to user visual recognition. As an example, assume that the display unit 320 of the display device 300 is a display device with a configuration and size that is visible to the user at a 60-degree viewing angle. Furthermore, the number of pixels in the display unit 320 is calculated as 1920 pixels in the horizontal direction. From the user's perspective, the speed at which the activity of the subject displayed on the display unit 320 is recognized is considered to be approximately 1 to 10 minutes per second. Even calculated at 10 minutes per second, this is 3 pixels / 30 frames, so it can be said that the activity of the subject can be recognized as long as there is 0.1 pixels / frame of activity. Therefore, for example, under such conditions, the activity speed of the imaging device 100, such as 1 pixel / frame of activity being displayed on the display unit 320, is set as the predetermined threshold. Such a predetermined threshold is just one example. It can be set appropriately according to the performance of the display unit 320 of the display device 300, the configuration and size of the display unit 320 that determines the viewing angle, and other conditions.

[0138] Here, Figure 5 This is a flowchart of an information processing method according to a variation of Implementation 1. In this example, with Figure 4 Compared to the flowchart of the information processing method shown, the method for obtaining the activity speed of the imaging device 100 is different. Furthermore, in this example, compared to... Figure 4 Compared to the flowchart of the information processing method shown, this describes a more detailed process. For example, such detailed processing can also be performed in the above embodiment.

[0139] like Figure 5 As shown, in the information processing method according to Modification 1, firstly, the image acquisition unit 210 of the information processing device 200 acquires a compressed image from the imaging device 100 (S21). Next, the image acquisition unit 210 of the information processing device 200 determines whether an image of the previous frame exists (S22). If an image of the previous frame does not exist (S22: No), the process returns to step S21. On the other hand, if an image of the previous frame exists (S22: Yes), a motion vector is generated together with the newly acquired image (S23). The motion vector is information that represents the speed of movement of the object (i.e., the subject) in the two frames as the magnitude of the vector. Based on the motion vector, the relative speed of the imaging device 100 with respect to the object can be calculated. In other words, the activity speed here is derived based on the matching processing of the subject contained in the first compressed image within a first period corresponding to a certain frame and the subject contained in the second compressed image within a second period corresponding to a frame preceding the first period. Therefore, in this example, even without the speed sensor 400, the activity speed of the imaging device 100 can be obtained by deriving it from the image.

[0140] like Figure 5 As shown, based on the activity velocity obtained from the motion vector, it is determined whether the condition that the activity velocity is below a predetermined threshold is met (S24). Furthermore, if the determination unit 260 determines that the condition is met (S24: Yes), it first determines whether restoration processing of a previous frame or other compressed image is currently underway (S25). Restoration processing is a relatively resource-intensive process, making parallel processing difficult. Therefore, step S25 is provided to avoid parallel processing. If it is determined that restoration processing of a past compressed image is not currently underway (S25: No), restoration processing of the newly acquired compressed image is performed (S26). If it is determined that restoration processing of a past compressed image is currently underway (S25: Yes), step S26 is skipped.

[0141] Next, the restoration calculation unit 240 determines whether a hyperspectral image (HS image) has been restored (S27). If it is determined that a hyperspectral image has been restored (S27: Yes), the most recent hyperspectral image is set as the display image (S28). If it is determined that no hyperspectral image has been restored (S27: No), the process proceeds to step S30.

[0142] On the other hand, if the determination unit 260 determines that the condition is not met (S24: No), it deletes the hyperspectral image that has completed the restoration process (S29) and sets the most recent compressed image as the display image (S30). Furthermore, the restoration calculation unit 240 outputs the hyperspectral image or compressed image set as the display image. The output hyperspectral image or compressed image is displayed on the display unit 320 of the display device 300.

[0143] Figure 6 This is a flowchart of the information processing method involved in Variation 2 of Implementation 1. Figure 7 This is a diagram showing an example of an image related to a variation 2 of embodiment 1. In this example, with Figure 5The display image differs from the flowchart of the information processing method shown. Specifically, after determining "yes" in step S27, the subject is analyzed based on the hyperspectral image, and an image representing the analysis result is generated. Here, based on the hyperspectral image, a concentration image is generated by converting the concentration of each pixel into a brightness value according to the predetermined component concentration contained in the subject (e.g., the lycopene concentration in the case of a tomato). Then, instead of step S28, the step of setting the generated concentration image as the display image is performed (S33). Furthermore, if "yes" is determined in step S25, it is determined whether a concentration image already exists (S34). If a concentration image exists (S34: Yes), the process proceeds to step S33; if a concentration image does not exist (S34: No), the process proceeds to step S27.

[0144] When displaying a concentration image, for example, such as Figure 7 As shown, it is possible to distinguish between the "leaves" and "fruits" of a tomato, which are the subjects in an image, based on lycopene concentration (in this case, the dotted-lined areas represent high concentrations). It is also possible to distinguish between parts of the "fruit" with high and low lycopene concentrations. Alternatively, it is possible to distinguish between fruits with high and low lycopene concentrations. Furthermore, the tomato is just one example of the subject; any object can be used as the subject. Based on hyperspectral image analysis, specific components contained in the object are analyzed, and a two-dimensional concentration image is generated by converting the concentration of the specific components into brightness values ​​as the analysis result. The generation of the concentration image can be performed by deriving information from one or more spectroscopic images constituting the hyperspectral image using the brightness value information of each pixel. In addition, algorithms, image filters, and trained models can be used to generate the concentration image using a hyperspectral image as input. Such information, including algorithms, image filters, and trained models, is stored in storage unit 250 and can be read out as needed.

[0145] Next, implementation method 2 will be described. Figure 8 This is a functional configuration diagram of the shooting system involved in Implementation Method 2. Figure 8 The imaging system 1000a shown differs from Embodiment 1 in that: the information processing device 200 only has a determination unit 260; the display device 300 only has a display unit 320; and the image is directly output from the imaging device 100 to the display device 300. In this embodiment, the imaging device 100, targeting a two-dimensional subject in the image space, uses an image sensor composed of multiple pixel groups arranged in the two-dimensional direction, each consisting of a unit pixel group that transmits light of various wavelengths, to capture the entire image space at all wavelengths in a snapshot manner. In other words, the imaging device 100 can acquire multiple images corresponding to four or more wavelengths in a single capture.

[0146] Each unit pixel contains filters that, for example, possess optical properties that transmit light at any wavelength between 460 and 650 nm. For instance, in a 16-band filter system, the first filter in each unit pixel has a transmittance peak at a wavelength of 465 nm. The second filter has a transmittance peak at a wavelength of 474 nm. The third filter has a transmittance peak at a wavelength of 485 nm. The fourth filter has a transmittance peak at a wavelength of 496 nm. The fifth filter has a transmittance peak at a wavelength of 546 nm. The sixth filter has a transmittance peak at a wavelength of 534 nm. The seventh filter has a transmittance peak at a wavelength of 522 nm. The eighth filter has a transmittance peak at a wavelength of 510 nm. The ninth filter has a transmittance peak at a wavelength of 586 nm. The tenth filter has a transmittance peak at a wavelength of 578 nm. The eleventh filter has a transmittance peak at a wavelength of 562 nm. Furthermore, the 12th filter has a transmittance peak at a wavelength of 548 nm. The 13th filter has a transmittance peak at a wavelength of 630 nm. The 14th filter has a transmittance peak at a wavelength of 624 nm. The 15th filter has a transmittance peak at a wavelength of 608 nm. The 16th filter has a transmittance peak at a wavelength of 600 nm. Thus, the filters contained in each unit pixel have peak transmittance values ​​corresponding to specific wavelengths. Alternatively, the imaging device 100 in this embodiment, targeting a two-dimensional subject in the image space, receives light of various wavelengths dispersed in one direction (either longitudinal or transverse) in a row of pixels, accumulating charge in a row-sized quantity. Then, it moves to the next pixel row and repeats the operation of receiving light of various wavelengths for that row. Thus, the imaging device 100 captures the entire image space at all wavelengths in a spatially scanning manner.

[0147] Alternatively, in this embodiment, the imaging device 100 receives one of the wavelengths of light that has been split in the image space for a two-dimensional subject, and performs charge accumulation throughout the entire image space. Then, by repeatedly performing the operation of receiving the next of the wavelengths of light that have been split, the imaging device 100 performs imaging of the entire image space at all wavelengths in a scanning manner along the wavelength direction.

[0148] Therefore, although this embodiment does not involve processing such as acquiring compressed images and / or reconstructing images from compressed images, the device size is relatively large compared to the number of pixels, which easily leads to an increase in the number of operation steps for the imaging device 100. Alternatively, generating and acquiring hyperspectral images captured across the entire image space at all wavelengths may itself require a large number of scanning steps. Therefore, similar to the embodiments described above, there is a problem that the provision of hyperspectral images is difficult to keep up with.

[0149] Therefore, in this embodiment, the speed of the imaging device 100 is also detected by a speed sensor, and based on the speed of the imaging device 100, the system switches between capturing a hyperspectral image and outputting it to the display device 300, or capturing one or more alternative images to replace the hyperspectral image and outputting them to the display device 300.

[0150] Specifically, the 1000a camera system, such as Figure 9 It works as shown. Figure 9 This is a flowchart of the information processing method involved in Implementation Method 2. For example... Figure 9 As shown, firstly, the activity speed of the imaging device 100 is obtained from the speed sensor 400 (S41). The determination unit 260 determines whether the condition that the obtained activity speed is below a predetermined threshold is met (S42). Furthermore, if the determination unit 260 determines that the condition is met (S42: Yes), it generates a control signal and outputs it to the imaging device 100, causing the imaging device 100 to take a picture with a first exposure time sufficient to generate a hyperspectral image.

[0151] As a result, a hyperspectral image is captured (generated and acquired) in the imaging device 100 with a first exposure time (S43). On the other hand, if the determination unit 260 determines that the condition is not met (S42: No), it generates a control signal and outputs it to the imaging device 100, causing the imaging device 100 to capture the image with a second exposure time that is shorter than the first exposure time and insufficient to generate a hyperspectral image. As a result, a substitute image to replace the hyperspectral image is captured (generated and acquired) in the imaging device 100 with a second exposure time (S44). The imaging device 100 directly outputs the acquired image to the display device 300 for display on the display unit 320 (S45).

[0152] Alternate images are, for example, images containing fewer bands of information than the number of bands contained in a hyperspectral image. Specific examples of alternative images include binary black-and-white images or RGB images. Alternate images generated by shorter exposure times reduce the exposure time (reducing the difference between the first and second exposure times) by combining the charge of multiple wavelengths of light into a single brightness value, or by skipping the scanning of more than one wavelength of information contained in the hyperspectral image, or the scanning of more than one pixel (row) in the image space, thereby reducing the steps until the image is provided, and thus the image provision follows the activity speed of the imaging device 100.

[0153] Specifically, when combining the charges of multiple wavelengths of light into a single brightness value, any of the following methods are applicable: spatial scanning, wavelength scanning, and snapshot scanning. In other words, regardless of the method used, it is possible to combine the charges of multiple wavelengths of light into a single brightness value to capture a replacement image. Furthermore, when skipping scans of more than one wavelength, wavelength scanning is applicable. That is, using wavelength scanning allows for capturing a replacement image by skipping scans of more than one wavelength. Similarly, when skipping scans of more than one pixel (row), spatial scanning is applicable. That is, using spatial scanning allows for capturing a replacement image by skipping scans of more than one pixel (row).

[0154] As an example, when the imaging device 100 acquires a hyperspectral image in snapshot mode, if the shortest exposure time for the first exposure is Ts and the number of bands containing information in the hyperspectral image is B, the exposure time required to combine the bands of the hyperspectral image to form a black and white image is Ts / B seconds for the band to accumulate charge. For example, if Ts = 1 / 2 and B = 16, a black and white binary image can be obtained at 1 / 32 of a second (approximately 30 fps).

[0155] When the imaging device 100 acquires a hyperspectral image by scanning in the spatial direction, similarly, by combining the charges of each band of the hyperspectral spectrum to obtain a charge of a single wavelength, for example, if Ts=5 and B=150, a black-and-white binary image can be obtained at 1 / 30 of a second (30fps). Furthermore, by halving the number of rows corresponding to the number of scans in the spatial direction, a black-and-white binary image can be obtained at 1 / 60 of a second (60fps).

[0156] Next, a variation of Implementation Method 2 will be described. Figure 10 This is a functional configuration diagram of the shooting system involved in a variation of implementation method 2. Figure 10The imaging system 1000b shown differs from Embodiment 1 in that: the information processing device 200 only includes an image acquisition unit 210, a storage unit 250, and a determination unit 260; the restoration calculation unit 240 is replaced by an output unit 270 that only outputs images; and the display device 300 only includes a display unit 320. Similar to Embodiment 2, the imaging device 100 in this embodiment, for a two-dimensional subject in the image space, uses an image sensor composed of unit pixel groups arranged in a two-dimensional direction that further arrange multiple pixel groups in a two-dimensional direction to capture the entire image space at all wavelengths in a snapshot manner. Alternatively, the imaging device 100 may also capture the entire image space at all wavelengths by scanning in the spatial direction or the wavelength direction.

[0157] Compared to Embodiment 2 described above, in this modified example, a concentration image based on a hyperspectral image is generated and output instead of outputting a hyperspectral image. Therefore, the information processing apparatus 200 includes an image acquisition unit 210 that temporarily acquires an image captured by the imaging device 100, and a storage unit 250 that stores algorithms used when generating the concentration image. Furthermore, it includes an output unit 270 for outputting an image from the information processing apparatus 200 to the display device 300.

[0158] The imaging system 1000b involved in this variation is as follows: Figure 11 It works as shown. Figure 11 This is a flowchart of an information processing method involved in a variation of Implementation 2. Figure 11 The flowchart shown is Figure 9 The difference between the flowchart shown and the one described is the inclusion of step S46. Specifically, in the imaging system 1000b of this variant, after acquiring a hyperspectral image at a first exposure time and sending it to the information processing device 200, a concentration image is generated from the hyperspectral image in the information processing device 200 (S46). Then, the process proceeds to step S45. Furthermore, in this variant, since the generated hyperspectral image or alternative image is first acquired by the information processing device 200, a motion vector can be generated and the movement speed of the imaging device 100 can be obtained even without a velocity sensor 400.

[0159] Next, implementation method 3 will be described. Figure 12 This is a functional configuration diagram of the shooting system involved in Implementation Method 3. Figure 12The difference between the imaging system 1000c shown and Embodiment 2 is that an imaging drone 100c is used instead of the imaging device 100. Here, as in Embodiment 2, a camera that uses a snapshot method or a scanning method in the spatial or wavelength direction is envisioned, but it can also be configured in the same way as Embodiment 1 to acquire a compressed image, perform a reconstruction process in the information processing device 200, and generate a hyperspectral image.

[0160] Furthermore, the connection between the camera drone 100c and the information processing device 200, as well as the connection between the camera drone 100c and the display device 300, are both achieved through wireless communication. That is to say, transceivers (not shown) are respectively installed between the camera drone 100c and the information processing device 200, and between the camera drone 100c and the display device 300.

[0161] The camera drone 100c in this embodiment also includes a control circuit 110c instead of the control circuit 110, and also has a built-in speed sensor 400. The control circuit 110c also has the function of controlling the autonomous movement of the camera drone 100c. In addition to the handheld shooting device 100 described above, the same problem arises in autonomously moving cameras such as the camera drone 100c: the provision of hyperspectral images cannot keep up with the movement speed of the camera. Therefore, the determination unit 260 can switch between acquiring a hyperspectral image and acquiring a substitute image based on the movement speed of the camera drone 100c, similar to Embodiment 2. In addition, similar to Embodiment 1, a compressed image can be acquired from the camera drone 100c, and it is possible to switch between performing a reconstruction process to generate a hyperspectral image and directly using the compressed image. Alternatively, the determination unit 260 can be mounted on the camera drone 100c, and the shooting system 1000c can be implemented by the camera drone 100c and the display device 300 alone.

[0162] Next, implementation method 4 will be described. Figure 13 This is a flowchart of the information processing method involved in Embodiment 4. In Embodiment 4, with Figure 4 Compared to the flowchart shown, the difference is that if the result in step S13 is "no", then steps S51 and S52 are executed.

[0163] The computation time for restoring a hyperspectral image can be reduced by decreasing the number of spectral bands to be restored. Therefore, in situations with high activity speeds, where it is determined that there is no time to perform full-spectral band restoration before the image is displayed, the number of spectral bands to be restored can be reduced to shorten the restoration time, and the restored spectral band image can be displayed within that shortened time.

[0164] In step S13, if the condition is not met (S13: No), as described above, it can be determined that there is not enough time to perform the full-spectrum band restoration process. That is, if the full-spectrum band restoration result is waited for before the image is displayed, a large deviation will occur between the previous state of the imaging device 100 and the displayed image. In this case, in this embodiment, a specific spectral band to be restored is selected (S51).

[0165] The selection of the spectral band is preset, such as by the user inputting a specified value at any time, such as at the start of shooting, or by the manufacturer specifying it. That is, in step S51, the setting value is read out, and the spectral band corresponding to the setting value is selected.

[0166] As mentioned above, the number of selected spectral bands, in other words, the number of spectral bands to be restored, needs to be less than the total number of spectral bands. The number of spectral bands to be selected is preset, such as by user input or specified by the manufacturer. Furthermore, the number of selected spectral bands can vary according to the speed of the imaging device 100; the faster the speed, the fewer bands are selected, and the slower the speed, the more bands are selected. In this case, a priority can be preset for each spectral band, and the number of bands to be restored can be selected starting with the highest priority bands according to the speed. Alternatively, a set of spectral bands to be restored corresponding to the speed can be predetermined. The priority and set of spectral bands used here can also be arbitrarily set by user input. For example, the set of spectral bands might correspond to red (R), green (G), and blue (B) bands respectively.

[0167] Then, for the spectral band selected as described above, an image of that spectral band is generated through a reconstruction process (S52).

[0168] Next, implementation method 5 will be described. Figure 14 This is a flowchart of the information processing method involved in Embodiment 5. In Embodiment 5, with Figure 4 The difference between the flowchart shown is that if the result in step S13 is "no", then step S61 is executed.

[0169] Without performing computationally intensive processing such as iterative calculations or AI-based restoration on compressed images, it is possible to generate simplified images (hereinafter referred to as "simplified images") that correspond to multiple spectral bands, although lacking in precision. Therefore, in situations with high activity speeds, where it is determined that there is no time to perform full-spectral band restoration before image display, simplified images with lower precision are generated and displayed.

[0170] In step S13, if the condition is not met (S13: No), as described above, it can be determined that there is not enough time to perform the full-spectrum band restoration process. That is, if the full-spectrum band restoration result is waited for before the image is displayed, a large deviation will occur between the previous state of the imaging device 100 and the displayed image. In this case, in this embodiment, a simplified image corresponding to multiple spectral bands is generated, although the accuracy is insufficient (S61). As described above, in the generation of simplified images, as long as the principle of less processing volume and faster processing speed is prioritized over higher accuracy, any existing technology can be used.

[0171] The following are several examples of simple image generation and processing, explained in detail. Figures 15-18 This is a flowchart related to the simplified image generation process in the information processing method involved in Implementation Method 5. Figure 15 This is a flowchart that schematically illustrates an example of a simple image generation process.

[0172] In the generation of a simplified image, firstly, a restoration table stored in a storage device such as storage unit 250 is obtained (S101). The restoration table is table information represented as a matrix: a three-dimensional matrix in which the depth represents the bands and the horizontal and vertical dimensions represent the pixel values ​​of each mask matrix; or a two-dimensional matrix in which the horizontal dimension represents the bands and the vertical dimension represents the pixel values ​​of multiple pixels contained in each mask matrix.

[0173] In each mask matrix, pixel values ​​can be normalized, for example, by the maximum grayscale value corresponding to the number of bits, representing a range of 0 to 1. Alternatively, pixel values ​​can also be represented by a grayscale range corresponding to the number of bits. In the case of 8 bits, pixel values ​​are 0 to 255, with a maximum grayscale value of 255.

[0174] Each mask matrix is, for example, an n-row, m-column two-dimensional matrix representing the two-dimensional distribution of multiple numerical values. Using this data format, it is easy to understand how the pixel values ​​are distributed in two dimensions for a given band.

[0175] Alternatively, each mask matrix can also be an n×m row, 1 column one-dimensional matrix, i.e., a vector, formed by arranging multiple values ​​in one dimension. This data format allows multiple mask matrices corresponding to multiple bands to be represented as two-dimensional matrices.

[0176] The form of the restoration table is not limited to the examples above. For instance, the restoration table can also be represented as an n×N row m column or n row m×N column two-dimensional matrix formed by arranging N two-dimensional matrices vertically or horizontally. Alternatively, the restoration table can also be represented as an n×m×N row 1 column one-dimensional matrix formed by arranging N one-dimensional matrices vertically.

[0177] By using the operations of the restoration table, multiple simplified images based on the compressed image are generated (S102). The multiple simplified images include simplified images respectively corresponding to bands W1, W2, …, W M The corresponding simplified images. The multiple simplified images generated in step S102 are also referred to as "multiple simplified images of the restoration table type".

[0178] The multiple simplified images are generated by performing operations on the compressed image based on multiple mask matrices included in the restoration table. Each simplified image is generated using the compressed image and a corresponding one of the mask matrices of the same band. More specifically, each simplified image is generated by multiplying each pixel included in the compressed image by a corresponding matrix element among the multiple matrix elements included in the one mask matrix.

[0179] Let the matrix element at the i-th row and j-th column included in the simplified image W k be S k (i, j), the matrix element at the i-th row and j-th column included in the compressed image be C(i, j), and the matrix element at the i-th row and j-th column included in the mask matrix corresponding to the band W k be M(i, j, k). In this case, S k (i, j) = C(i, j) × M(i, j, k).

[0180] It is also possible that, unlike this embodiment, in the case where it is determined that the condition that the moving speed is below a predetermined threshold is not satisfied, a hyperspectral image is not generated through the reconstruction process, but instead, for example, an image having a smaller number of spectral bands than the number of spectral bands included in the predetermined hyperspectral image to be generated is generated for display even in such a situation. For example, when the number of spectral bands included in the compressed image is a relatively large number such as 20 or more, and the number of spectral bands included in the hyperspectral image is correspondingly a relatively large number such as N close to 20, in the case where the above condition is not satisfied, it is also possible to perform the reconstruction process only on M spectral bands less than N (i.e., M < N), and generate and display an image. Here, M is, for example, 3 (M = 3) corresponding to red, green, and blue, etc.

[0181] Figure 16 is a flowchart schematically showing an example of another processing method for generating a simplified image.

[0182] In the generation of the simplified image here, firstly, the three-color weight matrix stored in a storage device such as storage unit 250 is obtained (S201). Details regarding the method for generating these weight matrices will be explained later. The weight matrix contains the following information. In this example, a hyperspectral image is generated using a learning model independent of the imaging device 100. As mentioned earlier, the hyperspectral image contains multiple reconstructed images corresponding to multiple bands respectively. More specifically, the multiple reconstructed images corresponding to multiple bands are four or more reconstructed images corresponding to four or more bands respectively.

[0183] The weight matrix is ​​generated from two or more weight matrices based on the restoration table, and corresponds to two or more specific bands. Since it is generated from the restoration table, it can be said that the two or more weight matrices contain encoded information reflecting the optical transmission spectrum of the coded mask. These two or more weight matrices are used to generate multiple simplified images. Each weight matrix contains the pixel values ​​of multiple pixels as multiple matrix elements. Each matrix element corresponds to one pixel among the multiple pixels contained in the compressed image.

[0184] The specific two or more bands are two or more bands contained within the multiple bands in the restoration table. Therefore, the number of specific two or more bands is less than the number of multiple bands in the restoration table. Alternatively, the number of multiple bands can be read as the number of four or more bands. Furthermore, parameters for generating a simplified image can also be used here.

[0185] The following uses three specific bands commonly used in color images, namely the red, green, and blue bands, as two or more specific bands. The center wavelength of the red band is in the range of 620 nm to 750 nm, the center wavelength of the green band is in the range of 495 nm to 570 nm, and the center wavelength of the blue band is in the range of 450 nm to 495 nm. The width of each band within the target wavelength domain W can be, for example, less than 10 nm. For example, the red band is 681 nm to 690 nm, the green band is 538 nm to 547 nm, and the blue band is 468 nm to 467 nm. All bands within the target wavelength domain W have the same width.

[0186] The weight matrix corresponding to the red band is also called the "red weight matrix", the weight matrix corresponding to the green band is also called the "green weight matrix", and the weight matrix corresponding to the blue band is also called the "blue weight matrix". The weight matrices of red, green and blue are also called the "three-color weight matrix".

[0187] The three-color weight matrix is ​​represented as a three-dimensional matrix where depth represents the bands, and the horizontal and vertical dimensions represent the pixel values ​​of the multiple pixels contained in each weight matrix. Alternatively, the three weight matrices can also be represented as two-dimensional matrices where the horizontal dimension represents the bands and the vertical dimension represents the pixel values ​​of the multiple pixels contained in each weight matrix. After obtaining the weight matrix (S201), multiple simplified images based on the compressed image are generated by operating on the three-color weight matrix (S202). The multiple simplified images generated in step S202 are also referred to as "multiple simplified images of the weight matrix type".

[0188] Figure 17 This is a flowchart detailing the actions in step S202. In step S202, the actions of steps S202a to S202c, as shown below, are performed.

[0189] <Step S202a>

[0190] By weighting the compressed image according to a three-color weight matrix, three pseudo-images corresponding to the three color bands are generated. In this specification, these pseudo-images are simply referred to as "pseudo-images." Each pseudo-image is generated using a weight matrix corresponding to the compressed image and the bands of the same color. More specifically, each pseudo-image is generated by multiplying each pixel in the compressed image by the corresponding element of the weight matrix.

[0191] <Step S202b>

[0192] Next, the resolution of each pseudo-image is reduced. More specifically, the pixel values ​​of multiple pixels contained in each pseudo-image are averaged in an n×n image unit. n is a parameter used for resolution reduction, which is also used when generating the three-color weight matrix based on the restoration table. n can be, for example, greater than 2 and less than 10.

[0193] <Step S202c>

[0194] Next, the dynamic range of pixel values ​​of multiple pixels contained in each pseudo-image after resolution reduction is aligned (made consistent), and the pseudo-image is output as a simplified image. The dynamic range can be, for example, a range of pixel values ​​from 0 to 1 after normalization to the maximum grayscale value corresponding to the number of bits. Alternatively, the dynamic range can also be, for example, a range of grayscale values ​​corresponding to the number of bits. For example, if the averaged pixel value falls outside the range of image brightness values, such as 0 to 255, the dynamic range can be aligned by applying the minimum or maximum brightness value to unify it within a certain range. The minimum value is applied when the averaged pixel brightness value is lower than the minimum value, and the upper limit of the brightness value is applied when the averaged pixel brightness value exceeds the upper limit of the brightness value.

[0195] in addition, Figure 18 This is a flowchart that schematically illustrates an example of a method for generating a weight matrix that is performed before executing a simplified image generation process. In generating the weight matrix, the actions shown in steps S301 to S303 are performed.

[0196] <Step S301>

[0197] Retrieve the recovery table from the storage device.

[0198] <Step S302>

[0199] Next, the three-color weight matrices are determined as follows. Each weight matrix is ​​determined in the following way: for the pixel values ​​of multiple pixels contained in the mask matrix corresponding to the same color band in the restoration table, high pixel values ​​are assigned high weights, and low pixel values ​​are assigned low weights.

[0200] The table showing band W k The corresponding matrix contains matrix elements divided into n×n pixel units. The same applies to the matrix elements in the three-color weight matrix. The definition of n is as follows: Figure 17 As described in step S202b. In steps S202b and S302, the same n is used. If n=4, then one unit contains 16 pixels.

[0201] For the band W in the restoration table k For the corresponding matrix, set the matrix elements in each unit of n×n pixels to M (u, v, k). For the red weight matrix, set the matrix elements in each unit of n×n pixels to S. R (u, v), for the green weight matrix, set the matrix elements in each unit of n×n pixels as S. G (u, v), for the blue weight matrix, set the matrix elements in each unit of n×n pixels as S. B (u, v).

[0202] S R (u, v) can be determined, for example, by minimizing the function of equation (4) while satisfying equation (5). In equations (4) and (5), k R Indicates the red band.

[0203]

[0204]

[0205] S G (u, v) can be determined, for example, by minimizing the function of the following equation (6) while satisfying the following equation (7). In equations (6) and (7), k GIndicates the green band.

[0206]

[0207]

[0208] S B (u, v) can be determined, for example, by minimizing the function of equation (8) while satisfying equation (9). In equations (8) and (9), k B Indicates the blue band.

[0209]

[0210]

[0211] S generated as described above R (u, v), S G (u, v) and S B (u, v) assume that the pixels within each unit of an n×n pixel array have the same spectrum. Therefore, when generating a simplified image, step S202b is performed to reduce the resolution of each pseudo-image.

[0212] <Step S303>

[0213] Next, the balance of the matrix elements in each weight matrix is ​​adjusted as follows. Using a typical color image as a sample image, the balance of the matrix elements in each weight matrix can be adjusted to satisfy the following condition: the image obtained by weighting the sample image using each weight matrix is ​​close to an image of the same color contained in the sample image. The image obtained by weighting the sample image using each weight matrix is ​​generated by multiplying each pixel in the sample image by the corresponding matrix element in the weight matrix.

[0214] Alternatively, using a compressed image as a sample image, the balance of multiple matrix elements within each weight matrix can be adjusted to satisfy the following condition: In the image obtained by weighting the sample image using each weight matrix, the average pixel value—that is, the average pixel value—is the same regardless of the red, green, and blue colors. To satisfy this condition, at least one of the three-color weight matrices needs to be multiplied by a correction constant.

[0215] The correction constant can be set, for example, in the following manner: making the average pixel value of the image obtained by weighting the sample image with the weight matrix of a certain color consistent with the average pixel value of the image obtained by weighting the sample image with the weight matrices of other colors. Alternatively, the correction constant can be set in the following manner: making the average pixel value of the image obtained by weighting the sample image with each weight matrix consistent with a specific value, such as half of the maximum pixel value.

[0216] It is also possible to not generate a hyperspectral image through the reconstruction process when it is determined that the condition that the moving speed is below the predetermined threshold is not satisfied, as in this embodiment, but to generate a simple image that can be generated even in such a situation without going through the reconstruction process for display. For example, when the number of spectral bands included in the compressed image is a relatively large number such as 20 or more, and the number of spectral bands included in the hyperspectral image is a relatively large number such as N close to 20 corresponding thereto, the simple image can include N spectral bands that are the same as the number of spectral bands included in the hyperspectral image, or can also include M spectral bands less than N (i.e., M < N). Here, M is, for example, 3 (M = 3) corresponding to red, green, and blue, etc.

[0217] (Other embodiments)

[0218] As described above, the information processing method has been described based on the embodiments and modified examples, but the information processing method related to the present disclosure is not limited to the above embodiments and modified examples. Other embodiments achieved by combining any constituent elements in the above embodiments and modified examples and / or modified examples obtained by making various modifications that those skilled in the art can think of within the scope not departing from the gist of the present disclosure for the above embodiments and modified examples are also included in the present disclosure.

[0219] For example, in the above embodiment, the hyperspectral image is defined as an image including 4 or more spectroscopic images, but it can also be defined as an image including 5 or more, 6 or more, 7 or more, 8 or more, 9 or more, or 10 or more spectroscopic images.

[0220] In addition, in the imaging device 100 related to the above embodiments and modified examples, the filter array 130 is used as the encoding element, but it is not limited thereto. For example, a meta - lens described in Non - Patent Document 1 can be used instead of the filter array 130 related to the above embodiments and modified examples. The meta - lens includes a plurality of regions with different transmission spectra. In addition, for example, an image sensor described in Non - Patent Document 2 can be used instead of the filter array 130 and the image sensor 120 related to the above embodiments and modified examples. The image sensor processes the sensing area in such a way that light having a predetermined spectrum is obtained for each pixel.

[0221] Industrial applicability

[0222] This disclosure is useful for situations involving the use of hyperspectral images.

[0223] Explanation of reference numerals in the attached figures

[0224] 100 Shooting device; 100c Shooting drone; 110, 110c Control circuit; 120 Image sensor; 130 Filter array; 130a, 130b Filter; 140 Optical system; 200 Information processing device; 210 Image acquisition unit; 220 Information acquisition unit; 230 Parameter determination unit; 240 Recovery calculation unit; 250 Storage unit; 260 Judgment unit; 300 Display device; 310 Input unit; 320 Display unit; 1000 Shooting system.

Claims

1. An information processing method, which is an information processing method executed by a computer, comprising: The subject is photographed using a camera, and a first compressed image is obtained during the first period. The first compressed image is a compressed image containing multiple pixels and is used to generate a hyperspectral image through a reconstruction process. The data of each of the multiple pixels contains information of more than four bands. Obtain the camera's motion speed; Determine whether the condition that the activity speed is below a predetermined threshold is met during the first period; If it is determined that the condition is not met during the first period, the generation of the hyperspectral image based on the first compressed image through the reconstruction process will not be performed.

2. The information processing method according to claim 1, In a second period earlier than the first period, a second compressed image is obtained as the compressed image; If it is determined that the condition is met during the first period, the hyperspectral image based on the first compressed image or the hyperspectral image based on the second compressed image is generated through the reconstruction process.

3. The information processing method according to claim 1, If it is determined that the condition is met during the first period, the hyperspectral image is generated through the reconstruction process based on the first compressed image. When the generation of the hyperspectral image based on the first compressed image through the reconstruction process is completed at the end of the first period, an image related to the generated hyperspectral image is output for display on the display device. If the generation of the hyperspectral image based on the first compressed image through the reconstruction process is not completed at the end of the first period, the first compressed image is output and displayed by the display device.

4. The information processing method according to claim 3, The image associated with the hyperspectral image is either the hyperspectral image itself or an image representing the analysis results of the subject obtained based on the analysis of the hyperspectral image.

5. The information processing method according to claim 1, If it is determined that the condition is not met during the first period, the first compressed image is output and displayed on the display device.

6. The information processing method according to claim 2, The activity speed is obtained as the relative activity speed between the camera and the subject derived from the matching process of the subject contained in the first compressed image and the subject contained in the second compressed image.

7. The information processing method according to claim 1, The activity speed is obtained as a detection result detected by a sensor that detects the speed of the camera.

8. An information processing method, which is an information processing method executed by a computer, comprising: Obtain the camera's movement speed. Determine whether the condition that the activity speed is below a predetermined threshold is met; If the conditions are met, the camera is used to take a picture of the subject at the first exposure time to obtain a hyperspectral image containing multiple images that correspond to more than four bands respectively. If the condition is not met, the camera is used to take a picture of the subject with a second exposure time that is shorter than the first exposure time, so as to obtain a substitute image to replace the hyperspectral image.

9. The information processing method according to claim 8, The alternative image contains fewer images than the plurality of images contained in the hyperspectral image.

10. The information processing method according to claim 8, The camera detects light passing through multiple optical filters. Each of the plurality of optical filters has a peak transmittance that corresponds one-to-one with any one of the four or more bands.

11. The information processing method according to claim 8, The camera generates the hyperspectral image by scanning in the wavelength direction or the spatial direction.

12. The information processing method according to claim 1, If it is determined that the condition is not met during the first period, a display image containing information of three or fewer bands is generated and output based on the first compressed image.

13. The information processing method according to claim 12, In a second period earlier than the first period, a second compressed image is obtained as the compressed image; If it is determined that the condition is not met during the first period, the display image based on the first compressed image or the display image based on the second compressed image is generated.

14. The information processing method according to claim 12, The displayed image is not generated through the reconstruction process.

15. An information processing method, executed by a computer, comprising: The subject is photographed using a camera, and a first compressed image is obtained during the first period. The first compressed image is a compressed image containing multiple pixels and is used to generate a hyperspectral image through a reconstruction process. The data of each of the multiple pixels contains information of more than four bands. Obtain the camera's motion speed; Determine whether the condition that the activity speed is below a predetermined threshold is met during the first period; If it is determined that the condition is met during the first period, a first screen is displayed based on multiple images corresponding to each of the N bands. If it is determined that the condition is not met during the first period, a second frame is displayed based on (i) the compressed image or (ii) multiple images corresponding to each of the M bands, wherein M <N。 16. The information processing method according to claim 15, If it is determined that the conditions are not met during the first period, the plurality of images corresponding to each of the M bands are generated through the reconstruction process based on the compressed image, and the second screen is displayed.

17. The information processing method according to claim 15, If it is determined that the condition is not met during the first period, the plurality of images corresponding to each of the M bands are generated without going through the reconstruction process based on the compressed image, and the second screen is displayed.

18. A shooting system, comprising: The image acquisition unit uses a camera to capture images of the subject and acquires a first compressed image during the first period. The first compressed image is a compressed image containing multiple pixels and is used to generate a hyperspectral image through a reconstruction process. The data of each of the multiple pixels contains information of more than four bands. The information acquisition unit acquires the movement speed of the camera; The determination unit determines whether the condition that the activity speed is below a predetermined threshold is met during the first period; as well as, The reconstruction unit, if it determines that the conditions are not met during the first period, does not generate the hyperspectral image based on the first compressed image obtained through the reconstruction process.

19. A shooting system, comprising: The acquisition unit acquires the speed of the camera's movement; The determination unit determines whether the condition that the activity speed is below a predetermined threshold is met; and Image acquisition unit, The image acquisition unit, If the conditions are met, the camera is used to capture an image of the subject at a first exposure time to obtain a hyperspectral image containing multiple pixels, each of which contains information from four or more spectral bands. If the condition is not met, the camera is used to take a picture of the subject with a second exposure time that is shorter than the first exposure time, so as to obtain a substitute image to replace the hyperspectral image.

Images