Information processing method and information processing device

Abstract

This information processing method executed by a computer: acquires a compressed image which includes a plurality of pixels, where data of each of the plurality of pixels includes information of four or more wavelength bands (S110); acquires image quality adjustment information for adjusting the image quality of four or more spectral images which are four or more spectral images generated by executing a reconstruction operation based on the compressed image and which respectively correspond to the four or more wavelength bands (S130); determines a value of an operation parameter used in the reconstruction operation on the basis of the image quality adjustment information (S140); and generates the four or more spectral images by using the determined value of the operation parameter to execute the reconstruction operation on the compressed image (S150).

Description

Information processing method and information processing device

[0001] The present disclosure relates to an information processing method and an information processing device.

[0002] A technology has been proposed that applies compressed sensing to a camera that captures hyperspectral images having spectral information in more wavelength bands than red, green, and blue (four or more wavelength bands). Compressed sensing is a technology that restores more data from acquired data with a small number of samples. For example, in Patent Documents 1 and 2, a compressed image is captured using a filter array including a plurality of filters having different wavelength dependencies, and four or more spectral images (hyperspectral images) that correspond one-to-one to the four or more wavelength bands are restored based on the captured compressed image.

[0003] International Publication No. 2021 / 192891 Patent No. 7262003 JP 2018-26781 A

[0004] “World's first technology to combine metalens and AI with a conventional digital camera to capture hyperspectral images and videos – Combining optical technology and AI to transform a ‘normal camera’ into a ‘camera that can see the properties of objects’”, [online], October 24, 2022, Nippon Telegraph and Telephone Corporation, [Retrieved June 15, 2023], Internet <URL: https: / / group.ntt / jp / newsrelease / 2022 / 10 / 24 / 221024a.html> Ahasan Ahamed, et.al., “Reconstruction-based spectroscopy using CMOS image sensors with random photon-trapping nanostructure per sensor”, Proc. SPIE 11971, High-Speed ​​Biomedical Imaging and Spectroscopy VII, 1197106, 2 March 2022 Ana Serrano, et.al., “Convolutional Sparse Coding for High Dynamic Range Imaging”, Computer Graphics Forum Volume 35, Issue 1 2, pp. 153-163, 27 May 2016, Toshiki Sonoda, et.al., “High-Speed ​​Imaging using CMOS Image Sensor with Quasi Pixel-Wise Exposure”, Proceedings of IEEE International Conference on Computational Photography (ICCP), pp. 1-11, May 2016

[0005] However, with the above-described conventional techniques, it is difficult to set appropriate values ​​for the calculation parameters used in the restoration calculation of compressed sensing, and it is difficult to balance the image quality of the restored image and the calculation speed.

[0006] The present disclosure provides an information processing method and information processing device that can effectively perform reconstruction calculations for compressed sensing by setting appropriate values ​​for calculation parameters.

[0007] An information processing method according to one aspect of the present disclosure is an information processing method executed by a computer, which acquires a compressed image including a plurality of pixels, wherein data for each of the plurality of pixels includes information on four or more wavelength bands; acquires image quality adjustment information for adjusting the image quality of four or more spectral images corresponding to the four or more wavelength bands, which are generated by performing a restoration operation based on the compressed image; determines values ​​of calculation parameters to be used in the restoration operation based on the image quality adjustment information; and generates the four or more spectral images by performing the restoration operation on the compressed image using the determined values ​​of the calculation parameters.

[0008] This comprehensive or specific aspect may be realized by a system, an apparatus, an integrated circuit, a computer program, or a computer-readable recording medium, or by any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. The computer-readable recording medium includes, for example, a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory).

[0009] According to the present disclosure, it is possible to effectively perform reconstruction calculations for compressed sensing by setting appropriate values ​​for calculation parameters.

[0010] FIG. 1 is a functional configuration diagram of an imaging system according to an embodiment. FIG. 2 is a schematic diagram of an imaging device according to an embodiment. FIG. 3A is a schematic diagram of a filter array according to an embodiment. FIG. 3B is a diagram showing an example of a transmission spectrum of a filter according to an embodiment. FIG. 3C is a diagram showing an example of a transmission spectrum of another filter according to an embodiment. FIG. 3D is a diagram showing an example of a transmittance of a first wavelength band of a filter array according to an embodiment. FIG. 3E is a diagram showing an example of a transmittance of a second wavelength band of a filter array according to an embodiment. FIG. 4 is a flowchart of an information processing method according to an embodiment. FIG. 5 is a diagram showing an example of a graphical user interface (GUI) according to an embodiment. FIG. 6 is a diagram showing an example of a conversion table according to an embodiment. FIG. 7 is a diagram showing an example of a GUI according to Modification 1. FIG. 8 is a diagram showing an example of a GUI according to Modification 2. FIG. 9 is a diagram showing an example of a GUI according to Modification 2. FIG. 10 is a diagram showing an example of a GUI according to Modification 3. FIG. 11 is a flowchart of a parameter determination process according to Modification 3. FIG. 12 is a diagram showing an example of a GUI according to Modification 4. FIG. 13 is a flowchart of an information processing method according to Modification 4.

[0011] (Outline of the Present Disclosure) Before describing the embodiments, an outline of the present disclosure will be described.

[0012] An information processing method according to a first aspect of the present disclosure is an information processing method executed by a computer, which acquires a compressed image including a plurality of pixels, wherein data for each of the plurality of pixels includes information on four or more wavelength bands; acquires image quality adjustment information for adjusting the image quality of four or more spectroscopic images corresponding to the four or more wavelength bands, which are generated by performing a restoration operation based on the compressed image; determines values ​​of calculation parameters to be used in the restoration operation based on the image quality adjustment information; and generates the four or more spectroscopic images by performing the restoration operation on the compressed image using the determined values ​​of the calculation parameters.

[0013] According to this, the values ​​of the calculation parameters are determined based on the image quality adjustment information. In other words, since the values ​​of the calculation parameters are determined by providing the image quality adjustment information, there is no need to directly set the values ​​of the calculation parameters, and the difficulty of setting the values ​​of the calculation parameters can be reduced. As a result, it is possible to effectively perform restoration calculations for compressed sensing of hyperspectral images by appropriately setting the values ​​of the calculation parameters.

[0014] An information processing method according to a second aspect of the present disclosure is an information processing method according to the first aspect, further comprising displaying a graphical user interface for acquiring the image quality adjustment information, and the image quality adjustment information being acquired via the graphical user interface.

[0015] This allows the image quality adjustment information to be acquired via a graphical user interface, allowing the user to easily input the image quality adjustment information, further reducing the difficulty of setting the values ​​of the calculation parameters.

[0016] An information processing method according to a third aspect of the present disclosure is the information processing method according to the second aspect, further comprising displaying the compressed image and the four or more generated spectral images.

[0017] This allows four or more spectral images restored based on the image quality adjustment information to be displayed, allowing the user to visually confirm the restoration results and easily determine whether the values ​​of the calculation parameters determined based on the image quality adjustment information are appropriate.

[0018] An information processing method according to a fourth aspect of the present disclosure is the information processing method according to the third aspect, further displaying a required time for the restoration calculation.

[0019] This displays the time required for the restoration calculation, allowing the user to visually check the restoration time and easily determine whether the values ​​of the calculation parameters determined based on the image quality adjustment information are appropriate.

[0020] An information processing method according to a fifth aspect of the present disclosure is an information processing method according to any one of the first to fourth aspects, wherein the image quality adjustment information includes a first priority value that is a priority of the image quality of the four or more spectroscopic images, a second priority value that is a priority of the speed of the restoration calculation, or a combination of the first priority value and the second priority value.

[0021] This allows the image quality adjustment information to include a priority of image quality and / or a priority of the speed of the reconstruction calculation, so that information that is intuitively understandable to the user can be used as the image quality adjustment information, which further reduces the difficulty of setting the values ​​of the calculation parameters and enables the reconstruction calculation of compressed sensing to be performed more effectively.

[0022] An information processing method according to a sixth aspect of the present disclosure is the information processing method according to the fifth aspect, wherein the calculation parameters include a first parameter corresponding to a degree of influence of regularization in the reconstruction calculation.

[0023] This allows the value of the first parameter corresponding to the degree of influence of regularization to be determined based on the image quality adjustment information. In the restoration calculation, the degree of influence of regularization affects the image quality of the restored image and the calculation speed. Therefore, by determining the value of the first parameter corresponding to the degree of influence of regularization based on the image quality adjustment information, it is possible to achieve a balance between the image quality of the restored image and the calculation speed, and it becomes possible to perform the restoration calculation for compressed sensing more effectively.

[0024] An information processing method according to a seventh aspect of the present disclosure is an information processing method according to the sixth aspect, in which the value of the first parameter is determined so that the influence of the regularization decreases as the first priority increases or the second priority decreases.

[0025] According to this, when image quality is prioritized or when the speed of the restoration calculation is not prioritized, the influence of regularization can be reduced. If the influence of regularization is low, image quality improves but the calculation speed decreases. Therefore, when image quality is prioritized or when the speed of the restoration calculation is not prioritized, by reducing the influence of regularization, it is possible to achieve consistency between the image quality adjustment information and the image quality and calculation speed, and it is possible to realize intuitive operation for the user.

[0026] An information processing method according to an eighth aspect of the present disclosure is an information processing method according to any one of the fifth to seventh aspects, wherein the calculation parameters include a second parameter corresponding to the number of iterations of the iterative calculation included in the reconstruction calculation.

[0027] According to this, it is possible to determine a value of the second parameter corresponding to the number of iterations of the iterative calculation based on the image quality adjustment information. In the reconstruction calculation, the number of iterations affects the image quality and the calculation speed of the reconstructed image. Therefore, by determining the value of the second parameter corresponding to the number of iterations based on the image quality adjustment information, it is possible to achieve a balance between the image quality and the calculation speed of the reconstructed image, and it is possible to more effectively perform the reconstruction calculation of the compressed sensing.

[0028] An information processing method according to a ninth aspect of the present disclosure is an information processing method according to the eighth aspect, in which the value of the second parameter is determined so that the number of iterations increases as the first priority becomes higher or the second priority becomes lower.

[0029] According to this, when image quality is prioritized or when the speed of the restoration calculation is not prioritized, the number of iterations of the iterative calculation can be increased. A large number of iterations improves image quality but reduces the calculation speed. Therefore, when image quality is prioritized or when the speed of the restoration calculation is not prioritized, by increasing the number of iterations, it is possible to achieve consistency between the image quality adjustment information and the image quality and calculation speed, and to realize intuitive operation for the user.

[0030] An information processing method according to a tenth aspect of the present disclosure is an information processing method according to any one of the first to ninth aspects, wherein the image quality adjustment information includes at least one of information indicating color reproducibility in the four or more spectral images and information indicating the size of a structure possessed by a subject in the four or more spectral images.

[0031] This allows the image quality adjustment information to include information indicating color reproducibility and / or structure size, making it possible to use information that is intuitively understandable to the user as the image quality adjustment information, thereby further reducing the difficulty of setting the values ​​of the calculation parameters and enabling more effective reconstruction calculations for compressed sensing.

[0032] An information processing method according to an eleventh aspect of the present disclosure is an information processing method according to any one of the first to tenth aspects, wherein the calculation parameters include a first parameter corresponding to the influence of regularization in the restoration calculation, and the value of the first parameter is determined based on the average luminance value of the compressed image and the image quality adjustment information.

[0033] According to this method, the value of the first parameter corresponding to the influence of regularization is determined based on the average luminance value of the compressed image. If the average luminance value of the compressed image is large, the influence of the residual term on the sum of the residual term and the regularization term minimized in the restoration calculation tends to increase. Therefore, by adjusting the influence of regularization based on the average luminance value of the compressed image, it is possible to achieve a balance between the residual term and the regularization term.

[0034] An information processing device according to a twelfth aspect of the present disclosure includes an image acquisition unit that acquires a compressed image including a plurality of pixels, wherein data for each of the plurality of pixels includes information on four or more wavelength bands; an information acquisition unit that acquires image quality adjustment information for adjusting the image quality of four or more spectral images generated by performing a restoration operation based on the compressed image, the four or more spectral images corresponding to the four or more wavelength bands, respectively; a parameter determination unit that determines values ​​of calculation parameters to be used in the restoration operation based on the image quality adjustment information; and a restoration operation unit that generates the four or more spectral images by performing the restoration operation on the compressed image using the determined values ​​of the calculation parameters.

[0035] This makes it possible to achieve the same effect as the above-described information processing method in an information processing device.

[0036] An information processing method according to a thirteenth aspect of the present disclosure is an information processing method executed by a computer, which acquires a compressed image including a plurality of pixels, each having a different exposure time or amount of transmitted light, acquires image quality adjustment information for adjusting the image quality of a high dynamic range image having a higher dynamic range than the compressed image, which is generated by performing a restoration operation based on the compressed image, determines values ​​of calculation parameters to be used in the restoration operation based on the image quality adjustment information, and generates the high dynamic range image by performing the restoration operation on the compressed image using the determined values ​​of the calculation parameters.

[0037] According to this, the values ​​of the calculation parameters are determined based on the image quality adjustment information. In other words, since the values ​​of the calculation parameters are determined by providing the image quality adjustment information, there is no need to directly set the values ​​of the calculation parameters, and the difficulty of setting the values ​​of the calculation parameters can be reduced. As a result, it is possible to appropriately set the values ​​of the calculation parameters and effectively perform restoration calculations for compressed sensing of high dynamic range images.

[0038] An information processing method according to a fourteenth aspect of the present disclosure is an information processing method executed by a computer, which acquires a compressed image including a plurality of pixels, each having a different exposure pattern for coded exposure for the plurality of pixels, acquires image quality adjustment information for adjusting the image quality of a plurality of images (temporal super-resolution images) corresponding to different times that are generated by performing a restoration operation based on the compressed image, determines values ​​of calculation parameters to be used in the restoration operation based on the image quality adjustment information, and generates the plurality of images by performing the restoration operation on the compressed image using the determined values ​​of the calculation parameters.

[0039] According to this, the values ​​of the calculation parameters are determined based on the image quality adjustment information. In other words, since the values ​​of the calculation parameters are determined by providing the image quality adjustment information, there is no need to directly set the values ​​of the calculation parameters, and the difficulty of setting the values ​​of the calculation parameters can be reduced. As a result, it is possible to set appropriate values ​​for the calculation parameters and effectively perform restoration calculations for compressed sensing of temporal super-resolution images.

[0040] An information processing method according to a fifteenth aspect of the present disclosure is an information processing method executed by a computer, which acquires a compressed image including a plurality of pixels, wherein data for each of the plurality of pixels includes image information from a plurality of viewpoints; acquires image quality adjustment information for adjusting the image quality of a plurality of images (multi-view images) corresponding to different viewpoints that are generated by performing a restoration operation based on the compressed image; determines values ​​of calculation parameters to be used in the restoration operation based on the image quality adjustment information; and generates the plurality of images by performing the restoration operation on the compressed image using the determined values ​​of the calculation parameters.

[0041] According to this, the values ​​of the calculation parameters are determined based on the image quality adjustment information. In other words, since the values ​​of the calculation parameters are determined by providing the image quality adjustment information, there is no need to directly set the values ​​of the calculation parameters, and the difficulty of setting the values ​​of the calculation parameters can be reduced. As a result, it is possible to set appropriate values ​​for the calculation parameters and effectively perform restoration calculations for compressed sensing of multi-view images.

[0042] (Embodiments) Hereinafter, embodiments will be specifically described with reference to the drawings.

[0043] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, the arrangement and connection of the components, steps, and the order of steps shown in the following embodiments are merely examples and are not intended to limit the technology of the present disclosure.

[0044] Furthermore, each figure is a schematic diagram and is not necessarily an exact illustration. Therefore, for example, the scales of the figures do not necessarily match. Furthermore, in each figure, substantially the same components are given the same reference numerals, and duplicated explanations may be omitted or simplified.

[0045] In the following, terms indicating the relationship between elements, such as parallel or perpendicular, terms indicating the shape of elements, such as rectangle or circle, and numerical ranges are not expressions that only express a strict meaning, but are expressions that also include a substantially equivalent range, for example, a difference of about a few percent.

[0046] In the following description, all or part of a circuit, unit, or device, or all or part of a functional block in a block diagram, may be implemented by one or more electronic circuits, including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). The LSI or IC may be integrated on a single chip or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated on a single chip. While the terms LSI and IC are used here, the term may be changed depending on the degree of integration, and may be referred to as a system LSI, a VLSI (very large scale integration), or an ULSI (ultra large scale integration). Field programmable gate arrays (FPGAs), which are programmed after the LSI is manufactured, or reconfigurable logic devices (RLDs), which can reconfigure connections within the LSI or set up circuit partitions within the LSI, can also be used for the same purpose.

[0047] Furthermore, all or part of a circuit, unit, or device, or all or part of a functional block in a block diagram, may be implemented by a software program. In this case, the software program is recorded on one or more non-transitory storage media such as a ROM, optical disk, hard disk drive, etc., and when the software program is executed by a processor, the functions specified in the software program are performed by the processor and peripheral devices. A system or device may include one or more non-transitory storage media in which a software program is stored, a processor, and necessary hardware devices (e.g., memory, interface, etc.).

[0048] [Functional Configuration of Imaging System 1000] First, the functional configuration of imaging system 1000 according to this embodiment will be specifically described with reference to Fig. 1. Fig. 1 is a functional configuration diagram of imaging system 1000 according to this embodiment. Note that Fig. 1 shows an exemplary functional configuration of imaging system 1000, and the functional configuration of imaging system 1000 is not limited to that shown in Fig. 1.

[0049] 1, the imaging system 1000 includes an imaging device 100, an information processing device 200, and a display device 300. The imaging device 100, the information processing device 200, and the display device 300 included in the imaging system 1000 will be described in detail below.

[0050] The imaging device 100 has a configuration similar to that of the imaging device disclosed in Patent Document 1, and is capable of capturing a compressed image including a plurality of pixels. Each of the plurality of pixels included in the compressed image includes information of four or more wavelength bands. The imaging device 100 includes a control circuit 110 and an image sensor 120.

[0051] The control circuit 110 controls the image sensor 120 to cause the image sensor 120 to generate a compressed image.

[0052] The image sensor 120 is a monochrome photodetector having a plurality of photodetection elements arranged in a matrix. Examples of the image sensor 120 include a charge-coupled device (CCD) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, an infrared array image sensor, a terahertz array image sensor, and a millimeter-wave array image sensor. The image sensor 120 does not have to be a monochrome photodetector and may be a color photodetector. The wavelength range detectable by the image sensor 120 is not particularly limited and may be, for example, visible light, ultraviolet light, infrared light, terahertz waves, or any combination thereof.

[0053] 1, the imaging device 100 further includes a filter array 130 and an optical system 140. The configuration of the imaging device 100 including the filter array 130 and the optical system 140 will be described later with reference to FIG.

[0054] The information processing device 200 is communicably connected to the imaging device 100 and the display device 300 via wired and / or wireless communication. The information processing device 200 includes an image acquisition unit 210, an information acquisition unit 220, a parameter determination unit 230, a reconstruction calculation unit 240, and a storage unit 250.

[0055] The image acquisition unit 210 can acquire a compressed image from the imaging device 100 and store it in the storage unit 250. Note that the image acquisition unit 210 does not have to acquire the compressed image directly from the imaging device 100, and may acquire the compressed image via another device or the like.

[0056] The information acquisition unit 220 can acquire image quality adjustment information from the display device 300. Note that the information acquisition unit 220 does not have to acquire the image quality adjustment information directly from the display device 300, and may acquire the compressed image via another device or the like.

[0057] The parameter determination unit 230 can determine the values ​​of the calculation parameters used in the restoration calculation based on the image quality adjustment information. That is, the parameter determination unit 230 can determine, based on the image quality adjustment information, one or more values ​​that correspond one-to-one to one of the calculation parameters used in the restoration calculation. The image quality adjustment information refers to information for adjusting the image quality of four or more spectral images obtained by the restoration calculation. For example, the image quality adjustment information may include a first priority value that indicates the priority of the image quality of the four or more spectral images, a second priority value that indicates the priority of the speed at which the four or more spectral images are generated from the compressed image, or a combination of the first priority value and the second priority value. The calculation parameters may be a first parameter (a weighting coefficient τ, described later) that corresponds to the influence of regularization in the restoration calculation and / or a second parameter that corresponds to the number of iterations of the iterative calculation included in the restoration calculation.

[0058] The reconstruction calculation section 240 can generate four or more spectral images by performing reconstruction calculations on the compressed image using the values ​​of the calculation parameters determined by the parameter determination section 230 .

[0059] The restoration calculation performed in this embodiment may be the same as the restoration calculation described in Patent Document 1 or 2. Specifically, four or more spectroscopic images may be restored based on the following equation (1).

[0060]

[0061] Here, g is data representing a compressed image, and is expressed, for example, as a one-dimensional array (i.e., vector). If the compressed image is an image of n×m pixels, data g is expressed as a one-dimensional array having n×m elements. f is data representing w spectral images that correspond one-to-one to w wavelength bands, and is expressed, for example, as a one-dimensional array. f1 is the wavelength band W １ The spectral image data corresponding to the wavelength band W ２ , fw is the wavelength band W １ , fw are data of spectral images corresponding to the wavelength bands W of the filter array 130. Each of f1, f2, ..., fw is represented by, for example, a one-dimensional array. If each spectral image is an image of nxm pixels, each of f1, f2, ..., fw is represented by a one-dimensional array having nxm elements, and data f is represented by a one-dimensional array having nxmxw elements. H is a matrix with nxm rows and nxmxw columns, and may be referred to as a system matrix. H is a matrix of ... １ Transmission spectrum of the filter array 130 wavelength band W ２ The transmission spectrum of the filter array 130 wavelength band W ｗ The data f satisfying the formula (1) can be estimated using a compressed sensing technique, and specifically, can be estimated by the formula (2).

[0062]

[0063] Equation (2) expresses finding f that minimizes the sum of the first and second terms in the parentheses. Data f as a final solution can be calculated by converging the solution through recursive iterative calculations.

[0064] The first term in the parentheses in equation (2) represents the sum of squares of the difference between Hf obtained by system transformation of data f in the estimation process using matrix H and data g, and is a so-called residual term. Here, the sum of squares is used, but instead of the sum of squares, the sum of absolute values ​​or the square root of the sum of squares may be used. The sum of squares of the difference between Hf and data g is expressed as follows: (g １ -r １ ) × (g １ -r １ ) + ... + (g ｎ×ｍ -r ｎ×ｍ ) × (g ｎ×ｍ -r ｎ×ｍ ) where g = (g １ ...g ｎ×ｍ ) Ｔ , Hf = (r １ ...r ｎ×ｍ ) Ｔ is.

[0065] The second term in parentheses in Equation (2) is a regularization term, sometimes called a stabilization term. Φ(f) represents a constraint on the regularization of f and is a function reflecting the sparse information of the data f. This function has the effect of smoothing or stabilizing the data f. Φ(f) can be expressed, for example, by a discrete cosine transform (DCT), a wavelet transform, a Fourier transform, a total variation (TV), or any combination thereof. τ is a weighting coefficient for the regularization term and corresponds to the influence of regularization in the reconstruction calculation. The larger the value of τ, the greater the influence of regularization, the greater the amount of redundant data removed, and the stronger the convergence of the solution in the iterative calculation. Conversely, the smaller the value of τ, the less the influence of regularization, the less redundant data removed, and the weaker the convergence of the solution in the iterative calculation.

[0066] In Equation (2), τ may be used as a first parameter. Furthermore, the number of iterations of the recursive iterative calculation of Equation (2) may be used as a second parameter. The reconstruction calculation unit 240 can generate four or more spectroscopic images by performing the reconstruction calculation of Equation (2) using the values ​​of the first parameter and the second parameter.

[0067] The storage unit 250 can store a compressed image and / or four or more spectroscopic images (reconstructed images), etc. The storage unit 250 can be implemented using, for example, a hard disk drive and / or a solid state drive.

[0068] The display device 300 is a user interface and includes an input unit 310 and a display unit 320. The display device 300 can be implemented using, for example, a tablet computer, a smartphone, or a desktop computer.

[0069] The input unit 310 can receive image quality adjustment information from a user. The input unit 310 can be implemented using, for example, a touch screen, a touch pad, a mouse, a keyboard, or any combination thereof.

[0070] The display unit 320 can display a graphical user interface (GUI) for acquiring image quality adjustment information. The display unit 320 can be implemented using, for example, a liquid crystal display (LCD) and / or an organic light-emitting diode (OLED) display.

[0071] 1 illustrates an exemplary functional configuration of the imaging system 1000, and the functional configuration of the imaging system 1000 is not limited to that illustrated in FIG. 1. For example, part or all of the imaging device 100 may be included in the information processing device 200. Also, for example, part or all of the display device 300 may be included in the information processing device 200. Also, for example, the information processing device 200 may be divided into multiple devices, and may be realized by, for example, a cloud server.

[0072] [Configuration of Image Capturing Apparatus 100] Next, the configuration of the image capturing apparatus 100 will be described with reference to Fig. 2. Fig. 2 is a schematic diagram of the image capturing apparatus 100 according to the embodiment.

[0073] The imaging device 100 has a configuration similar to that of the imaging devices disclosed in Patent Documents 1 and 2. Specifically, the imaging device 100 includes a control circuit 110, an image sensor 120, a filter array 130, and an optical system 140. Note that the control circuit 110 is not shown in FIG. 2 .

[0074] The filter array 130 is disposed on the optical path of light incident from the object 70, which is the subject, and is disposed between the optical system 140 and the image sensor 120 in FIG. 2 . The filter array 130 functions as the encoding element of Patent Document 1. The filter array 130 may be integrated with the image sensor 120. Note that the arrangement of the filter array 130 is not limited to the arrangement shown in FIG. 2 . For example, the filter array 130 may be disposed between the optical system 140 and the image sensor 120, but away from the image sensor 120. Alternatively, for example, the filter array 130 may be disposed between the object 70 and the optical system 140. Alternatively, for example, the filter array 130 may be disposed within the optical system 140.

[0075] The optical system 140 is disposed on the optical path of light incident from the object 70, and in FIG. 2 is disposed between the object 70 and the filter array 130. The optical system 140 includes at least one lens and can form an image of the object 70 on the imaging surface of the image sensor 120 via the filter array 130. Note that the configuration and arrangement of the optical system 140 are not limited to the configuration and arrangement shown in FIG. 2. For example, the optical system 140 may be disposed between the filter array 130 and the image sensor 120. Alternatively, for example, the optical system 140 may include multiple lenses lined up on the optical path. In this case, the filter array 130 may be disposed between adjacent lenses of the multiple lenses.

[0076] [Configuration of Filter Array 130] The filter array 130 includes a plurality of filters. The number of the plurality of filters may be n×m. The n×m filters are１１ ,···,filter ｎｍ Contains filters １１ Wavelength band W １ ~Transmission spectrum S in wavelength band Ww １１ , ..., wavelength band W of filter nm １ ~Transmission spectrum S in wavelength band Ww ｎｍ may all be different, or the transmission spectrum S １１ , ..., transmission spectrum S ｎｍ Some of the elements may be the same.

[0077] filter １１ Wavelength band W １ Transmittance S １１Ｗ１ , ..., transmittance S of the wavelength band Ww of the filter 11 １１Ｗｗ ,···,filter ｎｍ Wavelength band W １ Transmittance S ｎｍＷ１ , ..., wavelength band W of filter nm ｗ Transmittance S ｎｍＷｗ may all be different, or the transmittance S １１Ｗ１ ,..., transmittance S １１Ｗｗ ,..., transmittance S ｎｍＷ１ ,..., transmittance S ｎｍＷｗ may be the same. w may be an integer of 4 or more. In the present disclosure, transmittance may mean light transmittance.

[0078] Wavelength band W α The transmittance of the filter β in the above formula may be expressed by the following formula (3):

[0079]

[0080] Wαmin is the wavelength band W α Wαmax is the minimum wavelength value of the wavelength band W α is the maximum wavelength value, h(λ) is a function indicating the transmission spectrum, and λ is the wavelength.

[0081] In addition, the wavelength band W α The transmittance of the filter β in the wavelength band W is not limited to the formula (3). αThe transmittance of the filter β in the wavelength band W may be the transmittance obtained by dividing the formula (3) by (Wαmax-Wαmin). α The transmittance of the filter β in the wavelength band W α The frequency λ that represents α０ Transmittance h (λ α０ ) may be used. α０ is Wαmin≦λ α０ ≦Wαmax. For example, the wavelength band W α The center frequency may be ((Wαmax−Wαmin) / 2).

[0082] The configuration of the filter array 130 will be described with reference to Figures 3A to 3E. Figure 3A is a schematic diagram of the filter array 130 according to an embodiment. The filter array 130 includes a filter 130a and a filter 130b. Figure 3B shows an example of the transmission spectrum of the filter 130a. Figure 3C' shows an example of the transmission spectrum of the filter 130b.

[0083] FIG. 3D shows the wavelength band W of the filter array 130 according to the embodiment. １ 3E is a diagram illustrating an example of the transmittance of the wavelength band W of the filter array 130 according to the embodiment. ２ 3D and 3E , the shading of each region represents the transmittance of the filter, with lighter regions representing higher transmittance and darker regions representing lower transmittance.

[0084] The filter array 130 includes a plurality of filters arranged in a matrix. In the example shown in FIG. 3A , the filter array 130 includes 48 filters arranged in 6 rows and 8 columns. Filter 130a is the filter arranged in the upper left of the 48 filters, and filter 130b is the filter arranged in the lower right of the 48 filters. Note that the number of filters included in the filter array 130 is not limited to 48. For example, the number of filters included in the filter array 130 may be approximately the same as the number of pixels of the image sensor 120, and may be determined depending on the application, for example, in a range from several tens to several tens of millions.

[0085] The wavelength dependence of the transmittance of the multiple filters included in the filter array 130 is different from one another. For example, in the filter 130a, １ The transmittance of the wavelength band W ２ On the other hand, in the filter 130b, the transmittance of the wavelength band W １ The transmittance of the wavelength band W ２ In other words, the wavelength dependency of the transmittance of the filter 130a is different from the wavelength dependency of the transmittance of the filter 130b. １ and W ２ The transmittance of the other wavelength bands included in the four or more wavelength bands will be illustrated and described below, and the illustration and description of the transmittance of the other wavelength bands will be omitted.

[0086] [Information Processing Method] Next, an information processing method in the imaging system 1000 configured as above will be described with reference to Figures 4 to 6. Figure 4 is a flowchart of the information processing method according to the embodiment.

[0087] [Steps S110 to S130] First, the image acquisition unit 210 of the information processing device 200 acquires a compressed image from the imaging device 100 (S110). The display unit 320 of the display device 300 displays a GUI for acquiring image quality adjustment information (S120). The input unit 310 of the display device 300 receives input of the image quality adjustment information via the GUI, and the information acquisition unit 220 of the information processing device 200 acquires the image quality adjustment information from the display device 300 (S130).

[0088] 5 is a diagram showing an example of a GUI according to an embodiment. Note that in FIG. 5, the labels "Compressed Image" and "Restored Image (Current)" indicate areas where images are displayed, and may not be displayed in the actual GUI.

[0089] For example, the display unit 320 displays the GUI shown in FIG. 5 in step S120, and the input unit 310 receives image quality adjustment information via the slider 30 in step S130. The compressed image acquired in step S110 is displayed in the compressed image 10. The slider 30 is a GUI component for setting image quality adjustment information, and sets image quality adjustment information corresponding to the horizontal position of the indicator 30a. In FIG. 5, the indicator 30a can set values ​​for the image quality priority (first priority) of four or more spectral images (i.e., hyperspectral images) and the priority (second priority) of the speed of the restoration calculation as image quality adjustment information. For example, when the indicator 30a is located at the left end, the slider 30 sets the first priority to 0% and the second priority to 100%. On the other hand, when the indicator 30a is located at the right end, the slider 30 sets the first priority to 100% and the second priority to 0%. For example, when the indicator 30a is positioned in the center, the slider 30 sets the first priority to 50% and the second priority to 50%.

[0090] In this embodiment, the higher the first priority, the larger the first priority value, but it may also be smaller. Similarly, the higher the second priority, the larger the second priority value, but it may also be smaller. Furthermore, the second priority value is dependent on the first priority value, and conversely, the first priority value is dependent on the second priority value. Therefore, it is not necessary to acquire both the first priority value and the second priority value, and only one of the first priority value and the second priority value may be acquired.

[0091] 5 shows an exemplary configuration of the GUI, and the configuration of the GUI is not limited to that shown in FIG. 5. For example, other GUI components (e.g., buttons and / or text boxes) may be used instead of the slider 30. Also, the compressed image 10, the current restored image 20, the current restoration time 40, or any combination thereof may not be displayed.

[0092] [Step S140] The parameter determination unit 230 of the information processing device 200 determines the values ​​of the calculation parameters used in the restoration calculation based on the image quality adjustment information (S140).

[0093] 6 is a diagram illustrating an example of a conversion table according to an embodiment. For example, the parameter determination unit 230 determines the value of the influence degree of regularization and the value of the number of iterations by referring to the conversion table illustrated in FIG.

[0094] 6, the value of the regularization influence is set so that the higher the image quality priority (first priority), the lower the influence of regularization. For example, the highest image quality priority value "100%" is set to a regularization influence value of "0.1," and the lowest image quality priority value "0%" is set to a regularization influence value of "1.0." This allows the parameter determination unit 230 to determine the value of the first parameter corresponding to the influence of regularization so that the higher the image quality priority, the lower the influence of regularization.

[0095] 6, the regularization influence value is set so that the lower the speed priority (second priority), the lower the influence of regularization. For example, the regularization influence value "0.1" is set for the lowest speed priority value "0%," and the regularization influence value "1.0" is set for the highest speed priority value "100%." ​​This allows the parameter determination unit 230 to determine the value of the first parameter corresponding to the influence of regularization so that the lower the speed priority, the lower the influence of regularization.

[0096] 6, the value of the number of iterations is set so that the higher the image quality priority, the greater the number of iterations. For example, the highest image quality priority value "100%" is set to the value of the number of iterations "100," and the lowest image quality priority value "0%" is set to the value of the number of iterations "100." This allows the parameter determination unit 230 to determine the value of the second parameter corresponding to the number of iterations so that the higher the image quality priority, the greater the number of iterations of the iterative calculation.

[0097] 6, the value of the number of iterations is set so that the lower the speed priority, the greater the number of iterations. For example, the lowest speed priority value "0%" is set to the value of the number of iterations "1000," and the highest speed priority value "100%" is set to the value of the number of iterations "100." This allows the parameter determination unit 230 to determine the value of the second parameter corresponding to the number of iterations so that the lower the speed priority, the greater the number of iterations of the iterative operation.

[0098] It is not necessary to use a conversion table to determine the parameter. For example, the value of τ as the first parameter may be determined based on the image quality priority P using the following equation (4).

[0099]

[0100] In this way, since the values ​​of the calculation parameters are determined based on the image quality adjustment information, there is no need to directly set values ​​for the calculation parameters, which makes it easier to set the values ​​of the calculation parameters.

[0101] [Step S150] The reconstruction calculation unit 240 of the information processing device 200 generates four or more spectroscopic images by performing a reconstruction calculation on the compressed image using the determined values ​​of the calculation parameters (S150). Specifically, the reconstruction calculation unit 240 applies the value of the first parameter (τ) and the value of the second parameter (the number of iterations) determined in step S140 to equation (2) to estimate data f.

[0102] [Step S160] The display unit 320 of the display device 300 displays the restored image and the restoration time (S160). For example, as shown in FIG. 5 , the current restored image 20 and the current restoration time 40 are displayed. Here, the displayed current restored image 20 and the current restoration time 40 correspond to the current horizontal position of the indicator 30a. That is, a superimposed image of four or more spectral images restored based on the current image quality adjustment information is displayed as the current restored image 20, and the time required for the restoration calculation of the four or more spectral images is displayed as the current restoration time 40. Note that four or more spectral images may be displayed instead of or in addition to the superimposed image.

[0103] As described above, the GUI according to this modification allows the user to recognize the current restored image 20 and the current restoration time 40, and to easily set image quality adjustment information. For example, if the image quality of the current restored image 20 is poor, the user can reset the image quality priority to a higher level. Also, for example, if the current restoration time 40 is long, the user can reset the speed priority to a higher level.

[0104] 5 is an example, and the steps and the order of the steps are not limited to those shown in Fig. 5. For example, steps S120 and S160 are optional steps and may not be included in the information processing method according to the present disclosure.

[0105] (Modification 1) Next, Modification 1 of the embodiment will be described. This modification is different from the above embodiment mainly in the GUI. The GUI according to this modification will be described below with reference to FIG. 7, focusing on the differences from the above embodiment.

[0106] Fig. 7 is a diagram showing an example of a GUI according to Modification 1. In Fig. 7, the labels "Compressed Image", "Restored Image (Current)", and "Restored Image (Previous)" indicate areas where images are displayed, and may not be displayed on the actual GUI.

[0107] 7 further displays the previous restored image 21 and the previous restoration time 41. Here, the displayed previous restored image 21 and the previous restoration time 41 correspond to the previous horizontal position of the indicator 30a. In other words, a hyperspectral image in which four or more spectral images restored based on the previous image quality adjustment information are superimposed is displayed as the previous restored image 21, and the time required for the restoration calculation of the four or more spectral images is displayed as the previous restoration time 41.

[0108] This allows the user to compare the current restored image 20 and the current restoration time 40 with the previous restored image 21 and the previous restoration time 41, and easily set image quality adjustment information. For example, if the previous restored image 21 and the previous restoration time 41 are superior to the current restored image 20 and the current restoration time 40, the user can reset the image quality priority and the speed priority to the previous image quality priority and the speed priority values. As a result, the restoration calculation for compressed sensing can be performed more effectively.

[0109] (Modification 2) Next, Modification 2 of the embodiment will be described. This modification is different from the above embodiment mainly in the GUI. The GUI according to this modification will be described below, focusing on the differences from the above embodiment, with reference to FIGS. 8 and 9.

[0110] 8 and 9 are diagrams showing an example of a GUI according to Modification 2. In Fig. 8 and 9, the labels "Compressed Image", "Restored Image (Current)", and "Restored Image (Previous)" indicate areas where images are displayed, and may not be displayed on the actual GUI.

[0111] 8 further displays a button 45. The button 45 is a button for switching between the result of the current restoration calculation and the result of the previous restoration calculation. The result of the current restoration calculation means the result of the restoration calculation performed based on the image quality adjustment information corresponding to the current horizontal position of the indicator 30 a, and the result of the previous restoration calculation means the result of the restoration calculation performed based on the image quality adjustment information corresponding to the previous horizontal position of the indicator 30 a.

[0112] Specifically, by pressing the button 45 labeled "Previous Parameters" in Fig. 8, the result of the current restoration operation is switched to the result of the previous restoration operation. As a result, as shown in Fig. 9, the current restored image 20 and the current restoration time 40 are changed to the previous restored image 21 and the previous restoration time 41, and the label of the button 45 is changed to "Current Parameters." Furthermore, by pressing the button 45 labeled "Current Parameters" in Fig. 9, the result of the previous restoration operation is switched to the result of the current restoration operation. As a result, as shown in Fig. 8, the previous restored image 21 and the previous restoration time 41 are changed to the current restored image 20 and the current restoration time 40, and the label of the button 45 is changed to "Previous Parameters."

[0113] As described above, the GUI of this modified example allows the user to switch between displaying the current restored image 20 and current restoration time 40 and the previous restored image 21 and previous restoration time 41, and allows the image size to be larger than that shown in Figure 7, thereby improving visibility for the user.

[0114] (Variation 3) Next, Variation 3 of the embodiment will be described. This variation differs from the above embodiment mainly in the GUI. The GUI and related processing of this variation will be described below, focusing on the differences from the above embodiment, with reference to FIGS. 10 and 11 .

[0115] Fig. 10 is a diagram showing an example of a GUI according to this modification. Note that in Fig. 10, the labels "Compressed Image" and "Restored Image (Current)" indicate areas where images are displayed, and may not be displayed in the actual GUI.

[0116] 10, a radio button 31 is displayed instead of the slider 30. The radio button 31 is a GUI component that allows a user to select one of a speed priority option 31 a and an image quality priority option 31 b, which are mutually exclusive. The user can select either the speed priority or the image quality priority via the radio button 31.

[0117] The parameter determination process (step S140 in FIG. 4) when either speed priority or image quality priority is selected via such a GUI will be described with reference to FIG. 11. FIG. 11 is a flowchart of the parameter determination process (step S140 in FIG. 4) according to this modified example.

[0118] First, the parameter determination unit 230 determines whether the average luminance value of the compressed image is smaller than a threshold value (S141). That is, the parameter determination unit 230 calculates the average value of the luminance values ​​of multiple pixels included in the compressed image, and determines whether the calculated average value is smaller than a threshold value. The threshold value may be a value determined empirically and / or experimentally in advance.

[0119] Here, if the average luminance value of the compressed image is smaller than the threshold value (Yes in S141), the parameter determination unit 230 sets τ to the first parameter (weighting coefficient τ of the regularization term) corresponding to the influence of regularization in the restoration calculation. １ (0 < τ １ On the other hand, if the average luminance value of the compressed image is not smaller than the threshold value (No in S141), the parameter determination unit 230 sets τ ２ (0 < τ ２ ) is set (S143). １ and τ ２ is τ １ <τ ２ That is, the parameter determination unit 230 sets the initial value of the first parameter so that the influence of regularization is low when the average luminance value of the compressed image is low.

[0120] The parameter determination unit 230 determines whether the image quality adjustment information acquired via the GUI indicates that image quality is to be prioritized (S144). Specifically, the parameter determination unit 230 determines whether the image quality priority option 31b has been selected using the radio button 31 in FIG. 10.

[0121] Here, if the image quality adjustment information indicates that image quality is prioritized, that is, if the image quality priority option 31b is selected (Yes in S145), the parameter determination unit 230 updates the value of the first parameter so as to reduce the influence of regularization (S145). For example, the parameter determination unit 230 updates the value of the first parameter by multiplying the first parameter set in step S142 or S143 by a coefficient c (0<c<1). In other words, when the image quality priority is high and the speed priority is low, the parameter determination unit 230 updates the value of the first parameter so as to reduce the influence of regularization. Furthermore, the parameter determination unit 230 sets the second parameter (number of iterations) to N １ is set (S146).

[0122] On the other hand, if the image quality adjustment information does not indicate that image quality is prioritized, that is, if the speed priority option 31a is selected (No in S145), the parameter determination unit 230 sets the second parameter (number of iterations) to N. ２ is set (S147). １ and N ２ is N １ >N ２ That is, the parameter determination unit 230 sets the value of the second parameter so that the number of iterations is reduced when the image quality priority is low and the speed priority is high.

[0123] Note that multiplying the value of the first parameter by the coefficient c in step S145 is an example, and the process of updating the value of the first parameter in step S145 is not limited to this. For example, the parameter determination unit 230 may subtract a constant a (a > 0) from the value of the first parameter. Also, for example, the parameter determination unit 230 may update the value of the first parameter by performing both multiplication (coefficient c) and subtraction (constant a).

[0124] As described above, with the GUI according to this modification, the values ​​of the calculation parameters are determined by selecting either speed priority or image quality priority, which makes it easier to set the values ​​of the calculation parameters, thereby enabling more effective reconstruction calculation of compressed sensing.

[0125] (Modification 4) Next, Modification 4 of the embodiment will be described. This modification is different from the above embodiment mainly in the GUI. The GUI and related processing of this modification will be described below, focusing on the differences from the above embodiment, with reference to FIGS. 12 and 13 .

[0126] Fig. 12 is a diagram showing an example of a GUI according to this modification. Note that in Fig. 12, the labels "Compressed Image" and "Restored Image (Current)" indicate areas where images are displayed, and may not be displayed in the actual GUI.

[0127] In the GUI of Fig. 12, a button set 32 ​​including three buttons 32a to 32c is displayed instead of the slider 30. Button 32a is a GUI component for instructing that speed be given priority over the current image quality adjustment information. Button 32b is a GUI component for instructing that the image quality of the restored image be given priority over the current image quality adjustment information. Button 32c is a GUI component for instructing that the current image quality adjustment information be confirmed. The user can confirm the image quality adjustment information by repeatedly pressing buttons 32a and 32b until an appropriate restored image and restoration time are obtained, and finally pressing button 32c.

[0128] An information processing method when image quality adjustment information is set via such a GUI will be described with reference to Fig. 13. Fig. 13 is a flowchart of the information processing method according to this modified example. Note that steps S110 to S160 in Fig. 13 are the same as those in Fig. 4, so their description will be omitted. Steps S310 to S350, which are executed after step S160, will be described below.

[0129] If button 32b is pressed (Yes in S310), the image quality adjustment information is updated (S320) so that the value of the image quality priority (first priority) increases, that is, the value of the speed priority (second priority) decreases, and the process returns to step S140.

[0130] If button 32b is not pressed (No in S310) and button 32a is pressed (Yes in S330), the image quality adjustment information is updated so that the image quality priority value decreases, i.e., the speed priority value increases (S340), and the process returns to step S140.

[0131] If button 32b is not pressed (No in S310), button 32a is not pressed (No in S330), and button 32c is pressed (Yes in S350), the information processing method ends. If button 32b is not pressed (No in S310), button 32a is not pressed (No in S330), and button 32c is not pressed (No in S350), the process returns to step S310.

[0132] As described above, with the GUI according to this modification, image quality adjustment information can be updated by repeatedly pressing the two buttons 32 a and 32 b, making it easier to fine-tune the values ​​of the calculation parameters, thereby enabling more effective reconstruction calculations for compressed sensing.

[0133] (Variation 5) Next, Variation 5 of the embodiment will be described. This variation differs from the above embodiment mainly in that information indicating color reproducibility in four or more spectral images is used instead of speed priority (first priority) and image quality priority (second priority). The GUI and related processing according to this variation will be described below with reference to FIG. 5, focusing on the differences from the above embodiment.

[0134] In the GUI according to this modification, the labels "Color reproducibility: low" and "Color reproducibility: high" are displayed instead of the "Speed ​​priority" and "Image quality priority" labels in Fig. 5. Therefore, when the indicator 30a is positioned at the left end, the slider 30 sets the image quality adjustment information that indicates the lowest color reproducibility. On the other hand, when the indicator 30a is positioned at the right end, the slider 30 sets the image quality adjustment information that indicates the highest color reproducibility.

[0135] The parameter determination unit 230 may determine a value of a first parameter corresponding to the influence of regularization such that the influence of regularization increases as color reproducibility increases. The parameter determination unit 230 may also determine a value of a second parameter corresponding to the number of iterations such that the number of iterations increases as color reproducibility increases. Conversely, the parameter determination unit 230 may determine a value of the first parameter such that the influence of regularization decreases as color reproducibility decreases. The parameter determination unit 230 may also determine a value of the second parameter such that the number of iterations decreases as color reproducibility decreases.

[0136] As described above, the GUI according to this modification allows the value of the calculation parameter to be determined by setting the level of color reproducibility. The color reproducibility depends on the influence of regularization and the number of iterations. For example, the lower the influence of regularization, the higher the color reproducibility. Therefore, by determining the value of the calculation parameter based on color reproducibility, the restoration calculation of compressed sensing can be performed more effectively.

[0137] (Variation 6) Next, Variation 5 of the embodiment will be described. This variation differs from the above embodiment mainly in that information indicating the size of structures possessed by a subject in four or more spectroscopic images is used instead of the speed priority (first priority) and image quality priority (second priority). The GUI and related processing according to this variation will be described below with reference to FIG. 5, focusing on the differences from the above embodiment.

[0138] In the GUI according to this modification, the labels "Structure size: small" and "Structure size: large" are displayed instead of the "Speed ​​priority" and "Image quality priority" labels in Fig. 5. Therefore, when the indicator 30a is positioned at the left end, the slider 30 sets the image quality adjustment information indicating the smallest structure size. On the other hand, when the indicator 30a is positioned at the right end, the slider 30 sets the image quality adjustment information indicating the largest structure size.

[0139] The parameter determination unit 230 may determine a value of a first parameter corresponding to the influence of regularization such that the influence of regularization decreases as the size of the structure decreases. The parameter determination unit 230 may also determine a value of a second parameter corresponding to the number of iterations such that the number of iterations increases as the size of the structure decreases. Conversely, the parameter determination unit 230 may determine a value of the first parameter such that the influence of regularization increases as the size of the structure increases. The parameter determination unit 230 may also determine a value of the second parameter such that the number of iterations decreases as the size of the structure increases.

[0140] As described above, the GUI according to this modification allows the values ​​of the calculation parameters to be determined by setting the size of the structure. To properly restore the structure, the influence of regularization and the number of iterations must be determined according to the size of the structure. For example, the lower the influence of regularization, the smaller the size of the structure that can be restored. Therefore, by determining the values ​​of the calculation parameters based on the size of the structure, the restoration calculation for compressed sensing can be performed more effectively.

[0141] (Other Embodiments) While the information processing method has been described above based on the embodiments and modifications, the information processing method according to the present disclosure is not limited to the above embodiments and modifications. The present disclosure also includes other embodiments realized by combining any of the components in the above embodiments and modifications, and modifications obtained by applying various modifications that would occur to a person skilled in the art to the above embodiments and modifications without departing from the spirit of the present disclosure.

[0142] For example, any combination of the above-described embodiment, modification 5, and modification 6 may be used. In other words, the image quality adjustment information may include any combination of image quality priority values, speed priority values, color reproducibility values, and structure size values.

[0143] In the above embodiments and variations, a hyperspectral image is defined as an image that includes four or more spectroscopic images, but it may also be defined as an image that includes five or more, six or more, seven or more, eight or more, nine or more, or ten or more spectroscopic images.

[0144] Furthermore, in the imaging device 100 according to the above embodiment and modified example, the filter array 130 is used as an encoding element, but this is not limited to this. For example, the metalens described in Non-Patent Document 1 may be used instead of the filter array 130 according to the above embodiment and modified example. Furthermore, for example, the CMOS image sensor described in Non-Patent Document 2 may be used instead of the filter array 130 and image sensor 120 according to the above embodiment and modified example.

[0145] Furthermore, although the information processing methods according to the above-described embodiments and modifications have been applied to compressed sensing of hyperspectral images, they are not limited to this. For example, they may be applied to compressed sensing of high dynamic range images, temporal super-resolution images, or images from a multi-eye camera.

[0146] For example, when applied to compressed sensing of high dynamic range images, the value of the parameter used in the calculation of equation (1) disclosed in Non-Patent Document 3 may be determined based on image quality adjustment information. In this case, ε in equation (1) of Non-Patent Document 3 corresponds to τ in equation (2) of the present disclosure. Note that a compressed image of a high dynamic range image can be obtained, for example, by using LCoS (Liquid Crystal on Silicon) and a beam splitter to change the exposure time for each pixel. Furthermore, for example, a compressed image may be obtained by installing multiple ND filters with different transmittances at multiple pixels. In the compressed image obtained in this manner, the exposure times or transmitted light amounts for the multiple pixels are different.

[0147] Furthermore, for example, when applied to compressed sensing of temporal super-resolution images, the values ​​of the parameters used in the calculation of equation (3) disclosed in Non-Patent Document 4 may be determined based on the image quality adjustment information. In this case, ε in equation (3) of Non-Patent Document 4 corresponds to τ in equation (2) of the present disclosure. Note that a compressed image of a temporal super-resolution image can be obtained by, for example, changing the exposure pattern within one frame on a pixel-by-pixel basis using a special CMOS sensor or LCoS. In the compressed image obtained in this manner, the exposure patterns of the coded exposures for multiple pixels are different.

[0148] Furthermore, for example, when applied to compressed sensing of multi-view images, the value of a parameter used in the calculation of equation (6) disclosed in Patent Document 3 may be determined based on image quality adjustment information. In this case, λ in equation (8) of Patent Document 3 corresponds to τ in equation (2) of the present disclosure. Note that a compressed image of a multi-view image can be obtained, for example, by receiving images of multiple viewpoints with a single image sensor through a mask with different transmittances on a pixel-by-pixel basis. In the compressed image obtained in this manner, data for each of the multiple pixels includes image information of multiple viewpoints.

[0149] The present disclosure can be used for image restoration using compressed sensing.

[0150] 10 Compressed image 20 Current restored image 21 Previous restored image 30 Slider 30a Indicator 31 Radio button 31a Speed ​​priority option 31b Image quality priority option 32 Button set 32a, 32b, 32c, 45 Button 40 Current restoration time 41 Previous restoration time 70 Object 100 Imaging device 110 Control circuit 120 Image sensor 130 Filter array 130a, 130b Filter 140 Optical system 200 Information processing device 210 Image acquisition unit 220 Information acquisition unit 230 Parameter determination unit 240 Restoration calculation unit 250 Memory unit 300 Display device 310 Input unit 320 Display unit 1000 Imaging system

Claims

1. An information processing method executed by a computer, comprising: acquiring a compressed image including a plurality of pixels, wherein data for each of the plurality of pixels includes information of four or more wavelength bands; acquiring image quality adjustment information for adjusting the image quality of four or more spectroscopic images corresponding to the four or more wavelength bands, which are generated by performing a restoration operation based on the compressed image; determining values ​​of calculation parameters to be used in the restoration operation based on the image quality adjustment information; and generating the four or more spectroscopic images by performing the restoration operation on the compressed image using the determined values ​​of the calculation parameters.

2. The information processing method according to claim 1, further comprising displaying a graphical user interface for acquiring the image quality adjustment information, the image quality adjustment information being acquired via the graphical user interface.

3. The information processing method according to claim 2, further comprising displaying the compressed image and the four or more spectral images thus generated.

4. The information processing method according to claim 3, further comprising displaying a time required for the restoration calculation.

5. An information processing method according to any one of claims 1 to 4, wherein the image quality adjustment information includes a first priority value which is a priority of the image quality of the four or more spectral images, a second priority value which is a priority of the speed of the restoration calculation, or a combination of the first priority value and the second priority value.

6. The information processing method according to claim 5, wherein the calculation parameters include a first parameter corresponding to a degree of influence of regularization in the restoration calculation.

7. The information processing method according to claim 6, wherein the value of the first parameter is determined such that the influence of the regularization decreases as the first priority is higher or as the second priority is lower.

8. The information processing method according to claim 5, wherein the calculation parameters include a second parameter corresponding to the number of iterations of an iterative calculation included in the restoration calculation.

9. The information processing method according to claim 8, wherein the value of the second parameter is determined such that the number of iterations increases as the first priority is higher or as the second priority is lower.

10. An information processing method according to any one of claims 1 to 4, wherein the image quality adjustment information includes at least one of information indicating color reproducibility in the four or more spectral images and information indicating the size of structures possessed by the subject in the four or more spectral images.

11. An information processing method according to any one of claims 1 to 4, wherein the calculation parameters include a first parameter corresponding to the degree of influence of regularization in the restoration calculation, and the value of the first parameter is determined based on an average luminance value of the compressed image and the image quality adjustment information.

12. An information processing device comprising: an image acquisition unit that acquires a compressed image including a plurality of pixels, wherein data for each of the plurality of pixels includes information on four or more wavelength bands; an information acquisition unit that acquires image quality adjustment information for adjusting the image quality of four or more spectroscopic images generated by performing a restoration calculation based on the compressed image, the four or more spectroscopic images corresponding to the four or more wavelength bands respectively; a parameter determination unit that determines values ​​of calculation parameters to be used in the restoration calculation based on the image quality adjustment information; and a restoration calculation unit that generates the four or more spectroscopic images by performing the restoration calculation on the compressed image using the determined values ​​of the calculation parameters.

Images