Image processing method and image processing system

Abstract

This image processing method includes: acquiring a compressed image (S101); determining one or more wavelength band groups (S102); determining, for each of the one or more wavelength band groups, one or more combinations each consisting of one or more wavelength bands combined into one wavelength band, on the basis of at least one of the number and the total width of the one or more wavelength bands constituting the wavelength band group (S103); and generating, on the basis of the compressed image, a plurality of spectroscopic images corresponding to a plurality of wavelength bands combined on the basis of the one or more combinations (S104).

Description

Image processing method and image processing system

[0001] The present disclosure relates to an image processing method and the like.

[0002] Patent Literature 1 discloses generating multiple two-dimensional image data corresponding to multiple wavelength bands included in one or more subwavelength ranges specified by a user. Patent Literature 1 also discloses combining multiple component bands. Patent Literature 2 also discloses generating multiple two-dimensional image data corresponding to multiple specified wavelength bands determined based on reference spectral data. Patent Literature 2 also discloses reducing the amount of calculation required for restoration processing by combining multiple component bands into one band.

[0003] Furthermore, Non-Patent Documents 1 and 2 describe techniques related to hyperspectral images.

[0004] International Publication No. 2021 / 192891 International Publication No. 2023 / 282069

[0005] "World's first technology has been established to capture hyperspectral images and videos by combining a metalens and AI with a regular digital camera - Combining optical technology and AI to transform a 'regular camera' into a 'camera that can see the properties of objects'" [online], October 24, 2022, Nippon Telegraph and Telephone Corporation, [Retrieved November 16, 2023], Internet <URL: https: / / group.ntt / jp / newsrelease / 2022 / 10 / 24 / 221024a.html> Ahasan Ahamed et. al., "Reconstruction-based spectroscopy using CMOS image sensors with random photon-trapping" “nanostructure per sensor”, Proc. SPIE 11971, High-Speed ​​Biomedical Imaging and Spectroscopy VII, 1197106, 2 March 2022

[0006] However, combining multiple wavelength bands may result in an inappropriate spectral image, for example, a large reconstruction error may occur depending on the number or total width of multiple wavelength bands combined into one wavelength band.

[0007] Therefore, an image processing method and the like that can generate an appropriate spectral image while reducing the processing load is provided.

[0008] An image processing method according to one aspect of the present disclosure includes acquiring a compressed image in which spectral information corresponding to a plurality of wavelength bands is compressed; determining one or more wavelength band groups, each of which is composed of one or more wavelength bands to be combined from among the plurality of wavelength bands; determining, for each of the one or more wavelength band groups, one or more combinations, each of which is composed of one or more wavelength bands to be combined into one wavelength band from among the one or more wavelength bands that make up the wavelength band group, based on at least one of the number and total width of the one or more wavelength bands that make up the wavelength band group; and generating, based on the compressed image, a plurality of spectroscopic images corresponding to the plurality of wavelength bands combined based on the one or more combinations.

[0009] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0010] An image processing method according to an aspect of the present disclosure makes it possible to generate an appropriate image while reducing the processing load.

[0011] 10 is a graph showing an example of the magnitude of a restoration error occurring in a band of interest due to band synthesis. FIG. 10 is a block diagram showing an example of a configuration of an image processing system in an embodiment. FIG. 10 is a conceptual diagram showing an example of a configuration of an imaging device in an embodiment. FIG. 10 is a conceptual diagram showing an example of a configuration of a filter array in an embodiment. FIG. 10 is a graph showing an example of a transmission spectrum of a filter in an embodiment. FIG. 10 is a graph showing an example of a transmission spectrum of another filter in an embodiment. FIG. 10 is a conceptual diagram showing an example of a transmittance of a first wavelength band of a filter array in an embodiment. FIG. 10 is a conceptual diagram showing an example of a transmittance of a second wavelength band of a filter array in an embodiment. FIG. 10 is a flowchart showing an example of an operation of an image processing system in an embodiment. FIG. 10 is a flowchart showing a first specific example of the operation of an image processing system in an embodiment. FIG. 10 is a graph showing an example of a correct spectrum restored using original mask data. FIG. 10 is a graph showing an example of a correct spectrum restored using mask data converted by averaging. FIG. 10 is a graph showing an example of a correct spectrum restored using mask data converted by addition. FIG. 10 is a flowchart showing band synthesis processing in an embodiment. FIG. 10 is a flowchart showing a second specific example of the operation of an image processing system in an embodiment. FIG. 10 is a conceptual diagram showing a first example of an operation screen of a user interface in an embodiment. FIG. 10 is a conceptual diagram showing a second example of an operation screen of a user interface in an embodiment. FIG. 10 is a conceptual diagram showing a fourth example operation screen of a user interface in an embodiment. FIG. 11 is a flowchart showing a third specific example of the operation of an image processing system in an embodiment. FIG. 12 is a flowchart showing a fourth specific example of the operation of an image processing system in an embodiment. FIG. 13 is a flowchart showing a fifth specific example of the operation of an image processing system in an embodiment. FIG. 14 is a relationship diagram showing an example of the relationship between wavelength and maximum synthesis bandwidth in an embodiment. FIG. 15 is a flowchart showing an example of a process for determining the maximum number of synthesis bands in an embodiment. FIG. 16 is a flowchart showing a sixth specific example of the operation of an image processing system in an embodiment. FIG. 17 is a relationship diagram showing an example of the relationship between the maximum processing time and the maximum number of bands to be restored in an embodiment.10 is a diagram showing the relationship between a plurality of wavelength bands W1, ..., Ww, a plurality of wavelength bands Wc1, ..., Wcp, and a plurality of wavelength bands Wd1, ..., Wdr in a modified example of the embodiment. ｔ 10 is a diagram illustrating an example of a method for generating a matrix H according to a modified example of the embodiment.

[0012] For example, an RGB camera detects light and generates an RGB image represented by information in three wavelength bands corresponding to red, green, and blue. On the other hand, a hyperspectral camera detects light and generates a hyperspectral image represented by information in four or more wavelength bands. Here, wavelength bands may simply be referred to as bands. Hyperspectral cameras and hyperspectral images are used in various fields, such as food inspection, biological testing, medical drug development, and mineral analysis.

[0013] Furthermore, from the viewpoint of cost and flexibility, it has been proposed to apply compressed sensing to hyperspectral cameras, i.e., to restore a hyperspectral image from a compressed image.

[0014] Here, a compressed image is an image in which spectral information corresponding to multiple bands is compressed for each region, e.g., an image in which information from four or more bands is superimposed. A hyperspectral image can be restored from the compressed image according to sparsity. That is, multiple spectral images corresponding to the multiple bands are restored from the compressed image. The regions can be, for example, pixels included in the image.

[0015] Restoring an image represented by information of four or more bands, such as a hyperspectral image, from a compressed image requires a huge amount of calculation. This results in a large processing load, as well as high power and energy consumption. Therefore, as described in Patent Documents 1 and 2, multiple bands may be synthesized (aggregated). This can reduce the processing load.

[0016] However, combining multiple bands may result in an inappropriate spectral image. For example, combining multiple bands may result in a decrease in wavelength resolution. Also, combining multiple bands may result in an inappropriate spectral image corresponding to the designated band of interest.

[0017] Here, to obtain a spectroscopic image of a band of interest among the multiple bands, the remaining multiple bands may be synthesized. This allows for efficient acquisition of a spectroscopic image of the band of interest. However, even in this case, synthesis of the remaining multiple bands may result in a large restoration error in the spectroscopic image of the band of interest.

[0018] 1 is a graph showing an example of the magnitude of a restoration error that occurs in a band of interest due to band synthesis. Specifically, FIG. 1 shows the results of a simulation of restoration errors that occur in the band of interest due to synthesis of multiple bands different from the band of interest.

[0019] In this simulation, the wavelength range of the imaging target is 445 nm to 725 nm. The wavelength range includes 28 bands. The 28 bands have wavelengths (specifically, center wavelengths) of 450 nm, 460 nm, ..., 720 nm, and each band has a bandwidth of 10 nm. Of the 28 bands, four bands having wavelengths of 450 nm, 580 nm, 590 nm, and 720 nm are bands of interest. Band synthesis is not performed on the band of interest, but may be performed on the other bands.

[0020] When multiple bands are combined into one band, element values ​​corresponding to the multiple bands combined into one band are combined in an operation matrix for generating multiple spectral images from a compressed image. That is, combining multiple bands corresponds to combining multiple element values ​​in the operation matrix. Also, here, a band having a wavelength of X (nm) (specifically, a center wavelength) may be referred to as an X (nm) band.

[0021] In addition, in the band synthesis in the example of FIG. 1, six conditions are applied: no synthesis, max2, max3, max4, max6, and max12.

[0022] Specifically, no compositing corresponds to no compositing and corresponds to restoring the original 28 bands.

[0023] max2 corresponds to a maximum of two bands being combined into one band, and corresponds to the restoration of 16 bands. The 16 bands correspond to 450, [460, 470], [480, 490], [500, 510], [520, 530], [540, 550], [560, 570], 580, 590, [600, 610], [620, 630], [640, 650], [660, 670], [680, 690], [700, 710], and 720 (nm). Here, [a, ..., b] indicate the combined bands of a, ..., b.

[0024] max3 corresponds to a maximum of three bands being combined into one band, and corresponds to the restoration of 12 bands: 450, [460, 470, 480], [490, 500, 510], [520, 530, 540], [550, 560, 570], 580, 590, [600, 610, 620], [630, 640, 650], [660, 670, 680], [690, 700, 710], and 720 (nm).

[0025] max4 corresponds to a maximum of four bands being combined into one band, and corresponds to the restoration of 10 bands, which correspond to 450, [460, 470, 480, 490], [500, 510, 520, 530], [540, 550, 560, 570], 580, 590, [600, 610, 620, 630], [640, 650, 660, 670], [680, 690, 700, 710], and 720 (nm).

[0026] The max6 value corresponds to a maximum of six bands being combined into one band, and corresponds to the restoration of eight bands, which are 450, [460, 470, ..., 510], [520, 530, ..., 570], 580, 590, [600, 610, ..., 650], [660, 670, ..., 710], and 720 (nm).

[0027] max12 corresponds to a maximum of 8 bands being combined into one band, and corresponds to the restoration of 6 bands, which correspond to 450, [460, 470, ..., 570], 580, 590, [600, 610, ..., 710], and 720 (nm).

[0028] The reconstruction error under each condition is shown in Figure 1. The specific calculation of the reconstruction error in Figure 1 is as follows.

[0029] First, a compressed image is created using the correct spectrum of the color chart and the mask data of the prototype camera, and then a spectrum (spectral image) is restored from the compressed image. The spectrum is restored by changing the number of iterations of the iterative calculation to 200, 500, 1000, 1500, and 2000.

[0030] Then, the mean absolute error (MAE) is calculated as the restoration error between the restored spectrum and the correct spectrum for the four target bands of 450, 580, 590, and 720 (nm) and for the 12 regions of the color chart. Specifically, the 12 regions are six regions of cyan, magenta, yellow, red, green, and blue, and six regions of monochrome.

[0031] In calculating the MAE, first, the spectral value at 450 nm (the correct spectrum) is extracted for each of the 12 regions of the color chart. Next, the intensity value of each of the 12 regions is calculated in the restored image at 450 nm. Then, for each of the 12 regions, the MAE is obtained as the restoration error between the correct spectral value and the intensity value of the restored image. The average of the restoration errors for the 12 regions is then defined as the error at 450 nm.

[0032] The above calculation is also performed for the other three bands of interest, specifically, 580, 590, and 720 (nm). The average of the errors for 450, 580, 590, and 720 (nm) is then obtained as the MAE in Fig. 1. More specifically, in Fig. 1, the ratio (%) of the MAE value to the range of possible intensity values ​​is expressed as the MAE.

[0033] The above calculations are performed for all conditions and all iterations to obtain the simulation results shown in Fig. 1. The simulation results shown in Fig. 1 reveal a new problem: when differences in bandwidth occur during band synthesis, the restoration error increases. In particular, the restoration error is large at max12 and max6. In other words, when multiple bands different from the band of interest are synthesized in large units, the restoration error of the band of interest increases.

[0034] For example, as the maximum number of synthesis bands increases, the maximum synthesis bandwidth also increases, and the difference between the bandwidth of the band of interest and the bandwidth of the synthesis band increases. This increases the difference between the element values ​​of the band of interest and the element values ​​of the synthesis band in the calculation matrix for generating multiple spectral images from a compressed image, which is expected to cause a difference in the influence of noise between them and increase the condition number. Therefore, it is expected that it will be difficult to converge on a solution through iterative calculations, and the restoration error will increase.

[0035] As described above, the processing load for restoring multiple spectral images corresponding to multiple bands is large, and band synthesis, which is used to reduce the processing load, may result in an inappropriate spectral image being obtained.

[0036] Therefore, the image processing method of Example 1 includes obtaining a compressed image in which spectral information corresponding to a plurality of wavelength bands is compressed; determining one or more wavelength band groups, each of which is composed of one or more wavelength bands to be combined from among the plurality of wavelength bands; determining, for each of the one or more wavelength band groups, one or more combinations, each of which is composed of one or more wavelength bands to be combined into one wavelength band from among the one or more wavelength bands that make up the wavelength band group, based on at least one of the number and total width of the one or more wavelength bands that make up the wavelength band group; and generating, based on the compressed image, a plurality of spectroscopic images corresponding to the plurality of wavelength bands combined based on the one or more combinations.

[0037] This makes it possible to determine an appropriate combination for synthesis depending on the number or total width of one or more wavelength bands included in each wavelength band group. Therefore, it becomes possible to synthesize multiple wavelength bands based on the appropriate combination to generate a spectral image. Therefore, it becomes possible to generate an appropriate spectral image while reducing the processing load.

[0038] Furthermore, the image processing method of Example 2 may be the image processing method of Example 1, in which the plurality of wavelength bands include one or more designated wavelength bands, and the one or more wavelength bands constituting each of the one or more wavelength band groups are one or more contiguous wavelength bands among the plurality of wavelength bands that do not include the one or more designated wavelength bands.

[0039] This makes it possible to suppress a decrease in wavelength resolution for a specified wavelength band, thereby enabling the generation of an appropriate spectral image corresponding to the specified wavelength band.

[0040] The image processing method of Example 3 may be the image processing method of Example 2, further comprising determining the one or more designated wavelength bands based on an input provided by a user or based on an object in the compressed image.

[0041] This makes it possible to appropriately determine the designated wavelength band, and therefore to appropriately generate a spectral image corresponding to the designated wavelength band to be analyzed.

[0042] Furthermore, the image processing method of Example 4 may be any of the image processing methods of Examples 1 to 3, and in determining the one or more combinations, the number of the one or more wavelength bands constituting each of the one or more combinations may be set to a threshold value or less.

[0043] This makes it possible to prevent the number of one or more wavelength bands constituting each combination from increasing, thereby preventing a decrease in wavelength resolution and an increase in reconstruction error, thereby enabling the generation of an appropriate spectral image.

[0044] Furthermore, the image processing method of Example 5 may be the image processing method of Example 4, in which the threshold value is set to be larger for each of the one or more wavelength band groups as the wavelength corresponding to the wavelength band group becomes shorter.

[0045] This allows the number of one or more wavelength bands constituting each combination to be reduced in the long wavelength region where the correlation between bands is high, and the number of one or more wavelength bands constituting each combination to be increased in the short wavelength region where the correlation between bands is low. Therefore, the randomness of the wavelength information is maintained, which may enable the generation of an appropriate spectral image. Furthermore, the influence of noise is suppressed, which may enable the efficient suppression of an increase in reconstruction error.

[0046] The image processing method of Example 6 may be the image processing method of Example 4 or 5, in which the threshold value is 4 or less.

[0047] This makes it possible to reduce the number of one or more wavelength bands constituting each combination to 4 or less. Therefore, it becomes possible to appropriately suppress a decrease in wavelength resolution and an increase in reconstruction error.

[0048] Furthermore, the image processing method of Example 7 may be any of the image processing methods of Examples 4 to 6, further including estimating a restoration error occurring in a spectroscopic image of a specified wavelength band among the plurality of spectroscopic images according to the number of the one or more wavelength bands constituting each of the one or more combinations, and determining the threshold according to the restoration error.

[0049] This makes it possible to appropriately determine the threshold for the number of one or more wavelength bands that make up each combination in accordance with the reconstruction error, thereby making it possible to appropriately suppress an increase in the reconstruction error.

[0050] Furthermore, the image processing method of Example 8 may be any of the image processing methods of Examples 1 to 3, and in determining the one or more combinations, the total width of the one or more wavelength bands that make up each of the one or more combinations may be set to be equal to or less than a threshold value.

[0051] This makes it possible to prevent the total width of one or more wavelength bands constituting each combination from becoming too large. This makes it possible to prevent a decrease in wavelength resolution and an increase in reconstruction error. This makes it possible to generate an appropriate spectral image.

[0052] Furthermore, the image processing method of Example 9 may be the image processing method of Example 8, in which the threshold value is set to be larger for each of the one or more wavelength band groups as the wavelength corresponding to the wavelength band group becomes shorter.

[0053] This makes it possible to reduce the total width of one or more wavelength bands constituting each combination in the long wavelength region where the correlation between bands is high, and to increase the total width of one or more wavelength bands constituting each combination in the short wavelength region where the correlation between bands is low. Therefore, the randomness of the information for wavelengths is maintained, which may make it possible to generate an appropriate spectral image. Furthermore, the influence of noise is suppressed, which may make it possible to efficiently suppress an increase in reconstruction error.

[0054] The image processing method of Example 10 may be the image processing method of Example 8 or 9, in which the threshold is 40 nm or less.

[0055] This allows the total width of one or more wavelength bands constituting each combination to be 40 nm or less, thereby making it possible to appropriately suppress a decrease in wavelength resolution and an increase in reconstruction error.

[0056] In addition, the image processing method of Example 11 may be any of the image processing methods of Examples 8 to 10, further including estimating a restoration error occurring in a spectroscopic image of a specified wavelength band among the plurality of spectroscopic images according to a total width of the one or more wavelength bands constituting each of the one or more combinations, and determining the threshold according to the restoration error.

[0057] This makes it possible to appropriately determine the threshold for the total width of one or more wavelength bands that make up each combination in accordance with the reconstruction error, thereby making it possible to appropriately suppress an increase in the reconstruction error.

[0058] Furthermore, the image processing method of Example 12 may be the image processing method of any one of Examples 1 to 11, wherein the compressed image is generated by photographing a subject with a camera including an optical element, and generating the plurality of spectral images includes: acquiring first encoded information, which is encoded information reflecting a spatial distribution of a transmission spectrum of the optical element and corresponds to the plurality of wavelength bands before synthesis based on the one or more combinations; generating, based on the first encoded information and the one or more combinations, second encoded information, which is encoded information reflecting a spatial distribution of the transmission spectrum of the optical element and corresponds to the plurality of wavelength bands after synthesis based on the one or more combinations; and generating the plurality of spectral images based on the compressed image and the second encoded information.

[0059] This makes it possible to generate second encoded information corresponding to multiple wavelength bands after combination from first encoded information corresponding to multiple wavelength bands before combination, thereby efficiently generating multiple spectral images corresponding to multiple wavelength bands after combination.

[0060] Furthermore, the image processing method of Example 13 may be the image processing method of Example 12, wherein the first encoded information includes a plurality of values ​​corresponding to the plurality of wavelength bands, and generating the second encoded information includes, for each of the one or more combinations, calculating at least one of a sum and an average of one or more values, from the plurality of values ​​included in the first encoded information, that correspond to the one or more wavelength bands that constitute the combination.

[0061] This makes it possible to appropriately combine one or more values ​​corresponding to one or more wavelength bands constituting one combination into one value corresponding to the one combination, and therefore makes it possible to appropriately generate second encoded information corresponding to multiple wavelength bands after combination from first encoded information corresponding to multiple wavelength bands before combination.

[0062] Furthermore, the image processing method of Example 14 may be an image processing method including: acquiring a compressed image in which spectral information corresponding to a plurality of wavelength bands is compressed; acquiring time limit information indicating a limit on a processing time for generating a plurality of spectroscopic images based on the compressed image; determining one or more wavelength band groups, each of which is composed of one or more wavelength bands to be combined, from among the plurality of wavelength bands; determining, for each of the one or more wavelength band groups, based on the time limit information, one or more combinations, each of which is composed of one or more wavelength bands to be combined into one wavelength band, from among the one or more wavelength bands that make up the wavelength band group; and generating, based on the compressed image, the plurality of spectroscopic images corresponding to the plurality of wavelength bands combined based on the one or more combinations.

[0063] This makes it possible to determine an appropriate combination for synthesis for each of one or more wavelength band groups, taking into account processing time limitations. Therefore, it becomes possible to synthesize multiple wavelength bands based on the appropriate combination to generate a spectral image. Therefore, it becomes possible to generate an appropriate spectral image while reducing the processing load.

[0064] Moreover, the image processing system of Example 15 includes an image sensor that acquires a compressed image in which spectral information corresponding to a plurality of wavelength bands is compressed, and a processing circuit, wherein the processing circuit determines one or more wavelength band groups, each composed of one or more wavelength bands to be combined, from among the plurality of wavelength bands, and for each of the one or more wavelength band groups, determines one or more combinations, each composed of one or more wavelength bands to be combined into one wavelength band, from among the one or more wavelength bands that make up the wavelength band group, based on at least one of the number and total width of the one or more wavelength bands that make up the wavelength band group, and generates, based on the compressed image, a plurality of spectroscopic images corresponding to the plurality of wavelength bands combined based on the one or more combinations.

[0065] This makes it possible to determine an appropriate combination for synthesis depending on the number or total width of one or more wavelength bands included in each wavelength band group. Therefore, it becomes possible to synthesize multiple wavelength bands based on the appropriate combination to generate a spectral image. Therefore, it becomes possible to generate an appropriate spectral image while reducing the processing load.

[0066] Furthermore, the image processing method of Example 16 may be an image processing method that includes determining one or more wavelength band groups, each of which is composed of one or more wavelength bands to be combined, from among a plurality of wavelength bands, and determining, for each of the one or more wavelength band groups, one or more combinations, each of which is composed of one or more wavelength bands to be combined into one wavelength band, from among the one or more wavelength bands that make up the wavelength band group, based on at least one of the number and total width of the one or more wavelength bands that make up the wavelength band group.

[0067] This makes it possible to determine an appropriate combination for synthesis depending on the number or total width of one or more wavelength bands included in each wavelength band group. Therefore, it becomes possible to control synthesis of multiple wavelength bands and generation of a spectral image based on the appropriate combination. This makes it possible to support a reduction in processing load and generation of an appropriate spectral image. This makes it possible to generate an appropriate spectral image while reducing the processing load.

[0068] Furthermore, these comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0069] Hereinafter, embodiments will be described with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, the arrangement and connection of the components, steps, the order of steps, and the like shown in the following embodiments are merely examples and are not intended to limit the scope of the claims.

[0070] 2 is a block diagram showing an example of the configuration of an image processing system according to an embodiment. As shown in Fig. 2, the image processing system 100 includes an image sensor 111 and a processing circuit 121. The image processing system 100 may further include a display device 130.

[0071] The image processing system 100 may also include an imaging device 110 and an image processing device 120. The imaging device 110 may also include an image sensor 111. The image processing device 120 may also include a processing circuit 121. Each of the imaging device 110 and the image processing device 120 may also include a memory and a user interface. The display device 130 may also constitute the user interface of the image processing device 120.

[0072] The image sensor 111 acquires an image by detecting an optical signal for each pixel and generating an image represented by a plurality of optical signals corresponding to a plurality of pixels. Here, the image sensor 111 acquires a compressed image. Specifically, for example, the image sensor 111 acquires a compressed image in which information of four or more wavelength bands is superimposed for each pixel based on a filter array described below. Note that a wavelength band may be simply referred to as a band.

[0073] The image sensor 111 may be a monochrome photodetector having a plurality of photodetection elements arranged in a matrix. More specifically, the image sensor 111 may be a charge-coupled device (CCD) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, an infrared array image sensor, a terahertz array image sensor, or a millimeter wave array image sensor.

[0074] Alternatively, the image sensor 111 may be a color-type photodetector. The wavelength range detectable by the image sensor 111 is not limited, and may be, for example, visible light, ultraviolet light, infrared light, terahertz waves, or any combination thereof.

[0075] The processing circuitry 121 is a circuit that performs information processing. For example, the processing circuitry 121 generates a hyperspectral image represented by information of four or more wavelength bands by performing a restoration operation on a compressed image. For example, the hyperspectral image is composed of four or more spectral images corresponding to four or more wavelength bands. The restoration operation may be the same as the restoration operation described in Patent Documents 1 and 2. Specifically, four or more spectral images may be generated as the hyperspectral image based on the following equation (1):

[0076]

[0077] Here, g is data representing a compressed image and is represented, for example, by a one-dimensional array (i.e., a vector). If the compressed image is an image of n×m pixels, data g is represented by a one-dimensional array having n×m elements. f is data representing w spectral images corresponding one-to-one to w wavelength bands and is represented, for example, by a one-dimensional array. When w spectral images constitute a hyperspectral image, w is an integer greater than or equal to 4.

[0078] Also, f １ , f ２ , ..., f ｗ are the wavelength bands W １ Spectral image data corresponding to the wavelength band W ２ Spectral image data corresponding to the wavelength band W ｗ The spectral image data corresponds to the

[0079] Data f １ , f ２ , ..., f ｗ Each of the spectral images is represented by, for example, a one-dimensional array. If each spectral image is an image of n×m pixels, the data f １ , f ２ , ..., f ｗ are represented by a one-dimensional array having n×m elements, and data f is represented by a one-dimensional array having n×m×w elements. H is a matrix with n×m rows and n×m×w columns, called the system matrix, and corresponds to the mask data.

[0080] The matrix H is the wavelength band W of the filter array described below. １ Transmission spectrum of wavelength band W ２ Transmission spectrum of wavelength band W ｗ The transmittance may be determined based on the transmittance spectrum of the light emitting element.

[0081] The data f that satisfies equation (1) can be estimated using a compressed sensing technique, specifically, by equation (2).

[0082]

[0083] Equation (2) expresses finding data f that minimizes the sum of the first and second terms in the parentheses. Data f can be calculated as final calculation result data by converging the calculation result data through recursive iterative calculation.

[0084] The first term in the parentheses in equation (2) represents the sum of squares of the difference between data g and data Hf obtained by transforming data f in the estimation process using matrix H, and is a so-called residual term. Although the sum of squares is used here, the sum of absolute values, the square root of the sum of squares, or the like may be used instead of the sum of squares.

[0085] The sum of squares of the difference between data Hf and data g is (g １ -r １ ) × (g １ -r １ ) + ... + (g ｎ×ｍ -r ｎ×ｍ ) × (g ｎ×ｍ -r ｎ×ｍ ) where g １ , ..., g ｎ×ｍ is g = (g １ ...g ｎ×ｍ ) Ｔ is an element of data g expressed as １ , ..., r ｎ×ｍ is Hf = (r １ ...r ｎ×ｍ ) Ｔ The elements of data Hf are expressed as follows:

[0086] The second term in the parentheses in Equation (2) is a regularization term, sometimes called a stabilization term. Φ(f) represents a constraint on the regularization of f and is a function that reflects the sparse information of the data f. This function has the effect of smoothing or stabilizing the data f. Φ(f) can be expressed, for example, by a discrete cosine transform (DCT), a wavelet transform, a Fourier transform, a total variation (TV), or any combination thereof.

[0087] τ is a weighting coefficient for the regularization term, and corresponds to the influence of regularization in the reconstruction calculation. The larger the value of τ, the greater the influence of regularization, and the stronger the convergence of the solution in the iterative calculation. Conversely, the smaller the value of τ, the less the influence of regularization, and the weaker the convergence of the solution in the iterative calculation.

[0088] Note that the image encoded by an optical element such as a filter array may be captured on the image sensor in a blurred state, for example, a point image corresponding to one pixel of the compressed image may be blurred and spread to a region of pixels surrounding that pixel.

[0089] The display device 130 is a display device for displaying information. A compressed image, a monochrome image, an RGB image, a hyperspectral image, or the like is displayed on the display device 130 by the processing circuit 121. The display device 130 may be, for example, a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display.

[0090] A graphical user interface (GUI) may be displayed on the display device 130. The display device 130 may also be a touch panel. A user may input information to the display device 130. The processing circuitry 121 may acquire information from the user via the display device 130. That is, the display device 130 may be an input / output device. Alternatively, the processing circuitry 121 may acquire information from the user via an input device different from the display device 130.

[0091] 2 shows an example of the configuration of the image processing system 100, but the configuration of the image processing system 100 is not limited to the example of the configuration in FIG. 2. For example, multiple devices may be integrated into one device, or one device may be distributed among multiple devices. Furthermore, multiple distributed devices may be able to communicate with each other via wired or wireless communication.

[0092] The processing circuitry 121 corresponds to an acquirer that acquires an image from the image sensor 111, a combiner that performs band composition, a generator that generates an image, and a display controller that displays the image on the display device 130. Instead of the processing circuitry 121, the image processing device 120 may include some or all of these components.

[0093] Fig. 3 is a conceptual diagram showing an example of the configuration of the imaging device 110 shown in Fig. 2. The imaging device 110 may have a configuration similar to that of the imaging devices disclosed in Patent Documents 1 and 2. For example, the imaging device 110 includes an image sensor 111, a filter array 112, and an optical system 113.

[0094] The filter array 112 is disposed on the optical path of light incident from the subject, and is disposed between the optical system 113 and the image sensor 111. The filter array 112 functions as the encoding element described in Patent Documents 1 and 2. The filter array 112 may be integrated with the image sensor 111.

[0095] The arrangement of the filter array 112 is not limited to the arrangement shown in Fig. 3. For example, the filter array 112 may be arranged between the optical system 113 and the image sensor 111, but away from the image sensor 111. Alternatively, for example, the filter array 112 may be arranged between the subject and the optical system 113. Alternatively, for example, the filter array 112 may be arranged within the optical system 113.

[0096] The optical system 113 is disposed on the optical path of light incident from the subject, and is disposed between the subject and the filter array 112. The optical system 113 includes at least one lens, and can form an image of the subject on the imaging surface of the image sensor 111 via the filter array 112.

[0097] The configuration and arrangement of the optical system 113 are not limited to those shown in Fig. 3. For example, the optical system 113 may be arranged between the filter array 112 and the image sensor 111. Furthermore, for example, the optical system 113 may include a plurality of lenses arranged on the optical path. In this case, the filter array 112 may be arranged between adjacent lenses of the plurality of lenses.

[0098] The filter array 112 and the optical system 113 are examples of optical elements. The image sensor 111 may detect the optical signal through an optical element different from the filter array 112 and the optical system 113. The optical element may be an object expressed as an optical member, an optical mechanism, an optical device, or the like.

[0099] 4 is a conceptual diagram showing an example of the configuration of the filter array 112 shown in FIG. 3. The filter array 112 is made up of a plurality of filters F arranged in a matrix. １１ , ..., F ｎｍ In the example shown in FIG. 4, the filter array 112 includes 48 filters F １１ , ..., F ｎｍ The filter F １１ is 48 filters F １１ , ..., F ｎｍ The filter F is located at the top left of the ｎｍ is 48 filters F １１ , ..., F ｎｍ This is the filter located at the bottom right of the list.

[0100] The filter F included in the filter array 112 １１ , ..., F ｎｍ The number of filters F included in the filter array 112 is not limited to 48. １１ , ..., F ｎｍ The number may be approximately the same as the number of pixels of the image sensor 111, and may be determined depending on the application within the range of, for example, several tens to several tens of millions.

[0101] For example, the filter array 112 may include n×m filters F corresponding to n×m pixels. １１ , ..., F ｎｍ Waveband W １ , ..., W ｗ In this case, the filter F １１ , ..., F ｎｍ are the transmission spectrum S １１ , ..., S ｎｍ The transmission spectrum S １１ , ..., S ｎｍAlternatively, the transmission spectrum S １１ , ..., S ｎｍ may be the same. Here, the transmission spectrum may refer to the light transmittance spectrum.

[0102] n×m filters F １１ , ..., F ｎｍ is w wavelength bands W １ , ..., W ｗ For n×m×w transmittances S １１Ｗ１ , ..., S ｎｍＷｗ The transmittance S １１Ｗ１ , ..., S ｎｍＷｗ Alternatively, the transmittance S １１Ｗ１ , ..., S ｎｍＷｗ Here, transmittance may refer to light transmittance.

[0103] Wavelength band W α The transmittance of the filter β in the above formula may be expressed by the following formula (3):

[0104]

[0105] where Wαmin is the wavelength band W α is the minimum wavelength value of the wavelength band W α is the maximum wavelength value of h(λ), and h(λ) is a function indicating the transmission spectrum, where λ is the wavelength.

[0106] In addition, the wavelength band W α The transmittance of the filter β in the wavelength band W is not limited to the formula (3). α The transmittance of the filter β in the wavelength band W may be the transmittance obtained by dividing the formula (3) by (Wαmax-Wαmin). α The transmittance of the filter β in the wavelength band W α The wavelength λ that represents α０ Transmittance h (λ α０ ) may also be used.

[0107] Here, the wavelength λ α０ is Wαmin≦λ α０For example, the wavelength λ α０ is the wavelength band W α The center wavelength may be ((Wαmax−Wαmin) / 2).

[0108] FIG. 5 shows the filter F shown in FIG. １１ 6 shows an example of the transmission spectrum of the filter F shown in FIG. ｎｍ 7 is an example of the transmission spectrum of the wavelength band W of the filter array 112 shown in FIG. １ 8 is a diagram showing an example of the transmittance of the wavelength band W of the filter array 112 shown in FIG. ２ 7 and 8, the shading of each region represents the transmittance of the filter, with lighter regions representing higher transmittance and darker regions representing lower transmittance.

[0109] A plurality of filters F included in the filter array 112 １１ , ..., F ｎｍ The wavelength dependence of the transmittance of each filter is different. １１ In the wavelength band W １ The transmittance of the wavelength band W ２ On the other hand, the transmittance of the filter F ｎｍ In the wavelength band W １ The transmittance of the wavelength band W ２ The transmittance of the filter F is approximately the same as that of the filter F. １１ The wavelength dependence of the transmittance of the filter F ｎｍ The wavelength dependence of the transmittance is different from that of the

[0110] Here, w wavelength bands W １ , ..., W ｗ Two of the wavelength bands W １ and W ２ The transmittance of the w wavelength bands W １ , ..., W ｗ The transmittance of other wavelength bands will not be shown or explained.

[0111] For example, the mask data may include w wavelength bands W １ , W２ , ..., W ｗ Specifically, the mask data is expressed as w matrices corresponding to w wavelength bands W １ , W ２ , ..., W ｗ is expressed by a matrix of transmittances (transmittance matrix) with n rows and m columns corresponding to pixels with n rows and m columns. Then, by changing the representation format from the mask data expressed by w matrices each having n rows and m columns, a single matrix H with n×m rows and n×m×w columns is obtained.

[0112] Also, here, for example, the image sensor 111 acquires the compressed image through a plurality of light-receiving regions having a plurality of transmission spectra. Specifically, the image sensor 111 acquires the compressed image through the filter array 112. The plurality of light-receiving regions may correspond to a plurality of filters included in the filter array 112, respectively.

[0113] Alternatively, the metalens described in Non-Patent Document 1 may be used instead of the filter array 112. In this case, the wavelength transmittance varies depending on the location. The light receiving region corresponds to such a location. Alternatively, the CMOS image sensor described in Non-Patent Document 2 may be used instead of the filter array 112. In this case, the sensing region is processed so that a predetermined transmittance is obtained for each pixel. The light receiving region corresponds to such a sensing region.

[0114] The light receiving region may be expressed as an optical element, a modulation element, an encoding element, an optical processing region, a masking region, etc. Furthermore, for example, the optical element, the modulation element, or the encoding element may correspond to a plurality of light receiving regions.

[0115] 9 is a flowchart showing an example of the operation of the image processing system 100 shown in Fig. 2. In this example, the image sensor 111 of the imaging device 110 acquires a compressed image by generating a compressed image, and the processing circuit 121 of the image processing device 120 acquires the compressed image from the image sensor 111 (S101). The operation of acquiring the compressed image may correspond to the operation of the image sensor 111 acquiring the compressed image by generating the compressed image, or may correspond to the operation of the processing circuit 121 acquiring the compressed image from the image sensor 111.

[0116] The compressed image includes information corresponding to multiple wavelength bands within the wavelength range of the imaged object. Specifically, in the compressed image, spectral information corresponding to the multiple wavelength bands is compressed for each pixel. The compressed image may include information corresponding to four or more wavelength bands. The multiple wavelength bands of the compressed image may be multiple contiguous wavelength bands within the wavelength range of the imaged object. Furthermore, the multiple wavelength bands of the compressed image may have the same bandwidth, and each wavelength band may also be referred to as a unit wavelength band.

[0117] Then, the processing circuitry 121 of the image processing device 120 determines one or more wavelength band groups, each of which is composed of one or more wavelength bands to be combined, from among the multiple wavelength bands (S102). For example, the processing circuitry 121 determines multiple wavelength bands to be combined from among multiple wavelength bands in the wavelength range of the image capture target, and determines each wavelength band group composed of one or more of the multiple wavelength bands to be combined. The wavelength band groups may be composed of one or more consecutive wavelength bands to be combined.

[0118] A wavelength band group may be expressed as a wavelength band set. One or more wavelength bands to be combined that are included in a wavelength band group are not limited to being combined into one wavelength band. N wavelength bands to be combined that are included in a wavelength band group may be combined into M wavelength bands. Here, for example, N and M are natural numbers that satisfy N≧M≧1. Note that when N=M, the N wavelength bands are maintained as they are.

[0119] Next, the processing circuitry 121 of the image processing device 120 determines one or more combinations for each wavelength band group based on at least one of the number and total width of the one or more wavelength bands that make up the wavelength band group (S103). Each combination is a combination of one or more wavelength bands, and is composed of one or more wavelength bands that are combined into a single wavelength band from among the one or more wavelength bands that make up the wavelength band group. For example, the processing circuitry 121 groups the one or more wavelength bands that make up the wavelength band group into one or more combinations.

[0120] Specifically, for each wavelength band group, the processing circuitry 121 may increase the number of one or more combinations the greater the number or total width of one or more wavelength bands constituting the wavelength band group. Furthermore, when the number or total width of one or more wavelength bands in a wavelength band group is equal to or less than a threshold, the processing circuitry 121 may determine a combination including all wavelength bands in the wavelength band group as the combination of one or more wavelength bands to be combined into one wavelength band.

[0121] Furthermore, the processing circuitry 121 may determine, for each wavelength band group, at least one of the number and total width of one or more wavelength bands in each combination in accordance with at least one of the number and total width of one or more wavelength bands constituting the wavelength band group.The processing circuitry 121 may then determine one or more wavelength bands in each combination based on the determination result of at least one of the number and total width of one or more wavelength bands in each combination.

[0122] When one band is combined with another band, the band is maintained as is. In other words, combining one band with another band corresponds to maintaining the band as is. Furthermore, a combination of one or more wavelength bands may be expressed as a wavelength band set.

[0123] Then, the processing circuitry 121 generates, based on the compressed image, a plurality of spectral images corresponding to the plurality of wavelength bands synthesized based on one or more combinations of the wavelength band groups (S104). That is, the processing circuitry 121 synthesizes a plurality of wavelength bands based on one or more combinations of the wavelength band groups, and generates, based on the compressed image, a plurality of spectral images corresponding to the plurality of synthesized wavelength bands. For example, a restoration method for compressed sensing may be used to generate the spectral images.

[0124] This makes it possible to determine an appropriate combination for synthesis depending on the number or total width of one or more wavelength bands included in each wavelength band group. Therefore, it becomes possible to synthesize multiple wavelength bands based on the appropriate combination to generate a spectral image. Therefore, it becomes possible to generate an appropriate spectral image while reducing the processing load.

[0125] For example, one or more designated wavelength bands may be included in multiple wavelength bands. Each wavelength band group may be composed of one or more consecutive wavelength bands that do not include any of the one or more designated wavelength bands among the multiple wavelength bands. This makes it possible to suppress a decrease in wavelength resolution for each designated wavelength band. Therefore, it becomes possible to generate an appropriate spectral image corresponding to each designated wavelength band.

[0126] Furthermore, one or more designated wavelength bands may be determined based on, for example, input by a user or based on the object in the compressed image, thereby enabling each designated wavelength band to be appropriately determined and thus enabling appropriate generation of a spectral image corresponding to each designated wavelength band to be analyzed.

[0127] Furthermore, for example, the processing circuitry 121 may set the number of one or more wavelength bands constituting each combination to be equal to or less than a threshold. Specifically, the processing circuitry 121 may determine one or more combinations for setting the number of one or more wavelength bands constituting each combination to be equal to or less than a threshold, based on the number of one or more wavelength bands constituting the wavelength band group.

[0128] This makes it possible to prevent the number of one or more wavelength bands constituting each combination from increasing, thereby preventing a decrease in wavelength resolution and an increase in reconstruction error, thereby enabling the generation of an appropriate spectral image.

[0129] Furthermore, for example, the threshold value may be set to be larger for each wavelength band group as the wavelength corresponding to that wavelength band group becomes shorter. Here, the wavelength corresponding to the wavelength band group may be the center wavelength of the wavelength band group, the maximum wavelength of the wavelength band group, or the minimum wavelength of the wavelength band group.

[0130] This makes it possible to reduce the number of one or more wavelength bands constituting each combination in the long wavelength region where the correlation between bands is high, and to increase the number of one or more wavelength bands constituting each combination in the short wavelength region where the correlation between bands is low. Therefore, the randomness of the information for wavelengths is maintained, and it may be possible to generate an appropriate spectroscopic image. Furthermore, it may be possible to suppress the influence of noise in the reconstruction calculation, and efficiently suppress an increase in reconstruction error.

[0131] Furthermore, for example, the threshold value may be equal to or less than 4. This makes it possible to set the number of one or more wavelength bands constituting each combination to 4 or less. Therefore, it becomes possible to appropriately suppress a decrease in wavelength resolution and an increase in restoration error.

[0132] Furthermore, for example, a reconstruction error occurring in a spectroscopic image of a designated wavelength band may be estimated according to the number of one or more wavelength bands constituting each combination. Then, a threshold may be determined according to the reconstruction error. The processing circuitry 121 may estimate the reconstruction error and determine the threshold, or another device or component may estimate the reconstruction error and determine the threshold.

[0133] This makes it possible to appropriately determine the threshold for the number of one or more wavelength bands that make up each combination in accordance with the reconstruction error, thereby making it possible to appropriately suppress an increase in the reconstruction error.

[0134] Furthermore, for example, the processing circuitry 121 may set the total width of one or more wavelength bands constituting each combination equal to or less than a threshold value. Specifically, the processing circuitry 121 may determine one or more combinations for setting the total width of one or more wavelength bands constituting each combination equal to or less than a threshold value, based on the total width of one or more wavelength bands constituting the wavelength band group.

[0135] This makes it possible to prevent the total width of one or more wavelength bands constituting each combination from becoming too large. This makes it possible to prevent a decrease in wavelength resolution and an increase in reconstruction error. This makes it possible to generate an appropriate spectral image.

[0136] Furthermore, for example, the threshold value may be set to be larger for each wavelength band group as the wavelength corresponding to that wavelength band group becomes shorter. Here, the wavelength corresponding to the wavelength band group may be the center wavelength of the wavelength band group, the maximum wavelength of the wavelength band group, or the minimum wavelength of the wavelength band group.

[0137] This makes it possible to reduce the total width of one or more wavelength bands constituting each combination in the long wavelength region where the correlation between bands is high, and to increase the total width of one or more wavelength bands constituting each combination in the short wavelength region where the correlation between bands is low. Therefore, the randomness of the information for wavelengths is maintained, which may make it possible to generate an appropriate spectroscopic image. Furthermore, the influence of noise in the reconstruction calculation may be suppressed, which may make it possible to efficiently suppress an increase in reconstruction error.

[0138] Furthermore, for example, the threshold value may be 40 nm or less. This allows the total width of one or more wavelength bands constituting each combination to be 40 nm or less. Therefore, it is possible to appropriately suppress a decrease in wavelength resolution and an increase in restoration error.

[0139] Furthermore, for example, a restoration error occurring in a spectroscopic image of a designated wavelength band may be estimated according to the total width of one or more wavelength bands constituting each combination. Then, a threshold may be determined according to the restoration error. The processing circuitry 121 may estimate the restoration error and determine the threshold, or another device or component may estimate the restoration error and determine the threshold.

[0140] This makes it possible to appropriately determine the threshold for the total width of one or more wavelength bands that make up each combination in accordance with the reconstruction error, thereby making it possible to appropriately suppress an increase in the reconstruction error.

[0141] Furthermore, for example, the compressed image may be generated by capturing an image of a subject with a camera including an optical element. Furthermore, the processing circuitry 121 may acquire first encoded information. Here, the first encoded information is encoded information that reflects the spatial distribution of the transmission spectrum of the optical element and corresponds to multiple wavelength bands before synthesis based on one or more combinations.

[0142] The processing circuitry 121 may then generate second encoded information based on the first encoded information and one or more combinations within each wavelength band group. Here, the second encoded information is encoded information that reflects the spatial distribution of the transmission spectrum of the optical element and corresponds to multiple wavelength bands after synthesis based on one or more combinations within each wavelength band group. The processing circuitry 121 may then generate multiple spectral images based on the compressed image and the second encoded information.

[0143] This makes it possible to generate second encoded information corresponding to multiple wavelength bands after combination from first encoded information corresponding to multiple wavelength bands before combination, thereby efficiently generating multiple spectral images corresponding to multiple wavelength bands after combination.

[0144] Note that the coding information such as the first coding information and the second coding information may correspond to mask data, a system matrix (H), etc. The camera may correspond to the imaging device 110. The optical elements may correspond to the filter array 112, the optical system 113, etc. The image sensor 111 in the imaging device 110, which is a camera, may capture an image of a subject via the filter array 112, the optical system 113, etc., which are optical elements in the camera, to generate a compressed image.

[0145] Furthermore, for example, the first encoded information may include a plurality of values ​​corresponding to a plurality of wavelength bands. In generating the second encoded information, the processing circuitry 121 may calculate, for each combination, at least one of the sum and the average of one or more values, among the plurality of values ​​included in the first encoded information, that correspond to one or more wavelength bands constituting the combination.

[0146] This makes it possible to appropriately combine one or more values ​​corresponding to one or more wavelength bands constituting one combination into one value corresponding to the one combination, and therefore makes it possible to appropriately generate second encoded information corresponding to multiple wavelength bands after combination from first encoded information corresponding to multiple wavelength bands before combination.

[0147] Furthermore, for example, the processing circuitry 121 may acquire time limit information indicating a processing time limit for generating multiple spectral images based on the compressed image, and the processing circuitry 121 may determine one or more combinations for each wavelength band group based on the time limit information.

[0148] That is, the processing circuitry 121 may determine one or more combinations each made up of one or more wavelength bands to be combined into one wavelength band based on the time limit information, instead of at least one of the number and total width of one or more wavelength bands that make up the wavelength band group. Alternatively, the processing circuitry 121 may determine one or more combinations each made up of one or more wavelength bands to be combined into one wavelength band based on the time limit information, in addition to at least one of the number and total width of one or more wavelength bands that make up the wavelength band group.

[0149] This makes it possible to determine an appropriate combination for synthesis for each of one or more wavelength band groups, taking into account processing time limitations. Therefore, it becomes possible to synthesize multiple wavelength bands based on the appropriate combination to generate a spectral image. Therefore, it becomes possible to generate an appropriate spectral image while reducing the processing load.

[0150] Furthermore, for example, the acquisition of the compressed image (S101) and the generation of the spectral image (S104) may be omitted, or the acquisition of the compressed image (S101) and the generation of the spectral image (S104) may be performed by another device or another system.

[0151] Even in this embodiment, it is possible to determine an appropriate combination for synthesis depending on the number or total width of one or more wavelength bands included in each wavelength band group. Therefore, it is possible to appropriately control the synthesis of multiple wavelength bands and the generation of a spectral image based on the appropriate combination. This makes it possible to reduce the processing load and support the generation of an appropriate spectral image. This makes it possible to generate an appropriate spectral image while reducing the processing load.

[0152] FIG. 10 is a flowchart showing a first specific example of the operation of the image processing system 100 shown in FIG.

[0153] In this example, the processing circuitry 121 acquires a compressed image (S201). Specifically, the image sensor 111 acquires the compressed image by generating a compressed image, and the processing circuitry 121 acquires the compressed image from the image sensor 111. The processing circuitry 121 also acquires mask data (S202). For example, the processing circuitry 121 acquires the mask data from a memory in which the mask data is stored in advance.

[0154] The processing circuitry 121 also acquires the maximum number of combined bands (S203). Here, the maximum number of combined bands refers to a threshold value for the number of one or more bands that can be combined into one band, and refers to the upper limit of the number of one or more bands that can be combined into one band. As described above, when one band is combined, the band is maintained as is. In other words, combining one band corresponds to maintaining the band as is.

[0155] For example, the processing circuitry 121 may obtain the maximum number of synthesis bands based on an input made by a user via a user interface (UI). In this case, it is possible to set an optimal maximum number of synthesis bands based on the user's knowledge, specifically, past restoration results, etc.

[0156] Alternatively, the processing circuitry 121 may obtain the maximum number of synthesis bands from a memory that stores the maximum number of synthesis bands in advance. The maximum number of synthesis bands may be determined based on a restoration error estimated by simulation and stored in the memory.

[0157] Specifically, in the example of Fig. 1, the restoration errors for max2, max3, and max4 are almost the same as those without synthesis. On the other hand, the restoration errors increase significantly for max6 and max12. Therefore, the maximum number of synthesis bands may be determined to be 4 and stored in advance in memory. This can suppress the increase in restoration error.

[0158] However, the maximum number of synthesis bands may be greater than or less than 4. Specifically, the maximum number of synthesis bands may be 8, 6, or 2.

[0159] Furthermore, for example, the maximum number of composite bands may be determined based on values ​​set for each pixel and each band in the mask data. Here, the values ​​set for each pixel and each band in the mask data may be determined by calibration. If the transmittance distribution of one band in the mask data is not sufficiently different from the transmittance distribution of other bands, it will be difficult to restore the information of those bands. Therefore, the maximum number of composite bands may be determined based on the randomness of the values ​​in the mask data.

[0160] The processing circuitry 121 also acquires non-synthesis band information indicating non-synthesis bands (S204). The non-synthesis bands are bands different from the bands to be synthesized and are not synthesized with other bands. The non-synthesis bands may be attention bands that are bands that are focused on by the user. The non-synthesis bands may be designated bands that are designated by the user or in accordance with the subject. In the present disclosure, the non-synthesis bands, attention bands, and designated bands may be interchangeable.

[0161] Specifically, the processing circuitry 121 may acquire non-synthesis band information input by a user to a user interface. For example, a band of interest may be selected by the user in the user interface. This determines the non-synthesis band.

[0162] The non-synthesis band may be determined according to the subject. Specifically, the non-synthesis band information may be associated with information about the subject and stored in a memory. Then, the non-synthesis band information associated with the information about the subject may be acquired by inputting the information about the subject to a user interface by a user.

[0163] Next, the processing circuitry 121 converts the mask data (S205). Specifically, the processing circuitry 121 may determine, based on the non-synthesis band information, a band different from the non-synthesis band as a band to be synthesized. Then, the processing circuitry 121 may determine a combination of one or more bands to be synthesized into one band from the one or more bands to be synthesized. Here, the processing circuitry 121 may determine one or more combinations such that the number of one or more bands included in each combination is equal to or less than the maximum number of synthesis bands.

[0164] The processing circuitry 121 may then convert the mask data based on one or more combinations. For example, when combining multiple bands into one band, the processing circuitry 121 integrates multiple values ​​corresponding to the multiple bands in the mask data into a single value corresponding to the single band. In other words, combining multiple bands into a single band may correspond to integrating multiple values ​​corresponding to the multiple bands into a single value. The integration of the multiple values ​​may be performed by averaging the multiple values ​​or by adding (summing) the multiple values.

[0165] Next, the processing circuitry 121 performs a restoration calculation process on the compressed image using the converted mask data (S206). Specifically, the processing circuitry 121 performs a restoration calculation process corresponding to the above-mentioned equation (2) using the converted mask data. This results in spectral information such as a plurality of spectroscopic images (spectral images) corresponding to the plurality of bands synthesized for each combination.

[0166] Then, the processing circuitry 121 outputs the data obtained by the restoration calculation process (S207). For example, the processing circuitry 121 may display the multiple spectroscopic images on the display device 130 or may store them in memory. The processing circuitry 121 may display the spectral information obtained from the multiple spectroscopic images on the display device 130 or may store them in memory.

[0167] Specifically, the following conversion method may be used in the conversion of the mask data (S205): For example, the mask data is expressed as M(u, v, ch), which indicates the transmittance of each pair of wavelength band and pixel, where ch indicates the wavelength band index and (u, v) indicates the pixel.

[0168] When N wavelength bands from wavelength band index CHS to wavelength band index CHE are combined by averaging, the combined mask data is expressed by the following equation (4).

[0169]

[0170] Furthermore, when N wavelength bands from wavelength band index CHS to wavelength band index CHE are synthesized by addition, the synthesized mask data is expressed by the following equation (5).

[0171]

[0172] 11 is a graph showing an example of a correct spectrum reconstructed using the original mask data. In this example, the luminance values ​​of each band are reconstructed using the original mask data without band synthesis. Specifically, the luminance value of each band is 1000.

[0173] 12 is a graph showing an example of a ground truth spectrum restored using mask data converted by averaging. In this example, the luminance values ​​of each band are restored using mask data obtained by averaging multiple values ​​corresponding to multiple bands. Specifically, the 540 nm band and the 560 nm band are combined, and the 640 nm band and the 660 nm band are combined.

[0174] By combining two bands into one band, the bandwidth is doubled. Therefore, the brightness values ​​in the band also double. Since the mask data is converted by averaging, the doubled brightness values ​​are restored as is, as in the example of FIG. 12. Therefore, the brightness in the band may increase, and a spectrum different from the expected one may be restored.

[0175] On the other hand, when the average is used, the difference between the multiple values ​​corresponding to the multiple bands in the mask data becomes relatively small, and it is expected that the solution in the reconstruction calculation process will be stable, that is, the solution will be more likely to converge, and the reconstruction error will be smaller.

[0176] 13 is a graph showing an example of a ground truth spectrum restored using mask data converted by addition. In this example, the luminance values ​​of each band are restored using mask data obtained by combining multiple values ​​corresponding to multiple bands by addition. Specifically, in this example, the 540 nm band and the 560 nm band are combined, and the 640 nm band and the 660 nm band are combined.

[0177] By combining two bands into one band, the bandwidth is doubled. Therefore, the brightness values ​​in the band are also doubled. Since the mask data is converted by addition, the doubled brightness values ​​are restored by halving them, as in the example of FIG. 13 . In other words, when addition is used, the dynamic range of all spectra is constant, regardless of the bandwidth. Therefore, the brightness of the combined band may not increase, and the expected spectrum may be restored.

[0178] On the other hand, when addition is used, the difference between the values ​​corresponding to the bands in the mask data becomes relatively large, and the solution in the reconstruction calculation process is expected to become unstable, making it difficult for the solution to converge and increasing the reconstruction error.

[0179] Fig. 14 is a flowchart showing the band synthesis process performed by the image processing system 100 shown in Fig. 2. The band synthesis process is performed, for example, in the mask data conversion process (S205) shown in Fig. 10.

[0180] First, the processing circuitry 121 determines whether a non-synthesis band exists (S301). For example, the processing circuitry 121 determines whether a non-synthesis band exists based on non-synthesis band information acquired in advance. The processing circuitry 121 may determine that a non-synthesis band exists when a non-synthesis band is specified, or may determine that a non-synthesis band does not exist when a non-synthesis band is not specified.

[0181] If it is determined that there is no non-combination band (No in S301), the processing circuitry 121 performs composition for each combination of bands that is equal to or less than the maximum number of combination bands for the band group that includes all bands (S302). For example, the processing circuitry 121 determines each combination that includes bands that are equal to or less than the maximum number of combination bands, and combines one or more bands included in each combination into one band.

[0182] If it is determined that a non-synthesis band exists (Yes in S301), the processing circuitry 121 performs synthesis for each band group that does not include a non-synthesis band (S303). For example, the processing circuitry 121 determines each band group that does not include a non-synthesis band and includes one or more consecutive bands, and synthesizes one or more bands included in each band group into a single band.

[0183] Next, the processing circuitry 121 determines whether the number of bands in each band group is equal to or less than the maximum number of bands to be synthesized (S304). If the number of bands in each band group is equal to or less than the maximum number of bands to be synthesized (Yes in S304), the processing circuitry 121 ends the band synthesis process.

[0184] If the number of bands in each band group is not equal to or less than the maximum number of synthesis bands (No in S304), the processing circuitry 121 performs recombination for each combination of bands equal to or less than the maximum number of synthesis bands for band groups whose number of bands exceeds the maximum number of synthesis bands (S305). For example, the processing circuitry 121 determines each combination of bands in the band group that includes bands equal to or less than the maximum number of synthesis bands, and recombines one or more bands included in each combination into a single band.

[0185] In one example, the original mask data corresponds to 25 bands of 430, 440, ..., 670 (nm). The non-synthesized bands are three bands of 510, 560, and 570 (nm). The maximum number of synthesized bands is four.

[0186] In this case, since the non-synthesis bands are the three bands of 510, 560, and 570 (nm), it is determined that a non-synthesis band exists (Yes in S301). Then, synthesis is performed for each band group that does not include a non-synthesis band (S303). Specifically, excluding the non-synthesis bands, three band groups of [430, 440, ..., 500], [520, 530, 540, 550], and [580, 590, ..., 670] (nm) are determined, and synthesis is performed for each band group.

[0187] As a result, the 25 bands are synthesized into six bands of [430, 440, ..., 500],

[510] , [520, 530, 540, 550],

[560] ,

[570] , and [580, 590, ..., 670] (nm).

[0188] The number of bands in the band group [430, 440, ..., 500] (nm) is 8. The number of bands in the band group [580, 590, ..., 670] (nm) is 10. Therefore, the number of bands in each of these two band groups exceeds the maximum number of synthesis bands (No in S304). Therefore, for these two band groups, recombination is performed for each combination of bands that is equal to or less than the maximum number of synthesis bands.

[0189] Specifically, for the band group [430, 440, ..., 500] (nm), two combinations, [430, 440, 450, 460] and [470, 480, 490, 500] (nm), are determined, and recombination is performed for each combination. Also, for the band group [580, 590, ..., 670] (nm), three combinations, [580, 590, 600, 610], [620, 630, 640], and [650, 660, 670] (nm), are determined, and recombination is performed for each combination.

[0190] As a result, the 25 bands are synthesized into nine bands: [430, 440, 450, 460], [470, 480, 490, 500],

[510] , [520, 530, 540, 550],

[560] ,

[570] , [580, 590, 600, 610], [620, 630, 640], and [650, 660, 670] (nm). The mask data is converted according to the above band synthesis.

[0191] 15 is a flowchart showing a second specific example of the operation of the image processing system 100 shown in FIG. 2. In this example, compared to the example of FIG. 10, the process of acquiring non-synthesis band information (S204) is omitted. In other words, there are no non-synthesis bands. The other processes are the same as the example of FIG. 10. In this example, all bands are determined to be bands to be synthesized and are synthesized for each combination based on the maximum number of synthesis bands. This reduces the number of multiple spectral images to be restored, improving the restoration speed.

[0192] In one example, the original mask data corresponds to 25 bands of 430, 440, ..., 670 (nm). There are no non-synthesized bands. The maximum number of synthesized bands is 4.

[0193] In this case, it is determined that no non-combination bands exist (No in S301 in FIG. 14). Then, for the band group including all bands, composition is performed for each combination of bands that is equal to or less than the maximum number of combination bands (S302 in FIG. 14). Specifically, seven combinations are determined: [430, 440, 450, 460], [470, 480, 490], [500, 510, 520, 530], [540, 550, 560], [570, 580, 590, 600], [610, 620, 630], and [640, 650, 660, 670] (nm), and composition is performed for each combination.

[0194] As a result, the 25 bands are synthesized into seven bands: [430, 440, 450, 460], [470, 480, 490], [500, 510, 520, 530], [540, 550, 560], [570, 580, 590, 600], [610, 620, 630], and [640, 650, 660, 670] (nm). The mask data is converted according to the above band synthesis.

[0195] Fig. 16 is a conceptual diagram showing an example of a first operation screen on the user interface of the image processing device 120 shown in Fig. 2. For example, the processing circuitry 121 displays an operation screen such as that shown in Fig. 16 on the user interface of the image processing device 120 in order to acquire non-synthesis band information and the maximum number of synthesis bands.

[0196] Specifically, the user is prompted to select a wavelength of interest and specify the maximum number of synthesis bands. The wavelength of interest corresponds to a non-synthesis band. In this example, the wavelength of interest can be selected from 430, 440, ..., 670 (nm). Note that each wavelength displayed in this example represents a band that includes that wavelength.

[0197] For example, by clicking on a square on the operation screen, a check mark will appear in the square. Also, 4 will be displayed as the initial value for the maximum number of synthesis bands. The value of the maximum number of synthesis bands can be updated by inputting an input. When "Cancel" is clicked, the display of the operation screen will close and the operation will end.

[0198] Fig. 17 is a conceptual diagram showing an example of a second operation screen on the user interface of the image processing device 120 shown in Fig. 2. In this example, 510, 560, and 570 (nm) are selected as wavelengths of interest.

[0199] Fig. 18 is a conceptual diagram showing an example of a third operation screen in the user interface of the image processing device 120 shown in Fig. 2. Clicking "Settings" in the example of Fig. 17 displays an operation screen such as that shown in Fig. 18. In Fig. 18, different hatching corresponds to different colors. Multiple bands are displayed in different colors for each combination of one or more bands that are combined into one band based on the non-combination band information and the maximum number of combination bands.

[0200] You are then prompted to confirm whether or not to perform the combining. If "OK" is clicked, the multiple bands are combined for each displayed combination, and the mask data is converted based on each combination. If "Cancel" is clicked, each band is maintained, and the mask data is maintained without being converted.

[0201] Fig. 19 is a conceptual diagram showing an example of a fourth operation screen in the user interface of the image processing device 120 shown in Fig. 2. For example, in the example of Fig. 16, clicking "Settings" without selecting a wavelength of interest displays an operation screen such as that shown in Fig. 19. In Fig. 19, similar to the example of Fig. 18, different hatching corresponds to different colors. Multiple bands are displayed in different colors for each combination of one or more bands that are combined into one band based on the maximum number of combined bands.

[0202] The example in FIG. 19 is similar to the example in FIG. 18, but the combination is different because the wavelength of interest is not selected.

[0203] Fig. 20 is a flowchart showing a third specific example of the operation of the image processing system 100 shown in Fig. 2. In this example, compared to the example of Fig. 10, a process of determining whether or not to convert mask data (S211) is added, and if the mask data is not converted, multiple processes (S203, S204, and S205) are skipped. The other processes are the same as those in the example of Fig. 10.

[0204] Specifically, in this example, the processing circuitry 121 determines whether or not to convert the mask data (S211).

[0205] If the processing circuitry 121 determines that the mask data should be converted (Yes in S211), it obtains the maximum number of synthesis bands (S203), obtains non-synthesis band information (S204), and converts the mask data (S205). On the other hand, if the processing circuitry 121 determines that the mask data should not be converted (No in S211), it skips obtaining the maximum number of synthesis bands (S203), obtaining non-synthesis band information (S204), and converting the mask data (S205).

[0206] This may reduce processing, and therefore the load on processing circuitry 121.

[0207] For example, processing circuitry 121 may determine whether to convert the mask data based on an input made by a user on a user interface. Alternatively, a selection of whether to apply a mask synthesis process may be made on the user interface. Then, if application of the mask synthesis process is selected, processing circuitry 121 may determine to convert the mask data.

[0208] Fig. 21 is a flowchart showing a fourth specific example of the operation of the image processing system 100 shown in Fig. 2. In this example, unlike the example of Fig. 10, instead of obtaining the maximum number of synthesis bands (S203), the maximum synthesis bandwidth is obtained (S221). Then, mask data is converted based on the maximum synthesis bandwidth and non-synthesis band information (S222). The other processing is the same as in the example of Fig. 10.

[0209] Here, the maximum combined bandwidth means a threshold value for the total width of one or more bands combined into one band, and means an upper limit for the total width of one or more bands combined into one band.

[0210] For example, if the maximum combined bandwidth is 50 nm and each pre-combination bandwidth is 10 nm, five or fewer bands can be combined into one band, and if each pre-combination bandwidth is 20 nm, two or fewer bands can be combined into one band. In other words, if the maximum combined bandwidth is 50 nm, five or fewer bands can be combined into one band for 10-nm-wide mask data, and two or fewer bands can be combined into one band for 20-nm-wide mask data.

[0211] Furthermore, for example, the processing circuitry 121 may obtain the maximum synthesis bandwidth based on an input made by a user via a user interface (UI). In this case, it is possible to set an optimal maximum synthesis bandwidth based on the user's knowledge, specifically, past restoration results, etc.

[0212] Alternatively, the processing circuitry 121 may obtain the maximum synthesis bandwidth from a memory in which the maximum synthesis bandwidth is stored in advance. The maximum synthesis bandwidth may be determined according to a restoration error estimated by simulation and stored in the memory.

[0213] Specifically, based on the example of FIG. 1 , the maximum synthesis bandwidth may be determined to be 40 nm and stored in advance in memory. This can suppress an increase in reconstruction error. However, the maximum synthesis bandwidth may be greater than or less than 40 nm. Specifically, the maximum synthesis bandwidth may be 80 nm, 60 nm, or 20 nm.

[0214] Furthermore, for example, the maximum composite bandwidth may be determined based on values ​​set for each pixel and each band in the mask data. Here, the values ​​set for each pixel and each band in the mask data may be determined by calibration. If the transmittance distribution of one band in the mask data is not sufficiently different from the transmittance distribution of other bands, it will be difficult to restore the information of those bands. Therefore, the maximum composite bandwidth may be determined based on the randomness of the values ​​in the mask data.

[0215] The conversion of mask data (S222) in the example of Fig. 21 is performed in the same manner as the conversion of mask data (S205) in the example of Fig. 10. However, instead of the maximum number of synthesis bands, a maximum synthesis bandwidth is used. Specifically, in the example of Fig. 10, the processing circuitry 121 sets the number of one or more bands to be synthesized into one band to be equal to or less than the maximum number of synthesis bands. Instead, in the example of Fig. 21, the processing circuitry 121 sets the total width of one or more bands to be synthesized into one band to be equal to or less than the maximum synthesis bandwidth.

[0216] Fig. 22 is a flowchart showing a fifth specific example of the operation of the image processing system 100 shown in Fig. 2. In this example, compared to the example in Fig. 21, instead of acquiring the maximum synthesis bandwidth (S221), non-synthesis band information is acquired (S204) and then the maximum synthesis bandwidth is determined (S231). The other processing is the same as in the example in Fig. 21.

[0217] For example, in the long wavelength region (low frequency region), it is assumed that the correlation of spectral information between bands is high, and in the short wavelength region (high frequency region), it is assumed that the correlation of spectral information between bands is low.

[0218] Therefore, in determining the maximum synthesis bandwidth (S231), the processing circuitry 121 may reduce the maximum synthesis bandwidth in the long wavelength region and increase the maximum synthesis bandwidth in the short wavelength region. This is expected to maintain the randomness of the transmission spectrum and allow the reconstruction calculation to be performed appropriately. This is also expected to reduce the influence of noise in the reconstruction calculation, make it easier for the solution to converge, and reduce the reconstruction error.

[0219] Specifically, in determining the maximum synthesis bandwidth (S231), the processing circuitry 121 may determine a band group including one or more consecutive bands that do not include a non-synthesis band based on the non-synthesis band information. The processing circuitry 121 may determine the maximum synthesis bandwidth for the band group based on the wavelengths (e.g., center wavelengths) of the band group.

[0220] For example, a lookup table in which wavelengths and maximum synthesis bandwidths are associated may be stored in the memory. The processing circuitry 121 may then determine the maximum synthesis bandwidth by referencing the lookup table and deriving the maximum synthesis bandwidth from the wavelengths of the band group. If multiple band groups exist, the maximum synthesis bandwidth may be determined for each band group.

[0221] Fig. 23 is a relationship diagram showing an example of the relationship between wavelength and maximum synthesis bandwidth for determining the maximum synthesis bandwidth in Fig. 22. That is, the relationship shown in Fig. 23 may be used as a reference table to be referred to in determining the maximum synthesis bandwidth (S231) in Fig. 22.

[0222] For example, if the wavelengths of the bands are greater than or equal to 400 nm and less than or equal to 600 nm, the maximum combined bandwidth is determined to be 40 nm, if the wavelengths of the bands are greater than 600 nm and less than or equal to 700 nm, the maximum combined bandwidth is determined to be 30 nm, and if the wavelengths of the bands are greater than 700 nm and less than or equal to 800 nm, the maximum combined bandwidth is determined to be 20 nm.

[0223] Although the bandwidth (specifically, the total bandwidth) is used in FIGS. 22 and 23 and the description thereof, the number of bands may be used instead of the bandwidth.

[0224] Furthermore, the number of bands in the present disclosure may be interpreted as a bandwidth, and the bandwidth in the present disclosure may be interpreted as the number of bands. Furthermore, the number of bands and the bandwidth may be used in combination. That is, each combination of one or more bands to be combined into one band may be determined so that the conditions for the maximum number of combined bands and the maximum combined bandwidth are satisfied, and multiple bands may be combined for each combination.

[0225] 24 is a flowchart showing an example of a process for determining the maximum number of synthesis bands used in the image processing system 100 shown in FIG. 2. The process for determining the maximum number of synthesis bands may be performed by the processing circuitry 121 of the image processing device 120, or by another component, another device, or another system. Although an example of the maximum number of synthesis bands is shown here, the maximum synthesis bandwidth may also be determined using a similar procedure.

[0226] First, a variable M is set to 1 (S401). A compressed image is acquired (S402). Mask data is acquired (S403). A spectral image is restored from the compressed image using the mask data as a reference image (S404). Non-synthesis band information is acquired (S405). The maximum number of synthesis bands is set to an initial value (S406). For example, the initial value is 2.

[0227] Then, the mask data is converted based on the non-composite band information and the maximum number of composite bands (S407). The example of Fig. 14 may be applied to the conversion of the mask data. Next, using the converted mask data, a spectral image is restored from the compressed image as a comparison image (S408).

[0228] Next, the error between the reference image and the comparison image is calculated (S409). For example, the error between the reference image and the comparison image may be calculated using the mean square error (MSE), mean absolute error (MAE), or structural similarity index measure (SSIM) between the images of the non-synthesized bands of the restored spectral images.

[0229] Then, it is determined whether the error between the reference image and the comparison image is equal to or less than a certain value (S410).

[0230] If the error between the reference image and the comparison image is equal to or smaller than a certain value (Yes in S410), the variable M is set to the current maximum number of synthesis bands (S412). Then, it is determined whether the maximum number of synthesis bands can be increased (S413). Whether the maximum number of synthesis bands can be increased may correspond to whether the maximum number of synthesis bands is smaller than the maximum number of consecutive bands excluding non-synthesis bands.

[0231] If the maximum number of synthesis bands can be increased (Yes in S413), the maximum number of synthesis bands is increased (S414). For example, 1 is added to the maximum number of synthesis bands. After the maximum number of synthesis bands is increased, the process is repeated from the conversion of mask data (S407).

[0232] In addition, if the mask data does not change from the previous time in the mask data conversion (S407) even when 1 is added to the maximum number of synthesis bands in increasing the maximum number of synthesis bands (S414), the maximum number of synthesis bands may be increased until the mask data changes (S414). In other words, if the combination does not change from the previous time even when 1 is added to the maximum number of synthesis bands, the maximum number of synthesis bands may be increased until the combination changes.

[0233] If the error between the reference image and the comparison image is not below a certain value (No in S410), or if the maximum number of composite bands cannot be increased (No in S413), the final maximum number of composite bands is determined to be variable M (S411), and the process of determining the maximum number of composite bands is completed.

[0234] For example, the mask data corresponds to a wavelength range of 400 to 700 nm and has 15 bands, each having a bandwidth of 20 nm. The center wavelengths of the 15 bands are 410, 430, ..., 690 nm. The non-composite bands are 410, 490, and 610 nm.

[0235] First, a spectral image is restored from the compressed image with the 15 bands remaining as it is as a reference image (S404).

[0236] Next, if the current maximum number of synthesized bands is 2, the 15 bands are synthesized into 10 bands:

[410] ,

[430] , [450, 470],

[490] ,

[510] , [530, 550], [570, 590],

[610] , [630, 650], and [670, 690] (nm). Then, using the mask data obtained by synthesizing the 15 bands into 10 bands, a spectral image is restored from the compressed image as a comparison image (S408).

[0237] Then, the reference image and the comparison image are compared in the non-combination bands of 410, 490, and 610 (nm) (S409 and S410). If the error is not equal to or less than a certain value (No in S410), the final maximum number of composite bands is determined to be 1 (S411). Note that if the maximum number of composite bands is 1, no compositing is performed.

[0238] If the error is equal to or smaller than a certain value (Yes in S410), the current maximum number of synthesized bands is increased to 3 (S414). If the current maximum number of synthesized bands is 3, the 15 bands are synthesized into 8 bands:

[410] , [430, 450, 470],

[490] , [510, 530], [550, 570, 590],

[610] , [630, 650], and [670, 690] (nm). Then, using the mask data obtained by synthesizing the 15 bands into 8 bands, a spectral image is restored from the compressed image as a comparison image (S408).

[0239] Then, the reference image and the comparison image are compared in the non-composite bands of 410, 490, and 610 (nm) (S409 and S410). If the error is not less than a certain value (No in S410), the final maximum number of composite bands is determined to be 2 (S411).

[0240] If the error is equal to or smaller than a certain value (Yes in S410), the current maximum number of synthesized bands is increased to 4 (S414). If the current maximum number of synthesized bands is 4, the 15 bands are synthesized into seven bands:

[410] , [430, 450, 470],

[490] , [510, 530], [550, 570, 590],

[610] , and [630, 650, 670, 690] (nm). Then, using the mask data obtained by synthesizing the 15 bands into the seven bands, a spectral image is restored from the compressed image as a comparison image (S408).

[0241] Then, the reference image and the comparison image are compared in the non-composite bands of 410, 490, and 610 (nm) (S409 and S410). If the error is not less than a certain value (No in S410), the final maximum number of composite bands is determined to be 3 (S411).

[0242] If the error is equal to or smaller than a certain value (Yes in S410), the current maximum number of synthesized bands is increased to 5 (S414). If the current maximum number of synthesized bands is 5, the 15 bands are synthesized into 6 bands:

[410] , [430, 450, 470],

[490] , [510, 530, ..., 590],

[610] , and [630, 650, 670, 690] (nm). Then, using the mask data obtained by synthesizing the 15 bands into 6 bands, a spectral image is restored from the compressed image as a comparison image (S408).

[0243] Then, the reference image and the comparison image are compared in the non-composite bands of 410, 490, and 610 (nm) (S409 and S410). If the error is not less than a certain value (No in S410), the final maximum number of composite bands is determined to be 4 (S411).

[0244] If the error is equal to or smaller than a certain value (Yes in S410), the maximum number of synthesis bands cannot be increased (No in S413), and therefore the final maximum number of synthesis bands is determined to be 5 (S411). Note that since the maximum number of consecutive bands excluding non-synthesis bands is 5 and the current maximum number of synthesis bands is 5, it is determined that the maximum number of synthesis bands cannot be increased.

[0245] The maximum number of synthesis bands is determined according to the above-described determination process, whereby an appropriate maximum number of synthesis bands is determined based on the restoration error.

[0246] The processing time of the restoration calculation process is proportional to the number of bands to be restored. Here, the number of bands to be restored means the number of one or more bands to be restored, and can also be expressed as the number of one or more bands after synthesis, the number of one or more bands after conversion, or the number of one or more spectroscopic images to be restored. Reducing the number of bands to be restored is an effective way to reduce the processing time.

[0247] However, as the number of bands to be restored decreases, the number and total width of one or more bands combined into one band increases, which may result in a decrease in wavelength resolution and an increase in restoration error.

[0248] In the above examples, multiple wavelengths are synthesized into multiple bands to be restored, while suppressing an increase in the number and total width of one or more bands synthesized into one band. Therefore, it is possible to reduce processing time while suppressing a decrease in wavelength resolution and an increase in restoration error. Below, another example is shown in which processing time is reduced while suppressing a decrease in wavelength resolution and an increase in restoration error.

[0249] Fig. 25 is a flowchart showing a sixth specific example of the operation of the image processing system 100 shown in Fig. 2. In this example, unlike the example of Fig. 10, instead of obtaining the maximum number of synthesis bands (S203), the maximum processing time is obtained (S241). Then, mask data is converted based on the maximum processing time and non-synthesis band information (S242). The other processing is the same as in the example of Fig. 10.

[0250] Here, the maximum processing time refers to a threshold value for the processing time of the reconstruction calculation process (S206) and refers to the upper limit of the processing time of the reconstruction calculation process (S206). In other words, the maximum processing time corresponds to the processing time allowed for the reconstruction calculation process (S206). The processing circuitry 121 may acquire the maximum processing time based on an input made by a user on a user interface (UI). Alternatively, the processing circuitry 121 may acquire the maximum processing time from a memory in which the maximum processing time is pre-stored.

[0251] The conversion of mask data (S242) in the example of Fig. 25 is performed in the same manner as the conversion of mask data (S205) in the example of Fig. 10. However, the maximum processing time is used instead of the maximum number of synthesis bands.

[0252] In the mask data conversion (S242), the processing circuit 121 determines the maximum number of bands to be restored based on the maximum processing time. Here, the maximum number of bands to be restored means a threshold for the number of one or more bands to be restored, meaning the upper limit of the number of one or more bands to be restored. Then, the processing circuit 121 synthesizes multiple bands into bands to be restored that are equal to or less than the maximum number of bands to be restored.

[0253] Specifically, the processing circuitry 121 determines one or more combinations, each including one or more bands, such that the number of one or more combinations does not exceed the maximum number of bands to be restored, or the total width of the one or more bands included in each combination is minimized.The processing circuitry 121 then synthesizes the multiple bands for each combination.

[0254] Fig. 26 is a relationship diagram showing an example of the relationship between the maximum processing time and the maximum number of bands to be restored in the image processing system 100 shown in Fig. 2. As shown in Fig. 26, the maximum number of bands to be restored is determined depending on the maximum processing time.

[0255] For example, the mask data corresponds to a wavelength range of 400 to 700 (nm) and has 15 bands each having a bandwidth of 20 nm, with the center wavelengths of the 15 bands being 410, 430, ..., 690 (nm).

[0256] Here, when the maximum processing time is 0.15 seconds, the maximum number of bands to be restored is 10. When the non-synthesized bands are 410, 490, and 610 (nm), the bands to be restored are 10 bands:

[410] ,

[430] , [450, 470],

[490] ,

[510] , [530, 550], [570, 590],

[610] , [630, 650], and [670, 690] (nm).

[0257] When the maximum processing time is 0.05 seconds, the maximum number of bands to be restored is 5. When the non-synthesized bands are 410 and 490 (nm), the bands to be restored are the five bands

[410] , [430, 450, 470],

[490] , [510, 530, ..., 590], and [610, 630, ..., 690] (nm).

[0258] When the maximum processing time is 0.3 seconds, the maximum number of bands to be restored is 15. When the non-synthesized bands are 410, 490, and 610 (nm), the bands to be restored are the original 15 bands of

[410] ,

[430] , ...,

[690] (nm).

[0259] The mask data is transformed by combining multiple bands into the band to be restored as described above.

[0260] 25 and 26 may be used in combination with at least one of the maximum number of synthesis bands and the maximum synthesis bandwidth described with reference to Figures 10 to 24. In other words, each combination of one or more bands to be synthesized into one band may be determined so that the conditions of the maximum processing time, the maximum number of synthesis bands, and the maximum synthesis bandwidth are satisfied, and multiple bands may be synthesized for each combination.

[0261] Furthermore, in the above description, an example is shown in which multiple wavelength bands before combining have the same bandwidth, but the multiple wavelength bands before combining do not have to have the same bandwidth. Also, an example is shown in which one or more wavelength bands included in each wavelength band group are contiguous, but the one or more wavelength bands included in each wavelength band group do not have to be contiguous. Similarly, an example is shown in which one or more wavelength bands included in each combination are contiguous, but the one or more wavelength bands included in each combination do not have to be contiguous.

[0262] Although the image processing system and the like have been described according to the embodiments, the aspects of the image processing system and the like are not limited to the embodiments. Modifications conceivable by those skilled in the art may be made to the embodiments, and multiple components in the embodiments may be combined in any manner.

[0263] For example, a process performed by a specific component in an embodiment may be performed by another component instead of the specific component. Furthermore, the order of multiple processes may be changed, or multiple processes may be performed in parallel. Furthermore, ordinal numbers such as "first" and "second" used in the description may be changed, removed, or newly assigned as appropriate. These ordinal numbers do not necessarily correspond to a meaningful order, and may be used to identify elements.

[0264] Also, for example, a phrase "at least one of a first element, a second element, and a third element" corresponds to the first element, the second element, the third element, or any combination thereof.

[0265] Furthermore, a method including steps performed by each component of an image processing system or the like may be performed by any system or device. In other words, this method may be performed by the image processing system or the like described above, or by another system or device.

[0266] For example, a part or all of the method may be executed by a computer including a processor, a memory, an input / output circuit, etc. In this case, the method may be executed by the computer executing a program for causing the computer to execute the method.

[0267] For example, the above program causes a computer to execute an image processing method including: acquiring a compressed image in which spectral information corresponding to a plurality of wavelength bands is compressed for each pixel; determining one or more wavelength band groups, each composed of one or more wavelength bands to be combined, from among the plurality of wavelength bands; determining, for each of the one or more wavelength band groups, one or more combinations, each composed of one or more wavelength bands to be combined into one wavelength band, from among the one or more wavelength bands that make up the wavelength band group, based on at least one of the number and total width of the one or more wavelength bands that make up the wavelength band group; and generating, based on the compressed image, a plurality of spectroscopic images corresponding to the plurality of wavelength bands combined based on the one or more combinations.

[0268] Furthermore, for example, the above program may cause a computer to execute an image processing method including: acquiring a compressed image in which spectral information corresponding to a plurality of wavelength bands is compressed for each pixel; acquiring time limit information indicating a limit on a processing time for generating a plurality of spectroscopic images based on the compressed image; determining one or more wavelength band groups, each of which is composed of one or more wavelength bands to be combined, from among the plurality of wavelength bands; determining, for each of the one or more wavelength band groups, based on the time limit information, one or more combinations, each of which is composed of one or more wavelength bands to be combined into one wavelength band, from among the one or more wavelength bands that make up the wavelength band group; and generating, based on the compressed image, the plurality of spectroscopic images corresponding to the plurality of wavelength bands combined based on the one or more combinations.

[0269] Furthermore, for example, the above program may cause a computer to execute an image processing method that includes determining one or more wavelength band groups, each of which is composed of one or more wavelength bands to be combined from among a plurality of wavelength bands, and determining, for each of the one or more wavelength band groups, one or more combinations, each of which is composed of one or more wavelength bands to be combined into one wavelength band from among the one or more wavelength bands that make up the wavelength band group, based on at least one of the number and total width of the one or more wavelength bands that make up the wavelength band group.

[0270] The above program may also be recorded on a non-transitory computer-readable recording medium such as a CD-ROM.

[0271] Furthermore, each component of the image processing system may be configured with dedicated hardware, general-purpose hardware that executes the above-mentioned programs, or a combination of these. The general-purpose hardware may be configured with a memory in which the programs are recorded and a general-purpose processor that reads and executes the programs from the memory. Here, the memory may be a semiconductor memory or a hard disk, and the general-purpose processor may be a CPU.

[0272] Furthermore, the dedicated hardware may be configured with a memory and a dedicated processor, etc. For example, the dedicated processor may refer to the memory and execute the above-described method.

[0273] Furthermore, each component of the image processing system or the like may be an electric circuit. These electric circuits may form a single electric circuit as a whole, or may be separate electric circuits. These electric circuits may correspond to dedicated hardware, or may correspond to general-purpose hardware that executes the above-mentioned programs or the like.

[0274] (Others) The following modifications of the embodiment may be made.

[0275] FIG. 27 shows a plurality of wavelength bands W １ , ..., W ｗ , multiple wavelength bands W ｃ１ , ..., W ｃｐ , multiple wavelength bands W ｄ１ , ..., W ｄｒ FIG.

[0276] (a) acquiring image data g, wherein an image sensor having n×m pixels images an object through a filter array, thereby generating the image data g, the filter array including a plurality of filters, the plurality of filters having different transmittances in each of a first plurality of wavelength bands; (b) selecting a first designated wavelength band W ｂ１ , second designated wavelength band W ｂ２ determining the first designated wavelength band W ｂ１ is one of the first plurality of wavelength bands, and the second designated wavelength band W ｂ２ is one of the first plurality of wavelength bands, and the first designated wavelength band W ｂ１ is the second designated wavelength band Ｗｂ２ Unlike the first designated wavelength band W ｂ１ and the second designated wavelength band W ｂ２ One or more wavelength bands W ｃ１ , W ｃ２, ..., W ｃ（ｐ－１） , W ｃｐ The first designated wavelength band W ｂ１ and the maximum wavelength of the wavelength band W ｃ１ The minimum wavelength of the wavelength band W is the same. ｃ１ and the maximum wavelength of the wavelength band W ｃ２ The minimum wavelength of the wavelength band W is the same. ｃ（ｐ－１） and the maximum wavelength of the wavelength band W ｃｐ The minimum wavelength of the wavelength band W is the same. ｃｐ and the maximum wavelength of the second designated wavelength band W ｂ２ The minimum wavelength of the wavelength band W is the same. ｃ１ Hand width ΔW ｃ１ is the wavelength band W ｃ１ The minimum wavelength and the wavelength band W ｃ１ the difference in the maximum wavelength between the wavelength bands W ｃ２ Hand width ΔW ｃ２ is the wavelength band W ｃ２ The minimum wavelength and the wavelength band W ｃ２ the difference in the maximum wavelength of the wavelength band W ｃ（ｐ－１） Hand width ΔW ｃ（ｐ－１） is the wavelength band W ｃ（ｐ－１） The minimum wavelength and the wavelength band W ｃ（ｐ－１） the difference in the maximum wavelength between the wavelength bands W ｃｐ Hand width ΔW ｃｐ is the wavelength band W ｃｐ The minimum wavelength and the wavelength band W ｃｐ (c) (Δw ｃ１ +ΔW ｃ２ +...+W ｃ（ｐ－１） +ΔW ｃｐ ) is greater than a predetermined value ΔWpdm, the wavelength band W ｄ１ and the minimum wavelength of the wavelength band W ｄ１ Maximum wavelength of the wavelength band W ｄ２ and the minimum wavelength of the wavelength band W ｄ２ Maximum wavelength of, ..., wavelength band W ｄｉ and the minimum wavelength of the wavelength band Wｄｉ Maximum wavelength of the wavelength band W ｄ（ｉ＋１） and the minimum wavelength of the wavelength band W ｄ（ｉ＋１） Maximum wavelength of the wavelength band Wd ｒ and the minimum wavelength of the wavelength band W ｄｒ and p is greater than r, and the wavelength band W ｄ１ Hand width ΔW ｄ１ is the wavelength band W ｄ１ The minimum wavelength and the wavelength band W ｄ１ and is equal to or less than ΔWpdm, Ｗｄ２ Hand width ΔW ｄ２ is the wavelength band W ｄ２ The minimum wavelength and the wavelength band W ｄ２ and is equal to or less than ΔWpdm, ..., the wavelength band W ｄｉ Hand width ΔW ｄｉ is the wavelength band W ｄｉ The minimum wavelength and the wavelength band W ｄｉ and is equal to or less than ΔWpdm, and ｄ（ｉ＋１） Hand width ΔW ｄ（ｉ＋１） is the wavelength band W ｄ（ｉ＋１） The minimum wavelength and the wavelength band W ｄ（ｉ＋１） and is equal to or less than ΔWpdm, ..., the wavelength band W ｄｒ Hand width ΔW ｄｒ is the wavelength band W ｄｒ The minimum wavelength and the wavelength band W ｄｒ and is equal to or less than ΔWpdm, and W ｃ１ Is W ｄ１ Included in W ｃ１ Is W ｄ２ , ..., W ｄｒ Not included in ..., W ｃ（ｊ－1） Is W ｄ（ｋ－１） Included in W ｃ（ｊ－１） Is W ｄ１ , ..., W ｄ（ｋ－２） Not included in W ｃ（ｊ－１） Is W ｄｋ , ..., Wｄｒ Not included in W ｃｊ Is W ｄｋ Included in W ｃｊ Is W ｄ１ , ..., W ｄ（ｋ－１） Not included in W ｃｊ Is W ｄ（ｋ＋１） , ..., W ｄｒ Not included in W ｃ（ｊ＋１） Is W ｄｋ Included in W ｃ（ｊ＋１） Is W ｄ１ , ..., W ｄ（ｋ－１） Not included in W ｃ（ｊ＋１） Is W ｄ（ｋ＋１） , ..., W ｄｒ Not included in ..., W ｃｐ Is W ｄｒ Included in W ｃｐ Is W ｄ１ , ..., W ｄ（ｒ-１） (d) W ｄ１ Image data f corresponding to ｄ１ , ..., W ｄｒ Image data f corresponding to ｄｒ , the image data g of the compressed image, and the matrix H Ｄ Generated based on

[0277]

[0278]

[0279] An image processing method.

[0280] (Method of Generating Matrix H) The method of generating matrix H may be as follows. Fig. 29 is a diagram showing an example of a method of generating matrix H in a modified example of the embodiment. An example of the method of generating matrix H will be described using Fig. 29.

[0281] (S1000) t is set to 1.

[0282] (S1100) The light source is wavelength band W ｔ Optical IW corresponding to ｔ FIG. 28 shows the light IW ｔ FIG. 1 is a diagram illustrating an example of light IW. ｔ is the wavelength band W ｔ The light intensity at SWｔ and the light IWt is in the wavelength band W ｔ The light intensity outside the wavelength band W is 0. ｔ The minimum wavelength of λmin (W ｔ ), the maximum wavelength of the wavelength band Wt is λmax (W ｔ )

[0283] The light source includes a white laser light source, a monochromator, and an integrating sphere.

[0284] The white laser may be a high-intensity light source that outputs white light. The wavelength range of the white light is the wavelength band W １ , ..., wavelength band W ｗ Includes:

[0285] A monochromator is a device that selectively extracts specific wavelength components from white light.

[0286] The light output from the monochromator is incident on an integrating sphere. The integrating sphere converts the incident light into light with a uniform intensity distribution and outputs it. This output light is called light IW ｔ is.

[0287] (S1200) The light It is incident on an image sensor having n×m pixels via a filter array. As a result, the image sensor outputs image data gW ｔ Generate.

[0288] (S1300) Memory (gW ｔ , S.W. ｔ ) is recorded.

[0289] The value of t plus 1 is set as the new t.

[0290] (S1400) If t is equal to or greater than (W+1), step S1500 is executed. If t is other than W, step S1100 is executed.

[0291] (S1500) Recorded (gW １ , S.W. １ ), ~, (gW ｗ , S.W. ｗ ) is used to determine the matrix H. At this time, the light IW ｔ is the wavelength band W ｔ The light intensity of SW ｔ, wavelength band W ｔ Note that the light intensity other than g is 0. In other words, in equation (6), g is gW １ In the case of f ２ , f ３ , ..., f ｗ is an all-zero vector, f １ Each value of n × m components included in SW １ , g is gW ２ In the case of f １ , f ３ , ..., f ｗ is an all-zero vector, f ２ Each value of n × m components included in SW ２ , ..., g is gW ｗ In the case of f １ , f ２ , ..., f （ｗ-１） is an all-zero vector, f ｗ Each value of n × m components included in SW ｗ is.

[0292] (Method of generating matrix H') Ａ , H Ｃ Using W １ , ~, W ｗ W ｄ１ , ~, W ｄｒ The matrix H' can be generated using a similar method as shown in FIG.

[0293] The present disclosure is applicable to an image processing method for restoring an image, and can be used in image processing systems, imaging systems, camera systems, analysis systems, recognition systems, and the like.

[0294] REFERENCE SIGNS LIST 100 Image processing system 110 Imaging device 111 Image sensor 112 Filter array 113 Optical system 120 Image processing device 121 Processing circuit 130 Display device

Claims

1. An image processing method comprising: acquiring a compressed image in which spectral information corresponding to a plurality of wavelength bands is compressed; determining one or more wavelength band groups, each composed of one or more wavelength bands to be combined from among the plurality of wavelength bands; determining, for each of the one or more wavelength band groups, one or more combinations, each composed of one or more wavelength bands to be combined into one wavelength band from among the one or more wavelength bands that make up the wavelength band group, based on at least one of the number and total width of the one or more wavelength bands that make up the wavelength band group; and generating, based on the compressed image, a plurality of spectral images corresponding to the plurality of wavelength bands combined based on the one or more combinations.

2. The image processing method according to claim 1, wherein the plurality of wavelength bands include one or more designated wavelength bands, and the one or more wavelength bands constituting each of the one or more wavelength band groups are one or more contiguous wavelength bands among the plurality of wavelength bands that do not include the one or more designated wavelength bands.

3. The image processing method of claim 2, further comprising determining the one or more designated wavelength bands based on input provided by a user or based on objects in the compressed image.

4. An image processing method according to any one of claims 1 to 3, wherein in determining the one or more combinations, the number of the one or more wavelength bands constituting each of the one or more combinations is set to a threshold value or less.

5. The image processing method according to claim 4, wherein the threshold value is set to be larger for each of the one or more wavelength band groups as the wavelength corresponding to that wavelength band group becomes shorter.

6. The image processing method according to claim 4, wherein the threshold value is 4 or less.

7. The image processing method according to claim 4, further comprising: estimating a restoration error occurring in a spectroscopic image of a specified wavelength band among the plurality of spectroscopic images in accordance with the number of the one or more wavelength bands constituting each of the one or more combinations; and determining the threshold in accordance with the restoration error.

8. An image processing method according to any one of claims 1 to 3, wherein in determining the one or more combinations, the total width of the one or more wavelength bands constituting each of the one or more combinations is set to a threshold value or less.

9. The image processing method according to claim 8, wherein the threshold value is set to be larger for each of the one or more wavelength band groups as the wavelength corresponding to that wavelength band group becomes shorter.

10. The image processing method according to claim 8, wherein the threshold value is 40 nm or less.

11. The image processing method according to claim 8, further comprising: estimating a restoration error occurring in a spectroscopic image of a specified wavelength band among the plurality of spectroscopic images in accordance with the total width of the one or more wavelength bands constituting each of the one or more combinations; and determining the threshold in accordance with the restoration error.

12. The image processing method according to any one of claims 1 to 3, wherein the compressed image is generated by photographing a subject with a camera including an optical element, and generating the plurality of spectral images includes: acquiring first encoded information, which is encoded information reflecting the spatial distribution of the transmission spectrum of the optical element and which corresponds to the plurality of wavelength bands before synthesis based on the one or more combinations; generating, based on the first encoded information and the one or more combinations, second encoded information, which is encoded information reflecting the spatial distribution of the transmission spectrum of the optical element and which corresponds to the plurality of wavelength bands after synthesis based on the one or more combinations; and generating the plurality of spectral images based on the compressed image and the second encoded information.

13. The image processing method of claim 12, wherein the first encoded information includes a plurality of values ​​corresponding to the plurality of wavelength bands, and generating the second encoded information includes calculating, for each of the one or more combinations, at least one of the sum and average of one or more values ​​from the plurality of values ​​included in the first encoded information that correspond to the one or more wavelength bands that make up the combination.

14. An image processing method comprising: acquiring a compressed image in which spectral information corresponding to a plurality of wavelength bands is compressed; acquiring time limit information indicating a limit on the processing time for generating a plurality of spectroscopic images based on the compressed image; determining one or more wavelength band groups, each consisting of one or more wavelength bands to be combined, from among the plurality of wavelength bands; determining, for each of the one or more wavelength band groups, based on the time limit information, one or more combinations, each consisting of one or more wavelength bands to be combined into one wavelength band, from among the one or more wavelength bands constituting the wavelength band group; and generating, based on the compressed image, the plurality of spectroscopic images corresponding to the plurality of wavelength bands combined based on the one or more combinations.

15. An image processing system comprising: an image sensor that acquires a compressed image in which spectral information corresponding to a plurality of wavelength bands is compressed; and a processing circuit, wherein the processing circuit: determines one or more wavelength band groups, each composed of one or more wavelength bands to be combined from among the plurality of wavelength bands; for each of the one or more wavelength band groups, determines one or more combinations, each composed of one or more wavelength bands to be combined into one wavelength band from among the one or more wavelength bands that make up the wavelength band group, based on at least one of the number and total width of the one or more wavelength bands that make up the wavelength band group; and generates, based on the compressed image, a plurality of spectral images corresponding to the plurality of wavelength bands combined based on the one or more combinations.

Images