Information processing device, information processing method, and program

Abstract

One aspect of the present invention is an information processing device that comprises: an acquisition unit that acquires an encoded image obtained by optically compressing a spectral image; and a reconstruction unit that produces a reconstructed image from the encoded image by computation. The reconstruction unit uses an alternating direction method of multipliers to solve a reconstruction problem as a convex quadratic function minimization problem, expressing an observation matrix for the alternating direction method of multipliers as a minimization problem that uses a zero padding matrix, a fast Fourier transform matrix, an inverse fast Fourier transform matrix, a color reduction matrix, an observation model, a function, and a variable that is expressed using an extended Lagrangian function, switching and thereby updating the variable, and applying a Woodbury inverse matrix formula to each element of the matrix used during the updating to reduce the fast Fourier transform matrix and the inverse fast Fourier transform matrix.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program technology.

[0002] The human eye and general color cameras capture visible light in three bands (blue, green, and red). On the other hand, spectral imaging technology, which captures images in more bands, can easily distinguish between elements that are difficult for the human eye to distinguish. For example, hemoglobin, a component of blood that transports oxygen, has slightly different absorption characteristics (absorption spectrum) for each wavelength of light when oxygen is bound or unbound. By utilizing this property, it is possible to estimate blood oxygen saturation from spectral images. However, the complexity of spectral imaging devices and the long imaging times have hindered their widespread adoption.

[0003] Recently, a compressed spectral imaging technique using a wavelength-dependent PSF (Point Spread Function) metalens, which is capable of high-speed imaging with a simple device configuration, has been proposed (see, for example, Non-Patent Document 1). A wavelength-dependent PSF metalens is a metalens designed to have properties such that the PSF shape varies significantly depending on the wavelength. A metalens is also a lens created using an optical element called an optical metasurface. The technology described in Non-Patent Document 1 reconstructs a spectral image from an encoded image captured using this wavelength-dependent PSF metalens and an image sensor. This technique is also referred to as "hyperspectral compressed imaging technology" in this specification.

[0004] Harumitsu Sogabe, "Compressed Spectral Imaging Using Wavelength-Dependent PSF Metalens," Journal of the Institute of Image Information and Television Engineers, vol. 76, no. 2, pp. 234-239, Mar. 2022.

[0005] However, in the conventional technology, there is a problem that it takes a long time to reconstruct a captured encoded image into a spectral image. In view of the above circumstances, an object of the present invention is to provide a technology that can reduce the processing time when reconstructing an encoded image into a spectral image.

[0006] One aspect of the present invention is an information processing device that includes an acquisition unit that acquires an encoded image obtained by optically compressing a spectral image and capturing it, and a reconstruction unit that reconstructs a reconstructed image from the encoded image through calculation, wherein the reconstruction unit uses alternating direction multipliers to solve a reconstruction problem as a convex quadratic function minimization problem, and in the alternating direction multipliers, expresses an observation matrix as a minimization problem using a zero-padding matrix, a fast Fourier transform matrix, an inverse fast Fourier transform matrix, a color reduction matrix, an observation model, a function, and variables, expresses the variables using an extended Lagrangian function, updates the variables by switching between them, and reduces the fast Fourier transform matrix and the inverse fast Fourier transform matrix by applying Woodbury's inverse matrix formula to each element of the matrix used during updating.

[0007] One aspect of the present invention is an information processing method of an information processing device, in which the information processing device acquires an encoded image obtained by optically compressing and capturing a spectral image, reconstructs a reconstructed image from the encoded image by calculation, and in the reconstruction, uses alternating direction multipliers to solve the reconstruction problem as a convex quadratic function minimization problem, and in the alternating direction multipliers, expresses an observation matrix in the alternating direction multipliers as a minimization problem using a zero-padding matrix, a fast Fourier transform matrix, a fast Fourier transform matrix, a color reduction matrix, an observation model, a function, and variables, expresses the variables using an extended Lagrangian function, updates the variables by switching between them, and applies Woodbury's inverse matrix formula to each element of the matrix used during the update, thereby reducing the fast Fourier transform matrix and the inverse fast Fourier transform matrix.

[0008] One aspect of the present invention is a program that causes a computer to function as the information processing device described above.

[0009] According to the present invention, it is possible to reduce the processing time required to reconstruct an encoded image into a spectral image.

[0010] FIG. 1 is a diagram for explaining an overview of compressive spectrum imaging using a wavelength-dependent PSF metalens. FIG. 2 is an image diagram of an observation model g, a two-dimensional fast Fourier transform matrix F, and sensor sensitivity. FIG. 3 is a diagram showing an example of the configuration of an information processing device of an embodiment. FIG. 4 is a diagram showing an example of the configuration of a reconstruction unit of an embodiment, where a deep layer is expanded. FIG. 5 is a diagram showing an example of the configuration of each stage of the reconstruction unit, where a deep layer is expanded. FIG. 6 is a flowchart of processing performed by an information processing device of an embodiment.

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail with reference to the drawings and formulas.

[0012] [Overview of Compressive Spectral Imaging Using Wavelength-Dependent PSF Metalens] First, an overview of compressive spectral imaging using a wavelength-dependent PSF metalens will be described with reference to FIGS. 1 and 2. FIG. 1 is a diagram for explaining the overview of compressive spectral imaging using a wavelength-dependent PSF metalens. FIG. 2 is an image diagram of an observation model g, a two-dimensional fast Fourier transform matrix F, and sensor sensitivity. In FIG. 2, the image indicated by reference symbol g150 is an image diagram of the observation model g, the image indicated by reference symbol g160 is an image diagram of the two-dimensional fast Fourier transform matrix F, and reference symbol g170 is an image diagram of sensor sensitivity. In reference symbol g170, the horizontal axis is wavelength (nm), line g171 is the sensor sensitivity for R, line g172 is the sensor sensitivity for B, and line g173 is the sensor sensitivity for G.

[0013] The camera g110 in FIG. 1 has a metalens g111 and an image sensor g112. The metalens g111 is configured with a nanostructure pattern as indicated by the reference symbol g113. The camera g110 captures an image of a subject g101 and outputs a compressed encoded image (color image) g121. Note that the image is compressed using the metalens g111. In other words, the encoded image is an image obtained by optically compressing and encoding a spectral image.

[0014] The image reconstruction unit reconstructs the encoded image g121 using, for example, a trained neural network (symbol g131). The image reconstruction unit outputs the reconstructed hyperspectral image g141. Note that a hyperspectral image is an image that captures more color information (wavelengths) than a normal color image that mimics the human eye.

[0015] Here, the observation model g for compressive imaging using a metalens is formulated as in the following equation (1): In each equation below, bold characters represent vectors or matrices.

[0016]

[0017] In equation (1), Φ is the observation matrix, W is the subtractive color matrix, and P is the metalens matrix. The vector f is expressed by the following equation (2), which is a hyperspectral image with height H, width W, and number of bands Λ. R (double-underlined character) is a set of all real numbers.

[0018]

[0019] The vector g is expressed by the following equation (3) and is an RGB (red, green, blue) image with height H and width W.

[0020]

[0021] The observation matrix Φ = WP is expressed by the following equation (4) and is composed of a metalens matrix P and a subtractive matrix W.

[0022]

[0023] In the embodiments, the "problem of estimating a hyperspectral image f from a given RGB image g and observation matrix Φ" is referred to as the "hyperspectral image reconstruction problem." Because the number of rows in the observation matrix Φ is smaller than the number of columns, the reconstruction problem is an underdetermined problem. When the hyperspectral image reconstruction problem is formulated as a regularization problem using a function R (the following equation (5)) to solve, the problem of estimating a hyperspectral image f can be formulated as shown in the following equation (6).

[0024]

[0025]

[0026] In equation (6), τ is a positive real number and is a parameter that controls the strength of regularization. ２ ２ is the vector... ２ It is the square of the norm.

[0027] To solve the hyperspectral image reconstruction problem using the Alternating Direction Method of Multiplier (ADMM) (see Reference 1), we define an augmented Lagrangian function, introduce an auxiliary variable s = f, and replace equation (6) with equation (7).

[0028] Reference 1: Jian-Feng Cai, Ke Wei, “Alternating Direction Method of Multiplier”, in Handbook of Numerical Analysis, 2018, <Internet search; 2023.12.18>, https: / / www.sciencedirect.com / topics / mathematics / alternating-direction-method-of-multipliers

[0029]

[0030] For this optimization problem, the extended Lagrange function L (cursive) is defined as the following equation (8).

[0031]

[0032] In equation (8), μ is a hyperparameter, and u is the dual variable as shown in equation (9).

[0033]

[0034] This extended Lagrange function is optimized by alternately switching variables as shown in the following equation (10): where argmin is the minimum point set.

[0035]

[0036] The first update equation f of equation (10) ｋ＋１ Specifically, the following equation (11) is written, which is a convex quadratic function optimization problem (quadratic programming problem). Note that a convex quadratic function optimization problem is a problem of optimizing a convex quadratic function on a polyhedron.

[0037]

[0038] The solution to equation (11) can be analytically obtained by the following equation (12): In equation (12), I is a unit matrix, and the superscript T represents a transpose.

[0039]

[0040] In equation (12), the matrix Φ Ｔ Since the size of Φ+μI is very large, it cannot be performed on current computers due to the spatial and time complexity. Therefore, the gradient direction update is performed approximately by the steepest descent method without directly solving the convex quadratic function optimization problem. That is, the first update formula f in formula (10) is ｋ＋１ Now, equation (12) is approximated by the gradient method as in equation (13). The update equation is given by introducing a positive real step parameter ε as in equation (13). In equation (13), ∇ is the nabla operator.

[0041]

[0042] Furthermore, the second update formula s in formula (10) ｋ＋１ Specifically, this is written as the following equation (14).

[0043] In formula (14), prox ｆ is a proximity map for the function f. According to the theory of PnP (Plug-and-Play) regularization, the proximity map outputs the noise removal result of the input signal, so it can be replaced with any noise remover to perform the regularization function. For example, if U-net is used as a noise remover, s ｋ＋１can be formulated as the third equation in equation (14). In the third equation in equation (14), D (cursive) is a function that represents noise removal by U-net. U-net is a type of FCN (fully convolution network), and is a network for estimating image segmentation (where an object is located). In this way, the second update equation s in equation (10) ｋ＋１ Here, we use the function D (cursive) to represent noise removal.

[0044] From the above, the solution of the reconstruction problem using the ADMM is the iterative repetition of the update formula from formula (10) to the following formula (15).

[0045]

[0046] The solution f of the reconstruction problem is obtained by incrementing the update formula (15) from k=0 and ending the calculation at an arbitrary k=k-1. ｋ However, with this solution method, the accuracy of the solution improves as the number of iterations k increases, but the calculation time also increases. For this reason, deep neural networking is introduced to reduce calculation time while maintaining accuracy. Deep neural networking is a framework that uses a relatively small amount of training data to optimally design the hyperparameters of an iterative optimization algorithm, significantly reducing the number of iterations while maintaining accuracy.

[0047] In the deep expansion, we first allow the hyperparameters μ and ε of the ADMM solution to change with each k, and then ｋ , ε ｋ (k=0, ..., K-1). Appropriate values ​​for these hyperparameters and the weight parameters of the U-net are acquired through learning.

[0048] In this calculation method, the two-dimensional FFT (Fast Fourier Transform) and IFFF (Inverse Fast Fourier Transform) become bottlenecks in terms of computational complexity. The metalens matrix P constituting the observation matrix Φ is written as the following equation (16):

[0049]

[0050] In equation (16), Z is the zero-padding matrix shown in equation (17) below, F is the two-dimensional fast Fourier transform matrix shown in equation (18) below, and p is the frequency response of the metalens shown in equation (19) below. Also, diag(·) is a square diagonal matrix. Zero padding is the process of adding blank pixels of appropriate values ​​around an image, and is the process of filling the image with zeros. C (double-underlined character) is the set of all complex numbers.

[0051]

[0052]

[0053]

[0054] In equations (17) to (19), the height H' and width W' are the height and width extended by zero padding, and where L is the filter kernel size of the metalens, H' = H + L and W' = W + L. At each stage, Φ Ｔ Φf ｋ Since it is necessary to calculate Φ, two-dimensional FFT and IFFT must be performed four times at each stage, which becomes a bottleneck in calculation time. Ｔ Φ is the matrix of the first term of equation (12) or the like that solves the first update equation.

[0055] (Description, etc.) Next, the notation, etc. used in the reduction method of this embodiment will be described. Ｎ , a zero matrix of M rows and N columns is O Ｍ×Ｎ When there is a matrix A with I rows and J columns and a matrix B with K rows and L columns, the Crocker product (X in a white circle) (the following equation (20)) of matrix A and matrix B is a matrix with IK rows and JL columns, and each component a ｉ，ｊ , b ｋ，ｌ Using the formula (20), each component of the Crocker product (formula (20)) is defined as the following formula (21): The Crocker product is a binary operation defined between matrices of any size.

[0056]

[0057]

[0058] The Crocker product (Equation (20)) can be written as the following Equation (22) in block matrix notation.

[0059]

[0060] Using this Crocker product, the metalens matrix P and the color-reduction matrix W that make up the observation matrix Φ = WP can be described in more detail as follows: First, since the metalens matrix P is given by equation (16), the zero-padding matrix Z that makes up this matrix can be written as the following equation (23).

[0061]

[0062] In formula (23), Z Ｈ’×Ｈ and Z Ｗ’×Ｗ Each is a zero-padded matrix of a one-dimensional signal. Ｈ’×Ｈ is the following equation (24), and Z Ｗ’×Ｗ is expressed by the following equation (25).

[0063]

[0064]

[0065] Furthermore, the two-dimensional fast Fourier transform matrix F is given by the following equation (26).

[0066]

[0067] In formula (26), F Ｈ’ is the following equation (27), which is a one-dimensional fast Fourier transform matrix of length H′, and F Ｗ’ is the following equation (28) and is a one-dimensional fast Fourier transform matrix of W′.

[0068]

[0069]

[0070] Moreover, the color-reduction matrix W can be written as the following equation (29).

[0071]

[0072] In equation (29), Ω is the sensor sensitivity matrix as shown in equation (30) below.

[0073]

[0074] Here, the Cartesian product of the matrices that make up the block diagonal matrix is ​​written as the following equation (31) using the Cartesian product symbol (a plus sign (+) in a circle).

[0075]

[0076] Furthermore, it is assumed that the zero-padding matrix Z satisfies the approximation of the following equation (32).

[0077]

[0078] Using an approximation of equation (32), the following equation (33) holds.

[0079]

[0080] In addition, in the formula (33), W′ is given by the following formula (34).

[0081]

[0082] [Method for reducing the amount of calculation] In this embodiment, the first update is not a gradient direction update, but a convex quadratic function minimization problem is solved to reduce the number of times that fast Fourier transforms and inverse fast Fourier transforms are performed, thereby reducing the amount of calculation. Furthermore, in this embodiment, processing such as noise removal is performed on the second update formula.

[0083] (First Update Equation) First, the first update equation will be described. In order to solve the convex quadratic function minimization problem, the first update equation is obtained from the reconstruction equation of the ADMM in this embodiment. The reconstruction problem is expressed by the following equation (35) when the observation matrix Φ is specified.

[0084]

[0085] Here, h = Zf. Ｔ Z = I ＨＷΛ Since this holds, f = Z Ｔ h. Therefore, the reconstruction problem of equation (35) is equivalent to the following equation (36):

[0086]

[0087] Furthermore, when the auxiliary variable s=h is introduced, equation (36) is equivalent to the following equation (37).

[0088]

[0089] Here, the extended Lagrange function L (cursive) is defined as the following equation (38).

[0090]

[0091] Furthermore, the extended Lagrange function L (cursive) is optimized by alternately switching variables as shown in the following equation (39): Note that equation (39) can be expressed as the following equation (40).

[0092]

[0093]

[0094] When the first update equation of equation (40) is specifically written using the above definitions and descriptions, it becomes the following equation (41).

[0095]

[0096] FZW of equation (41) Ｔ WZ Ｔ F Ｈ is expressed by the following equation (42).

[0097]

[0098] In formula (42), Z Ｎ’×Ｎ Z Ｎ’×Ｎ Ｔ is approximated by the following equation (43), and equation (42) can be approximated by the following equation (44).

[0099]

[0100]

[0101] Using the approximation of equation (44), h ｋ＋１ can be approximated as in the following equation (45).

[0102]

[0103] The first term of equation (45) (diag(p － ) ...) －１is a matrix as shown in the following equation (46). Each element A of the matrix is ​​as shown in the following equation (47).

[0104]

[0105]

[0106] In equation (46), focusing on the diagonal blocks, when Woodbury's inverse matrix formula (see, for example, M.A. Woodbury, "Inverting Modified Matrices", Department of Statistics, Princeton University, 1950) is applied to calculate the inverse matrix of the Λ × Λ matrix, each element can be expressed as in the following equation (48).

[0107]

[0108] Here, using equation (34), each element is expressed as in the following equation (49): In equation (49), each element B of the matrix is ​​expressed as in the following equation (50).

[0109]

[0110]

[0111] In addition, in equation (49), each element of the matrix (μ ３ +Ωdiag(|p ｉ，ｊ | ２ Ω Ｔ )) －１ is a 3-row, 3-column matrix. Therefore, the first update formula h ｋ＋１ can be approximated as in the following equation (51): In equation (51), each element C of the matrix is ​​expressed by the following equation (52).

[0112]

[0113]

[0114] (Second Update Formula) Next, the second update formula of formula (40) will be described. The second update formula of formula (40) can be expressed as the following formula (53).

[0115]

[0116] Here, the above-mentioned Z Ｔ If Z=I is used, and the above-mentioned theorem is used, the following equation (54) holds true for equation (53).

[0117]

[0118] If the proximity mapping calculation is PnP regularization D (cursive) by U-net, then equation (54) becomes the following equation (55).

[0119]

[0120] (Update Equation, Parameters) As described above, as a result of approximating and substituting the first update equation and the second update equation, the update equation for solving the reconstruction problem is given by the following equation (56). Note that the weight parameters and the dual variable u are parameters that are learned in advance.

[0121]

[0122] In this embodiment, the update formula (56) is incremented from k=0 and the calculation is terminated at an arbitrary k=K−1, thereby obtaining the solution f ｋ =Z Ｔ h ｋ It should be noted that deep evolution allows the hyperparameter μ to change at each k, and μ ｋ (k=0, ..., K-1).

[0123] By approximating the first update equation as described above, the first update equation can be calculated by fast Fourier transform, element product with the frequency response of the filter, color subtraction, 3-row 3-column product for each pixel, transpose of color subtraction, element product with the frequency response of the filter, difference, and inverse fast Fourier transform.The first update equation before approximation is expressed as equation (42) using the fast Fourier transform matrix F and the inverse fast Fourier transform matrix F Ｈ On the other hand, the first update equation after approximation is the fast Fourier transform matrix F and the inverse fast Fourier transform matrix F as shown in equation (51). Ｈ In this way, according to this embodiment, for the first update equation, the number of times of fast Fourier transform and inverse fast Fourier transform can be reduced from four to two. As a result, according to this embodiment, the amount of calculation during reconstruction can be reduced.

[0124] As described above, in this embodiment, in compressed spectral imaging using a wavelength-dependent PSF metalens, the following processing is performed to solve the problem of estimating a hyperspectral image f from a given RGB image g and observation matrix Φ (hyperspectral image reconstruction problem). I. Formulate using a function R (Equations (5) and (6)). II. In order to solve the hyperspectral image reconstruction problem using the alternating direction multiplier method (ADMM), an augmented Lagrangian function is defined (Equation (8)). This allows the optimization update equation to be obtained (Equation (10)). III. The first update equation f of the update equation in II is ｋ＋１ This is done by using approximation etc. when solving a convex quadratic function optimization problem (quadratic programming problem) (Equation (51)). IV. The second update equation s ｋ＋１ When solving, we use a function D (cursive) that represents noise removal (Equation (55)). V. The solution method in ADMM of III and IV is accelerated by deep layer expansion.

[0125] [Configuration example of information processing device] Next, a configuration example of the information processing device 20 that performs the above-mentioned processing will be described. Fig. 3 is a diagram showing a configuration example of the information processing device of this embodiment. As shown in Fig. 3, the information processing device 20 includes, for example, an acquisition unit 21, a reconstruction unit 22, an output unit 23, and a storage unit 24.

[0126] The acquisition unit 21 acquires image data from the imaging device 10. As described with reference to FIG. 1 , the imaging device 10 includes, for example, a metalens g111 and an image sensor g112. The image data acquired by the acquisition unit 21 is, for example, an encoded image (color image) g obtained by optically compressing a spectral image f. The imaging device 10 and the information processing device 20 are connected via wire or wirelessly.

[0127] The reconstructor 22 generates a reconstructed image (hyperspectral image) by solving the image data acquired by the acquirer 21 using, for example, the ADMM solution as described above. An example of the configuration of the reconstructor 22 will be described later.

[0128] The output unit 23 outputs the reconstructed image generated by the reconstruction unit 22 to an external device. The external device is, for example, an image display device, a personal computer, a smartphone, or a tablet terminal.

[0129] The storage unit 24 stores mathematical expressions, thresholds, programs, etc. that are used by each unit of the information processing device 20 for processing.

[0130] The information processing device 20 may also be included in the imaging device 10. Furthermore, all or part of the processing performed by the information processing device 20 may be processed on the cloud. In this case, the information processing device 20 includes a communication unit.

[0131] The information processing device 20 is configured using a processor such as a CPU (Central Processing Unit) and a memory. The information processing device 20 functions as an acquisition unit 21, a reconstruction unit 22, and an output unit 23 by the processor executing a program. All or part of the functions of the information processing device 20 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, and a semiconductor storage device (e.g., an SSD: Solid State Drive), as well as storage devices such as a hard disk or semiconductor storage device built into a computer system. The above program may be transmitted via a telecommunications line.

[0132] [Configuration Example of Reconstruction Unit] Next, a configuration example of the reconstruction unit 22 will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a diagram showing a deeply expanded configuration example of the reconstruction unit of this embodiment. Fig. 5 is a diagram showing a deeply expanded configuration example of each stage of the reconstruction unit.

[0133] 4, the reconstruction unit 22 includes K stages 220-1 to 220-K (K is an integer equal to or greater than 2). The output of the first stage 220-1 is connected to the input of the second stage 220-2, ..., and the output of the (K-1)th stage 220-(K-1) is connected to the input of the Kth stage 220-K.

[0134] The first stage 220-1 includes an initial variable h ０ , initial auxiliary function s ０ , initial dual variable u ０ , hyperparameter μ １ , and weight parameters are input. The first stage 220-1 updates the variable h １ , auxiliary function s １ , dual variable u １ The K-th stage 220-K outputs a variable h Ｋ－１ , auxiliary function s Ｋ－１ , dual variable u Ｋ－１ , hyperparameter μ Ｋ , and weight parameters are input. The Kth stage 220-K inputs the variable h Ｋ , auxiliary function s Ｋ , dual variable u Ｋ Then, the zero-padding matrix Z Ｔ Multiply by h ｋ The reconstructed solution is f ｋ Convert it to and output it.

[0135] As shown in FIG. 5 , each stage includes, for example, a convex quadratic function optimization unit 221, a first calculation unit 222, a first matrix calculation unit 223, a noise reduction unit 224, a second matrix calculation unit 225, a third matrix calculation unit 226, a second calculation unit 227, and a third calculation unit 228.

[0136] The convex quadratic function optimization unit 221 includes a variable h ｋ , auxiliary function s ｋ , dual variable u ｋ , and the hyperparameter μ Ｋ＋１ The convex quadratic function optimization unit 221 performs optimization processing using the above-mentioned formulas and approximations to update the variable h ｋ＋１ Output.

[0137] The first calculation unit 222 calculates the variable h ｋ＋１ From the dual variable u ｋ Subtract the variable h ｋ＋１ From the dual variable u ｋ The result of subtraction is output to the first matrix calculation unit 223, the third matrix calculation unit 226, and the third calculation unit 228.

[0138] The first matrix calculation unit 223 calculates the variable h ｋ＋１ From the dual variable u ｋ The result of subtracting Ｔ The result of multiplication (Z Ｔ (h ｋ＋１ -u ｋ )) to the noise elimination unit 224.

[0139] The noise removal unit 224 receives the calculation result (Z Ｔ (h ｋ＋１ -u ｋ )) and weighting parameters are input. The noise elimination unit 224 performs noise elimination using the above-mentioned formulas and approximations, and outputs the noise elimination result (D (cursive) (Z Ｔ (h ｋ＋１ -u ｋ ))) is output to the second matrix calculation unit 225.

[0140] The second matrix calculation unit 225 calculates the noise-removed result (D(cursive)(Z Ｔ (h ｋ＋１ -u ｋ ))) by the zero-padding matrix Z to obtain the result (ZD (cursive) (Z Ｔ (h ｋ＋１ -u ｋ ))).

[0141] The third matrix calculation unit 226 calculates the variable h ｋ＋１ From the dual variable u ｋ The result of subtracting (I-ZZ Ｔ ) multiplied by the result ((I-ZZ Ｔ ) (h ｋ＋１ -u ｋ )) to the second calculation unit 227.

[0142] The second calculation unit 227 calculates the (ZD (cursive) (Z Ｔ (h ｋ＋１ -u ｋ ))) the third matrix calculation unit 226 outputs ((I-ZZ Ｔ ) (h ｋ＋１ -u ｋ )) and outputs the result of the addition to the second calculation unit 227, and the updated auxiliary function s ｋ＋１ Output as

[0143] The third calculation unit 228 calculates the updated auxiliary function s output by the second calculation unit 227. ｋ＋１ The first calculation unit 222 outputs (h ｋ＋１ -u ｋ ) and update the result of the subtraction to the dual variable u ｋ＋１ Output as

[0144] 4 and 5 are merely examples, and the present invention is not limited to these. For example, each stage may include other components.

[0145] [Example of Processing Procedure] Next, a description will be given of an example of processing procedure performed by the information processing device 2. Fig. 6 is a flowchart of processing performed by the information processing device of this embodiment.

[0146] (Step S1) The acquisition unit 21 acquires image data, which is, for example, an optically compressed encoded image, from the image capturing device 10.

[0147] (Step S2) The reconstructor 22 initializes parameters and values ​​used for counting (for example, k).

[0148] (Step S3) The reconstruction unit 22 updates the update formula by performing processing at each stage as shown in FIG. 4 using processing using an approximation formula and noise removal processing for the update formula. The reconstruction unit 22 increments the update formula sequentially from k=0 and ends the calculation at an arbitrary k=k−1, thereby obtaining a solution f of the reconstruction problem. ｋ get.

[0149] (Step S4) The reconstructing unit 22 determines whether or not the predetermined number of updates k has been completed. If the updates have been completed (step S4; YES), the reconstructing unit 22 proceeds to the processing of step S5. If the updates have not been completed (step S4; NO), the reconstructing unit 22 returns to the processing of step S3.

[0150] (Step S5) The reconstructing unit 22 outputs the reconstructed image data (hyperspectral image) to an external device.

[0151] The information processing device 20 configured as described above reconstructs a reconstructed image through calculations from an encoded image captured by optically compressing a spectral image. The information processing device 20 then uses an alternating direction multiplier method to solve the reconstruction problem as a convex quadratic function minimization problem. In the alternating direction multiplier method, the observation matrix is ​​expressed as a minimization problem using a zero-padding matrix, a fast Fourier transform matrix, a fast Fourier transform matrix, a color reduction matrix, an observation model, a function, and variables. The variables are expressed using an extended Lagrangian function, and the variables are updated by switching between them. The Woodbury inverse matrix formula is applied to each element of the matrix used during the update, thereby reducing the fast Fourier transform matrix and the inverse fast Fourier transform matrix. The information processing device 20 also performs noise reduction processing during the update. This allows the information processing device 20 of this embodiment to eliminate two fast Fourier transforms and inverse fast Fourier transforms, thereby reducing the processing time required to reconstruct an encoded image into a spectral image.

[0152] (Modification) Note that, in the above example, the acquisition unit 21 acquires an encoded image captured by an imaging device 10 having a metalens g111 and an image sensor g112, but the structure of the imaging device 10 is not limited to this. The acquisition unit 21 may acquire an encoded image from an imaging device 10 with a different structure, or may acquire an encoded image from a device other than the imaging device 10.

[0153] In the above example, the problem of reconstructing a hyperspectral image is solved using the Alternating Direction Method of Multipliers (ADMM), but the method used is not limited to this. For example, a method in which another algorithm is added to the ADMM, or a method in which the ADMM is improved may also be used.

[0154] Furthermore, the approximation formulas described above are merely examples, and other approximation formulas may be used.

[0155] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0156] The present invention is applicable to, for example, an imaging device, an image processing device, an image analysis device, and the like.

[0157] 10...imaging device, 20...information processing device, 21...acquisition unit, 22...reconstruction unit, 23...output unit, 24...storage unit, 221...convex quadratic function optimization unit, 222...first calculation unit, 223...first matrix calculation unit, 224...noise removal unit, 225...second matrix calculation unit, 226...third matrix calculation unit, 227...second calculation unit, 228...third calculation unit

Claims

1. An acquisition unit that optically compresses a spectral image to obtain an encoded image, and a reconstruction unit that reconstructs a reconstructed image by calculation from the encoded image, wherein the reconstruction unit uses the alternating direction multiplier method to solve the reconstruction problem as a convex quadratic function minimization problem, and represents the observation matrix as a minimization problem using a zero-padding matrix, a fast Fourier transform matrix, an inverse fast Fourier transform matrix, a color reduction matrix, an observation model, a function, and variables in the alternating direction multiplier method, represents the variables using an extended Lagrangian function, updates the variables by switching them mutually, and reduces the fast Fourier transform matrix and the inverse fast Fourier transform matrix by applying the Woodbury inverse matrix formula to each element in the matrix used at the time of update. Information processing apparatus.

2. The information processing apparatus according to claim 1, wherein the reconstruction unit updates the first variable by solving a convex quadratic function optimization problem, multiplies the second variable by the zero-padding matrix transposed with respect to the result of updating the first variable, performs noise removal processing on the multiplied result, multiplies the noise-removed result by the zero-padding matrix, and adds a result obtained by subtracting a result of multiplying the result of multiplying by the zero-padding matrix by the zero-padding matrix transposed from the identity matrix.

3. An information processing method for an information processing apparatus, wherein the information processing apparatus obtains an encoded image obtained by optically compressing a spectral image, reconstructs a reconstructed image by calculation from the encoded image, and in the reconstruction, uses the alternating direction multiplier method to solve the reconstruction problem as a convex quadratic function minimization problem, represents the observation matrix as a minimization problem using a zero-padding matrix, a fast Fourier transform matrix, an inverse fast Fourier transform matrix, a color reduction matrix, an observation model, a function, and variables in the alternating direction multiplier method, represents the variables using an extended Lagrangian function, updates the variables by switching them mutually, and reduces the fast Fourier transform matrix and the inverse fast Fourier transform matrix by applying the Woodbury inverse matrix formula to each element in the matrix used at the time of update. Information processing method.

4. A program that causes a computer to function as the information processing apparatus according to claim 3.

Images