A hyperspectral computational imaging method, system, and related equipment for multispectral acquisition

Abstract

This invention relates to the field of computational optical imaging and provides a hyperspectral computational imaging method, system, and related equipment for multispectral acquisition. The method includes: acquiring filter parameters; constructing a spectral simulation reconstruction network; setting a minimum reconstruction error and using the filter parameters to set variables for the annealing algorithm, constructing a simulated transmittance curve to train the spectral simulation reconstruction network, calculating the error until the annealing algorithm calculates the optimal solution and obtains the optimal parameters; training the spectral reconstruction network based on the optimal parameters, simultaneously selecting corresponding filters to form spectral sampling units, and sampling the target spectrum to obtain a mixed spectral image; registering the mixed spectral image to obtain a registered spectral image; and inputting the registered spectral image into the spectral reconstruction network to obtain a reconstructed spectral image. This invention improves the spectral acquisition speed, solves the registration problem of snapshot-type spectral acquisition, and further improves the accuracy of hyperspectral computational imaging.

Description

Technical Field

[0001] This invention belongs to the field of computational optical imaging, and in particular relates to a hyperspectral computational imaging method, system and related equipment for multispectral acquisition. Background Technology

[0002] Hyperspectral imaging is a sophisticated technique that captures and analyzes spectra point-by-point within a spatial region. By analyzing the spectrum, the composition and components of substances can be understood, while imaging techniques can obtain information on the contours and textures of substances. Spectral imaging systems that combine spectral analysis and imaging can measure the chemical composition of the corresponding substance for each pixel in an image, enabling the detection and identification of high-value targets against complex backgrounds. Computational optical imaging, on the other hand, is an emerging research field that achieves specific imaging functions and characteristics by jointly optimizing optical systems and signal processing. It mainly uses methods such as illumination and optical system modulation to establish a transformation or modulation model between the target scene and the observed image, and then uses mathematical methods such as solving inverse problems to perform imaging through computational inversion.

[0003] In existing technologies, pushbroom-type spectral imaging systems used in remote sensing and industrial online inspection consist of two modules: spectral image acquisition and spectral image analysis. During spectral image acquisition, a slit divides the target into one-dimensional lines, and a grating unfolds the spectrum of each point of the one-dimensional line in space to form a spectral plane of the one-dimensional line. An area array image sensor is used to acquire the image of the spectral plane to obtain the spectral data of each point of the one-dimensional line. To further acquire the spectral data of the target's two-dimensional image, the camera needs to pushbroom relative to the target, thereby stitching together the spectral data of the one-dimensional line to obtain a two-dimensional image. The scanning time for 1000 lines usually takes tens of seconds. Therefore, when the amount of spectral image data is large, the data transmission and processing require a long time. On the other hand, spectral imaging technology based on AOTF (Acousto-Optic Tunable Filter) has a switching time on the order of microseconds. However, AOTF itself is expensive, has a small aperture, a small viewing angle, and a thickness of over 100mm. It also requires a high-voltage driver, resulting in poor integration performance. SSCSI (spatial-spectral encoded compressive hyperspectral imaging) proposed by Arce et al. of the University of Delaware uses a dispersive element to translate images of different spectral bands in the horizontal direction. Then, a color filter array is used to modulate the frequency of the translated image in the spectral domain. This enables the acquisition of measurement values ​​for compressed sensing multispectral imaging without moving system components. After that, spectral reconstruction algorithms such as OMP (orthogonal matching pursuit) are used to reconstruct the spectrum. The problem is that its color filter array is expensive, and the iteration required to reconstruct a single spectral pixel using a target-based spectral reconstruction algorithm is also very time-consuming, making it impossible to achieve rapid spectral analysis. Summary of the Invention

[0004] This invention provides a hyperspectral computational imaging method, system, and related equipment for multispectral acquisition, aiming to solve the problems of traditional hyperspectral computational imaging technology being unable to quickly acquire and analyze spectra, and having complex hardware processes and high costs.

[0005] In a first aspect, embodiments of the present invention provide a hyperspectral computational imaging method for multispectral acquisition, the method comprising the following steps:

[0006] Obtain the parameters of at least 8 filters;

[0007] Construct a spectral simulation and reconstruction network;

[0008] Set a minimum reconstruction error, and set the variables controlled by the annealing algorithm in a group using the parameters of at least 8 of the filters. Construct a simulated transmittance curve using the parameters of the filters, and iteratively train the spectral simulation reconstruction network. Calculate the error of each iteration until the annealing algorithm calculates the optimal solution. Obtain the parameters of at least 8 of the filters corresponding to the optimal solution, and record them as the optimal parameters.

[0009] The corresponding filter is selected according to the optimal parameters, and a spectral reconstruction network with the same structure as the spectral simulation reconstruction network is trained according to the actual transmittance curve of the filter. At the same time, the selected filters are used to form a spectral sampling unit, and the target spectrum is sampled through the spectral sampling unit to obtain a mixed spectral image.

[0010] The mixed spectral image is registered to obtain a registered spectral image;

[0011] The registered spectral image is input into the spectral reconstruction network to obtain the reconstructed spectral image.

[0012] Furthermore, the spectral simulation reconstruction network includes an input layer, a network convolutional layer, a dimensionality reduction layer, a fully connected layer, and an output layer. The network convolutional layer includes three sets of cross-layer connected feature extraction structures, the fully connected layer includes three sets of fully connected structures, and the dimension of the output layer is the same as the dimension of the target spectrum.

[0013] Furthermore, the step of setting a minimum reconstruction error, setting variables controlled by the annealing algorithm using the parameters of at least eight of the filters as a group, constructing a simulated transmittance curve using the parameters of the filters, iteratively training the spectral simulation reconstruction network, calculating the error of each iteration, until the annealing algorithm calculates the optimal solution, obtaining the parameters of at least eight of the filters corresponding to the optimal solution, and recording them as the optimal parameters, includes the following sub-steps:

[0014] Set the minimum reconstruction error, enter the iterative training process, and set the annealing algorithm with the parameters of at least 8 of the filters as a group of variables;

[0015] The simulated transmittance curve is set according to the parameters of the filter, and a simulated sampling matrix is ​​constructed using the simulated transmittance curve;

[0016] The simulated sampling spectrum is obtained by performing matrix multiplication on the simulated sampling matrix, the preset relative spectrum, and the target spectrum.

[0017] The simulated sampled spectrum is input into the spectral simulation and reconstruction network for training to obtain the simulated reconstructed spectrum;

[0018] The reconstruction error between the simulated reconstructed spectrum and the target spectrum is calculated to complete one iteration of the training process. At this point, the reconstruction error is compared with the minimum reconstruction error, wherein:

[0019] If the reconstruction error is not less than the minimum reconstruction error, then the parameters of the filter controlled by the annealing algorithm are adjusted, and the next round of the iterative training process begins.

[0020] If the reconstruction error is less than the minimum reconstruction error, then the value of the reconstruction error is assigned to the minimum reconstruction error, and the parameters of the filter controlled by the corresponding annealing algorithm are saved.

[0021] Furthermore, after the step of assigning the value of the reconstruction error to the minimum reconstruction error and saving the corresponding parameters of the filter controlled by the annealing algorithm if the reconstruction error is less than the minimum reconstruction error, the method further includes:

[0022] The annealing algorithm is used to calculate whether the current variable has reached the optimal solution, where:

[0023] If the annealing algorithm does not obtain the optimal solution, the parameters of the filter controlled by the annealing algorithm are adjusted, and the next round of the iterative training process begins.

[0024] If the annealing algorithm obtains the optimal solution, the iterative training process is stopped, and the parameters of at least 8 of the filters corresponding to the optimal solution are obtained, denoted as the optimal parameters.

[0025] Furthermore, the spectral simulation reconstruction network is set to a batch size of 350 and a maximum number of iterations of 800 in the iterative training process, and the reconstruction error is calculated as mean absolute error.

[0026] Furthermore, the corresponding filter is selected based on the optimal parameters, and a spectral reconstruction network with the same structure as the spectral simulation and reconstruction network is trained based on the actual transmittance curve of the filter. Simultaneously, the selected filters are used to form a spectral sampling unit, and the target spectrum is sampled through the spectral sampling unit to obtain a mixed spectral image. This includes the following sub-steps:

[0027] The spectral reconstruction network is constructed, and the spectral reconstruction network has the same structure as the spectral simulation reconstruction network. The corresponding filter is selected through the optimal parameters, and the actual transmittance curve is constructed according to the filter. The spectral reconstruction network is trained and the trained spectral reconstruction network is saved.

[0028] The target spectrum is sampled using a spectral sampling unit, wherein each selected filter is assembled into a different lens and all the lenses are set into a combination structure with parallel optical axes. Each lens obtains a spectral feature image with a specific gray value through sampling.

[0029] The set of spectral feature images obtained by sampling all the lenses is taken as the mixed spectral image.

[0030] Furthermore, the step of registering the mixed spectral image to obtain a registered spectral image includes the following sub-steps:

[0031] The image plane distance and image plane pixel distance are calculated based on the image plane, focal length, object distance, the interval between the lenses, and the pixel size.

[0032] Construct an image plane coordinate relationship matrix between the lenses based on the image plane distance and the image plane pixel distance;

[0033] Using any of the spectral feature images as a projection image, the corresponding points between the projection image and all other spectral feature images are obtained according to the image plane coordinate relationship matrix and using a preset corresponding point algorithm.

[0034] The projection transformation matrix between the projected image and all other spectral feature images is calculated based on the corresponding points, wherein the robustness of the projection transformation matrix is ​​ensured by using the RANSAC algorithm.

[0035] The corresponding spectral feature image is projected and the black borders are cropped according to the projection transformation matrix to obtain the registered spectral image registered with the projected image.

[0036] In a second aspect, embodiments of the present invention also provide a hyperspectral computational imaging system, comprising:

[0037] A filter parameter acquisition module is used to acquire parameters of at least 8 filters;

[0038] The network building module is used to construct spectral reconstruction networks;

[0039] The filter selection module is used to set the minimum reconstruction error, and set the variables controlled by the annealing algorithm in a group of parameters of at least 8 of the filters, and iteratively train the spectral simulation reconstruction network, calculate the error of each iteration, until the annealing algorithm calculates the optimal solution, obtains the parameters of at least 8 of the filters corresponding to the optimal solution, and records them as the optimal parameters.

[0040] The sampling module is used to train a spectral reconstruction network with the same structure as the spectral simulation reconstruction network according to the optimal parameters, and at the same time select the filter corresponding to the optimal parameters to form a spectral sampling unit, and sample the target spectrum through the spectral sampling unit to obtain a mixed spectral image;

[0041] The registration module is used to register the mixed spectral image to obtain a registered spectral image;

[0042] The spectral reconstruction module is used to input the registered spectral image into the spectral reconstruction network to obtain a reconstructed spectral image.

[0043] Thirdly, embodiments of the present invention also provide a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the hyperspectral computational imaging method for multispectral acquisition as described in any of the above embodiments.

[0044] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the hyperspectral computational imaging method for multispectral acquisition as described in any of the above embodiments.

[0045] The beneficial effects achieved by this invention are as follows: by using compressed sensing technology to acquire mixed spectra, the number of data points is reduced by an order of magnitude, thereby increasing the speed of spectral acquisition. At the same time, it also uses a real-time registration algorithm to solve the registration problem of snapshot-type spectral acquisition, and uses neural networks for optimization during spectral reconstruction, further improving the accuracy of hyperspectral computational imaging. Attached Figure Description

[0046] Figure 1 This is a flowchart of the steps for hyperspectral computational imaging provided in an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of the structure of the spectral simulation reconstruction network provided in an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the overall process for obtaining the final transmittance curve provided in an embodiment of the present invention;

[0049] Figure 4 This is a flowchart of a sub-step in step S103 of the hyperspectral computational imaging method for multispectral acquisition provided in this embodiment of the invention.

[0050] Figure 5 This is a schematic diagram of the simulated transmittance curve provided in an embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram of the simulation sampling process provided in an embodiment of the present invention;

[0052] Figure 7 This is a flowchart of the sub-step of step S104 in the hyperspectral computational imaging method for multispectral acquisition provided in this embodiment of the invention;

[0053] Figure 8 This is a schematic diagram of the lens arrangement of the spectral sampling unit provided in an embodiment of the present invention;

[0054] Figure 9 This is a schematic diagram of the imaging scene and registration algorithm of the spectral sampling unit provided in an embodiment of the present invention;

[0055] Figure 10 This is a flowchart of a sub-step in step S105 of hyperspectral computational imaging provided in an embodiment of the present invention;

[0056] Figure 11 This is a schematic diagram of the structure of the hyperspectral computational imaging system 200 provided in an embodiment of the present invention;

[0057] Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0059] Please refer to Figure 1 , Figure 1 This is a flowchart of the steps for hyperspectral computational imaging provided in an embodiment of the present invention, specifically including the following steps:

[0060] S101. Obtain the parameters of at least 8 filters.

[0061] Specifically, in this embodiment of the invention, at least 8 of the filters have different peaks. Obtaining data from at least 8 of the filters is based on the fact that in actual applications, it is preferable to use at least 8 filters to achieve a better sampling effect.

[0062] S102. Construct a spectral simulation and reconstruction network.

[0063] For details, please see Figure 2 , Figure 2This is a schematic diagram of the spectral simulation and reconstruction network provided in an embodiment of the present invention. The spectral simulation and reconstruction network includes an input layer, a convolutional layer, a dimensionality reduction layer, a fully connected layer, and an output layer. In this embodiment, the dimension of the input layer is (batch_size, 8, 1), corresponding to a set of parameters for eight filters that need to be input into the spectral simulation and reconstruction network. Unless otherwise specified, in this embodiment, the simulated transmittance curve is constructed using eight filters as a group as the input to the spectral simulation and reconstruction network. The selection of eight filters as a group ensures sufficient accuracy in spectral sampling. In practical applications, the more filters there are, the higher the accuracy is achieved, but the corresponding hardware cost will be higher. The eight filters selected in this embodiment represent a trade-off between sampling accuracy and other factors in practical applications, and are not intended to limit the invention. The convolutional layer includes three groups... The cross-layer connection feature extraction structure consists of a 1D convolution, three 1D convolution layers, a Dropout layer, a cross-layer connection, and an upsampling layer arranged sequentially. The cross-layer connection is used to perform a residual connection between the data before the first 1D convolution and the data processed by the Dropout layer. The dimensionality reduction layer is used to reduce the dimensionality of the data to one dimension, and then it is further processed by the three fully connected layers. Finally, the data is output through the output layer. For example, in this embodiment, the batch size is set to 350. In the input layer, the data dimension is 8. The data is processed according to the order of the spectral simulation reconstruction network. After the dimensionality reduction layer, the output data dimension is 2048. After the fully connected layer, the data dimension is 101. Therefore, the output data dimension is (batch size, 101).

[0064] In this embodiment of the invention, a preset mean squared error (MSE) is used as the loss function. The standard MSE formula is:

[0065]

[0066] in, For the true sample values, The predicted value output by the spectral simulation reconstruction network is used as the loss function. To meet the actual needs of the target spectrum, the weights of key wavelengths need to be increased. For example, in this embodiment of the invention, the weights of wavelengths 450nm, 555nm, 660nm, 750nm, and 840nm are increased, further transforming the formula of the loss function into:

[0067]

[0068] In addition, the spectral simulation reconstruction network uses Adam as the optimizer.

[0069] S103. Set the minimum reconstruction error, and set the variables controlled by the annealing algorithm in a group using the parameters of at least 8 of the filters. Construct a simulated transmittance curve using the parameters of the filters, and iteratively train the spectral simulation reconstruction network. Calculate the error of each iteration until the annealing algorithm calculates the optimal solution. Obtain the parameters of at least 8 of the filters corresponding to the optimal solution, and record them as the optimal parameters.

[0070] Please refer to the following at the same time Figure 3 and Figure 4 , Figure 3 This is a schematic diagram of the overall process for obtaining the final transmittance curve provided in an embodiment of the present invention. Figure 4 This is a flowchart of a sub-step in step S103 of the hyperspectral computational imaging method for multispectral acquisition provided in this embodiment of the invention, specifically including the following sub-steps:

[0071] S1031. Set the minimum reconstruction error and enter the iterative training process. Set the annealing algorithm with the parameters of at least 8 of the filters as a group of variables.

[0072] The annealing algorithm is a probability-based optimization algorithm originally used to describe the process of heating a solid to a sufficiently high temperature and then slowly cooling it. During heating, the particles inside the solid become disordered as the temperature rises, and the internal energy increases. During the cooling process, the particles gradually become more ordered, reaching an equilibrium state at each temperature, and finally reaching the ground state at room temperature, where the internal energy is reduced to a minimum. The algorithm calculates the optimal solution for the equilibrium state based on this physical phenomenon. In this embodiment of the invention, a minimum reconstruction error for the output of the spectral simulation reconstruction network is first set. Then, the iterative training process of the spectral simulation reconstruction network begins. The parameters of at least eight filters are used as a group as variables to control the annealing algorithm, and the annealing algorithm is configured.

[0073] S1032. Set the simulated transmittance curve according to the parameters of the filter, and construct a simulated sampling matrix through the simulated transmittance curve.

[0074] The simulated transmittance curves are constructed based on the parameters of at least eight of the filters. For example, please refer to... Figure 5 , Figure 5 This is a schematic diagram of the simulated transmittance curve provided in an embodiment of the present invention. Figure 5Taking the first set of data as an example, the simulated transmittance curve can be represented as a vector [2, 710, 750, 840, 890] based on the number of peaks and the start and end bands of each peak. The start and end bands of each simulated transmittance curve are extended by 10 nm from the peak. The simulated sampling matrix is ​​constructed based on the simulated transmittance curve. Specifically, the simulated transmittance curve includes the transmittance data of 8 filters. Before matrix multiplication, the data of these 8 filters need to be constructed into a matrix form similar to the target spectrum according to the compressed sensing principle, i.e., the simulated sampling matrix.

[0075] S1033. Perform matrix multiplication calculation on the simulation sampling matrix and the preset relative spectrum with the target spectrum to obtain the simulation sampling spectrum.

[0076] Based on the principle of compressed sensing, the compressed acquisition of spectral signals during the spectral acquisition process satisfies the following formula:

[0077] F = S·R + noise

[0078] Where F represents the acquired mixed signal, i.e., the simulated sampling spectrum, S represents the true spectrum of each pixel, R represents the compressed sampling matrix, S and R are multiplied by matrix, and noise represents the unavoidable noise signal during the acquisition process, including sensor noise, noise caused by stray light from the filter, etc. For details, please refer to... Figure 6 , Figure 6 This is a schematic diagram of the simulation sampling process provided in an embodiment of the present invention. In this embodiment, S is specifically the target spectrum, which is an existing spectrum with complete sampling data and reconstructed data. In this embodiment, it is used as comparison data. R is the simulation difference matrix, and L is the noise signal. L is specifically composed of the lens, the sensor, and the relative spectrum of the target spectrum. In the matrix multiplication calculation process, R and L are multiplied point by point, and then multiplied by S to finally obtain the simulation sampling spectrum F.

[0079] S1034. Input the simulated sampled spectrum into the spectral simulation reconstruction network for training to obtain the simulated reconstructed spectrum.

[0080] The simulated sampled spectrum is input as training data into the spectral simulation and reconstruction network, and the spectral simulation and reconstruction network performs spectral reconstruction to obtain the simulated reconstructed spectrum based on the simulated sampled spectrum.

[0081] S1035. Calculate the reconstruction error between the simulated reconstructed spectrum and the target spectrum to complete one iteration of the training process. At this time, the reconstruction error is compared with the minimum reconstruction error.

[0082] Specifically, the simulated reconstructed spectrum is obtained based on the calculated spectral sampling matrix, therefore it has a reconstruction error compared to the existing target spectrum. The reconstruction error is calculated using the MAE (Mean Absolute Error) algorithm, which satisfies the following formula:

[0083]

[0084] Where n is determined by the number of spectral curves in the target spectrum, and in this embodiment of the invention, the value is 8; actual is the value of the spectral curve in the target spectrum; predicted is the value of the spectral curve in the reconstructed spectrum; the reconstruction error between the simulated reconstructed spectrum and the target spectrum is calculated using MAE and denoted as m1. By comparing the magnitude of the reconstruction error and the minimum reconstruction error, the following cases are further distinguished:

[0085] 1035a. If the reconstruction error is not less than the minimum reconstruction error, then adjust the parameters of the filter controlled by the annealing algorithm and start the next round of the iterative training process.

[0086] If the reconstruction error is not less than the minimum reconstruction error, then the parameters of at least 8 filters controlled by the annealing algorithm are adjusted, the parameters of at least one of the filters are replaced with different parameters, and the iterative training process of step S1032 above is restarted to calculate the reconstruction error of the simulated reconstructed spectrum obtained based on the parameters of another set of filters.

[0087] 1035b. If the reconstruction error is less than the minimum reconstruction error, then the value of the reconstruction error is assigned to the minimum reconstruction error, and the parameters of at least 8 filters controlled by the corresponding annealing algorithm are saved.

[0088] If the reconstruction error is less than the minimum reconstruction error, then the value of the reconstruction error is assigned to the minimum reconstruction error, that is, the value of the minimum reconstruction error is changed to m1. Then, the parameters of at least eight filters used in the current iterative training process are saved, and it is further calculated whether the parameters of the current at least eight filters have reached the optimal solution in the annealing algorithm, that is, whether using the current filter parameters to calculate the spectral simulation reconstruction network can achieve the minimum reconstruction error. The calculation is further divided according to whether the optimal solution is obtained in the annealing algorithm:

[0089] 1036a. If the annealing algorithm does not obtain the optimal solution, the parameters of the filter controlled by the annealing algorithm are adjusted, and the next round of the iterative training process begins.

[0090] If the annealing algorithm does not obtain the optimal solution, the parameters of the filter controlled by the annealing algorithm are adjusted, the parameters of at least one of the filters are replaced with different parameters, and the iterative training process of step S1032 above is restarted to calculate the reconstruction error of the simulated reconstructed spectrum obtained based on the parameters of another set of filters, and further calculate the optimal solution.

[0091] 1036b. If the annealing algorithm obtains the optimal solution, the iterative training process is stopped, and the parameters of at least 8 of the filters corresponding to the optimal solution are obtained, denoted as the optimal parameters.

[0092] If the annealing algorithm obtains the optimal solution, the iterative training process is stopped. The parameters of at least 8 filters stored in 1035b are used as the optimal parameters. For example, the maximum number of iterations of the iterative training process is set to 800 so that the annealing algorithm will not excessively pursue the optimal solution when the amount of data is too large.

[0093] S104. Select the corresponding filter according to the optimal parameters, and train a spectral reconstruction network with the same structure as the spectral simulation reconstruction network according to the actual transmittance curve of the filter. At the same time, form a spectral sampling unit with the selected filter, and sample the target spectrum through the spectral sampling unit to obtain a mixed spectral image.

[0094] Please refer to Figure 7 , Figure 7 This is a flowchart of a sub-step in step S104 of the hyperspectral computational imaging method for multispectral acquisition provided in this embodiment of the invention, specifically including the following sub-steps:

[0095] S1041. Construct the spectral reconstruction network, which has the same structure as the spectral simulation reconstruction network. Select the corresponding filter through the optimal parameters, construct the actual transmittance curve based on the filter, train the spectral reconstruction network, and save the trained spectral reconstruction network.

[0096] Specifically, the spectral reconstruction network has the same structure as the spectral simulation reconstruction network constructed in step S102. However, the spectral simulation reconstruction network uses different filter parameters during the training phase, which affects the parameters and weights in the structure. In order to further improve the progress of spectral reconstruction, the filter with corresponding parameters is selected using the obtained optimal parameters. Unlike the above-mentioned method of constructing the simulated transmittance curve, the actual parameters of the filter are obtained using actual measurement and other methods, and the actual transmittance curve is constructed. Based on the actual transmittance curve, the spectral reconstruction network is trained according to the method of steps S1033 to S1034, so that the spectral reconstruction network can obtain accurate results when performing spectral reconstruction with the optimal parameters.

[0097] S1042. The target spectrum is sampled using a spectral sampling unit, wherein the spectral sampling unit is a combination structure in which each selected filter is assembled into a different lens and all the lenses are set to have parallel optical axes. Each lens obtains a spectral feature image with a specific gray value through sampling.

[0098] The optimal parameters correspond to eight filters with different parameters. These filters are assembled into lenses of the same specifications, creating a parallel optical axis combination structure between the lenses, thus obtaining the spectral sampling unit. In this embodiment, the parallel optical axis combination structure refers to a structure where the lenses are arranged on the same plane, and the optical axes of each lens are parallel but not coincident. Please refer to [reference needed]. Figure 8 , Figure 8 This is a schematic diagram of the lens arrangement of the spectral sampling unit provided in an embodiment of the present invention. The lenses of the spectral sampling unit form a 3×3 matrix structure, and the spacing between adjacent lenses is equal. In this embodiment of the present invention, only 8 sets of lenses are required. Therefore, one lens position can be selected to be covered. In order to facilitate further calculation of the imaging error caused by the lens spacing, it is preferable to cover one lens position at the corner of the matrix. In addition, 8 lenses are only one optimal embodiment, and the number is not limited to this. The spectral sampling unit with more than 8 lenses needs to pay extra attention to whether the imaging range of all lenses is within the range of the imaging target. Therefore, when more lenses are needed to form the spectral sampling unit, their combination position and shape should be as close as possible, and the spacing between the lenses should be the same.

[0099] S1043. The set of spectral feature images obtained by sampling all the lenses is taken as the mixed spectral image.

[0100] Each lens in the spectral sampling unit is equipped with the filter. Under the influence of the filter, each lens will capture a spectral feature image with a specific gray value. The set of spectral feature images obtained by the eight lenses is used as the mixed spectral image.

[0101] S105. Register the mixed spectral image to obtain a registered spectral image.

[0102] Please refer to the following at the same time Figure 9 and Figure 10 , Figure 9 This is a schematic diagram of the imaging scene and registration algorithm of the spectral sampling unit provided in an embodiment of the present invention. Figure 10 This is a flowchart of a sub-step in step S105 of hyperspectral computational imaging provided in an embodiment of the present invention, specifically including the following sub-steps:

[0103] S1051. Calculate the image plane distance and image plane pixel distance based on the image plane of the lens, focal length, object distance, the interval between the lenses, and the pixel size.

[0104] Specifically, the imaging planes of any two adjacent lenses in the spectral sampling unit are defined as S1 and S2, the focal length of each lens is f, the object distance from the lens to the imaging object is d, the interval between two adjacent lenses is x, the pixel size is δu, the image plane distance is u, and the image plane pixel distance is p. Then, according to the imaging principle, the image plane distance u satisfies the following formula:

[0105]

[0106] Furthermore, the pixel distance p of the image plane satisfies the following formula:

[0107]

[0108] S1052. Construct an image plane coordinate relationship matrix between the lenses based on the image plane distance and the image plane pixel distance.

[0109] According to the formula for the image plane pixel distance p, when the interval x, the focal length f, and the pixel size δu are constant, the image plane pixel distance p and the object distance d are uniquely correlated. Therefore, when the object distance and the plane where the object is located are fixed, two adjacent lenses will capture two images located at different center points in the same plane, and the distance between the pixel coordinates in the two images is also a constant. Therefore, the image plane coordinate relationship matrix based on the image plane coordinates can be constructed according to the image plane corresponding to the lens.

[0110] S1053. Using any one of the spectral feature images as a projection image, and based on the image plane coordinate relationship matrix and using a preset corresponding point algorithm, obtain the corresponding points between the projection image and all other spectral feature images.

[0111] The spectral feature image captured by any one of the eight lenses is used as the projection image. Combined with the image plane coordinate relationship matrix, a preset correspondence point algorithm is used to obtain the correspondence points between the projection image and all other spectral feature images. Specifically, the preset correspondence point algorithm can be a Scale Invariant Feature Transform (SIFT) algorithm or a corner detection algorithm. The SIFT algorithm is a commonly used correspondence point extraction algorithm that can be used for targets with a large number of size-invariant features. The corner detection algorithm requires the target to be a checkerboard pattern image. In this embodiment, all the spectral feature images meet the requirements of the above algorithms. By using the correspondence point algorithm, the correspondence points between the projection image and any other spectral feature image can be obtained. Besides the projection image, there are seven other spectral feature images, therefore, there are seven sets of correspondence points. The projection transformation matrix between the projected image and all other spectral feature images is calculated using an image projection transformation method. Specifically, the image projection transformation method is a commonly used image processing method used to project the original image onto a new plane to obtain a projected image. For example, the spectral feature image captured by the lens located at the center of the matrix structure of the spectral sampling unit is taken as the projected image, and the spectral feature images captured by the other 7 lenses located around the matrix are calculated to obtain 7 sets of projection transformation matrices.

[0112] S1054. Calculate the projection transformation matrix between the projected image and all other spectral feature images based on the corresponding points, wherein the robustness of the projection transformation matrix is ​​ensured by using the RANSAC algorithm.

[0113] The projection transformation matrix between the projected image and the other spectral feature images is calculated according to the corresponding points. In this embodiment of the invention, seven projection transformation matrices can be obtained. At the same time, the RANSAC algorithm (Random Sample Consensus Algorithm) is used to process the projection transformation matrix in the process of obtaining the projection transformation matrix to ensure the robustness of the projection transformation matrix.

[0114] S1055. Based on the projection transformation matrix, the corresponding spectral feature image is projected and the black border is cropped to obtain the registered spectral image registered with the projected image.

[0115] By calculating the seven sets of projection transformation matrices, the spectral feature images are subjected to projection transformation and black border removal respectively, so that all the spectral feature images correspond to the positions of the projected images, thereby completing the registration of the spectral feature images and obtaining the registered spectral images.

[0116] S106. Input the registered spectral image into the spectral reconstruction network to obtain the reconstructed spectral image.

[0117] The registered spectral image obtained in the above steps is used as input data to the spectral reconstruction network obtained in step S1041 above, and hyperspectral computational imaging is performed through the spectral reconstruction network to output the reconstructed spectral image corresponding to the registered spectral feature image.

[0118] The beneficial effects achieved by this invention are as follows: by using compressed sensing technology to acquire mixed spectra, the number of data points is reduced by an order of magnitude, thereby increasing the speed of spectral acquisition. At the same time, it also uses a real-time registration algorithm to solve the registration problem of snapshot-type spectral acquisition, and uses neural networks for optimization during spectral reconstruction, further improving the accuracy of hyperspectral computational imaging.

[0119] This invention also provides a hyperspectral computational imaging system, please refer to... Figure 11 , Figure 11 This is a schematic diagram of the structure of a hyperspectral computational imaging system 200 provided in an embodiment of the present invention. The hyperspectral computational imaging system 200 includes:

[0120] The filter parameter acquisition module 201 is used to acquire the parameters of at least 8 filters;

[0121] Network building module 202 is used to build a spectral simulation and reconstruction network;

[0122] The filter selection module 203 is used to set the minimum reconstruction error, and set the variables controlled by the annealing algorithm in a group of parameters of at least 8 of the filters, and iteratively train the spectral simulation reconstruction network, calculate the error of each iteration, until the annealing algorithm calculates the optimal solution, obtains the parameters of at least 8 of the filters corresponding to the optimal solution, and records them as the optimal parameters.

[0123] The sampling module 204 is used to select the corresponding filter according to the optimal parameters, and train a spectral reconstruction network with the same structure as the spectral simulation reconstruction network according to the actual transmittance curve of the filter. At the same time, the selected filters are used to form a spectral sampling unit, and the target spectrum is sampled through the spectral sampling unit to obtain a mixed spectral image.

[0124] Registration module 205 is used to register the mixed spectral image to obtain a registered spectral image;

[0125] The spectral reconstruction module 206 is used to input the registered spectral image into the spectral reconstruction network to obtain a reconstructed spectral image.

[0126] The hyperspectral computational imaging system 200 can implement the steps in the hyperspectral computational imaging method with multispectral acquisition as described in the above embodiments, and can achieve the same technical effect. Refer to the description in the above embodiments, which will not be repeated here.

[0127] This invention also provides a computer device, please refer to... Figure 12 , Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. The computer device 300 includes: a memory 302, a processor 301, and a computer program stored in the memory 302 and executable on the processor 301.

[0128] The processor 301 calls the computer program stored in the memory 302 to execute the steps in the park management method provided in this embodiment of the invention. Please refer to... Figure 1 Specifically, it includes:

[0129] S101. Obtain the parameters of at least 8 filters.

[0130] S102. Construct a spectral simulation and reconstruction network.

[0131] Furthermore, the spectral simulation reconstruction network includes an input layer, a network convolutional layer, a dimensionality reduction layer, a fully connected layer, and an output layer. The network convolutional layer includes three sets of cross-layer connected feature extraction structures, the fully connected layer includes three sets of fully connected structures, and the dimension of the output layer is the same as the dimension of the target spectrum.

[0132] S103. Set the minimum reconstruction error, and set the variables controlled by the annealing algorithm in a group using the parameters of at least 8 of the filters. Construct a simulated transmittance curve using the parameters of the filters, and iteratively train the spectral simulation reconstruction network. Calculate the error of each iteration until the annealing algorithm calculates the optimal solution. Obtain the parameters of at least 8 of the filters corresponding to the optimal solution, and record them as the optimal parameters.

[0133] Furthermore, the step of setting a minimum reconstruction error, setting variables controlled by the annealing algorithm in a group based on the parameters of at least eight of the filters, iteratively training the spectral simulation reconstruction network, calculating the error of each iteration, until the annealing algorithm calculates the optimal solution, obtaining the parameters of at least eight of the filters corresponding to the optimal solution, and recording them as the optimal parameters, includes the following sub-steps:

[0134] Set the minimum reconstruction error, enter the iterative training process, and set the annealing algorithm with the parameters of at least 8 of the filters as a group of variables;

[0135] The simulated transmittance curve is set according to the parameters of the filter, and a simulated sampling matrix is ​​constructed using the simulated transmittance curve;

[0136] The simulated sampling spectrum is obtained by performing matrix multiplication on the simulated sampling matrix, the preset relative spectrum, and the target spectrum.

[0137] The simulated sampled spectrum is input into the spectral simulation and reconstruction network for training to obtain the simulated reconstructed spectrum;

[0138] The reconstruction error between the simulated reconstructed spectrum and the target spectrum is calculated to complete one iteration of the training process. At this point, the reconstruction error is compared with the minimum reconstruction error, wherein:

[0139] If the reconstruction error is not less than the minimum reconstruction error, then the parameters of the filter controlled by the annealing algorithm are adjusted, and the next round of the iterative training process begins.

[0140] If the reconstruction error is less than the minimum reconstruction error, then the value of the reconstruction error is assigned to the minimum reconstruction error, and the parameters of the filter controlled by the corresponding annealing algorithm are saved.

[0141] Furthermore, after the step of assigning the value of the reconstruction error to the minimum reconstruction error and saving the corresponding parameters of the filter controlled by the annealing algorithm if the reconstruction error is less than the minimum reconstruction error, the method further includes:

[0142] The annealing algorithm is used to calculate whether the current variable has reached the optimal solution, where:

[0143] If the annealing algorithm does not obtain the optimal solution, the parameters of the filter controlled by the annealing algorithm are adjusted, and the next round of the iterative training process begins.

[0144] If the annealing algorithm obtains the optimal solution, the iterative training process is stopped, and the parameters of at least 8 of the filters corresponding to the optimal solution are obtained, denoted as the optimal parameters.

[0145] Furthermore, the spectral simulation reconstruction network is set to a batch size of 350 and a maximum number of iterations of 800 in the iterative training process, and the reconstruction error is calculated as mean absolute error.

[0146] S104. Select the corresponding filter according to the optimal parameters, and train a spectral reconstruction network with the same structure as the spectral simulation reconstruction network according to the actual transmittance curve of the filter. At the same time, form a spectral sampling unit with the selected filter, and sample the target spectrum through the spectral sampling unit to obtain a mixed spectral image.

[0147] Furthermore, the corresponding filter is selected based on the optimal parameters, and a spectral reconstruction network with the same structure as the spectral simulation and reconstruction network is trained based on the actual transmittance curve of the filter. Simultaneously, the selected filters are used to form a spectral sampling unit, and the target spectrum is sampled through the spectral sampling unit to obtain a mixed spectral image. This includes the following sub-steps:

[0148] The spectral reconstruction network is constructed, and the spectral reconstruction network has the same structure as the spectral simulation reconstruction network. The corresponding filter is selected through the optimal parameters, and the actual transmittance curve is constructed according to the filter. The spectral reconstruction network is trained and the trained spectral reconstruction network is saved.

[0149] The target spectrum is sampled using a spectral sampling unit, wherein each selected filter is assembled into a different lens and all the lenses are set into a combination structure with parallel optical axes. Each lens obtains a spectral feature image with a specific gray value through sampling.

[0150] The set of spectral feature images obtained by sampling all the lenses is taken as the mixed spectral image.

[0151] S105. Register the mixed spectral image to obtain a registered spectral image.

[0152] Furthermore, the step of registering the mixed spectral image to obtain a registered spectral image includes the following sub-steps:

[0153] The image plane distance and image plane pixel distance are calculated based on the image plane, focal length, object distance, the interval between the lenses, and the pixel size.

[0154] Construct an image plane coordinate relationship matrix between the lenses based on the image plane distance and the image plane pixel distance;

[0155] Using any of the spectral feature images as a projection image, the corresponding points between the projection image and all other spectral feature images are obtained according to the image plane coordinate relationship matrix and using a preset corresponding point algorithm.

[0156] The projection transformation matrix between the projected image and all other spectral feature images is calculated based on the corresponding points, wherein the robustness of the projection transformation matrix is ​​ensured by using the RANSAC algorithm.

[0157] The corresponding spectral feature image is projected and the black borders are cropped according to the projection transformation matrix to obtain the registered spectral image registered with the projected image.

[0158] S106. Input the registered spectral image into the spectral reconstruction network to obtain the reconstructed spectral image.

[0159] The computer device 300 provided in this embodiment of the invention can implement the steps in the hyperspectral computational imaging method for multispectral acquisition as described in the above embodiments, and can achieve the same technical effect. Refer to the description in the above embodiments, which will not be repeated here.

[0160] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes and steps of the hyperspectral computational imaging method for multispectral acquisition provided in this invention, and achieves the same technical effect. To avoid repetition, these will not be described again here.

[0161] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0162] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0163] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0164] The embodiments of the present invention have been described above with reference to the accompanying drawings. The disclosed embodiments are merely preferred embodiments of the present invention. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many equivalent changes in form without departing from the spirit and scope of the claims of the present invention, and all such changes are within the protection scope of the present invention.

Claims

1. A hyperspectral computational imaging method using multispectral acquisition, characterized in that, The method includes the following steps: Obtain the parameters of at least 8 filters; Construct a spectral simulation and reconstruction network; Set a minimum reconstruction error, and set the variables controlled by the annealing algorithm in a group using the parameters of at least 8 of the filters. Construct a simulated transmittance curve using the parameters of the filters, and iteratively train the spectral simulation reconstruction network. Calculate the error of each iteration until the annealing algorithm calculates the optimal solution. Obtain the parameters of at least 8 of the filters corresponding to the optimal solution, and record them as the optimal parameters. The corresponding filter is selected according to the optimal parameters, and a spectral reconstruction network with the same structure as the spectral simulation reconstruction network is trained according to the actual transmittance curve of the filter. At the same time, the selected filters are used to form a spectral sampling unit, and the target spectrum is sampled through the spectral sampling unit to obtain a mixed spectral image. The mixed spectral image is registered to obtain a registered spectral image; The registered spectral image is input into the spectral reconstruction network to obtain the reconstructed spectral image; The process includes selecting the corresponding filter based on the optimal parameters, training a spectral reconstruction network with the same structure as the spectral simulation and reconstruction network based on the actual transmittance curve of the filter, assembling the filters into spectral sampling units, and sampling the target spectrum through these units to obtain a mixed spectral image. This process includes the following sub-steps: The spectral reconstruction network is constructed, and the spectral reconstruction network has the same structure as the spectral simulation reconstruction network. The corresponding filter is selected through the optimal parameters, and the actual transmittance curve is constructed according to the filter. The spectral reconstruction network is trained and the trained spectral reconstruction network is saved. The target spectrum is sampled using a spectral sampling unit, wherein each selected filter is assembled into a different lens and all the lenses are set into a combination structure with parallel optical axes. Each lens obtains a spectral feature image with a specific gray value through sampling. The set of spectral feature images obtained by sampling all the lenses is taken as the mixed spectral image; The step of registering the mixed spectral image to obtain a registered spectral image includes the following sub-steps: The image plane distance and image plane pixel distance are calculated based on the image plane, focal length, object distance, spacing between the lenses, and pixel size of the lens. Construct an image plane coordinate relationship matrix between the lenses based on the image plane distance and the image plane pixel distance; Using any of the spectral feature images as a projection image, the corresponding points between the projection image and all other spectral feature images are obtained according to the image plane coordinate relationship matrix and using a preset corresponding point algorithm. The projection transformation matrix between the projected image and all other spectral feature images is calculated based on the corresponding points, wherein the robustness of the projection transformation matrix is ​​ensured by using the RANSAC algorithm. The corresponding spectral feature image is projected and the black borders are cropped according to the projection transformation matrix to obtain the registered spectral image registered with the projected image.

2. The hyperspectral computational imaging method for multispectral acquisition as described in claim 1, characterized in that, The spectral simulation reconstruction network includes an input layer, a network convolutional layer, a dimensionality reduction layer, a fully connected layer, and an output layer. The network convolutional layer includes three sets of cross-layer connected feature extraction structures, the fully connected layer includes three sets of fully connected structures, and the dimension of the output layer is the same as the dimension of the target spectrum.

3. The hyperspectral computational imaging method for multispectral acquisition as described in claim 1, characterized in that, The step of setting a minimum reconstruction error, setting variables controlled by the annealing algorithm using parameters of at least eight filters as a group, constructing a simulated transmittance curve using the filter parameters, iteratively training the spectral simulation reconstruction network, calculating the error of each iteration, until the annealing algorithm calculates the optimal solution, obtaining the parameters of at least eight filters corresponding to the optimal solution, and recording them as the optimal parameters, includes the following sub-steps: Set the minimum reconstruction error, enter the iterative training process, and set the annealing algorithm with the parameters of at least 8 of the filters as a group of variables; The simulated transmittance curve is set according to the parameters of the filter, and a simulated sampling matrix is ​​constructed using the simulated transmittance curve; The simulated sampling spectrum is obtained by performing matrix multiplication on the simulated sampling matrix, the preset relative spectrum, and the target spectrum. The simulated sampled spectrum is input into the spectral simulation and reconstruction network for training to obtain the simulated reconstructed spectrum; The reconstruction error between the simulated reconstructed spectrum and the target spectrum is calculated to complete one iteration of the training process. At this point, the reconstruction error is compared with the minimum reconstruction error, wherein: If the reconstruction error is not less than the minimum reconstruction error, then the parameters of the filter controlled by the annealing algorithm are adjusted, and the next round of the iterative training process begins. If the reconstruction error is less than the minimum reconstruction error, then the value of the reconstruction error is assigned to the minimum reconstruction error, and the parameters of the filter controlled by the corresponding annealing algorithm are saved.

4. The hyperspectral computational imaging method for multispectral acquisition as described in claim 3, characterized in that, If the reconstruction error is less than the minimum reconstruction error, then after assigning the value of the reconstruction error to the minimum reconstruction error and saving the corresponding parameters of the filter controlled by the annealing algorithm, the method further includes: The annealing algorithm is used to calculate whether the current variable has reached the optimal solution, where: If the annealing algorithm does not obtain the optimal solution, the parameters of the filter controlled by the annealing algorithm are adjusted, and the next round of the iterative training process begins. If the annealing algorithm obtains the optimal solution, the iterative training process is stopped, and the parameters of at least 8 of the filters corresponding to the optimal solution are obtained, denoted as the optimal parameters.

5. The hyperspectral computational imaging method for multispectral acquisition as described in claim 4, characterized in that, The spectral simulation reconstruction network is set to a batch size of 350 and a maximum number of iterations of 800 in the iterative training process. The reconstruction error is calculated as the mean absolute error.

6. A hyperspectral computational imaging system with multispectral acquisition, characterized in that, include: A filter parameter acquisition module is used to acquire parameters of at least 8 filters; The network building module is used to construct spectral simulation and reconstruction networks; The filter selection module is used to set the minimum reconstruction error and set the variables controlled by the annealing algorithm in a group of parameters of at least 8 of the filters. The module constructs a simulated transmittance curve through the parameters of the filters and iteratively trains the spectral simulation reconstruction network, calculates the error of each iteration, until the annealing algorithm calculates the optimal solution, obtains the parameters of at least 8 of the filters corresponding to the optimal solution, and records them as the optimal parameters. The sampling module is used to select the corresponding filter according to the optimal parameters, and train a spectral reconstruction network with the same structure as the spectral simulation reconstruction network according to the actual transmittance curve of the filter. At the same time, the selected filters are used to form a spectral sampling unit, and the target spectrum is sampled through the spectral sampling unit to obtain a mixed spectral image. The registration module is used to register the mixed spectral image to obtain a registered spectral image; The spectral reconstruction module is used to input the registered spectral image into the spectral reconstruction network to obtain a reconstructed spectral image. The sampling module is further used for: The spectral reconstruction network is constructed, and the spectral reconstruction network has the same structure as the spectral simulation reconstruction network. The corresponding filter is selected through the optimal parameters, and the actual transmittance curve is constructed according to the filter. The spectral reconstruction network is trained and the trained spectral reconstruction network is saved. The target spectrum is sampled using a spectral sampling unit, wherein each selected filter is assembled into a different lens and all the lenses are set into a combination structure with parallel optical axes. Each lens obtains a spectral feature image with a specific gray value through sampling. The set of spectral feature images obtained by sampling all the lenses is taken as the mixed spectral image; The registration module is also used for: The image plane distance and image plane pixel distance are calculated based on the image plane, focal length, object distance, spacing between the lenses, and pixel size of the lens. Construct an image plane coordinate relationship matrix between the lenses based on the image plane distance and the image plane pixel distance; Using any of the spectral feature images as a projection image, the corresponding points between the projection image and all other spectral feature images are obtained according to the image plane coordinate relationship matrix and using a preset corresponding point algorithm. The projection transformation matrix between the projected image and all other spectral feature images is calculated based on the corresponding points, wherein the robustness of the projection transformation matrix is ​​ensured by using the RANSAC algorithm. The corresponding spectral feature image is projected and the black borders are cropped according to the projection transformation matrix to obtain the registered spectral image registered with the projected image.

7. A computer device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the hyperspectral computational imaging method for multispectral acquisition as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the hyperspectral computational imaging method for multispectral acquisition as described in any one of claims 1 to 5.

Images