Spectrum reconstruction and camera spectrum sensitivity optimization system and method

Abstract

The invention relates to a spectrum reconstruction and camera spectrum sensitivity optimization system and method, and the method comprises the steps: setting a sensitivity-parameter device, a multispectral image generation module, a multispectral camera and a spectrum reconstruction module, and enabling the sensitivity-parameter device to carry out the spectrum reconstruction through a fusion algorithm of a convolution and residual network, and performing wave band mapping on the hyperspectral data extracted from the database to obtain a plurality of sensitivity values. Moreover, the sensitivity-parameter device can obtain filter structure parameters corresponding to the sensitivity values through a finite difference time domain method according to the sensitivity values, and the filters contained in the multispectral camera are in one-to-one correspondence with the filter structure parameters obtained by the sensitivity-parameter device. The sensitivity-parameter device not only realizes high-efficiency automatic acquisition of structural parameters of the filter, but also optimizes the spectral sensitivity of the camera, so that the combination of the multispectral camera and the spectral reconstruction module remarkably improves the spectral reconstruction precision.

Description

Technical Field

[0001] This invention relates to the field of computer modeling and systems technology, and more specifically, to a spectral reconstruction and camera spectral sensitivity optimization system and method. Background Technology

[0002] In recent years, with the development of spectral reconstruction technology, spectral reconstruction techniques for recovering hyperspectral information from single red-green-blue (RGB) images or multispectral images have become an efficient hyperspectral imaging solution. Traditional hyperspectral imaging techniques typically require specialized hyperspectral cameras, which are expensive and have slow data acquisition speeds. In contrast, spectral reconstruction techniques can use conventional red-green-blue (RGB) cameras or multispectral cameras to generate hyperspectral data through algorithms, thereby reducing equipment costs and improving data acquisition efficiency.

[0003] In spectral reconstruction technology, the final accuracy of the spectral reconstruction is mainly determined by the channel selection of the multispectral camera. Specifically, through reasonable channel selection, the accuracy of spectral reconstruction can be improved by more than 33%. Channel selection for a multispectral camera is essentially an optimization of the camera's spectral sensitivity. Traditional spectral sensitivity optimization methods are mostly based on manual design constraints, using genetic or evolutionary algorithms to search for the optimal filter combination from a limited pool of candidate filters. The drawback of these methods is that their improvement in spectral reconstruction accuracy is limited.

[0004] In response, existing technologies disclose a method based on an efficient convolutional neural network (CNN). This method jointly selects the best camera spectral response values ​​from candidate datasets and fits a mapping to recover the hyperspectral image. At the same time, the method uses a convolutional layer structure to simulate the spectral response of the filter and achieves high-precision spectral reconstruction through neural network optimization.

[0005] However, due to the characteristics of the convolutional layer structure, inconsistencies in the spectral response of the same pixel often occur when simulating the spectral response of a filter, making it difficult to complete the subsequent thin film design after spectral reconstruction. Furthermore, changes in the input scene alter the optimal spectral response curve, limiting the transfer efficiency and reconstruction accuracy of convolutional neural networks used for spectral reconstruction, ultimately failing to improve reconstruction accuracy. Summary of the Invention

[0006] The technical problem to be solved by the present invention is how to overcome the technical defects of existing convolutional neural network methods in spectral reconstruction technology, such as inconsistent spectral responses of the same pixel and inability to improve reconstruction accuracy. In order to overcome this technical defect, the present invention provides a spectral reconstruction and camera spectral sensitivity optimization system and method, specifically including a spectral reconstruction and camera spectral sensitivity optimization system and a spectral reconstruction and camera spectral sensitivity optimization method.

[0007] This invention provides a spectral reconstruction and camera spectral sensitivity optimization system, comprising: The sensitivity-parameter device is configured to perform band mapping on hyperspectral data extracted from the database using a fusion algorithm of convolution and residual networks to obtain multiple sensitivity values; then, based on each sensitivity value, the filter structure parameters corresponding to each sensitivity value are obtained through the finite-difference time-domain method. The multispectral image generation module is electrically connected to the sensitivity-parameter device and is configured to substitute each sensitivity value into a discrete form of the channel response value calculation formula to obtain the response value image corresponding to each sensitivity value, and to integrate the response value images corresponding to all sensitivity values ​​into a multispectral data image. A multispectral camera is set up to acquire multispectral images of the scene; The spectral reconstruction module is electrically connected to both the multispectral image generation module and the multispectral camera. It is configured to execute a hyperspectral reconstruction algorithm to obtain a reconstructed hyperspectral image using the hyperspectral data image during parameter optimization, and to obtain a hyperspectral image using the multispectral image of the scene during non-parametric optimization. The number of filters in the multispectral camera is equal to the number of sensitivity values, and the filters correspond one-to-one with the filter structure parameters obtained by the sensitivity-parameter device.

[0008] The spectral reconstruction and camera spectral sensitivity optimization system disclosed in this invention, by setting up a sensitivity-parameter device, a multispectral image generation module, a multispectral camera, and a spectral reconstruction module, replaces the convolutional neural network in the efficient convolutional neural network-based method with the sensitivity-parameter device. Through a fusion algorithm of convolution and residual networks, it performs band mapping on the hyperspectral data extracted from the database to obtain multiple sensitivity values. Furthermore, the sensitivity-parameter device can obtain the filter structure parameters corresponding to each sensitivity value using the finite-difference time-domain method. The number of filters in the multispectral camera equals the number of sensitivity values, and there is a one-to-one correspondence between the filters and the filter structure parameters obtained by the sensitivity-parameter device. Therefore, in this invention, the structural parameters of the filters in the multispectral camera are provided by the sensitivity-parameter device. Thus, the sensitivity-parameter device not only achieves highly efficient automatic acquisition of filter structure parameters but also optimizes the camera's spectral sensitivity. This significantly improves the spectral reconstruction accuracy of the combination of the multispectral camera and the spectral reconstruction module. The multispectral image generation module avoids the defect of inconsistent spectral responses of the same pixel when simulating the spectral response of the filter, solving the problem of low accuracy in traditional methods. Furthermore, this sensitivity optimization system utilizes conventional multispectral cameras, a fusion algorithm of convolutional and residual networks, and the finite-difference time-domain method to replace expensive hyperspectral cameras and their filter optimization strategies. Moreover, since the fusion algorithm of convolutional and residual networks is applied to spectral reconstruction with the goal of obtaining sensitivity values, the parameters of the algorithm model can be optimized according to the set loss function, which can ensure transfer effectiveness and improve robustness, thereby reducing hardware costs while improving data acquisition efficiency.

[0009] In one possible implementation, the sensitivity-parameter device includes: The sensitivity device is a network structure consisting of multiple band mapping modules connected in parallel. All the band mapping modules are electrically connected to the multispectral image generation module. Each band mapping module is configured to perform a fusion algorithm of convolution and residual network. Each band mapping module obtains a sensitivity value to form multiple sensitivity values. The parameter device, which is electrically connected to all the band mapping modules, is configured to obtain the filter structure parameters corresponding to each sensitivity value using a time-domain finite-difference method based on the Fabry-Perot multibeam interferometry principle, according to each sensitivity value.

[0010] The sensitivity device is a network structure composed of multiple band mapping modules connected in parallel, while the parameter device obtains the filter structure parameters corresponding to each sensitivity value through the finite-difference time-domain method. This allows the sensitivity-parameter device, which has the above structure and function, to obtain multiple sensitivity values ​​to optimize the spectral sensitivity of the camera filter. At the same time, based on the Fabry-Perot (FP) multibeam interference principle, it uses the interference effect of multiple beams under specific conditions to achieve selective light transmission simulation, obtain the filter structure parameters corresponding to each sensitivity value, provide suitable filter structure data for the multispectral camera, and further improve the accuracy of hyperspectral image reconstruction.

[0011] In one possible implementation, the band mapping module is a network structure formed by sequentially connecting convolutional layer units, residual networks, linear layer units, and fully connected layer units. This scheme, which uses the fusion of convolutional layers, residual networks, linear layers, and fully connected layers to perform band mapping, can significantly improve the accuracy of spectral reconstruction during the optimization of camera spectral sensitivity, thus solving the technical problem of low accuracy in traditional methods.

[0012] In one possible implementation, the discrete form of the channel response value is calculated as follows: , , In the formula, Representing the The response value image corresponding to the sensitivity values ​​obtained by the band mapping module; Represents wavelength; Represents the multispectral camera with respect to the first The overall spectral sensitivity of the filter is set to be equal to that of the first filter in the calculation. The sensitivity values ​​obtained by the band mapping module; The spectral power distribution representing the light source; Represents the spectral reflectance of the target in the scene; Representing the The system noise corresponding to the sensitivity values ​​obtained by the band mapping module.

[0013] The above-mentioned calculation formula simulates the spectral response of the filter. Its computational complexity is small and its accuracy is controllable. This enables the multispectral image generation module that executes the calculation formula to obtain the response value image corresponding to each sensitivity value with low loss and high accuracy. Furthermore, with the cooperation of the band mapping module, it further overcomes the technical defect of inconsistent spectral response of the same pixel when simulating the spectral response of the filter.

[0014] In one possible implementation, the multispectral camera comprises cameras arranged sequentially along the propagation direction of a beam of reflected light from the scene: A filter wheel module includes a filter wheel, a servo motor, and filters arranged in a ring on the filter wheel in a number equal to the sensitivity value. The filter wheel is mounted on the output shaft of the servo motor, which drives the filter wheel to rotate so that one of the filters is positioned on the propagation path of the reflected light. Lens; The detector, electrically connected to the spectral reconstruction module, is configured to receive the output beams from each of the filters to the lens in order to acquire multispectral images of the scene.

[0015] In one possible implementation, the hyperspectral reconstruction algorithm is the MST++ model algorithm, which is applicable to most spectral reconstruction modules, has good robustness, and can significantly improve the accuracy of hyperspectral reconstruction by optimizing the camera's spectral sensitivity.

[0016] Another technical solution of the present invention is to provide a method for spectral reconstruction and camera spectral sensitivity optimization, comprising the following steps: S1: Collect multiple hyperspectral data and use the loss function formed by the linear combination of the loss function of the spectral reconstruction network and the loss function of the polynomial fitting to optimize the parameters of the sensitivity-parameter device and the spectral reconstruction module. S2: Input hyperspectral data into the optimized sensitivity-parameter device to obtain multiple filter structure parameters; S3: Prepare multiple filters based on the obtained filter structure parameters to build a multispectral camera, and use the multispectral camera to acquire multispectral images of the scene; S4: The spectral reconstruction module with optimized parameters input to the multispectral image of the scene obtains a hyperspectral image.

[0017] The spectral reconstruction and camera spectral sensitivity optimization method disclosed in this invention first optimizes the parameters of the sensitivity-parameter device and the spectral reconstruction module using a loss function formed by a linear combination of the spectral reconstruction network loss function and the polynomial fitting loss function. This parameter optimization of the loss function makes the sensitivity values ​​obtained by the optimized sensitivity-parameter device more consistent with the thin-film design concept when used as the camera spectral sensitivity, and also facilitates iteration in the backpropagation of the deep neural network. Subsequently, multiple filter structure parameters are obtained using the sensitivity-parameter device, and multiple filters are fabricated based on these parameters to build a multispectral camera. The multispectral camera is then used to acquire multispectral images of the scene, ultimately completing the hyperspectral image reconstruction. Since the sensitivity-parameter device obtains multiple sensitivity values ​​by band mapping the hyperspectral data extracted from the database through a fusion algorithm of convolution and residual networks, and can also obtain the corresponding filter structure parameters for each sensitivity value using the finite-difference time-domain method, it not only achieves highly efficient automatic acquisition of filter structure parameters but also optimizes the camera spectral sensitivity. This significantly improves the spectral reconstruction accuracy of the combination of the multispectral camera and the spectral reconstruction module, solving the problem of low accuracy in traditional methods. Furthermore, this sensitivity optimization system utilizes conventional multispectral cameras, a fusion algorithm of convolutional and residual networks, and the finite-difference time-domain method to replace expensive hyperspectral cameras and their filter optimization strategies. Moreover, since the fusion algorithm of convolutional and residual networks is applied to spectral reconstruction with the goal of obtaining sensitivity values, the parameters of the algorithm model can be optimized according to the set loss function, which can ensure transfer effectiveness and improve robustness, thereby reducing hardware costs while improving data acquisition efficiency.

[0018] In one possible implementation, step S1 includes the following steps: S11: Extract multiple hyperspectral data from the database, set up multiple band mapping modules, and set model parameters for multiple band mapping modules and spectral reconstruction modules, with different model parameters set for different band mapping modules; S12: Simultaneously input each hyperspectral data into the multiple band mapping modules set in step S11 to obtain their respective sensitivity value combinations. S13: Combine the sensitivity values ​​of each hyperspectral data and input them into the multispectral image generation module to obtain their respective multispectral data images; S14: Input the multispectral data images of each hyperspectral data into the spectral reconstruction module to obtain their respective reconstructed hyperspectral images. Substitute each hyperspectral data, the sensitivity value combination of each hyperspectral data, and the reconstructed hyperspectral image into the loss function formed by the linear combination of the spectral reconstruction network loss function and the polynomial fitting loss function to obtain the loss function value of each hyperspectral data. S15: Determine whether the maximum value of all loss function values ​​obtained in step S14 is less than a preset value. If so, the current model parameters of the multiple band mapping modules are taken as their optimal model parameters, and the current model parameters of the spectral reconstruction module are taken as their optimal model parameters, so as to obtain the multiple band mapping modules and the spectral reconstruction module with optimized parameters, and then step S17 is executed. If not, proceed to the next step; S16, determine if the current loop count has reached the peak. If so, the model parameters of the multiple band mapping modules in the loop with the smallest maximum value of all loss function values ​​obtained in step S14 are taken as their optimal model parameters, and the model parameters of the spectral reconstruction module in that loop are taken as their optimal model parameters, so as to obtain the multiple band mapping modules and the spectral reconstruction module with optimized parameters, and then step S17 is executed. If not, perform a translation transformation with a specified step size and / or a scaling operation with a specified magnification on the current model parameters of the multiple band mapping modules and the spectral reconstruction module to obtain the model parameters of the multiple band mapping modules and the spectral reconstruction module in the next loop, and then return to execute step S12 to enter the next loop; S17: Electrically connect the output terminals of the multiple band mapping modules after parameter optimization to the input terminal of the parameter device to obtain the parameter-optimized sensitivity-parameter device.

[0019] This method of jointly training the band mapping module and the spectral reconstruction module not only achieves efficient optimization of camera spectral sensitivity and enhances the robustness and adaptability of camera spectral reconstruction, but also further ensures that the multispectral image generation module will not exhibit inconsistent spectral responses of the same pixel when simulating the spectral response of the filter.

[0020] In one possible implementation, the loss function formed in step S14 is calculated as follows: , , , In the formula, Represents the loss function; This represents the loss function of the spectral reconstruction network; This represents the polynomial fitting loss function; This represents the hyperspectral data input to the band mapping module in step S12. Reconstructed hyperspectral images representing hyperspectral data; The matrix represents the combination of sensitivity values ​​of all hyperspectral data input to the band mapping module in step S12; Represents the matrix The output result after polynomial fitting.

[0021] Using the loss function described above allows the optimized camera spectral sensitivity to better align with thin-film design principles, while also facilitating iteration during backpropagation in deep neural networks.

[0022] In one possible implementation, the norm used in both the loss function of the spectral reconstruction network and the loss function of the polynomial fitting is the 1-norm, which can reduce the computational complexity during parameter optimization. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of a spectral reconstruction and camera spectral sensitivity optimization system disclosed in an embodiment of the present invention; Figure 2 This is a schematic diagram of the operation process of a spectral reconstruction and camera spectral sensitivity optimization system disclosed in an embodiment of the present invention; Figure 3 This is a schematic diagram of the band mapping module structure disclosed in the embodiments of the present invention; Figure 4 This is a schematic diagram illustrating the multispectral image generation principle of the multispectral camera disclosed in this embodiment of the invention; Figure 5 This is a schematic diagram illustrating the parameter optimization of the sensitivity-parameter device and the spectral reconstruction module using a loss function as disclosed in the embodiments of the present invention. Figure 6 This is a flowchart of the method disclosed in the embodiments of the present invention; Figure 7 This is a flowchart of step S1 disclosed in the embodiments of the present invention; Figure 8 These are the qualitative test results of the spectral reconstruction bands disclosed in the embodiments of the present invention; Figure 9 This is the transmittance curve of the filter disclosed in the embodiments of the present invention. Detailed Implementation

[0024] First, those skilled in the art should understand that these embodiments are merely used to explain the technical principles of the embodiments of this application and are not intended to limit the scope of protection of the embodiments of this application. Those skilled in the art can make adjustments as needed to adapt to specific application scenarios.

[0025] In the description of the embodiments of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "electrical connection" and "establishing an electrical connection relationship" should be interpreted broadly, that is, it should be understood that both or more parties have an electrical relationship, which can be achieved through wires, radio connections, or a combination of both; it can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of this application based on the specific circumstances. Furthermore, a combination of discrete bands with 3-15 discrete bands and a resolution greater than 20 nm is called multispectral, and a combination of discrete bands with more than 15 bands and a spectral resolution less than or equal to 10 nm is called hyperspectral.

[0026] In the description of the embodiments of this application, it should be noted that, unless otherwise explicitly specified and limited, whenever it is mentioned that a module or device is optimized by or using a certain loss function, it refers to the series of processes in which the model parameters of the module or device are taken as the parameters to be determined, and the optimal value of the loss function is obtained as the goal. The specific value of the parameters to be determined is obtained by solving the optimization model solving algorithm (such as particle swarm optimization algorithm, greedy algorithm, Newton's method, back gradient method), and the specific value of the parameters to be determined is used as the model parameters of the module or device.

[0027] In the embodiments of this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0028] The present application will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0029] See Figures 1-9 This application discloses a spectral reconstruction and camera spectral sensitivity optimization system. Figure 1This is a schematic diagram of the system structure. The system includes a sensitivity-parameter device, a multispectral image generation module, a multispectral camera, and a spectral reconstruction module. The multispectral image generation module is electrically connected to the sensitivity-parameter device, and the spectral reconstruction module is electrically connected to both the multispectral image generation module and the multispectral camera. The number of filters in the multispectral camera equals the number of sensitivity values, and the filter structure parameters obtained by the sensitivity-parameter device correspond one-to-one. That is, in this embodiment, the filter structure parameters of all filters on the multispectral camera are obtained through the sensitivity-parameter device. Figure 2 The process from "filter design" to "multispectral camera" in the text represents this idea.

[0030] See Figure 1 In this system, a multispectral camera is configured to acquire multispectral images of the scene. The multispectral camera used in this embodiment includes a filter wheel module, a lens, and a detector arranged sequentially along the propagation direction of a beam of reflected light from the scene. The filter wheel module comprises a filter wheel, a servo motor, and filters arranged in a ring around the filter wheel in a number equal to the sensitivity value. The filter wheel is mounted on the output shaft of the servo motor, which drives the filter wheel to rotate so that one of the filters is positioned in the propagation path of the reflected light. The detector is configured to receive the output beams from each filter to the lens to acquire multispectral images of the scene.

[0031] See Figure 1 In this system, the sensitivity-parameter device is configured to perform band mapping on the hyperspectral data extracted from the database using a fusion algorithm of convolution and residual networks to obtain multiple sensitivity values; subsequently, based on each sensitivity value, the filter structure parameters corresponding to each sensitivity value are obtained through the finite-difference time-domain method. Specifically, in this embodiment, the sensitivity-parameter device includes a sensitivity device and a parameter device, wherein the parameter device is electrically connected to the sensitivity device.

[0032] See Figure 1 and Figure 2 In the sensitivity-parameter device, the sensitivity device is a network structure composed of multiple band mapping modules connected in parallel. Each band mapping module is configured to execute a fusion algorithm of convolution and residual networks, such as... Figure 2 As shown, in this embodiment, the number of band mapping modules used is 3 (the principle of other values ​​is the same as that of this embodiment), and each band mapping module obtains a sensitivity value, so that the sensitivity device outputs 3 sensitivity values, which are denoted as... And the band mapping module is denoted as ,like Figure 5As shown, all band mapping modules are electrically connected to the multispectral image generation module. Each band mapping module is a network structure formed by sequentially connecting convolutional layer units, residual networks, linear layer units, and fully connected layer units. This scheme, which integrates convolutional layers, residual networks, linear layers, and fully connected layers, and outputs sensitivity values ​​through fully connected layer units to perform band mapping, significantly improves spectral reconstruction accuracy during camera spectral sensitivity optimization, thus solving the technical problem of low accuracy in traditional methods.

[0033] In the sensitivity-parameter device, the parameter device is electrically connected to all band mapping modules simultaneously. The parameter device is configured to obtain the filter structure parameters corresponding to each sensitivity value using a finite-difference time-domain method based on the Fabry-Perot multibeam interferometry principle. In specific calculations, the material refractive index, thickness, sequence structure, number of periods, substrate or incident medium, spectral target, incident angle or polarization angle, and grid accuracy can be set, and boundary conditions can also be further configured. In this embodiment, TiO2 and SiO2 are used as the materials. The specific calculation process is existing technology and will not be elaborated here.

[0034] In this system, the multispectral image generation module is configured to substitute each sensitivity value into a discrete-form formula for calculating the channel response value to obtain the response value image corresponding to each sensitivity value, and then integrate the response value images corresponding to all sensitivity values ​​into a single multispectral data image. The multispectral image generation module essentially simulates the imaging mechanism of a multispectral camera. In this embodiment, the discrete-form formula for calculating the channel response value is: , , In the formula, Representing the The response value image corresponding to the sensitivity values ​​obtained by each band mapping module; It represents the wavelength, and its value range is the wavelength range of visible light; Representing the multispectral camera regarding the first The overall spectral sensitivity of the filter is set to be equal to that of the first filter in the calculation. Sensitivity values ​​obtained from each band mapping module; The spectral power distribution representing the light source; Represents the spectral reflectance of the target in the scene; Representing the The system noise corresponding to the sensitivity values ​​obtained by each band mapping module.

[0035] To facilitate better understanding by those skilled in the art, the derivation process of the discrete-form channel response value calculation formula will be explained below: Let wavelength... The unit is nm, such as Figure 4 As shown, this illustrates the spectral power distribution (or energy) of a target (object) relative to a light source in the scene. After reflection, the light passes through a filter in the multispectral camera, and finally through the camera lens and is captured by the detector. During this process, the reflected light from the target in the scene passes through the... After filtering, the response image acquired by the detector of the multispectral camera can be represented as: , in This refers to the spectral power distribution (SPD) of the light source. The spectral reflectance of the target in the scene. Indicates the first The spectral transmittance of each filter. Let be the spectral transfer function of the lens. Let be the spectral response function of the detector. This indicates system noise. Because... and These are generally fixed values. For ease of calculation, this embodiment will... The overall spectral sensitivity of a multispectral camera is considered, including the detector's spectral response function, the filter's spectral transmittance, and the lens's spectral transfer function. (For the band mapping module...) After setting the model parameters, let The value is equal to that of the band mapping module. Output result, i.e., the first The sensitivity value obtained by the band mapping module is used to determine the reflected light's path through the first band. After filtering, the response image obtained by the detector can be represented as: , Will Considered as the target spectral reflectance intensity, The value range is taken from the wavelength range of visible hyperspectral data. According to the definition of the Riemann integral, Converting to discrete form, the formula for calculating the discrete channel response value is: , The above formula can be used to determine the sensitivity values ​​obtained from the input hyperspectral image and the band mapping module. Generate multispectral data images.

[0036] See Figure 1 and Figure 2In this system, the spectral reconstruction module is configured to execute a hyperspectral reconstruction algorithm to obtain a reconstructed hyperspectral image using hyperspectral data images during parameter optimization, and to obtain a hyperspectral image using multispectral images of the scene during non-parametric optimization. In this embodiment, the hyperspectral reconstruction algorithm used is the MST++ model algorithm, which has wide adaptability and can be used in most spectral reconstruction processes. Furthermore, the accuracy of spectral reconstruction can be significantly improved by optimizing the camera's spectral sensitivity.

[0037] See Figure 2 , Figure 3 , Figure 5 , Figure 6 and Figure 7 The following section will further disclose a method for spectral reconstruction and camera spectral sensitivity optimization corresponding to this system. Figure 6 Here is a flowchart of the method, which includes the following steps: S1: Collect multiple hyperspectral data and optimize the parameters of the sensitivity-parameter device and the spectral reconstruction module using a loss function formed by a linear combination of the loss function of the spectral reconstruction network and the loss function of the polynomial fitting.

[0038] exist Figure 2 , Figure 3 and Figure 4 In this context, the acquired hyperspectral dataset is denoted as... , , Where H, W, and C represent the height, width, and number of bands of the hyperspectral data, respectively. This embodiment uses the ARAD1K hyperspectral dataset, with H, W, and C of 482, 512, and 31, respectively, and a total of 900 hyperspectral data points. .

[0039] See Figure 7 In this embodiment, step S1 includes the following steps: S11: Extract multiple hyperspectral data from the database, set up multiple band mapping modules, and configure model parameters for both the band mapping and spectral reconstruction modules. Different model parameters are set for different band mapping modules. As per the above convention, the number of band mapping modules is 3.

[0040] S12: Simultaneously input the hyperspectral data into the multiple band mapping modules set in step S11 to obtain their respective sensitivity value combinations. For the first... Hyperspectral data The combination of its sensitivity values ​​is denoted as .

[0041] S13: Input the combined sensitivity values ​​of each hyperspectral data point into the multispectral image generation module to obtain their respective multispectral data images. (For the combined sensitivity values...) In the multispectral image generation module, the response value images are obtained sequentially using discrete channel response value calculation formulas. The resulting three response value images are: Subsequently, the multispectral image generation module integrates these three response value images into a multispectral data image, denoted as... .

[0042] S14: Input the multispectral data images of each hyperspectral data into the spectral reconstruction module to obtain their respective reconstructed hyperspectral images. Substitute each hyperspectral data, the sensitivity value combination of each hyperspectral data, and the reconstructed hyperspectral image into the loss function formed by the linear combination of the spectral reconstruction network loss function and the polynomial fitting loss function to obtain the loss function value of each hyperspectral data.

[0043] The formula for calculating the loss function formed in step S14 is as follows: , , , In the formula, Represents the loss function; The loss function of the spectral reconstruction network; This represents the loss function for polynomial fitting. This represents the hyperspectral data input to the band mapping module in step S12. Reconstructed hyperspectral images representing hyperspectral data; This represents the matrix formed by combining the sensitivity values ​​of all hyperspectral data input to the band mapping module in step S12. Represents the matrix The output result after polynomial fitting.

[0044] The specific calculation details are further disclosed below: For multispectral data images The reconstructed hyperspectral image obtained by inputting it into the spectral reconstruction module is denoted as Therefore, for the first Hyperspectral data Its sensitivity values ​​are combined as follows Its reconstructed hyperspectral image is .

[0045] First of all, let , To obtain the first spectral reconstruction network loss function using the spectral reconstruction network loss function The loss function value of the spectral reconstruction network for hyperspectral data This allows us to obtain the loss function values ​​of the spectral reconstruction network for all hyperspectral data in the hyperspectral dataset, thus yielding the sequence. Note: To reduce computational complexity, the loss function used in this embodiment for the spectral reconstruction network uses the 1-norm, i.e. ; As is well known, hyperspectral data can be represented in tensor form, because hyperspectral datasets... Therefore, the first Hyperspectral data It can be represented as: , , In the formula, The first in hyperspectral data The response value is located at the pixel position in the image. The response value (also known as pixel intensity). , , , .

[0046] Similarly, the first Hyperspectral data The reconstructed hyperspectral image can be represented as: , , In the formula, To reconstruct the first hyperspectral image The response value is located at the pixel position in the image. The response value. Therefore, when the norm used in the loss function of the spectral reconstruction network is the 1-norm, , The loss function value of the spectral reconstruction network for all hyperspectral data can be obtained by following the above calculation formula.

[0047] Following that, regarding the first Hyperspectral data Sensitivity value combination The resulting vector can be represented as: , This yields a vector formed by combining the sensitivity values ​​of all hyperspectral data in the hyperspectral dataset, thus creating a vector sequence. Now we need to process the vector sequence. Perform polynomial fitting, that is, based on the sequence Find a fitting polynomial function such that this polynomial function is similar to... The sum of distances is minimized. Let's assume this fitted polynomial function is: , Let the coefficient vector of this fitted polynomial be: , At the same time , , , Solving the coefficients using the least squares method We can obtain: , in, This represents the transpose operation of a matrix, which yields the fitted polynomial function. Note that during the calculation process, if the matrix... If duplicate rows appear in the matrix, the duplicate rows need to be deleted to ensure the matrix is ​​intact. Reversible.

[0048] In this embodiment, the norm used in the polynomial fitting loss function is the 1-norm. Therefore, the polynomial fitting loss function value for all hyperspectral data in the hyperspectral dataset is: .

[0049] Finally, using the polynomial fitting loss function value obtained earlier and the spectral reconstruction network loss function value of all hyperspectral data in the hyperspectral dataset, combined with the calculation formula of the loss function, the loss function value of each hyperspectral data is obtained, thus yielding a sequence of loss function values. ,in, , In the formula, For the first The loss function value of the hyperspectral data.

[0050] S15: Determine whether the maximum value of all loss function values ​​obtained in step S14 is less than a preset value. If so, the current model parameters of the multiple band mapping modules are taken as their optimal model parameters, and the current model parameters of the spectral reconstruction module are taken as their optimal model parameters, so as to obtain the multiple band mapping modules and the spectral reconstruction module with optimized parameters, and then step S17 is executed. If not, proceed to the next step.

[0051] Let's assume the current number is the [number]. In the next iteration, all loss function values ​​obtained through step S14 are... As one implementation method, the preset value set in this embodiment is 0.005. Therefore, when the inequality is satisfied... When the maximum value of all loss function values ​​obtained in step S14 is less than the preset value, it is determined that the maximum value is less than the preset value.

[0052] S16, determine if the current loop count has reached the peak. If so, the model parameters of the multiple band mapping modules in the loop with the smallest maximum value of all loss function values ​​obtained in step S14 are taken as their optimal model parameters, and the model parameters of the spectral reconstruction module in that loop are taken as their optimal model parameters, so as to obtain the multiple band mapping modules and the spectral reconstruction module with optimized parameters, and then step S17 is executed. If not, perform a translation transformation with a specified step size and / or a scaling operation with a specified magnification on the current model parameters of the multiple band mapping modules and the spectral reconstruction modules to obtain the model parameters of the multiple band mapping modules and the spectral reconstruction modules in the next loop, and then return to execute step S12 to enter the next loop.

[0053] As one implementation method, the peak value set in this embodiment is 1000, while in the... In this iteration, the maximum value of all loss function values ​​obtained in step S14 is: , When the number of iterations reaches 1000, first calculate the minimum value of the sequence of maximum values ​​of all loss function values ​​obtained in step S14 during these 1000 iterations. The calculation formula is as follows: , In the formula, This represents the minimum value of the sequence of maximum values ​​of the loss function to be obtained. Then find... The corresponding number of iterations is determined, and the model parameters of multiple band mapping modules in that iteration are taken as their optimal model parameters. The model parameters of the spectral reconstruction module in that iteration are also taken as their optimal model parameters, so as to obtain the multiple band mapping modules and the spectral reconstruction module with optimized parameters.

[0054] S17: Electrically connect the output of these multiple band mapping modules after parameter optimization to the input of the parameter device to obtain the parameter-optimized sensitivity-parameter device.

[0055] S2: Input hyperspectral data into the optimized sensitivity-parameter device to obtain multiple filter structure parameters.

[0056] Specifically, in this embodiment, the structural parameters of the three obtained filters are shown in Table 1: Table 1: Filter Structural Parameters

[0057] S3: Prepare multiple filters based on the obtained filter structure parameters to build a multispectral camera, and use the multispectral camera to acquire multispectral images of the scene.

[0058] S4: The spectral reconstruction module takes the multispectral image of the scene as input parameters and optimizes them to obtain a hyperspectral image.

[0059] The technical effects of the spectral reconstruction and camera spectral sensitivity optimization system disclosed in this embodiment will be described in detail below. This embodiment compares the system with current mainstream spectral reconstruction models, including HSCNN+, HRNet, Restormer, and MST++ models. In quantitative testing, their root mean square error (RMSE), mean relative absolute error (MRAE), and peak signal-to-noise ratio (PSNR) were calculated. The obtained quantitative test results are shown in Table 2. Table 2: Comparison of Spectral Reconstruction Results

[0060] As can be seen from Table 2, the spectral reconstruction accuracy was significantly improved after the system in this embodiment optimized the camera's spectral sensitivity.

[0061] Meanwhile, this embodiment also compared the performance test results; the qualitative test results are available for reference. Figure 8 By comparing the system with the true value and observing the difference between the two, it can be seen that the error of the optimized system in this embodiment is significantly reduced and the spectral reconstruction results are significantly improved.

[0062] Furthermore, in this embodiment, a transmittance test is performed on a filter fabricated based on the three filter structure parameters output by the sensitivity-parameter device. The obtained transmittance image is shown below. Figure 9 As shown, from Figure 9 As can be seen, the transmittance of these three filters is independent of each other and each has its own characteristics, which is in line with the actual needs.

[0063] The spectral reconstruction and camera spectral sensitivity optimization system disclosed in this embodiment, by setting up a sensitivity-parameter device, a multispectral image generation module, a multispectral camera, and a spectral reconstruction module, replaces the convolutional neural network in the efficient convolutional neural network-based method with the sensitivity-parameter device. Through a fusion algorithm of convolution and residual networks, the hyperspectral data extracted from the database is band-mapped to obtain multiple sensitivity values. Furthermore, the sensitivity-parameter device can obtain the filter structure parameters corresponding to each sensitivity value using the finite-difference time-domain method. The structural parameters of the filters contained in the multispectral camera are provided by the sensitivity-parameter device. Therefore, the set sensitivity-parameter device not only achieves efficient automatic acquisition of filter structure parameters but also optimizes the camera's spectral sensitivity. This significantly improves the spectral reconstruction accuracy of the combination of the multispectral camera and the spectral reconstruction module. The multispectral image generation module avoids the defect of inconsistent spectral responses of the same pixel when simulating the spectral response of the filter, solving the problem of low accuracy in traditional methods. Furthermore, this sensitivity optimization system utilizes conventional multispectral cameras, a fusion algorithm of convolutional and residual networks, and the finite-difference time-domain method to replace expensive hyperspectral cameras and their filter optimization strategies. Moreover, since the fusion algorithm of convolutional and residual networks is applied to spectral reconstruction with the goal of obtaining sensitivity values, the parameters of the algorithm model can be optimized according to the set loss function, which can ensure transfer effectiveness and improve robustness, thereby reducing hardware costs while improving data acquisition efficiency.

[0064] In the description of the embodiments of this application, it should be noted that the terms "inner" and "outer" and other terms indicating direction or positional relationship are based on the direction or positional relationship shown in the drawings. This is only for the convenience of description and does not indicate or imply that the device or component must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this application.

[0065] In the description of this application, the references to terms such as "an embodiment," "some embodiments," "in this embodiment," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0066] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A spectral reconstruction and camera spectral sensitivity optimization system, comprising: The sensitivity-parameter device is configured to perform a convolution and residual network fusion algorithm to map the hyperspectral data extracted from the database to obtain a plurality of sensitivity values; and then obtain filter structure parameters corresponding to each of the sensitivity values by using a time domain finite difference method according to each of the sensitivity values. The multispectral image generation module is electrically connected to the sensitivity-parameter device and is configured to substitute each of the sensitivity values into a discrete form of a channel response value calculation formula to obtain a response value image corresponding to each of the sensitivity values, and integrate all the response value images corresponding to the sensitivity values into a multispectral data image. The multispectral camera is configured to capture a multispectral image of a scene. The spectral reconstruction module is electrically connected to the multispectral image generation module and the multispectral camera, and is configured to perform a hyperspectral reconstruction algorithm to obtain a reconstructed hyperspectral image by using the hyperspectral data image when parameters are optimized, and obtain a hyperspectral image by using the multispectral image of the scene when parameters are not optimized. The number of filters included in the multispectral camera is equal to the number of the sensitivity values, and each filter included in the multispectral camera corresponds to the filter structure parameters obtained by the sensitivity-parameter device. The sensitivity-parameter device includes:

2. The system for spectral reconstruction and camera spectral sensitivity optimization of claim 1, wherein, The sensitivity device is a network structure formed by connecting a plurality of band mapping modules in parallel, all of the band mapping modules are electrically connected to the multispectral image generation module, each of the band mapping modules is configured to perform a convolution and residual network fusion algorithm, and each of the band mapping modules obtains a sensitivity value to form a plurality of sensitivity values. The parameter device is electrically connected to all of the band mapping modules and is configured to obtain filter structure parameters corresponding to each of the sensitivity values by using a time domain finite difference method based on a Fabry-Perot multi-beam interference principle according to each of the sensitivity values. The band mapping module is a network structure formed by connecting a convolution layer unit, a residual network, a linear layer unit and a full connection layer unit in sequence.

3. The spectral reconstruction and camera spectral sensitivity optimization system of claim 2, wherein, The discrete form of the channel response value calculation formula is:

4. The spectral reconstruction and camera spectral sensitivity optimization system of claim 2 or 3, wherein, In the formula, ， ， The multispectral camera includes, in sequence along a propagation direction of reflected light of the scene: representing the first response value image corresponding to the sensitivity value obtained by the waveband mapping module Represents wavelength; Represents the multispectral camera with respect to the first The overall spectral sensitivity of the filter is set to be equal to that of the first filter in the calculation. The sensitivity values ​​obtained by the band mapping module; The spectral power distribution representing the light source; Represents the spectral reflectance of the target in the scene; representing the system noise corresponding to the sensitivity value obtained by the waveband mapping module.

5. The system for spectral reconstruction and camera spectral sensitivity optimization of claim 4, wherein, A filter wheel module including a filter wheel, a servo motor and a plurality of filters arranged in a ring around the filter wheel, the number of the filters being equal to the number of the sensitivity values, the filter wheel being arranged on an output shaft of the servo motor, and the servo motor being used to drive the filter wheel to rotate so that one of the filters is located on a propagation path of the reflected light; A lens; A detector electrically connected to the spectral reconstruction module and configured to receive output light beams of each of the filters to the lens to capture a multispectral image of the scene. The hyperspectral reconstruction algorithm is an MST++ model algorithm.

6. The system for spectral reconstruction and camera spectral sensitivity optimization of claim 1 or 2 or 3 or 5, wherein, The spectral reconstruction and camera spectral sensitivity optimization system according to any one of claims 1-6 includes the following steps:

7. A method for spectral reconstruction and camera spectral sensitivity optimization, characterized in that, S1: capturing a plurality of hyperspectral data, and performing parameter optimization on the sensitivity-parameter device and the spectral reconstruction module by using a loss function formed by a linear combination of a spectral reconstruction network loss function and a polynomial fitting loss function. ​ S2: input a hyperspectral data to the parameter-optimized sensitivity-parameter device to obtain a plurality of filter structure parameters; S3: prepare a plurality of filters according to the obtained plurality of filter structure parameters to build a multispectral camera, and use the multispectral camera to collect a multispectral image of a scene; S4: input the multispectral image of the scene to the parameter-optimized spectral reconstruction module to obtain a hyperspectral image.

8. The method of spectral reconstruction and camera spectral sensitivity optimization of claim 7, wherein, The step S1 includes the following steps: S11: extract a plurality of hyperspectral data from a database, set a plurality of band mapping modules, and set model parameters for the plurality of band mapping modules and the spectral reconstruction module, and different model parameters are set for different band mapping modules; S12: input each hyperspectral data to the plurality of band mapping modules set in the step S11 to obtain a respective sensitivity value combination; S13: input the sensitivity value combination of each hyperspectral data to a multispectral image generation module to obtain a respective multispectral data image; S14: input the multispectral data image of each hyperspectral data to a spectral reconstruction module to obtain a respective reconstructed hyperspectral image, and substitute each hyperspectral data, the sensitivity value combination of each hyperspectral data, and the reconstructed hyperspectral image into a loss function formed by a linear combination of a spectral reconstruction network loss function and a polynomial fitting loss function to obtain a loss function value of each hyperspectral data; S15: determine whether the maximum value of all the loss function values obtained in the step S14 is less than a preset value, if yes, take the current model parameters of the plurality of band mapping modules as the optimal model parameters thereof, take the current model parameters of the spectral reconstruction module as the optimal model parameters thereof, to obtain the parameter-optimized plurality of band mapping modules and the parameter-optimized spectral reconstruction module, and then perform the step S17; if no, perform the next step; S16: determine whether the current cycle number reaches a peak value, if yes, take the model parameters of the plurality of band mapping modules in the cycle in which the minimum maximum value of all the loss function values is obtained as the optimal model parameters thereof, take the model parameters of the spectral reconstruction module in the cycle as the optimal model parameters thereof, to obtain the parameter-optimized plurality of band mapping modules and the parameter-optimized spectral reconstruction module, and then perform the step S17; if no, perform a specified step translation and / or a specified scale operation on the current model parameters of the plurality of band mapping modules and the spectral reconstruction module to obtain the model parameters of the plurality of band mapping modules and the spectral reconstruction module in the next cycle, and then perform the step S12 again to enter the next cycle; S17: electrically connect the output end of the parameter-optimized plurality of band mapping modules to the input end of the parameter device to obtain a parameter-optimized sensitivity-parameter device.

9. The method of claim 8, wherein, The calculation formula of the loss function formed in the step S14 is as follows: ， ， ， In the formula, Represents the loss function; representing the spectral reconstruction network loss function; representing the polynomial fit loss function; representing hyperspectral data input to the band mapping module in step S12; reconstructed hyperspectral image representative of the hyperspectral data; a matrix representing the combination of the sensitivity values of all hyperspectral data input to the band mapping module in step S12; representative of the matrix output results after polynomial fitting.

10. The method of spectral reconstruction and camera spectral sensitivity optimization of claim 9, wherein, The norm used in the spectral reconstruction network loss function and the polynomial fitting loss function is a 1-norm.

Images