Optimization method, reconstruction method and device of diffraction and refraction hybrid snapshot hyperspectral imaging system

Abstract

This application provides an optimization method, reconstruction method, and device for a hybrid diffraction and refraction snapshot hyperspectral imaging system. The optimization method includes: using a genetic algorithm with a fitness function consisting of a weighted difference between spectral Fisher information and spatial second moments to globally search and optimize the structural parameters of diffractive optical elements, obtaining initial parameters; and jointly optimizing the structural parameters of the diffractive optical elements and the hyperspectral reconstruction network based on the initial parameters, a differentiable optical propagation model, and hyperspectral training data. This application can solve the problems of existing diffractive optical element optimization easily getting trapped in local minima, and the difficulty of single diffractive elements simultaneously considering spectral dispersion and spatial focusing, resulting in poor imaging quality and low optimization efficiency. It can avoid the structural parameter optimization process of diffractive optical elements getting trapped in local minima, and can significantly improve the optimization convergence speed and final reconstruction accuracy, thereby achieving high light throughput, high signal-to-noise ratio, and high-precision hyperspectral reconstruction.

Description

Technical Field

[0001] This application relates to the fields of machine learning and optical imaging technology, and in particular to optimization methods, reconstruction methods and devices for hybrid diffraction and refraction snapshot hyperspectral imaging systems. Background Technology

[0002] Hyperspectral imaging technology can acquire the spectral information of a target, interpreting the basic information composition of a scene in the wavelength dimension of light, and is widely used in agriculture, medicine, and art authentication. Traditional hyperspectral imaging systems mostly use pushbroom or tunable filter-based line-by-line or band-by-band scanning. While this time-modulation method can achieve high spectral resolution, it suffers from slow imaging speed and low temporal resolution, and is prone to motion artifacts when capturing dynamic scenes, severely limiting its real-time performance and application in dynamic environments. To solve this problem, snapshot hyperspectral imaging technology has emerged, which can directly acquire a complete three-dimensional data cube within a single exposure time.

[0003] Currently, among various snapshot architectures, computational imaging systems based on diffractive optical elements (DOEs) have become a research hotspot due to their advantages such as small size, light weight, low cost, and ease of integration. However, existing lensless imaging systems based on monolithic diffractive elements face serious performance bottlenecks in practical applications. First, the system suffers from inherent functional overload: a single DOE must simultaneously handle two conflicting optical tasks—strong spectral dispersion (encoding) and high spatial focusing (imaging)—making it difficult to balance spatial and spectral resolution. Second, limited by the physical limits of diffraction efficiency, zero-order light (undiffracted light) passes directly through the DOE to the sensor, forming extensive background haze, severely reducing image contrast and signal-to-noise ratio, and causing a decrease in luminous flux, making the system almost unusable in low-light environments. Furthermore, due to the lack of effective optical focusing, the point spread function of pure diffraction systems is severely diffused, resulting in extremely blurry images and the loss of a large amount of high-frequency spatial details.

[0004] At the level of system design and optimization methods, although the end-to-end deep learning frameworks that have emerged in recent years have achieved joint optimization of optics and algorithms, they still face key challenges: the optimization surface of diffractive optics is highly non-convex, and traditional gradient-based optimization algorithms are prone to getting trapped in local minima, making it difficult to find the globally optimal optical structure. At the same time, existing methods usually treat the optical layer as a "black box," lacking physical interpretability and making it difficult to effectively integrate optical prior knowledge into the optimization process, resulting in low optimization efficiency, slow convergence speed, and the final system performance being limited by the choice of initial values.

[0005] Therefore, there is an urgent need for an optimization method for snapshot hyperspectral imaging systems that can balance spectral encoding and spatial focusing, avoid gradient optimization from getting trapped in local minima, and have physical interpretability. Summary of the Invention

[0006] In view of this, embodiments of this application provide an optimization method, reconstruction method, and apparatus for a hybrid diffraction and refraction snapshot hyperspectral imaging system to eliminate or improve one or more defects existing in the prior art.

[0007] One aspect of this application provides an optimization method for a hybrid diffraction and refraction snapshot hyperspectral imaging system, the imaging system including a refractive lens, a diffractive optical element disposed at the aperture stop of the refractive lens, and an image sensor, the method comprising: A genetic algorithm is used to perform a global search optimization on the structural parameters of the diffractive optical element, using a function composed of the weighted difference between spectral Fisher information and the second spatial moment as the fitness function, to obtain the initial optimized structural parameters of the diffractive optical element; wherein, the spectral Fisher information is used to quantify the sensitivity of the point spread function to wavelength, and the second spatial moment is used to quantify the energy dispersion of the point spread function over a wide wavelength range. Based on the initial optimized structural parameters, the differentiable optical propagation model corresponding to the imaging system, and the hyperspectral training data, the structural parameters of the diffractive optical element and a hyperspectral reconstruction network are jointly optimized to obtain the target optimized structural parameters for optimizing the diffractive optical element in the imaging system, as well as the optimized hyperspectral reconstruction network.

[0008] In some embodiments of this application, before performing a global search optimization on the structural parameters of the diffractive optical element, the method further includes: The height map of the diffractive optical element is discretized into M×M pixel units to obtain a discretized height map; wherein each pixel unit corresponds to a height value; M is a positive integer greater than 1; The discretized height map is parameterized into a rotationally symmetric structure composed of multiple concentric rings and used as the structural parameter of the diffractive optical element; wherein each concentric ring corresponds to a discrete height value; Furthermore, a total generalized pupil function is constructed; wherein the total generalized pupil function is the product of the aperture function, the phase of the refractive lens, and the modulation of the diffractive optical element; Based on the angular spectrum method, the square of the Fourier transform modulus of the total generalized pupil function is used as the point spread function, thereby establishing the differentiable optical propagation model corresponding to the imaging system.

[0009] In some embodiments of this application, a genetic algorithm is used to perform a global search optimization of the structural parameters of the diffractive optical element, using a function composed of the weighted difference between spectral Fisher information and the second spatial moment as the fitness function. The initial optimized structural parameters of the diffractive optical element include: Based on the differentiable optical propagation model, the point spread function of the imaging system in each wavelength channel is calculated; wherein, the wavelength channel refers to the discrete spectral interval divided within the operating wavelength band of the imaging system; Based on the point spread function of each wavelength channel, the spectral Fisher information is calculated; Furthermore, based on the point spread function intensity distribution calculated using the differentiable optical propagation model, the second-order spatial moment is calculated: Substituting the spectral Fisher information and the spatial second moment into the fitness function, the fitness function value is calculated; The fitness function includes: in, This represents the fitness function value; This represents the spectral Fisher information. Represents the balance coefficient. Denotes the second moment in space; With the goal of minimizing the fitness function value, the structural parameters of the diffractive optical element are iteratively optimized using a genetic algorithm, and the optimized structural parameters are output as the initial optimized structural parameters of the diffractive optical element.

[0010] In some embodiments of this application, the step of jointly optimizing the structural parameters of the diffractive optical element and a hyperspectral reconstruction network based on the initial optimized structural parameters, the differentiable optical propagation model corresponding to the imaging system, and hyperspectral training data to obtain the target optimized structural parameters for optimizing the diffractive optical element in the imaging system, and the optimized hyperspectral reconstruction network, includes: The initial optimized structural parameters are used as the initial values ​​for the structural parameters of the diffractive optical element; The differentiable optical propagation model is cascaded with a hyperspectral reconstruction network to serve as an end-to-end differentiable optimization model. Using hyperspectral training data, and through a backpropagation algorithm with the objective of minimizing a preset total loss function, the structural parameters of the diffractive optical element and the network weights of the hyperspectral reconstruction network in the end-to-end differentiable optimization model are jointly optimized for at least one iteration. The total loss function is a weighted sum of a quantization loss function and a preset reconstruction loss function. The quantization loss function characterizes the difference between the structural parameters of the diffractive optical element and preset discrete step values. If the current iteration meets the preset convergence condition, the optimized structural parameters are output as the target optimized structural parameters, and the optimized hyperspectral reconstruction network is also output.

[0011] In some embodiments of this application, the weight coefficient of the quantization loss function in the total loss function gradually increases with each iteration, so that the structural parameters of the diffractive optical element gradually converge to the discrete step value.

[0012] In some embodiments of this application, The hyperspectral reconstruction network is a spatial-spectral dual-attention network; The spatial-spectral dual attention network adopts a U-shaped architecture, including an encoder, a decoder, and skip connections; The encoder is used to receive input compressed image data and extract feature maps through downsampling operations, wherein the feature map output by the last layer of the encoder is used as the encoded feature output. The decoder is used to receive the encoded features, restore the spatial resolution through upsampling operations, and output the reconstructed hyperspectral data cube; The skip connection is used to pass the feature maps output by the layers other than the last layer in the encoder to the corresponding scale layer in the decoder; Each layer in the encoder and the decoder contains at least one spatial-spectral attention module; The spatial-spectral attention module includes a spatial path and a spectral path: The spatial path is used to receive the input feature map, extract spatial features through a 3×3 convolutional layer, and output spatially enhanced features; The spectral path is used to receive the same input feature map of the spatial path, extract spectral features through a window-based multi-head self-attention mechanism, and output spectral enhancement features; wherein, the window size of the window-based multi-head self-attention mechanism is set according to the diffusion range of the point spread function of the imaging system. The spatial-spectral attention module is also used to fuse the spatial enhancement features with the spectral enhancement features and output the fused feature map.

[0013] A second aspect of this application provides a hyperspectral image reconstruction method, comprising: The target optimized structural parameters obtained by the optimization method of the diffraction and refraction hybrid snapshot hyperspectral imaging system provided in the first aspect are configured in the diffraction optical element of the diffraction and refraction hybrid snapshot hyperspectral imaging system. A single compressed grayscale image is acquired using the image sensor in the configured imaging system; The compressed grayscale image is input into the optimized hyperspectral reconstruction network obtained by the optimization method of the diffraction and refraction hybrid snapshot hyperspectral imaging system, so that the optimized hyperspectral reconstruction network outputs the reconstructed hyperspectral data cube.

[0014] A third aspect of this application provides an optimized apparatus for a hybrid diffraction and refraction snapshot hyperspectral imaging system, the imaging system comprising a refractive lens, a diffractive optical element disposed at the aperture stop of the refractive lens, and an image sensor, the optimized apparatus comprising: The first-stage optimization module uses a genetic algorithm to perform a global search optimization of the structural parameters of the diffractive optical element, using a function composed of the weighted difference between spectral Fisher information and the second spatial moment as the fitness function, to obtain the initial optimized structural parameters of the diffractive optical element; wherein, the spectral Fisher information is used to quantify the sensitivity of the point spread function to wavelength, and the second spatial moment is used to quantify the energy dispersion of the point spread function over a wide wavelength range; The second-stage optimization module is used to jointly optimize the structural parameters of the diffractive optical element and a hyperspectral reconstruction network based on the initial optimized structural parameters, the differentiable optical propagation model corresponding to the imaging system, and the hyperspectral training data, to obtain the target optimized structural parameters for optimizing the diffractive optical element in the imaging system, and the optimized hyperspectral reconstruction network.

[0015] A fourth aspect of this application provides a hyperspectral image reconstruction apparatus, comprising: An imaging system optimization module is used to configure the target optimized structural parameters obtained by the optimization method of the diffraction and refraction hybrid snapshot hyperspectral imaging system provided by the first aspect to the diffraction optical element of the diffraction and refraction hybrid snapshot hyperspectral imaging system. The image acquisition module is used to acquire a single compressed grayscale image using the image sensor in the configured imaging system; The image reconstruction module is used to input the compressed grayscale image into the optimized hyperspectral reconstruction network obtained by the optimization method of the diffraction and refraction hybrid snapshot hyperspectral imaging system, so that the optimized hyperspectral reconstruction network outputs the reconstructed hyperspectral data cube.

[0016] The fifth aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the optimization method of the diffraction and refraction hybrid snapshot hyperspectral imaging system described in the first aspect and / or the hyperspectral image reconstruction method described in the second aspect.

[0017] The sixth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the optimization method for the diffraction and refraction hybrid snapshot hyperspectral imaging system described in the first aspect and / or the hyperspectral image reconstruction method described in the second aspect.

[0018] The seventh aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the optimization method for the diffraction and refraction hybrid snapshot hyperspectral imaging system described in the first aspect and / or the hyperspectral image reconstruction method described in the second aspect.

[0019] The optimization method for the hybrid snapshot hyperspectral imaging system provided in this application, by constructing a hybrid optical architecture consisting of a refractive lens and a diffractive optical element located at its aperture stop, can decouple spectral dispersion and spatial focusing functions. This allows the refractive lens to undertake the tasks of light collection and focusing, while the diffractive optical element focuses on spectral encoding. This can suppress zero-order background haze, improve the system's light throughput and signal-to-noise ratio, and compress the diffusion radius of the point spread function, thus preserving more spatial high-frequency details for subsequent reconstruction. In the optimization process, a genetic algorithm is used to globally search the structural parameters of the diffractive optical element. The fitness function is the weighted difference between spectral Fisher information and spatial second moment. Spectral Fisher information quantifies the sensitivity of the point spread function to wavelength, guiding the diffractive optical element to produce wavelength-discriminable diffraction patterns to enhance spectral encoding capabilities. Spatial second moment quantifies the energy dispersion of the point spread function over a wide wavelength range, constraining energy concentration towards the optical axis. The weighted difference between the two forms a mutually restraining physical driving objective, thereby guiding the genetic algorithm to quickly locate the globally optimal attraction region in the discrete structural space, avoiding the pitfalls of traditional gradient optimization that easily get trapped in local minima. The initial optimized structural parameters obtained from the global search are used as the initialization parameters for joint optimization. Based on a differentiable optical propagation model and hyperspectral training data, end-to-end joint optimization is performed on the structural parameters of the diffractive optical element and the hyperspectral reconstruction network. This utilizes initial values ​​that are close to the optimal solution to accelerate the convergence process and improve the performance of the final optimized structural parameters and reconstruction network, enabling the imaging system to achieve high-precision hyperspectral data cube reconstruction.

[0020] Additional advantages, objectives, and features of this application will be set forth in part in the description which follows, and will in part become apparent to those skilled in the art upon review of the following description, or may be learned by practice of the application. The objectives and other advantages of this application can be realized and obtained by means of the structures specifically pointed out in the specification and drawings.

[0021] Those skilled in the art will understand that the purposes and advantages that can be achieved with this application are not limited to those specifically described above, and that the above and other purposes that this application can achieve will be more clearly understood from the following detailed description. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. The components in the drawings are not drawn to scale but are merely for illustrating the principles of this application. For ease of illustration and description of certain parts of this application, corresponding portions in the drawings may be enlarged, i.e., may appear larger relative to other components in an exemplary device actually manufactured according to this application. In the drawings: Figure 1 This is a schematic diagram of the structure of a hyperspectral imaging system with a hybrid diffraction and refraction snapshot according to an embodiment of this application.

[0023] Figure 2 This is a schematic diagram of the first process of optimizing a hyperspectral imaging system with a hybrid diffraction and refraction snapshot according to an embodiment of this application.

[0024] Figure 3 This is a schematic diagram of the second process of the optimization method for the diffraction and refraction hybrid snapshot hyperspectral imaging system in one embodiment of this application.

[0025] Figure 4 This is a schematic diagram of the third process of the optimization method for the diffraction and refraction hybrid snapshot hyperspectral imaging system in one embodiment of this application.

[0026] Figure 5(a) is a schematic diagram of the structure of a spatial-spectral dual attention network (S2DA-Net) in one embodiment of this application.

[0027] Figure 5(b) is a schematic diagram of the spatial-spectral transformation module (S2TB) in one embodiment of this application.

[0028] Figure 5(c) is a schematic diagram showing the design details of the dual-path (spatial convolution + spectral window attention) design of the spatial-spectral attention module (S2AB) in one embodiment of this application.

[0029] Figure 6 This is a flowchart illustrating a hyperspectral image reconstruction method according to an embodiment of this application.

[0030] Figure 7 This is a comparison diagram of the training convergence dynamics of the two-stage optimization strategy and the traditional random initialization strategy in an application example of this application.

[0031] Figure 8 This is a comparison diagram of the training convergence dynamics of the two-stage optimization strategy and the traditional random initialization strategy in an application example of this application.

[0032] Figure 9(a) is a physical image of the prototype machine in an application example of this application.

[0033] Figure 9(b) is a comparison of the light flux and the original data in an application example of this application.

[0034] Figure 9(c) is a comparison of the spectral curve reconstruction results of four selected points in an application example of this application.

[0035] Figure 9(d) is a comparison of the resolution plate imaging results in an application example of this application.

[0036] Figure 9(e) is a schematic diagram of the normalized current response of the image sensor over time in an application example of this application.

[0037] Figure 9(f) is a comparison of the RGB synthesis and reconstruction results of a complex real-world scene in an application example of this application.

[0038] Figure 10 This is a schematic diagram of the structure of an optimized device for a hybrid snapshot hyperspectral imaging system of diffraction and refraction provided in an embodiment of this application.

[0039] Figure 11 This is a schematic diagram of the structure of a hyperspectral image reconstruction apparatus provided in an embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain this application, but are not intended to limit it.

[0041] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the structures and / or processing steps closely related to the scheme according to this application are shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0042] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0043] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0044] In the following description, embodiments of the present application will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0045] To address issues such as single-chip DOE overload (dispersion and focusing conflict), low luminous flux, poor signal-to-noise ratio, severe PSF diffusion, loss of spatial details, and gradient optimization's susceptibility to local minima and lack of physical interpretability, this application provides an optimization method for a diffraction and refraction hybrid snapshot hyperspectral imaging system, a hyperspectral image reconstruction method, an optimization device for the diffraction and refraction hybrid snapshot hyperspectral imaging system for executing the optimization method, a hyperspectral image reconstruction device for executing the hyperspectral image reconstruction method, an electronic device, a computer-readable storage medium, and a computer program product. These are applicable to fields requiring real-time acquisition of high-dimensional spectral information, such as remote sensing, biomedical diagnostics, precision agriculture, and consumer electronics.

[0046] The following examples will provide a detailed description.

[0047] Based on this, embodiments of this application provide an optimization method for a hybrid diffraction and refraction snapshot hyperspectral imaging system, which can be implemented by an optimization device for such a system. Specific details are as follows: See Figure 1 The hardware structure of the hybrid diffraction and refraction snapshot hyperspectral imaging system includes a refractive lens, a diffractive optical element (DOE), and a CMOS image sensor. The refractive lens has an aperture stop, and the diffractive optical element is embedded at the aperture stop of the refractive lens. Incident light first passes through the refractive lens, which is responsible for efficient light collection and spatial focusing. The diffractive optical element, located at the aperture stop, applies wavelength-dependent phase modulation to the passing light beam, achieving precise spectral dispersion. The refractive lens focuses the undiffracted zero-order light into a compact spot, preventing it from spreading into background haze (i.e., stray background light). The light beam, modulated by both the refractive lens and the diffractive optical element, finally reaches the CMOS image sensor, which acquires a single compressed grayscale image. This hybrid architecture allows the refractive lens to perform the primary focusing task, while the diffractive optical element focuses on spectral encoding, thereby improving light throughput and signal-to-noise ratio while providing high-quality raw data for subsequent hyperspectral image reconstruction.

[0048] See Figure 2 Based on the above-mentioned diffraction and refraction hybrid snapshot hyperspectral imaging system, the embodiments of the optimization method for the diffraction and refraction hybrid snapshot hyperspectral imaging system provided in this application include the following: Step 100: Using a genetic algorithm, a function consisting of the weighted difference between spectral Fisher information and spatial second moment is used as the fitness function to perform a global search optimization of the structural parameters of the diffractive optical element, thereby obtaining the initial optimized structural parameters of the diffractive optical element; wherein, the spectral Fisher information is used to quantify the sensitivity of the point spread function to wavelength, and the spatial second moment is used to quantify the energy dispersion of the broadband point spread function.

[0049] It should be noted that in step 100, the Genetic Algorithm (GA) is a global search optimization algorithm that simulates natural selection and genetic mechanisms, iteratively evolving the population through operations such as selection, crossover, and mutation. The fitness function is a quantitative indicator used to evaluate the quality of a set of structural parameters; in this embodiment, it is the weighted difference between spectral Fisher information and the spatial second moment. Spectral Fisher information measures the point spread function's ability to distinguish different wavelengths; a larger value indicates that a small wavelength change can cause a significant change in the point spread function. The spatial second moment measures the spatial dispersion of the point spread function's energy; a smaller value indicates more concentrated energy. The weighted difference is used to multiply the spectral Fisher information and the spatial second moment by a balance coefficient and then subtract them to balance spectral encoding ability and spatial focusing quality. The point spread function (PSF) is the response function of an optical system to a point light source, describing the distribution of light intensity on the image plane. In this embodiment, structural parameters refer to the height values ​​of each concentric ring of the diffractive optical element, which determine the phase modulation amount.

[0050] Specifically, in step 100, the structural parameters of the diffractive optical element are encoded as chromosomes, and the population is randomly initialized. For each individual, based on the current structural parameters, the point spread function of each wavelength channel is calculated using an optical propagation model, and then the spectral Fisher information and spatial second moment are calculated. The individual's quality is evaluated based on the fitness function value. Subsequently, the next generation of the population is generated through genetic operations such as selection, crossover, and mutation. The above iterative process is repeated until the fitness value converges or a preset number of generations is reached. Finally, the individual with the best fitness in the population is decoded, and its corresponding structural parameters are output as the initial optimized structural parameters.

[0051] It is understandable that when faced with the optimization problem of a diffraction-refractive hybrid system, those skilled in the art usually adopt the following conventional approach: (1) directly use the end-to-end gradient descent method to jointly optimize optical parameters and network weights; (2) if a genetic algorithm is used, imaging quality evaluation indicators (such as PSNR, SSIM) or a single physical quantity (such as the reciprocal of spectral resolution, the full width at half maximum of the point spread function) are usually selected as the fitness function. The optimization method provided in this application is the first to identify that "spectral separability" and "spatial focusing" under the hybrid architecture are a pair of physical contradictions that need to be specially balanced, and creatively selects quantitative indicators that can respectively characterize these two physical characteristics: spectral Fisher information and spatial second moment, and then fuses the two into a single fitness function through weighted difference to guide the genetic algorithm to directly search for the optimal solution in the discrete structure space. The existing technology has not disclosed that spectral Fisher information and spatial second moment are used as the fitness function of the genetic algorithm in the form of weighted difference, nor has it revealed that the fitness function can simultaneously guide the genetic algorithm to balance the inherent contradiction between spectral encoding capability and spatial focusing quality under the hybrid architecture. Specifically: (1) Existing technologies usually regard DOE optimization as a parameter fitting problem with a single objective, or simply take imaging quality as the optimization objective. This application, for the first time, explicitly regards wavelength sensitivity and energy dispersion as two mutually restraining physical quantities, and points out that in the hybrid architecture, the fitness function of the genetic algorithm should directly reflect this restraining relationship, rather than using the reconstruction quality index of the back end.

[0052] (2) Both spectral Fisher information and spatial second moment are known mathematical concepts, but it is not conventional in the field to simultaneously introduce them into the fitness function of a genetic algorithm and adopt a weighted difference form. When faced with the optimization of hybrid systems, those skilled in the art lack the motivation to combine Fisher information (usually used for parameter estimation) and second moment (usually used for beam quality evaluation) into a single fitness function, because they belong to different physical quantities in the spectral and spatial dimensions, and have different dimensions.

[0053] (3) The design of the fitness function is not independent of the optical architecture. The introduction of the second spatial moment term is precisely because the refractive lens has already undertaken most of the focusing task, and at this point, it is necessary to constrain the dispersion of the DOE so as not to excessively destroy the focusing effect. Without the refractive lens, simply minimizing the second spatial moment would severely suppress the dispersion capability of the DOE, leading to spectral encoding failure. Therefore, this fitness function is specifically tailored for the hybrid architecture of refractive focusing and DOE dispersion, and applying it to a pure diffraction system will not achieve the same effect. This design concept, which is deeply coupled with the optimization target, goes far beyond the scope of conventional parameter tuning.

[0054] Therefore, compared to existing technologies such as disclosed hybrid architectures, genetic algorithm-optimized DOEs, and end-to-end differentiable optimization, the technique provided in this application, which uses the weighted difference between spectral Fisher information and the second-order spatial moment as the fitness function of the genetic algorithm, is not something that those skilled in the art would readily conceive of when facing the optimization problem of a diffraction-refractive hybrid snapshot hyperspectral imaging system. This solution requires cross-disciplinary knowledge of optical physics, genetic algorithms, and spectral imaging, and is specifically designed to address the unique contradictions of the hybrid architecture.

[0055] Step 200: Based on the initial optimized structural parameters, the differentiable optical propagation model corresponding to the imaging system, and the hyperspectral training data, jointly optimize the structural parameters of the diffractive optical element and a hyperspectral reconstruction network to obtain the target optimized structural parameters for optimizing the diffractive optical element in the imaging system, and the optimized hyperspectral reconstruction network.

[0056] The Differentiable Optical Propagation Model (DOP) is a mathematical model that simulates the propagation of light from an object through refracting lenses and diffractive optical elements to an image sensor, with all computational modules supporting gradient backpropagation. Hyperspectral Training Data consists of known hyperspectral image datasets used to train the reconstruction network, typically containing multiple spatial-spectral three-dimensional data cubes. Joint Optimization is used to simultaneously update the structural parameters of the diffractive optical elements and the network weights of the hyperspectral reconstruction network, enabling them to adapt collaboratively. The Hyperspectral Reconstruction Network is a deep learning network used to recover a complete hyperspectral data cube from a single compressed grayscale image.

[0057] Specifically, in step 200, firstly, the initial optimized structural parameters output in step 100 are used as the initial values ​​for the structural parameters of the diffractive optical element. Then, the differentiable optical propagation model is cascaded with a hyperspectral reconstruction network to be trained, forming an end-to-end differentiable optimization model. Using hyperspectral training data, the structural parameters of the diffractive optical element and the network weights of the hyperspectral reconstruction network are iteratively updated using a backpropagation algorithm, with the goal of minimizing a preset total loss function (e.g., the mean square error between the reconstructed image and the ground truth). When the loss function converges or reaches a preset number of iterations, optimization stops, and the final structural parameters are output as the target optimized structural parameters, while the trained hyperspectral reconstruction network is also output. These target optimized structural parameters can be used to actually fabricate diffractive optical elements, and the optimized hyperspectral reconstruction network is used to reconstruct a hyperspectral data cube from compressed grayscale images acquired by a sensor.

[0058] As described above, the optimization method for the hybrid diffraction and refraction snapshot hyperspectral imaging system provided in this application can solve the problems of existing diffraction optical element optimization easily getting trapped in local minima, and single diffraction elements having difficulty simultaneously taking into account spectral dispersion and spatial focusing, resulting in poor imaging quality and low optimization efficiency. By guiding the genetic algorithm to perform a global search through a physical perception fitness function, the optimization process of the structural parameters of the diffraction optical element can avoid getting trapped in local minima, providing good initial values ​​for subsequent joint fine-tuning. Combining the hybrid architecture to increase light throughput can significantly improve the optimization convergence speed and the final reconstruction accuracy, thereby achieving high light throughput, high signal-to-noise ratio, and high-precision hyperspectral reconstruction.

[0059] To further address the problems in existing methods where the structural parameters of diffractive optical elements have excessively high dimensionality and lack physical constraints, resulting in a huge optimization search space, non-differentiable optical propagation models, and inability to jointly optimize with the backend reconstruction network, this application provides an optimization method for a hybrid diffraction and refraction snapshot hyperspectral imaging system. (See [link to relevant documentation]). Figure 3 The optimization method for the diffraction and refraction hybrid snapshot hyperspectral imaging system, prior to step 100, specifically includes the following: Step 010: Discretize the height map of the diffractive optical element into M×M pixel units to obtain a discretized height map; wherein each pixel unit corresponds to a height value; M is a positive integer greater than 1.

[0060] Step 020: Parameterize the discretized height map into a rotationally symmetric structure composed of multiple concentric rings and use it as the structural parameter of the diffractive optical element; wherein each concentric ring corresponds to a discrete height value.

[0061] The height map is a two-dimensional array describing the longitudinal height of each position on the surface of the diffractive optical element. The height value determines the amount of phase retardation applied to the incident light at that position. The pixel unit is the smallest discrete unit of the height map, corresponding to the smallest feature size in actual photolithography. The concentric ring is a ring centered on the optical axis, with all positions on the same ring having equal height values, used to reduce the dimensionality of optimization variables. The rotationally symmetric structure represents the surface profile of the diffractive optical element, which varies only with radial distance and is independent of angle, significantly reducing the number of structural parameters.

[0062] And, step 030: construct the total generalized pupil function; wherein the total generalized pupil function is the product of the aperture function, the phase of the refractive lens, and the modulation of the diffractive optical element.

[0063] Step 040: Based on the angular spectrum method, the square of the Fourier transform modulus of the total generalized pupil function is used as the point spread function, thereby establishing the differentiable optical propagation model corresponding to the imaging system.

[0064] The generalized pupil function is a complex function describing the modulation of the amplitude and phase of the incident light wave by the optical system on the pupil plane (usually the plane where the aperture stop is located), denoted as P(u,v,λ), where (u,v) are the pupil plane coordinates and λ is the wavelength. The aperture function describes the light-transmitting region on the pupil plane, typically taking values ​​of 1 (within the light-transmitting region) or 0 (outside the light-transmitting region). The refractive lens phase is a quadratic phase term introduced by the refractive lens to achieve light convergence; its mathematical expression is: f is the focal length of the lens. Modulation in diffractive optical elements is the wavelength-dependent phase delay introduced by the diffractive optical element. Let f be... Let DOE be in the pupil plane coordinates Discrete height maps at the location. Wavelength-dependent phase delay introduced by DOE. for: in, The refractive index of the material. A refractive lens introduces a second-order phase term. Used for focusing. According to the thin lens approximation, the total generalized pupil function... It is the product of the aperture function, lens phase, and modulation of the diffractive optical element.

[0065] The angular spectrum method is a numerical calculation method for light wave propagation based on Fourier transform, capable of accurately calculating the diffraction process of the light field from the pupil plane to the sensor plane. The point spread function (PSF) represents the response of an optical system to a point source; in this embodiment, it is given by the square of the Fourier transform modulus of the total generalized pupil function. That is, the point spread function (PSF) reaching the sensor plane. Derived from the angular spectrum method, it is expressed as the square of the Fourier transform modulus of the pupil function: in,( x , y ) represents the sensor plane coordinates, and F represents the two-dimensional Fourier transform.

[0066] The differentiable optical propagation model is a mathematical model constructed based on the aforementioned angular spectrum method and the total generalized pupil function. All operations support gradient backpropagation, and it is used to calculate the point spread function and simulate the imaging process in subsequent joint optimization. The differentiable optical propagation model also describes the compressed grayscale image acquired by the image sensor as a convolutional superposition of the scene spectral cube and the wavelength-dependent point spread function. That is, the compressed grayscale image acquired by the image sensor... Scene spectral cube Convolutional superposition of wavelength-dependent PSFs: As can be seen from the above description, the optimization method of the hyperspectral imaging system of the diffraction and refraction hybrid snapshot provided in this application reduces the optimization variables to a tractable scale by using concentric ring rotational symmetry parameterization, and establishes a differentiable optical propagation model based on the angular spectrum method, providing an accurate and differentiable imaging simulation basis for subsequent global search of genetic algorithms and end-to-end joint optimization.

[0067] A two-stage optical design method based on physical perception: Addressing the challenge of a highly nonlinear mapping between phase modulation and imaging quality in diffractive optical elements (DOEs), and the problem of optimization surfaces containing numerous local minima, traditional gradient-based end-to-end optimization methods are prone to getting trapped in suboptimal solutions. To this end, this application proposes a two-stage optimization strategy combining physical-driven global search with end-to-end fine-tuning. Through the organic combination of discrete and continuous optimization, it achieves a balance between high performance and manufacturability. To further improve the effectiveness and applicability of optimizing hybrid diffraction and refraction snapshot hyperspectral imaging systems, an optimization method for a hybrid diffraction and refraction snapshot hyperspectral imaging system is provided in an embodiment of this application. Based on this, see Figure 4 Step 100 of the optimization method for the diffraction and refraction hybrid snapshot hyperspectral imaging system specifically includes the following: Step 110: Calculate the point spread function of the imaging system in each wavelength channel according to the differentiable optical propagation model; wherein, the wavelength channel refers to the discrete spectral interval divided within the working band of the imaging system.

[0068] Step 120: Calculate the spectral Fisher information based on the point spread function of each wavelength channel.

[0069] And, step 130: calculate the spatial second moment based on the point spread function intensity distribution obtained from the differentiable optical propagation model.

[0070] Step 140: Substitute the spectral Fisher information and the spatial second moment into the fitness function to calculate the fitness function value.

[0071] Step 150: With the goal of minimizing the fitness function value, the structural parameters of the diffractive optical element are iteratively optimized using a genetic algorithm, and the optimized structural parameters are output as the initial optimized structural parameters of the diffractive optical element.

[0072] Specifically, in the step of using a genetic algorithm to perform a global search optimization of the structural parameters of the diffractive optical element, using a function composed of the weighted difference between spectral Fisher information and the second spatial moment as the fitness function, to obtain the initial optimized structural parameters of the diffractive optical element, the specific implementation method is as follows: To quickly pinpoint the approximate range of the global optimal solution within the discrete space, the structural parameters of the diffractive optical element are first set to be determined by... N A rotationally symmetric discrete structure consisting of concentric rings. For this discrete structure, a fitness function with specific physical constraints is defined to guide the search: in, This represents the fitness function value; This represents the spectral Fisher information. Represents the balance coefficient. This represents the second-order moment in space. The fitness function is not a universal image quality evaluation metric, but rather a physically guided constraint built upon the spectral compression transmission mechanism under hybrid optical paths, containing two mutually balancing quantitative metrics: (1) Spectral Fisher information ( ): Where C is the number of wavelength channels. Let be the normalized point spread function for the Cth wavelength channel; The Frobenius norm is used to maximize the system's spectral encoding capability. Based on wavelength difference calculations, this metric quantifies the sensitivity of the point spread function (PSF) morphology to wavelength variations, requiring that even small wavelength changes cause significant alterations in the PSF diffraction pattern, thus ensuring effective separation of information from different wavelength bands. Fisher information essentially measures the system's sensitivity to minute changes. Physically, it forces the DOE to produce diffraction spots with significantly different morphologies when faced with extremely small wavelength changes, thereby maximizing the spectral encoding diversity and reducing the difficulty of subsequent network decoupling of spectral aliasing from a physical source.

[0073] (2) Spatial second moment ( ): in, Let (xc, yc) be the band cumulative intensity distribution of the point spread function, and (xc, yc) be the coordinates of the energy centroid. This is a focusing constraint term specific to hybrid optical paths. To combat the spot enlargement problem introduced by strong dispersion in DOE, a second-order spatial moment is introduced. This metric quantifies the energy diffusion of the wideband PSF, constraining energy to concentrate towards the optical axis to achieve high signal-to-noise ratio imaging in conjunction with the refractive lens. It calculates the dispersion radius of the entire light spot. During optimization, it forces the diverging diffracted light energy back towards the center of the optical axis. This guides the optimization algorithm to maintain a relatively compact light spot as much as possible in conjunction with the refractive lens, while ensuring spectral separability, thus guaranteeing high luminous throughput and high spatial resolution of the system.

[0074] To minimize the fitness function value With the goal of optimizing the structure parameters of the diffractive optical element through iterative optimization using a genetic algorithm, the optimized structure parameters are output as the initial optimized structure parameters of the diffractive optical element.

[0075] As described above, the optimization method for the diffraction and refraction hybrid snapshot hyperspectral imaging system provided in this application firstly utilizes spectral Fisher information to quantify the sensitivity of the point spread function to wavelength changes, guiding the genetic algorithm to prioritize the search for structural parameters that significantly differentiate the shape of the point spread function across different wavelength channels. This enhances the system's spectral encoding capability from a physical source and reduces the difficulty of correcting spectral aliasing during subsequent reconstruction. Secondly, the spatial second moment quantifies the energy dispersion of the point spread function across a wide band, guiding the genetic algorithm to simultaneously constrain energy concentration towards the optical axis during optimization. This, combined with the refractive lens in the hybrid architecture, maintains a compact spot shape, which is beneficial for improving the system's luminous flux and spatial resolution. By fusing two mutually constraining physical indices into a single fitness function, the genetic algorithm can simultaneously consider spectral separability and spatial focusing quality in a discrete structural space, avoiding the pitfalls of traditional gradient optimization methods that fall into local minima due to non-convex surfaces, and quickly locking onto the attraction domain where the global optimum is located. The initial optimized structural parameters obtained in this way serve as the starting point for subsequent joint fine-tuning, shortening the convergence rounds required for the second-stage optimization and improving the overall performance of the final imaging system.

[0076] Furthermore, existing purely data-driven methods typically treat optical layers as black boxes, lacking physical interpretability, and often optimize height profiles in continuous space, ignoring discretization and quantization constraints in actual photolithography. This neglect of manufacturing processes leads to a significant simulation-to-physical gap, causing a substantial performance degradation of the designed structure after actual fabrication. Therefore, to further address the problem that existing end-to-end optimization methods optimize the structural parameters of diffractive optical elements in continuous space, neglecting discretization and quantization constraints in actual photolithography, resulting in a significant performance degradation of the optimized continuous surface after quantization and a severe "simulation-to-physical" gap, this application provides an optimization method for a hybrid diffraction and refraction snapshot hyperspectral imaging system, see [link to relevant documentation]. Figure 4 Step 200 of the optimization method for the diffraction and refraction hybrid snapshot hyperspectral imaging system specifically includes the following: Step 210: Use the initial optimized structural parameters as the initial values ​​of the structural parameters of the diffractive optical element; Step 220: Cascade the differentiable optical propagation model with a hyperspectral reconstruction network to form an end-to-end differentiable optimization model; Step 230: Using hyperspectral training data, through backpropagation algorithm and with the objective of minimizing the preset total loss function, jointly optimize the structural parameters of the diffractive optical element and the network weights of the hyperspectral reconstruction network in the end-to-end differentiable optimization model for at least one iteration; wherein, the total loss function is a weighted sum of the quantization loss function and the preset reconstruction loss function; the quantization loss function is used to characterize the difference between the structural parameters of the diffractive optical element and the preset discrete step value; Step 240: If the current iteration meets the preset convergence condition, output the optimized structural parameters as the target optimized structural parameters, and output the optimized hyperspectral reconstruction network.

[0077] If the current iteration does not meet the preset convergence condition, then the joint optimization of the next iteration is performed based on the optimized structural parameters output by the current iteration (i.e., the hyperspectral training data is reused to update the structural parameters and network weights through the backpropagation algorithm until the convergence condition is met).

[0078] During joint optimization, the preset convergence conditions can be any one or more of the following combinations: (1) Loss function threshold: When the total loss function value (or reconstruction loss on the validation set) is lower than the preset threshold, it is determined to be converged.

[0079] (2) Iteration cycle limit: When the number of iteration cycles reaches the preset maximum number of cycles (e.g., epoch=100), the optimization stops.

[0080] (3) Rate of change of loss function: When the decrease of the total loss function over several consecutive rounds (e.g., 10 consecutive epochs) is less than a preset proportion (e.g., the rate of change is less than 0.1%), it is determined to be converged.

[0081] (4) Parameter change: When the absolute value of the update of the structural parameters (or network weights) of the diffractive optical element in continuous iteration is less than the preset threshold, it is determined to be converged.

[0082] (5) Early Stopping: Monitor reconstruction quality metrics (such as PSNR) on the validation set. When the metrics no longer improve for several consecutive rounds, stop optimization and revert to the best model.

[0083] In one example, a combination of the second and third methods mentioned above can be used: set the maximum number of iterations to 100, and terminate the optimization early if the total loss decreases by less than 0.1% over 10 consecutive iterations.

[0084] Specifically, the step of jointly optimizing the structural parameters of the diffractive optical element and a hyperspectral reconstruction network based on the initial optimized structural parameters, the differentiable optical propagation model corresponding to the imaging system, and the hyperspectral training data is implemented as follows: Traditional end-to-end optimization typically uses the mean squared error (MSE) of the reconstructed image directly as the loss function. However, in diffractive optics, the mapping from the height map of a diffractive optical element to the final image's MSE is an extremely non-convex "black box." If the initial diffraction pattern of the diffractive optical element is too large, the information from each wavelength will intertwine into a featureless background stray light, causing the gradient direction of the neural network to become chaotic, ultimately remaining stuck in a local minimum.

[0085] This application utilizes a genetic algorithm in the first stage to perform a global search in the discrete space, locking down the "attraction region" where the global optimum is located. Upon entering the second stage, the initial optimized structural parameters (discrete height vector) are... Expanded to continuous learnable tensors , which serves as the initial value for the structural parameters of the diffractive optical element. This initialization method is equivalent to injecting prior knowledge of physical optics (i.e., excellent spectral separation and spatial focusing) into the network weights at the beginning of neural network training, so that the joint optimization in the second stage only needs to be corrected within the physically feasible region, effectively avoiding gradient optimization from falling into local traps.

[0086] To address the disconnect between design and manufacturing, this application integrates the hierarchical constraints of photolithography into an end-to-end differentiable training process. Specifically, this is achieved by constructing a quantization loss function, which characterizes the difference between the structural parameters of the diffractive optical element and preset discrete step values. The expression is: Wherein, H is the structural parameter (continuous height value tensor) of the diffractive optical element. It is a quantization operator used to convert continuous height values Mapped to the nearest allowable lithographic step .

[0087] During backpropagation, the quantization loss function generates a gradient, forcing continuous structural parameters to converge on the optimization surface towards physically manufacturable discrete points. This process simulates the approximation process from ideal design to actual manufacturing. The weighted sum of the quantization loss function and the reconstruction loss function is used as the total loss function. Using hyperspectral training data, the backpropagation algorithm is employed to jointly optimize the structural parameters of the diffractive optical element and the network weights of the hyperspectral reconstruction network in the end-to-end differentiable optimization model, with the goal of minimizing the total loss function. When a preset convergence condition is met, the optimized structural parameters are output as the target optimized structural parameters, and the optimized hyperspectral reconstruction network is also output. The resulting diffractive optical element surface shape not only boasts optimal optical performance but also directly matches the processing depth of the lithography machine, eliminating the performance gap between simulation and reality.

[0088] As can be seen from the above description, the optimization method of the hyperspectral imaging system of the diffraction and refraction hybrid snapshot provided in this application embodiment can force the structural parameters of the diffraction optical element to converge to the preset discrete step value during the optimization process by constructing a quantization loss function. This allows the final output target optimized structural parameters to directly match the processing accuracy of the photolithography process, thereby eliminating the performance gap between simulation and physical objects and ensuring that the processed diffraction optical element can maintain the optical performance of the design stage.

[0089] Furthermore, if strong quantization constraints are applied in the early stages of optimization, the structural parameters may fall into the vicinity of discrete points too early, making it impossible to fully explore the optimal solution in the continuous space, resulting in suboptimal final performance. Based on this, in the optimization method of a hybrid diffraction and refraction snapshot hyperspectral imaging system provided in this application embodiment, the weight coefficient of the quantization loss function in the total loss function gradually increases with the iteration rounds, so that the structural parameters of the diffraction optical element gradually converge to the discrete step value.

[0090] Specifically, in the joint optimization process, the weighting coefficient of the quantization loss function in the total loss function The number of iterations gradually increases. Specifically, in the early stages of training, the number of iterations increases. Setting it to a smaller value allows the structural parameter H of the diffractive optical element to fully explore a broad solution space in a continuous space; it is gradually increased in the later stages of training. The value of H causes the structural parameter H of the diffractive optical element to gradually "harden," that is, gradually converge to the preset discrete step value. Through this progressive hardening strategy, the final output optimized structural parameters maintain optimal optical performance while strictly meeting the discretization requirements of photolithography.

[0091] As described above, the optimization method for the hybrid diffraction and refraction snapshot hyperspectral imaging system provided in this application, by gradually increasing the weight coefficient of the quantization loss function with each iteration, maintains a relatively small constraint strength in the early stage of optimization, allowing the structural parameters to fully explore a broad solution space in a continuous space; in the later stage of optimization, the constraints are gradually strengthened, causing the structural parameters to gradually harden to a preset discrete step value. This progressive hardening strategy can balance global optimization capability with manufacturing process compliance, achieving manufacturability while ensuring optimal optical performance.

[0092] To address the issues in existing snapshot hyperspectral imaging systems based on monolithic diffraction elements, such as severe front-end optical blurring and a large point spread function (PSF) diffusion range, which forces back-end reconstruction algorithms to employ computationally intensive global attention or large-kernel convolutional networks to recover spatial details, making it difficult to meet the real-time processing requirements of edge devices; and the fact that existing network designs do not fully utilize the characteristics of the hybrid optical front-end that has already compressed the PSF diffusion range, leading to wasted computational resources, this application provides an optimization method for a diffraction and refraction hybrid snapshot hyperspectral imaging system. The hyperspectral reconstruction network in this optimization method is a spatial-spectral dual-attention network: the spatial-spectral dual-attention network adopts a U-shaped architecture, including an encoder, a decoder, and skip connections.

[0093] (1) The encoder is used to receive input compressed image data and extract feature maps through downsampling operation, wherein the feature map output by the last layer of the encoder is used as the encoded feature output; (2) The decoder is used to receive the encoded features, restore the spatial resolution through upsampling operations, and output the reconstructed hyperspectral data cube; (3) The skip connection is used to pass the feature maps output by the other layers in the encoder except the last layer to the corresponding scale layer in the decoder.

[0094] Spatial-Spectral Dual Attention Reconstruction Network (S2DA-Net): Existing standalone DOE systems have extremely large point spread function (PSF) diffusion ranges, necessitating the inclusion of computationally complex global deblurring modules (such as GlobalTransformer) in the back-end reconstruction network. In contrast, the front-end optical system of this application significantly compresses the PSF radius through a refraction-diffraction hybrid design, with the optical layer essentially handling most of the "spatial focusing" work. Based on this unique physical characteristic, this application designs a lightweight spatial-spectral dual attention network. This network design abandons the redundant global spatial attention mechanism, concentrating computational resources on solving the spectral aliasing problem caused by diffraction, achieving deep coupling between the algorithm architecture and the front-end optical characteristics. The network input is a single compressed grayscale image captured by the sensor, and the output is a reconstructed hyperspectral data cube. The network architecture adopts a U-shaped architecture, with the core component being the spatial-spectral attention module (S2AB), meaning that each layer in the encoder and decoder contains at least one spatial-spectral attention module. The spatial-spectral attention module includes a spatial path and a spectral path: (1) The spatial path is used to receive the input feature map, extract spatial features through a 3×3 convolutional layer, and output spatial enhancement features.

[0095] Specifically, since the spatial sharpness of the image has been ensured by the refractive lens in the hybrid optical path, the spatial path can extract sufficient spatial details using only a lightweight 3×3 convolution, without the need to introduce a complex global deblurring module, thereby significantly reducing the number of model parameters and computational load.

[0096] (2) The spectral path is used to receive the same input feature map of the spatial path, extract spectral features through a window-based multi-head self-attention mechanism, and output spectral enhancement features; wherein, the window size of the window-based multi-head self-attention mechanism is set according to the diffusion range of the point spread function of the imaging system. Specifically, to address the local spectral aliasing characteristics caused by DOE dispersion, this spectral path utilizes a window-based multi-head self-attention mechanism. The size of the attention window is set according to the maximum diffraction dispersion range of the optical system, and the window size W is set as follows: in, The average diameter of the 98% energy circle of the system's PSF is used. This design, while saving computational resources, enables the decoding of local spectral correlations and the precise separation of overlapping spectral signals.

[0097] The spatial-spectral attention module is also used to fuse the spatial enhancement features with the spectral enhancement features and output the fused feature map.

[0098] Specifically, as shown in Figure 5(a), the spatial-spectral dual attention network adopts a U-shaped architecture, including an encoder, a decoder, and skip connections.

[0099] (1) Encoder: Receives the input compressed image data (single grayscale image) and sequentially passes it through multiple S2TBs (Spatial-Spectral Transformer Blocks). Each S2TB contains a downsampling operation (as shown in Figure 5(b)), which gradually reduces the spatial resolution of the feature map through stride convolution or pooling, while increasing the number of channels. The feature map output from the last S2TB of the encoder is used as the encoded feature output.

[0100] (2) Decoder: Receives the encoded features and sequentially processes them through multiple S2TBs. Each S2TB contains an upsampling operation (as shown in Figure 5(b)), which gradually restores the spatial resolution of the feature map through transposed convolution or interpolation until it reaches the size of the input image. The last S2TB of the decoder outputs a reconstructed hyperspectral data cube.

[0101] (3) Skip connection: The feature maps output by the S2TBs other than the last S2TB in the encoder (i.e. the feature maps after downsampling at each stage) are passed to the corresponding scale S2TB in the decoder and spliced ​​or added with the feature maps after upsampling at the current stage of the decoder to fuse shallow spatial details.

[0102] Each S2TB contains one or more spatial-spectral attention modules (S2AB), the structure of which is shown in Figure 5(c).

[0103] Furthermore, referring to Figure 5(b), the internal structure of the S2TB is shown. Its processing flow is as follows: (1) Down sampling or up sampling: In the encoder stage, the spatial resolution of the input feature map is first reduced by a downsampling layer (such as a convolution with a stride of 2), and the downsampled feature map is output.

[0104] In the decoder stage, the spatial resolution of the input feature map is first improved by an upsampling layer (such as transposed convolution or bilinear interpolation), and the upsampled feature map is output.

[0105] If the S2TB is located in the deepest layer of the encoder or the shallowest layer of the decoder, this step can be omitted.

[0106] (2) Spatial-Spectral Attention Module (S2AB): The feature map obtained in the previous step is input into one or more cascaded S2ABs. The detailed structure of each S2AB is shown in Figure 5(c).

[0107] (3) 1×1 convolution (CONV1x1): The feature map output by S2AB is adjusted by 1×1 convolution, and the residuals are added or concatenated with the feature map passed by the skip connection (if any), and finally the result of S2TB is output.

[0108] Furthermore, referring to Figure 5(c), the detailed structure of S2AB is shown. The spatial-spectral attention module includes a spatial path and a spectral path, with both paths sharing the same input feature map.

[0109] (1) Spatial path: Received input feature map .

[0110] The cells are passed through two 3×3 convolutional layers in sequence, with a GELU activation function sandwiched in between (labeled as GELL in Figure 5(c)).

[0111] Output spatial enhancement features (feature maps output by spatial paths) .

[0112] (2) Spectral path: The same input feature map of the spatial path is collected.

[0113] First, layer normalization (LayerNorm) is performed.

[0114] Features are projected onto the dimensions of query, key, and value through a linear transformation. This represents the projection weights (learnable parameters in the linear transformation layer).

[0115] A window-based multi-head self-attention mechanism is implemented, with the window size set according to the diffusion range of the point spread function of the imaging system.

[0116] The attention output is then projected back to the original number of channels using a linear transformation.

[0117] Perform a residual connection with the input feature map (addition operation A).

[0118] Output spectral enhancement features (feature map of spectral path output) .

[0119] (3) Feature fusion: spatial enhancement features Add the spectral enhancement features element by element.

[0120] Optionally, the number of channels can be adjusted using a 1×1 convolution (CONV1x1).

[0121] Output the fused feature map .

[0122] in, represents the relative position bias (a learnable parameter used to encode the relative position in multi-head self-attention); Softmax represents the normalized exponential function (an activation function that converts attention scores into a probability distribution).

[0123] As described above, the optimization method for the diffraction and refraction hybrid snapshot hyperspectral imaging system provided in this application adopts a U-shaped architecture and a spatial-spectral dual attention module. Spatial feature extraction is performed by a lightweight 3×3 convolution, while spectral feature extraction is performed by window self-attention. The window size is set according to the diffusion range of the point spread function of the front-end optical system, so that the computational complexity of the network matches the physical characteristics of the front end. This can significantly reduce the number of parameters and computational load while ensuring the accuracy of hyperspectral reconstruction, and meet the real-time processing requirements of the edge end. At the same time, skip connections preserve multi-scale spatial details, further improving the spatial resolution and spectral fidelity of the reconstructed image.

[0124] Based on the above-described optimization method for a hybrid diffraction and refraction snapshot hyperspectral imaging system, and to further address the problems in practical use of existing snapshot hyperspectral imaging systems—either poor reconstruction quality due to unoptimized front-end optics or overly complex back-end networks that prevent real-time processing and hinder high-precision, low-latency hyperspectral image reconstruction on edge devices—this application also provides an embodiment of a hyperspectral image reconstruction method, see [link to embodiment]. Figure 6 The hyperspectral image reconstruction method specifically includes the following: Step 300: The target optimized structural parameters obtained by the optimization method of the aforementioned diffraction and refraction hybrid snapshot hyperspectral imaging system are configured in the diffraction optical element of the diffraction and refraction hybrid snapshot hyperspectral imaging system.

[0125] Step 400: Acquire a single compressed grayscale image using the image sensor in the configured imaging system.

[0126] Step 500: Input the compressed grayscale image into the optimized hyperspectral reconstruction network obtained by the optimization method of the diffraction and refraction hybrid snapshot hyperspectral imaging system, so that the optimized hyperspectral reconstruction network outputs the reconstructed hyperspectral data cube.

[0127] It is understood that in the hyperspectral image reconstruction method, the configuration refers to converting the target optimized structural parameters obtained from the optimization method of the diffraction and refraction hybrid snapshot hyperspectral imaging system provided in the aforementioned embodiments into actual processing data, processing the diffraction optical element into the corresponding surface morphology through photolithography, and installing it at the aperture stop of the diffraction and refraction hybrid snapshot hyperspectral imaging system. In specific implementation, the configured system already possesses optimized spectral dispersion and spatial focusing performance.

[0128] The optimization method of the hyperspectral image reconstruction method of this application for the diffraction and refraction mixed snapshot hyperspectral imaging system can refer to the processing flow of the optimization method of the diffraction and refraction mixed snapshot hyperspectral imaging system in the foregoing embodiments. Its function will not be repeated here, but can be referred to the detailed description of the above-mentioned optimization method of the diffraction and refraction mixed snapshot hyperspectral imaging system.

[0129] Taking agricultural product disease detection as an example, the user first selects the target optimized structural parameters (e.g., the height values ​​of 64 concentric rings) obtained through optimization methods. The image is fed into a photolithography machine to fabricate the corresponding diffractive optical element on a silicon dioxide glass substrate. The diffractive optical element is then embedded in the aperture stop of a refractive lens (focal length 50mm) and combined with a CMOS image sensor (pixel size 5μm) to form a complete imaging system.

[0130] In actual testing, the user points the system at the leaf to be tested and presses the shutter. The image sensor acquires a compressed grayscale image with a resolution of 1024×1024 within a single exposure (e.g., 10 milliseconds). This image is transmitted in real time to an edge computing device (such as a mobile phone processor), which has a hyperspectral reconstruction network (S2DA-Net) pre-programmed with an optimized method. The edge device inputs the compressed grayscale image into the network, which performs forward inference (taking approximately 30 milliseconds) and outputs a hyperspectral data cube with a size of 1024×1024×31 (31 spectral channels, covering 400-700nm). The user can then extract the continuous spectral curve of any pixel from this cube to determine whether the leaf is infected with disease.

[0131] As can be seen from the above description, the hyperspectral image reconstruction method provided in this application provides a physical basis for high light throughput and high signal-to-noise ratio in the front-end optical system by configuring the optimized target structure parameters on the diffractive optical element; and with the help of a lightweight hyperspectral reconstruction network, it can recover a high-quality hyperspectral data cube from a single compressed grayscale image in real time on an edge device.

[0132] To further illustrate the above embodiments, this application also provides an optimization method for the diffraction and refraction hybrid snapshot hyperspectral imaging system provided in the embodiments of this application to obtain the target optimized structural parameters of the diffraction optical element and the optimized hyperspectral reconstruction network, and to build a prototype system for diffraction-refraction hybrid snapshot hyperspectral imaging, the hardware parameters of which are shown in Table 1.

[0133] Table 1 Hardware parameters of the experimental system for diffractive optical elements The system operates in the visible light band (400nm–700nm). The optimized target structural parameters are configured on diffractive optical elements, and the optimized hyperspectral reconstruction network is deployed on computing units (GPUs or edge computing devices).

[0134] Based on this, this application example also provides training data and optimization process: Training data: The KAUST public hyperspectral dataset was used for training, with a total of 100 epochs. The CAVE dataset was used as the test set to verify the reconstruction performance.

[0135] Optimization process: Performed according to the two-stage optimization method provided in the aforementioned embodiments: The first stage involves parameterizing the diffractive optical element into a rotationally symmetric discrete structure of 64 concentric rings. Using the weighted difference between spectral Fisher information and the second spatial moment as the fitness function, a genetic algorithm is employed for global search to obtain the initial optimized structural parameters.

[0136] The second stage involves using the initial optimized structural parameters as the initial values ​​of a continuously learnable tensor. Based on the differentiable optical propagation model and hyperspectral training data, the structural parameters of the diffractive optical element and the hyperspectral reconstruction network are jointly optimized. A quantization loss function constraint (progressive hardening strategy) is introduced to finally obtain the target optimized structural parameters and the optimized hyperspectral reconstruction network (spatial-spectral dual attention network S2DA-Net).

[0137] Comparison Method: For comparison, the traditional random initialization strategy (without global search by genetic algorithm) is used to directly perform end-to-end joint optimization, and the existing advanced method MST++ (multi-stage spectral Transformer) is used as the baseline for comparing the performance of the reconstruction network.

[0138] The experimental results are as follows: (1) Comparison of convergence speed like Figure 7As shown, when using the two-stage optimization strategy (physical constraint initialization + process coordination) of this application, the convergence speed of the second-stage joint optimization reaches stability in about 14 epochs; while when using the traditional random initialization strategy, it takes about 29 epochs to converge. The method of this application improves the convergence speed by about 50%.

[0139] (2) Comparison of reconstruction accuracy On the CAVE test set, the optimized S2DA-Net achieved a peak signal-to-noise ratio (PSNR) of 35.58 dB for reconstructing hyperspectral images, a 1.28 dB improvement compared to the traditional random initialization strategy (PSNR of 34.3 dB). Furthermore, compared to the MST++ method, which relies on a global Transformer, S2DA-Net achieves significantly lower parameter count (0.55M vs. higher) and computational cost (33.21 GFLOPs vs. higher) while maintaining higher reconstruction quality (PSNR 35.58 dB vs. 34.3 dB), thus achieving an optimal balance between performance and efficiency.

[0140] like Figure 8 As shown, Res-Unet represents the Residual U-shaped Network (a neural network based on residual connections and a U-shaped architecture); MST++ represents the Multi-Stage Spectral Transformer Network (an advanced method for hyperspectral reconstruction); HDNet represents the High-Resolution Network (a network architecture that preserves high-resolution features); Restormer represents the Residual Transformer Network (an image restoration model combining residual connections and Transformers); S2DA-Net represents the Spatial-Spectral Dual Attention Network (the hyperspectral reconstruction network proposed in this application); Scene 1 to Scene 10 represent scenes 1 to 10 (different hyperspectral image scenes in the test dataset); FLOPs represents the number of floating-point operations (a metric for measuring the computational cost of the model, usually measured in GBLOPs, i.e., billions of floating-point operations); Params represents the number of parameters (the number of trainable parameters in the model, usually measured in M, i.e., millions), for example, "30.76 / "0.952" represents the PSNR (Peak Signal-to-Noise Ratio, in dB) on the left and the SSIM (Structural Similarity Index) on the right, in the format "PSNR / SSIM". Average represents the average of PSNR (in dB) and SSIM (dimensionless) across all test scenarios.

[0141] in, Figure 8The pairs of numbers under each scene row (e.g., "30.76 / 0.952") represent the PSNR (dB) and SSIM (dimensionless) of the model reconstruction result in that scene. The numbers under "FLOPs / Params" in the last row correspond to the computational cost (GFLOPs) and parameter count (M) of each method.

[0142] In other words, the experimental results show that this dedicated network achieves the optimal balance between performance and efficiency. After 100 rounds of training on the publicly available spectral dataset KAUST and testing on the CAVE dataset, the results are as follows: Figure 8 As shown, compared to advanced methods that rely on global Transformers (such as MST++), S2DA-Net reduces the number of parameters to approximately 0.55M and the computational cost to only 33.21 GFLOPs, while achieving a higher reconstruction quality (PSNR) of 35.58 dB. This clearly demonstrates that lightweight networks tailored to physical optics characteristics are superior to general-purpose, large-scale networks.

[0143] (3) System imaging performance As shown in Figures 9(a) to 9(f), Figure 9(a) shows the prototype, including the illumination source, the refractive lens (lens) with embedded diffractive optical elements, the image sensor, and the scene target. The left side of Figure 9(a) is a partial schematic diagram of the lens with embedded diffractive optical elements, and the right side is a photograph of the prototype. The red arrows indicate the correspondence between the whole machine and the partial structure. Figure 9(b) shows the comparison between the luminous flux and the original data, where points A, B, C, and D are four selected points (e.g., different areas of the blade, different color blocks, etc.). Figure 9(c) shows the comparison of the spectral curve reconstruction of the four selected points.

[0144] Figure 9(d) shows the comparison of the resolution plate imaging results. In Figure 9(d), the blue line represents the spectral curve reconstructed by the diffraction-refraction hybrid system (DOE-lens system) proposed in this application; the red line represents the spectral curve reconstructed by the existing monolithic DOE system. The reference value (usually a black or green line) is the true ground spectrum (such as a reference spectrum directly measured using a spectrometer). The comparison shows that the blue line matches the reference value significantly better than the red line, indicating that the system in this application has higher spectral fidelity.

[0145] Figure 9(e) shows the normalized current response curve of the image sensor over time (used to characterize the sensor noise level or dark current characteristics). In Figure 9(e), the blue line represents the hybrid system of this application, the red line represents the monolithic DOE system, and the green line represents the background / reference baseline; pixels represent pixels.

[0146] Figure 9(f) shows a comparison of the RGB composite reconstruction results for complex real-world scenes, comparing the reconstruction results of a single-chip DOE system (first row) and the DOE-lens system of this application (second row) in complex real-world scenes. Each row, from left to right, shows: the RGB composite image, and single-band reconstructed images of the 450nm, 500nm, 550nm, 600nm, and 650nm wavelength channels. As seen in the first row, the RGB composite image of the single-chip DOE system is blurry and color-distorted, with significant noise and loss of detail in each wavelength channel. As seen in the second row, the RGB composite image of the system of this application is clear and has natural colors, maintaining rich texture information, sharp edges, and low noise levels in each wavelength channel. The system of this application demonstrates significantly better reconstruction quality than the single-chip DOE system in all channels.

[0147] In other words, actual filming was conducted using a prototype: Luminous flux: Compared with a standalone DOE system, the luminous flux of the hybrid system in this application is increased by approximately 4.7 times.

[0148] Resolution: It can clearly distinguish high-frequency details of the USAF1951 resolution board (as shown in Figure 9(d)).

[0149] Spectral fidelity: In complex real-world scenarios, the reconstructed spectral curves at four selected points closely match the real spectra (as shown in Figure 9(c)).

[0150] Image quality: The reconstructed RGB composite image exhibits excellent color reproduction and low noise characteristics (as shown in Figure 9(f)).

[0151] In other words, this application was validated by building a prototype, with the system operating in the 400-700nm (visible light band) wavelength range. Compared to a standalone DOE system, the hybrid system of this application increases the light throughput by approximately 4.7 times. In real-world shooting scenarios, this system can clearly distinguish high-frequency details of the USAF1951 resolution panel and exhibits excellent color reproduction and low noise characteristics in complex scene reconstruction.

[0152] Therefore, this application provides a physically-aware diffraction-refractive hybrid snapshot hyperspectral imaging system and its optimization method. Specifically, it involves a hardware design of a snapshot hyperspectral imaging system based on a diffraction-refractive hybrid architecture, a physically-aware two-stage optical element optimization strategy, and a supporting lightweight neural network reconstruction algorithm. This application uses spectral Fisher information and the second-order spatial moment as the fitness function of a genetic algorithm for a globally search guided by physical constraints. This aims to solve problems existing in current monolithic diffractive optical element hyperspectral imaging systems, such as the conflict between spectral dispersion and spatial focusing, low light flux, and the tendency of design methods to get trapped in local minima. The refractive lens is responsible for efficient light collection and clear spatial focusing, while the DOE located at the aperture stop focuses solely on precise spectral dispersion. This design focuses the undiffracted zero-order light into a compact point, rather than spreading it across the entire image, thereby significantly improving the light flux (approximately 4.7 times) and signal-to-noise ratio.

[0153] From a software perspective, this application also provides an optimization apparatus for a mixed diffraction and refraction snapshot hyperspectral imaging system, comprising all or part of an optimization method for such a system. The imaging system includes a refractive lens, a diffractive optical element disposed at the aperture stop of the refractive lens, and an image sensor. See [link to relevant documentation]. Figure 10 The optimization device for the diffraction and refraction hybrid snapshot hyperspectral imaging system specifically includes the following components: The first-stage optimization module 10 is used to perform a global search optimization of the structural parameters of the diffractive optical element using a genetic algorithm, with a function consisting of the weighted difference between spectral Fisher information and the second spatial moment as the fitness function, to obtain the initial optimized structural parameters of the diffractive optical element; wherein, the spectral Fisher information is used to quantify the sensitivity of the point spread function to wavelength, and the second spatial moment is used to quantify the energy dispersion of the broadband point spread function.

[0154] The second-stage optimization module 20 is used to jointly optimize the structural parameters of the diffractive optical element and a hyperspectral reconstruction network based on the initial optimized structural parameters, the differentiable optical propagation model corresponding to the imaging system, and the hyperspectral training data, to obtain the target optimized structural parameters for optimizing the diffractive optical element in the imaging system, and the optimized hyperspectral reconstruction network.

[0155] The embodiments of the optimization device for the diffraction and refraction hybrid snapshot hyperspectral imaging system provided in this application can be used to execute the processing flow of the optimization method for the diffraction and refraction hybrid snapshot hyperspectral imaging system in the above embodiments. Its functions will not be repeated here, but can be referred to the detailed description of the above-described optimization method for the diffraction and refraction hybrid snapshot hyperspectral imaging system.

[0156] The optimization device for the diffraction and refraction hybrid snapshot hyperspectral imaging system can perform the optimization of the system in either a server or a client device. The specific choice depends on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations in this regard. If all operations are performed in the client device, the client device may further include a processor for the specific processing of the optimization of the diffraction and refraction hybrid snapshot hyperspectral imaging system.

[0157] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0158] The server and the client device can communicate using any suitable network protocol, including those not yet developed as of the date of this application. Such network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Furthermore, such network protocols may also include RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer Protocol) protocols used on top of the aforementioned protocols.

[0159] From a software perspective, this application also provides an apparatus for performing all or part of a hyperspectral image reconstruction method, see [link to relevant documentation]. Figure 11 The hyperspectral image reconstruction device specifically includes the following components: Imaging system optimization module 30 is used to configure the target optimized structural parameters obtained by the optimization method of the diffraction and refraction hybrid snapshot hyperspectral imaging system provided by the first aspect to the diffraction optical element of the diffraction and refraction hybrid snapshot hyperspectral imaging system. Image acquisition module 40 is used to acquire a single compressed grayscale image using the image sensor in the configured imaging system; The image reconstruction module 50 is used to input the compressed grayscale image into the optimized hyperspectral reconstruction network obtained by the optimization method of the diffraction and refraction hybrid snapshot hyperspectral imaging system, so that the optimized hyperspectral reconstruction network outputs the reconstructed hyperspectral data cube.

[0160] The embodiments of the hyperspectral image reconstruction apparatus provided in this application can be used to execute the processing flow of the hyperspectral image reconstruction method embodiments described above. Its functions will not be repeated here, but can be referred to the detailed description of the hyperspectral image reconstruction method embodiments described above.

[0161] The hyperspectral image reconstruction portion of the aforementioned hyperspectral image reconstruction apparatus can be performed on either a server or a client device. The choice can be made based on the processing capabilities of the client device and the limitations of the user's usage scenario. This application does not impose any limitations in this regard. If all operations are performed on the client device, the client device may further include a processor for the specific processing of the hyperspectral image reconstruction.

[0162] This application also provides an electronic device, which may include a processor, a memory, a receiver, and a transmitter. The processor is used to execute the optimization method and / or hyperspectral image reconstruction method of the diffraction and refraction hybrid snapshot hyperspectral imaging system mentioned in the above embodiments. The processor and memory can be connected via a bus or other means, taking a bus connection as an example. The receiver can be connected to the processor and memory via wired or wireless means.

[0163] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0164] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the optimization method and / or hyperspectral image reconstruction method of the diffraction and refraction mixed snapshot hyperspectral imaging system in the embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the optimization method and / or hyperspectral image reconstruction method of the diffraction and refraction mixed snapshot hyperspectral imaging system in the above method embodiments.

[0165] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0166] The one or more modules are stored in the memory, and when executed by the processor, they execute the optimization method and / or hyperspectral image reconstruction method of the diffraction and refraction hybrid snapshot hyperspectral imaging system in the embodiment.

[0167] In some embodiments of this application, the user equipment may include a processor, a memory, and a transceiver unit. The transceiver unit may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected via a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0168] As one implementation method, the functions of the receiver and transmitter in this application can be implemented by transceiver circuits or dedicated transceiver chips, and the processor can be implemented by dedicated processing chips, processing circuits or general-purpose chips.

[0169] As another implementation approach, the server provided in this application embodiment can be implemented using a general-purpose computer. That is, the program code implementing the processor, receiver, and transmitter functions is stored in memory, and the general-purpose processor implements the processor, receiver, and transmitter functions by executing the code in memory.

[0170] This application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the steps of the aforementioned optimization method and / or hyperspectral image reconstruction method for a diffraction and refraction hybrid snapshot hyperspectral imaging system. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0171] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned optimization method and / or hyperspectral image reconstruction method for a mixed diffraction and refraction snapshot hyperspectral imaging system.

[0172] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave.

[0173] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0174] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0175] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to the embodiments of this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An optimization method for a hybrid diffraction and refraction snapshot hyperspectral imaging system, characterized in that, The imaging system includes a refractive lens, a diffractive optical element disposed at the aperture stop of the refractive lens, and an image sensor; the method includes: A genetic algorithm is used to perform a global search optimization on the structural parameters of the diffractive optical element, using a function composed of the weighted difference between spectral Fisher information and the second spatial moment as the fitness function, to obtain the initial optimized structural parameters of the diffractive optical element; wherein, the spectral Fisher information is used to quantify the sensitivity of the point spread function to wavelength, and the second spatial moment is used to quantify the energy dispersion of the point spread function over a wide wavelength range. Based on the initial optimized structural parameters, the differentiable optical propagation model corresponding to the imaging system, and the hyperspectral training data, the structural parameters of the diffractive optical element and a hyperspectral reconstruction network are jointly optimized to obtain the target optimized structural parameters for optimizing the diffractive optical element in the imaging system, as well as the optimized hyperspectral reconstruction network.

2. The optimization method for the hybrid diffraction and refraction snapshot hyperspectral imaging system according to claim 1, characterized in that, Before performing a global search optimization on the structural parameters of the diffractive optical element, the method further includes: The height map of the diffractive optical element is discretized into M×M pixel units to obtain a discretized height map; wherein each pixel unit corresponds to a height value; M is a positive integer greater than 1; The discretized height map is parameterized into a rotationally symmetric structure composed of multiple concentric rings and used as the structural parameter of the diffractive optical element; wherein each concentric ring corresponds to a discrete height value; Furthermore, a total generalized pupil function is constructed; wherein the total generalized pupil function is the product of the aperture function, the phase of the refractive lens, and the modulation of the diffractive optical element; Based on the angular spectrum method, the square of the Fourier transform modulus of the total generalized pupil function is used as the point spread function, thereby establishing the differentiable optical propagation model corresponding to the imaging system.

3. The optimization method for the hybrid diffraction and refraction snapshot hyperspectral imaging system according to claim 1, characterized in that, The genetic algorithm is used, with a fitness function consisting of a weighted difference between spectral Fisher information and the second spatial moment, to globally search and optimize the structural parameters of the diffractive optical element. The initial optimized structural parameters of the diffractive optical element include: Based on the differentiable optical propagation model, the point spread function of the imaging system in each wavelength channel is calculated; wherein, the wavelength channel refers to the discrete spectral interval divided within the operating wavelength band of the imaging system; Based on the point spread function of each wavelength channel, the spectral Fisher information is calculated; Furthermore, based on the point spread function intensity distribution calculated using the differentiable optical propagation model, the second-order spatial moment is calculated: Substituting the spectral Fisher information and the spatial second moment into the fitness function, the fitness function value is calculated; The fitness function includes: in, This represents the fitness function value; This represents the spectral Fisher information. Represents the balance coefficient. Denotes the second moment in space; With the goal of minimizing the fitness function value, the structural parameters of the diffractive optical element are iteratively optimized using a genetic algorithm, and the optimized structural parameters are output as the initial optimized structural parameters of the diffractive optical element.

4. The optimization method for the hybrid diffraction and refraction snapshot hyperspectral imaging system according to claim 1, characterized in that, Based on the initial optimized structural parameters, the differentiable optical propagation model corresponding to the imaging system, and hyperspectral training data, the structural parameters of the diffractive optical element and a hyperspectral reconstruction network are jointly optimized to obtain the target optimized structural parameters for optimizing the diffractive optical element in the imaging system, and the optimized hyperspectral reconstruction network, including: The initial optimized structural parameters are used as the initial values ​​for the structural parameters of the diffractive optical element; The differentiable optical propagation model is cascaded with a hyperspectral reconstruction network to serve as an end-to-end differentiable optimization model. Using hyperspectral training data, and through a backpropagation algorithm with the objective of minimizing a preset total loss function, the structural parameters of the diffractive optical element and the network weights of the hyperspectral reconstruction network in the end-to-end differentiable optimization model are jointly optimized for at least one iteration. The total loss function is a weighted sum of a quantization loss function and a preset reconstruction loss function. The quantization loss function characterizes the difference between the structural parameters of the diffractive optical element and preset discrete step values. If the current iteration meets the preset convergence condition, the optimized structural parameters are output as the target optimized structural parameters, and the optimized hyperspectral reconstruction network is also output.

5. The optimization method for the hybrid diffraction and refraction snapshot hyperspectral imaging system according to claim 4, characterized in that, The weight coefficient of the quantization loss function in the total loss function gradually increases with each iteration, so that the structural parameters of the diffractive optical element gradually converge to the discrete step value.

6. The optimization method for the hybrid diffraction and refraction snapshot hyperspectral imaging system according to claim 1, characterized in that, The hyperspectral reconstruction network is a spatial-spectral dual-attention network; The spatial-spectral dual attention network adopts a U-shaped architecture, including an encoder, a decoder, and skip connections; The encoder is used to receive input compressed image data and extract feature maps through downsampling operations, wherein the feature map output by the last layer of the encoder is used as the encoded feature output. The decoder is used to receive the encoded features, restore the spatial resolution through upsampling operations, and output the reconstructed hyperspectral data cube; The skip connection is used to pass the feature maps output by the layers other than the last layer in the encoder to the corresponding scale layer in the decoder; Each layer in the encoder and the decoder contains at least one spatial-spectral attention module; The spatial-spectral attention module includes a spatial path and a spectral path: The spatial path is used to receive the input feature map, extract spatial features through a 3×3 convolutional layer, and output spatially enhanced features; The spectral path is used to receive the same input feature map of the spatial path, extract spectral features through a window-based multi-head self-attention mechanism, and output spectral enhancement features; wherein, the window size of the window-based multi-head self-attention mechanism is set according to the diffusion range of the point spread function of the imaging system. The spatial-spectral attention module is also used to fuse the spatial enhancement features with the spectral enhancement features and output the fused feature map.

7. A hyperspectral image reconstruction method, characterized in that, include: The target optimized structural parameters obtained by the optimization method of the hybrid snapshot hyperspectral imaging system of diffraction and refraction as described in any one of claims 1 to 6 are configured in the diffractive optical element of the hybrid snapshot hyperspectral imaging system of diffraction and refraction. A single compressed grayscale image is acquired using the image sensor in the configured imaging system; The compressed grayscale image is input into the optimized hyperspectral reconstruction network obtained by the optimization method of the diffraction and refraction hybrid snapshot hyperspectral imaging system, so that the optimized hyperspectral reconstruction network outputs the reconstructed hyperspectral data cube.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the optimization method of the diffraction and refraction hybrid snapshot hyperspectral imaging system as described in any one of claims 1 to 6, and / or implements the hyperspectral image reconstruction method as described in claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the optimization method for the diffraction and refraction hybrid snapshot hyperspectral imaging system as described in any one of claims 1 to 6, and / or implements the hyperspectral image reconstruction method as described in claim 7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the optimization method for the diffraction and refraction hybrid snapshot hyperspectral imaging system as described in any one of claims 1 to 6, and / or implements the hyperspectral image reconstruction method as described in claim 7.

Images