Hyperspectral imaging method and system based on FPGA

Abstract

The invention discloses a hyperspectral imaging method and system based on an FPGA, and belongs to the field of hyperspectral imaging, and the system comprises a data collection and preprocessing module, a compressed sensing module, a sparse reconstruction module and an output module. Various components such as a broadband light source, an imaging optical system, an Echel grating diffraction lens for wavelength separation, a CMOS detector, an ADC (analog-to-digital converter) and a calculation and control unit are integrated in the hardware design, and the components work cooperatively and are used for efficient hyperspectral data collection and processing. According to the system, diffraction imaging is enhanced by using a chirped fiber bragg grating (CFBG), and hyperspectral image reconstruction is carried out by adopting a generative adversarial network (GANs) method. According to the hyperspectral imaging method and system based on the FPGA, compressed sensing, sparse reconstruction and the enhancement technology based on the GAN are effectively combined, and original hyperspectral data are reconstructed with high precision while low hardware occupation is kept and the real-time processing capacity is ensured.

Description

Technical Field

[0001] The present invention relates to the field of hyperspectral imaging, and in particular to a hyperspectral imaging method and system based on FPGA. Background Art

[0002] Hyperspectral imaging systems are used to capture spectral information in different wavelength ranges, thus providing detailed insights into the composition and characteristics of the observed scene. Traditional systems usually require bulky hardware and complex setups, which limits their practical applications. In addition, high-dimensional hyperspectral data requires a lot of computing resources to process, making real-time imaging difficult.

[0003] Therefore, it is urgent to study a method to address these challenges by integrating FPGA-based data processing, compressed sensing, and generative adversarial network (GAN) enhancement technology to achieve efficient and high-precision hyperspectral imaging. Summary of the invention

[0004] The purpose of the present invention is to provide an FPGA-based hyperspectral imaging method and system, which effectively combines compressed sensing, sparse reconstruction and GAN-based enhancement technology to reconstruct the original hyperspectral data with high precision while maintaining low hardware occupancy and ensuring real-time processing capabilities.

[0005] To achieve the above objectives, the present invention provides a hyperspectral imaging method and system based on FPGA, comprising the following modules:

[0006] Data acquisition and preprocessing module: collects hyperspectral data from the target scene using a broadband light source and an optical sensor; preprocesses the collected data to improve signal quality and prepare for subsequent analysis, including noise reduction, optical filtering, and analog signal conditioning;

[0007] Compressed sensing module: including a random measurement matrix generator, which generates a random basis for compressing input hyperspectral data by dimensionality reduction;

[0008] Sparse reconstruction module: Iteratively reconstructs the original hyperspectral signal from the compressed data by selecting the dictionary element that best matches the measured value, implementing a matching pursuit algorithm on the FPGA; wherein the reconstruction process uses the parallel processing capability of the FPGA for real-time data processing, and the reconstruction process is terminated when a predefined error threshold is reached; the sparse reconstruction module includes a dictionary or transform basis of wavelet or discrete cosine transform (DCT) functions, sparsely represents the hyperspectral data, and applies a greedy algorithm based on matching pursuit and orthogonal matching pursuit groups;

[0009] Output Module: Visualize the reconstructed and enhanced hyperspectral images in real time and transmit the enhanced hyperspectral images for further analysis and remote storage.

[0010] Preferably, it includes:

[0011] Spectral dispersion unit: including Echelle grating diffraction lens and chirped fiber Bragg grating CFBG;

[0012] Digital micromirror device (DMD) array: includes multiple DMDs working in parallel, each of which applies a binary mask to the incident light and modulates the reflected light or deflected light according to the mirror direction, thereby selectively processing different spectral bands;

[0013] Detector unit: including a CMOS detector and an analog-to-digital converter ADC;

[0014] Enhancement unit based on generative adversarial network GAN: including generator network, discriminator network and training loss function;

[0015] Processing unit: Handles multiple data processing stages including compressed sensing, sparse reconstruction, and data augmentation, thereby optimizing throughput and reducing power consumption.

[0016] Preferably, the data acquisition and preprocessing module includes:

[0017] Broadband light source: selected from the group of halogen lamps, light-emitting diodes LEDs and laser diodes, which will evenly illuminate the target scene over a wide spectrum;

[0018] Pre-processing filters: Includes optical bandpass filters and polarizers that selectively allow desired spectral components while reducing interference from unwanted light sources.

[0019] Preferably, the compressed sensing module is implemented based on an FPGA, projecting the collected data onto a random basis, thereby reducing data acquisition time and storage requirements; the compressed sensing module reduces the number of measurements required by at least 10 times while maintaining at least 90% reconstruction accuracy, thereby minimizing data acquisition time.

[0020] Preferably, in the enhancement unit based on the generative adversarial network GAN, the generator network is implemented through a codec architecture to generate high-fidelity hyperspectral images from sparse reconstruction data, and the generator captures spatial and spectral features; the spectral attention layer in the generator network ensures that spectral correlation is maintained during the reconstruction process; the jump connection between the encoder and the decoder in the generator network is used to maintain spatial details during the reconstruction process;

[0021] The discriminator network is a patch-based classifier that evaluates whether the generated hyperspectral image is real or synthetic by assigning probability values ​​to local patches of the image;

[0022] The training loss function includes adversarial loss, L1 loss and spectral consistency loss, which ensures that the generated images are realistic and maintain spectral integrity.

[0023] Preferably, in the spectral dispersion unit, the Echelle grating diffraction lens is used to separate the incident broadband light into multiple spectral components; the chirped fiber Bragg grating CFBG introduces position-dependent light modulation to improve the spectral resolution by dispersing the incident light based on wavelength.

[0024] Preferably, the digital micromirror device (DMD) array operates at a switching speed of microseconds to achieve high-speed modulation and parallel processing of hyperspectral data.

[0025] Preferably, in the detector unit, the CMOS detector converts the input spectral components into electronic signals, wherein the CMOS detector provides high sensitivity to detect weak spectral signals; and the analog-to-digital converter ADC converts the analog electronic signals from the CMOS detector into digital data suitable for FPGA-based processing.

[0026] The present invention also provides an operation method of a hyperspectral imaging system based on FPGA, comprising the following steps:

[0027] S1, use broadband light source to illuminate the target scene;

[0028] S2, modulating the incident light using a plurality of digital micromirror devices (DMDs), each of which applies a binary mask to the incident light to selectively process different spectral components;

[0029] S3, using Echelle grating diffraction lens and chirped fiber Bragg grating CFBG to disperse the modulated light and separate the spectral components with high precision;

[0030] S4, using a CMOS detector to acquire the scattered light, converting the optical signal into an electronic signal, and using an analog-to-digital converter ADC to convert the electronic signal into a digital form;

[0031] S5, compressing digital data using random measurement matrices, where the compressed sensing process reduces the dimensionality of the data by projecting the data onto a random basis;

[0032] S6. reconstructing the compressed data using a matching pursuit algorithm implemented on the FPGA, wherein the algorithm iteratively approximates the original hyperspectral signal by selecting elements from a predefined dictionary;

[0033] S7. Enhance the reconstructed hyperspectral image using a generative adversarial network (GAN), wherein the generative adversarial network (GAN) includes a generator and a discriminator, and is trained to improve image quality and maintain spectral integrity;

[0034] S7.1. Apply spectral normalization on the generator network in the Generative Adversarial Network (GAN) to stabilize training and ensure consistency between spectral channels.

[0035] S7.2. Using a patch-based discriminator in a generative adversarial network (GAN) to evaluate the local authenticity of generated hyperspectral image patches, thereby ensuring that high-frequency details are accurately reconstructed.

[0036] S8. Output the enhanced hyperspectral image for visualization or further analysis.

[0037] Preferably, the compressed sensing process uses a random projection matrix to reduce the number of required measurements by 80% to reduce data acquisition time and storage requirements; the generative adversarial network GAN is trained using a combination of adversarial loss, L1 loss and spectral consistency loss to ensure that the generated images are both realistic and spectrally accurate; the digital micromirror device DMD operates at a switching speed of less than 5 microseconds to achieve fast modulation and high-speed data acquisition.

[0038] Therefore, the present invention adopts the above-mentioned FPGA-based hyperspectral imaging method and system, which effectively combines compressed sensing, sparse reconstruction and GAN-based enhancement technology to reconstruct the original hyperspectral data with high precision while maintaining low hardware occupancy and ensuring real-time processing capabilities.

[0039] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a system block diagram of an embodiment of a hyperspectral imaging method and system based on FPGA of the present invention;

[0041] Figure 2 It is a spectral dispersion process diagram of an embodiment of a hyperspectral imaging method and system based on FPGA of the present invention;

[0042] Figure 3 The present invention is a flowchart of a data acquisition and processing workflow of an FPGA-based hyperspectral imaging method and system embodiment. DETAILED DESCRIPTION

[0043] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0044] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0045] Embodiment 1

[0046] like Figure 1 As shown, a hyperspectral imaging method and system based on FPGA consists of four main modules:

[0047] Data acquisition and preprocessing module: collects data from the scene through optical sensors and preprocesses it for efficient analysis.

[0048] Compressed Sensing Module: Compresses input data using a random measurement matrix to reduce the dimensionality of the captured hyperspectral image.

[0049] Sparse reconstruction module: reconstructs the compressed data using efficient sparse reconstruction algorithms (such as matching pursuit) to restore the original hyperspectral signal.

[0050] Output module: Output the reconstructed hyperspectral image for further analysis or display.

[0051] Among them, the following key components and subsystems are included:

[0052] Digital Micromirror Arrays (DMDs): This system utilizes multiple DMDs working in parallel to achieve high-speed, flexible data acquisition. Each DMD modulates the incident light by reflecting or deflecting the beam according to the mirror orientation, enabling selective processing of different spectral bands. The microsecond switching speed of DMDs significantly increases the overall throughput of the system.

[0053] Chirped Fiber Bragg Grating (CFBG): Hyperspectral imaging systems employ chirped fiber Bragg gratings (CFBG) to improve spectral resolution. CFBGs are designed to introduce position-dependent light modulation, dispersing wavelengths in a way that allows precise wavelength separation. This provides a mechanism for efficiently separating incident spectral components, achieving finer spectral resolution prior to subsequent component processing.

[0054] Echelle Grating Diffraction Lens: This system uses an Echelle Grating Diffraction Lens for high-resolution spectral separation. The grating is designed to diffract incident light at steep angles, effectively separating different wavelengths with high precision. This element is essential for processing broadband ranges, and the steep diffraction angle helps separate the incident broadband light into different spectral components for subsequent processing.

[0055] The basic principle of spectral dispersion using Echelle grating diffraction lens and chirped fiber Bragg grating (CFBG) is as follows:

[0056] In hyperspectral imaging, the system must decompose incident broadband light containing multiple wavelength components into separate spectral channels. This makes it possible to obtain multi-wavelength information for each pixel. Traditional dispersive elements, such as prisms or standard gratings, usually have difficulty achieving high resolution while maintaining a compact form factor. The combined use of Echelle gratings and CFBGs provides a viable solution for improving spectral resolution and achieving a more compact system.

[0057] An Echelle grating is a high-order diffraction grating that has a larger groove spacing and operates at higher diffraction orders (high m values). Its groove geometry allows for clean spectral separation and high resolution even at high diffraction orders. Compared to standard gratings, Echelle gratings have fewer grooves per unit length, but the groove angles are steeper, allowing them to produce clean and well-separated spectral lines.

[0058] In this system, the Echelle grating is combined with the lens element to form a diffractive lens assembly, which not only disperses the spectrum but also images the separated spectral components onto the detector. Light of different wavelengths is spatially separated on the focal plane, while allowing the downstream detector to read the information of each wavelength band pixel by pixel.

[0059] A fiber Bragg grating (FBG) is a refractive index modulated structure written into an optical fiber. The FBG reflects a specific wavelength that satisfies the Bragg condition:

[0060] λ B =2n eff Λ (1);

[0061] Among them, λ B is the Bragg wavelength, n eff is the effective refractive index of the fiber, and Λ is the grating period.

[0062] A "chirped" fiber Bragg grating (CFBG) means that the grating period is not constant, but varies along the fiber axis. This variation creates a position-dependent modulation, causing different wavelengths to be reflected or delayed at different points in the fiber. As a result, the CFBG can introduce additional spectral dispersion or group delay. In this system, the CFBG works in conjunction with an Echelle grating to further improve spectral resolution. The Echelle grating provides spatial dispersion of wavelengths, while the CFBG imparts a wavelength-dependent delay or modulation, enhancing the overall spectral discrimination capability.

[0063] In summary, the Echelle grating processes the spatial separation of the spectrum, and the CFBG further increases the spectral domain modulation. The combination of the two significantly improves the spectral resolution and performance of the hyperspectral imaging system.

[0064] Using the spectral dispersion of an Echelle grating, the basic diffraction equation of the grating is:

[0065] mλ=d(sinθ i +sinθ m ) (2);

[0066] where m is the diffraction order (usually higher for Echelle gratings), λ is the wavelength, d is the grating constant (groove spacing), and θ i is the angle of incidence, θ m is the diffraction angle of the mth order.

[0067] Echelle gratings are typically operated at high diffraction orders to achieve greater dispersion and finer spectral resolution. When combined with lens elements, the diffracted wavelengths are focused onto the detector plane, with each wavelength mapped to a different spatial location.

[0068] The intensity distribution on the detector plane can be expressed as:

[0069] S(x')=∫∫I(x,λ)h(x,λ)dxdλ (3);

[0070] Where S(x') is the light intensity distribution at x' on the detector plane. I(x,λ) is the spectral intensity distribution of the system input, including the diffraction effect of the Echelle grating. h(x,λ) is the optical transfer function of the system, which describes the modulation and transfer characteristics of the optical system to the input spectrum. In summary, I(x,λ) and h(x,λ) describe the diffraction effect of the Echelle grating and the subsequent modulation and transfer effect of the imaging optical system, respectively. Through appropriate design, a nearly one-to-one mapping relationship between wavelength λ and detector position x' is established.

[0071] For a CFBG, the grating period Λ(z) varies along the fiber axis z:

[0072] Λ(z)=Λ0+ΔΛ(z) (4);

[0073] Where Λ0 is the initial period and ΔΛ(z) is the spatial variation (chirp). The Bragg wavelength at position z is:

[0074] λ B (z) = 2n eff Λ(z)λ B (5);

[0075] Since the grating period varies with z, different wavelengths encounter their Bragg reflection points at different locations, resulting in wavelength-dependent delay or dispersion. The effect of CFBG can be described by the coupled mode equation or transfer matrix. By solving these equations, the reflectivity and group delay for each wavelength can be obtained. The group delay τ(λ) can be approximately expressed as:

[0076] τ(λ)≈dτdλ(λ-λ o ) (6);

[0077] Where dτdλ is related to the chirp rate of CFBG, λ represents the wavelength, and λ o represents the initial wavelength.

[0078] Combined with the spatial dispersion of the Echelle grating, the CFBG imposes an additional wavelength-dependent modulation. Therefore, the measured signal y after passing through the DMD mask and the dispersion element is k for:

[0079] y k =∫∫S(x')M k (x')dx' (7);

[0080] Among them, M k (x') represents the binary modulation pattern of the DMD, and the function S(x') already includes the spatial dispersion from the Echelle grating and the spectral modulation from the CFBG.

[0081] The following is the preparation method of Echelle grating diffraction lens:

[0082] First, grating preparation is performed. Echelle gratings are manufactured by precision ruling or lithography. Traditional methods involve mechanical ruling on metallized substrates (such as aluminum-coated glass), while modern micro-nano manufacturing techniques use electron beam lithography and dry etching to achieve nanometer-level groove accuracy.

[0083] Then, high-order optimization is performed. Echelle gratings require precise control of groove depth and flame angle to optimize efficiency at high diffraction orders.

[0084] Finally, the Echelle grating must be integrated with a lens element to form a diffractive lens assembly. This may involve aligning and mounting the grating in the optical system so that the diffracted beams can be properly focused and separated at the detector plane.

[0085] The following is a method for preparing chirped fiber optic grids (CFBGs):

[0086] The first step is to prepare the light by selecting high-quality optical fiber with a stable refractive index. The fiber coating and environment must be controlled to ensure uniformity.

[0087] Then proceed to UV writing of FBG. FBG is usually written by ultraviolet (UV) light together with a phase mask. The interference pattern of the mask produces a periodic refractive index modulation in the fiber core.

[0088] Next, we implement chirping. To create a CFBG, the grating period must vary along the length of the fiber. This can be achieved by using a non-uniform phase mask, or by applying a spatially varying strain or temperature gradient during the writing process. As a result, the grating period gradually changes along the fiber, creating a chirped pattern.

[0089] Finally, post-processing and packaging. After writing is completed, the CFBG usually needs to be packaged to stabilize its characteristics. This may include fixing the fiber to a mechanical support or placing it in a protective housing to ensure that changes in temperature or strain do not distort the chirp profile.

[0090] Therefore, in this hyperspectral imaging system, the Echelle grating diffraction lens array provides high-order spatial dispersion, separates broadband light into different wavelength bands, and maps each wavelength to a different position on the detector. Its mathematical model is based on the grating equation and the optical transfer function of the system.

[0091] CFBG introduces additional wavelength-dependent delay and dispersion, enabling the system to achieve higher spectral resolution. Its characteristics are described by coupled mode equations and dispersion parameters. By adjusting the chirp characteristics of CFBG, we can finely control the spectral broadening and resolution improvement.

[0092] The fabrication process involves precision engineering. Echelle gratings are fabricated using advanced photolithography or mechanical scribing techniques to ensure the correct flame angle and groove depth. CFBGs are fabricated by using UV interference patterns and controlled conditions to precisely write chirp patterns in the fiber.

[0093] By combining the spatial dispersion of the Echelle grating with the spectral modulation of the CFBG, the system obtains a higher quality initial spectral signal, which supports more efficient compressed sensing, sparse reconstruction, and enhancement based on GAN technology. This integrated approach provides a more robust and high-resolution FPGA-based real-time hyperspectral imaging system. This integrated approach provides more powerful and high-resolution support for FPGA-based real-time hyperspectral imaging systems.

[0094] The light diffracted by the Echelle grating is projected onto a CMOS detector, which converts the incoming light into an electronic signal. The CMOS detector design has high sensitivity and can effectively capture weak spectral components. The signal output by the detector is then converted to digital form through an ADC (analog to digital converter) to be processed in the FPGA hardware.

[0095] In FPGA-based data processing, the FPGA is responsible for managing multiple data processing stages, including compressed sensing and sparse reconstruction. The compressed sensing algorithm implemented on the FPGA uses a random measurement matrix to obtain a compressed representation of the hyperspectral data. By projecting the data onto a random basis, compressed sensing reduces the number of samples required while retaining key information. This not only minimizes data acquisition time, but also reduces storage requirements.

[0096] After compressed sensing, the FPGA performs sparse reconstruction to restore the original signal. This is achieved through a matching pursuit algorithm, which finds the optimal sparse representation of the signal by iteratively selecting the dictionary element that best matches the measured value. The parallel processing capabilities of the FPGA significantly speed up the reconstruction process, making real-time data processing possible.

[0097] This system uses generative adversarial networks (GANs) to improve the quality of reconstructed hyperspectral images. GANs consist of two neural networks - a generator and a discriminator - which are trained in an adversarial setting.

[0098] The generator takes sparsely reconstructed data as input and aims to generate high-fidelity hyperspectral images. It adopts an encoder-decoder architecture that can effectively capture spatial and spectral features. On the other hand, the discriminator evaluates whether the generated image is real or synthetic and provides feedback to the generator to help it improve its output over time.

[0099] GANs are trained using a combination of adversarial loss, L1 loss, and spectral consistency loss. The adversarial loss ensures that the generated images are increasingly realistic, while the L1 loss minimizes the pixel-level difference between the generated images and the real images. The spectral consistency loss helps maintain the spectral integrity of the generated images, which is critical for hyperspectral applications.

[0100] The workflow of a hyperspectral imaging system based on FPGA of the present invention is as follows:

[0101] Step 1, data acquisition: The system first illuminates the scene with a broadband light source. Then, the DMDs modulate the reflected light, and then the CFBG and Echelle grating are used for wavelength separation.

[0102] The input hyperspectral scene is represented as a three-dimensional data cube I(x,λ), where x=(x,y) represents the spatial coordinates and λ represents the wavelength. The scene can be represented as a function:

[0103]

[0104] where Λ is the spectral domain.

[0105] Spectral dispersion by Echelle grating and CFBG:

[0106] After passing through the Echelle grating and chirped fiber Bragg grating (CFBG), the light field is dispersed so that each wavelength λ is mapped to a different spatial position on the detector plane. We can express the dispersion signal on the detector plane with the following formula:

[0107] S(x′)=∫ Λ I(x,λ)h(x,λ)dλ (9);

[0108] Among them, I(x,λ) represents the spectral intensity distribution at position x and wavelength λ on the input plane of the optical system (usually at the Echelle grating). It contains the diffraction of the incident spectrum by the Echelle grating and the separated light intensity information. The Echelle grating disperses light of different wavelengths to different spatial positions through high-order diffraction (usually higher diffraction orders). Each wavelength λ is diffracted to a specific angle at the grating, thereby achieving wavelength separation in space. In the lens system behind the Echelle grating, the diffracted light of different wavelengths is focused to different positions x′ on the detector plane. Therefore, I(x,λ) actually describes the spatial and wavelength mapping relationship of the grating and lens system to the incident spectrum. It contains the light intensity information of each wavelength λ at each position x, which is derived from the original spectrum of the light source and the diffraction efficiency of the grating. h(x,λ) is the optical transfer function of the system, which describes the modulation and transfer characteristics of the optical system to the input spectrum. By designing and optimizing h(x,λ), the optical performance of the system is improved, such as reducing distortion, improving resolution, and enhancing signal contrast. During the data processing, h(x,λ) is used to correct the light intensity distribution S(x′) on the detector to compensate for the optical distortion and modulation effect introduced by the system, thereby obtaining more accurate spectral information.

[0109] The detector then measures the intensity associated with S(x').

[0110] Digital micromirror arrays (DMDs) use a binary mask M k (x′)∈{0,1}, modulate the incident light before detection. For a given mode index k, the captured measurement is:

[0111] y k =∫S(x')M k (x')dx' (10);

[0112] Step 2: Compressed sensing and signal detection. The separated light components are detected by a CMOS detector and converted into digital signals by an ADC. These digital signals are input to the FPGA, which performs compressed sensing to reduce the amount of data.

[0113] The goal of compressed sensing is to reduce the number of detections while retaining key information. The scene is initially represented as a high-dimensional vector (obtained by vectorization of the hyperspectral cube), through the random projection matrix Measurement, where M<<N. The measurement process is:

[0114] y=Φx+e (11);

[0115] in, is the compressed measurement vector, is a noise term. Each DMD pattern corresponds to a row of the matrix Φ.

[0116] Hyperspectral signals are usually sparse in a suitable transform domain (e.g., wavelet transform or DCT basis). Assume:

[0117] x = Ψα (12);

[0118] Where Ψ is a known dictionary or basis and α is a sparse coefficient vector. The measurement model becomes:

[0119] y=ΦΨα+e (13);

[0120] Step 3: Sparse reconstruction: FPGA uses the matching pursuit algorithm to reconstruct the original hyperspectral data from the compressed data. The sparse reconstruction problem aims to solve:

[0121]

[0122] Among them, ∈ is the noise tolerance and subject to means "limited by". It means that the L0 norm of the sparse coefficient vector α (that is, the number of non-zero elements in α) is minimized under the premise of satisfying the constraint ||y-ΦΨα||≤∈. This means that among all α vectors that may satisfy the measurement error not exceeding the tolerance ∈, those with the least non-zero elements are selected to achieve efficient signal reconstruction and data compression. ||α|| o Count the number of non-zero elements in α. In practical applications, greedy algorithms such as Matching Pursuit or Orthogonal Matching Pursuit are usually used to identify the dictionary element that is most relevant to the current residual.

[0123] The residual is updated and the process is repeated until the stopping criterion is met.

[0124] Then, the reconstructed hyperspectral data cube is:

[0125]

[0126] in, Reconstruction data. Represents the reconstructed sparse coefficient vector.

[0127] Step 4: GAN-based image reconstruction enhancement. The sparsely reconstructed image is further enhanced by the trained Generative Adversarial Network (GAN). The generator can produce high-quality images, and the discriminator ensures their authenticity. The reconstructed data may have artifacts or loss of fine details, so the Generative Adversarial Network (GAN) is used to optimize and improve the reconstruction quality.

[0128] A generator G is a function:

[0129]

[0130] Will rebuild the data (or a rearranged spatial-spectral cube) as input and outputs an enhanced hyperspectral image.

[0131] The discriminator D is a binary classifier:

[0132]

[0133] It outputs a probability value indicating whether the input hyperspectral image patch is real (from a real dataset) or synthetic (from the generator).

[0134] GAN training involves minimizing the following loss function:

[0135]

[0136] in, Represents the real sample x real The expected value of x real Represents a real hyperspectral image data sample. Reconstruction data expected value. Represents the generator G to reconstruct data The output of is the generated enhanced hyperspectral image. D represents the discriminator, which is a binary classifier used to distinguish between real images and generated images.

[0137] (1) Additional loss items

[0138] The loss can be expressed as:

[0139]

[0140] (2) Loss of spectral consistency

[0141] Hyperspectral data must maintain accurate spectral characteristics. We define:

[0142]

[0143] in, Denotes the extraction of spectral features of given spatial pixels, F(·) denotes the function for extracting spectral features of given spatial pixels, for example, wavelet transform or discrete cosine transform (DCT) basis. E[·] denotes the expected operator, i.e., the average value of all samples. ∑ x represents the sum of all spatial pixels x. It represents the square of the norm, that is, the square of the Euclidean distance, which is used to measure the similarity between two spectral feature vectors. is the spectral consistency loss of the generator G, which is used to ensure that the generated hyperspectral image is consistent with the real image in terms of spectral characteristics. The system ensures that the spectral information of the generated image is not distorted, maintaining the accuracy and details of the hyperspectral data.

[0144] The total loss of the generator is:

[0145]

[0146] Among them, λ GAN ,λ L1 ,λ spectral is the weighting factor.

[0147] The GAN network architecture is as follows:

[0148] (1) Overall architecture

[0149] The generator network can be constructed as an encoder-decoder or U-Net style architecture to capture global context and details. For hyperspectral images, the generator should also maintain spectral consistency across channels.

[0150] Encoder: A series of convolutional layers with spectral-spatial kernels (e.g., 3D convolutions can be used if computationally feasible, or 2D convolutions applied sequentially to each band), followed by a downsampling operation (stride > 1 or pooling) to reduce dimensionality.

[0151] Neck: A residual block or attention module that operates on a smaller spatial dimension but with full spectral range.

[0152] Decoder: Symmetric upsampling layers (transposed convolution or convolution after interpolation) to reconstruct the spatial resolution. Skip connections from encoder to decoder help preserve good spatial information.

[0153] Output layer: The final convolutional layer that maps the output back to the hyperspectral domain with the desired spectral bands.

[0154] (2) Key attributes: Use spectral normalization or instance normalization to stabilize the training process.

[0155] A spectral attention layer is introduced to ensure that spectral correlations are preserved.

[0156] Use non-linear activations (e.g., ReLU or Leaky ReLU) between layers.

[0157] (3) Discriminator architecture

[0158] Patch-based discriminator: A patch-based discriminator (Patch GAN) can be used. Instead of outputting a single scalar, it outputs a probability matrix where each element corresponds to a local patch of the hyperspectral image. This helps to enhance the authenticity of high-frequency details.

[0159] Input: Hyperspectral image (real or generated).

[0160] Convolution: Multiple 2D convolutional layers with increasing feature channels and downsampling with a stride of 2 to reduce the spatial size and increase the receptive field.

[0161] Non-linear activation: Leaky ReLU activation function is used after each convolution.

[0162] Last layer: A convolutional layer maps the intermediate representation to a probability map. Each element in this output map represents the probability of being “real” or “fake” for a local region.

[0163] (4) Processing of spectral dimensions

[0164] To handle the spectral dimension, the following approaches can be taken:

[0165] Treat each spectral band as a channel input and use 2D convolution.

[0166] Use 3D convolution for joint spectral-spatial feature extraction (computationally expensive).

[0167] (5) Training process

[0168] Adversarial training loop:

[0169] Discriminant update: Given a batch of real hyperspectral samples x real And the generated sample: By minimizing To update D.

[0170] The generator updates the above definition Update G to ensure the visual authenticity and spectral consistency of the image.

[0171] The following is a summary of the models and architectures:

[0172] (1) Optical and Compressed Sensing Models

[0173] The hyperspectral data is captured by DMD mode to form a random measurement, which is mathematically expressed as y = ΦΨα + e.

[0174] (2) Sparse reconstruction model

[0175] Use matching pursuit algorithm to recover To find out So that y=ΦΨα is satisfied.

[0176] (3) GAN Enhanced Model

[0177] Using generator G and discriminator D. Adversarial loss Combined with L1 loss and spectral consistency loss, the generator G is guided to improve the quality of reconstructed hyperspectral data.

[0178] (4) GAN architecture

[0179] Generator: Adopts the encoder-decoder (U-Net) style, containing convolutional layers, skip connections, and attention modules to maintain spatial and spectral fidelity.

[0180] Discriminator: Patch-based discriminator (PatchGAN) performs local authenticity evaluation through convolutional layers.

[0181] Step 5: Output and visualization. The final hyperspectral image is sent to the output module for visualization or further processing. This system ensures real-time display capabilities and is suitable for applications that require immediate results.

[0182] Based on this, the present invention can be applied to the following fields:

[0183] Medical imaging: Real-time hyperspectral imaging can provide enhanced tissue differentiation, aiding precise medical diagnosis and surgical procedures.

[0184] Remote Sensing: This system can be used in areas such as environmental monitoring, agriculture, and mineral exploration, where spectral information is essential for identifying different materials.

[0185] Industrial inspection: Hyperspectral imaging helps with quality control and defect detection in the manufacturing process by capturing subtle spectral differences to identify material variations.

[0186] By integrating advanced technologies such as compressed sensing, sparse reconstruction, and enhancement based on generative adversarial networks (GANs), the present invention provides a powerful tool for real-time hyperspectral imaging in a wide range of applications. The FPGA-based architecture ensures high efficiency, low power consumption, and can effectively meet the needs of modern hyperspectral imaging tasks.

[0187] Therefore, the present invention adopts the above-mentioned FPGA-based hyperspectral imaging method and system, which has the following technical advantages:

[0188] (1) Parallel data processing: The use of FPGA hardware enables highly parallel data processing, which is essential for real-time processing of large amounts of hyperspectral data.

[0189] (2) Improved spectral resolution: The integration of CFBG and Echelle grating enables precise spectral separation, ensuring high image quality and fine spectral details.

[0190] (3) Real-time reconstruction: By utilizing GANs and matching pursuit algorithms, the system can efficiently reconstruct hyperspectral images, which is suitable for real-time applications.

[0191] (4) High energy efficiency: Compared with traditional CPU- or GPU-based systems, FPGA technology reduces power consumption, making the solution suitable for portable or field applications.

[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A hyperspectral imaging system based on FPGA, characterized in that: Includes the following modules: Data acquisition and preprocessing module: collects hyperspectral data from the target scene using a broadband light source and an optical sensor; preprocesses the collected data to improve signal quality and prepare for subsequent analysis, including noise reduction, optical filtering, and analog signal conditioning; Compressed sensing module: including a random measurement matrix generator, which generates a random basis for compressing input hyperspectral data by dimensionality reduction; Sparse reconstruction module: It iteratively reconstructs the original hyperspectral signal from the compressed data by selecting the dictionary element that best matches the measured value, implementing the matching pursuit algorithm on the FPGA; The reconstruction process uses the parallel processing capability of FPGA to perform real-time data processing, and the reconstruction process is terminated when a predefined error threshold is reached; the sparse reconstruction module includes a dictionary or transform basis of wavelet or discrete cosine transform (DCT) function, which sparsely represents the hyperspectral data, and applies a greedy algorithm based on matching pursuit and orthogonal matching pursuit group; Output Module: Visualize the reconstructed and enhanced hyperspectral images in real time and transmit the enhanced hyperspectral images for further analysis and remote storage.

2. The FPGA-based hyperspectral imaging system according to claim 1, characterized in that: include: Spectral dispersion unit: including Echelle grating diffraction lens and chirped fiber Bragg grating CFBG; Digital micromirror device (DMD) array: includes multiple DMDs working in parallel, each of which applies a binary mask to the incident light and modulates the reflected light or deflected light according to the mirror direction, thereby selectively processing different spectral bands; Detector unit: including a CMOS detector and an analog-to-digital converter ADC; Enhancement unit based on generative adversarial network GAN: including generator network, discriminator network and training loss function; Processing unit: Handles multiple data processing stages including compressed sensing, sparse reconstruction, and data augmentation, thereby optimizing throughput and reducing power consumption.

3. The FPGA-based hyperspectral imaging system according to claim 1, characterized in that: The data acquisition and preprocessing module includes: Broadband light source: selected from the group of halogen lamps, light-emitting diodes LEDs and laser diodes, which will evenly illuminate the target scene over a wide spectrum; Pre-processing filters: Includes optical bandpass filters and polarizers that selectively allow desired spectral components while reducing interference from unwanted light sources.

4. The FPGA-based hyperspectral imaging system according to claim 1, characterized in that: The compressed sensing module is based on an FPGA implementation that projects the collected data onto a random basis, thereby reducing data acquisition time and storage requirements; the compressed sensing module reduces the number of measurements required by at least 10 times while maintaining at least 90% reconstruction accuracy, thereby minimizing data acquisition time.

5. The FPGA-based hyperspectral imaging system according to claim 2, characterized in that: In the enhancement unit based on the generative adversarial network (GAN), the generator network is implemented through a codec architecture to generate high-fidelity hyperspectral images from reconstruction data, and the generator captures spatial and spectral features; the spectral attention layer in the generator network ensures that spectral correlation is maintained during the reconstruction process; the jump connection between the encoder and decoder in the generator network is used to maintain spatial details during the reconstruction process; The discriminator network is a patch-based classifier that evaluates whether the generated hyperspectral image is real or synthetic by assigning probability values ​​to local patches of the image; The training loss function includes adversarial loss, L1 loss and spectral consistency loss, which ensures that the generated images are realistic and maintain spectral integrity.

6. The FPGA-based hyperspectral imaging system according to claim 2, characterized in that: In the spectral dispersion unit, the Echelle grating diffraction lens is used to separate the incident broadband light into multiple spectral components; the chirped fiber Bragg grating CFBG introduces position-dependent light modulation to improve the spectral resolution by dispersing the incident light based on wavelength.

7. The FPGA-based hyperspectral imaging system according to claim 2, characterized in that: The digital micromirror device (DMD) array operates at a switching speed of microseconds to achieve high-speed modulation and parallel processing of hyperspectral data.

8. The FPGA-based hyperspectral imaging system according to claim 2, characterized in that: In the detector unit, the CMOS detector converts the input spectral components into electronic signals, wherein the CMOS detector provides high sensitivity to detect weak spectral signals; the analog-to-digital converter ADC converts the analog electronic signals from the CMOS detector into digital data suitable for FPGA-based processing.

9. The method for operating a hyperspectral imaging system based on FPGA according to any one of claims 1 to 8, characterized in that: The following steps are involved: S1, use broadband light source to illuminate the target scene; S2, modulating the incident light using a plurality of digital micromirror devices (DMDs), each of which applies a binary mask to the incident light to selectively process different spectral components; S3, using Echelle grating diffraction lens and chirped fiber Bragg grating CFBG to disperse the modulated light and separate the spectral components with high precision; S4, using a CMOS detector to acquire the scattered light, converting the optical signal into an electronic signal, and using an analog-to-digital converter ADC to convert the electronic signal into a digital form; S5, compressing digital data using random measurement matrices, where the compressed sensing process reduces the dimensionality of the data by projecting the data onto a random basis; S6. reconstructing the compressed data using a matching pursuit algorithm implemented on the FPGA, wherein the algorithm iteratively approximates the original hyperspectral signal by selecting elements from a predefined dictionary; S7. Enhance the reconstructed hyperspectral image using a generative adversarial network (GAN), wherein the generative adversarial network (GAN) includes a generator and a discriminator, and is trained to improve image quality and maintain spectral integrity; S7.

1. Apply spectral normalization on the generator network in the Generative Adversarial Network (GAN) to stabilize training and ensure consistency between spectral channels. S7.

2. Using a patch-based discriminator in a generative adversarial network (GAN) to evaluate the local authenticity of generated hyperspectral image patches, thereby ensuring that high-frequency details are accurately reconstructed. S8. Output the enhanced hyperspectral image for visualization or further analysis.

10. The method for operating a hyperspectral imaging system based on FPGA according to claim 9, characterized in that: The compressed sensing process uses a random projection matrix to reduce the number of required measurements by 80% to reduce data acquisition time and storage requirements; the generative adversarial network (GAN) is trained using a combination of adversarial loss, L1 loss, and spectral consistency loss to ensure that the generated images are both realistic and spectrally accurate; the digital micromirror device (DMD) operates at a switching speed of less than 5 microseconds to achieve fast modulation and high-speed data acquisition.

Images