An intelligent imaging system based on electrochromic and multispectral collaboration

Abstract

The application provides an intelligent imaging system based on electrochromic and multispectral cooperation, and belongs to the technical field of multispectral imaging, and is used for solving the problems that the filter wheel is large in volume, slow in switching, and difficult to balance miniaturization, real-time performance and high spectral accuracy in the related art. The system comprises a miniaturized switchable filter wheel, an image sensor and a controller. The filter wheel is integrated with an electrochromic narrow-band filter station and a multi-band synchronous acquisition filter area. The controller constructs an implicit spectral radiation field model and updates parameters thereof online through variational inference, outputs a posterior mean and a posterior variance, calculates expected information gain of each candidate acquisition action based on the posterior variance, and executes an optimal action, so as to realize adaptive cooperation of narrow-band high-precision acquisition and wide-band rapid acquisition. The system significantly improves compactness, real-time performance and spectral reconstruction quality of multispectral imaging.

Description

Technical Field

[0001] This application relates to the technical field of multispectral imaging, and more particularly to an intelligent imaging system based on electrochromic and multispectral synergy. Background Technology

[0002] Multispectral imaging technology can acquire spectral information of targets in different bands and is widely used in fields such as monitoring of equipment in new energy power plants and fault diagnosis of microgrid equipment cabinets. Traditional multispectral imaging systems typically use a filter wheel driven by a stepper motor to achieve band switching by rotating multiple narrowband filters, or use liquid crystal tunable filters and other solutions.

[0003] In existing technologies, stepper motor-driven filter wheel structures are bulky and slow in switching speed, making them difficult to adapt to confined spaces such as microgrid equipment cabinets, and unable to meet the requirements of dynamic scene monitoring with high real-time requirements. Some solutions use electrochromic filters to replace mechanical wheels, but these still suffer from low spectral sampling efficiency and the inability to simultaneously achieve both narrowband high precision and broadband real-time performance. Furthermore, traditional multispectral imaging algorithms rely on a serial processing mode of registration followed by reconstruction, which has high hardware requirements, poor real-time performance, and lacks the ability to adaptively optimize acquisition strategies.

[0004] Therefore, existing technologies have significant shortcomings in miniaturization, real-time performance, and intelligent decision-making in multispectral imaging systems. There is an urgent need for an imaging system that can achieve rapid switching, multi-mode collaborative acquisition, and is equipped with efficient intelligent algorithms. Summary of the Invention

[0005] This application provides an intelligent imaging system based on electrochromic and multispectral synergy, which can achieve high-quality real-time reconstruction and adaptive acquisition decision of hyperspectral images in a way that deeply integrates a compact hardware structure with intelligent algorithms.

[0006] Firstly, this application provides an intelligent imaging system based on electrochromic and multispectral synergy. It includes: A miniaturized switchable filter wheel integrates an electrochromic filter station and a multi-band synchronous acquisition filter area. The electrochromic filter station switches its transmission band by applying voltage, and the multi-band synchronous acquisition filter area transmits both visible light and near-infrared light in a single exposure. An image sensor that receives light signals incident through a filter wheel and outputs raw image data; A controller performs the following steps: acquiring multiple frames of raw image data output by an image sensor and the acquisition parameters corresponding to each frame, including the currently used filter type, applied voltage value, and exposure start and end times; constructing an implicit spectral radiation field model, parameterized by a deep neural network, to characterize the radiation intensity at any spatial coordinate and wavelength position in the scene; updating the parameters of the implicit spectral radiation field model online through variational inference based on the multiple frames of raw image data and acquisition parameters, and outputting the posterior mean and posterior variance of the model at each spatial coordinate and wavelength position; calculating the expected information gain of each candidate acquisition action based on the posterior variance of the current implicit spectral radiation field model, where candidate acquisition actions include switching the filter wheel to a designated electrochromic filter station to acquire one frame, or switching to a multi-band synchronous acquisition filter area to acquire one frame; selecting the candidate acquisition action with the largest expected information gain and controlling the filter wheel to execute the action.

[0007] By adopting the above technical solution, the miniaturized switchable filter wheel integrates an electrochromic filter and a multi-band synchronous acquisition filter, realizing the synergy of narrowband high-precision acquisition and broadband fast acquisition. The controller can reconstruct a continuous hyperspectral radiation field from sparse observations by constructing an implicit spectral radiation field model and using variational inference for online updates. At the same time, it can make active decisions based on the expected information gain calculated by the posterior variance, realizing adaptive optimization of the acquisition strategy, significantly reducing hardware complexity and improving the real-time performance and reconstruction quality of spectral imaging.

[0008] Furthermore, the diameter of the filter wheel is less than 20 mm, and the response speed of the electrochromic filter station is less than 0.1 seconds; the multi-band synchronous acquisition filter area adopts a micro-nano structure with alternating checkerboard or strip arrangement, wherein the first region transmits visible light from 400 nm to 700 nm, and the second region transmits near-infrared light from 800 nm to 1000 nm.

[0009] By adopting the above technical solutions, the miniaturized size and fast response capability enable the system to be adapted to small spaces such as microgrid equipment cabinets. The synchronous acquisition filter area of ​​the micro-nano structure realizes parallel acquisition of single frame dual bands, effectively avoiding switching delay.

[0010] Furthermore, the implicit spectral radiation field model is implemented using a multilayer perceptron with a sinusoidal activation function. Its input is normalized spatial coordinates and wavelength, and its output is the corresponding radiation intensity value. The model obtains prior parameters through pre-training on a hyperspectral dataset in the offline stage, and performs posterior updates based on the prior parameters during online runtime variational inference.

[0011] By adopting the above technical solution, the multilayer perceptron with sinusoidal activation function can effectively fit high-frequency spectral information. The combination of offline pre-training and online variational inference reduces the amount of online computation and improves the model convergence speed and reconstruction accuracy.

[0012] Furthermore, variational inference includes treating the parameters of the implicit spectral radiation field model as random variables following a mean-field Gaussian distribution, and updating the mean and variance of this distribution by maximizing the lower bound of evidence. The lower bound of evidence includes the expected log-likelihood term and the KL divergence term, wherein the expected log-likelihood term is calculated based on the error between the original image data and the predicted values ​​of the forward imaging model, which includes the filter spectral response, the electrochromic transmittance dynamic function, and the sensor integration process.

[0013] By adopting the above technical solution, the hardware physical model (filter response, electrochromic dynamics, integral process) is embedded into the Bayesian inference framework, which quantifies the uncertainty of the model parameters. At the same time, the KL divergence constraint ensures that the posterior distribution does not deviate from the prior, thereby enhancing the generalization ability of the model.

[0014] Furthermore, the electrochromic transmittance dynamic function is as follows: the transmittance increases exponentially from zero to steady-state transmittance over time, and the rate of increase is controlled by the time constant; the steady-state transmittance and the time constant are used as scalable parameters of the implicit spectral radiation field model and are estimated together with the parameters of the deep neural network in variational inference.

[0015] By adopting the above technical solution, the dynamic characteristics of electrochromic materials are incorporated into the radiation field model as learnable parameters, which effectively compensates for switching transient errors, avoids the independent compensation steps in traditional methods, and simplifies the system structure.

[0016] Furthermore, the method for calculating the expected information gain includes: for each candidate acquisition action, the expected reduction of the posterior variance of the implicit spectral radiation field model by the observation data obtained after the simulation acquisition, and using this expected reduction as the expected information gain of the action; the posterior variance characterizes the uncertainty of the implicit spectral radiation field model under the current observation data.

[0017] By adopting the above technical solution, Bayesian experimental design is introduced into multispectral acquisition decision-making. The expected reduction in the model's posterior variance is used as the information gain, driving the system to actively acquire bands that minimize uncertainty, thereby achieving optimized allocation of sensing resources.

[0018] Furthermore, the calculation of the expected information gain adopts an approximation strategy: only the posterior variance reduction on the band corresponding to the candidate acquisition action is considered, and the posterior variance reduction is estimated by single-step gradient descent after a virtual observation of the band.

[0019] By adopting the above technical solution, the first-order approximation strategy significantly reduces computational complexity, enabling active decision-making to run in real time on embedded platforms while maintaining high decision effectiveness.

[0020] Furthermore, the controller includes a heterogeneous computing unit consisting of an FPGA and an ARM, wherein the FPGA is used to receive the raw image data output from the image sensor and perform dark level correction and flat field correction, and the ARM unit is used to perform variational inference and calculation of expected information gain; the FPGA and ARM share the corrected image data through an AXI bus.

[0021] By adopting the above technical solutions, the heterogeneous computing architecture separates the underlying image preprocessing from the high-level intelligent inference. The parallel processing capability of the FPGA ensures low latency of the data flow, while the ARM processes complex probabilistic models. The two work together to meet the real-time requirements.

[0022] Furthermore, the filter wheel is also equipped with a stepper motor and a sliding brush ring: the stepper motor drives the filter wheel to rotate, so that the designated electrochromic filter station or multi-band synchronous acquisition filter area is aligned with the photosensitive surface of the image sensor; the sliding brush ring transmits the voltage output by the controller to the electrochromic filter station currently aligned with the sensor, while the other electrochromic filter stations are not powered.

[0023] By adopting the above technical solution, the sliding brush ring avoids cable tangling, supplies power only to the currently operating electrochromic filter station, reduces system power consumption, and improves reliability.

[0024] Furthermore, the controller also outputs an image of the posterior mean of the implicit spectral radiation field model in a specified band, and calculates the normalized difference index based on the image. When the normalized difference index is lower than a preset threshold, an abnormal alarm signal is output. At the same time, the region where the posterior variance exceeds the preset threshold is marked as a low confidence region.

[0025] By adopting the above technical solution, the reconstruction results can be directly applied to downstream detection tasks, and the uncertainty information can be used to mark low confidence areas and prompt manual review, thereby enhancing the practicality and reliability of the system.

[0026] In summary, this application has at least the following beneficial effects: An intelligent imaging system integrating an electrochromic filter and a multi-band synchronous acquisition filter is provided, achieving miniaturization, rapid switching and real-time hyperspectral reconstruction; By using implicit spectral radiation field models and variational inference, the hardware physical characteristics are deeply integrated with Bayesian active learning, which significantly improves the quality of spectral imaging and the efficiency of data acquisition. By utilizing an active decision-making strategy driven by posterior variance, adaptive switching of acquisition modes was achieved, balancing narrowband high precision with broadband high real-time performance, filling the technological gap in multispectral imaging systems for edge computing scenarios.

[0027] It should be understood that the description in the Summary Section is not intended to limit the key or essential features of the embodiments of this application, nor is it intended to restrict the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0028] The above and other features, advantages, and aspects of the embodiments of this application will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A schematic diagram of an intelligent imaging system based on electrochromic and multispectral synergy is shown in an embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0031] This application provides an intelligent imaging system based on electrochromic and multispectral synergy. Through the synergistic design of electrochromic and multi-band synchronous acquisition filters, combined with implicit spectral radiation field variational inference and Bayesian active decision-making, it achieves miniaturized, high real-time, and high-precision spectral imaging and adaptive acquisition.

[0032] This application discloses an intelligent imaging system based on electrochromic and multispectral synergy. This system is particularly suitable for hyperspectral imaging scenarios requiring rapid response in confined spaces, such as monitoring equipment in new energy power plants and diagnosing faults in microgrid equipment cabinets.

[0033] Figure 1 A schematic diagram of an intelligent imaging system based on electrochromic and multispectral synergy is shown in an embodiment of this application.

[0034] The system comprises a miniaturized switchable filter wheel, an image sensor, and a controller. In actual deployment, the imaging system is encapsulated within an aluminum alloy housing measuring approximately 25 mm × 30 mm × 20 mm. A protective window is located at the front of the housing, embedding an AR anti-reflective coating to ensure a transmittance of at least 98% for a wide wavelength range of light. The miniaturized switchable filter wheel, image sensor, and controller are stacked in order from object to image: the outermost layer is the protective window, followed by a miniature fixed-focus lens. Behind the lens is the miniaturized switchable filter wheel, which is then attached to the photosensitive surface of the image sensor. The controller is located on a PCB board on the back of the image sensor and connected to the sensor via a flexible ribbon cable.

[0035] The miniaturized switchable filter wheel comprises an electrochromic filter station and a multi-band synchronous acquisition filter area. The electrochromic filter station switches its transmission band by applying voltage, while the multi-band synchronous acquisition filter area transmits both visible and near-infrared light in a single exposure. In this embodiment, the filter wheel has a diameter of 15 mm and a thickness of 4 mm, with a 4 mm diameter through-hole at its center to reduce weight. Four electrochromic filter stations are evenly spaced on the wheel, each corresponding to an independent electrochromic filter. The multi-band synchronous acquisition filter area is fixed in the center of the wheel, coaxial with the center of the image sensor's photosensitive surface. Due to the coaxial design, the central multi-band synchronous acquisition filter area remains aligned with the photosensitive surface regardless of the filter wheel's rotation angle, eliminating the need for additional alignment. When a specific narrowband spectrum needs to be acquired, a stepper motor drives a rotating disk to move the target electrochromic filter station in front of the sensor's photosensitive area. At this point, the station precisely covers and shields the central multi-band synchronous acquisition filter area, and the optical path passes only through this narrowband filter. When rapid acquisition of dual-band information is required, the disk rotates to the gap between two adjacent electrochromic filter stations, directly exposing the central multi-band synchronous acquisition filter area to the optical path, thus simultaneously acquiring images of both visible and near-infrared broadband channels in a single frame. Each electrochromic filter station employs a sandwich structure, consisting of, from top to bottom, ITO conductive glass, a tungsten trioxide electrochromic layer, a lithium niobate ion conductive layer, a nickel oxide ion storage layer, and another layer of ITO conductive glass. By applying voltage to the upper and lower ITO layers, the transmission characteristics of the electrochromic material change, thereby achieving narrowband filtering. The voltage is output from the controller via a DAC and transmitted to the current station through a sliding brush ring; other stations are not powered to reduce power consumption.

[0036] Furthermore, the diameter of the filter wheel is less than 20 mm, and the response speed of the electrochromic filter station is less than 0.1 seconds. The multi-band synchronous acquisition filter area adopts a micro-nano structure with alternating checkerboard or strip arrangements, wherein the first region transmits visible light from 400 nm to 700 nm, and the second region transmits near-infrared light from 800 nm to 1000 nm. Preferably, in this embodiment, the diameter of the filter wheel is 15 mm, the voltage switching response time of the electrochromic filter is 0.05 seconds, the period of the micro-nano structure is 10 micrometers, and the side length of each cell in the checkerboard is the same as the period. The peak transmittance in the visible light region is greater than 90%, and the peak transmittance in the near-infrared region is greater than 85%. This design enables the system to simultaneously acquire image information from two broadband channels, visible light and near-infrared light, in a single frame without any mechanical movement, and subsequently separate the two independent channel images using a software de-mosaic algorithm.

[0037] Furthermore, the filter wheel is equipped with a stepper motor and a sliding brush ring: the stepper motor drives the filter wheel to rotate, aligning the designated electrochromic filter station or multi-band synchronous acquisition filter area with the photosensitive surface of the image sensor; the sliding brush ring transmits the voltage output from the controller to the electrochromic filter station currently aligned with the sensor, while the other electrochromic filter stations remain unpowered. The stepper motor measures 3.5 mm × 3.5 mm × 2 mm and is coupled to the wheel shaft via a lead screw. The sliding brush ring, located at the wheel shaft, consists of a fixed part and a rotating part. The fixed part connects to the controller's DAC output line, while the rotating part has four sets of contacts, each connected to the ITO electrode leads of one of the four electrochromic filter stations. When the wheel rotates, only the contact corresponding to the station currently aligned with the sensor is connected to the fixed brush, while the other stations remain in a floating state, thus avoiding crosstalk and energy waste. The controller's built-in voltage monitoring loop uses a precision sampling resistor to provide real-time feedback of the actual applied voltage, compares it with the DAC set value, and performs closed-loop correction to ensure that the voltage error is less than ±0.01 volts.

[0038] The image sensor is used to receive light signals incident through the filter wheel and output raw image data. This embodiment uses a 2-megapixel CMOS rolling shutter sensor with a 12-bit ADC. The sensor size is 1 / 3 inch, and its optical format is compatible with the front-end miniature fixed-focus lens. The sensor's master clock frequency is 50 MHz, and it transmits raw RAW data via a two-channel MIPI CSI-2 interface at a rate of 800 megabits per second per channel. The sensor integrates a temperature sensor to monitor the chip's operating temperature. This temperature value is read by the controller via the I2C bus for subsequent temperature compensation in the electrochromic dynamic model.

[0039] The controller performs the following steps: acquiring multiple frames of raw image data output by the image sensor and the acquisition parameters corresponding to each frame, including the currently used filter type, applied voltage value, and exposure start and end times; constructing an implicit spectral radiation field model, parameterized by a deep neural network, to characterize the radiation intensity at any spatial coordinate and wavelength position in the scene; updating the parameters of the implicit spectral radiation field model online through variational inference based on the multiple frames of raw image data and acquisition parameters, and outputting the posterior mean and posterior variance of the model at each spatial coordinate and wavelength position; calculating the expected information gain of each candidate acquisition action based on the posterior variance of the current implicit spectral radiation field model, including switching the filter wheel to a designated electrochromic filter station to acquire one frame, or switching to a multi-band synchronous acquisition filter area to acquire one frame; selecting the candidate acquisition action with the largest expected information gain and controlling the filter wheel to execute the action. Before executing the above steps, the controller will first perform system initialization: load the prior parameters of the offline pre-trained implicit spectral radiation field model, set the initial acquisition mode to multi-band synchronous acquisition of the filter area, and continuously acquire five frames of images to provide initial observation data, thereby quickly starting the variational inference process.

[0040] The algorithm executed by the controller will be explained in detail below, based on specific mathematical principles.

[0041] The controller first acquires multiple frames of raw image data output by the image sensor, and records the raw response value of the k-th frame image at pixel position x,y as... .in and These are the row and column indices of the image, respectively, ranging from 0 to H-1 and 0 to W-1, where H is the image height (1080 pixels in this embodiment) and W is the image width (1920 pixels in this embodiment). The acquisition parameters for each image frame include: the type of filter currently used. ,in The value can be selected from four narrowband stations, ECF1 to ECF4, or the MSF broadband mode; the voltage value applied to the electrochromic filter. (Only when) (Valid only for ECF station); Exposure start and end times and The time is measured in seconds and recorded by a high-precision timer inside the controller. Simultaneously, the controller reads the sensor temperature via the I2C bus. The unit is Celsius.

[0042] The controller constructs an implicit spectral radiation field model, implemented using a multilayer perceptron with a sinusoidal activation function, to characterize the radiation intensity at arbitrary spatial coordinates and wavelength locations within the scene. Specifically, this model is defined as a function... ,in To normalize to The normalization formula for the image space coordinates of the interval is: , ; To normalize to Wavelength values ​​within the range (actual wavelength range 400 nm to 1000 nm, normalized formula) ), output Relative radiation intensity, dimensionless. Network parameters. This includes the weights and biases of all layers. The network employs a four-layer fully connected structure, with 256 neurons per layer, and uses a sine activation function. To enhance the model's ability to fit high-frequency details, the input coordinates are first position-encoded. Where, represents the number of frequency bands encoded. Take respectively and The encoded feature vectors have a dimension of 60 (3 coordinates × 2 × 10), and are concatenated before being input into the network. The model is pre-trained offline using large hyperspectral datasets (e.g., ICVL, Harvard Refractance database), and the prior distribution of network parameters, i.e., the initial mean, is obtained by minimizing the mean squared error loss. and variance When running online, the controller initiates variational inference based on prior parameters.

[0043] The controller updates the parameters of the implicit spectral radiation field model online through variational inference based on multiple frames of raw image data and acquisition parameters. The core idea of ​​variational inference is to update the model parameters... Treating them as random variables, we assume their posterior distribution follows a Gaussian distribution with mean: ,in Let be the variational parameters to be optimized. Update them by maximizing the lower bound of evidence (ELBO). ELBO is defined as:

[0044] in For all historical observation data, The prior distribution is taken from the offline pre-training results. Let KL divergence be the expected log-likelihood. To calculate the expected log-likelihood, a forward imaging model needs to be defined. For the k-th frame observation, the forward imaging model predicts... The calculation comprehensively considers the filter spectral response, the electrochromic transmittance dynamic function, and the sensor integration process:

[0045] in This refers to the exposure time. For filter type The spectral response function, pre-calibrated using a spectrophotometer, is used for each narrowband station of the ECF. It is approximately a Gaussian curve with center wavelengths of 450nm, 550nm, 650nm, and 750nm, and a half-width at half-maximum of 20nm. For the MSF mode, it is a constant (takes a value of 1) in the two intervals of 400-700nm and 800-1000nm, and zero in other intervals. Let be the dynamic function of electrochromic transmittance, when When the workstation is ECF, the function is expressed as:

[0046] in The starting moment of voltage application (recorded by the controller). For parameters that need to be estimated online, Steady-state transmittance (dimensionless, ranging from 0 to 1). This is the time constant (in seconds). If If it is in MSF mode, then... due to model parameters... The electrochromic parameters are already included; in practice, they will be... and as We'll learn about the extended section together. Sensor temperature. It can be used to correct the time constant: ,in The baseline time constant (in seconds) is a learnable parameter for each ECF station; The temperature coefficient (unit: degree Celsius) was pre-calibrated to 0.03 through experiments. For reference temperature, 25 degrees Celsius is used in this embodiment. In actual calculations, the above time integral and wavelength integral are approximated by discretization: the exposure interval is divided into three equal segments, and the wavelength domain is sampled at 5nm intervals, i.e., the wavelength sampling interval. Then the integration is transformed into a summation:

[0047] in As the integration weights, since uniform sampling is used using the middle rectangle method, we take... The controller employs reparameterization techniques and stochastic gradient variational Bayes for optimization: the gradient is calculated for each new observation that arrives. ,renew and And cut to ensure The online update performs one or more gradient steps (typically 1 to 3 steps). After the update is complete, the controller outputs the posterior mean of the implicit spectral radiation field model at each spatial coordinate and wavelength position. and posterior variance The posterior mean is the reconstructed hyperspectral cube, and the posterior variance characterizes the uncertainty of the model under the current observation data.

[0048] The controller calculates the expected information gain for each candidate acquisition action based on the posterior variance of the current implicit spectral radiation field model. Candidate acquisition actions include switching the filter wheel to one of the four ECF narrowband positions to acquire one frame (action). ), or switch to MSF to capture one frame (action) The expected information gain is defined as the expected reduction in the posterior uncertainty (entropy) of the model after acquiring new observation data. Let the current variational posterior be... For candidate actions The corresponding observation data (That is, the image frame) follows a predicted distribution. The expected information gain is:

[0049] in This is the differential entropy. Direct calculation is complex; this embodiment adopts an approximation strategy: it only considers the reduction in posterior variance on the band corresponding to the candidate acquisition action, and uses single-step gradient descent after a virtual observation to estimate it. Specifically, for candidate ECF narrowband actions... (band) The controller simulates adding a virtual observation to this band, with the observed value set as the current posterior mean. The variance of the observation noise is set to .in This is a constant pre-calibrated based on sensor dark noise and readout noise; in this embodiment, it is set to 0.01 (corresponding to the quantization noise level of a 12-bit ADC). Then, a variational gradient descent is performed on the virtual observation to obtain a roughly updated posterior variance. The approximate expected information gain of this action is:

[0050] For MSF actions Then, the sum of the variance reductions in the visible light band (center wavelength 550nm) and the near-infrared band (center wavelength 900nm) should be considered simultaneously:

[0051] in , The controller calculates the values ​​of all five candidate actions, selects the action corresponding to the maximum value as the next frame acquisition action, and sends the corresponding voltage command to the DAC via GPIO. Simultaneously, it controls the stepper motor (when switching filter types is required) to rotate. This approximation method can be completed within 0.5 milliseconds on an embedded platform.

[0052] Furthermore, the controller includes a heterogeneous computing unit comprising an FPGA and an ARM processor. The FPGA receives the raw image data output from the image sensor and performs dark level correction and flat field correction, while the ARM processor performs variational inference and calculates the expected information gain. The FPGA and ARM processor share the corrected image data via an AXI bus. The FPGA has a built-in correction lookup table and dark level correction coefficients. Flat field correction coefficient These are pre-stored in the FPGA's block RAM during the system's factory calibration phase. For each frame of RAW image, the FPGA processes each pixel in parallel, using the following calculation formula: ,in These are the original pixel values. This is the corrected output. The corrected image data is stored in a shared buffer of DDR4 memory via AXI direct memory access. The ARM core reads this data through the AXI bus for subsequent variational inference and active decision-making. The entire pipeline design ensures that all processing is completed within a 30-millisecond cycle per frame, meeting real-time requirements.

[0053] Furthermore, the implicit spectral radiation field model is implemented using a multilayer perceptron with a sinusoidal activation function. Its inputs are normalized spatial coordinates and wavelength, and its output is the corresponding radiation intensity value. In the offline stage, the model obtains prior parameters through pre-training on a hyperspectral dataset, and during online runtime, variational inference updates the posterior parameters based on these prior parameters. The specific implementation of this part has been detailed in the controller step described above and will not be repeated here.

[0054] Furthermore, variational inference involves treating the parameters of the implicit spectral radiation field model as random variables following a Gaussian distribution with a mean field, and updating the mean and variance of this distribution by maximizing the lower bound of evidence. The lower bound of evidence includes the expected log-likelihood term and the KL divergence term, where the expected log-likelihood term is calculated based on the error between the original image data and the predicted values ​​from the forward imaging model. The forward imaging model includes the filter spectral response, the electrochromic transmittance dynamic function, and the sensor integration process. The mathematical formulas and parameter definitions described above have been elaborated in detail in the controller step; please refer to the preceding text.

[0055] Furthermore, the electrochromic transmittance dynamic function is as follows: transmittance increases exponentially from zero to steady-state transmittance over time, with the rate of increase controlled by the time constant. The steady-state transmittance and time constant, as scalable parameters of the implicit spectral radiation field model, are estimated together with the parameters of the deep neural network in variational inference. The specific form of this dynamic function and its application in the forward model have been given earlier using LaTeX formulas, and the parameters... , , All of these have clear definitions.

[0056] Furthermore, the method for calculating the expected information gain includes: for each candidate acquisition action, the expected reduction in the posterior variance of the implicit spectral radiation field model based on the observed data obtained after simulated acquisition, and using this expected reduction as the expected information gain for that action; the posterior variance characterizes the uncertainty of the implicit spectral radiation field model under the current observed data. Its approximate calculation strategy has been described in detail above, including steps such as virtual observation, single-step gradient descent, and variance integration.

[0057] Furthermore, the calculation of the expected information gain employs an approximation strategy: only the posterior variance reduction on the band corresponding to the candidate acquisition action is considered, and the posterior variance reduction is estimated by single-step gradient descent after a virtual observation of that band. The above has been fully disclosed in the controller step.

[0058] Furthermore, the controller also outputs an image of the posterior mean of the implicit spectral radiation field model in a specified band, and calculates the normalized difference index based on this image. When the normalized difference index is lower than a preset threshold, an abnormal alarm signal is output; simultaneously, regions with posterior variance exceeding a preset threshold are marked as low-confidence regions. In the hot spot detection application of the new energy power plant in this embodiment, the controller first extracts the posterior mean image with a wavelength of 850 nanometers from the radiation field. And the posterior mean image at a wavelength of 650 nm Calculate the normalized difference exponent:

[0059] Preset threshold Based on statistical analysis of historical hot spot data and normal background, the NDHI was calibrated to 0.05. Its physical meaning is: when NDHI is less than 0.05, it indicates that the near-infrared reflectance is significantly lower than the red light reflectance, consistent with the spectral characteristics of hot spots causing chlorophyll attenuation or equipment overheating. When the location of the pixel is identified as an abnormal hot spot, an alarm signal is output. Simultaneously, a posterior variance image with a wavelength of 850 nm is extracted. Set variance threshold The coefficient 0.1 is an empirical value used to filter out the top 10% of regions with the largest variance as low-confidence regions. (The remaining text appears to be incomplete and possibly contains errors.) Areas with low confidence are marked as such in the output image, prompting maintenance personnel to manually verify these areas. The system transmits a real-time video stream, which integrates pseudo-color hotspot markers and low-confidence markers, to the host computer via Ethernet or Wi-Fi, thus completing the entire intelligent imaging and diagnostic process.

[0060] In the intelligent imaging system based on electrochromic and multispectral synergy described in the above embodiments, the system operates as follows: After power-on, the controller first loads the prior parameters of the offline pre-trained implicit spectral radiation field model and controls the stepper motor to rotate the filter wheel so that the multi-band synchronous acquisition filter area is aligned with the photosensitive surface of the image sensor. Multiple frames of dual-band images are continuously acquired as initial observation data to quickly initiate the variational inference process. Within each frame acquisition cycle, the image sensor outputs raw RAW image data to the FPGA. The FPGA performs dark level correction and flat field correction in parallel and writes the corrected image to shared memory via the AXI bus. The ARM core reads the corrected image and the corresponding acquisition parameters (filter type, voltage value, exposure start and end times, sensor temperature). Based on the posterior distribution of the current implicit spectral radiation field model, the model parameters are updated by maximizing the lower bound of evidence to obtain the posterior mean (i.e., the reconstructed hyperspectral cube) and posterior variance (uncertainty quantification). Subsequently, the ARM core uses posterior variance to calculate the approximate expected information gain for each candidate acquisition action (four ECF narrowband stations and MSF mode), selects the action with the highest gain, outputs the corresponding voltage through the DAC and sliding brush ring, and controls the stepper motor to switch the filter wheel, driving the acquisition of the next frame. This process repeats, with the system completing acquisition, correction, variational inference update, active decision-making, and filter wheel switching within approximately 30 milliseconds per frame, achieving real-time, high-quality reconstruction of hyperspectral images. Finally, the system extracts the image of a specified band from the reconstructed radiation field and calculates the normalized difference exponent. If the exponent falls below a preset threshold, an abnormal alarm is output, along with a low-confidence region marker indicating a posterior variance exceeding the threshold, for maintenance personnel to verify. The entire system achieves a dynamic balance between hyperspectral accuracy, temporal resolution, and adaptive acquisition capabilities through deep integration of hardware and algorithms.

[0061] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A smart imaging system based on electrochromic and multispectral synergy, characterized in that, include: A miniaturized switchable filter wheel integrates an electrochromic filter station and a multi-band synchronous acquisition filter area. The electrochromic filter station switches its transmission band by applying voltage, and the multi-band synchronous acquisition filter area transmits both visible light and near-infrared light in a single exposure. An image sensor that receives light signals incident through a filter wheel and outputs raw image data; A controller performs the following steps: acquiring multiple frames of raw image data output by an image sensor and the acquisition parameters corresponding to each frame, including the currently used filter type, applied voltage value, and exposure start and end times; constructing an implicit spectral radiation field model, parameterized by a deep neural network, to characterize the radiation intensity at any spatial coordinate and wavelength position in the scene; updating the parameters of the implicit spectral radiation field model online through variational inference based on the multiple frames of raw image data and acquisition parameters, and outputting the posterior mean and posterior variance of the model at each spatial coordinate and wavelength position; calculating the expected information gain of each candidate acquisition action based on the posterior variance of the current implicit spectral radiation field model, where candidate acquisition actions include switching the filter wheel to a designated electrochromic filter station to acquire one frame, or switching to a multi-band synchronous acquisition filter area to acquire one frame; selecting the candidate acquisition action with the largest expected information gain and controlling the filter wheel to execute the action.

2. The intelligent imaging system based on electrochromic and multispectral synergy according to claim 1, characterized in that, The diameter of the filter wheel is less than 20 mm, and the response speed of the electrochromic filter station is less than 0.1 seconds. The multi-band synchronous acquisition filter area adopts a micro-nano structure with alternating checkerboard or strip arrangement. The first area transmits visible light from 400 nm to 700 nm, and the second area transmits near-infrared light from 800 nm to 1000 nm.

3. The intelligent imaging system based on electrochromic and multispectral synergy according to claim 1, characterized in that, The implicit spectral radiation field model is implemented using a multilayer perceptron with a sinusoidal activation function. Its input is normalized spatial coordinates and wavelength, and its output is the corresponding radiation intensity value. The model obtains prior parameters through pre-training on a hyperspectral dataset in the offline stage, and performs posterior updates based on the prior parameters during online runtime variational inference.

4. The intelligent imaging system based on electrochromic and multispectral synergy according to claim 1, characterized in that, Variational inference involves treating the parameters of the implicit spectral radiation field model as random variables following a mean Gaussian distribution, and updating the mean and variance of this distribution by maximizing the lower bound of evidence. The lower bound of evidence includes the expected log-likelihood term and the KL divergence term, where the expected log-likelihood term is calculated based on the error between the original image data and the predicted values ​​of the forward imaging model. The forward imaging model includes the filter spectral response, the electrochromic transmittance dynamic function, and the sensor integration process.

5. The intelligent imaging system based on electrochromic and multispectral synergy according to claim 4, characterized in that, The electrochromic transmittance dynamic function is as follows: the transmittance increases exponentially from zero to steady-state transmittance over time, and the rate of increase is controlled by the time constant. The steady-state transmittance and the time constant are used as scalable parameters of the implicit spectral radiation field model and are estimated together with the parameters of the deep neural network in variational inference.

6. The intelligent imaging system based on electrochromic and multispectral synergy according to claim 1, characterized in that, The method for calculating the expected information gain includes: for each candidate acquisition action, the expected reduction of the posterior variance of the implicit spectral radiation field model by the observation data obtained after the simulation acquisition, and using this expected reduction as the expected information gain of the action; the posterior variance characterizes the uncertainty of the implicit spectral radiation field model under the current observation data.

7. The intelligent imaging system based on electrochromic and multispectral synergy according to claim 6, characterized in that, The expected information gain is calculated using an approximation strategy: only the posterior variance reduction on the band corresponding to the candidate acquisition action is considered, and the posterior variance reduction is estimated by single-step gradient descent after a virtual observation of the band.

8. The intelligent imaging system based on electrochromic and multispectral synergy according to claim 1, characterized in that, The controller includes a heterogeneous computing unit consisting of an FPGA and an ARM. The FPGA receives raw image data from the image sensor and performs dark level correction and flat field correction, while the ARM unit performs variational inference and calculation of expected information gain. The FPGA and ARM share the corrected image data via an AXI bus.

9. The intelligent imaging system based on electrochromic and multispectral synergy according to claim 1, characterized in that, The filter wheel is also equipped with a stepper motor and a sliding brush ring: the stepper motor drives the filter wheel to rotate, so that the designated electrochromic filter station or multi-band synchronous acquisition filter area is aligned with the photosensitive surface of the image sensor; the sliding brush ring transmits the voltage output by the controller to the electrochromic filter station currently aligned with the sensor, and the other electrochromic filter stations are not powered.

10. The intelligent imaging system based on electrochromic and multispectral synergy according to claim 1, characterized in that, The controller also outputs an image of the posterior mean of the implicit spectral radiation field model in a specified band, and calculates the normalized difference index based on the image. When the normalized difference index is lower than a preset threshold, an abnormal alarm signal is output. At the same time, regions with posterior variance exceeding a preset threshold are marked as low confidence regions.

Images