A sensing and computing integrated spectral camera processor system for edge-oriented real-time analysis

Abstract

The application discloses a sensing and computing integrated spectral camera processor system for edge real-time analysis, and realizes efficient edge computing through hardware-algorithm-accelerator collaborative innovation. The hardware system adopts a three-level board card architecture: the core board integrates a programmable logic unit and a processing system, and real-time processing of spectral data collected by a CMOS board; the interface board accurately controls power timing. In the lightweight classification network, the SSFE branch converts frequency domain filtering into cyclic matrix multiplication; the HFViT branch replaces layer normalization with batch normalization; cooperate with 8-bit full integer quantization strategy to compress the model and reserve the input spectral discrimination features. The accelerator designs time overlap scheduling, and executes SSFE matrix multiplication in the HFViT calculation window period; the outer product pulsation architecture improves the matrix operation efficiency; and unified nonlinear kernel multiplexing calculation resources. The system supports real-time classification of push-broom spectral data, and the results are directly connected to the sorting equipment, and meets the needs of edge embedded deployment.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of deep learning and embedded image processing, and particularly relates to a sensing and computing integrated spectral camera processor system for edge-side real-time analysis. BACKGROUND

[0002] The hyperspectral imaging system has high spatial resolution and continuous spectral sampling capability, and is widely used in the fields of non-contact and non-destructive classification and detection of chemical substances such as medicine. However, the traditional push-broom hyperspectral system usually adopts a “sensing-computing” separation architecture, and needs to transmit the collected spectral data to an industrial computer through a high-speed interface for post-processing. This architecture not only introduces a large bandwidth and delay overhead, but also relies on a general-purpose processor with high power consumption, which is difficult to meet the application requirements of real-time, low power consumption and compactness on the edge side.

[0003] Currently, a variety of processor platforms have been used for hyperspectral image processing, including CPU, GPU, ASIC and FPGA. Among them, the CPU has flexible architecture but limited computing power; the GPU has high parallel computing power but low energy efficiency; the ASIC has the advantage of high customization, but has poor flexibility and high development cost. In comparison, the FPGA has the advantages of reconfigurable logic structure, fine parallelism and low power consumption, and becomes an ideal choice for building a special computing platform in the field of hyperspectral imaging. Existing research has implemented a variety of traditional hyperspectral algorithms on FPGA, such as hybrid decomposition algorithm based on image space reconstruction, target detection based on Gram-Schmidt orthogonalization, and anomaly detection based on multivariate Gaussian model, which have achieved certain results in real-time and resource efficiency. However, these methods generally rely on traditional analysis models and are difficult to extract complex inter-spectral nonlinear relationships and spatial context information, resulting in low accuracy in fine spectral classification. In recent years, with the rapid development of deep learning, the visual Transformer (ViT) architecture has gradually become a new trend in hyperspectral image analysis due to its global receptive field and powerful spatial-spectral modeling capability. At the same time, existing research has begun to deploy lightweight CNN networks to the FPGA platform to achieve low-power inference using its parallel structure. However, existing deployments still focus on CNN and RNN structures, and there is still a lack of research on deploying ViT models on FPGA. In addition, existing models based on ViT improvements usually have deep attention structures and irregular data flows, which pose great challenges to hardware implementation.

[0004] In view of the above problems, the present application provides a sensing and computing integrated spectral camera processing scheme for edge-side real-time analysis. SUMMARY

[0005] In view of the above technical problems, the present application provides a sensing and computing integrated spectral camera processor system for edge-side real-time analysis.

[0006] The technical scheme adopted by the present application to solve its technical problems is:

[0007] A sensing and computing integrated spectral camera processor system for edge real-time analysis, comprising a hardware processing system, a lightweight hyperspectral classification network PBViT, and a PBViT hardware accelerator;

[0008] The hardware processing system comprises a core board, an interface board, and a CMOS board, wherein the core board integrates a programmable logic unit FPGA and a processing system PS, and collects hyperspectral data through a MIPI CSI-2 interface connected with the CMOS board; the interface board is loaded with an STM32 microcontroller, and manages a multi-voltage domain power supply timing and an external interface; and the CMOS board is configured with a CMOS sensor;

[0009] The lightweight hyperspectral classification network PBViT takes a spatial patch generated from a hyperspectral image of a measured object as input, and outputs a category prediction result, which is output to an external display system through a PS side UART interface, the network comprises a frequency domain encoding branch SSFE and a hardware-friendly ViT branch HFViT which are executed in parallel, the frequency domain encoding branch converts frequency domain filtering into cyclic matrix multiplication, and the hardware-friendly ViT branch replaces layer normalization with batch normalization, and the network is implemented by full-integer quantization through 8-bit quantization perception training;

[0010] The PBViT hardware accelerator integrates a linear calculation core, a nonlinear calculation core, and a residual path core, performs the SSFE branch in parallel during Softmax calculation of the HFViT through a time overlap scheduling strategy, adopts a streaming matrix multiplication architecture based on outer product, and time-division multiplexes BN, quantization, Softmax, and GELU operations through a unified nonlinear calculation core.

[0011] Preferably, the core board in the hardware processing system is built-in with a spectral preprocessing module and a PBViT accelerator, is configured with a DDR4 cache array and a 10G Ethernet communication interface, and uploads spectral data to an upper computer through a UDP protocol;

[0012] The interface board provides an RJ45 Ethernet port, a JTAG debugging interface, an STLINK interface, a USB-to-UART interface, a 12V power supply interface, and an SD card slot;

[0013] The CMOS board realizes compatible communication between 1.8V I2C and 1.2V I / O Bank of the core board through a PCA9306 level conversion chip.

[0014] Preferably, data collection of the core board comprises:

[0015] DPHY resources are used to decouple LP and HS signals, and an IDELAY primitive is used to complete phase alignment of clock and data;

[0016] Byte alignment is achieved by Bitslip technique, and multi-channel synchronization is realized by SoT packet;

[0017] The spectral preprocessor module adopts a pipeline architecture to complete the operations of pixel binning, whiteboard correction and spectral smoothing;

[0018] The preprocessed spectral lines are written into the DDR4 chip cache through the built-in DDR controller and assembled into a 9x9 spatial patch, denoted as:

[0019] ;

[0020] wherein, represents a spectral curve, and the dimension size is 63, represents the number of spectral curves contained in the spatial patch.

[0021] Preferably, the STM32 microcontroller of the interface board monitors the Power-Good signals of each voltage regulator and outputs the Enable signal to control the power-on sequence of the core board and the CMOS board according to the preset timing; the integrated voltage regulator converts the 12V input into multiple voltage domains for power supply.

[0022] Preferably, in the frequency domain encoding branch, it is assumed that the input image block is wherein, represents the spatial dimension, is the number of spectra, and the branch sequentially performs the following frequency domain filtering operations in the spatial and spectral dimensions:

[0023] ;

[0024] wherein, is a conjugate symmetric learnable filter, represents element-wise multiplication, is a Fourier transform, is an inverse Fourier transform, is an input image block, is an image block output after frequency domain filtering;

[0025] According to the convolution theorem, the above formula can be equivalently converted into a matrix multiplication form:

[0026] ;

[0027] wherein, represents a circulant matrix constructed based on the inverse transform of This transform is used in the inference process of the network and can reuse the standard matrix multiplication circuit.

[0028] Preferably, each ViTBlock of the HFViT is composed of the following two substructures:

[0029] ;

[0030] ;

[0031] wherein, is the input, is the intermediate output after multi-head attention, is the final output, denotes the batch normalization operation, is the multi-head self-attention, is a two-layer feedforward network with GELU activation.

[0032] Preferably, the quantization strategy of the quantization-aware training process includes Token-level quantization for input activation, layer-level quantization for intermediate activation, and channel-level quantization for weights;

[0033] Quantization perturbation is introduced in the quantization-aware training process to make the model adapt to the error caused by quantization. The quantization adopts a symmetric linear strategy:

[0034] ;

[0035] wherein is a floating-point number, is a scaling factor obtained by the min-max algorithm, is the quantization bit width, and the clip function limits the output within the representable range, is used to round the input to an integer.

[0036] Preferably, the time overlap scheduling strategy is as follows:

[0037] In the process of performing Softmax in the HFViT branch, there is a delay in the exponential calculation and normalization operation, so the linear calculation module will be in an idle state. The cyclic matrix multiplication of the SSFE branch is performed in parallel during this idle period.

[0038] The BN operation of the SSFE is processed using the idle GELU path DSP resources in the nonlinear calculation core.

[0039] Preferably, the linear calculation core adopts a 32x32 outer product systolic array; one column of matrix A and one row of matrix B are input per cycle, and the corresponding outer product is calculated directly in the processing array; through a double 8-bit operand packing strategy, two multiplications are performed in parallel within a single DSP48E2 unit; the array supports streaming data scheduling with results propagating to the right or downward.

[0040] Preferably, the unified nonlinear calculation core performs BN, quantization, Softmax and GELU operations through time division multiplexing hardware resources, wherein BN and quantization are fused into a channel-level multiplication operation, and second-degree polynomial integer approximation is adopted for Softmax and GELU.

[0041] The present application proposes an edge-end real-time analysis sensing and computing integrated spectral camera processor system, which realizes three breakthroughs through hardware-algorithm-accelerator collaborative innovation: the hardware system adopts a three-level board card architecture, the core board integrates spectral preprocessing and accelerators, the CMOS board carries a 400-700nm high quantum efficiency sensor to accurately capture the characteristic spectrum of the measured object, and the interface board controls the power timing through an STM32 to ensure stable operation in complex environments; the PBViT network fuses double branches: SSFE converts frequency domain filtering into cyclic matrix multiplication to avoid FFT resource consumption, HFViT eliminates real-time calculation overhead through batch normalization, and cooperates with 8-bit full integer quantization to compress the model while retaining the fitting ability of the model; the accelerator design initiates time overlap scheduling, performs SSFE matrix multiplication in parallel during the Softmax empty window period of HFViT, reduces delay, and cooperates with a double 8-bit operand packing to unify the nonlinear core to fuse BN and quantization and integer approximate Softmax and GELU, thereby reducing resource consumption. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 Fig. 1 is a structural schematic diagram of a hyperspectral camera circuit hardware processing system in an embodiment of the present application;

[0043] Figure 2 Fig. 2 is a principle schematic diagram of a lightweight hyperspectral classification network PBViT in an embodiment of the present application;

[0044] Figure 3 Fig. 3 is a structural schematic diagram of a PBViT hardware accelerator in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to enable personnel in the art to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the drawings.

[0046] In one embodiment, a sensing and computing integrated spectral camera processor system for edge-end real-time analysis includes a hardware processing system, a lightweight hyperspectral classification network PBViT, and a PBViT hardware accelerator.

[0047] The hardware processing system comprises a core board, an interface board and a CMOS board, wherein the core board integrates a programmable logic unit FPGA and a processing system PS, collects hyperspectral data through a CMOS board connected through a MIPI CSI-2 interface; the interface board carries an STM32 microcontroller, manages multi-voltage domain power supply timing and external interfaces; and the CMOS board is configured with a CMOS sensor.

[0048] The lightweight hyperspectral classification network PBViT takes spatial patches generated from the spectral image of the measured object as input and outputs a category prediction result, which is output to an external display system through a PS side UART interface, and the network comprises a frequency domain encoding branch SSFE and a hardware-friendly ViT branch HFViT which are executed in parallel, the frequency domain encoding branch converts frequency domain filtering into cyclic matrix multiplication, and the hardware-friendly ViT branch replaces layer normalization with batch normalization, and the network is implemented by full-integer quantization through 8-bit quantization perception training;

[0049] The PBViT hardware accelerator integrates linear computing cores, nonlinear computing cores and residual path cores, performs the SSFE branch in parallel during the Softmax calculation of the HFViT through a time overlap scheduling strategy, adopts a streaming matrix multiplication architecture based on outer product, and time-division multiplexes BN, quantization, Softmax and GELU operations through a unified nonlinear computing core.

[0050] Specifically, the first aspect of the present application proposes a hyperspectral camera circuit hardware processing system, as shown in Figure 1 As shown, it comprises a single camera hardware processor composed of a core board, an interface board and a CMOS board. The core board adopts Xilinx ZYNQ Ultrascale+ MPSoC ZU19EG as a core processor, integrates a power management module, a memory (including PS / PL DDR, QSPI memory, eMMC memory), an optical interface (SFP+) and other related peripheral interfaces. The interface board carries an STM32F103RBT6 single-chip microcomputer for power timing control and management of external interfaces (ETH, UART, SD, etc.). The CMOS board carries a Siliti SC130GS CMOS sensor to provide high-precision image acquisition and communicate with the core board through a 4-lane MIPI interface.

[0051] The second aspect of the present application proposes a lightweight hyperspectral classification network PBViT which is composed of a hardware-friendly ViT (HFViT) and a spatial-spectral frequency domain encoding (SSFE) branch, the input spectral line flows into the PL end DDR4 after passing through a spectral preprocessing module to be cached as a spatial patch, and finally the PBViT is used to predict the category of the measured object. Based on quantization perception training (QAT), the model is quantized to 8-bit integer type, which is convenient for subsequent FPGA accelerator deployment.

[0052] The third aspect of the present application proposes a FPGA-based PBViT accelerator. Through a time-overlapping scheduling strategy, the prediction time of SSFE is hidden in the inference process of HFViT, and the calculation resources of HFViT are completely reused. A linear calculation kernel based on outer product is designed for all matrix multiplication operations in the network. A general nonlinear calculation kernel is designed for all nonlinear operations such as quantization, dequantization, Softmax, normalization, GELU, etc. in the network.

[0053] In one embodiment, the core board in the hardware processing system is built-in with a spectrum preprocessing module and a PBViT accelerator, and is configured with a DDR4 cache array and a 10G Ethernet communication interface, and uploads spectrum data to an upper computer through a UDP protocol;

[0054] The interface board provides an RJ45 Ethernet port, a JTAG debugging interface, an STLINK interface, a USB-to-UART interface, a 12V power supply interface, and an SD card slot.

[0055] The CMOS board realizes compatible communication between 1.8V I²C and 1.2V I / O Bank of the core board through a PCA9306 level conversion chip.

[0056] Specifically, the core board is the core unit of the optical circuit hardware system, and undertakes tasks such as spectrum reception, on-chip calculation, and communication management. Based on the ZU19EG chip platform, the core board integrates programmable logic (FPGA) and processing system (PS), and has powerful computing power, memory bandwidth, and high-speed communication capability, which is suitable for real-time processing requirements in push-broom hyperspectral imaging. The spectrum data is collected through the CMOS board, and the on-board photosensitive device is the SC130GS CMOS image sensor chip of Silterra. The chip has 1.3 million pixels, global shutter characteristics, and high quantum efficiency in the range of 400-700 nm. It can output 10-bit spectrum data at a speed of 120 frames per second through a four-channel MIPI CSI-2 interface. The 27MHz input clock required by the sensor is provided by the on-board crystal oscillator. In addition, the I²C interface level used by the CMOS is 1.8V, while the VCCO of the SoC side I / O Bank is 1.2V, so a PCA9306 level conversion chip is added in the design to realize bidirectional voltage compatible communication.

[0057] In one embodiment, the data acquisition of the core board includes:

[0058] The DPHY resource is used to decouple the LP and HS signals, and the IDELAY primitive is used to complete the phase alignment of the clock and data.

[0059] The Bitslip technology is used to complete byte alignment, and the SoT packet is used to realize multi-channel synchronization.

[0060] The spectral preprocessor module adopts a pipeline architecture to complete the operations of pixel binning, whiteboard correction and spectral line smoothing.

[0061] The preprocessed spectral lines are written into the DDR4 chip buffer through the built-in DDR controller and assembled into a 9x9 spatial patch. The patch is represented as:

[0062]

[0063] wherein, represents the spectral curve, and the dimension size is 63, represents the number of spectral curves contained in the spatial patch.

[0064] Specifically, in terms of spectral reception, the FPGA side performs high-speed data acquisition through the MIPI CSI-2 interface, decouples the LP and HS signals using the DPHY resource, and completes the phase alignment of the clock and data through the IDelay primitive. Subsequently, byte alignment is completed using the Bitslip technology, and multi-channel alignment is achieved with the help of the SoT package. In terms of on-chip computing, the system integrates a spectral preprocessor module and a PBViT accelerator. The spectral preprocessor module adopts a pipeline architecture to complete the operations of pixel binning, whiteboard correction and spectral line smoothing. The processed data is written into the DDR4 chip buffer through the built-in DDR controller. The FPGA side is configured with four MT40A1G16RC DDR4 chips, totaling 8GB, with a working frequency of 1600MHz, for data caching for hyperspectral processing. The PBViT accelerator is used for real-time classification of spectral images. In terms of data communication, control signals are debugged by the host through the JTAG interface, or CMOS registers are configured through the I2C interface; data transmission adopts the GTH transceiver to build a 10G Ethernet, and combines the UDP protocol to transmit spectral image streams to the upper computer. At the same time, the classification results of the PBViT are sent to the external system through the PS side UART interface for display and recording. In addition, the core board integrates QSPI and eMMC memories, which are used for program startup and configuration file storage, respectively; the PS part is also equipped with an 8GB DDR4 memory, which provides the memory resources required for software running of the ARM core.

[0065] In one embodiment, the STM32 microcontroller of the interface board monitors the Power-Good signals of each voltage regulator and outputs the Enable signal to control the power-up sequence of the core board and the CMOS board according to the preset timing; the integrated step-down voltage regulator converts the 12V input into multiple voltage domains for power supply.

[0066] ​Specifically, to adapt to multiple voltage domains inside the system, the interface board is designed with multiple sets of voltage reduction stabilizers to convert the 12V input voltage into different voltages required by each module. Considering the strict power-on timing requirements of ZU19EG, an STM32F103 microcontroller is integrated in the interface board to realize power management and power-on control. The controller can monitor the PG (Power-Good) signals of each voltage stabilizer and output the EN (Enable) signal according to the set order to ensure reliable system startup.

[0067] Further, the present application proposes a PBViT network suitable for hyperspectral image classification, as shown in Figure 2 The overall structure contains two parallel branches: a frequency domain encoding branch (SSFE) and a hardware-friendly ViT branch (HFViT). The structure fuses the output results of the two branches through soft voting to achieve high-precision and low-delay push-broom hyperspectral image recognition, which is suitable for FPGA deployment.

[0068] In one embodiment, the feature distribution law of the hyperspectral image in the frequency domain is designed for the frequency domain encoding branch SSFE. In the frequency domain encoding branch, assuming that the input image block is wherein, represents the spatial dimension, is the number of spectra, and the branch sequentially performs the following frequency domain filtering operations in the spatial and spectral dimensions:

[0069] ;

[0070] wherein, is a conjugate symmetric learnable filter, represents element-wise multiplication, is a Fourier transform, is an inverse Fourier transform, is an input image block, is an image block output after frequency domain filtering;

[0071] According to the convolution theorem, the above formula can be equivalently converted into matrix multiplication form:

[0072] ;

[0073] wherein, represents a circulant matrix constructed based on the inverse transform of This transform is used in the inference process of the network and can reuse standard matrix multiplication circuits, thereby effectively avoiding the resource overhead caused by frequency domain transform.

[0074] In one embodiment, to reduce inference latency, the present application designs HFViT, which replaces layer normalization (LN) in standard ViT with batch normalization (BN) to avoid the overhead of calculating mean and variance in real time. Each ViTBlock of HFViT consists of the following two substructures:

[0075]

[0076]

[0077] wherein, is the input, is the intermediate output after multi-head attention, is the final output, represents the batch normalization operation, is multi-head self-attention, is a two-layer feedforward network with GELU activation. This module has good hardware mapping and is easy to implement in a pipeline.

[0078] In one embodiment, the quantization strategy of the quantization-aware training process includes Token-level quantization for input activation, layer-level quantization for intermediate activation, and channel-level quantization for weights.

[0079] In the quantization-aware training process, a quantization perturbation is introduced to make the model adapt to the error caused by quantization. The quantization uses a symmetric linear strategy:

[0080]

[0081] wherein is a floating-point number, is a scaling factor obtained by the min-max algorithm, is the quantization bit width, the clip function limits the output within the representable range, is used to round the input to an integer.

[0082] Specifically, to achieve full integer deployment, the present application uses quantization-aware training (QAT). The quantization granularity is crucial. The quantization granularity of the present application includes: Token-level quantization for input activation to preserve fine-grained spatial-spectral features; layer-level quantization for intermediate activation to reduce hardware deployment difficulty; and channel-level quantization for weights to maintain model expressiveness. In addition, element-wise operations such as BN, bias addition after linear layers, and quantization operations are fused into a unified computing unit in inference to avoid redundant calculations. Nonlinear functions (such as Softmax and GELU) use quadratic polynomial approximation.

[0083] ​​​Further, in order to realize efficient deployment under the limited logic and storage resources of FPGA, the application proposes a hyperspectral network accelerator with overlapping execution architecture, as shown in Figure 3 As shown in the figure, the accelerator supports real-time inference of the PBViT model. The accelerator includes three types of special computing modules: linear computing kernel, nonlinear computing kernel and residual path kernel. The modules realize intermediate data caching through on-chip SRAM, access external DDR4 for image input through MIG interface, and dynamically load model parameters through DMA. A finite state machine controls the data flow and memory scheduling of each module, thereby improving the overall hardware utilization.

[0084] In one embodiment, the time overlapping scheduling strategy is as follows:

[0085] In the process of executing Softmax in the HFViT branch, due to the delay of exponential calculation and normalization operation, the linear computing module will be in an idle state, and the cyclic matrix multiplication of the SSFE branch is executed in parallel during the idle period.

[0086] The BN operation of the SSFE is processed by using the idle GELU path DSP resource in the nonlinear computing kernel.

[0087] Specifically, the parallel execution during the idle period can effectively hide the inference delay. In addition, the SSFE output needs to be processed by BN, and the Softmax path in the corresponding nonlinear kernel has occupied the first DSP resource. To avoid conflicts, the idle DSP resource in the GELU path is used for BN calculation, thereby realizing resource reuse and timing coupling between modules.

[0088] In one embodiment, the linear computing kernel adopts a 32x32 outer product systolic array; one column of matrix A and one row of matrix B are input per cycle, and the corresponding outer product is calculated directly in the processing array; through a double 8-bit operand packing strategy, two multiplications are executed in parallel in a single DSP48E2 unit; the array supports streaming data scheduling with right or downward propagation of results.

[0089] Specifically, the application adopts an outer product-based streaming matrix multiplication architecture, which avoids the rearrangement overhead of input tensors in traditional structures. One column of matrix A and one row of matrix B are input per cycle, and the corresponding outer product is calculated directly in the processing array. The core structure is a 32x32 systolic array, each PE unit has integer multiplication and addition capability, local registers and result forwarding path. DSP48E2 resources are used for multiplication mapping, and a double 8-bit operand packing strategy is designed to execute two multiplications in parallel in each DSP, thereby saving half of the multiplication resources. The array supports right or downward propagation of results, which adapts to different data flow directions.

[0090] In one embodiment, the unified nonlinear calculation core performs BN, quantization, Softmax and GELU operations by time division multiplexing hardware resources, wherein the BN and quantization are fused into a channel-level multiplication and addition operation, and the Softmax and GELU adopt quadratic polynomial integer approximation.

[0091] Specifically, the nonlinear calculation core adopts a unified structure design and supports BN, quantization, Softmax, GELU and other operations by time division multiplexing. Since BN and quantization are both channel-level element-by-element operations in the inference stage, the system fuses them into one arithmetic path and realizes one multiplication and addition by using pre-stored fixed-point parameters. The bias addition in the linear layer is also combined with quantization and multiplexes the same hardware logic. To adapt to integer operations, the Softmax and GELU functions are approximated by polynomials, and integer multiplication and addition are used to replace floating-point operations of exponential and error functions, so as to reduce resource overhead. These approximation operations are all integrated into the unified nonlinear unit, reducing logic resource consumption.

[0092] In order to evaluate the accuracy and consistency of the hyperspectral image classification processing system proposed in the present application, the following three evaluation indexes are selected: overall accuracy (OA), average accuracy (AA) and Cohen's Kappa consistency coefficient (K).

[0093] Comparative test. The PBViT model proposed in the present application is compared with traditional machine learning models, CNN methods and network architectures based on ViT. The test data set uses traditional Chinese medicine data set, and the results are summarized in Table 1. The traditional machine learning model has poor overall classification performance due to the lack of modeling ability of spatial-spectral joint features. The CNN method performs better than the traditional model in accuracy, especially the three-dimensional convolutional network (3DCNN) can model the spatial and spectral dimensions at the same time, and achieves high accuracy. However, this type of network has large parameters and relies on complex data rearrangement operations, resulting in low deployment efficiency on hardware platforms. The model based on ViT usually achieves the best performance due to its global modeling capability. However, SSFTT and GCSViT introduce irregular network topology, which significantly increases the design complexity of the hardware accelerator. In contrast, the PBViT model proposed in the present application achieves the best performance in all indicators, and has the following advantages: regular network structure, least parameter quantity, and hardware-friendly.

[0094] Table 1 Comparative test

[0095]

[0096] Ablation and quantization experiments. To verify the independent contribution of each module proposed in the present application, ablation experiments were designed and carried out on the Chinese herbal medicine (CHM) dataset, and the results are shown in Table 2. First, after replacing the layer normalization (LayerNorm) in the ViT network with batch normalization (BatchNorm) to construct the HFViT structure, the overall accuracy (OA) of the model remained at 0.965, and the average accuracy (AA) increased from 0.914 to 0.921, indicating that BN as a hardware-friendly alternative can significantly reduce computational complexity without compromising performance. After introducing the frequency domain modeling branch SSFE, the model achieved an OA of 0.972, an AA of 0.916, and a Kappa coefficient of 0.937, verifying the effectiveness of frequency domain information in hyperspectral image classification. Further application of HFViT and SSFE jointly, the model's performance is optimal, AA from 0.921 significantly improved to 0.942, Kappa from 0.922 to 0.950, reflecting the synergistic gain effect between the two modules.

[0097] In addition, to verify the feasibility of deploying the model on FPGA, the HFViT and SSFE modules were quantized to 8-bit integers, respectively, denoted as HFViT* and SSFE*. Experimental results show that the performance of the quantized sub-modules decreases very little; after quantizing both modules, AA and Kappa decrease by only 0.3% and 0.2%, respectively. This result shows that the proposed model has good quantization robustness.

[0098] Table 2 Ablation and quantization experiments

[0099]

[0100] Accelerator cross-platform experiments. To verify the performance of the processor in the present application in terms of inference efficiency and power consumption control, inference speed and power consumption were compared on Intel i9-13900Hx CPU, RTX 4090 Laptop GPU, and FPGA accelerator based on ZU19EG platform in the present application. The experimental results are shown in Table 3. The CPU platform is limited by the serial computing structure, with the slowest inference speed; the GPU has a large number of floating-point operation units and can achieve the fastest inference speed, but its power consumption is high and is not suitable for edge deployment. In contrast, the FPGA accelerator proposed in the present application achieves a good balance between inference delay and power consumption, with an average inference time of 75.95% of the GPU, and a power consumption of only 14.42W, which is significantly lower than the GPU and CPU. In terms of energy efficiency, the inference efficiency of the system in the present application under unit power consumption is 6.24 times higher than that of the GPU and 63 times higher than that of the CPU, fully demonstrating its deployment advantages and energy efficiency in edge hyperspectral processing tasks.

[0101] Table 3 Cross-platform inference efficiency and power consumption comparison

[0102]

[0103] The application provides a spectral camera processor system for edge real-time classification, which realizes breakthrough progress of hyperspectral classification technology on the edge side through the trinity innovation of hardware architecture, lightweight network and special accelerator. At the hardware level, the three-level board architecture of the core board, the interface board and the CMOS board deeply integrates the spectral acquisition, real-time processing and precise control capabilities: the core board integrates the preprocessing module and the accelerator, directly processes the 400-700nm waveband data collected by the CMOS board, and the interface board strictly controls the multi-voltage domain timing through the STM32 microcontroller to ensure the stable operation of the system in the complex electromagnetic environment of the production site. At the algorithm level, the lightweight PBViT network has a unique double-branch collaborative architecture - the frequency domain coding branch SSFE converts the frequency domain filtering into a cyclic matrix multiplication, avoiding the resource bottleneck of traditional frequency domain transformation; the hardware-friendly ViT branch HFViT replaces the layer normalization with batch normalization, eliminating the real-time statistical quantity calculation delay; combined with the 8-bit full integer quantization strategy, the Token-level input quantization preserves spatial-spectral details, and the channel-level weight quantization maintains expression ability, which significantly compresses the model size while ensuring high-precision classification. At the acceleration level, the PBViT hardware accelerator performs SSFE matrix multiplication in parallel within the Softmax calculation window of HFViT through time overlap scheduling strategy, effectively hiding the branch calculation delay; the outer product pulsatile array architecture combined with the double 8-bit operand packing technology realizes the order of magnitude improvement of matrix multiplication; the unified nonlinear calculation core integrates BN, quantization, Softmax and GELU into a time division multiplexing unit, which greatly reduces the logic resource consumption. The system finally realizes the "acquisition-algorithm-control" full-process closed loop of spectral classification: the spectral data is processed on the chip in real time, and the classification results are directly connected to the sorting equipment through UART, providing a domestic and independent high-performance edge computing solution for intelligent manufacturing of chemicals such as medicine.

[0104] The above describes in detail the spectral camera processor system for edge real-time analysis provided by the application. The principles and implementation modes of the application are described by specific examples in this paper. The above examples are only used to help understand the core idea of the application. It should be noted that those skilled in the art can make some improvements and modifications to the application without departing from the principles of the application, and these improvements and modifications also fall within the protection scope of the claims of the application.

Claims

1. A sensor-processor integrated system for real-time edge analysis of a spectral camera, characterized in that, This includes a hardware processing system, the lightweight hyperspectral classification network PBViT, and the PBViT hardware accelerator; The hardware processing system includes a core board, an interface board, and a CMOS board. The core board integrates a programmable logic unit (FPGA) and a processing system (PS), and connects to the CMOS board via a MIPI CSI-2 interface to acquire hyperspectral data. The interface board is equipped with an STM32 microcontroller to manage the power supply timing of multiple voltage domains and external interfaces. The CMOS board is configured with CMOS sensors. The lightweight hyperspectral classification network PBViT takes a spatial patch generated from the spectral image of the object under test as input and outputs the category prediction result. It is output to an external display system through the UART interface on the PS side. The network includes a parallel frequency domain coding branch SSFE and a hardware-friendly ViT branch HFViT. The frequency domain coding branch converts frequency domain filtering into cyclic matrix multiplication, and the hardware-friendly ViT branch replaces layer normalization with batch normalization. The network achieves full integer quantization through 8-bit quantization-aware training. The PBViT hardware accelerator integrates linear computation cores, nonlinear computation cores, and residual path cores. It executes the SSFE branch in parallel during the Softmax computation of HFViT through a time-overlapping scheduling strategy. It adopts a streaming matrix multiplication architecture based on outer product and time-division multiplexes BN, quantization, Softmax, and GELU operations through a unified nonlinear computation core.

2. The system according to claim 1, characterized in that, The core board of the hardware processing system has a built-in spectral preprocessing module and a PBViT accelerator, and is equipped with a DDR4 cache array and a 10G Ethernet communication interface. It uploads spectral data to the host computer via the UDP protocol. The interface board provides an RJ45 Ethernet port, a JTAG debugging interface, an STLINK interface, a USB to UART interface, a 12V power supply interface, and an SD card slot; The CMOS board achieves compatible communication between the 1.8V I²C and the core board's 1.2V I / O Bank through the PCA9306 level conversion chip.

3. The system according to claim 2, characterized in that, Data acquisition for the core board includes: The DPHY resource is used to decouple the LP and HS signals, and the IDELAY primitive is used to complete the phase alignment of the clock and data. Byte alignment is achieved using Bitslip technology, and multi-channel synchronization is realized with the help of the SoT package; The spectral preprocessor module adopts a pipelined architecture to complete pixel merging, whiteboard correction, and spectral line smoothing operations. The preprocessed spectral lines are written to the DDR4 chip cache via the built-in DDR controller and then assembled into a 9×9 space patch. The patch is represented as follows: ； in, This represents a spectral curve with a dimension of 63. This indicates the number of spectral curves contained in a spatial patch.

4. The system according to claim 3, characterized in that, The STM32 microcontroller on the interface board monitors the Power-Good signal of each regulator and outputs the Enable signal according to the preset timing to control the power-on sequence of the core board and the CMOS board; the integrated buck regulator converts the 12V input into a multi-voltage domain power supply.

5. The system according to claim 4, characterized in that, In the frequency domain coding branch, assuming the input image patch is ,in, Indicates spatial dimension, Given the number of spectra, this branch performs the following frequency domain filtering operations sequentially in the spatial and spectral dimensions: ； in, It is a conjugate symmetric learnable filter. This represents element-wise multiplication. For Fourier transform, This is the inverse Fourier transform. For the input image patch, This is the image block output after frequency domain filtering; According to the convolution theorem, the above equation can be equivalently transformed into matrix multiplication form: ； in, Indicates based on The cyclic matrix is ​​constructed by the inverse transformation of the standard matrix multiplication circuit. This transformation is used in the inference process of the network and can reuse the standard matrix multiplication circuit.

6. The system according to claim 5, characterized in that, Each ViTBlock in HFViT consists of the following two substructures: ； ； in, For input, This is the intermediate output after multiple heads pay attention. For the final output, This indicates a batch normalization operation. For the sake of the bulls' self-attention, It is a two-layer feedforward network with GELU activation.

7. The system according to claim 6, characterized in that, The quantization strategy for the quantization perception training process includes token-level quantization for input activation, hierarchical quantization for intermediate activation, and channel-level quantization for weights. During quantization-aware training, quantization perturbations are introduced to allow the model to adapt to the errors introduced by quantization. A symmetric linear strategy is used for quantization. ； in For floating-point numbers, The scaling factor is obtained through the min-max algorithm. To quantize the bit width, the clip function limits the output to the representable range. Used to round the input to an integer.

8. The system according to claim 7, characterized in that, The time overlap scheduling strategy is as follows: During the execution of Softmax in the HFViT branch, the linear calculation module will be idle due to the delay in its exponent calculation and normalization operations. The cyclic matrix multiplication of the SSFE branch will be executed in parallel during this idle window. The BN operation of SSFE is processed using the idle GELU path DSP resources in the nonlinear computing core.

9. The system according to claim 8, characterized in that, The linear computation kernel uses a 32×32 outer product pulsating array; each cycle, one column of matrix A and one row of matrix B are input, and the corresponding outer product is directly calculated in the processing array; By employing a dual 8-bit operand packing strategy, two multiplications are performed in parallel within a single DSP48E2 unit; The array supports streaming data scheduling where results propagate to the right or down.

10. The system according to claim 9, characterized in that, The unified nonlinear computing kernel performs BN, quantization, Softmax, and GELU operations by time-division multiplexing hardware resources. BN and quantization are integrated into channel-level multiply-add operations, and quadratic polynomial integer approximations are used for Softmax and GELU.

Images