Satellite on-board fire point detection method and system based on hybrid heterogeneous architecture and storage medium
By using an ARM+FPGA hybrid heterogeneous architecture and a neural network LNN-Fire model, the timeliness and accuracy issues of satellite fire detection were solved, enabling real-time fire detection on satellites and improving processing efficiency and accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING UNIV OF AERONAUTICS & ASTRONAUTICS
- Filing Date
- 2026-03-30
- Publication Date
- 2026-07-21
Smart Images
Figure CN122432756A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of energy storage technology, specifically to a method, system, and storage medium for detecting on-board fire points based on a hybrid heterogeneous architecture. Background Technology
[0002] Satellite fire detection is currently a crucial technology for monitoring forest and grassland fires. The Medium Resolution Spectroscopic Imager II (MERSI-II) onboard the Fengyun-3D satellite (FY-3D) possesses multi-channel, high-resolution detection capabilities, providing abundant spectral data support for fire detection. Existing Fengyun satellite fire detection technologies all employ a ground-based post-processing model, following a process of "satellite scanning - data downlink - ground reception - centralized processing - product production - information dissemination" to achieve fire monitoring. This model suffers from significant timeliness deficiencies; the raw data collected by the satellite must undergo multiple stages of transmission and processing before fire information can be generated, resulting in substantial delays in fire emergency response times and failing to meet the real-time monitoring and emergency response needs of sudden fires.
[0003] Meanwhile, traditional fire detection uses a threshold method, requiring the design of detection rules specific to certain instruments, regions, and seasons based on empirical knowledge. This results in poor algorithm transferability, and detection accuracy is significantly affected by manually designed rules. Furthermore, traditional satellite data preprocessing employs a pure CPU architecture to perform geometric positioning and calibration calculations. The positioning algorithm involves numerous repetitive iterative operations, leading to high computational complexity and long processing times. The preprocessing time for a single frame of multi-channel data exceeds the satellite payload's scanning cycle, making real-time on-orbit processing impossible. In addition, the low resource utilization of a single hardware architecture makes it difficult to balance the flexibility of floating-point operations with the efficiency of iterative operations, further limiting the realization of real-time on-board fire detection. Summary of the Invention
[0004] To overcome the aforementioned shortcomings, a method, system, and computer-readable storage medium for onboard fire detection based on a hybrid heterogeneous architecture are proposed. This aims to solve, or at least partially solve, the technical problems of low processing efficiency and low accuracy in related technologies. It enables real-time onboard preprocessing of satellite payload data and fire detection, shortens emergency response time for fires, improves the accuracy of fire detection, reduces false alarm rates, and simultaneously enhances the resource utilization and processing efficiency of the onboard hardware platform.
[0005] In a first aspect, a method for detecting on-board fire points based on a hybrid heterogeneous architecture is provided, comprising: receiving raw data collected by a satellite; unpacking the raw data to obtain multi-channel digital quantized value data; performing calibration and positioning processing on the multi-channel digital quantized value data, wherein an ARM processor is used to perform single-cycle operations to handle floating-point operations and data scheduling for positioning and calibration, and an FPGA processor is used to perform repetitive iterative operations in parallel through hardware acceleration IP cores, the repetitive iterative operations including line-of-sight vector calculation and line-of-sight intersection point calculation to complete geometric positioning and calibration processing and obtain positioning and calibration data; the positioning and calibration data is used to perform fire point detection based on a neural network LNN-Fire model; and the fire point detection results are broadcast to a ground terminal.
[0006] Furthermore, the fire detection process of the LNN-Fire neural network model includes: The input layer receives data from key spectral channels after channel filtering, the key channels including thermal infrared channels and other auxiliary channels; The first channel attention mechanism is applied to the data of the key spectral channel, and channel weights are generated by the Sigmoid function and then multiplied and weighted with the data of the corresponding key spectral channel to obtain the purified spectral channel features. The channel feature linear extension layer enhances the information of the purified spectral channel data; The N-layer feature extraction module captures the deep features of the purified spectral channel data, including inter-channel features and intra-channel features; The second channel attention mechanism is applied to the deep features, and channel importance weights are generated by the Sigmoid function. These weights are then multiplied and weighted with the deep features to obtain the focused features. The linear classification layer outputs the probability value of the fire point based on the focusing features.
[0007] Furthermore, the channel selection process of the LNN-Fire neural network model includes: Pearson correlation coefficient was used to analyze the linear correlation between multiple raw spectral channels; Calculate the optimal index factor for each band combination, select the channel combination with the lowest inter-band correlation and the largest single-band standard deviation as the key spectral channel, and use the data of the key spectral channel as the input of the neural network LNN-Fire model.
[0008] Furthermore, the key spectral channels include four thermal infrared channels and / or auxiliary channels selected from the remaining 21 visible / near-infrared channels. The total number of input channels of the neural network LNN-Fire model can be configured to 4, 16, or 19 channels. A fire type pre-judgment branch is added to the input layer of the neural network LNN-Fire model. Based on the global land cover data type, the weight learning of the channel attention mechanism is automatically adjusted: for forest fires, the weight of the thermal infrared channel is increased; for grassland fires, the auxiliary feature weight of the near-infrared channel is increased; and for urban fires, the background suppression weight of the visible light channel is increased.
[0009] Furthermore, the geometric positioning and calibration process is completed using the MerAcAl model, which encapsulates the geometric positioning algorithm into a hardware acceleration IP core; the fire situation is predicted according to the type of the detection area, and the FPGA is dynamically activated or deactivated based on the prediction results.
[0010] Furthermore, a tiered decision-making process is implemented based on the probability values of fire points: high confidence is broadcast immediately, medium confidence triggers local re-verification, and low confidence is cached locally.
[0011] Furthermore, the local re-verification includes: Re-execute LNN-Fire inference on the medium-confidence pixel and its neighboring pixels, and fuse the output probabilities 9 times; By combining the spatial context of nearby confirmed fire points, a Conditional Random Field (CRF) post-processing method is used to optimize the boundary. If the verification passes, a "Confirmed Fire Point" message will be broadcast; if it fails, it will be marked as a false alarm and written to the local log; and / or In the local re-verification with medium confidence, multi-frame fusion with a time dimension is added: fire point probability fusion is performed on continuous multi-frame data of the same geographical area, and if the continuous multi-frame data are all medium confidence fire points, they are determined to be confirmed fire points.
[0012] Furthermore, the frequency and operating voltage of the ARM processor are dynamically adjusted according to the load of the hardware acceleration IP core; Monitor the utilization rate of the DSP computing unit of the FPGA acceleration core. When the utilization rate is greater than a preset value, increase the processing frequency of the ARM processor. When the utilization rate is less than the preset value, trigger the ARM processor to idle and sleep.
[0013] According to a second aspect of the present invention, an on-board fire detection system is provided, comprising: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program, which, when executed by the at least one processor, implements the on-board fire detection method based on a hybrid heterogeneous architecture as described in any of the above technical solutions.
[0014] According to a third aspect of the present invention, a computer-readable storage medium is also provided, wherein a plurality of program codes are stored, characterized in that the program codes are adapted to be loaded and run by a processor to perform the on-board fire detection method based on a hybrid heterogeneous architecture in any of the above-described technical solutions.
[0015] This application breaks with the traditional ground post-processing mode, adopting an ARM+FPGA heterogeneous architecture to integrate on-board data preprocessing and fire detection. The MercA1 model compresses the preprocessing time of a single frame and single channel to less than 100ms, and the LNN-Fire model has a single frame identification time of less than 0.05s. The overall single frame processing time is less than 1.2s, which is better than the 1.5s scanning cycle of the FY-3D satellite MERSI-II payload. Fire information can be directly broadcast to the ground handheld terminal via short messages, realizing end-to-end millisecond-level fire detection emergency response from satellite to user, achieving real-time on-board processing, and greatly improving timeliness.
[0016] A lightweight neural network LNN-Fire model is used to replace the traditional thresholding method. Combined with channel attention mechanism and band selection strategy, it automatically extracts deep features of spectral channels. The fire detection accuracy is higher than 92% and the false alarm rate is lower than 9%. Moreover, with 19-channel input, the accuracy can reach 98% and the false alarm rate is only 1.34%, which is significantly better than traditional detection methods, improving the accuracy of fire detection and reducing the false alarm rate.
[0017] The MerAcAl model deploys the module with the largest iterative computation on an FPGA to achieve parallel pipelined processing, while single-cycle operations are performed by an ARM processor, improving preprocessing speed by 4.5 times compared to traditional CPU solutions. At the same time, by dynamically adjusting the ARM frequency and voltage and linking them with the FPGA load, the model avoids wasting hardware resources, improves the overall operating efficiency of the onboard hardware platform, optimizes hardware resource utilization, and enhances processing efficiency.
[0018] The total number of input channels of the LNN-Fire model can be configured to 4, 16 or 19 channels, which can be flexibly adjusted according to the on-board hardware resources and detection accuracy requirements. The fire detection results adopt a hierarchical decision-making mechanism, which takes into account both real-time performance and detection accuracy. The local re-verification strategy for medium-confidence fire points further reduces the false alarm rate and improves the reliability of fire point information. The algorithm has high flexibility and engineering adaptability.
[0019] The hybrid heterogeneous on-board hardware platform adopts a modular board design, with each module communicating through the VPX internal bus. The board functions can be flexibly expanded according to the upgrade requirements of the satellite payload. Furthermore, the FPGA resources are not yet fully utilized. By optimizing resource configuration and parallel architecture, the computing efficiency can be further improved, and the hardware architecture has strong scalability. Attached Figure Description
[0020] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Wherein: Figure 1 This is a flowchart of a spaceborne fire detection method based on a hybrid heterogeneous architecture according to an embodiment of this application; Figure 2 This is a structural diagram of a spaceborne fire detection system based on a hybrid heterogeneous architecture according to an embodiment of this application; Figure 3 This is a screenshot of the test results according to one embodiment of this application. Detailed Implementation
[0021] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.
[0022] In the description of this application, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and may also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.
[0023] This embodiment provides an on-board fire detection method based on a hybrid heterogeneous architecture, applied to the on-board processing platform of the FY-3D satellite. This platform is an ARM+FPGA hybrid heterogeneous architecture, equipped with a MercA1 hardware acceleration model and an LNN-Fire intelligent fire detection model. Figure 1 As shown, the method includes the following steps: S1, Receive raw data collected by satellite.
[0024] The data interface module of the hybrid heterogeneous on-board hardware platform receives raw data collected by the FY-3D satellite MERSI-II via the satellite payload data link. The raw data contains detection data from 25 spectral channels, including 6 channels with a resolution of 250m and 19 channels with a resolution of 1000m. The raw data is transmitted to the data interface module at a code rate of 1.5 seconds / 3,728,628 bytes.
[0025] S2. Unpack the raw data to obtain multi-channel digital quantization value data.
[0026] The data interface module transmits the raw data to board 4 (DSP+FPGA architecture) of the data processing module via the VPX internal bus (SRIO / PCIE, switching bandwidth 1Gb / s). This board performs operations such as unpacking, frame synchronization, and data verification on the raw data, removes redundant information in the data transmission, and extracts the digital quantization (DN) data of each spectral channel to obtain multi-channel digital quantization data.
[0027] S3. Perform calibration and positioning processing on the multi-channel digital quantization data to obtain positioning and calibration data.
[0028] This step uses the MercAI model to complete the geometric positioning and calibration process. This model encapsulates the geometric positioning algorithm into a hardware-accelerated IP core (MERSI-Accelerate core), enabling collaborative computation between ARM and FPGA. The specific process is as follows: S31, the data processing module board 5 (ZYNQ chip, integrating ARM processor and FPGA processor) receives the unpacked multi-channel digital quantized value data, and the ARM processor performs floating-point operations and data scheduling for positioning and calibration, completing the operation that only needs to be performed once per cycle, including interpolation of satellite ephemeris and attitude, acquisition of external orientation parameters, etc. S32. The FPGA processor performs repetitive iterative calculations in parallel using hardware acceleration IP cores. The repetitive iterative calculations include line-of-sight vector calculation and the calculation of the intersection point of the line-of-sight axis with the Earth ellipsoid. The calculations are repeated 8192×40 times per pixel for the six 250m resolution channels and 2048×40 times per pixel for the nineteen 1000m resolution channels. The FPGA performs hardware acceleration processing on the 64×40 repetitive calculations. S33. Combine the digital elevation model (DEM) data to perform terrain correction on the geometric positioning results, calculate the intersection points on the ellipsoid and the actual intersection points of the additional DEM, and finally obtain the geographic coordinates, elevation, radiometric calibration values and other positioning and calibration data of each pixel.
[0029] In this step, the MercA1 model converts all channels into 10.75 channels with a resolution of 250m, resulting in a total preprocessing time of 1.059s, which is 4.5 times faster than the traditional CPU solution of 4.859s. The preprocessing time per frame per channel is compressed to 98.5ms, which is less than 100ms, meeting the real-time processing requirements of the satellite.
[0030] Simultaneously, this step also performs dynamic hardware resource scheduling: real-time monitoring of the DSP computing unit utilization of the FPGA acceleration core, with a preset utilization threshold of 70%; when the utilization is greater than 70%, the processing frequency and operating voltage of the ARM processor are automatically increased to improve data scheduling efficiency; when the utilization is less than 70%, the ARM processor is triggered to enter idle sleep mode to reduce power consumption and improve hardware resource utilization.
[0031] S4. Perform fire detection based on the LNN-Fire neural network model on the positioning and calibration data.
[0032] This step uses a lightweight LNN-Fire neural network model for fire detection. The model input is the localization and calibration data of key spectral channels after channel filtering, and the output is the probability value of the pixel to be judged as a fire point. The specific process is as follows: S41, Key Spectral Channel Screening The 25 raw spectral channels of the FY-3D satellite MERSI-II were screened to obtain the key spectral channels, specifically: S411. Pearson correlation coefficient analysis was used to analyze the linear correlation between 25 original spectral channels and to quantify the information redundancy between each channel. S412. Calculate the optimal index factor (OIF) for each band combination. The OIF combines the correlation between bands and the information content of a single band. The formula is as follows: Where SDi is the standard deviation of the i-th band, and Rij is the Pearson correlation coefficient between the i-th and j-th bands; S413. Select the channel combination with the least inter-band correlation and the largest single-band standard deviation as the key spectral channel. In this embodiment, the key spectral channels include 4 thermal infrared channels (channels 20, 21, 24, and 25, which are the core channels for fire detection) and 15 visible / near-infrared auxiliary channels, for a total of 19 channels as input to the LNN-Fire model.
[0033] A fire type pre-judgment branch is added to the input layer of the LNN-Fire neural network model. Based on global land cover data, the learning weights of the channel attention mechanism are automatically adjusted: for forest fires, the weight of the thermal infrared channel is increased; for grassland fires, the weight of the auxiliary features of the near-infrared channel is increased; and for urban fires, the background suppression weight of the visible light channel is increased. The fire type pre-judgment branch is added to the input layer of the LNN-Fire model. By matching global land cover data (forest / grassland / farmland / city) with satellite geographic coordinates, the weight learning strategy of the channel attention mechanism is automatically adjusted: for example, for forest fires, the weight of the thermal infrared channel (20 / 21 / 24 / 25) is strengthened; for grassland fires, the weight of the auxiliary features of the near-infrared channel is strengthened; and for urban fires, the background suppression weight of the visible light channel is increased.
[0034] The specific calculation process for fire point feature extraction and probability output of the S42 and LNN-Fire models is as follows: S421, The input layer receives positioning and calibration data from 19 key spectral channels, with a data dimension of 1×19; S422. The first channel attention mechanism is applied to key spectral channel data: the interaction relationship between channels is learned through two linear fully connected layers, and the weight value of each channel (weight range 0~1) is generated by the Sigmoid function. The weight value is multiplied and weighted with the localization and calibration data of the corresponding channel to suppress the features of information redundant channels and strengthen the fire point related features to obtain the purified spectral channel features with the dimension maintained at 1×19. S423. The channel feature linear expansion layer enhances the information of the purified spectral channel features: the feature dimension is expanded to 1×64 through linear transformation to fully explore the fine-grained features of a single channel; S424. The N-layer feature extraction module (N=3 in this embodiment) captures deep features: each feature extraction module includes a Batch Norm layer and a Leaky ReLU activation function in sequence. Through 3 layers of feature extraction, the inter-channel features (correlation features of different spectral channels) and intra-channel features (spectral features of a single channel) of the spectral data are fully captured to obtain deep features with a dimension of 1×64; S425. The second channel attention mechanism is applied to deep features: the channel importance weights are generated in the same way as the first channel attention mechanism, and the deep features are weighted and focused to strengthen the core deep features of the fire point, resulting in focused features with a dimension of 1×64. S426. The linear classification layer classifies the focused features: the focused features are mapped to a binary classification space (fire point / non-fire point) through linear transformation, and the probability value of the pixel to be judged as a fire point is output by the Sigmoid function, with the probability value ranging from 0 to 1.
[0035] In this embodiment, the processing performance of the LNN-Fire model with 19 input channels is as follows: accuracy 98%, false alarm rate 1.34%, computational load 0.69 G FLOPs, runtime 0.046 s, computational efficiency 15.00 G FLOPS, and single frame identification time less than 0.05 s, which meets the real-time processing requirements of the satellite.
[0036] S5. Classification and broadcasting of fire detection results.
[0037] Based on the fire probability values output by the LNN-Fire model, a hierarchical decision-making mechanism is executed. This embodiment presets probability thresholds: high confidence (probability ≥ 0.9), medium confidence (0.5 ≤ probability < 0.9), and low confidence (probability < 0.5). The specific processing is as follows: S51, High Confidence Fire Point The fire point is directly marked as "confirmed fire point". The data interface module encapsulates the geographical coordinates, probability value, resolution and other information of the fire point into a short message, which is broadcast to the ground handheld terminal in real time through the satellite broadcast channel to realize the real-time push of fire point information.
[0038] S52, Medium Confidence Firepoint Trigger a partial re-verification process to ensure detection accuracy, specifically as follows: S521. Re-execute the LNN-Fire model inference on the mid-confidence pixel and its 8 neighboring pixels (a total of 9 pixels) to obtain 9 fire probability values. Use the arithmetic mean method to fuse the 9 output probabilities to obtain the fused probability value. S522. Combining the spatial context of nearby confirmed fire points, **Conditional Random Field (CRF)** is used for post-processing to optimize the boundaries of fire point pixels and eliminate spatially isolated suspected fire points. S523. If the probability value after fusion is ≥0.8, it is marked as "confirmed fire point" and broadcast; if the probability value after fusion is <0.8, it is marked as a false alarm sample and written to the local log on the satellite for subsequent model iteration and optimization.
[0039] S53, Low-confidence fire point Directly mark as non-fire points, cache relevant data locally, do not broadcast, and cached data can be used as negative samples for model training.
[0040] S6. Fire detection results are broadcast to the ground terminal. The onboard processing platform will broadcast the "confirmed fire point" information after hierarchical decision-making via short message through the onboard antenna. Ground users within the satellite coverage area can receive real-time fire point information through handheld terminals, realizing end-to-end fire point detection and emergency response from satellite to user.
[0041] In the local re-verification of medium-confidence fire points, multi-frame fusion in the temporal dimension can be added: combining the orbital scanning characteristics of the FY-3D satellite, fire point probability fusion is performed on three consecutive frames of data from the same geographical area. If all three consecutive frames are medium-confidence fire points, they are directly determined as confirmed fire points; if they appear only in a single frame, they are determined as false alarms. At the same time, combined with spatial context (such as whether the fire point is distributed along the terrain slope and wind direction), CRF post-processing is used to optimize the spatial boundary of the fire point. The "single-frame single-pixel detection" is upgraded to spatiotemporal joint detection, which improves the accuracy of fire point detection by utilizing the characteristics of continuous satellite scanning, which is different from the traditional single-frame detection scheme.
[0042] It should be noted that in this embodiment, the fire probability can also be predicted based on the type of the detection area, and the FPGA can be dynamically activated or deactivated in blocks based on the prediction results. For example, based on the fire probability prediction of the scanning area of the MERSI-II payload (such as high fire probability in forest / grassland areas and low fire probability in ocean / desert areas), the FPGA acceleration cores can be dynamically activated / deactivated in blocks. For example, for high-probability fire areas, all 64×40 operation blocks of the FPGA are activated; for low-probability areas, only 32×40 operation blocks are activated, and the remaining resources enter low-power mode.
[0043] By linking hardware resource scheduling with the characteristics of fires in geographical areas, a more refined balance between power consumption and efficiency can be achieved than "simple utilization scheduling," adapting to the limited power supply resources on the satellite.
[0044] A dynamic pruning module that is aware of computing power can also be added to the LNN-Fire model to monitor the remaining computing power of the onboard hardware (ARM / FPGA utilization, memory usage) in real time. If the computing power is insufficient (e.g., processing multiple frames of data simultaneously), the hidden layer neurons of the feature extraction module are automatically pruned (e.g., reduced from 3 layers to 2 layers), and the dimension of the fully connected layers of the channel attention mechanism is reduced. If the computing power is sufficient, the complete model structure is restored. This achieves dynamic matching between the model structure and the real-time computing power on the satellite, which is different from the traditional "fixed-structure lightweight model" and represents an innovative application of neural networks in embedded scenarios on satellites.
[0045] The on-board fire detection method based on a hybrid heterogeneous architecture in this embodiment has a total processing time of 1.105s per frame, which is less than 1.2s and far better than the 1.5s scanning cycle of the FY-3D satellite MERSI-II payload, fully meeting the on-orbit real-time processing requirements.
[0046] The existing Fengyun satellite fire emergency service still uses the traditional model of "satellite scanning - data downlink - ground reception - centralized processing - product production - information dissemination," which reduces the timeliness and reach of the emergency service. To further improve the real-time performance of the fire emergency service, this system adopts an on-board processing approach, acquiring fire information at the satellite end and broadcasting it directly in real time via short messages. Users only need a single ground-based handheld terminal within its coverage area to obtain real-time fire information.
[0047] In this system, the on-board portion employs a hybrid heterogeneous on-board processing hardware platform to accelerate the preprocessing stage based on a rigorous geometric positioning algorithm, thereby meeting processing time requirements. Subsequently, a neural network-based fire detection algorithm is used to complete the detection of fire points, and the data is broadcast through a distribution channel.
[0048] This embodiment provides an on-board fire detection system applied to the FY-3D satellite to implement the on-board fire detection method based on a hybrid heterogeneous architecture described in Embodiment 1. This system is a hybrid heterogeneous on-board hardware platform, and its structural diagram is shown below. Figure 2 As shown.
[0049] The hardware platform includes a power supply module, a data interface module, a data processing module, an intelligent processing module, and a baseboard. Each module is connected to the baseboard slot as an independent board. Communication between boards is achieved via a VPX internal bus, with a bus exchange bandwidth not exceeding 1Gb / s (SRIO / PCIE). The power supply module (board 1 in the diagram) and its backup (board 2 in the diagram) provide power to the entire hardware platform. The data interface module (board 3 in the diagram) serves as the data transceiver interface for the entire platform, directly receiving raw data and related auxiliary data from the Fengyun satellite payload and outputting the processing results to the external data transmission section. The intelligent processing module (board 6 in the diagram) is based on the ZYNQ+AGX architecture, employing a single-board master / backup configuration. It uses the positioning and calibration data input from the data processing module to perform intelligent fire detection and outputs the results to the data interface module.
[0050] The data processing module consists of two boards (board 4 and board 5 in the diagram). Board 4 adopts a DSP+FPGA architecture with a single-board master / slave configuration, primarily responsible for unpacking the MERSI-II payload data from the FY-3D satellite. Board 5 mainly performs preprocessing tasks such as calibration and positioning of the unpacked MERSI-II payload data. Its core processor can be a ZYNQ chip, and it runs a Linux operating system. The ZYNQ chip integrates a high-speed ARM core processor and an FPGA processor. The ARM processor handles floating-point operations and data scheduling for positioning and calibration, while the FPGA processor serves as a digital circuit hardware acceleration module.
[0051] For the positioning algorithm, a rigorous geometric positioning model is adopted, which features high positioning accuracy and independence from control points. First, a detector vector model on the focal plane is established based on interior orientation elements. The resulting line-of-sight vector in the instrument coordinate system, along with the instrument's installation azimuth relative to the satellite, DEM reference, and scan angle, serve as static parameters. Then, external orientation parameters at the sampling time are obtained by interpolating the satellite ephemeris and attitude, transforming the line-of-sight vector from the satellite body coordinate system to the geocentric rotating coordinate system. The orbital coordinate system involved is related to Euler angles, and the geocentric inertial coordinate system can be associated with the satellite coordinate system through specified quaternions. Finally, the intersection points on the ellipsoid and the actual intersection points of the attached DEM are calculated. In addition to the pre-calculated static parameters, the positioning process typically requires performing a series of time-varying transformations pixel by pixel to ultimately obtain the geographic coordinates, elevation, and related angles of the topographically corrected pixels.
[0052] The extensive iterative computations in localization algorithms result in high computational complexity and consume most of the resources in the preprocessing stage. To reduce resource consumption and improve data processing efficiency, a hybrid heterogeneous architecture hardware acceleration model for localization algorithms (MerAcAI model) is designed based on the aforementioned algorithms. This model leverages the parallel and pipelined processing capabilities of FPGAs for repetitive computations, deploying the modules with the highest iterative computational load—the line-of-sight calculation and the line-of-sight intersection calculation—onto the FPGA to achieve hardware acceleration.
[0053] FY-3D / MERSI-II generates 25 channels of data within a 1.5s scan cycle: 6 channels with 250m resolution require 8192×40 repetitions per pixel, and 19 channels with 1000m resolution require 2048×40 repetitions per pixel. In the MERSI-II model, 64×40 repetitions are accelerated by the FPGA, while the operation requiring only one repetition per cycle is executed by the ARM. Simultaneously, the accelerated model is encapsulated as a hardware acceleration IP core (MERSI-Accelerate core) and accompanying driver, completing hardware implementation.
[0054] During the experimental testing, the actual code rate of the payload was strictly maintained at 1.5 seconds per 3,728,628 bytes. The test data was sent through a data simulation source to the platform's data interface module, which then transmitted it to the data processing module (ZYNQ) and intelligent processing module via the high-speed bus interface on the baseboard. The pre-processed data and fire point data results were returned to the test equipment through the data interface module.
[0055] To compare the functional accuracy and performance metrics of the MerAcAI model results and the results of traditional CPU processing, the preprocessing results under both modes were tested. Screenshots of the results are shown below. Figure 3 As shown in the figure, the hybrid heterogeneous method meets the functional requirements. The real-time processing performance indicators are shown in the table below, demonstrating that this model successfully solves the problem of on-orbit real-time processing that a single CPU cannot handle.
[0056] Traditional thresholding methods for fire detection typically require meticulously designed detection rules tailored to specific instruments, regions, and seasons based on empirical knowledge. This approach suffers from poor transferability and requires improvement in detection accuracy, making it unsuitable for real-time spaceborne fire detection. Therefore, a lightweight neural network-based intelligent fire detection model (LNN-Fire model) is designed. This model automatically extracts effective features of fire points based on the correlation of DN values between single channels and between channels, thereby achieving efficient and accurate identification.
[0057] For the rich bands of the FY-3D MERSI-II, in addition to the conventional thermal infrared channels (20, 21, 24, and 25) used for fire detection, other visible and near-infrared channels provide auxiliary information. To effectively evaluate the information contribution and combination potential of each band, the Pearson correlation coefficient was used to analyze the linearity between spectral channels, obtaining the optimal index factor (OIF) for individual bands and between bands. The OIF combines the correlation between bands and the information content of a single band. Theoretically, the smaller the correlation between bands and the larger the standard deviation of a single band, the richer the information content of the band combination.
[0058] After channel analysis and selection, the model accepts the chosen important channels as input. When identifying fire points, the proposed LNN-Fire model first utilizes a linear expansion layer of channel features to obtain information-rich features. Then, the model uses an N-layer feature extraction module to fully capture inter-channel and intra-channel features of the data. Finally, for the extracted high-level features, a linear layer and a classification function are used to output the probability that the pixel to be judged is a fire point. Furthermore, to more effectively learn important channel features, the model incorporates a channel attention mechanism in both the initial input layer and the final feature extraction layer. It uses two fully connected linear layers to learn the interaction relationships between channels, then uses a sigmoid function to obtain the importance weights of each channel, and finally performs a dot product operation between these weights and the input features to obtain weighted features.
[0059] This embodiment uses 25 channels of FY-3D MERSI-II L1 data. The training set contains 375 fire point samples and 375 non-fire point samples, while the test set contains 150 fire point samples and 150 non-fire point samples. The table below summarizes the detection performance of the LNN-Fire model with different channel combinations. Accuracy and false alarm rate are calculated using the ratio of detected / missed pixels to the actual number of fire point pixels.
[0060] Table 2 Detection results of the LNN-Fire model Accuracy (a core metric in fire detection, referring to the proportion of correctly detected fire points plus non-fire point samples out of the total sample). FAR, or False Alarm Rate, is the proportion of non-fire points that are mistakenly identified as fire points. It is a key performance indicator for fire detection. FLOPs, or Floating-Point Operations, represent the computational complexity of model inference. The smaller the value, the lighter the model and the more suitable it is for onboard computing power. Running Time (the time taken for a single frame / single fire detection, usually measured in seconds (s) or milliseconds (ms), directly reflects the onboard real-time processing performance). FLOPS stands for Floating-Point Operations Per Second, measured in GB / FLOPS. It represents the computational efficiency of the hardware and the model; the higher the value, the faster the processing speed.
[0061] Based on MERSI-II payload data from the FY-3D satellite, this invention proposes an integrated on-board processing solution, encompassing a hardware-accelerated positioning algorithm model (MerAcAI) and an intelligent fire detection model (LNN-Fire), enabling real-time on-orbit data preprocessing and fire product generation.
[0062] Hardware acceleration of the geometric localization algorithm for MERSI-II data using FPGA achieves a 4.5-fold increase in single-frame processing speed compared to traditional CPUs, while maintaining positioning accuracy. The single-channel processing time is reduced to less than 100ms. The LNN-Fire intelligent fire detection model demonstrates superior performance in testing, achieving an accuracy of up to 92%, a false alarm rate of 9%, and a single-frame identification time of less than 0.05s, significantly outperforming traditional thresholding methods. Overall, the total processing time per frame is less than 1.2s, better than the payload's 1.5s scan cycle, fully meeting the requirements for real-time on-orbit processing. Furthermore, the localization algorithm used in this method is based on a rigorous geometric model; introducing more efficient localization methods could further improve processing speed. Simultaneously, FPGA resources are not yet fully utilized; further optimization of resource allocation and parallel architecture allows for improvements in computational efficiency.
[0063] Another aspect of this application provides a computer-readable storage medium.
[0064] In one embodiment of a computer-readable storage medium according to this application, the computer-readable storage medium can be configured to store a program that performs the on-board fire detection method based on a hybrid heterogeneous architecture as described in the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described on-board fire detection method based on a hybrid heterogeneous architecture. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.
[0065] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application and therefore will also fall within the protection scope of this application.
[0066] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0067] The technical solution of this application has been described above with reference to one embodiment shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.
Claims
1. A method for detecting on-board fire points based on a hybrid heterogeneous architecture, characterized in that, include: Receive raw data collected by satellite; The original data is unpacked to obtain multi-channel digital quantized value data; The multi-channel digital quantized value data is subjected to calibration and positioning processing. An ARM processor is used to perform single-cycle operations to handle floating-point operations and data scheduling for positioning and calibration. An FPGA processor is used to perform repetitive iterative operations in parallel through hardware acceleration IP cores. The repetitive iterative operations include line-of-sight vector calculation and line-of-sight intersection with the Earth ellipsoid to complete the geometric positioning and calibration processing and obtain positioning and calibration data. Fire detection based on the LNN-Fire neural network model is performed on the positioning and calibration data; The fire detection results are broadcast to the ground terminal.
2. The method according to claim 1, characterized in that, The fire detection process of the LNN-Fire neural network model includes: The input layer receives data from key spectral channels after channel filtering, the key channels including thermal infrared channels and other auxiliary channels; The first channel attention mechanism is applied to the data of the key spectral channel, and channel weights are generated by the Sigmoid function and then multiplied and weighted with the data of the corresponding key spectral channel to obtain the purified spectral channel features. The channel feature linear extension layer enhances the information of the purified spectral channel data; The N-layer feature extraction module captures the deep features of the purified spectral channel data, including inter-channel features and intra-channel features; The second channel attention mechanism is applied to the deep features, and channel importance weights are generated by the Sigmoid function. These weights are then multiplied and weighted with the deep features to obtain the focused features. The linear classification layer outputs the probability value of the fire point based on the focusing features.
3. The method according to claim 2, characterized in that, The channel selection process of the LNN-Fire neural network model includes: Pearson correlation coefficient was used to analyze the linear correlation between multiple raw spectral channels; Calculate the optimal index factor for each band combination, select the channel combination with the lowest inter-band correlation and the largest single-band standard deviation as the key spectral channel, and use the data of the key spectral channel as the input of the neural network LNN-Fire model.
4. The method according to claim 2, characterized in that, The key spectral channels include 4 thermal infrared channels and / or auxiliary channels selected from the remaining 21 visible / near infrared channels. The total number of input channels of the neural network LNN-Fire model can be configured to 4, 16 or 19 channels. A fire type pre-judgment branch is added to the input layer of the LNN-Fire neural network model. Based on the global land cover data type, the learning weights of the channel attention mechanism are automatically adjusted: for forest fires, the weight of the thermal infrared channel is increased; for grassland fires, the weight of the auxiliary features of the near-infrared channel is increased; and for urban fires, the background suppression weight of the visible light channel is increased.
5. The method according to claim 1, characterized in that, The geometric positioning and calibration process is completed using the MerAcAl model, which encapsulates the geometric positioning algorithm into a hardware-accelerated IP core. The fire situation is predicted based on the type of the detection area, and the FPGA is dynamically activated or deactivated in blocks based on the prediction results.
6. The method according to claim 2, characterized in that, Hierarchical decision-making is performed based on the probability value of fire points: high confidence is broadcast immediately, medium confidence triggers local re-verification, and low confidence is cached locally.
7. The method according to claim 6, characterized in that, The local re-verification includes: Re-execute LNN-Fire inference on the medium-confidence pixel and its neighboring pixels, fusing the output probabilities 9 times. By combining the spatial context of neighboring confirmed fire points, a Conditional Random Field (CRF) post-processing method is used to optimize the boundary. If the verification passes, a "Confirmed Fire Point" message will be broadcast; if it fails, it will be marked as a false alarm and written to the local log; and / or In the local re-verification with medium confidence, multi-frame fusion with a time dimension is added: fire point probability fusion is performed on continuous multi-frame data of the same geographical area, and if the continuous multi-frame data are all medium confidence fire points, they are determined to be confirmed fire points.
8. The method according to claim 1, characterized in that, The frequency and operating voltage of the ARM processor are dynamically adjusted according to the load of the hardware acceleration IP core. Monitor the utilization rate of the DSP computing unit of the FPGA acceleration core. When the utilization rate is greater than a preset value, increase the processing frequency of the ARM processor. When the utilization rate is less than the preset value, trigger the ARM processor to idle and sleep.
9. A satellite fire detection system, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program, which, when executed by the at least one processor, implements the on-board fire detection method based on a hybrid heterogeneous architecture as described in any one of claims 1 to 8.
10. A computer-readable storage medium storing a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by a processor to perform the on-board fire detection method based on a hybrid heterogeneous architecture as described in any one of claims 1 to 8.