An image fusion-based edge-cloud collaborative intelligent fire monitoring system

Abstract

The present invention discloses an image fusion end-edge-cloud collaborative intelligent fire monitoring system, which relates to the field of intelligent fire protection technology. The system consists of several functional modules, including: a multimodal image fusion module, which generates a dynamic scanning priority map based on prior data, uses a dual-light fusion algorithm to distinguish natural heat sources from abnormal fire conditions, divides the scanning area according to the thermal risk level, and dynamically adjusts the thermal imaging resolution; a distributed edge computing module, which synchronizes cross-modal data in time and space through a multimodal feature alignment network, and dynamically allocates CUDA cores and CPU resources through adaptive computing scheduling; an improved artificial bee colony algorithm is used to dynamically allocate the bandwidth of multi-sensor data streams through a time-sharing multiplexing protocol to perform three-dimensional path planning; and an end-edge-cloud collaborative decision-making module builds a federated learning-driven model sharing network, in which each edge node trains a lightweight YOLOv5s pruning model based on local data.

Description

Technical Field

[0001] The present invention relates to the field of smart fire protection technology, and specifically to an image fusion end-edge-cloud collaborative smart fire protection fire monitoring system. Background Art

[0002] In the forest fire prevention and control system, fire monitoring is a core link. Its importance is not only reflected in the accuracy of real-time perception of fire intensity, but also in the ability to deeply analyze the dynamic evolution laws of fire scenes. Especially in large-scale forest fire scenes, the spatiotemporal evolution of fire conditions presents significant multidimensional coupling characteristics: on the one hand, the burning intensity in local areas is directly controlled by physical conditions such as vegetation type, terrain slope, and combustible load; on the other hand, the spread of fire in adjacent areas forms complex nonlinear correlations through mechanisms such as thermal radiation transfer, fire tornado generation, and flying fire propagation. This dual driving mechanism makes it difficult for traditional monitoring methods based on a single area to accurately predict the development trend of the fire scene, and it is urgent to build a dynamic monitoring model that integrates spatiotemporal correlation characteristics.

[0003] Existing technologies rely on data collected by satellite remote sensing, drone inspections, ground sensors and other equipment, which differ in temporal and spatial resolution, coordinate system and data format. How to achieve efficient collaboration remains a technical difficulty. The evolution of fire is affected by multiple factors such as vegetation type, terrain slope, wind speed and direction, and the mechanisms such as thermal radiation transfer and flying fire spread between adjacent areas are highly nonlinear. Traditional statistical models are difficult to accurately characterize their temporal and spatial correlations. Existing systems mostly rely on centralized cloud processing, which has data transmission delays and computing power bottlenecks, making it difficult to meet the stringent requirements of forest fire monitoring for low latency and high reliability. Summary of the Invention

[0004] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0005] The multimodal image fusion module generates a dynamic scanning priority map based on prior data, uses a dual-light fusion algorithm to distinguish natural heat sources from abnormal fire conditions, divides the scanning area according to the thermal risk level, and dynamically adjusts the thermal imaging resolution.

[0006] The distributed edge computing module uses a multimodal feature alignment network to synchronize cross-modal data in time and space, and dynamically allocates CUDA cores and CPU resources through adaptive computing scheduling. It also uses an improved artificial bee colony algorithm to dynamically allocate bandwidth for multi-sensor data streams through a time-sharing multiplexing protocol for three-dimensional path planning.

[0007] The device-edge-cloud collaborative decision-making module builds a federated learning-driven model sharing network. Each edge node trains a lightweight YOLOv5s pruning model based on local data, generates a global optimization model and synchronizes it in reverse, performs dynamic networking, and adjusts the density of drone formations in real time.

[0008] Furthermore, the process of generating the dynamic scanning priority map is as follows:

[0009] Based on the historical data of the fire point, including fire data, environmental information, and geographic location information, the system obtains the prior data and classifies the prior data into text information, image annotations, and sensor data. The prior data is structured through optical character recognition (OCR) of text information, image annotations, and sensor data, and multimodal fusion technology is used to integrate text, image, and sensor data.

[0010] YOLOv5s outputs bounding box coordinates to locate the fire point, Mask R-CNN segments the burned area, quantifies the burned area and fire line shape, rotates the fire satellite imagery, simulates smoke, and cleans and aligns the sensor data. Keywords are extracted from the reports extracted by optical character recognition (OCR) and converted into numerical vectors using TF-IDF or word embedding. ResNet is used to extract high-level features of the burned area, and the burn intensity is calculated based on the masked area.

[0011] Furthermore, the process of distinguishing between natural heat sources and abnormal fire conditions is as follows:

[0012] Collect visible light channel and thermal imaging channel data, and perform color space conversion on the visible light channel using a dehazing algorithm and image enhancement technology. Use the YOLOv5s model to detect flame or smoke features and extract morphological features.

[0013] The thermal imaging channel uses NSCT non-subsampled contourlet transform to extract temperature gradients and hot spot distribution; pulse-coupled neural networks are used to fuse high-frequency features, and adaptive fuzzy logic algorithms are used to process low-frequency information;

[0014] By utilizing dual-light fusion, the weights are dynamically adjusted according to the scene, and the DAF-Net domain adaptive dual-branch feature decomposition and fusion network is adopted. The distribution difference between infrared and visible light images is reduced through the multi-core maximum mean difference, and the Transformer-CNN structure is used to retain cross-modal features.

[0015] Furthermore, the process of dividing the scanning area according to the thermal risk level is as follows:

[0016] Random forests are used to score and categorize regional thermal risks. High-definition drone cameras capture images of the area, and NSCT is used to extract temperature gradients and hotspot distribution. Weights are dynamically adjusted based on the scenario, using the EMMA framework to preserve cross-modal features. Natural heat sources are excluded using temperature thresholds and morphological features, and the classification model is dynamically updated based on historical false positives.

[0017] Suspicious areas are locally magnified using AI algorithms. The drone's onboard NPU analyzes thermal imaging data in real time. If a temperature anomaly is detected, high-resolution mode is automatically triggered. The cloud uses the NSCT fusion model to process complex scenarios and dynamically optimize the resolution strategy. Otherwise, low-resolution pattern recognition continues.

[0018] The area is divided into grids, and the mean of each feature within the grid is extracted. The average humidity change rate over the past seven days is calculated, and the fire frequency over the same period of the past three years is also counted. If a fire occurred in the grid within 30 days, a binary feature is added; otherwise, it is 0. Based on historical data, if a fire occurred in the grid during the observation period, it is marked as high risk = 1, otherwise it is low risk = 0. Semi-supervised enhancement is performed, and K-Means clustering is used to generate pseudo labels for unlabeled areas. The XGBoost model is used to identify key risk drivers by sorting the built-in feature importance.

[0019] Furthermore, the process of performing spatiotemporal synchronization of cross-modal data is as follows:

[0020] Through timestamp calibration, different sensors are synchronized in time and their coordinate systems are aligned to the same reference system.

[0021] Furthermore, the process of dynamic allocation of CUDA cores and CPU resources is as follows:

[0022] Dynamically allocate GPU and CPU resources, prioritize real-time image processing tasks for fire monitoring, adjust resource allocation ratios based on real-time task requirements, and dynamically adjust scheduling strategies when sensor data streams fluctuate or network bandwidth changes.

[0023] Dynamic resource allocation strategy:

[0024] Use hardware performance counters to collect GPU and CPU load, temperature, and memory usage in real time; dynamically adjust resource allocation based on task queue status.

[0025] Furthermore, the process of dynamically allocating bandwidth to the multi-sensor data stream is as follows:

[0026] Collect bandwidth usage, latency, and packet loss rates for each sensor data stream, monitor network load between edge nodes and the cloud, extract thermal imaging video resolution and frame rate, and the characteristics and update frequency of temperature and humidity sensor data streams; identify thermal imaging data and ambient temperature data for fire monitoring; and classify sensor data streams based on traffic business models.

[0027] By dynamically assigning weights based on confidence or task requirements, the LSTM model is used to predict the data flow requirements of each sensor in the future. Combined with the traffic prediction model, the network load trend is evaluated and the bandwidth requirements of each sensor data flow are calculated.

[0028] Furthermore, the process of generating the three-dimensional path is as follows:

[0029] The starting and ending points are fixed, and the intermediate points are optimized using the ABC algorithm; an initial path is randomly generated, and the best path is selected based on the fitness value, and the location of the neighboring points is updated; other drones select paths based on probability for local search; if the path is not improved, a new path is randomly generated; spline interpolation is used to eliminate jagged paths; combined with lidar or visual sensors, the obstacle map is dynamically updated and path replanning is triggered; and at the same time, the coverage of high-risk areas is optimized during path planning.

[0030] Furthermore, the process of building a federated learning-driven model sharing network is as follows:

[0031] After obtaining the spatiotemporal alignment of multimodal data, the regional master node receives the gradients of all edge nodes, and then uses the federated averaging algorithm in federated learning to perform weighted aggregation to generate a global model. Weights are assigned according to task priorities, and hyperparameters such as the learning rate and regularization coefficient of the global model are adjusted based on the aggregation results.

[0032] Furthermore, the process of adjusting the density of the drone formation in real time is as follows:

[0033] A wireless mesh network is used, with each drone acting as a node, supporting multi-hop relay transmission. Based on a clustering algorithm, drones are divided into multiple clusters. A designated master node in each cluster is responsible for channel allocation. Scanning strategies are adjusted based on thermal risk scores, and the formation shape is dynamically adjusted by setting a navigator and a virtual navigation point. The drone's behavior rules are adjusted in real time based on environmental changes. At the same time, the cloud uses the NSCT fusion model to handle complex scenarios and dynamically optimize resolution strategies.

[0034] The present invention provides an image-fusion device-edge-cloud collaborative intelligent fire monitoring system, which has the following beneficial effects:

[0035] (1) The present invention generates a structured dynamic scanning priority map by utilizing multimodal data such as historical fire data, environmental information, and geographic location information, and through technologies such as OCR recognition and Mask R-CNN segmentation of burning areas. This improves the accuracy of fire point positioning and dynamically adjusts the risk level based on real-time meteorological data. High-risk areas are marked as priority scanning areas, while low-risk areas are scanned less frequently to save resources.

[0036] (2) The present invention effectively distinguishes natural heat sources from abnormal fire conditions through data collection and processing of visible light channels and thermal imaging channels, combined with the DAF-Net domain adaptive dual-branch feature decomposition and fusion network; adopts the Transformer-CNN structure to retain cross-modal features, and dynamically updates the classification model based on historical false alarm or missed alarm data, thereby ensuring the ability to capture small fire sources and reducing the false alarm rate.

[0037] (3) The present invention realizes spatiotemporal synchronization of cross-modal data through a multimodal feature alignment network, combines the dynamic allocation of CUDA cores and CPU resources, prioritizes the real-time image processing tasks of fire monitoring, and ensures low-latency response; dynamically adjusts the scheduling strategy in scenarios such as sensor data stream fluctuations and network bandwidth changes to avoid single-point resource bottlenecks; adopts an improved artificial bee colony algorithm for three-dimensional path planning, comprehensively considers factors such as path length, terrain cost, and energy consumption, and realizes the optimal path selection of the UAV formation, thereby improving coverage efficiency and reducing energy consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the system flow of the present invention. DETAILED DESCRIPTION

[0039] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0040] Example

[0041] See also Figure 1 The present application provides an image fusion device-edge-cloud collaborative intelligent fire monitoring system, which includes:

[0042] The multimodal image fusion module generates a dynamic scanning priority map based on prior data, uses a dual-light fusion algorithm to distinguish natural heat sources from abnormal fire conditions, divides the scanning area according to the thermal risk level, and dynamically adjusts the thermal imaging resolution.

[0043] Prior data:

[0044] Based on historical data of fire points, including fire data, environmental information, and geographic location information, historical fire data such as spatial distribution, burning intensity, and spread rate are combined with vegetation type, environmental information such as the flammability of pine trees, and geographic location information such as the heat retention characteristics of canyons or steep slopes. This prior data is then further classified into text information, image annotations, and sensor data to construct a thermal risk assessment model.

[0045] Historical fire data: including fire point distribution, combustion frequency, fire spread speed, etc.;

[0046] Environmental information: vegetation coverage, wind speed and direction, humidity, terrain elevation, etc.;

[0047] Geographic location information: Risk differences in different scenarios such as mountainous areas, forests, and industrial areas, such as high-risk areas around power transmission lines;

[0048] Prior data can be obtained through fire statistics released by fire or emergency management departments in various countries. For example, the Fire and Rescue Bureau of the Ministry of Emergency Management of China regularly publishes residential fire data, including fire causes, regional distribution, and casualty statistics. The U.S. National Interagency Fire Center provides annual records of fire area, number of fires, and rescue resource deployment, covering both wildfires and urban fires. Data can also be obtained through academic research and open databases.

[0049] Generate a dynamic scan priority map:

[0050] Structure the prior data through OCR recognition of text information, image annotation and sensor data, and use multimodal fusion technology to integrate text, image and sensor data;

[0051] Scanned fire reports, historical archive images, and on-site photos contain text information, such as fire start time, coordinates, and burned area. Use Tesseract OCR to detect text areas in images and generate text coordinate frames. Use pre-trained models like BERT to extract key fields and map them to structured fields. Correct OCR errors by integrating with a fire domain knowledge base.

[0052] YOLOv5s outputs bounding box coordinates to locate the fire point. Mask R-CNN segments the burned area, quantifies the burned area and fire line shape, rotates the fire satellite imagery, and simulates smoke to improve the model's robustness in low-visibility scenarios. Sensor data is also cleaned and aligned. Keywords such as "wind direction northeast" and "combustible material density 0.5 kg / m²" are extracted from the OCR-extracted report and converted into numerical vectors using TF-IDF or word embedding. ResNet is used to extract high-level features of the burned area, such as flame texture and smoke diffusion pattern. Burning intensity is calculated using the masked area. The fire point coordinates, satellite imagery, and sensor location are unified into the same projected coordinate system. The time series of each modality data are aligned based on the time of fire occurrence. The code is as follows:

[0053] # Pseudo code: Multimodal fusion based on attention mechanism

[0054] class MultimodalFusion(nn.Module):

[0055] def __init__(self):

[0056] super().__init__()

[0057] self.text_encoder = BertModel.from_pretrained('bert-base') #text encoding

[0058] self.image_encoder = ResNet50(pretrained=True) # Image encoding

[0059] self.sensor_lstm = nn.LSTM(input_size=5, hidden_size=128) #Sensor timing encoding

[0060] # Cross-modal attention layer

[0061] self.cross_attn = nn.MultiheadAttention(embed_dim=256, num_heads=4)

[0062] def forward(self, text, image, sensor):

[0063] text_feat = self.text_encoder(text).last_hidden_state[:,0,:]# [CLS] token

[0064] image_feat = self.image_encoder(image).flatten(1)

[0065] sensor_feat, _ = self.sensor_lstm(sensor)

[0066] # Concatenate features and calculate attention weights

[0067] combined = torch.cat([text_feat, image_feat, sensor_feat],dim=1)

[0068] attn_output, _ = self.cross_attn(combined, combined,combined)

[0069] return attn_output

[0070] Distinguishing between natural heat sources and abnormal fire conditions:

[0071] Data is collected separately from visible light and thermal imaging channels. The visible light channel uses a high-definition camera, with features like 30x optical zoom and fog-penetrating capabilities, to capture details such as flame shape, smoke outlines, and color changes. During the day, it can identify a fire of 1 x 3 square meters within 15 kilometers. Thermal imaging uses an uncooled vanadium oxide detector with a resolution of 640 x 512 and an operating wavelength of 8 to 14 μm. It can penetrate smoke and haze to detect temperature anomalies as small as 0.1°C. For example, in a dual-spectrum system for power transmission lines, thermal imaging can locate a fire source of 1 square meter within a 3-kilometer range.

[0072] And preprocess the collected visible light channel and thermal imaging channel:

[0073] The visible light channel uses a dehazing algorithm and image enhancement technology to convert the HSV to YUV color space to improve flame contrast. The YOLOv5s model is used to detect flames and smoke and extract morphological features such as flame vibration frequency and smoke diffusion speed.

[0074] The thermal imaging channel uses NSCT non-subsampled contourlet transform to extract temperature gradients and hot spot distribution; pulse-coupled neural networks are used to fuse high-frequency features, and adaptive fuzzy logic algorithms are used to process low-frequency information;

[0075] NSCT decomposition: multi-scale and multi-directional feature extraction;

[0076] NSCT is a multi-scale geometric analysis tool that implements image decomposition through a non-subsampled pyramid and a non-subsampled directional filter bank. In thermal imaging processing, NSCT has the following functions:

[0077] Perform thermal imaging preprocessing by inputting the original image of the thermal imaging channel and normalizing the image, such as mapping the temperature value to the range of [0,1]. Use Gaussian filtering or median filtering to eliminate noise. Perform NSCT multi-scale decomposition to decompose the thermal imaging image into low-frequency sub-bands and high-frequency sub-bands according to the scale.

[0078] Low-frequency subband: contains global temperature distribution information, such as the overall temperature gradient;

[0079] High-frequency sub-band: contains local detail information, such as hot spot edges and rapidly changing temperature areas;

[0080] Perform multi-directional decomposition on each high-frequency sub-band, such as 4 directions, to capture the hot spot texture features in different directions;

[0081] The temperature gradient is calculated through low-frequency coefficients, such as the temperature difference between adjacent pixels, reflecting the overall temperature change trend of thermal imaging; the hot spot distribution is extracted through high-frequency coefficients, such as the boundary, shape and density of local high-temperature areas, to extract the temperature gradient and hot spot distribution;

[0082] PCNN fuses high-frequency features to enhance hot spot details. The high-frequency subbands decomposed by NSCT are input into PCNN. PCNN enhances the contrast between hot spot edges and details through pulse coupling between neurons. The activation threshold of neurons is dynamically adjusted to highlight high-frequency areas. Multi-directional high-frequency subbands are fused to generate clearer hot spot edge information. The enhanced high-frequency features are output, and the edges and textures of the hot spot area are significantly enhanced to facilitate subsequent analysis.

[0083] Feature extraction and fusion: Through independent detection of visible light and thermal imaging, visible light detection: The YOLOv5s model is trained on more than 100,000 fire scene samples to identify flame morphology, such as color, shape, dynamic jitter, and smoke characteristics; thermal imaging detection: The brightness temperature threshold method and dual-band algorithm are used to distinguish between fire points and other heat sources.

[0084] Perform dual-light fusion and dynamically adjust the weights based on the scenario. For example, visible light is used as the primary focus during the day, while thermal imaging is used as the primary focus at night. For example, in a forest fire prevention system, visible light accounts for 70% of the weight during the day, while thermal imaging accounts for 90% of the weight at night.

[0085] The DAF-Net domain-adaptive dual-branch feature decomposition and fusion network is used to reduce the distribution differences between infrared and visible light images through multi-core maximum mean difference. The Transformer-CNN structure is used to preserve cross-modal features, such as the consistency of flame edges and hot spot positions.

[0086] Scan areas are divided by thermal risk level:

[0087] Using machine learning models, such as random forests or XGBoost, areas are scored for thermal risk and classified into high, medium, and low risk categories. High-risk areas, such as those with recent fires, are marked as priority areas for scanning, while low-risk areas are scanned less frequently to conserve resources.

[0088] The drone's high-definition camera captures details such as flame morphology and smoke contours. It uses an uncooled vanadium oxide detector operating in the 8-14μm wavelength range, capable of penetrating smoke and haze to detect temperature anomalies as small as 0.1°C. For example, in a dual-spectrum system for power transmission lines, thermal imaging can locate a fire source within a 1㎡ area within a 3km radius. The YOLOv5s model detects flames and smoke, extracts morphological features, and uses NSCT to extract temperature gradients and hot spot distribution.

[0089] Dynamically adjust weights based on the scenario, such as focusing on visible light during the day and thermal imaging at night, for weighted fusion. Use the EMMA framework or Transformer-CNN structure to retain cross-modal features. Eliminate natural heat sources through temperature thresholds and morphological features. Dynamically update the classification model based on historical false positives or missed reports. Use the highest resolution thermal imaging mode to ensure the capture of small fire sources. Switch to low-resolution mode to reduce power consumption. Use AI algorithms to locally magnify suspicious areas. Use the drone's NPU or GPU to analyze thermal imaging data in real time. If a temperature anomaly is detected, high-resolution mode is automatically triggered. The cloud calls the NSCT fusion model to process complex scenarios and dynamically optimize the resolution strategy.

[0090] By overlaying satellite remote sensing data with ground IoT sensors through the GIS system, a three-dimensional heat risk map is generated. The risk level is updated based on real-time meteorological data (wind speed, humidity), with higher scanning frequencies assigned to high-risk areas; conversely, low-resolution pattern recognition is continuously performed;

[0091] The area is divided into 1km×1km grids, and the mean of each feature within the grid is extracted. The average humidity change rate over the past 7 days is calculated. The fire frequency over the same period in the past 3 years is also counted. If a fire occurred in the grid within 30 days, a binary feature is added; otherwise, it is 0. Based on historical data, if a fire occurred in the grid during the observation period, it is marked as high risk (1), otherwise it is low risk (0). Semi-supervised enhancement is performed, and K-Means clustering is used to generate pseudo labels for unlabeled areas. The XGBoost model is used to identify key risk drivers by sorting the built-in feature importance. Code example:

[0092] import xgboost as xgb

[0093] from sklearn.model_selection import TimeSeriesSplit

[0094] # Time series cross validation (preventing data leakage)

[0095] tscv = TimeSeriesSplit(n_splits=5)

[0096] for train_idx, test_idx in tscv.split(X):

[0097] X_train, X_test = X.iloc[train_idx], X.iloc[test_idx]

[0098] y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]

[0099] # Training parameters (after optimization)

[0100] params = {

[0101] 'objective': 'binary:logistic',

[0102] 'max_depth': 8,

[0103] 'learning_rate': 0.05,

[0104] 'subsample': 0.7,

[0105] 'colsample_bytree': 0.6,

[0106] 'scale_pos_weight': 10 # Handling class imbalance (few high-risk samples)

[0107] }

[0108] model = xgb.train(params, xgb.DMatrix(X_train, y_train), num_boost_round=500)

[0109] # Validation AUC

[0110] preds = model.predict(xgb.DMatrix(X_test))

[0111] auc_score = roc_auc_score(y_test, preds)

[0112] By running in low-resolution mode 80% of the time, the overall energy consumption is reduced by 65%, the missed detection rate in high-risk areas is less than 3%, and the time from anomaly triggering to high-resolution imaging completion is less than 1 second, meeting the real-time warning requirements. Table 1: Dynamic scanning strategy;

[0113]

[0114] Dynamic adjustment of thermal imaging resolution is achieved through hardware-level pixel subgroup activation, threshold-triggered local super-resolution, risk-level-driven resolution mapping, deep learning super-resolution reconstruction, real-time detail enhancement, optical-electronic synergistic compensation, and adaptive energy consumption control. This technology enables intelligent switching of resolution between 0.1MP and 2MP for high-dynamic scenarios such as fire monitoring.

[0115] The distributed edge computing module synchronizes cross-modal data in time and space through a multimodal feature alignment network. It dynamically allocates CUDA cores and CPU resources through adaptive computing scheduling. It uses an improved artificial bee colony algorithm to dynamically allocate bandwidth for multi-sensor data streams through a time-sharing multiplexing protocol for three-dimensional path planning.

[0116] Multimodal Feature Alignment Network:

[0117] Through the multimodal feature alignment network, heterogeneous data from different sensors are temporally and spatially aligned to ensure consistency of timestamps and spatial coordinates of data from different modalities. Time deviations between sensors are eliminated through timestamp calibration. Spatial mapping: The coordinate systems of different sensors are aligned to the same reference system, such as geographic coordinates or device coordinates, to achieve physical consistency in data fusion.

[0118] Dynamic allocation of CUDA cores and CPU resources:

[0119] Dynamically allocate GPU and CPU resources to avoid idle or overloaded resources; prioritize real-time image processing tasks for fire monitoring to ensure low-latency response for fire monitoring tasks; adjust resource allocation ratios based on real-time task requirements to avoid single-point resource bottlenecks; and dynamically adjust scheduling strategies in scenarios such as sensor data flow fluctuations and network bandwidth changes.

[0120] In the distributed edge computing module, the dynamic allocation of CUDA cores and CPU resources is achieved by real-time monitoring of hardware loads, such as GPU utilization, CPU core occupancy, and task characteristics, and combining an improved artificial bee colony algorithm for multi-objective optimization to dynamically adjust resource allocation strategies. Computationally intensive tasks are preferentially assigned to CUDA cores for accelerated parallel processing, such as matrix operations in three-dimensional path planning, while logic-intensive tasks, such as path optimization logic, are controlled by the CPU. Through a time-sharing multiplexing protocol, the system dynamically divides bandwidth according to the priority of sensor data streams to ensure high-real-time tasks, such as priority transmission of lidar point cloud processing. At the same time, asynchronous memory transmission and shared memory optimization are used to reduce data copy delays between the GPU and CPU. During dynamic scheduling, the reinforcement learning model provides real-time feedback on hardware performance indicators and adaptively adjusts thread block division and task offloading strategies. For example, when the GPU is overloaded, some tasks are migrated to the CPU to maximize resource utilization and balance energy consumption, ultimately efficiently supporting complex tasks such as three-dimensional path planning in scenarios such as autonomous driving and industrial robots.

[0121] Task classification and prioritization:

[0122] High-computing load tasks such as image feature extraction and deep learning inference are dispatched to the GPU, leveraging CUDA parallel computing capabilities to accelerate processing. For example, inference tasks of the YOLOv5s model; real-time analysis of thermal images and multimodal data fusion;

[0123] Low-latency tasks such as sensor data preprocessing and protocol parsing are dispatched to the CPU to handle lightweight tasks to ensure the real-time performance of data streams. These tasks include sensor data format conversion, timestamp calibration, and cross-modal alignment.

[0124] Dynamic resource allocation strategy:

[0125] Hardware performance counters are used to collect real-time GPU and CPU load, temperature, memory usage, and other indicators. Resource allocation is dynamically adjusted based on task queue status, such as task priority and execution time estimate.

[0126] First, task prioritization is based on business needs or real-time requirements. For example, high-priority tasks, such as urgent decisions for 3D path planning, are prioritized for GPU core accelerated computing, while low-priority tasks, such as data log storage, are processed serially by the CPU. Second, execution time estimation uses prediction models trained on historical task data, such as LSTM or ARIMA, to estimate the required resources and execution time for a task, and dynamically adjust the resource allocation ratio. For example, if a task is expected to occupy 80% of the GPU's computing power, the system will reserve the corresponding resources and prevent other tasks from preempting them. Based on this, dynamic scheduling algorithms, such as the improved artificial bee colony algorithm or priority queue scheduling, combine hardware performance indicators, GPU utilization, CPU load, and task queue status to generate resource allocation plans in real time. High-priority tasks trigger a resource preemption mechanism, forcibly reclaiming GPU resources from lower-priority tasks. Tasks with longer execution times are dynamically adjusted through a time-sharing multiplexing protocol to ensure maximum resource utilization. Finally, the system continuously optimizes scheduling strategies through reinforcement learning models to adapt to task load fluctuations, such as sudden high-priority tasks, to achieve efficient resource utilization and stable service response.

[0127] Improved artificial bee colony algorithm:

[0128] To address the low search efficiency of the standard ABC algorithm in discrete space, a neighborhood search strategy based on the local smoothness of the path is introduced. For example, a point on the path is randomly selected, new points are generated in its neighborhood, and the path quality is evaluated using a probability density function. Adaptive learning is performed by dynamically adjusting parameters such as the number of leader bees and the probability of scout bees. This allows for rapid exploration of the global space in the early stages of the search, followed by a more refined search for the local optimal solution in the later stages. Furthermore, a differential evolution algorithm is introduced during the scouting phase to improve convergence speed.

[0129] Comprehensively consider path length, terrain cost such as steepness, energy consumption such as turning radius, threat avoidance such as obstacle distance and high-risk area scanning density;

[0130] Dynamic bandwidth allocation for multi-sensor data streams:

[0131] The tool collects metrics such as bandwidth usage, latency, and packet loss rate of each sensor data stream in real time. It monitors the network load between edge nodes and the cloud, extracts the resolution and frame rate of thermal imaging video, and the characteristics and update frequency of temperature and humidity sensor data streams. It also identifies thermal imaging data and ambient temperature data for fire monitoring. It also classifies sensor data streams based on traffic business models.

[0132] Give high priority to thermal imaging video and smoke detection signals;

[0133] Environmental data such as temperature, humidity, and air pressure are given medium priority;

[0134] Device status data such as sensor health status and log information are given a low priority, while thermal imaging data is given a higher priority based on real-time environmental conditions, such as high temperature warnings.

[0135] Dynamically assign weights based on confidence or task requirements. For example, the weight of the IMU increases during high-speed motion. Use the LSTM model to predict the data flow requirements of each sensor over a period of time, and combine it with the traffic prediction model to evaluate network load trends. Calculate the bandwidth requirements of each sensor data flow and evaluate the total available bandwidth of the network link.

[0136] By minimizing the delay of high-priority data flows and maximizing bandwidth utilization, an initial solution is generated as a bandwidth allocation scheme, such as:

[0137] Thermal imaging data is allocated 300 Mbps;

[0138] 100 Mbps for temperature and humidity data;

[0139] Other data is allocated 100 Mbps;

[0140] By dynamically adjusting the bandwidth allocation ratio and combining it with a time-division multiplexing protocol, the bandwidth is divided into time slices, and the length of the time slices is allocated according to priority. Statistical time-division multiplexing dynamically allocates time slices based on real-time needs, with high-priority data streams allocated more time slices. For example, thermal imaging data occupies 60% of the time slice. Low-priority data streams are transmitted only during idle periods, such as device status data when the thermal imaging system is idle. Buffer queues are used to temporarily store non-real-time data, such as temperature and humidity data, and batch transmit them during low-load periods. A token bucket algorithm is used to control burst traffic and avoid congestion. A time-division multiplexing protocol is applied to divide the total bandwidth into multiple equal-time slices, with high-priority data streams allocated continuous time slices and low-priority data streams allocated fragmented time slices. Time-division multiplexers are deployed at edge nodes to poll each sensor data stream according to the time slice, and low-latency time slot switching is achieved through hardware acceleration.

[0141] 3D path generation:

[0142] The path is represented by three-dimensional coordinate points (x, y, z), with fixed starting and ending points and intermediate points optimized using the ABC algorithm. An initial path is randomly generated to ensure that flight constraints such as minimum turning radius and maximum climb angle are met. The best path is selected based on the fitness value and the location of neighboring points is updated. Other drones select paths based on probability for local search. If the path is not improved, a new path is randomly generated. Spline interpolation is used to eliminate jagged paths. In combination with lidar or visual sensors, the obstacle map is dynamically updated and path replanning is triggered. At the same time, the path planning optimizes both coverage of high-risk areas and energy consumption, for example, deploying dense formations in mountainous areas and sparse formations in flat areas.

[0143] The device-edge-cloud collaborative decision-making module builds a federated learning-driven model sharing network. Each edge node trains a lightweight YOLOv5s pruning model based on local data, generates a global optimization model, and synchronizes it in reverse, performing dynamic networking and adjusting the density of drone formations in real time.

[0144] Federated learning enables collaborative model training across edge nodes, and dynamic networking allows for real-time adjustment of drone fleet density to optimize fire monitoring efficiency.

[0145] Build a federated learning-driven model sharing network:

[0146] Based on a lightweight design, redundant convolutional layers are removed from the YOLOv5s model, reducing computational complexity while maintaining detection accuracy. Each drone and ground sensor node uses locally collected multimodal data, including thermal imaging, visible light images, and environmental parameters, for model training. Fire characteristics such as flames and smoke are detected and the risk level is output.

[0147] After obtaining the spatiotemporal alignment of multimodal data, the regional master node receives the gradients of all edge nodes, and then uses the federated averaging algorithm in federated learning to perform weighted aggregation to generate a global model. Weights are assigned to edge nodes based on the data volume or task priority, with nodes in high-risk fire areas having higher weights. Hyperparameters such as the learning rate and regularization coefficient of the global model are adjusted based on the aggregation results. The performance of the global model is evaluated using a reserved validation dataset.

[0148] Dynamic networking and drone formation density adjustment:

[0149] A wireless mesh network is used, with each drone acting as a node, supporting multi-hop relay transmission. Drones are divided into multiple clusters based on a clustering algorithm, with a designated master node in each cluster responsible for channel allocation to reduce conflicts. Beidou / 5G integrated positioning is combined to improve positioning accuracy in complex environments. Based on the thermal risk score, dense formations are deployed in high-risk areas, scanning every 10 minutes. Sparse formations are used in low-risk areas, with the scanning cycle extended to 30 minutes. For example, in a forest fire prevention system, drone swarms automatically encrypt their formations at the early stages of a fire, covering a 15-kilometer radius. The formation shape is dynamically adjusted by setting a navigator and a virtual navigation point. The drone's behavior rules are adjusted in real time according to environmental changes. At the same time, the cloud calls the NSCT fusion model to handle complex scenarios and dynamically optimize the resolution strategy.

[0150] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. Those skilled in the art will appreciate that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution.

[0151] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, and may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment as needed.

[0152] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. An image fusion device-edge-cloud collaborative intelligent fire monitoring system, characterized by: The system includes: The multimodal image fusion module generates a dynamic scanning priority map based on prior data, uses a dual-light fusion algorithm to distinguish natural heat sources from abnormal fire conditions, divides the scanning area according to the thermal risk level, and dynamically adjusts the thermal imaging resolution. The distributed edge computing module uses a multimodal feature alignment network to synchronize cross-modal data in time and space, and dynamically allocates CUDA cores and CPU resources through adaptive computing scheduling. It also uses an improved artificial bee colony algorithm to dynamically allocate bandwidth for multi-sensor data streams through a time-sharing multiplexing protocol for three-dimensional path planning. The device-edge-cloud collaborative decision-making module builds a federated learning-driven model sharing network. Each edge node trains a lightweight YOLOv5s pruning model based on local data, generates a global optimization model, and synchronizes it in reverse, performing dynamic networking and adjusting the density of drone formations in real time. The process of generating a dynamic scanning priority map is as follows: Based on the historical data of the fire point, including fire data, environmental information, and geographic location information, the system obtains the prior data and classifies the prior data into text information, image annotations, and sensor data. The prior data is structured through optical character recognition (OCR) of text information, image annotations, and sensor data, and multimodal fusion technology is used to integrate text, image, and sensor data. YOLOv5s outputs bounding box coordinates to locate the fire point, Mask R-CNN segments the burned area, quantifies the burned area and fire line shape, rotates the fire satellite imagery and simulates smoke, and cleans and aligns the sensor data. Keywords are extracted from the reports extracted by optical character recognition (OCR) and converted into numerical vectors using TF-IDF or word embedding. ResNet is used to extract high-level features of the burned area, and the burn intensity is calculated based on the masked area.

2. The image fusion device-edge-cloud collaborative intelligent fire monitoring system according to claim 1 is characterized by: The process of distinguishing natural heat sources from abnormal fire conditions is as follows: Collect visible light channel and thermal imaging channel data, and perform color space conversion on the visible light channel using a dehazing algorithm and image enhancement technology. Use the YOLOv5s model to detect flame or smoke features and extract morphological features. The thermal imaging channel uses NSCT non-subsampled contourlet transform to extract temperature gradients and hot spot distribution; pulse-coupled neural networks are used to fuse high-frequency features, and adaptive fuzzy logic algorithms are used to process low-frequency information; By utilizing dual-light fusion, the weights are dynamically adjusted according to the scene, and the DAF-Net domain adaptive dual-branch feature decomposition and fusion network is adopted. The distribution difference between infrared and visible light images is reduced through the multi-core maximum mean difference, and the Transformer-CNN structure is used to retain cross-modal features.

3. The image fusion device-edge-cloud collaborative intelligent fire monitoring system according to claim 2 is characterized by: The process of dividing the scanning area by thermal risk level is as follows: Random forests are used to score and categorize regions for thermal risk. High-definition drone cameras are used to capture images of the area, and NSCT is used to extract temperature gradients and hot spot distribution. Weights are dynamically adjusted based on the scenario, and the EMMA framework is used to retain cross-modal features. By using temperature thresholds and morphological features, natural heat sources are excluded and the classification model is dynamically updated based on historical false positives; Suspicious areas are locally magnified using AI algorithms. The drone's onboard NPU analyzes thermal imaging data in real time. If a temperature anomaly is detected, high-resolution mode is automatically triggered. The cloud uses the NSCT fusion model to process complex scenarios and dynamically optimize the resolution strategy. Otherwise, low-resolution pattern recognition continues. The area is divided into grids, and the mean of each feature within the grid is extracted. The average humidity change rate over the past seven days is calculated, and the fire frequency over the same period of the past three years is also counted. If a fire occurred in the grid within 30 days, a binary feature is added; otherwise, it is 0. Based on historical data, if a fire occurred in the grid during the observation period, it is marked as high risk = 1, otherwise it is low risk = 0. Semi-supervised enhancement is performed, and K-Means clustering is used to generate pseudo labels for unlabeled areas. The XGBoost model is used to identify key risk drivers by sorting the built-in feature importance.

4. The image fusion device-edge-cloud collaborative intelligent fire monitoring system according to claim 1 is characterized by: The process of performing spatiotemporal synchronization of cross-modal data is as follows: Through timestamp calibration, different sensors are synchronized in time and their coordinate systems are aligned to the same reference system.

5. The image fusion device-edge-cloud collaborative intelligent fire monitoring system according to claim 4 is characterized in that: The process of dynamic allocation of CUDA cores and CPU resources is as follows: Dynamically allocate GPU and CPU resources, prioritize real-time image processing tasks for fire monitoring, adjust resource allocation ratios based on real-time task requirements, and dynamically adjust scheduling strategies when sensor data streams fluctuate or network bandwidth changes. Dynamic resource allocation strategy: Use hardware performance counters to collect GPU and CPU load, temperature, and memory usage in real time; dynamically adjust resource allocation based on task queue status.

6. The image fusion device-edge-cloud collaborative intelligent fire monitoring system according to claim 5 is characterized by: The process of dynamically allocating bandwidth to multi-sensor data streams is as follows: Collect bandwidth usage, latency, and packet loss rates for each sensor data stream, monitor network load between edge nodes and the cloud, extract thermal imaging video resolution and frame rate, and the characteristics and update frequency of temperature and humidity sensor data streams; identify thermal imaging data and ambient temperature data for fire monitoring; and classify sensor data streams based on traffic business models. Dynamically assign weights based on confidence or task requirements, use the LSTM model to predict the data flow requirements of each sensor in the future, and combine it with the traffic prediction model to evaluate network load trends; Calculate the bandwidth requirements of each sensor data stream.

7. The image fusion device-edge-cloud collaborative intelligent fire monitoring system according to claim 1 is characterized by: The process of generating the three-dimensional path is as follows: The starting and ending points are fixed, and the intermediate points are optimized using the ABC algorithm; an initial path is randomly generated, and the best path is selected based on the fitness value, and the location of the neighboring points is updated; other drones select paths based on probability for local search; if the path is not improved, a new path is randomly generated; spline interpolation is used to eliminate jagged paths; combined with lidar or visual sensors, the obstacle map is dynamically updated and path replanning is triggered; and at the same time, the coverage of high-risk areas is optimized during path planning.

8. The image fusion device-edge-cloud collaborative intelligent fire monitoring system according to claim 1 is characterized by: The process of building a federated learning-driven model sharing network is as follows: After obtaining the spatiotemporal alignment of multimodal data, the regional master node receives the gradients of all edge nodes, and then uses the federated averaging algorithm in federated learning to perform weighted aggregation to generate a global model. Weights are assigned according to task priorities, and the learning rate and regularization coefficient hyperparameters of the global model are adjusted based on the aggregation results.

9. The image fusion device-edge-cloud collaborative intelligent fire monitoring system according to claim 1 is characterized in that: The process of adjusting the density of the drone formation in real time is as follows: A wireless mesh network is used, with each drone acting as a node, supporting multi-hop relay transmission. Based on a clustering algorithm, drones are divided into multiple clusters. Within each cluster, a designated master node is responsible for channel allocation. Scanning strategies are adjusted based on thermal risk scores, and formation shape is dynamically adjusted by setting a navigator and virtual navigation points. The drone's behavior rules are adjusted in real time according to environmental changes; at the same time, the cloud calls the NSCT fusion model to process complex scenarios and dynamically optimize the resolution strategy.

Images