A ros-based dynamic modal switching forest fire point detection method and system

Abstract

The application discloses a kind of dynamic modal switching forest fire point detection method and system based on ROS, comprising: obtaining visible light image and infrared thermal imaging image under the same scene and publishing by ROS topic;The brightness entropy and local contrast mean of visible light image and the temperature distribution standard deviation of infrared thermal imaging image are calculated, visible light modal confidence and infrared modal confidence are generated and stored in historical confidence queue;Judge whether modal switching instruction is received, if it is non-automatic mode, then forcibly switch detection mode, otherwise, timing smoothing judgment is carried out;According to the determined detection mode, the corresponding detection sub-network is selected to detect fire point and publish the result.The application avoids modal frequent switching through timing smoothing judgment, improves flexibility through manual instruction and automatic decision cooperation, considers detection efficiency and precision through detection mode adaptive selection, and has the advantages of strong detection robustness, low computing overhead and easy deployment integration.

Description

Technical Field

[0001] This invention belongs to the field of forest fire prevention and multimodal visual detection technology, and in particular relates to a dynamic modal switching forest fire detection method and system based on ROS. Background Technology

[0002] Forest fires are characterized by their suddenness, destructiveness, and difficulty in control. Timely and accurate detection of fire points is crucial for effective fire control and minimizing losses. In recent years, visible light and infrared thermal imaging sensors mounted on drones and satellite platforms have become mainstream technologies for fire point detection. Visible light images provide rich spatial texture details, while infrared images are sensitive to thermal radiation and can effectively penetrate smoke to directly detect heat sources. The complementary advantages of these two technologies can significantly improve the accuracy and reliability of fire point detection. However, existing methods face the following three core problems in practical applications: First, the fusion strategy is rigid. Most existing multimodal fire detection methods use fixed fusion weights or a single fusion strategy, which cannot dynamically adjust the modal contribution based on actual environmental conditions. Under good lighting and unobstructed conditions, visible light images can provide sufficient detection information, and using complex cross-modal fusion models in this case would waste computational resources. However, under adverse conditions such as at night, smoke obstruction, and strong reflection, the performance of single visible light detection drops sharply, and it is necessary to rely on infrared modal information. Existing technologies have proposed environmental feature extraction methods based on brightness entropy, contrast, and temperature distribution standard deviation, and used them to generate continuous fusion weights for visible light and infrared. However, the fusion weights output by this method are used to perform weighted summation of dual-stream features, which is essentially still within the "always fused" framework and cannot completely avoid the computational overhead of the fusion model when computational resources are limited or under good lighting. In contrast, this invention proposes for the first time to use environmental features for discrete modal switching decisions, that is, to completely close the fusion branch and run only a lightweight single-modal detector, thereby significantly reducing the computational load while ensuring detection accuracy.

[0003] Second, modal switching is lacking and anti-shake mechanisms are absent. Existing methods consistently employ fixed detection strategies, lacking mechanisms for autonomous or controlled switching between different modalities, and also failing to achieve effective human-machine collaboration with UAV flight control systems or ground stations. Furthermore, simply switching based on single-frame features leads to frequent oscillations at scene edges, severely impacting the stability of detection results.

[0004] Third, the non-ideal alignment problem in cross-modal fusion is overlooked. Most existing methods assume that infrared and visible light images have been accurately registered. However, in actual UAV inspections, due to parallax caused by differences in the physical installation positions of sensors, physical time delay between thermal diffusion processes and visual representation, and differences in resolution between different sensors, non-ideal alignment problems generally exist between images at the spatial, temporal, and semantic levels.

[0005] Therefore, there is an urgent need for a forest fire detection method and system that can dynamically select the optimal detection strategy based on the environment, handle non-ideal alignment problems, support human-machine collaboration, and have anti-modal jitter capabilities. Summary of the Invention

[0006] To overcome the technical shortcomings of existing forest fire detection systems, such as fixed fusion strategies, inability to autonomously select detection modes, modal jitter caused by simple single-frame switching, and difficulty in handling non-ideal alignment of infrared and visible light images, a dynamic modal switching forest fire detection method and system based on ROS is provided.

[0007] The technical solution adopted by this invention to solve its technical problem is: A ROS-based method for detecting forest fire points with dynamic modal switching includes the following steps: S100: Acquire visible light images and infrared thermal images of the same scene and publish them via ROS topics; S200: Subscribe to visible light images and infrared thermal imaging images, calculate the brightness entropy and local contrast mean of the visible light images, and the temperature distribution standard deviation of the infrared thermal imaging images, generate visible light mode confidence and infrared mode confidence based on the brightness entropy, local contrast mean, and temperature distribution standard deviation, and store them in the historical confidence queue. S300: Determine whether a mode switching command has been received. If a mode switching command has been received and the mode switching command is not an automatic mode command, then force a switch to the corresponding detection mode according to the mode switching command and proceed to S500. If no mode switching command has been received or the mode switching command is an automatic mode command, then proceed to S400. The detection modes include visible light single-mode detection mode and cross-mode fusion detection mode. S400: Perform time-series smoothing judgment based on historical confidence queue. When the judgment result meets the preset mode switching conditions, switch the current detection mode; otherwise, maintain the current detection mode. S500: Based on the current detection mode determined by S300 or S400, select the corresponding detection sub-network to perform fire detection on visible light images and / or infrared thermal imaging images, and publish the fire detection results through the ROS topic. The fire detection results include fire location, confidence level, and bounding box information.

[0008] Preferably, the formula for calculating the luminance entropy in S200 is: in, For brightness entropy, To convert a visible light image to grayscale, the grayscale levels are... The probability of; The formula for calculating the mean local contrast is: in, This is a grayscale image obtained by converting a visible light image. Let L be the Laplace operator, and L be the pair of operators. The response image obtained after applying the Laplacian operator. This represents the local contrast average. To calculate the average value of all pixel values ​​in the input matrix; The formula for calculating the standard deviation of temperature distribution is: in, The standard deviation of the temperature distribution. This is an infrared thermal imaging image. and These are the height and width of the infrared thermal image, respectively. The average temperature of the infrared thermal imaging image. , These are the spatial coordinates of pixels in the image.

[0009] Preferably, in S200, the brightness entropy, local contrast mean, and temperature distribution standard deviation are concatenated into an environmental feature vector by a multilayer perceptron, which is mapped to the original modal confidence. The normalized visible light modal confidence and infrared modal confidence are output by the Softmax function, and the sum of the visible light modal confidence and infrared modal confidence is 1. The multilayer perceptron includes a fully connected layer, a ReLU activation layer, and a fully connected layer in sequence.

[0010] Preferably, in S200, the luminance entropy, the mean local contrast, and the standard deviation of temperature distribution are calculated once every N frames, where N is 5 to 10.

[0011] Preferably, the mode switching command in S300 is received through ROS topic or mode_switch. Command value 0 indicates a forced switch to visible light single-mode detection mode, command value 1 indicates a forced switch to cross-mode fusion detection mode, and command value 2 indicates an automatic mode command. When command value 0 or 1 is received, the system immediately switches to the corresponding detection mode and maintains it until a new command is received or the system restarts.

[0012] Preferably, the preset switching conditions in S400 are: When the visible light mode confidence of all frames in the historical confidence queue is greater than the first threshold and the infrared mode confidence is less than the second threshold, a switch to visible light single-mode detection mode is triggered; wherein, the value range of the first threshold is 0.6 to 0.8 and the value range of the second threshold is 0.2 to 0.4.

[0013] Preferably, S400 includes: If the current detection mode is cross-modal fusion detection mode and the preset switching conditions are met, then switch to visible light single-modal detection mode; If the current detection mode is visible light single-modal detection mode, and the M consecutive frames in the historical confidence queue do not meet the preset switching conditions, then switch back to cross-modal fusion detection mode; where M is a preset positive integer, with a value range of 3 to 10; Otherwise, maintain the current detection mode.

[0014] Preferably, the visible light single-modal detection mode performs fire detection through a visible light single-modal detection sub-network. The visible light single-modal detection sub-network is built based on the YOLO series models, uses the CSPDarknet backbone network to extract visible light image features, performs feature fusion through a path aggregation network, and outputs the fire detection results through the detection head. The visible light single-modal detection sub-network only takes visible light images as input. The cross-modal fusion detection mode performs fire detection on visible light images and infrared thermal imaging images through a cross-modal fusion detection sub-network. This sub-network uses a visible light backbone network and a thermal infrared backbone network to extract features from the visible light image and infrared thermal imaging image, respectively. These features are then processed sequentially by a learnable modal dictionary module, a multi-scale cross-modal alignment module, and a fire detection fusion module. The fire detection result is then output through the detection head. The learnable modality dictionary module constructs a learnable feature prototype dictionary for each modality, generates query weights through a spatial attention mechanism, reconstructs feature maps by weighted aggregation of dictionary prototypes, and fuses them with the original features through residual connections to compensate for cross-modal semantic differences. The multi-scale cross-modal alignment module predefines multiple mesh sizes with different granularities, performs multi-scale feature transformation on infrared and visible light features, calculates cross-scale similarity maps, generates alignment weights, and achieves implicit spatial alignment from the infrared feature space to the visible light feature space. The fire detection fusion module inputs aligned features into the detection head to generate an initial fire confidence heatmap. The heatmap is then upsampled at multiple scales. The upsampled heatmap is then concatenated with the aligned features at the corresponding scale along the channel dimension. After convolution processing and activation function generation, gate weights are generated. Based on the gate weights, the infrared features and visible light aligned features are adaptively weighted and fused to control the contribution ratio of different modal features at each scale, and the refined fused features are output.

[0015] Preferably, when switching from cross-modal fusion detection mode to visible light single-modal detection mode, the system maintains a cache of infrared feature maps from the most recent few frames to provide thermal history prior information when switching back to cross-modal fusion detection mode, thereby accelerating the fire point localization of the cross-modal fusion detection sub-network.

[0016] A ROS-based dynamic modal switching forest fire detection system includes: The image acquisition and publishing module is used to acquire visible light images and infrared thermal imaging images of the same scene and publish them via ROS topics; The environmental perception and modal decision-making module subscribes to visible light images and infrared thermal imaging images, calculates the brightness entropy and mean local contrast of the visible light images, and the standard deviation of the temperature distribution of the infrared thermal imaging images. Based on the brightness entropy, mean local contrast, and standard deviation of the temperature distribution, it generates visible light modal confidence scores and infrared modal confidence scores, and stores them in a historical confidence score queue. The environmental perception and modal decision-making module includes a temporal smoothing unit, which performs temporal smoothing judgment based on the historical confidence score queue. When the judgment result meets the preset modal switching conditions, the current detection mode is switched; otherwise, the current detection mode is maintained. The manual intervention interface is used to determine whether a mode switching command has been received. If a mode switching command is received and the command is not an automatic mode command, the system will force a switch to the corresponding detection mode according to the command and enter the heterogeneous detection engine. If no mode switching command is received or the command is an automatic mode command, the timing smoothing unit will be triggered to perform a judgment. The detection modes include visible light single-mode detection mode and cross-mode fusion detection mode. The heterogeneous detection engine is used to select the corresponding detection sub-network to perform fire detection on visible light images and / or infrared thermal imaging images according to the determined current detection mode, and output the fire detection results, which include fire location, confidence level and bounding box information. The system status publishing module is used to publish fire detection results via ROS topics.

[0017] This invention, by setting a temporal smoothing judgment step, switches the current detection mode only when the historical confidence queue meets the preset switching conditions, effectively avoiding frequent mode switching caused by single-frame noise. Simultaneously, it switches to the visible light single-modal detection mode when the conditions are met to reduce computational overhead. By setting a modal switching command judgment step, operators can inject forced switching commands in real time and skip the temporal smoothing judgment, allowing manual commands and automatic decision-making to work in tandem, achieving a complementary advantage of human experience and machine decision-making. It adaptively selects either the YOLO single-modal network or the cross-modal fusion detection network based on the current detection mode, balancing detection efficiency and all-weather detection accuracy. Cross-modal feature alignment in the fusion detection mode effectively overcomes spatial misalignment problems caused by sensor parallax, thermal diffusion delay, and resolution differences, improving detection robustness. Based on the ROS architecture, the modules of each step are decoupled, allowing for easy integration into existing UAV ground station systems, supporting remote monitoring and secondary development. Attached Figure Description

[0018] Figure 1 This is a flowchart of a ROS-based dynamic modal switching forest fire detection method in one embodiment of the present invention; Figure 2 This is a schematic diagram of the overall architecture of a ROS-based dynamic modal switching forest fire detection method in one embodiment of the present invention. Figure 3 This is a flowchart illustrating the workflow of the environmental perception and modal decision-making module and its temporal smoothing unit in one embodiment of the present invention. Figure 4 This is a schematic diagram of the heterogeneous detection engine architecture in one embodiment of the present invention; Figure 5 This is a schematic diagram of the learnable modality dictionary module in the cross-modal fusion detection subnetwork according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of the multi-scale cross-modal alignment module in the cross-modal fusion detection subnetwork in one embodiment of the present invention; Figure 7 This is a schematic diagram of the fire detection fusion module in a cross-modal fusion detection subnetwork according to an embodiment of the present invention; Figure 8 This is a main flowchart of a forest fire detection method in one embodiment of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0020] This invention provides a dynamic modal switching forest fire detection method based on ROS. It is implemented using a ROS distributed architecture, and the corresponding system is deployed in a node-based manner on a UAV or ground station computing platform. The flowchart of the entire method is as follows: Figure 1 As shown in the diagram, the overall architecture is as follows: Figure 2 As shown.

[0021] (a) S100: Acquire visible light images and infrared thermal images of the same scene and publish them through ROS topics.

[0022] Specifically, the drone platform is equipped with a visible light camera (1920×1080 resolution, 30fps) and an infrared thermal imaging camera (640×512 resolution, 25fps). Both cameras are connected to an onboard computing board (containing 32GB of RAM and a 2048-core CUDA GPU) via a USB 3.0 interface. The onboard computing board runs Ubuntu 20.04 and the ROS Noetic distribution. The visible light camera and the infrared thermal imaging camera respectively push H.264 encoded video streams via the GStreamer pipeline using the RTSP protocol.

[0023] (ii) S200: Subscribe to visible light images and infrared thermal imaging images, calculate the brightness entropy and local contrast mean of the visible light images, and the temperature distribution standard deviation of the infrared thermal imaging images, generate visible light mode confidence and infrared mode confidence based on the brightness entropy, local contrast mean and temperature distribution standard deviation, and store them in the historical confidence queue.

[0024] Specifically, the brightness map is calculated after converting the visible light image to a grayscale image, reflecting the overall brightness distribution of the image; the mean local contrast is the mean of the gradient magnitude calculated after applying Laplacian operator filtering to the grayscale image, reflecting the richness of the image's texture and the sharpness of its edges; the standard deviation of the infrared temperature distribution is used to reflect the degree of dispersion of the temperature field in the infrared image.

[0025] In one embodiment, the formula for calculating the luminance entropy in S200 is: in, For brightness entropy, To convert a visible light image to grayscale, the grayscale levels are... The probability of; The formula for calculating the mean local contrast is: in, This is a grayscale image obtained by converting a visible light image. Let L be the Laplace operator, and L be the pair of operators. The response image obtained after applying the Laplacian operator. This represents the local contrast average. To calculate the average value of all pixel values ​​in the input matrix; The formula for calculating the standard deviation of temperature distribution is: in, The standard deviation of the temperature distribution. This is an infrared thermal imaging image. and These are the height and width of the infrared thermal image, respectively. The average temperature of the infrared thermal imaging image. , These are the spatial coordinates of pixels in the image.

[0026] In one embodiment, in S200, the brightness entropy, local contrast mean, and temperature distribution standard deviation are concatenated into an environmental feature vector by a multilayer perceptron, which is mapped to the original modal confidence. The normalized visible light modal confidence and infrared modal confidence are output by the Softmax function, and the sum of the visible light modal confidence and infrared modal confidence is 1. The multilayer perceptron includes a fully connected layer, a ReLU activation layer, and a fully connected layer in sequence.

[0027] In one embodiment, the luminance entropy, mean local contrast, and standard deviation of temperature distribution in S200 are calculated every N frames, where N ranges from 5 to 10. The overall workflow diagram is as follows: Figure 3 As shown.

[0028] (iii) S300: Determine whether a mode switching command has been received. If a mode switching command has been received and the mode switching command is not an automatic mode command, then force a switch to the corresponding detection mode according to the mode switching command and proceed to S500; if no mode switching command has been received or the mode switching command is an automatic mode command, then proceed to S400; wherein, the detection modes include visible light single-mode detection mode and cross-mode fusion detection mode; In one embodiment, the mode switching command in S300 is received via ROS topic or mode_switch. Command value 0 indicates a forced switch to visible light single-mode detection mode, command value 1 indicates a forced switch to cross-mode fusion detection mode, and command value 2 indicates an automatic mode command. When command value 0 or 1 is received, the system immediately switches to the corresponding detection mode and maintains it until a new command is received or the system restarts.

[0029] Specifically, mode switching commands come from the ground station or remote controller. Manual commands have higher priority than autonomous decisions. That is, when a forced mode command is received, the system immediately switches to the specified mode and remains in that mode until a new command is received or the system restarts.

[0030] Specifically, upon system startup, the `camera_node` uses OpenCV's VideoCapture interface to read the RTSP stream and publishes the decoded image to the ` / camera / visible_raw` and ` / camera / thermal_raw` topics via `sensor_msgs / Image` messages. The `mode_decision_node` subscribes to these topics, calculates environmental features at preset intervals (every 5 frames by default), and maintains a historical confidence queue of length 5. This node updates the ` / mode_control` parameter through the ROS parameter server. The `detection_node` reads this parameter in each detection loop and dynamically instantiates the corresponding detection sub-network. Manual intervention is achieved via `rosservice call / set_mode "mode: 0"`, which immediately overwrites the parameter server's value and triggers the reconfiguration of the `detection_node`.

[0031] (iv) S400: Perform time-series smoothing judgment based on the historical confidence queue. When the judgment result meets the preset mode switching conditions, switch the current detection mode; otherwise, maintain the current detection mode. In one embodiment, the preset switching condition in S400 is: When the visible light mode confidence of all frames in the historical confidence queue is greater than the first threshold and the infrared mode confidence is less than the second threshold, a switch to visible light single-mode detection mode is triggered; wherein, the value range of the first threshold is 0.6 to 0.8 and the value range of the second threshold is 0.2 to 0.4.

[0032] In one embodiment, S400 includes: If the current detection mode is a cross-modal fusion detection mode and the preset switching conditions are met, then switch to the visible light single-modal detection mode. This mechanism effectively avoids frequent mode switching caused by single-frame noise or scene edges, and improves system stability. If the current detection mode is the visible light single-mode detection mode, and the M consecutive frames in the historical confidence queue do not meet the preset switching conditions (i.e., the visible light mode confidence of at least one frame is ≤ the first threshold, or the infrared mode confidence of at least one frame is ≥ the second threshold), then switch back to the cross-modal fusion detection mode; where M is a preset positive integer, and the value range is 3 to 10. Otherwise, the current detection mode is maintained. This mechanism effectively avoids frequent mode switching caused by single-frame noise or scene edges, while ensuring that the system can automatically resume fusion detection when environmental conditions deteriorate, thus improving system stability and environmental adaptability.

[0033] Specifically, in well-lit daytime scenes (luminance entropy > 6.5, mean local contrast > 35, temperature standard deviation < 5), the system stably maintains visible light single-modal detection mode with a single-frame inference time of 12ms (GPU) and an mAP@0.5 of 86.2%. When entering a smoke-covered area (luminance entropy drops to 4.2, mean local contrast drops to 18, and temperature standard deviation increases to 12), after 5 consecutive frames where the confidence level meets the switching conditions, the system automatically switches to cross-modal fusion detection mode, with a single-frame inference time of 28ms, an mAP@0.5 increase to 89.7%, and a 60% reduction in the false negative rate. After the smoke dissipates, the system smoothly switches back to single-modal mode, and no jitter in the detection results occurs during the switching process.

[0034] (V) S500: Based on the current detection mode determined by S300 or S400, select the corresponding detection sub-network to perform fire detection on visible light images and / or infrared thermal imaging images, and publish the fire detection results through the ROS topic. The fire detection results include fire location, confidence level, and bounding box information; wherein, the detection sub-network consists of a parallel visible light single-modal detection sub-network and a cross-modal fusion detection sub-network. A schematic diagram of the heterogeneous detection engine architecture is shown below. Figure 4 As shown.

[0035] In one embodiment, the visible light single-modal detection mode performs fire detection through a visible light single-modal detection sub-network. The visible light single-modal detection sub-network is built based on the YOLO series models, uses the CSPDarknet backbone network to extract visible light image features, performs feature fusion through a path aggregation network, and outputs the fire detection results through the detection head. The visible light single-modal detection sub-network only takes visible light images as input and outputs the fire location, confidence level, and bounding box information. This sub-network model has fewer parameters, faster inference speed, and is suitable for scenes with good lighting and simple backgrounds.

[0036] The cross-modal fusion detection mode uses an encoder-fusion-decoder architecture in its cross-modal fusion detection sub-network to detect fire points in visible light and infrared thermal imaging images. The cross-modal fusion detection sub-network extracts features from visible light and infrared images using a visible light backbone network and a thermal infrared backbone network, respectively. These features are then processed sequentially by a learnable modal dictionary module, a multi-scale cross-modal alignment module, and a fire point perception fusion module. The fire point detection result is then output through the detection head. The learnable modality dictionary module constructs a learnable feature prototype dictionary D∈R for each modality. K×C Where K is the number of dictionary atoms (K=64 in this example), C is the number of feature channels, and query weights A∈R are generated through a spatial attention mechanism. B ×K×NThe dictionary prototype is weighted and aggregated to reconstruct feature maps, which are then fused with the original features via residual connections to compensate for cross-modal semantic differences; a schematic diagram of the learnable modality dictionary module is shown below. Figure 5 As shown; The multi-scale cross-modal alignment module predefines multiple mesh sizes with different granularities (N=4, 8, 16 in this embodiment) to perform multi-scale feature transformations on infrared and visible light features, calculates cross-scale similarity maps, and generates alignment weights to achieve implicit spatial alignment from the infrared feature space to the visible light feature space; overcoming the spatial misalignment problem caused by sensor parallax and scale differences. A schematic diagram of the multi-scale cross-modal alignment module is shown below. Figure 6 As shown; The fire detection fusion module inputs aligned features into the detection head to generate an initial fire confidence heatmap. The heatmap is then upsampled at multiple scales. The upsampled heatmap is then concatenated with the aligned features at the corresponding scale along the channel dimension. After convolution processing and activation function generation, gate weights are generated. Based on the gate weights, the infrared features and visible light aligned features are adaptively weighted and fused to control the contribution ratio of different modal features at each scale, and the refined fused features are output.

[0037] Specifically, the structure of the fire detection fusion module is as follows: Figure 7 As shown. The alignment features output by the multi-scale cross-modal alignment module. Input a lightweight convolutional detector head to generate an initial fire confidence heatmap. This heatmap represents the probability of a fire point existing at each pixel location. Subsequently, [the following was performed]... Perform multi-scale upsampling to obtain features aligned with each scale. Upsampled confidence plot with matching spatial dimensions ,in Representing feature layers at different scales. The upsampled confidence map... Alignment features with corresponding scale By splicing along the channel dimension, we obtain [ , The weights are processed sequentially through convolutional layers, batch normalization layers, and ReLU activation functions, and then through convolutional layers and Sigmoid activation functions to generate gated weights. G , G The value range is (0, 1), which controls the contribution ratio of infrared features and visible light aligned features during fusion. Finally, through... To achieve adaptive weighted fusion, where Infrared characteristics, The visible light alignment features are used to complete the feature refinement guided by the fire detection task, and finally the fire detection results are output through the detection head.

[0038] Specifically, the parameters of the learnable modal dictionary module, the multi-scale cross-modal alignment module, and the fire point perception fusion module are obtained through end-to-end training. Training uses the FLAME2-DT+ dataset (containing augmented data with simulated spatial offsets, temporal offsets, and smoke occlusion), with the AdamW optimizer and an initial learning rate of 2e^(-1 / 2). -3 Cosine annealing scheduling, batch size 16, training for 300 rounds. The total loss function includes detection loss Ldet (CIoU + binary cross-entropy) and alignment auxiliary loss Lalign (maximizing the cosine similarity of features in the fire region and minimizing the similarity of the background region, with a weight coefficient λ=0.1).

[0039] In one embodiment, when switching from cross-modal fusion detection mode to visible light single-modal detection mode, the system maintains a cache of infrared feature maps from the most recent few frames to provide thermal history prior information when switching back to cross-modal fusion detection mode, thereby accelerating the fire point localization of the cross-modal fusion detection sub-network.

[0040] Finally, the current detection mode, timestamp, visible light modal confidence score, infrared modal confidence score, detection results (fire location, confidence score, bounding box coordinates), and environmental feature values ​​(luminance entropy, local contrast, temperature standard deviation) are published in real time via ROS Topic / system_status for ground station visualization, logging, or subsequent analysis. The overall method flowchart is as follows: Figure 8 As shown.

[0041] This invention discloses a ROS-based dynamic modal switching forest fire detection method and system. It can automatically select the optimal detection strategy based on actual environmental conditions, avoid frequent modal switching through a time-series smoothing scheme, effectively save computational resources while ensuring detection accuracy, and extend UAV cruise time. Simultaneously, a manual intervention interface supports real-time takeover mode selection by ground operators, enhancing the system's human-machine collaboration capabilities. The cross-modal fusion detection subnetwork effectively handles non-ideal alignment problems caused by sensor parallax, thermal diffusion delay, and resolution differences, exhibiting significantly better robustness in complex environments than existing methods. The system architecture of this invention is based on the ROS standard communication protocol, possessing good scalability and portability. It can be widely applied in forest fire prevention, UAV inspection, intelligent monitoring, and other fields, demonstrating significant industrial practical value and broad industrialization prospects.

[0042] Compared with the prior art, the present invention has the following advantages: First, it is computationally efficient and avoids modal jitter. By introducing a temporal smoothing scheme, it only switches to a lightweight single-modal mode when the conditions are consistently met across multiple frames. This avoids unnecessary fusion computation overhead and prevents frequent mode switching caused by noise in a single frame.

[0043] Second, it differs fundamentally from existing "continuous fusion" methods. This invention explicitly uses environmental features for discrete mode switching decisions, rather than continuous weighting, thereby completely shutting down the fusion branch when the lighting is good. This is a substantial breakthrough in the existing technological paradigm.

[0044] Third, the detection robustness is significantly enhanced. By introducing a multi-scale cross-modal alignment module, the system can effectively handle non-ideal alignment problems caused by sensor parallax, thermal diffusion delay, and resolution differences.

[0045] Fourth, it supports human-machine collaborative operation and features anti-jitter design. Real-time injection of manual commands is achieved through the ROS topic mechanism, with manual input taking precedence over automatic decision-making, thus improving system flexibility. Simultaneously, a timing smoothing scheme ensures decision-making stability in automatic mode.

[0046] Fifth, the system architecture is open and highly scalable. The ROS-based node-based design decouples the system modules, allowing for easy integration into existing UAV ground station systems and supporting remote monitoring and secondary development.

[0047] In one embodiment, a ROS-based dynamic modal switching forest fire detection system is also provided, comprising: The image acquisition and publishing module is used to acquire visible light images and infrared thermal imaging images of the same scene and publish them via ROS topics; The environmental perception and modal decision-making module subscribes to visible light images and infrared thermal imaging images, calculates the brightness entropy and mean local contrast of the visible light images, and the standard deviation of the temperature distribution of the infrared thermal imaging images. Based on the brightness entropy, mean local contrast, and standard deviation of the temperature distribution, it generates visible light modal confidence scores and infrared modal confidence scores, and stores them in a historical confidence score queue. The environmental perception and modal decision-making module includes a temporal smoothing unit, which performs temporal smoothing judgment based on the historical confidence score queue. When the judgment result meets the preset modal switching conditions, the current detection mode is switched; otherwise, the current detection mode is maintained. The manual intervention interface is used to determine whether a mode switching command has been received. If a mode switching command is received and the command is not an automatic mode command, the system will force a switch to the corresponding detection mode according to the command and enter the heterogeneous detection engine. If no mode switching command is received or the command is an automatic mode command, the timing smoothing unit will be triggered to perform a judgment. The detection modes include visible light single-mode detection mode and cross-mode fusion detection mode. The heterogeneous detection engine is used to select the corresponding detection sub-network to perform fire detection on visible light images and / or infrared thermal imaging images according to the determined current detection mode, and output the fire detection results, which include fire location, confidence level and bounding box information. The system status publishing module is used to publish fire detection results via ROS topics.

[0048] For specific limitations regarding the ROS-based dynamic modal switching forest fire detection system, please refer to the limitations of the ROS-based dynamic modal switching forest fire detection method described above, which will not be repeated here. The modules in the aforementioned ROS-based dynamic modal switching forest fire detection system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0049] The foregoing has provided a detailed description of the ROS-based dynamic modal switching forest fire detection method and system provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention; the descriptions of the embodiments are merely for the purpose of helping to understand the core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A ROS-based method for detecting forest fire points with dynamic modal switching, characterized in that, The method includes the following steps: S100: Acquire visible light images and infrared thermal images of the same scene and publish them via ROS topics; S200: Subscribe to visible light images and infrared thermal imaging images, calculate the brightness entropy and local contrast mean of the visible light images, and the temperature distribution standard deviation of the infrared thermal imaging images, generate visible light mode confidence and infrared mode confidence based on the brightness entropy, local contrast mean, and temperature distribution standard deviation, and store them in the historical confidence queue. S300: Determine whether a mode switching command has been received. If a mode switching command has been received and the mode switching command is not an automatic mode command, then force a switch to the corresponding detection mode according to the mode switching command and proceed to S500. If no mode switching command has been received or the mode switching command is an automatic mode command, then proceed to S400. The detection modes include visible light single-mode detection mode and cross-mode fusion detection mode. S400: Perform time-series smoothing judgment based on historical confidence queue. When the judgment result meets the preset mode switching conditions, switch the current detection mode; otherwise, maintain the current detection mode. S500: Based on the current detection mode determined by S300 or S400, select the corresponding detection sub-network to perform fire detection on visible light images and / or infrared thermal imaging images, and publish the fire detection results through the ROS topic. The fire detection results include fire location, confidence level, and bounding box information.

2. The method according to claim 1, characterized in that, The formula for calculating the luminance entropy in S200 is: in, For brightness entropy, To convert a visible light image to grayscale, the grayscale levels are... The probability of; The formula for calculating the mean local contrast is: in, This is a grayscale image obtained by converting a visible light image. Let L be the Laplace operator, and L be the pair of operators. The response image obtained after applying the Laplacian operator. This represents the local contrast average. To calculate the average value of all pixel values ​​in the input matrix; The formula for calculating the standard deviation of temperature distribution is: in, The standard deviation of the temperature distribution. This is an infrared thermal imaging image. and These are the height and width of the infrared thermal image, respectively. The average temperature of the infrared thermal imaging image. , These are the spatial coordinates of pixels in the image.

3. The method according to claim 2, characterized in that, In S200, the brightness entropy, local contrast mean, and temperature distribution standard deviation are concatenated into an environmental feature vector by a multilayer perceptron, which is mapped to the original modal confidence. The normalized visible light modal confidence and infrared modal confidence are output by the Softmax function, and the sum of the visible light modal confidence and infrared modal confidence is 1. The multilayer perceptron includes a fully connected layer, a ReLU activation layer, and a fully connected layer in sequence.

4. The method according to claim 3, characterized in that, In S200, the luminance entropy, mean local contrast, and standard deviation of temperature distribution are calculated once every N frames, where N ranges from 5 to 10.

5. The method according to claim 1, characterized in that, In the S300, mode switching commands are received via ROS topics or mode_switch. Command value 0 indicates a forced switch to visible light single-mode detection mode, command value 1 indicates a forced switch to cross-mode fusion detection mode, and command value 2 indicates an automatic mode command. When a command value of 0 or 1 is received, the system immediately switches to the corresponding detection mode and maintains it until a new command is received or the system restarts.

6. The method according to claim 1, characterized in that, The preset switching conditions in S400 are: When the visible light mode confidence of all frames in the historical confidence queue is greater than the first threshold and the infrared mode confidence is less than the second threshold, a switch to visible light single-mode detection mode is triggered; wherein, the value range of the first threshold is 0.6 to 0.8 and the value range of the second threshold is 0.2 to 0.

4.

7. The method according to claim 6, characterized in that, The S400 includes: If the current detection mode is cross-modal fusion detection mode and the preset switching conditions are met, then switch to visible light single-modal detection mode; If the current detection mode is visible light single-modal detection mode, and the M consecutive frames in the historical confidence queue do not meet the preset switching conditions, then switch back to cross-modal fusion detection mode; where M is a preset positive integer, with a value range of 3 to 10; Otherwise, maintain the current detection mode.

8. The method according to claim 7, characterized in that, The visible light single-modal detection mode performs fire detection through a visible light single-modal detection sub-network. The visible light single-modal detection sub-network is built based on the YOLO series models. It uses the CSPDarknet backbone network to extract visible light image features, performs feature fusion through a path aggregation network, and outputs the fire detection results through the detection head. The visible light single-modal detection sub-network only takes visible light images as input. The cross-modal fusion detection mode performs fire detection on visible light images and infrared thermal imaging images through a cross-modal fusion detection sub-network. This sub-network uses a visible light backbone network and a thermal infrared backbone network to extract features from the visible light image and infrared thermal imaging image, respectively. These features are then processed sequentially by a learnable modal dictionary module, a multi-scale cross-modal alignment module, and a fire detection fusion module. The fire detection result is then output through the detection head. The learnable modality dictionary module constructs a learnable feature prototype dictionary for each modality, generates query weights through a spatial attention mechanism, reconstructs feature maps by weighted aggregation of dictionary prototypes, and fuses them with the original features through residual connections to compensate for cross-modal semantic differences. The multi-scale cross-modal alignment module predefines multiple mesh sizes with different granularities, performs multi-scale feature transformation on infrared and visible light features, calculates cross-scale similarity maps, generates alignment weights, and achieves implicit spatial alignment from the infrared feature space to the visible light feature space. The fire detection fusion module inputs aligned features into the detection head to generate an initial fire confidence heatmap. The heatmap is then upsampled at multiple scales. The upsampled heatmap is then concatenated with the aligned features at the corresponding scale along the channel dimension. After convolution processing and activation function generation, gate weights are generated. Based on the gate weights, the infrared features and visible light aligned features are adaptively weighted and fused to control the contribution ratio of different modal features at each scale, and the refined fused features are output.

9. The method according to claim 8, characterized in that, When switching from cross-modal fusion detection mode to visible light single-modal detection mode, the system maintains a cache of infrared feature maps from the most recent frames to provide thermal history prior information when switching back to cross-modal fusion detection mode, thereby accelerating the fire point localization of the cross-modal fusion detection sub-network.

10. A ROS-based dynamic modal switching forest fire detection system, characterized in that, include: The image acquisition and publishing module is used to acquire visible light images and infrared thermal imaging images of the same scene and publish them via ROS topics; The environmental perception and modal decision-making module subscribes to visible light images and infrared thermal imaging images, calculates the brightness entropy and mean local contrast of the visible light images, and the standard deviation of the temperature distribution of the infrared thermal imaging images. Based on the brightness entropy, mean local contrast, and standard deviation of the temperature distribution, it generates visible light modal confidence scores and infrared modal confidence scores, and stores them in a historical confidence score queue. The environmental perception and modal decision-making module includes a temporal smoothing unit, which performs temporal smoothing judgment based on the historical confidence score queue. When the judgment result meets the preset modal switching conditions, the current detection mode is switched; otherwise, the current detection mode is maintained. The manual intervention interface is used to determine whether a mode switching command has been received. If a mode switching command is received and the command is not an automatic mode command, the system will force a switch to the corresponding detection mode according to the command and enter the heterogeneous detection engine. If no mode switching command is received or the command is an automatic mode command, the timing smoothing unit will be triggered to perform a judgment. The detection modes include visible light single-mode detection mode and cross-mode fusion detection mode. The heterogeneous detection engine is used to select the corresponding detection sub-network to perform fire detection on visible light images and / or infrared thermal imaging images according to the determined current detection mode, and output the fire detection results, which include fire location, confidence level and bounding box information. The system status publishing module is used to publish fire detection results via ROS topics.

Images