Fire image detection and segmentation method based on end-to-end unified framework and physical knowledge embedding

Abstract

The invention relates to the technical field of computer vision and fire monitoring, and provides a fire image detection and segmentation method based on an end-to-end unified framework and physical knowledge embedding. Aiming at the problems of computation redundancy, feature segmentation and strong dependence on visible light caused by traditional staged processing, the invention provides the following technical scheme: constructing an end-to-end network comprising an Officient Hybrid Ender encoder and a mask-dino decoder, and realizing global modeling and cross-scale feature fusion through a single-layer Transform; thermal imaging physical knowledge embedding is innovatively introduced, and three fusion modes of pixel-level addition, feature-level Embedding and interactive learning are adopted; a lightweight single-layer Transform architecture and a multi-task loss function are designed, and GIOU, Dice and a joint detection segmentation hybrid matching strategy are combined. According to the method, a bounding box and mask prediction are synchronously generated through a unified query mechanism, the adaptability of a low-illumination scene is enhanced by using thermal imaging data, and the training efficiency is improved by decoupling bounding box loss. According to the invention, the real-time performance, robustness and precision of fire monitoring are significantly improved in a complex scene.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and fire monitoring, and provides a fire image detection and segmentation method based on an end-to-end unified framework and physical knowledge embedding. Background Art

[0002] Fires are a frequent occurrence in today's society. Whether they are raging forest blazes caused by natural factors or human negligence, or urban fires caused by electrical failures or illegal use of fire, they pose a significant threat to people's lives, property, and the ecological environment. Traditional fire monitoring methods, such as smoke sensors and temperature sensors, have numerous limitations, often resulting in false alarms and missed alarms, making them unable to meet the urgent need for high-precision, real-time fire monitoring in modern society. However, the rapid development of computer vision technology and deep learning algorithms has brought new opportunities to the field of fire monitoring. By accurately detecting and segmenting key features such as fire points and smoke in images, early warning and rapid response to fires can be achieved, effectively reducing the damage caused by fires.

[0003] Limitations of traditional methods

[0004] In the field of fire monitoring, traditional image detection and segmentation methods use a phased processing model, separating the detection and segmentation tasks. Taking the classic Mask R-CNN algorithm as an example, it first detects candidate regions that may contain fire targets through a region proposal network (RPN), and then segments these candidate regions one by one. While this approach achieves target detection and segmentation to a certain extent, it suffers from significant computational redundancy. In practical applications, especially for fire monitoring scenarios that require real-time response, such as real-time monitoring of forest fires, the time delay caused by this phased processing makes it difficult for the system to meet the needs of rapid early warning, and the optimal time for fire fighting may be missed.

[0005] Traditional methods have limitations in feature utilization. Most rely on visible light imagery for analysis, which is susceptible to interference from lighting conditions and occlusion. At night or in dimly lit environments, the quality of visible light images degrades significantly, making it difficult to accurately extract fire-related features. The presence of obstructions such as smoke and branches at the fire scene also hinders the identification of fire points and smoke. Traditional methods lack in-depth utilization of the physical characteristics of fires, focusing solely on the visual features of images while ignoring important physical information such as thermal radiation and temperature changes generated during the fire process.

[0006] Traditional methods have limited adaptability in complex scenarios. When smoke and the surrounding background have similar colors and textures, traditional algorithms can easily misidentify smoke as background, or vice versa, resulting in a high rate of false positives. For small targets, such as the tiny ignition source in an early-stage fire, traditional methods often struggle to accurately detect them due to limitations in their detection mechanisms. This can lead to early fire signals being overlooked, delaying fire detection and resolution.

[0007] With the development of technology, end-to-end deep learning frameworks have gradually been applied to fire monitoring, but there are still some areas that need to be improved. The DETR series has achieved global modeling of target detection tasks by introducing the Transformer structure, abandoning traditional manually designed components such as anchor boxes and non-maximum suppression. When processing fire images, although the DETR series models can capture long-distance dependencies in the image, due to the high computational complexity of the Transformer's self-attention mechanism, the amount of calculation will increase significantly when faced with high-resolution images, resulting in reduced model operation efficiency. DETR also has shortcomings in small target detection performance. For small-sized fire spots or smoke that may appear in fire scenes, the detection effect is not ideal.

[0008] The YOLO (You Only Look Once) family of algorithms is widely used in the field of object detection for its rapid detection speed. While combining YOLO with ∪-Net for fire image detection and segmentation improves processing speed to a certain extent, this tandem detection and segmentation architecture still suffers from feature fragmentation. During information transfer, features extracted in the detection phase are not well shared with the segmentation phase, resulting in insufficient collaborative optimization between the two tasks. This prevents them from fully leveraging their respective strengths, impacting the final detection and segmentation accuracy.

[0009] To overcome these challenges, embedding physical knowledge into deep learning models is crucial. Thermal imaging data can reveal the temperature distribution of fire spots. Compared to visible light images, it is not restricted by lighting conditions. Even in low-light environments or under obstruction, it can clearly show the location and outline of fire targets. For nighttime forest fire monitoring, thermal imaging cameras can easily capture fires hidden in the trees, addressing the limitations of visible light images in such scenarios. Incorporating thermal imaging into the model helps it better understand the essential characteristics of fires and analyze and judge fire phenomena in the images from a physical perspective. Summary of the Invention

[0010] The purpose of the present invention is to solve the problems of computational redundancy, feature fragmentation and strong dependence on visible light caused by staged processing in traditional fire detection and segmentation methods. The present invention realizes the collaborative optimization of detection and segmentation tasks through an end-to-end unified framework combined with physical knowledge embedding (such as thermal imaging data fusion), thereby improving robustness, real-time performance and accuracy in complex scenarios.

[0011] In order to achieve the above-mentioned purpose, the present invention adopts the following technical means:

[0012] The present invention provides a fire image detection and segmentation method based on an end-to-end unified framework and physical knowledge embedding, comprising the following steps:

[0013] Step 1: Build an end-to-end unified network architecture, which includes an encoder module and a decoder module:

[0014] The encoder module uses the Efficient Hybrid Encoder to model the global dependencies of the input multi-scale feature maps, processes the high-level semantic feature maps through the AIFI module of the single-layer Transformer encoder, and combines the cross-scale feature fusion module CCFM to progressively fuse high-level semantic features with mid- and low-level detail features;

[0015] The decoder module, based on the mask-dino cascade prediction mechanism, generates bounding box predictions and mask predictions synchronously through a unified query mechanism;

[0016] Step 2: Implement physical knowledge embedding, including:

[0017] Direct addition embedding method: Thermal imaging image I t With visible light image I v At the pixel level, the formula is:

[0018] I 融合 =λI t +(1-λ)I v

[0019] Among them, λ is a weight parameter;

[0020] Embedding method: embed the thermal imaging image feature F t and visible light image feature F v Fusion, the formula is:

[0021] F 融合 =αF t +βF v

[0022] Among them, α and β are weight coefficients;

[0023] Interactive learning embedding method: Through real-time interaction between thermal imaging and visible light image features, the formula is:

[0024] F 融合 =αF t +(1-α)F v

[0025] α=σ(W·[F t ; F v ])

[0026] Where α is a learnable parameter and W is the weight matrix;

[0027] Step 3: Optimize the network through lightweight design and multi-task loss function. The lightweight design uses a single Transformer layer, and the multi-task loss function includes a hybrid matching strategy of GIOU loss, Dice loss, and joint detection and segmentation loss.

[0028] Step 4: Output the detection frame and segmentation mask of the fire target to achieve real-time fire monitoring and early warning.

[0029] In the above solution, the construction of the end-to-end unified network architecture described in step 1 further includes:

[0030] Step 1.1: The encoder module processes the input multi-scale feature map and performs global dependency modeling on the high-semantic feature map through the AIFI module of the single-layer Transformer encoder. At the same time, it combines the cross-scale feature fusion module CCFM to achieve progressive upsampling, specifically including:

[0031] Starting from the S5 feature map, the high-level semantic features of the S5 feature map are sequentially fused with the mid- and low-level detail features of the S4 feature map and the S3 feature map through skip connections to form a multi-scale feature interaction;

[0032] In the fusion process, feature map addition or splicing operations are used to avoid information attenuation;

[0033] Step 1.2: The decoder module initializes anchor box and mask predictions through a unified query mechanism based on the mask-dino cascade prediction mechanism, specifically including:

[0034] Two prediction heads are introduced at the encoder output, one for detection and the other for segmentation tasks:

[0035] The classification score of each tag is used as a confidence indicator to filter high-confidence features and input them into the decoder;

[0036] The filtered features are used for bounding box regression and dot product operation with high-resolution feature maps to predict segmentation masks;

[0037] The predicted bounding boxes and masks are supervised by the ground truth and serve as the initial anchor boxes for the decoder;

[0038] The segmentation mask is implemented by the dot product operation between the query and the high-resolution feature map, and only the semantic similarity between pixels needs to be evaluated;

[0039] By unifying the initial stage after query selection, the accuracy of mask prediction is higher than that of bounding box prediction, and a better anchor box initialization scheme is derived based on the mask.

[0040] In the above scheme, step 1.1 includes:

[0041] 1.1.1: Extract multi-scale information from the three feature maps output by the backbone network: S3 with 1 / 8 resolution, S4 with 1 / 16 resolution, and S5 with 1 / 32 resolution.

[0042] S5 is used as a high-level semantic feature to perform channel compression through 1×1 convolution and then flattened into a sequence form with a dimension of [d, HW] to adapt to the sequence processing mode of Transformer, where HW represents the height and width of the spatial resolution of the feature map;

[0043] The flattened S5 feature sequence is subjected to two-dimensional sinusoidal position encoding to preserve the spatial position information;

[0044] The AIFI module of a single-layer Transformer encoder performs global self-attention calculation on the S5 features to model long-range dependencies;

[0045] 1.1.2: Progressive upsampling and feature fusion are achieved through the cross-scale feature fusion module CCFM:

[0046] The self-attention-enhanced S5 features are upsampled to S4 resolution through bilinear interpolation and added element-wise to the S4 features that have not been processed by Transformer;

[0047] After upsampling to S3 resolution, it is concatenated with the S3 features and channel aligned through 1×1 convolution.

[0048] In the above scheme, step 3 includes:

[0049] Step 3.1: The lightweight design uses a single-layer Transformer encoder layer and a feature fusion module, specifically including:

[0050] The AIFI module of a single-layer Transformer encoder is used to model global dependencies of high-semantic feature maps, avoiding computational redundancy caused by deep stacking.

[0051] The feature fusion module uses progressive upsampling to concatenate the S5 features of high-level semantic features with the S3 and S4 features of mid- and low-level detail features, and then aligns the channels.

[0052] Step 3.2: The multi-task loss function includes a hybrid matching strategy of GIOU loss, Dice loss and joint detection and segmentation loss, specifically including:

[0053] The loss function consists of two parts: detection branch loss and segmentation branch loss;

[0054] After determining the correspondence between the prediction and the real object through Hungarian matching, the weighted loss is calculated for the prediction that matches successfully;

[0055] The hybrid matching strategy of joint detection and segmentation loss refers to adding segmentation loss to the matching cost to ensure that the predicted box is consistent with the mask space and semantics;

[0056] The matching cost formula is: cls L cls +λ box L box +λ mask L mask λ

[0057] Where: cls ,λ box ,λ mask is the weight coefficient of classification loss, bounding box loss and mask loss;

[0058] L cls , L box and L mask They are classification loss, bounding box loss and mask loss respectively;

[0059] Hybrid matching strategy: In the original binary matching, in addition to the bounding box loss and classification loss, the mask prediction loss L is introduced mask , to enhance the accuracy and consistency of matching results for a single query;

[0060] Decoupled bounding box prediction: During the hybrid matching process, the bounding box loss of the background class is set to the mean of the entity class loss, specifically:

[0061] The bounding box prediction process of the background category is still used to locate the valid area and extract features with the help of deformable attention, but its bounding box prediction loss is no longer calculated;

[0062] In the hybrid matching process, the bounding box loss of the background category is set to the mean of the instance-level category bounding box loss, where the instance-level category bounding box loss is the bounding box loss of all non-background category instances. The formula is as follows:

[0063]

[0064] in is the mean of instance-level category bounding box losses, is the bounding box loss for each instance-level category, N is the number of instance-level categories, and the instance-level category bounding box loss refers to the loss of other categories except the background class;

[0065] This decoupled design can speed up training and bring additional performance improvements to panoptic segmentation tasks.

[0066] Because the present invention adopts the above technical means, it has the following beneficial effects:

[0067] 1. Propose a unified detection and segmentation framework to achieve coordinated detection of fire points and smoke

[0068] Traditional fire detection methods typically separate the detection and segmentation of fire points and smoke, resulting in insufficient feature extraction and information utilization, which affects overall detection accuracy. This paper proposes an end-to-end unified framework based on the combination of object detection and image segmentation. This framework enables the two tasks to share deep features and enhances the synergy of detection and segmentation through an efficient information exchange mechanism, enabling accurate detection and segmentation of fire points and smoke in various environments.

[0069] 2. Introducing physical knowledge embedding to improve the robustness of fire detection

[0070] Existing deep learning-based fire detection methods primarily rely on data-driven feature learning, which is susceptible to background interference (such as solar reflection and industrial smoke), reducing detection accuracy. To address this issue, this paper embeds the physical characteristics of fire points and smoke (such as the temperature distribution of fire points) into the deep learning framework, enabling the model to use physical prior information to identify fire areas, thereby improving the ability to distinguish non-fire targets and enhancing detection reliability.

[0071] 3. Traditional fire detection methods typically handle the detection and segmentation of fire points and smoke separately. This separation can easily lead to insufficient feature extraction, information silos, and wasted computational resources, making it difficult to strike a balance between feature expression and detection accuracy. To this end, this paper proposes a unified detection and segmentation model that organically integrates object detection and image segmentation tasks into a unified framework. This model fully leverages the complementary strengths of detection and segmentation, achieving deep feature sharing and efficient information exchange, significantly improving the recognition of fire points and smoke in various complex scenarios. Experimental results on the drone fire dataset show that compared with the single-task target detection model RT-DETR, the average AP of fire points and smoke are improved by 13.4% and 3.8% respectively, the average IOU is improved by 14.9% and 4.4% respectively, and the FPS is improved by 183.3% compared with the baseline network MaskDINO; compared with the single-task target detection model RT-DETR, the average AP of fire point and smoke detection are improved by 19.9% ​​and 38.3% respectively; compared with the single-task instance segmentation Mask-RCNN, the average IOU is improved by 17.5% and 31.5% respectively.

[0072] 4. Traditional deep learning methods rely primarily on image data to drive learning features, while ignoring prior information about the physical properties of fires. This makes the model susceptible to interference from non-fire targets. This paper embeds the physical properties of fires (such as temperature distribution) into the deep learning framework to guide the model to more accurately identify fire areas. This paper proposes three embedding methods for thermal imaging images: direct additive embedding;

[0073] Embedding method; interactive learning embedding method. Experimental results on a drone fire dataset show that all three physical embedding methods improve performance compared to the baseline model to varying degrees. The interactive learning embedding method achieves the greatest improvement, with fire point detection and segmentation improving by 9.95% AP and 9.24% IOU, respectively, and smoke detection improving by 5.25% AP and 3.23% IOU, respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 This is a diagram of the end-to-end unified framework network structure of the present invention;

[0075] Figure 2 Three physical embedding methods of the present invention

[0076] Figure 3 This is an example of an autonomous dataset of the present invention;

[0077] Figure 4 A visualization diagram of fire point and smoke detection and segmentation of the end-to-end unified framework algorithm of the present invention;

[0078] Figure 5 The box-loss curve diagram of the end-to-end unified framework algorithm training process of the present invention;

[0079] Figure 6 This is a seg-loss curve diagram of the end-to-end unified framework algorithm training process of the present invention;

[0080] Figure 7 The AP curve diagram of the end-to-end unified framework algorithm training process of the present invention;

[0081] Figure 8 Visualization diagrams of fire point and smoke detection and segmentation using the three physical embedding methods of the present invention;

[0082] Figure 9 The AP curve diagram of the training process of the three physical embedding methods of the present invention. DETAILED DESCRIPTION

[0083] The following is a detailed description of the embodiments of the present invention. Although the present invention will be described and illustrated in conjunction with certain specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, modifications or equivalent substitutions of the present invention are intended to fall within the scope of the claims of the present invention.

[0084] In addition, in order to better illustrate the present invention, numerous specific details are given in the following detailed description. It will be understood by those skilled in the art that the present invention can also be implemented without these specific details.

[0085] The present invention provides a fire image detection and segmentation method based on an end-to-end unified framework and physical knowledge embedding, comprising the following steps:

[0086] Step 1: Build an end-to-end unified network architecture, which includes an encoder module and a decoder module:

[0087] The encoder module uses the Efficient Hybrid Encoder to model the global dependencies of the input multi-scale feature maps, processes the high-level semantic feature maps through the A-level module of the single-layer Transformer encoder, and combines it with the cross-scale feature fusion module CCFM to progressively fuse high-level semantic features with mid- and low-level detail features;

[0088] The Efficient Hybrid Encoder takes multi-scale feature maps extracted by a backbone network (such as ResNet and HGNet) as input. Specifically, three feature maps, S3 (1 / 8 resolution), S4 (1 / 16 resolution), and S5 (1 / 32 resolution), are output by the backbone network and fed into the encoder module. S5, as a high-level semantic feature, has a larger receptive field but lower spatial resolution, while S3 and S4 retain more detailed information. First, the S5 feature map undergoes channel compression (for example, from 2048 dimensions to 256 dimensions) via a 1×1 convolution. It is then flattened into a sequence (of dimension [d, HW]) to accommodate the Transformer's sequential processing mode. To preserve spatial position information, the encoder applies a two-dimensional sinusoidal positional encoding to the flattened feature sequence, enabling the model to perceive the relative positional relationships between feature points. Unlike traditional DETR, which uses a six-layer Transformer encoder, the Efficient Hybrid Encoder uses only a single Transformer encoder layer (comprising a multi-head self-attention module and a feed-forward network (FFN). This design is based on the observation that high-level features (S5) already possess strong semantic information. Using the Attention-based Intra-scale Feature Interaction (AJIU) module, a single-layer global self-attention mechanism, it can effectively model long-range dependencies between feature points, avoiding the computational redundancy associated with deep stacking. Specifically, the self-attention mechanism dynamically aggregates global contextual information for each feature point through interactive computations between query, key, and value matrices, thereby improving the representation of occluded and small objects. To leverage the advantages of multi-scale features, the encoder introduces a Cross-Scale Feature Fusion Module (CCFM). The self-attention-enhanced S5 features are first upsampled to S4 resolution via bilinear interpolation and element-wise added to the unprocessed S4 features. They are then upsampled again to S3 resolution, concatenated with the S3 features, and channel-aligned via a 1×1 convolution. This process effectively combines high-level semantic information with mid- and low-level spatial details through progressive upsampling and feature fusion, ultimately outputting an enhanced feature map containing multi-scale contextual information.

[0089] The decoder module, based on the mask-dino cascade prediction mechanism, generates bounding box predictions and mask predictions simultaneously through a unified query mechanism. While inheriting the advantages of the DINO framework's cascade prediction, this decoder introduces a mask prediction branch, forming a feature sharing and joint optimization architecture for detection and segmentation tasks.

[0090] Step 2: Implement physical knowledge embedding, including:

[0091] Direct Additive Embedding: The core idea of ​​direct additive embedding is to directly superimpose the thermal image with other images (such as visible light images) at the pixel level. Specifically, for the thermal image I t and visible light image I v , we first resize them to the same size and resolution, and then overlay them according to the following formula;

[0092] I 融合 =λI t +(1-λ)I v

[0093] Here, λ is a weight parameter that controls the relative contributions of thermal and visible light images during the fusion process. The value range of the weight parameter λ is [0, 1], and its physical meaning is the contribution ratio of thermal information to the fused image. When λ = 1, a pure thermal image is output, while when λ = 0, a pure visible light image is output.

[0094] Embedding method: This method first extracts features from thermal images and then embeds them into the subsequent processing steps of the model to achieve deeper physical knowledge embedding:

[0095] F 融台 =αF t +βF v

[0096] The extracted thermal imaging image features F t and the feature F of the visible light image v Fusion is performed, where α and W are weight coefficients used to balance the contribution of the two features.

[0097] Interactive learning embedding method: This method breaks the traditional independent feature extraction mode and introduces interactive operations in Backbone network processing. It enhances the representation ability of key targets through real-time interactive supplementation of thermal imaging and visible light image features:

[0098] F 融合 =αF t +(1-α)F v

[0099] α=σ(W·[F t ; F v ])

[0100] Where α is a learnable parameter and W represents .

[0101] Step 3: Optimize the network through lightweight design and multi-task loss function. The lightweight design uses a single Transformer layer, and the multi-task loss function includes a hybrid matching strategy of GIOU loss, Dice loss, and joint detection and segmentation loss.

[0102] The loss function consists of two parts: detection branch loss and segmentation branch loss. After determining the correspondence between the prediction and the real object through Hungarian matching, a weighted loss is calculated for the prediction that matches successfully. Traditional DETR only uses detection loss (classification + box) for Hungarian matching, and the segmentation result may be inconsistent with the detection box. The hybrid matching strategy of joint detection and segmentation loss refers to adding segmentation loss to the matching cost. Adding segmentation loss to the matching cost can ensure that the matched prediction box is spatially and semantically consistent with the mask. The matching cost is:

[0103] λ cls L cls +λ box L box +λ mask L mask

[0104] Among them L cls , L box and L mask They correspond to classification loss, bounding box loss, and mask loss respectively, and λ is the corresponding weight coefficient.

[0105] Step 3: Output the detection frame and segmentation mask of the fire target to achieve real-time fire monitoring and early warning.

[0106] The construction of the end-to-end unified network architecture described in step 1 further includes:

[0107] Step 1.1: The encoder module processes the input multi-scale feature map and models the global dependency of the high-semantic feature map through the A-old module of the single-layer Transformer encoder. At the same time, it combines the cross-scale feature fusion module CCFM to achieve progressive upsampling, which includes:

[0108] Starting from the S5 feature map, the high-level semantic features of the S5 feature map are sequentially fused with the mid- and low-level detail features of the S4 feature map and the S3 feature map through skip connections to form a multi-scale feature interaction;

[0109] In the fusion process, feature map addition or splicing operations are used to avoid information attenuation;

[0110] Step 1.2: The decoder module initializes anchor box and mask predictions through a unified query mechanism based on the mask-dino cascade prediction mechanism, specifically including:

[0111] We introduce three prediction heads at the encoder output, one for classification, one for detection, and one for segmentation. It is worth noting that these prediction heads have the same structure as the prediction heads in the decoder. The classification score of each token is used as a confidence indicator to filter features with higher confidence and input them into the decoder as content queries. These filtered features are not only used for bounding box regression, but also perform dot product operations with high-resolution feature maps to predict segmentation masks. The predicted bounding boxes and masks will be supervised by the corresponding ground truth and serve as the initial anchor boxes for the decoder.

[0112] Segmentation mask prediction is achieved through the dot product operation between the query and the high-resolution feature map, which only requires evaluating the semantic similarity between pixels. In contrast, object detection requires directly regressing the bounding box coordinates of the object in the image. Therefore, in the initial stage after unified query selection, the accuracy of mask prediction is much higher than that of bounding box prediction. Based on this property, after unified query selection, we derive bounding boxes from the predicted masks and use them as a more optimized anchor box initialization scheme in the decoder. Through this efficient task collaboration mechanism, the enhanced anchor box initialization can significantly improve object detection performance.

[0113] The cross-scale feature fusion module (CCFM) of the Efficient Hybrid Encoder uses progressive upsampling to concatenate high-level semantic features (S5) with mid- and low-level detail features (S3 / S4) and then align the channels. The mask-dino decoder uses a unified query mechanism to screen high-confidence features, initialize anchor box and mask predictions, and perform hybrid matching and decoupled bounding box predictions. Hybrid matching means that in addition to the bounding box loss and classification loss in the original binary matching, we introduce a mask prediction loss to enhance the accuracy and consistency of the matching results for a single query. Decoupled bounding box prediction means that during the hybrid matching process, the bounding box loss of the background category is set to the mean of the instance-level category bounding box loss. This decoupled design can speed up training.

[0114] The lightweight design of the end-to-end unified framework includes a single-layer Transformer encoder layer and a feature fusion module.

[0115] The method improves the generalization of the model through data augmentation strategies, including multi-scale scaling, color perturbation, random erasure, and cross-modal enhancement, to adapt to complex scenarios such as smoke occlusion and sudden changes in lighting.

[0116] The decoupled prediction optimization strategy of the method includes removing the bounding box loss for background categories to reduce redundant computations, and enhancing anchor box initialization through masks, leveraging the high accuracy of mask prediction to optimize the detection task. In the panoptic segmentation task, predicting bounding boxes for background categories (such as the sky) is not only unnecessary but also inefficient. Since the bounding box shapes derived from the true masks of these categories are often highly irregular and usually cover the entire image, they may be misleading for entity-level detection and segmentation tasks. Therefore, we remove the bounding box loss and bounding box matching for the background category. Specifically, although the bounding box prediction process for the background category is still used to locate the valid area and extract features with the help of deformable attention, its bounding box prediction loss is no longer calculated; in the hybrid matching process, the bounding box loss for the background category is set to the mean of the entity category loss. This decoupled design can speed up training and bring additional performance improvements to the panoptic segmentation task.

[0117] The real-time optimization of the proposed method is achieved through an efficient hybrid encoder, which reduces computational redundancy while ensuring the applicability of the model in real-time UAV monitoring scenarios.

[0118] The experimental validation of the method is based on the fusion of FLAME and BowFire public datasets with autonomously collected drone visible light and thermal imaging data, totaling 11,500 images, including complex scenes such as smoke occlusion and sudden changes in lighting.

[0119] Example 1

[0120] 1. Core Innovations

[0121] 1.1 End-to-End Unified Architecture

[0122] In order to solve the computational redundancy and feature fragmentation problems in the traditional staged processing mode, the present invention proposes an end-to-end unified architecture. This architecture uses an Efficient Hybrid Encoder and a mask-dino decoder, such as Figure 1 As shown in Figure 3, the joint optimization of detection and segmentation tasks is achieved.

[0123] The Efficient Hybrid Encoder combines the Transformer's self-attention mechanism with multi-scale feature interaction to reduce computational redundancy. Unlike traditional Transformer architectures, it effectively reduces computational complexity by specifically processing feature maps at different scales. When processing high-resolution images, the traditional Transformer's self-attention computational complexity increases dramatically as the image size increases. However, the Efficient Hybrid Encoder performs self-attention calculations only on high-level semantic feature maps, while combining multi-scale fusion of low-level feature maps. This ensures the capture of global image information while significantly reducing computational complexity, enabling the model to operate more efficiently when processing complex images.

[0124] The mask-dino decoder improves the consistency of detection and segmentation tasks through a unified query mechanism and dynamic convolution strategy. In traditional architectures where detection and segmentation are separated, the two tasks use different feature extraction and processing methods, resulting in insufficient information sharing. The mask-dino decoder, on the other hand, uses a unified query embedding to simultaneously predict the object's bounding box and mask, enabling collaborative optimization of detection and segmentation tasks within the same framework. It also employs a dynamic convolution strategy to adaptively adjust convolution kernel parameters based on different task requirements and input features, further improving the model's adaptability to diverse scenarios and targets.

[0125] 1.2 Physical Knowledge Embedding

[0126] Fire is a complex physical phenomenon, involving multiple physical processes such as heat transfer, combustion reactions, and gas diffusion. The features generated by these physical processes, such as temperature distribution, thermal radiation, and smoke diffusion patterns, provide rich information for fire detection and segmentation. Traditional fire detection methods based on physical models, such as infrared detection technology based on thermal radiation principles, can leverage these physical features to detect fires, but they often suffer from limited detection range and poor adaptability to complex scenarios. While deep learning models possess powerful feature learning capabilities, they lack a deep understanding of the physical processes of fire and are prone to misjudgment in some unique situations.

[0127] Embedding physical knowledge into deep learning models can give full play to the advantages of both. Physical knowledge can provide prior information for the model, helping the model to better understand the physical nature of fire, thereby improving its adaptability to complex scenes and detection accuracy. Deep learning models can use their powerful learning capabilities to automatically extract and learn the relationship between physical features and image features, achieving more accurate detection and segmentation of fire points and smoke. When a fire occurs, according to the physical principles of thermal radiation, the fire point will radiate infrared rays of a specific wavelength. By embedding this physical knowledge into the deep learning model, the model can more accurately identify the location and range of the fire point by learning the characteristics of thermal radiation in infrared images.

[0128] There are three embedding methods for the physical model embedding module. The first is the direct addition method, which directly superimposes the physical information represented by the thermal image with other image information and inputs it into the Backbone network. The second is the Embedding embedding method, in which the thermal image first passes through the Backbone network to extract features, and then the features are embedded into the subsequent processing link of the model through feature fusion. The third is interactive learning embedding, in which the thermal image and other images pass through the Backbone network respectively, and then interactive learning and embedding are carried out between the two to achieve effective integration of physical information. Thermal imaging images can reflect physical information such as the thermal radiation characteristics of the fire point, and the physical characteristics contained therein are introduced into the model through these embedding methods. The three physical embedding methods are as follows:

[0129] Direct Additive Embedding: The core idea of ​​direct additive embedding is to directly superimpose the thermal image with other images (such as visible light images) at the pixel level. Specifically, for the thermal image I t and visible light image I v , we first resize them to the same size and resolution, and then overlay them according to the following formula;

[0130] I 融合 =λI t +(1-λ)I v

[0131] Here, λ is a weight parameter that controls the relative contributions of the thermal image and visible light image during the fusion process. The weight parameter λ ranges from [0, 1] and represents the contribution ratio of the thermal image information to the fused image. When λ<λ, a pure thermal image is output; when λ<λ, a pure visible light image is output.

[0132] Embedding method: This method first extracts features from thermal images and then embeds them into the subsequent processing steps of the model to achieve deeper physical knowledge embedding:

[0133] F 融合=αF t +βF v

[0134] The extracted thermal imaging image features F t and the feature F of the visible light image v Fusion is performed, where α and W are weight coefficients used to balance the contribution of the two features.

[0135] Interactive learning embedding method: This method breaks the traditional independent feature extraction mode and introduces interactive operations in Backbone network processing. It enhances the representation ability of key targets through real-time interactive supplementation of thermal imaging and visible light image features:

[0136] F 融合 =αF t +(1-α)F v

[0137] α=σ(W·[F t ; F v ])

[0138] where α is a learnable parameter.

[0139] 2. Implementation Methods

[0140] 2.1 Network Architecture Design

[0141] 2.1.1 End-to-End Unified Framework

[0142] The end-to-end unified framework of the present invention mainly consists of two parts: an encoder and a decoder. Through a series of innovative designs, it realizes efficient collaborative processing of fire image detection and segmentation tasks.

[0143] In the encoder part, such as Figure 1As shown in the figure, the input is multi-scale feature maps, including S3, S4, and S5 feature maps. These feature maps at different scales contain rich image information from low-level to high-level layers. The S3 feature map retains more detailed information, while the S4 and S5 feature maps have higher semantic information. Processing these multi-scale feature maps through a single-layer Transformer encoder (AIFI module) effectively models global dependencies. Unlike the traditional multi-layer Transformer architecture, the AIFI module adopts a lightweight design, using only a single Transformer layer, significantly reducing computational overhead. When processing high-resolution images, the computational overhead of traditional multi-layer Transformers increases dramatically with image size, resulting in low model efficiency. The AIFI module, however, performs targeted processing on feature maps at different scales, performing self-attention calculations only on high-semantic feature maps (such as the S5 feature map) while integrating information from feature maps at other scales. This ensures that global image information is captured while significantly reducing computational complexity, enabling the model to operate more efficiently when processing complex images.

[0144] The Cross-Scale Feature Fusion Module (CCFM) plays a key role in the encoder. In fire images, high-level semantic features can help identify the overall shape and category of fire spots and smoke, while mid- and low-level detail features help capture subtle information such as the edges of fire spots and the texture of smoke. The CCFM module starts with the S5 feature map and gradually upsamples, fusing the semantic information of the F5 feature map with the detail information of the S4 and S3 feature maps in sequence. During the fusion process, the CCFM module uses a skip connection method to directly add or splice feature maps of different scales. This allows feature information to be efficiently transferred between different scales, avoiding information loss and attenuation, thereby providing a richer and more comprehensive feature representation for subsequent detection and segmentation tasks.

[0145] The decoder part is based on the cascade prediction mechanism of mask-dino, and initializes the anchor box and mask prediction through a unified query mechanism. In the traditional architecture where detection and segmentation are separated, the detection task usually uses the anchor box to predict the position of the target, and the segmentation task uses the mask to determine the shape of the target. This separation method leads to insufficient information sharing between the two tasks and poor synergy. In the unified framework of the present invention, the query embeddings of the detection and segmentation tasks are unified through a unified query mechanism, so that the model can simultaneously predict the bounding box and mask of the target. When processing fire images, the model can obtain the location information and shape information of the fire point and smoke based on the unified query embedding, thereby realizing the close integration of detection and segmentation tasks.

[0146] To further enhance the consistency of detection and segmentation tasks, this paper decouples bounding box prediction and introduces a hybrid matching strategy.

[0147] 2.1.2 Physical Knowledge Embedding

[0148] The physical knowledge embedding module is another key component of the present invention. It effectively fuses the information of thermal imaging and visible light images through interactive learning embedding and weighted fusion techniques, thereby improving the model's ability to understand and utilize the physical characteristics of fires. Figure 2 shown.

[0149] In terms of interactive learning embedding, thermal images and visible light images are separately input into Backbone, which has an interactive mechanism. Thermal images can reflect the temperature distribution of a fire point and clearly show the location and approximate range of the fire source even at night or when obscured by smoke. Visible light images contain rich texture and color information and, in sufficient light, can provide detailed information about the fire scene, such as the shape of the fire point, the color of the smoke, and the direction of its movement. In Backbone's middle layer, the dynamic fusion of temperature and texture information is achieved by adding the features of the thermal and visible light images. In forest fire monitoring, thermal images can accurately locate the fire source, while visible light images can display the texture and morphology of surrounding trees. By adding these features, the model can simultaneously utilize both types of information to more accurately determine the occurrence and development of a fire, as well as the impact of the surrounding environment on the fire.

[0150] The weighted fusion module further integrates the interactive features and feeds them into subsequent detection and segmentation tasks. The importance of thermal imaging and visible light data for fire detection and segmentation may vary in different scenarios. In scenes with dense smoke, thermal imaging data, unaffected by smoke obstruction, can more accurately reflect the location of the fire source. Therefore, the weight of thermal imaging features can be appropriately increased. In scenes with ample light and less smoke, the texture information in visible light data is more helpful in identifying fire points and smoke details. Therefore, the weight of visible light features can be increased accordingly. The weighted fusion module introduces learnable weight parameters, allowing the model to automatically adjust the weights of thermal imaging and visible light features based on the specific scenario, thereby achieving more accurate detection and segmentation of fire targets in different scenarios. These weight parameters are continuously optimized during training using a backpropagation algorithm. This allows the model to adaptively balance the contributions of different modal features based on the characteristics of the input image, enhancing its ability to capture key features such as fire point edges and smoke diffusion patterns, thereby improving the accuracy and reliability of fire detection and segmentation.

[0151] 2.2 Training Strategy

[0152] In terms of training strategy, this paper adopts a series of carefully designed methods to ensure that the model can accurately detect and segment fire targets in complex scenarios, while improving the generalization ability and real-time performance of the model.

[0153] The design of the loss function is crucial to the training strategy. For detection tasks, this paper employs a combination of L1 loss and GIOU loss. L1 loss primarily measures the absolute error in coordinates between the predicted bounding box and the ground-truth bounding box, effectively guiding the model to learn accurate target location information. GIOU loss further considers the degree of overlap between the predicted and ground-truth bounding boxes, as well as their relative positional relationship. Even when the predicted and ground-truth boxes do not completely overlap, it accurately reflects the distance and positional relationship between them. In fire detection, the shape and position of fire spots and smoke may vary. GIOU loss can better adapt to such variations, improving target detection accuracy. For segmentation tasks, cross-entropy loss and Dice loss are employed. Cross-entropy loss is commonly used in classification tasks. It measures the difference between predicted probabilities and true labels, guiding the model to learn the correct category information. Dice loss focuses on image segmentation tasks. By calculating the similarity between the predicted mask and the true mask, it can better capture detailed information about the target in the image and is particularly effective for segmenting irregularly shaped targets such as smoke and fire spots. To enhance the synergy between detection and segmentation tasks, this paper introduces a hybrid matching strategy to jointly optimize the detection and segmentation losses. The loss of the detection task and the loss of the segmentation task are weighted and combined, and the parameters of the detection and segmentation tasks are adjusted simultaneously during the training process, so that the model can achieve good performance on both tasks.

[0154] In order to improve the generalization ability of the model, the present invention adopts a variety of data enhancement techniques. Multi-scale scaling is one of the commonly used methods. It simulates fire scenes at different distances and viewing angles by scaling the input image at different ratios, enabling the model to learn fire characteristics at different scales. Color perturbation increases the diversity of the image by changing the color attributes such as brightness, contrast, and saturation of the image, making the model more adaptable to fire images under different lighting and color conditions. Random erasure technology simulates image occlusion by randomly erasing a part of the image, enabling the model to learn the ability to identify fire targets from partial information.

[0155] 2.3 Experimental Verification

[0156] In order to verify the effectiveness and superiority of the method proposed in the present invention, a series of rigorous experiments were carried out as follows.

[0157] This paper uses the publicly available fire detection datasets FLAME and BowFire, combined with fire video data collected from real-world scenarios to ensure training data diversity and generalization. The datasets cover fire points and smoke samples from various environments, including indoor fires, forest fires, and industrial fires. The datasets are specifically composed as follows: FLAME provides 2,500 annotated images covering forest fires; BowFire contains 5,800 images of fires in indoor and industrial environments; and the self-constructed dataset uses drones to capture 3,200 multi-scale fire scene images, including complex conditions such as smoke obstruction and dynamic lighting. All data is divided into training, validation, and test sets with a ratio of 8:1:1.

[0158] The evaluation of performance indicators is a key part of the experiment. For the fire point detection task, the average precision (AP) is used as the main indicator. The experimental results show that after using the interactive learning embedding technology, the AP of fire point detection reached 44.2%. Compared with the classic MaskDINO method, it has increased by 7.0%. This shows that the method of the present invention has significantly improved the accuracy of fire point detection, can more accurately identify and locate fire points, and reduce missed detections and false detections. In the smoke segmentation task, the intersection over union (IoU) is used as an evaluation indicator. The experimental results show that the smoke segmentation IoU of the method of the present invention reached 51.2%, an increase of 1.6% compared with the baseline method. This shows that the method of the present invention can more accurately segment the smoke area, capture the boundary and shape of the smoke more accurately, and provide a more reliable basis for the subsequent analysis of the smoke diffusion range and trend. In practical applications, especially in scenarios such as real-time monitoring of drones, inference speed is also an important performance indicator.

[0159] All experiments in this paper were conducted on a GPU server equipped with five GPUs. The PyTorch deep learning framework was used for model training. Model training was performed on a high-performance computing server, using mixed-precision training to improve training efficiency. The detailed server configuration is shown in the table below:

[0160]

[0161] 3. Experimental Results and Analysis

[0162] 3.1 Performance Comparison

[0163] In order to intuitively demonstrate the performance advantages of the proposed method, a comprehensive performance comparison is conducted with MaskDINO and a baseline method (without embedding). The specific results are shown in the following table:

[0164]

[0165] As can be seen from the data in the table, in terms of fire point AP, the method of the present invention reached 44.2, which is 7.0 higher than MaskDINO and 4.0 higher than the baseline method. This fully proves that the present invention has made significant progress in the accuracy of fire point detection, can more accurately identify and locate fire points, and effectively reduce missed detections and false detections. In terms of the smoke IoU indicator, the segmentation performance of the method of the present invention is 51.2, which is 5.6 higher than MaskDINO and 1.6 higher than the method without physical embedding. This shows that the present invention performs well in the smoke segmentation task, can more accurately segment the smoke area, and capture the smoke boundary and shape more accurately, providing a more reliable basis for the subsequent analysis of the smoke diffusion range and trend. In terms of inference speed, the method of the present invention can still reach 34FPS when including the segmentation task, which is much higher than MaskDINO's 12FPS, and meets the real-time requirements of scenarios such as real-time monitoring of drones. This means that in practical applications, the method of the present invention can quickly process and analyze the collected images, issue fire warnings in a timely manner, and buy valuable time for fire fighting and control.

[0166] like Figure 4 This is a visualization of the end-to-end unified framework proposed by the present invention to achieve fire point and smoke detection and segmentation. It can be seen that the model can accurately detect and segment fire points and smoke in fire images. Figure 5 、 6 , 7, it can be clearly observed that the target detection and segmentation framework proposed in the present invention shows better comprehensive performance than MaskDINO during the training process. Specifically, in the comparison of the key performance indicator AP (average precision), the proposed method (red dotted line) maintains a similar level to MaskDINO (blue solid line) at the beginning of training, which shows that the search capabilities of the two in the initial parameter space are at the same baseline. However, as the training epochs advance (especially from the 20th epoch), the AP value of the proposed method gradually shows a stable upward trend, and after the 50th epoch, it is stably maintained in the range of 68.5±0.3, while the AP curve of MaskDINO shows obvious fluctuations after the 40th epoch, and finally stabilizes at the level of 66.2±0.7. This difference reflects that the proposed method has a greater advantage in capturing complex target features during long-term training, and its designed dynamic feature fusion module effectively alleviates the problem of feature attenuation in the traditional two-stage method.

[0167] A comparative analysis of segmentation loss (seg-loss) and box regression loss (box-loss) reveals that both methods exhibit a decreasing trend with increasing training epochs, but the proposed method demonstrates significant advantages in terms of convergence speed and final loss value. Specifically, in the seg-loss curve, the proposed method reduces the loss to 0.85 by the 10th epoch, a 30.9% reduction compared to MaskDINO's 1.23 at the same time. This is attributed to the effective integration of features from different levels through its multi-scale semantic alignment module. Regarding box-loss, the proposed method achieves a loss of 0.42 by the 30th epoch, a 27.6% reduction compared to MaskDINO's 0.58. This is primarily attributed to the IoU-aware regression head, which more accurately captures the geometric features of object boundaries. Notably, the proposed method's loss curve exhibits lower variance in the later stages of training (epochs 80-100), indicating greater stability in its optimization process, effectively avoiding the overfitting problem common in traditional methods.

[0168] like Figure 8The visualization diagrams of fire point and smoke detection and segmentation using the three embedding methods proposed in the present invention are shown. The first row is the original visible light image, the second row is the baseline reasoning result diagram, the third row is the physical embedding method reasoning result diagram, the fourth row is the Embedding reasoning result diagram, and the fifth row is the interactive learning embedding reasoning result diagram. The visualization results of the three embedding methods proposed in the present invention on the fire point and smoke detection and segmentation tasks clearly show the performance differences of different methods. The original visible light image in the first row provides a real scene reference for subsequent results, showing the original state of the target to be detected and segmented under natural lighting conditions. The baseline reasoning result diagram in the second row serves as a basic reference. Its detection and segmentation effects are relatively limited. It is not accurate enough in capturing the fire point and smoke features in complex scenes, and some areas are missed or mis-segmented. The physical embedding method reasoning result diagram in the third row improves the shortcomings of the baseline to a certain extent. The physical characteristics assist the model to better identify fire points and smoke, and the segmentation boundaries are clearer, but there is still room for improvement. The Embedding inference result graph in the fourth row further improves detection and segmentation accuracy, can more accurately locate fire points and smoke areas, has stronger adaptability to fire points and smoke features of different shapes and distributions, and has a more refined segmentation effect. The interactive learning embedding inference result graph in the fifth row shows the best performance. The interactive learning mechanism enables the model to be more in-depth in feature extraction and semantic understanding, and the detection of fire points and smoke is more accurate. The segmentation boundaries are highly consistent with the actual targets, and it has good robustness to complex backgrounds and lighting changes. Overall, with the continuous optimization of the embedding method, the detection and segmentation effects of fire points and smoke have gradually improved. The interactive learning embedding method has shown significant advantages in complex scenarios.

[0169] Figure 9 The AP performance comparison of the baseline and three physical embedding methods during the training iteration process is shown. The horizontal axis represents the training round (0-300 epochs), and the vertical axis is the performance index (range 0.0-0.6). The four sets of curves all show a rapid upward trend in the initial stage (0-50 epochs), among which the Embedding embedding method has a faster convergence speed than other algorithms in the early stage. When the training round exceeds 150 epochs, the performance of each method tends to be stable. The baseline series of algorithms finally stabilize in the range of 0.5-0.45, while the interactive learning method maintains its leading advantage with an index value of approximately 0.59. The experimental results show that the improved method proposed in this paper surpasses the baseline solution in both model convergence efficiency and final performance, verifying the effectiveness of the algorithm innovation.

[0170] 3.2 Ablation Experiment

[0171] To further explore the specific contributions of each key component to the model's performance, we conducted a series of carefully designed ablation experiments. By gradually removing or replacing specific components in the model, we observed changes in metrics such as the average precision (AP) of fire point detection and the intersection over union (IoU) of smoke segmentation, thereby clarifying the importance and mechanism of each component.

[0172] The Efficient Hybrid Encoder plays a crucial role in the model. Ablation experiments show that removing the Efficient Hybrid Encoder significantly reduces the model's fire point AP by 3.1%. This demonstrates that the Efficient Hybrid Encoder effectively integrates the Transformer self-attention mechanism with multi-scale feature interactions, reducing computational redundancy and significantly improving the model's fire point detection capabilities. In practical applications, it helps the model better capture global information and multi-scale features in the image, accurately identifying the location and status of fire points.

[0173] The Cross-Scale Feature Fusion (CCFM) module also significantly impacts model performance. When the CCFM module is removed, the IoU for smoke segmentation decreases significantly, by 2.4%. This demonstrates that the CCFM module, through progressive upsampling, fuses high-level semantic features with mid- and low-level detail features, effectively improving smoke segmentation accuracy. When processing smoke images, the CCFM module leverages the strengths of features at different scales to accurately segment smoke regions, providing strong support for subsequent analysis of smoke diffusion.

[0174] Interactive learning embedding is one of the core innovations of this invention. Experimental results show that compared with the method of directly adding thermal imaging and visible light features, the use of interactive learning embedding technology can improve the fire point AP by 2.1% and the smoke Io∪ by 1.0%. This fully proves that interactive learning embedding can more effectively capture key features such as the edge of the fire point and the smoke diffusion morphology through cross-modal interactive learning and weighted feature fusion, thereby significantly improving the performance of the model in fire detection and segmentation tasks. In practical applications, interactive learning embedding technology can enable the model to better integrate the information of thermal imaging and visible light images, accurately judge the development trend of the fire, and provide a more accurate basis for fire prevention and control.

Claims

1. A fire image detection and segmentation method based on an end-to-end unified framework and physical knowledge embedding, characterized by: The following steps are involved: Step 1: Build an end-to-end unified network architecture, which includes an encoder module and a decoder module: The encoder module uses the Efficient Hybrid Encoder to model the global dependencies of the input multi-scale feature maps, processes the high-level semantic feature maps through the AIFI module of the single-layer Transformer encoder, and combines the cross-scale feature fusion module CCFM to progressively fuse high-level semantic features with mid- and low-level detail features; The decoder module, based on the mask-dino cascade prediction mechanism, generates bounding box predictions and mask predictions synchronously through a unified query mechanism; Step 2: Implement physical knowledge embedding, including: Direct addition embedding method: thermal imaging image I t With visible light image I v At the pixel level, the formula is: I 融合 =λI t +(1-λ)I v Among them, λ is a weight parameter; Embedding method: embed the thermal imaging image feature F t and visible light image feature F v Fusion, the formula is: F 融合 =αF t +βF v Among them, α and β are weight coefficients; Interactive learning embedding method: Through real-time interaction between thermal imaging and visible light image features, the formula is: F 融合 =αF t +(1-α)F v α=σ(W·[F t ;F v ]) Where α is a learnable parameter and W is the weight matrix; Step 3: Optimize the network through lightweight design and multi-task loss function. The lightweight design uses a single Transformer layer, and the multi-task loss function includes a hybrid matching strategy of GIOU loss, Dice loss, and joint detection and segmentation loss. Step 4: Output the detection frame and segmentation mask of the fire target to achieve real-time fire monitoring and early warning.

2. The method according to claim 1, wherein: The construction of the end-to-end unified network architecture described in step 1 further includes: Step 1.1: The encoder module processes the input multi-scale feature map and performs global dependency modeling on the high-semantic feature map through the AIFI module of the single-layer Transformer encoder. At the same time, it combines the cross-scale feature fusion module CCFM to achieve progressive upsampling, specifically including: Starting from the S5 feature map, the high-level semantic features of the S5 feature map are sequentially fused with the mid- and low-level detail features of the S4 feature map and the S3 feature map through skip connections to form a multi-scale feature interaction; In the fusion process, feature map addition or splicing operations are used to avoid information attenuation; Step 1.2: The decoder module initializes anchor box and mask predictions through a unified query mechanism based on the mask-dino cascade prediction mechanism, specifically including: Two prediction heads are introduced at the encoder output, one for detection and the other for segmentation tasks: The classification score of each tag is used as a confidence indicator to filter high-confidence features and input them into the decoder; The filtered features are used for bounding box regression and dot product operation with high-resolution feature maps to predict segmentation masks; The predicted bounding boxes and masks are supervised by the ground truth and serve as the initial anchor boxes for the decoder; The segmentation mask is implemented by the dot product operation between the query and the high-resolution feature map, and only the semantic similarity between pixels needs to be evaluated; By unifying the initial stage after query selection, the accuracy of mask prediction is higher than that of bounding box prediction, and a better anchor box initialization scheme is derived based on the mask.

3. The method according to claim 1, wherein: Step 1.1 includes: 1.1.1: Extract multi-scale information from the three feature maps output by the backbone network: S3 with 1 / 8 resolution, S4 with 1 / 16 resolution, and S5 with 1 / 32 resolution. S5 is used as a high-level semantic feature to perform channel compression through 1×1 convolution and then flattened into a sequence form with a dimension of [d, HW] to adapt to the sequence processing mode of Transformer, where HW represents the height and width of the spatial resolution of the feature map; The flattened S5 feature sequence is subjected to two-dimensional sinusoidal position encoding to preserve the spatial position information; The AIFI module of a single-layer Transformer encoder performs global self-attention calculation on the S5 features to model long-range dependencies; 1.1.2: Progressive upsampling and feature fusion are achieved through the cross-scale feature fusion module CCFM: The self-attention-enhanced S5 features are upsampled to S4 resolution through bilinear interpolation and added element-wise to the S4 features that have not been processed by Transformer; After upsampling to S3 resolution, it is concatenated with the S3 features and channel aligned through 1×1 convolution.

4. The method according to claim 1, wherein: Step 3 includes: Step 3.1: The lightweight design uses a single-layer Transformer encoder layer and a feature fusion module, specifically including: The AIFI module of a single-layer Transformer encoder is used to model global dependencies of high-semantic feature maps, avoiding computational redundancy caused by deep stacking. The feature fusion module uses progressive upsampling to concatenate the S5 features of high-level semantic features with the S3 and S4 features of mid- and low-level detail features, and then aligns the channels. Step 3.2: The multi-task loss function includes a hybrid matching strategy of GIOU loss, Dice loss and joint detection and segmentation loss, specifically including: The loss function consists of two parts: detection branch loss and segmentation branch loss; After determining the correspondence between the prediction and the real object through Hungarian matching, the weighted loss is calculated for the prediction that matches successfully; The hybrid matching strategy of joint detection and segmentation loss refers to adding segmentation loss to the matching cost to ensure that the predicted box is consistent with the mask space and semantics; The matching cost formula is: cls L cls +λ box L box +λ mask L mask λ Where: cls ,λ box ,λ mask is the weight coefficient of classification loss, bounding box loss and mask loss; L cls , L box and L mask They are classification loss, bounding box loss and mask loss respectively; Hybrid matching strategy: In the original binary matching, in addition to the bounding box loss and classification loss, the mask prediction loss L is introduced mask , to enhance the accuracy and consistency of matching results for a single query; Decoupled bounding box prediction: During the hybrid matching process, the bounding box loss of the background class is set to the mean of the entity class loss, specifically: The bounding box prediction process of the background category is still used to locate the valid area and extract features with the help of deformable attention, but its bounding box prediction loss is no longer calculated; In the hybrid matching process, the bounding box loss of the background category is set to the mean of the instance-level category bounding box loss, where the instance-level category bounding box loss is the bounding box loss of all non-background category instances. The formula is as follows: in is the mean of the instance-level category bounding box loss, is the bounding box loss for each instance-level category, N is the number of instance-level categories, and the instance-level category bounding box loss refers to the loss of other categories except the background class; This decoupled design can speed up training and bring additional performance improvements to panoptic segmentation tasks.

Images