Fire behavior tracking method based on time sequence correlation optimization

Abstract

The invention relates to the technical field of multi-target tracking, in particular to a fire behavior tracking method based on time sequence correlation optimization, which comprises the following steps: acquiring a fire behavior image of a target area; the fire behavior image is input into a preset fire behavior tracking model, a fire behavior tracking result of the target area is output, and the fire behavior tracking model is used for detecting a flame target, extracting time sequence information and physical characteristics of the flame target and conducting matching optimization, and fire behavior tracking is achieved by generating a virtual track. The method has good performance in tracking accuracy and tracking stability.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-target tracking, and in particular to a fire tracking method based on time series correlation optimization. Background Art

[0002] Forests are a vital component of Earth's ecosystem, playing a crucial role in maintaining ecological balance and supporting life. Currently, fire monitoring and suppression are largely manual, relying on forest rangers to regularly patrol designated routes and periodically monitor fires from towers. Once a fire is discovered, fire suppression plans are developed and manually extinguished. This approach suffers from limitations such as limited coverage, delayed response times, and high risks to personnel safety. In recent years, drone technology, owing to its advantages of flexibility, maneuverability, and low-cost, wide-area coverage, has become a core tool for forest fire monitoring. Drone technology, with its wide-area coverage and maneuverability, has significantly improved fire monitoring efficiency. However, accurate fire spread tracking and drone-based suppression rely heavily on the robustness of the underlying algorithms. Forest fire scenarios present challenges such as highly nonlinear motion due to the combined motion of flame targets and drones, and trajectory confusion and target loss caused by the dynamic blur of flames and the similarity of nearby fires. These problems make it difficult for traditional tracking methods to perform well when applied to forest fire scenarios. However, the use of fire tracking methods based on time series correlation optimization can greatly help people achieve accurate and continuous fire tracking, thereby improving the performance of fire tracking.

[0003] In recent years, scholars at home and abroad have conducted in-depth analysis and research on multi-target tracking and fire tracking methods. In 2023, Zeng et al. proposed OC-SORT to address the problems existing in SORT. This method further focuses on the occlusion problem and introduces a motion prediction consistency mechanism based on SORT, so that the tracker can maintain stable tracking in the case of target occlusion, rapid target movement or drastic posture changes. In 2024, Zhang et al. proposed DiffMOT, which describes the multi-target tracking problem as a generation problem based on the diffusion model, and performs target matching and trajectory prediction through the diffusion process, thereby improving the robustness and accuracy of tracking. In 2021, Adnan Khalil et al. used multi-color space to accurately segment fire-like objects, and detected moving fire pixels based on background modeling to further embed target tracking methods to achieve stable tracking of fires. In 2024, Daoud Z et al. used improved YOLOv3 combined with Kalman filtering to achieve accurate fire detection and tracking. That same year, Sun Niao and others, focusing on fire scenarios, improved the YOLOv8 detection algorithm, which currently achieves a good balance between accuracy and speed, and combined it with the DeepSort method to achieve precise fire tracking. However, fire tracking methods are still in their infancy and have considerable room for improvement. Current optimization approaches for fire tracking methods primarily focus on improving detector performance and refining traditional tracking paradigms. While this has improved tracking accuracy in specific scenarios, existing methods are unable to effectively address complex nonlinear motion, dynamic flame disturbances, and identity loss caused by the similarity of adjacent fire points. Summary of the Invention

[0004] The purpose of the present invention is to provide a fire tracking method based on time series correlation optimization, which can effectively complete the fire tracking task and improve the tracking accuracy and robustness.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A fire tracking method based on time series correlation optimization, comprising:

[0007] Acquire fire images of the target area;

[0008] The fire image is input into a preset fire tracking model, and the fire tracking result of the target area is output, wherein the fire tracking model is used to detect the flame target, extract the timing information and physical characteristics of the flame target and perform matching optimization, and realize fire tracking by generating a virtual trajectory.

[0009] Optionally, the fire tracking model includes:

[0010] A target detection module, configured to detect a flame target in an input image using a detector;

[0011] The time series association module is used to extract the time series information of the flame target and generate the trajectory by combining the spatial information;

[0012] Physical feature modeling module, used to extract and model the physical features of the flame target and construct a joint representation;

[0013] A hierarchical association strategy module is used to divide the scene, and perform multi-level association matching on the scene in combination with the flame target and the corresponding trajectory and joint representation, and output the trajectory matching result;

[0014] The virtual trajectory generation module is used to perform trajectory prediction on the unmatched trajectory, obtain a virtual trajectory, input it into the hierarchical association strategy module again for multi-level association matching, and output the final trajectory matching result.

[0015] Optionally, the detector in the target detection module adopts a YOLOX detector, which extracts multi-scale features through CSPDarknet53, then uses PAFPN to fuse multi-scale features, and finally uses a decoupling head to output the detection frame and position information of the flame target.

[0016] Optionally, the time sequence association module extracts the time sequence information of the flame target and generates a trajectory in combination with the spatial information, including:

[0017] Extracting long-range motion features through the temporal coding stage and performing global modeling to generate temporal coding features, wherein the temporal coding features include motion information and position information;

[0018] The motion information and position information are weightedly fused through a dynamic gating mechanism in the motion decoding stage to generate fused information and a trajectory;

[0019] The fused information is input into the feedforward network to further extract features and classify them, and output the confidence level of the association between the trajectory and the detection box.

[0020] Optionally, the physical feature modeling module extracts and models the physical features of the flame target to construct a joint representation, including:

[0021] ROI region of interest segmentation is used to divide several areas containing the flame target, and the areas are filtered and screened using HSV color space and Gaussian filtering;

[0022] Use the Sobel operator to perform contour detection calculation on the processed area to obtain the flame target contour information;

[0023] Corner point detection is performed on the flame target contour information to obtain contour corner points, and optical flow calculation is performed on the contour corner points using a sparse optical flow method to obtain optical flow information.

[0024] Optionally, the hierarchical association strategy module divides the scene, and performs multi-level association matching on the scene in combination with the flame target and the corresponding trajectory and joint representation, and outputs a trajectory matching result, including:

[0025] Compare the confidence level of the correlation between the trajectory and the detection frame with the preset detection threshold to obtain high-score detection frames and low-score detection frames;

[0026] Perform the first-stage matching on the high-scoring detection frame, output the unmatched track, and perform the second-stage matching on the low-scoring detection frame in combination with the unmatched track, and output the track matching result.

[0027] Optionally, performing first-stage matching on the high-score detection frame includes:

[0028] The trajectories are divided based on the preset scene distinction threshold to obtain complex scene trajectories and simple scene trajectories;

[0029] The complex scene trajectories are associated according to multi-level matching, and the simple scene trajectories are associated according to IoU matching, and unmatched trajectories are output. The unmatched trajectories are filtered based on a preset duplicate detection threshold, and the retained unmatched trajectories are output.

[0030] Optionally, performing a second-stage matching on the low-scoring detection frame in combination with the unmatched track includes:

[0031] After filtering the low-scoring detection frames using a preset filtering threshold, multi-level matching is performed in combination with the retained unmatched tracks, and a track matching result is output.

[0032] Optionally, the virtual trajectory generation module performs trajectory prediction on the unmatched trajectory, and obtaining the virtual trajectory includes:

[0033] Perform trajectory prediction on unmatched trajectories and output virtual detections;

[0034] performing a double verification on the virtual detection in combination with the trajectory, and performing motion correction on the virtual detection having a verification score lower than a preset threshold to obtain a corrected virtual detection;

[0035] The corrected virtual detection is spliced with the trajectory to output the virtual trajectory.

[0036] The beneficial effects of the present invention are:

[0037] The present invention proposes a fire tracking method based on time series association optimization, which can improve the accuracy and sustainability of fire tracking. The fire tracking model constructed by the present invention aims to solve the problem that the superposition of the movement of the observation platform and the flame produces complex nonlinear motion in fire tracking. The traditional tracking model is difficult to model due to its own limitations, resulting in reduced fire tracking accuracy. The target tracking is modeled as a dynamic association problem between trajectory and detection. First, a time series association module is designed. In the time series encoding stage, long-range motion features are extracted through a bidirectional Mamba architecture and a selective state space model to achieve global modeling of complex motion patterns. Secondly, in the motion decoding stage, a selective algorithm is designed to guide the evolution of time series features, and a gating mechanism is combined to realize the detection frame spatial information. Adaptive fusion with temporal features; In fire tracking, the similar appearance of flames makes it difficult to distinguish adjacent targets with a single motion representation, and the linear association strategy is easily affected by dynamic blur, which increases the risk of trajectory breakage and identity loss, and reduces the reliability of drone extinguishing guidance. The present invention designs a physical feature modeling module, and by fusing the contour detection algorithm with the sparse optical flow calculation, constructs a joint representation of temporal features and physical features to enhance the discrimination of adjacent targets; secondly, a hierarchical association strategy module is proposed, which optimizes the matching process through scene division, and designs a dynamic filtering mechanism to reduce the interference caused by repeated detection; finally, a virtual trajectory generation module is designed, and the target state of the missed detection period is interpolated through the kinematic model to ensure trajectory continuity. The present invention can effectively complete the fire tracking task and improve tracking accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 This is a flow chart of a fire tracking method based on time series correlation optimization according to an embodiment of the present invention;

[0040] Figure 2 Schematic diagram of the timing correlation module structure of an embodiment of the present invention;

[0041] Figure 3 Schematic diagram of the structure of the timing coding part in the timing association module according to an embodiment of the present invention;

[0042] Figure 4 Schematic diagram of the structure of the motion decoding part in the timing association module according to an embodiment of the present invention;

[0043] Figure 5 This is a schematic diagram of the structure of a physical feature modeling module according to an embodiment of the present invention;

[0044] Figure 6 A schematic diagram of the structure of a hierarchical association strategy module according to an embodiment of the present invention;

[0045] Figure 7 This is a schematic diagram of the structure of a virtual trajectory generation module according to an embodiment of the present invention;

[0046] Figure 8 A schematic diagram of the fire tracking model structure according to an embodiment of the present invention;

[0047] Figure 9 The HOTA value of each model in the embodiment of the present invention on the FireMOT dataset changes with the detection-association balance coefficient;

[0048] Figure 10 This is a graph showing changes in tracking accuracy and speed under different scene discrimination thresholds on the FireMOT dataset according to an embodiment of the present invention;

[0049] Figure 11 This is a comparison chart of the quantitative analysis of filtering thresholds on the FireMOT dataset according to an embodiment of the present invention;

[0050] Figure 12 This is a comparison chart of tracking results for frames 11-17 on the FireMOT dataset according to an embodiment of the present invention;

[0051] Figure 13 This is a comparison chart of tracking results of frames 509-512 on the FireMOT dataset according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] This embodiment provides a fire tracking method based on time series correlation optimization, such as Figure 1 As shown, including:

[0055] Acquire fire images of the target area;

[0056] The fire image is input into a preset fire tracking model, and the fire tracking result of the target area is output, wherein the fire tracking model is used to detect the flame target, extract the timing information and physical characteristics of the flame target and perform matching optimization, and realize fire tracking by generating a virtual trajectory.

[0057] Wherein, the fire tracking model includes:

[0058] A target detection module, configured to detect a flame target in an input image using a detector;

[0059] The time series association module is used to extract the time series information of the flame target and generate the trajectory by combining the spatial information;

[0060] Physical feature modeling module, used to extract and model the physical features of the flame target and construct a joint representation;

[0061] A hierarchical association strategy module is used to divide the scene, and perform multi-level association matching on the scene in combination with the flame target and the corresponding trajectory and joint representation, and output the trajectory matching result;

[0062] The virtual trajectory generation module is used to perform trajectory prediction on the unmatched trajectory, obtain a virtual trajectory, input it into the hierarchical association strategy module again for multi-level association matching, and output the final trajectory matching result.

[0063] Specifically, the fire tracking model constructed in this embodiment models target tracking as a dynamic association problem between trajectory and detection. First, a temporal association module is designed. In the temporal encoding stage, long-range motion features are extracted through a bidirectional Mamba architecture and a selective state space model to achieve global modeling of complex motion patterns. Secondly, in the motion decoding stage, a selective algorithm is designed to guide the evolution of temporal features, and a gating mechanism is combined to achieve adaptive fusion of detection frame spatial information and temporal features. A physical feature modeling module is designed. By fusing the contour detection algorithm with sparse optical flow calculation, a joint representation of temporal features and physical features is constructed to enhance the discrimination of neighboring targets. A hierarchical association strategy module is proposed. The matching process is optimized through scene division, and a dynamic filtering mechanism is designed to reduce the interference caused by repeated detection. Finally, a virtual trajectory generation module is designed. The target state of the missed detection period is interpolated through a kinematic model to ensure trajectory continuity. This model can effectively complete the fire tracking task and improve tracking accuracy and robustness.

[0064] Furthermore, the detector in the target detection module adopts the YOLOX detector, which extracts multi-scale features through CSPDarknet53, then uses PAFPN to fuse multi-scale features, and finally uses the decoupling head to output the detection frame and position information of the flame target.

[0065] Specifically, this embodiment selects YOLOX as the detector part, inputs the fire image into the detector, and adopts CSPDarknet53 and cross-stage partial connections in the backbone network part of the detector to reduce the amount of calculation while completing feature extraction. Subsequently, in the neck network, PAFPN is used to realize bidirectional multi-scale feature fusion to further fuse feature information. The decoupling head mechanism is used in the output network part to decouple the classification and regression tasks, realize the simultaneous output of the target detection category and position, and finally realize the detection target output through the detector inference calculation.

[0066] Furthermore, the time sequence association module extracts the time sequence information of the flame target and generates a trajectory in combination with the spatial information, including:

[0067] Extracting long-range motion features through the temporal coding stage and performing global modeling to generate temporal coding features, wherein the temporal coding features include motion information and position information;

[0068] The motion information and position information are weightedly fused through a dynamic gating mechanism in the motion decoding stage to generate fused information and a trajectory;

[0069] The fused information is input into the feedforward network to further extract features and classify them, and output the confidence level of the association between the trajectory and the detection box.

[0070] Specifically, in the temporal association module of this embodiment, its structure can be separated into a temporal encoding part and a motion decoding part. In this module, the temporal information of the target state in the historical frame is encoded to model the dynamic change of the target temporal sequence. Dynamic gating is further used to achieve adaptive weighting of temporal information and spatial information. This design provides a global perspective for the entire tracking process. Figure 2 It is the structural diagram of the timing association module.

[0071] The timing coding part is as follows Figure 3 As shown in Figure 2, the input in this section is historical trajectory information, and the output is temporal encoding features. The specific process is as follows: given trajectory information, position encoding is first performed to obtain an embedded sequence; after obtaining the embedded sequence, the sequence is used as the input of the temporal information encoding module, and temporal information is captured via the Bi-Mamba structure.

[0072] The Bi-Mamba structure implements bidirectional processing, enhancing the extraction of global contextual information from the embedded sequence. This overcomes the problem that the Mamba structure only processes the embedded sequence unidirectionally and may lose some motion information. Specifically, the embedded sequence obtained by position encoding is input into the Mamba structure as a forward sequence and a reverse sequence for forward and reverse encoding. The information of both is then fused to capture more comprehensive temporal information. After multi-layer bidirectional Mamba temporal information encoding iterative processing, the final output is a temporal encoding feature that can fully capture the long-range dependencies and motion context information in the motion trajectory.

[0073] The Mamba architecture has the following process: First, a fully connected layer is applied to the input embedding sequence to implement linear transformation and obtain a preliminary linear feature representation; then, a one-dimensional convolution operation is used to introduce local dependencies on these linear features and extract higher-level abstract features. Subsequently, the convolved features are nonlinearly transformed using the SiLU activation function to enhance the model's ability to express complex patterns. Next, these nonlinearly processed features will be passed to the state-space model (SSM) module, which aims to capture global dependencies in the input sequence and integrate long-distance information across time. Finally, in order to preserve the fine-grained information in the original embedding sequence and alleviate the gradient vanishing problem, the embedded sequence processed by the activation function is added to the features output by the SSM module through a residual connection to generate the final temporal coding features for the subsequent decoding process.

[0074] Motion decoding part such as Figure 4 As shown in Figure 2, in the motion decoding part, in order to accurately utilize the temporal coding features to optimize trajectory processing, the following processing flow is adopted. The input information includes the motion embedding containing position and motion information generated by the temporal coding module, the detection frame position information d i+1 , and the final location information P of the historical trajectory i First, P is encoded with cosine position coding. i Mapped to high-dimensional space, converted into high-dimensional trajectory position embedding H i , enhance the model's perception of location information. i Input the linear layer and split it by dimension to get the position feature p i and r i Among them, p i is fed into the OS-SSM module, which uses its temporal modeling capability to guide the motion embedding according to the input historical position information p i Output trajectory prediction feature p i+1 For r i , and treat it as a residual. Let r i The nonlinear transformation is introduced through the activation layer and then combined with p i+1Multiply and aggregate historical motion and historical position information to obtain the fusion feature p mix Then, put p mix and detection feature d i+1 Splicing. The gating mechanism is used for weighting during the splicing process. First, the gating weight is calculated:

[0075] G=σ(W·SILU(p mix ,d i+1 ))+b;

[0076] Among them, b is the bias term, and the splicing features are weighted fused after the calculation of the gating weights:

[0077] F=G⊙p mix +(1-G)⊙d i+1 ;

[0078] The spliced feature F combines the position of the current frame with the trajectory prediction feature to provide a more comprehensive target feature representation for the model. Finally, the feature F is input into the feedforward network. The FFN further extracts and classifies the features and outputs the confidence score S of the match between the target trajectory and the detection box. i .

[0079] During the decoding process of the decoder, in order to better utilize the complex dependencies in the temporal coding features to achieve motion pattern learning, this embodiment designs the OS-SSM module based on the SSM. As can be seen from the previous description of the state space model, the core part of state space modeling is the parameter matrix A, B, C, D used to describe the state. In OS-SSM, the temporal coding feature M is used as a guide to generate the parameterized matrix. The generation formula is as follows:

[0080] Δ,B,C←Linear(M);

[0081] Where Δ represents the time step. The zero-order hold method is then used to discretize the parameters A and B using the time step Δ:

[0082]

[0083] After discretization, the input position embedding p is guided by the temporal encoding feature M and the historical hidden state h of the input i The state transformation of , which realizes the feature prediction from the current position embedding to the next position state information in a single step:

[0084]

[0085] Furthermore, the physical feature modeling module extracts the physical features of the flame target and models them to construct a joint representation, including:

[0086] ROI region of interest segmentation is used to divide several areas containing the flame target, and the areas are filtered and screened using HSV color space and Gaussian filtering;

[0087] Use the Sobel operator to perform contour detection calculation on the processed area to obtain the flame target contour information;

[0088] Corner point detection is performed on the flame target contour information to obtain contour corner points, and optical flow calculation is performed on the contour corner points using a sparse optical flow method to obtain optical flow information.

[0089] Specifically, since the tracking target in the fire tracking scenario is flames, which lack a rigid structure and have highly similar dynamic textures, this embodiment proposes a physical feature modeling module (PFM) to distinguish flames using physical features. Figure 5 It is the structure diagram of the physical feature modeling module.

[0090] The physical feature modeling module achieves physical modeling of flame targets, supplementing their contour information with fine-grained contour optical flow motion information to improve the discrimination between different flame targets. The specific process is as follows: Each frame of the image is input along with the flame target detection box. To reduce background influence and reduce overhead, the input image is first segmented into regions of interest (ROIs). Within these regions of interest, the RGB color space is converted to HSV for easier flame target processing. To reduce background interference on the flame target, after completing the color space conversion, a color mask is constructed using the HSV range of empirical flames to achieve flame color filtering. Gaussian denoising is then performed, and the Sobel operator is used to extract contours from the denoised image. Corner detection is performed on the edge map obtained from contour extraction to obtain contour corners. Finally, sparse optical flow is used to calculate the optical flow of these corners to achieve physical modeling of the target.

[0091] Furthermore, the hierarchical association strategy module divides the scene, and performs multi-level association matching on the scene in combination with the flame target and the corresponding trajectory and joint representation, and outputs a trajectory matching result, including:

[0092] Compare the confidence level of the correlation between the trajectory and the detection frame with the preset detection threshold to obtain high-score detection frames and low-score detection frames;

[0093] Perform the first-stage matching on the high-scoring detection frame, output the unmatched track, and perform the second-stage matching on the low-scoring detection frame in combination with the unmatched track, and output the track matching result.

[0094] The first stage matching of the high-score detection frame includes:

[0095] The trajectories are divided based on the preset scene distinction threshold to obtain complex scene trajectories and simple scene trajectories;

[0096] The complex scene trajectories are associated according to multi-level matching, and the simple scene trajectories are associated according to IoU matching, and unmatched trajectories are output. The unmatched trajectories are filtered based on a preset duplicate detection threshold, and the retained unmatched trajectories are output.

[0097] The second stage matching of the low-scoring detection frame in combination with the unmatched track includes:

[0098] After filtering the low-scoring detection frames using a preset filtering threshold, multi-level matching is performed in combination with the retained unmatched tracks, and a track matching result is output.

[0099] Specifically, fire tracking involves complex scenarios, repeated detection interference caused by dynamic blur, and identity switching issues. Traditional models cannot achieve effective matching through linear association strategies. This embodiment proposes a hierarchical association policy (HAP) module to divide target scenarios and eliminate repeated interference. Figure 6 This is a structural diagram of the hierarchical association strategy module.

[0100] The core of HAP is to first classify the confidence of its detection frames, dividing them into high-confidence detection frames and low-confidence detection frames according to a threshold. These are used for first-stage matching and second-stage matching, respectively. Secondly, scene segmentation is achieved through trajectory classification. Trajectories in simple scenes use Intersection over Union (IoU) matching, which reduces time loss and prevents simple scene trajectories from being overtaken by complex scene trajectories. Complex scenes use multi-level matching to achieve association. Detection screening is then performed to resolve false detections and duplicate detections in unmatched high-scoring detections and low-scoring detections, thereby reducing interference from complex detection information during the tracking process.

[0101] Furthermore, the virtual trajectory generation module performs trajectory prediction on the unmatched trajectory, and obtaining the virtual trajectory includes:

[0102] Perform trajectory prediction on unmatched trajectories and output virtual detections;

[0103] performing a double verification on the virtual detection in combination with the trajectory, and performing motion correction on the virtual detection having a verification score lower than a preset threshold to obtain a corrected virtual detection;

[0104] The corrected virtual detection is spliced with the trajectory to output the virtual trajectory.

[0105] Specifically, dynamic occlusions such as irregular flame deformation can cause flame targets to go undetected at certain times. This can lead to premature trajectory termination due to multi-frame mismatches, as well as trajectory breakage caused by noise accumulation due to multi-frame mismatches. To address this, we propose a Virtual Trajectory Generation (VTG) module that interpolates the target state during missed detection periods using a kinematic model, ensuring trajectory continuity and compensating for detection artifacts. Figure 7 This is the structure diagram of the virtual trajectory generation module.

[0106] The VTG module uses the prediction generated by the Kalman filter as motion interpolation. Before generating a virtual trajectory, it needs to be double-verified by the TC module and PFM. If the double verification is passed, the corresponding virtual trajectory is generated and the trajectory is continuously maintained in the subsequent association process to facilitate subsequent matching. Specifically, for the inactive trajectory in the t-1 frame, if no new update is received in the matching of the next frame, the prediction p obtained by the Kalman filter in the last frame of the trajectory is used. t As a virtual inspection, in order to measure whether the prediction quality can be considered as a relatively accurate inspection, the TC module and physical feature matching are used to measure it. The measurement formula is as follows:

[0107] S vic =αS mamba (T t-1 ,p t )+βS phy (d t-1 ,p t );

[0108] Where α and β represent weights. In the matching association of the t-th frame, if a match is still not achieved, the trajectory is predicted and updated according to the virtual detection of the previous frame.

[0109] If the score of the virtual detection is lower than the set threshold, the optical flow υ and momentum m are used to correct it. The formula is as follows:

[0110] p t ′=p t +υ+m;

[0111] The modified virtual detection is adopted and a virtual trajectory is generated to complete the subsequent matching association.

[0112] Furthermore, the final fire tracking model constructed in this embodiment is as follows: Figure 8 As shown in the figure, the fire tracking dataset is used as the input of the fire tracking model based on time series association optimization, and a comparative experiment is carried out to evaluate the target tracking effect. At the same time, an ablation experiment is carried out to verify the effectiveness of the module.

[0113] The high-order tracking accuracy (HOTA), multi-target tracking accuracy (MOTA), identity F1 score (IDF1), association accuracy (AssA), transmission frames per second (FPS), and target detection accuracy (DetA) are used as evaluation indicators.

[0114] High-order tracking accuracy (HOTA) balances the effects of performing accurate detection, association, and localization, combining the evaluation of the three separate aspects into a unified metric to measure the tracker effect. It is calculated as:

[0115]

[0116] Object detection accuracy (DetA) is used to measure the impact of the detector and is calculated as follows:

[0117]

[0118] Among them, TP is the number of positive samples predicted by the model as positive samples, FP is the number of negative samples predicted by the model as positive samples, and FN is the number of positive samples predicted by the model as negative samples.

[0119] Association accuracy (AssA) is used to measure the ability of the tracking method to correctly associate detected targets in consecutive frames and is calculated as follows:

[0120]

[0121] Among them, TPA refers to the number of correct detections, FNA refers to the number of missed detections, and FPA refers to the number of false detections.

[0122] Multi-object tracking accuracy (MOTA) is a comprehensive indicator used to evaluate the detection quality of the tracker and the trajectory prediction effect. It is calculated as follows:

[0123]

[0124] Among them, IDSW (Identity Switches) reflects the total number of identity switches of the target during the tracking process.

[0125] Identity F1 score (IDF1), which is the harmonic mean of precision and recall, comprehensively reflects the performance of the tracking system in identity matching. It is calculated as follows:

[0126]

[0127] IDTP represents the number of target IDs correctly matched during identification. IDFP represents the number of target IDs incorrectly associated during identification, i.e., the number of different targets incorrectly matched to the same ID. IDFN represents the number of true targets incorrectly matched, i.e., the number of true targets that were not correctly associated.

[0128] The FireMOT dataset was used for model learning and performance verification. For the detector, the FireMOT dataset was used for training. The training set includes 30,375 images, and the training rounds were set to 300 rounds. The training was terminated early if the verification results did not fluctuate too much in the last 50 rounds. For the fire tracking model, the training batch size was set to 2048, Adam was used as the optimizer, the initial learning rate was set to 0.0001, and the training rounds were set to 300 rounds. Similarly, the training was terminated early if the verification results did not fluctuate too much in the last 50 rounds. Table 1 shows the performance index comparison results of each model on the FireMOT dataset, among which OC-SORT is the baseline model, TC-OCSORT is the model after adding the time series association module to the baseline model, and AO-OCSORT is the fire tracking model.

[0129] Table 1

[0130]

[0131] As can be seen from Table 1, this model has demonstrated competitive performance on this dataset. Overall, compared to the baseline model, this model has achieved the best tracking performance among this series of tracking models by adding a temporal association module, a physical feature modeling module, a hierarchical association strategy, and a virtual trajectory generation module. Among them, HOTA reached 74.8% and AssA reached 81.6%. This shows that this model can achieve good tracking performance in the face of fire scenarios. The IDF1 value of this model also reached the optimal 84.0%, verifying the model's ability to maintain identity during the tracking process. The above analysis further proves that this model can better complete the fire tracking task. In order to more intuitively feel the overall performance of this model, Figure 9 The HOTA value changes with the detection-association balance coefficient. The area enclosed by the curve represents the overall HOTA value. Curves closer to the upper right in this figure represent superior models. The AO-OCSORT curve, representing our model, maintains a leading position throughout the entire alpha variation, with an area under the curve reaching 74.8%, further demonstrating the model's superiority in fire scenarios.

[0132] To verify the impact of the temporal correlation module proposed in this embodiment on the performance of this model, the effects of adding the TC module and the bidirectional processing mechanism (Bi) on the overall model were verified on the FireMOT dataset. Table 2 shows a comparison of the model effects under different addition methods.

[0133] Table 2

[0134]

[0135] Table 2 shows that in scenarios such as fires, which involve extensive nonlinear motion, OC-SORT's matching and association performance is affected, resulting in relatively low performance. After adding the TC module, the HOTA value increased by 3.5%, demonstrating the effectiveness of the long-range temporal information captured by the TC module for tracking. The IDF1 and AssA values also increased by 5.7% and 6.8%, respectively, fully demonstrating the TC module's superior performance in matching trajectories to detections. Building on the TC module, we further optimized the extraction of temporal information and added the Bi module to capture contextual information, strengthening the capture of global dependencies within the TC structure. The results show that the addition of the Bi module increased the model's HOTA value by 0.9%, while IDF1 and AssA values increased by 1.9 percentage points and 2.3 percentage points, respectively, fully demonstrating the Bi module's ability to further explore motion patterns and enhance temporal information capture.

[0136] To further validate the performance of this module, we studied the input of the temporal association module and examined the impact of different input lengths on module performance. Table 3 shows the performance comparison of the model when different input lengths are selected.

[0137] Table 3

[0138]

[0139] Table 3 shows that different trajectory lengths contain different temporal information during tracking. Trajectories that are too short cannot represent the target's motion, while trajectories that are too long, when complex in parameters, can also suffer from noise accumulation and motion conflicts. To balance trajectory length and model performance, experiments were conducted on segmented input trajectory lengths. The model achieved optimal performance when the input trajectory length was 5. This indicates that, in the FireMOT dataset, a trajectory of five frames can effectively depict the model's staged motion and balance the information representation of local motion details and overall motion trends. Performance at a trajectory length of 3 shows only a slight decrease compared to a trajectory length of 5, indicating that trajectories closer to the detection box better represent the target's motion. Trajectories of length 3 can accurately reflect instantaneous motion states, but lack the ability to capture overall motion trends. As trajectory lengths increase, model performance gradually degrades. This is because overly long trajectories contain more than one motion model and contain excessive information redundancy, which can affect subsequent classification.

[0140] To verify the effectiveness of the physical feature modeling module, hierarchical association strategy module, and virtual trajectory generation module proposed in this example, an ablation comparison experiment was conducted on the FireMOT dataset, where "+" represents an added module. Table 4 shows the performance comparison results of the model when different modules are added.

[0141] Table 4

[0142]

[0143] Table 4 shows that the model's IDF1 index increased by 1.1% after adding the PFM module, indicating that the physical feature modeling module has certain advantages in achieving identity differentiation and recognizing different flame identities. This is because physical features utilize fine-grained flame features and contour information, which are significantly effective in distinguishing adjacent targets. Adding the HAP module also shows an overall improvement in model performance, with the HOTA value increasing by 0.6%, AssA increasing by 1.3%, and IDSW decreasing by 13%. This demonstrates the advantages of the hierarchical association strategy module in fire scenarios and also indirectly reflects the serious problem of duplicate detection caused by the dynamic blur of flames in fire scenarios. Finally, the VTG module was added, and its performance generally improved, with AssA increasing by 0.8%, verifying the effectiveness of this module.

[0144] In order to verify the impact of different setting parameters in the hierarchical association strategy module of this model on the overall performance of the model, quantitative ablation is performed on the scene division threshold, duplicate detection threshold, and filtering threshold to measure their impact on the model.

[0145] The purpose of scene classification is to measure the environment in which the trajectory is located. It determines which trajectories use simple IoU matching and which use a combination of the TC module, the PFM module, and multi-level IoU matching. Because the introduction of the TC module and the PFM module increases parameter calculation, this parameter significantly affects the balance between model accuracy and inference speed. Figure 10 This figure compares tracking accuracy and speed at different scene discrimination thresholds. As the scene discrimination threshold increases, more tracks begin to use multi-level matching to achieve more accurate matching associations. At the same time, due to the increased computation involved, the model's inference speed decreases. Specifically, HOTA increases with the scene discrimination threshold, while FPS decreases. The figure above shows that 0.5 is a critical threshold. Increasing from 0.4 to 0.5 only reduces FPS by 1.6 frames, while HOTA increases by 1.4%. In subsequent changes, HOTA growth slows, but FPS continues to decrease. In validation, to measure model accuracy and speed, a scene discrimination threshold of 0.5 was selected.

[0146] In fire scenarios, the same flame may be subject to high-scoring duplicate detections or mis-segmentation due to motion blur or partial obscuration by trees. Setting a duplicate detection threshold can mitigate this situation. However, a too low duplicate detection threshold will remove valid detection frames, leading to an increase in missed detections; a too high duplicate detection threshold will result in an excessive number of duplicate detection frames being retained, resulting in an increase in false detections. Therefore, it is important to select an appropriate threshold to achieve a balance between false detections and missed detections. Table 5 shows the impact of the duplicate detection threshold on model performance.

[0147] Table 5

[0148]

[0149] Table 5 shows that the number of false positives (FPs) increases steadily with increasing duplicate detection thresholds. This is because when the threshold is low, strict filtering effectively eliminates most false positives. However, as the threshold increases, only detections with extremely high IOU are considered duplicates, while other duplicates with low overlap are retained, resulting in an increase in FP. Meanwhile, the number of false negatives (FNs) decreases and then increases with threshold adjustment. When the threshold is low, overly strict filtering can mistakenly delete some valid detections, leading to a high FN. As the threshold is appropriately increased, valid detections are retained, reducing the FN. However, when the threshold is too high, the large number of retained duplicates disrupts the subsequent matching process, causing the FN to rise again. To achieve an optimal balance between FN, FP, and IDS, a duplicate detection threshold of 0.6 was chosen. With this setting, the optimal MOTA value reached 74.3%.

[0150] In order to verify the influence of the filtering threshold on the overall performance of this model, a quantitative analysis was conducted. Figure 11 This is a comparison chart of the filter threshold quantitative analysis, Figure 11 It can be seen that with the increase of the filtering threshold, the MOTA value of the model shows a trend of first increasing and then decreasing. In this model, a dynamic threshold method is adopted. Experiments show that the MOTA value of the model is optimal after adding the area dynamic threshold while setting 0.5 as the basic threshold.

[0151] like Figure 12 As shown in the OC-SORT tracking results, in frame 11, the two flames in the lower left corner of the image are assigned identities 2 and 4. In frame 13, both flames experience identity loss due to tracking drift caused by rapid viewpoint movement. In frame 15, track 2 is reactivated but assigned to the flame target, originally identified as 8, resulting in an identity switch. Subsequently, in frame 17, track 2 undergoes another identity switch and is assigned identity 4. The flame identified as 2 in frame 11 is reassigned identity 14. Its tracking performance degrades due to the rapid viewpoint movement. In contrast, AO-OCSORT assigns the two flames in the lower left corner identities 2 and 4 in frame 11. In frames 13, 15, and 17, flame 2 is stably tracked without identity switching. After frame 11, flame 4 is obscured by trees, preventing its identity from being misassigned. Furthermore, Flame No. 8 was tracked steadily without frequent identity switching as in OC-SORT.

[0152] like Figure 13 As shown in the OC-SORT tracking results, it can be seen that the flame target that produces a lot of smoke during the tracking process is assigned the identity 2 in frame 509. In the subsequent frames 510 and 511, the flame target is obscured by the smoke, resulting in missed detection. In frame 512, the target is redetected and re-tracked, but the trajectory is broken due to missed detection, and the target is assigned a new identity 19. In the tracking process using AO-OCSORT, in frame 509, the flame target that produces a lot of smoke is also assigned the identity 2. In frames 510 and 511, the trajectory identity is lost due to target occlusion. However, in frame 512, after the flame target is redetected, there is no identity switch, but the trajectory is reactivated.

[0153] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A fire tracking method based on time series correlation optimization, characterized in that: include: Acquire fire images of the target area; The fire image is input into a preset fire tracking model, and the fire tracking result of the target area is output, wherein the fire tracking model is used to detect the flame target, extract the timing information and physical characteristics of the flame target and perform matching optimization, and realize fire tracking by generating a virtual trajectory.

2. The fire tracking method based on time series correlation optimization according to claim 1 is characterized in that: The fire tracking model includes: A target detection module, configured to detect a flame target in an input image using a detector; The time series association module is used to extract the time series information of the flame target and generate the trajectory by combining the spatial information; Physical feature modeling module, used to extract and model the physical features of the flame target and construct a joint representation; A hierarchical association strategy module is used to divide the scene, and perform multi-level association matching on the scene in combination with the flame target and the corresponding trajectory and joint representation, and output the trajectory matching result; The virtual trajectory generation module is used to perform trajectory prediction on the unmatched trajectory, obtain a virtual trajectory, input it into the hierarchical association strategy module again for multi-level association matching, and output the final trajectory matching result.

3. The fire tracking method based on time series correlation optimization according to claim 2 is characterized in that: The detector in the target detection module adopts the YOLOX detector, which extracts multi-scale features through CSPDarknet53, then uses PAFPN to fuse multi-scale features, and finally uses a decoupling head to output the detection frame and position information of the flame target.

4. The fire tracking method based on time series correlation optimization according to claim 2 is characterized in that: The time sequence association module extracts the time sequence information of the flame target and generates a trajectory in combination with the spatial information, including: Extracting long-range motion features through the temporal coding stage and performing global modeling to generate temporal coding features, wherein the temporal coding features include motion information and position information; The motion information and position information are weightedly fused through a dynamic gating mechanism in the motion decoding stage to generate fused information and a trajectory; The fused information is input into the feedforward network to further extract features and classify them, and output the confidence level of the association between the trajectory and the detection box.

5. The fire tracking method based on time series correlation optimization according to claim 2 is characterized in that: The physical feature modeling module extracts the physical features of the flame target and models them to construct a joint representation, including: ROI region of interest segmentation is used to divide several areas containing the flame target, and the areas are filtered and screened using HSV color space and Gaussian filtering; Use the Sobel operator to perform contour detection calculation on the processed area to obtain the flame target contour information; Corner point detection is performed on the flame target contour information to obtain contour corner points, and optical flow calculation is performed on the contour corner points using a sparse optical flow method to obtain optical flow information.

6. The fire tracking method based on time series correlation optimization according to claim 2 is characterized in that: The hierarchical association strategy module divides the scene, and performs multi-level association matching on the scene in combination with the flame target and the corresponding trajectory and joint representation, and outputs the trajectory matching result, including: Compare the confidence level of the correlation between the trajectory and the detection frame with the preset detection threshold to obtain high-score detection frames and low-score detection frames; Perform the first-stage matching on the high-scoring detection frame, output the unmatched track, and perform the second-stage matching on the low-scoring detection frame in combination with the unmatched track, and output the track matching result.

7. The fire tracking method based on time series correlation optimization according to claim 6 is characterized in that: Performing the first stage matching on the high-score detection frame includes: The trajectories are divided based on the preset scene distinction threshold to obtain complex scene trajectories and simple scene trajectories; The complex scene trajectories are associated according to multi-level matching, and the simple scene trajectories are associated according to IoU matching, and unmatched trajectories are output. The unmatched trajectories are filtered based on a preset duplicate detection threshold, and the retained unmatched trajectories are output.

8. The fire tracking method based on time series correlation optimization according to claim 7 is characterized in that: Performing a second-stage matching on the low-scoring detection frame in combination with the unmatched track includes: After filtering the low-scoring detection frames using a preset filtering threshold, multi-level matching is performed in combination with the retained unmatched tracks, and a track matching result is output.

9. The fire tracking method based on time series correlation optimization according to claim 2 is characterized in that: The virtual trajectory generation module performs trajectory prediction on the unmatched trajectory, and obtaining the virtual trajectory includes: Perform trajectory prediction on unmatched trajectories and output virtual detections; performing a double verification on the virtual detection in combination with the trajectory, and performing motion correction on the virtual detection having a verification score lower than a preset threshold to obtain a corrected virtual detection; The corrected virtual detection is spliced with the trajectory to output the virtual trajectory.

Images