A damage rapid assessment method and system based on remote sensing video target detection and tracking

Abstract

The application provides a damage rapid evaluation method and system based on remote sensing video target detection and tracking, and the method comprises the following steps: obtaining target-containing video data to be processed; inputting the video data to be processed into a trained detection and tracking model to output detection and tracking results of a bounding box; obtaining target damage evaluation results of the video data to be processed based on the detection and tracking results of the bounding box; the detection and tracking model: processing a video frame through a multi-target detector based on multi-scale context block structure optimization to output a multi-scale fusion feature map and at least one bounding box information, wherein the bounding box information comprises a bounding box and a label category, and the label category comprises a target and damage; inputting multi-target detector output results of adjacent video frames into an appearance-enhanced multi-target tracker to output detection and tracking results of the bounding box.

Description

Technical Field

[0001] This invention belongs to the field of rapid damage assessment technology, specifically relating to a rapid damage assessment method and system based on remote sensing video target detection and tracking. Background Technology

[0002] Extensive research has been conducted both domestically and internationally in the field of video target detection and tracking, primarily encompassing single-target detection and tracking as well as multi-target detection and tracking. As the name suggests, single-target detection and tracking refers to continuously tracking a known target within a video sequence, typically achieved by updating the target's position in each video frame. Multi-target detection and tracking, on the other hand, is a more complex and challenging task, involving the simultaneous tracking of multiple targets within a video sequence, requiring consistent target identification over time. Traditional target detection methods suffer from drawbacks such as large data volumes and slow speeds, and have gradually been phased out in the development of target tracking. In recent years, the rapid development of deep learning technology has enabled target detection and tracking techniques to shine in various application areas. In the field of aerial Earth observation, previous researchers have utilized high spatial resolution satellite video data to address the challenge of tracking small targets in satellite videos that are subject to occlusion and significant lighting variations. They proposed a robust and lightweight target tracker using the Siamese network as the backbone, improving the accuracy of small target detection and real-time tracking performance. Meanwhile, addressing the challenge of low vehicle pixel ratios in satellite video, a method using historical models from each frame to update the satellite video tracker has been proposed. This addresses the difficulty of vehicle detection and tracking due to the limited vehicle features in satellite video, and the proposed scheme effectively improves the accuracy and success rate of the target tracker. Similarly, addressing the lack of detectors and ReID parallel collaborative learning in most current satellite video target trackers, a method is proposed to model the multi-target tracking problem as a graph-based information reasoning process. Experiments show that this method is very effective in improving the accuracy of multi-target tracking. Furthermore, in related research on fire smoke detection, researchers have proposed a VSmokeNet smoke detection algorithm based on ViBe and target detection, achieving accurate identification of smoke in videos. Similarly, addressing the insufficient processing accuracy and real-time performance of some current algorithms, some researchers have combined the YOLOv5 algorithm with an attention model to implement a fire smoke detection and recognition technique with the addition of the ACON activation function, effectively improving detection performance.

[0003] Research on target and damage monitoring information extraction and damage assessment has always been a key and challenging issue in domestic and international security monitoring and military applications. Video data offers a new approach to solving the integrated detection and dynamic tracking of targets and damage, thereby enabling rapid damage assessment. However, for a long time, research on targets and damage has been conducted separately, with few applied studies focusing on integrated target and damage detection, and most studies are based on information extraction from image data. The effective maintenance of a complete precision combat process depends on the effectiveness assessment of each strike. Without accurate and reliable target damage information, the attack effect cannot be judged in a timely manner, and subsequent combat operations cannot proceed smoothly. Informationized warfare is fast-paced, high-intensity, and the battlefield situation is complex and ever-changing. Coupled with the stringent requirements for accuracy, timely, rapid, and accurate battlefield assessment is essential to confirm the target damage status and extent, determine whether the strike has achieved its intended purpose, and provide a basis for formulating the next action plan and strike plan. Accurate assessment has become an indispensable and crucial link in the "reconnaissance, control, strike, and assessment" combat action chain. Accurate assessment is also a problem that both offensive and defensive sides must face and solve.

[0004] In summary, there is an urgent need to propose improvement strategies to address the problems of insufficient accuracy, feature loss during extraction, and slow detection efficiency in integrated target and damage detection. Furthermore, given the current lack of datasets in integrated target and damage detection research, it is imperative to construct a comprehensive dataset for this purpose. Ultimately, this will meet the requirements for accurate target and damage tracking, providing a strong basis for rapid damage assessment. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies where target and damage studies are conducted separately and lack accurate and reliable target and damage information, thus hindering timely assessment of attack effectiveness, this invention provides a rapid damage assessment method and system based on remote sensing video target detection and tracking. By constructing a detection and tracking model, using video frames as processing units, and employing a multi-target detector optimized with a multi-scale context block structure, global contextual dependencies are introduced during detection, resulting in more complete global information representation and more accurate target or damage detection results. Furthermore, an appearance-enhanced multi-target tracker strengthens the appearance representation of features, associating targets or damage with unique identifiers for more accurate target or damage tracking. Finally, by linking the detection and tracking results with abrupt video frame changes, rapid damage assessment is achieved.

[0006] According to one aspect of the present invention, a rapid damage assessment method based on remote sensing video target detection and tracking is provided, comprising:

[0007] Acquire the video data to be processed, which contains the target.

[0008] The video data to be processed is input into the trained detection and tracking model, which outputs the detection and tracking results of bounding boxes.

[0009] Based on the detection and tracking results of the bounding box, the target damage assessment results of the video data to be processed are obtained;

[0010] The training process for the detection and tracking model includes:

[0011] Construct a dataset for joint detection and tracking of targets and damage;

[0012] Construct a detection and tracking model: A multi-object detector based on a multi-scale context block structure optimization processes video frames and outputs a multi-scale fused feature map and at least one bounding box information, wherein the bounding box information includes the bounding box and the label category, and the label category includes target and damage; An appearance-enhanced multi-object tracker takes the output results of the multi-object detector of adjacent video frames as input and outputs the detection and tracking results of the bounding boxes.

[0013] The detection and tracking model is trained using a dataset for joint target and damage detection and tracking, and the trained model is output.

[0014] As a further technical solution, the multi-object detector includes a multi-object detection part and a multi-scale context block structure. The multi-object detection part is a backbone network-intermediate layer encoder-detector head integrated structure based on the Yolov5 model, which processes the input video frame and outputs feature maps at high, medium and low scales. The multi-scale context block structure processes the feature maps at different scales output by the multi-object detection part through three graph convolutional structure branches to obtain multi-granularity fusion feature maps at different scales. The multi-granularity fusion feature maps at different scales are fused to obtain a multi-scale fusion feature map, which is then processed by the task head to obtain at least one bounding box information in the input video frame.

[0015] As a further technical solution, the graph convolutional structure branch uses graph convolutional units to segment the feature map of the input branch according to a set number of segmentation regions, thereby obtaining enhanced feature maps at different granularity levels. Finally, the enhanced feature maps at different granularity levels are fused to obtain the multi-granularity fused feature map output by the current branch.

[0016] As a further technical solution, the multi-object tracker includes a feature extraction part and a tracking part. The feature extraction part takes the processing results of the multi-object detector in adjacent video frames as input, enhances appearance features, assigns a unique identifier to the bounding box, and uses it as the detection result of the bounding box. The tracking part continuously tracks the bounding box based on the unique identifier.

[0017] As a further technical solution, the feature extraction part takes the output results of the multi-object detector in adjacent video frames as input, concatenates the input and inputs it into the low-level feature extractor. After several downsampling processes, low-level features of different scales are obtained. The low-level features of different scales are fused by the cross-scale block embedding module to obtain cross-scale appearance features. The appearance feature extractor processes the cross-scale appearance features in sequence with several appearance transformer modules to output an appearance-enhanced feature map. Finally, the appearance-enhanced feature map is processed by global average pooling and a linear classifier to assign a unique identifier to the bounding box.

[0018] As a further technical solution, in the tracking part, the Hungarian matching algorithm is used to associate the bounding box with the unique identifier; motion estimation is performed based on the Kalman filter, and the bounding box information of the next video frame is predicted based on the bounding box information of the previous video frame in adjacent video frames; based on the bounding box association and motion estimation results, the bounding box is continuously tracked in combination with the bounding box detection results.

[0019] As a further technical solution, the steps for obtaining target damage assessment results based on bounding box detection and tracking results include: extracting abrupt video frames of bounding boxes from the bounding box detection and tracking results of the video data to be processed; when the bounding box tracking of the target in the abrupt video frame is lost, and the detection result of the bounding box corresponding to the damage is added, the abrupt video frame is identified as the detonation point of the target, and the damage assessment result of the target being hit is obtained; otherwise, the damage assessment result of the target not being hit is obtained.

[0020] According to another aspect of this specification, a rapid damage assessment system based on remote sensing video target detection and tracking is provided, comprising:

[0021] The video acquisition module is used to acquire video data containing the target to be processed;

[0022] The target and damage detection and tracking module is used to input the video data to be processed into the trained detection and tracking model and output the detection and tracking results of the bounding boxes;

[0023] The damage assessment module is used to obtain the target damage assessment results of the video data to be processed based on the detection and tracking results of the bounding box.

[0024] The training process of the detection and tracking model includes:

[0025] Construct a dataset for joint detection and tracking of targets and damage;

[0026] Construct a detection and tracking model: A multi-object detector based on a multi-scale context block structure optimization processes video frames and outputs a multi-scale fused feature map and at least one bounding box information, wherein the bounding box information includes the bounding box and the label category, and the label category includes target and damage; An appearance-enhanced multi-object tracker takes the output results of the multi-object detector of adjacent video frames as input and outputs the detection and tracking results of the bounding boxes.

[0027] The detection and tracking model is trained using a dataset for joint target and damage detection and tracking, and the trained model is output.

[0028] According to another aspect of this specification, an electronic device is provided, including a memory and a processor, the memory storing program instructions executed by the processor, the processor invoking the program instructions to perform a rapid damage assessment method based on remote sensing video target detection and tracking.

[0029] According to another aspect of this specification, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium storing computer instructions that cause the computer to execute a rapid damage assessment method based on remote sensing video target detection and tracking.

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention provides a detection and tracking model that uses video frames as processing units, a multi-target detector based on multi-scale context block structure optimization, introduces global context dependency in the detection process, and expresses global information more fully, thereby obtaining more accurate target or damage detection results, as well as an appearance-enhanced multi-target tracker that strengthens the appearance representation of features and associates targets or damage with unique identifiers to achieve more accurate target or damage tracking. In this way, dynamic detection and tracking of moving targets and damage are realized, which can compensate for military operations such as detection of sudden strikes.

[0031] This invention achieves rapid damage assessment by linking abrupt video frames with detection and tracking results. It can be used to predict the detonation point of a damage strike and make real-time judgments on whether a target has been hit, playing a crucial role in military decision-making. This invention effectively alleviates the problem of inaccurate target and damage detection in video data, achieving consistent detection and tracking of the target before and after damage, thereby enabling rapid assessment of whether the target has been hit.

[0032] The method of this invention has certain reference value for target tracking before and after sudden damage, and can be used for the detection of damage to intelligent transportation and military targets, filling the gaps and deficiencies in existing research on the research of target damage before and after damage and the detonation point. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 A schematic diagram of a rapid damage assessment method based on remote sensing video target detection and tracking provided in an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram of the network framework of the multi-target detector in an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of the structure of a multi-scale context block in an embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram of the network framework for the feature extraction part in an embodiment of the present invention;

[0038] Figure 5 This is a schematic diagram of appearance feature enhancement coding in an embodiment of the present invention;

[0039] Figure 6 This is a schematic diagram of the target and damage detection results in an embodiment of the present invention;

[0040] Figure 7 This is a schematic diagram of the target and damage tracking results in an embodiment of the present invention;

[0041] Figure 8 This is a schematic diagram of the rapid damage assessment results implemented in an embodiment of the present invention;

[0042] Figure 9 This is a schematic diagram of the structure of a rapid damage assessment system based on remote sensing video target detection and tracking, provided in an embodiment of the present invention.

[0043] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0044] It should be noted that:

[0045] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0046] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be decomposed, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0048] like Figure 1 As shown, a rapid damage assessment method based on remote sensing video target detection and tracking includes:

[0049] Step 1: Obtain the video data to be processed containing the target;

[0050] Step 2: Input the video data to be processed into the trained detection and tracking model, and output the detection and tracking results of the bounding boxes;

[0051] Step 3: Based on the detection and tracking results of the bounding box, obtain the target damage assessment results of the video data to be processed.

[0052] Preferably, step 1 also includes preprocessing of the remote sensing video data to be processed, which involves preprocessing the data after video frame conversion by deleting interfering video frames and converting the dataset format.

[0053] Furthermore, in step 2, the training process of the detection and tracking model includes:

[0054] Step 2-1: Construct a dataset for joint detection and tracking of targets and damage;

[0055] Step 2-2: Construct the detection and tracking model: Process video frames using a multi-object detector optimized based on a multi-scale context block structure, and output a multi-scale fused feature map and at least one bounding box information; Use an appearance-enhanced multi-object tracker to take the output results of the multi-object detectors of adjacent video frames as input, and output the detection and tracking results of the bounding boxes;

[0056] Steps 2-3: Train the detection and tracking model using the dataset for joint target and damage detection and tracking, and output the trained model.

[0057] In step 2-1, the dataset for joint target and damage detection and tracking includes: collected or simulated target and damage video data, and target and damage information annotations, wherein the target and damage information annotations consist of bounding boxes and category labels.

[0058] The damage information in this invention pertains to physical damage, namely, detecting and tracking smoke and fire in the video.

[0059] Specifically, the dataset for joint target and damage detection and tracking is a deep learning training dataset. The integrated target and damage detection and tracking dataset of this invention is obtained through online video collection, including infrared video and optical video. Since most online videos are news videos and many are synthetic, relevant preprocessing is required. Target and damage information is manually annotated using relevant annotation software. After annotation, the video is converted to frames, and preprocessing operations such as removing interfering video frames, dataset augmentation, and dataset format conversion are performed to obtain a complete integrated target and damage detection and tracking dataset, containing video target keyframes and target annotation information. Optionally, in a specific embodiment of this invention, annotations for the target and damage video data are created using DarkLabel 2.4 software. The annotation content includes: bounding boxes with X and Y coordinates of the upper left and lower right corners, and category labels (target or damage).

[0060] In step 2-2, the multi-object detector includes a multi-object detection part and a multi-scale context block structure. The multi-object detection part is a backbone network-intermediate layer encoder-detector head integrated structure based on the Yolov5 model, which processes the input video frame and outputs feature maps at high, medium, and low scales. The multi-scale context block structure processes the feature maps at different scales output by the multi-object detection part through three graph convolutional structure branches to obtain multi-granularity fusion feature maps at different scales. The multi-granularity fusion feature maps at different scales are fused to obtain a multi-scale fusion feature map, which is then processed by the task head to obtain at least one bounding box information in the input video frame.

[0061] The bounding box information includes the bounding box and label category: the bounding box includes the X and Y coordinates of the top-left and bottom-right corners of the target or damage, and the label category includes target and damage. Preferably, the bounding box information also includes a confidence score.

[0062] Based on the YOLOv5 model, the representation of video frame feature maps of targets and damage is effectively enhanced.

[0063] Specifically, the multi-object detection part is a YOLOv5-based backbone network-intermediate layer encoding-detection head integrated structure. The YOLOv5 model is divided into three parts: CSPDarknet as the backbone encoding network, PANet as the intermediate layer feature enhancement network, and a convolutional operation as the detection head part. This backbone network-intermediate layer encoding-detection head integrated structure processes the input video frames and outputs feature maps at high, medium, and low scales.

[0064] like Figure 2 As shown, the backbone coding network of the multi-object detection part adopts CSPDarknet, which sequentially passes through the first and second BottleNeckCSP (bottleneck cross-stage network) to extract image features of different scales from the input video frame. Then, it captures features of different scales through a spatial pyramid pooling (SPP) to enhance the robustness of the model to changes in target scale.

[0065] The intermediate feature enhancement network uses PANet, which enhances the semantic and spatial information of features by fusing features from different levels. Specifically, the processing results of the backbone coding network are passed through the third, fourth, fifth, sixth, and seventh BottleNeckCSPs in PANet to extract features at different scales. The third and fourth BottleNeckCSPs are followed by a 1×1 convolutional layer (Conv1×1), an upsampling layer (UpSample), and a concatenation layer (Concat, Concatenate). After reducing the feature channels through the 1×1 convolutional layer, the upsampling layer amplifies the resolution to match the introduced features to be concatenated. Finally, the concatenation layer concatenates the two, integrating multi-scale information as input to the next BottleNeckCSP layer. The fifth and sixth BottleNeckCSPs are connected to a 3×3 convolutional layer (Conv3×3) for further feature extraction and optimization, and a concatenation layer to concatenate the result of the 3×3 convolutional layer with the introduced features to be concatenated.

[0066] In this process, the splicing layers after the third and fourth BottleNeckCSPs respectively introduce the features extracted by the second and first BottleNeckCSPs as the features to be spliced. The splicing layers after the fifth and sixth BottleNeckCSPs respectively use the features extracted by the fourth and third BottleNeckCSPs, after channel adjustment processing by a 1×1 convolutional layer, as the features to be spliced.

[0067] The detection head part processes the features output by the fifth, sixth and seventh BottleNeckCSP through 1×1 convolution, and outputs feature maps of three different scales, O1, O2 and O3 (corresponding to high, medium and low scales).

[0068] Furthermore, the multi-scale context block structure processes the feature maps of different scales output by the multi-object detection part through three graph convolutional structure branches to enhance the feature representation. The output results of each branch are fused to obtain a multi-scale fused feature map. The multi-granularity fused feature maps of different scales are fused to obtain a multi-scale fused feature map. Through task head processing, at least one bounding box information in the input video frame is obtained.

[0069] Specifically, the three multi-scale feature maps O1, O2 and O3 output by the multi-target detection part lack global context dependency and the global information is not fully expressed, resulting in insufficient accuracy of the detection results. Therefore, an additional multi-scale context block structure is designed in the detection head to process the feature maps of three different scales and output the bounding box information of the target and the damage.

[0070] The multi-scale context block structure mainly consists of a graph convolutional unit (GCU), which includes three branches of graph convolutional structure, as shown in the specific structure below. Figure 3 As shown, the outputs O1, O2, and O3 from S2 are used as inputs to the graph convolutional structure branches to enhance the contextual dependencies of spatial features. This helps resolve long-distance contextual dependencies between pixels, strengthens spatial consistency between similar pixels, and reduces the distance between different pixels, thus ensuring the globality of target and damage information and outputting a multi-granularity fused feature map. The multi-granularity fused feature maps from each branch are then fused to obtain a multi-scale fused feature map. Finally, the task head performs coordinate regression and classification to obtain the target and damage detection results. Ultimately, target and damage detection is achieved.

[0071] Furthermore, the graph convolutional structure branch uses graph convolutional units to segment the feature map of the input branch according to a set number of segmentation regions, obtaining enhanced feature maps at different granularity levels. Finally, the enhanced feature maps at different granularity levels are fused to obtain the multi-granularity fused enhanced feature map output by the current branch.

[0072] Specifically, the calculation method for a single graph convolutional structure branch is as follows:

[0073] First, the features X (corresponding to feature maps O1, O2 and O3) after 1×1 convolution in the detection head layer are spatially reprojected to obtain the probability matrix Q, the encoded features Z and the adjacency matrix A;

[0074] Using the KMeans algorithm to implement region vectors Covariance Matrix The initialization is defined as follows: C1 represents the channel size of the input feature X, and V represents the feature... Figure X The number of regions that are divided is also the number of nodes in the graph structure.

[0075] To achieve multi-scale contextual remote dependencies, this invention sets the V value to 4 and 8 respectively. Figure 2 GCUi (corresponding to) Figure 2 The output features of GCU1, GCU2, and GCU3 are respectively the features containing their own feature X and the features when V=4. Features when V=8 The characteristics of the three are integrated.

[0076] Specifically, the specific algorithm for graph convolution units is shown in Table 1.

[0077] Table 1. Convolutional Unit Algorithm

[0078]

[0079] In S1, the probability matrix and encoding matrix The formula for calculating any element is as follows:

[0080]

[0081]

[0082] in, The probability that each pixel belongs to region (node) k, k∈[1,V-1], Let represent the feature vector of the character pixel in the i-th row and j-th column of the two-dimensional feature map. This represents the feature vector of the k-th graph node. Furthermore, It is the variance of all dimensions of node k, which is normalized to (0,1) using the Sigmoid function.

[0083] In step 2-2, the multi-object tracker includes a feature extraction part and a tracking part. The feature extraction part takes the processing results of the multi-object detector in adjacent video frames as input, enhances appearance features, assigns a unique identifier to the bounding box, and uses it as the detection result of the bounding box. The tracking part continuously tracks the bounding box based on the unique identifier.

[0084] Furthermore, in the feature extraction section, the output results of the multi-object detector in adjacent video frames are used as input. The input is concatenated and then fed into a low-level feature extractor. Through several downsampling processes, low-level features at different scales are obtained. The low-level features at different scales are fused through a cross-scale block embedding module to obtain cross-scale appearance features. The appearance feature extractor processes the cross-scale appearance features sequentially through several appearance transformer modules, outputting an appearance-enhanced feature map. Finally, the appearance-enhanced feature map is processed by global average pooling and a linear classifier to assign a unique identifier to the bounding box.

[0085] In the tracking part, the Hungarian matching algorithm is used to associate the bounding box with the unique identifier; motion estimation is performed based on the Kalman filter, and the bounding box information of the next video frame is predicted based on the bounding box information of the previous video frame in the adjacent video frames; the bounding box is continuously tracked based on the bounding box association and motion estimation results, combined with the bounding box detection results.

[0086] Specifically, such as Figure 4 As shown, the feature extraction part is a prerequisite for multi-target trackers, aiming to achieve global information and appearance-enhanced target and damage feature representations based on the appearance features of different targets and damage. This part utilizes the bounding box information of two consecutive frames of remote sensing video as input to achieve target and damage appearance feature representations with motion features, used to extract feature representations from the bounding boxes of detected targets or damage. These representations are used to uniquely identify targets or damage.

[0087] Step 2-2-1: Process the two frames output by the multi-target detector and concatenate the two input tensors x1 and x2 to form a larger input tensor x;

[0088] In step 2-2-2, the low-level feature extractor uses convolutional blocks (Conv) to downsample the input data multiple times for feature extraction, and uses the cross-scale patch embedding module to further enhance the feature map, outputting cross-scale appearance features.

[0089] The cross-scale block embedding module proposes to fuse information using multi-scale extended convolution. Let k represent the downsampling times in the feature extraction part of the multi-object tracker, H and W represent the height and width of the input features, and C be the original number of channels. The corresponding shape is... The low-level features, with a stride of Expanded convolution (expansion range is 1 to 1) All collected features are concatenated and fused using a linear layer to obtain cross-scale appearance features. Thus, the finer the granularity of the features, the more information they provide. This step yields fine-grained cross-scale appearance features, which are then fed into the appearance feature extractor.

[0090] Step 2-2-3, the Appearance Feature Extractor, uses several Appearance Transformer Blocks to process cross-scale appearance features sequentially. These Appearance Transformer Blocks are connected by downsampling blocks; this is the core of the appearance enhancement module structure, as shown below. Figure 5 As shown in (b). In the Appearance Transformer Block, the processing steps include inter-frame attention mechanism, multilayer perceptron (MLP), etc.

[0091] The Appearance Transformer Block implements enhanced appearance feature encoding through an inter-frame attention mechanism, as illustrated in the diagram below. Figure 5 As shown:

[0092] Assume the coding features of the previous layer of adjacent video frames are denoted as A0 and A1, respectively. Where A represents the encoding feature, the subscripts correspond to the order, 0 represents the previous video frame, and 1 represents the next video frame; This represents the feature size. Taking the appearance enhancement process of the previous video frame as an example, for any selected region in the previous video frame I0... Using this region and its spatial neighborhood in the subsequent video frame I1. This is used to generate the query matrix Q and the key / value (K / V) matrix, where N represents the neighborhood window size.

[0093]

[0094] in, , , It belongs to the linear projection matrix. Then, a dot product operation is performed on... and each position This is applied in the process, and then SoftMax is used to generate attention feature maps. The value at each position represents... The similarity value between it and its neighbors.

[0095]

[0096] To enhance appearance information, firstly, similar appearance information in the I1 frame is summarized, and then compared with... The fusion process is then performed, and finally, the appearance features in I0 are enhanced, mathematically represented as follows:

[0097]

[0098] Similarly, enhanced appearance features can also be obtained. Enhanced appearance features , It includes a blend of the appearances of similar regions in two different frames, which can provide more information about how the appearance changes between frames, and can be used to enhance the appearance feature representation of different categories of targets or damage in frames.

[0099] Step 2-2-4: After the last feature extraction stage (final Appearance Transformer Block), perform global average pooling on the features to obtain a feature vector of fixed size.

[0100] Step 2-2-5: Classify the feature vectors after global pooling using a simple linear classifier;

[0101] Step 2-2-6: Finally, return the final output of the model, outputting the unique identifier of the bounding box.

[0102] Specifically, the tracking part mainly includes Hungarian Assignment and Kalman Predict. This part takes the enhanced appearance feature map output by the feature extraction part as input. Based on its function, the tracking part can be divided into several sub-parts: target association, motion estimation, and target tracking.

[0103] In the target association section, the Hungarian matching algorithm is used to associate the detected bounding boxes in the current frame with the tracked bounding boxes; this helps determine which bounding boxes are the same bounding box across consecutive frames. Typically, each bounding box is associated with a unique ID to identify it during tracking.

[0104] In the motion estimation part, a Kalman filter is used for bounding box motion estimation, which can be used to predict the position of the bounding box in the next frame.

[0105] In the target tracking section, the Kalman filter method is used to update the bounding box state. Based on target association and motion estimation, the bounding box is continuously tracked in conjunction with the bounding box detection results. It is used to correlate the target's state with time steps.

[0106] Preferably, the tracking portion may further include a re-identification portion for handling changes in the appearance of the target or damage within the bounding box, such as crossing an occluded area or changing pose. By comparing the feature representations within the bounding boxes, the bounding boxes are more reliably reassociated with previously tracked bounding boxes, determining whether they belong to the same bounding box.

[0107] Specifically, in steps 2-3, during the training of the detection and tracking model using the dataset for joint target and damage detection and tracking, the hyperparameters of each network are fine-tuned multiple times, including the learning rate, batch size, and weight decay. After multiple parameter tunings, the optimal model is selected as the trained model to achieve stable target and damage tracking on effective video data, obtaining the tracking bounding box output, class label, and tracking ID number. Quantitative accuracy evaluation is then performed to verify the model performance.

[0108] The quantitative evaluation analysis of this invention is divided into two parts: target detection algorithm and target tracking algorithm. In the experiment, precision (P), recall (R), and mean average precision (mAP) are used as evaluation metrics for the improved YOLOv5 model detection algorithm. Multiple object tracking accuracy (MOTA) and multiple object tracking precision (MOTP) are selected to represent the accuracy of target tracking and the mismatch between the actual bounding box and the predicted bounding box, respectively. The specific calculation formulas are shown below:

[0109]

[0110]

[0111]

[0112]

[0113] In this dataset, TP refers to a positive sample predicted as positive by the model, TN refers to a negative sample predicted as negative by the model, FP refers to a negative sample predicted as positive by the model, and FN refers to a positive sample predicted as negative by the model. C0 refers to the classification category. The dataset of this invention has a total of 4 categories, so C0 is 4.

[0114]

[0115]

[0116] In this context, FP in the tracking process refers to the total number of false alarms, which is the total number of FPs in the entire video. The number of FPs in each frame needs to be summed. FN refers to the total number of missed alarms, which is the total number of FNs in the entire video. The number of FNs in each frame also needs to be summed. ID s This refers to the total number of ID transitions assigned to the target in the tracking trajectory, i.e., the sum of the number of ID transitions occurring in each frame. Furthermore, in MOTP, t represents the t-th frame, and d... t,i c represents the distance between the center point of the detection box and the center point of the ground truth detection box in frame t. t This represents the number of predicted bounding boxes that successfully match the actual bounding boxes in frame t. MOTA and MOTP together measure the performance of the algorithm in continuously tracking targets. The closer the MOTA value is to 1 and the closer the MOTP value is to 0, the higher the performance of the algorithm in continuously tracking targets.

[0117] In step 3, the step of obtaining the target damage assessment result based on the bounding box detection and tracking results includes: extracting the abrupt video frames of the bounding boxes from the bounding box detection and tracking results of the video data to be processed; when the bounding box tracking of the target in the abrupt video frame is lost, and the detection result of the bounding box corresponding to the damage is added, the abrupt video frame is identified as the detonation point of the target, and the damage assessment result of the target being hit is obtained; otherwise, the damage assessment result of the target not being hit is obtained.

[0118] Specifically, by extracting the transition frames, the detonation point of the target can be identified, thus determining whether the target has been hit and achieving rapid damage assessment. Final experiments demonstrate that this application can achieve damage assessment for single and multiple targets. Figure 6 The results show the detection of different types of targets and damage, presented as five consecutive frames containing abrupt changes before and after a target is hit. Figure 6 The first line shows a single aircraft target. Figure 6 The second line represents a single ship target. Figure 6 The third line contains multiple vehicle targets; Figure 7 Showing the Figure 6The target and damage tracking results show that, for the detection part, the precision is 0.93, the recall is 0.91, and the F1 score is 0.96; for the tracking part, the MOTA score is 0.92 and the MOTP score is 15.78, indicating that the proposed joint target and damage detection and tracking method based on remote sensing video has good performance and strong generalization ability. Figure 8 (Right) illustrates the specific process of extracting mutation frames from a single aircraft target. Figure 8 (Left) indicates the detonation point. For rapid damage assessment, this allows for quick determination of whether the target has been hit and the detonation point at the time of impact. This invention proposes a novel approach for the joint detection and tracking of moving targets and their damage information, and has significant reference value.

[0119] The implementation of the various embodiments of the present invention is based on programmed processing through a system with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of the present invention are encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of the present invention provide a rapid damage assessment system based on remote sensing video target detection and tracking. This system is used to execute a rapid damage assessment method based on remote sensing video target detection and tracking from the above method embodiments.

[0120] See Figure 9 The system includes:

[0121] The system comprises the following modules: a video acquisition module for acquiring video data containing targets; a target and damage detection and tracking module for inputting the video data into a trained detection and tracking model and outputting bounding box detection and tracking results; and a damage assessment module for obtaining target damage assessment results based on the bounding box detection and tracking results. The training process of the detection and tracking model includes: constructing a dataset for joint target and damage detection and tracking; constructing a detection and tracking model by processing video frames using a multi-target detector optimized based on a multi-scale context block structure, outputting a multi-scale fused feature map and at least one bounding box, wherein the bounding box information includes bounding boxes and label categories, with label categories including targets and damage; using an appearance-enhanced multi-target tracker, inputting the output results of the multi-target detector from adjacent video frames, and outputting bounding box detection and tracking results; and training the detection and tracking model using the dataset for joint target and damage detection and tracking, outputting the trained model.

[0122] It should be noted that the system embodiments provided by the present invention are used not only to implement the methods in the above method embodiments, but also to implement the methods in other method embodiments provided by the present invention. The only difference is that corresponding functional modules are set. The principle is basically the same as that of the above system embodiments provided by the present invention. As long as those skilled in the art can improve the modules in the above system embodiments by referring to the specific technical solutions in other method embodiments and combining technical features to obtain corresponding technical means and technical solutions composed of these technical means, on the basis of the above system embodiments, and on the premise of ensuring the practicality of the technical solutions, they can obtain corresponding system-like embodiments for implementing the methods in other method-like embodiments.

[0123] The method in this embodiment of the invention is implemented using an electronic device; therefore, it is necessary to introduce the relevant electronic device. For this purpose, embodiments of the present invention provide an electronic device, such as... Figure 10 As shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus, wherein the at least one processor, the communication interface, and the at least one memory communicate with each other via the communication bus. The at least one processor invokes logical instructions stored in the at least one memory to execute all or part of the steps of the methods provided in the foregoing method embodiments.

[0124] Furthermore, when the logical instructions in at least one of the aforementioned memories are implemented as software functional units and sold or used as independent products, they are stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, is embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (a personal computer, server, or network device) to execute all or part of the steps of the methods described in the various method embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks—various media for storing program code.

[0125] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, located in one place, or distributed across multiple network units. The purpose of this embodiment is achieved by selecting some or all of the modules according to actual needs. Those skilled in the art will understand and implement this without any inventive effort.

[0126] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0127] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0128] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0129] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A rapid damage assessment method based on remote sensing video target detection and tracking, characterized in that, include: Acquire the video data to be processed, which contains the target. The video data to be processed is input into the trained detection and tracking model, which outputs the detection and tracking results of bounding boxes. Based on the detection and tracking results of the bounding box, the target damage assessment results of the video data to be processed are obtained; The training process for the detection and tracking model includes: Construct a dataset for joint detection and tracking of targets and damage; Construct a detection and tracking model: A multi-object detector based on a multi-scale context block structure optimization processes video frames and outputs a multi-scale fused feature map and at least one bounding box information, wherein the bounding box information includes the bounding box and the label category, and the label category includes target and damage; An appearance-enhanced multi-object tracker takes the output results of the multi-object detector of adjacent video frames as input and outputs the detection and tracking results of the bounding boxes. The detection and tracking model is trained using a dataset for joint target and damage detection and tracking, and the trained model is output.

2. The rapid damage assessment method based on remote sensing video target detection and tracking as described in claim 1, characterized in that, The multi-object detector includes a multi-object detection part and a multi-scale context block structure. The multi-object detection part is a Yolov5-based backbone network-intermediate layer encoder-detector head integrated structure that processes the input video frame and outputs feature maps at high, medium, and low scales. The multi-scale context block structure processes the feature maps at different scales output by the multi-object detection part through three graph convolutional branches to obtain multi-granularity fused feature maps at different scales. The multi-granularity fused feature maps at different scales are then fused to obtain a multi-scale fused feature map. This multi-scale fused feature map is then processed by the task head to obtain at least one bounding box information in the input video frame.

3. The rapid damage assessment method based on remote sensing video target detection and tracking as described in claim 2, characterized in that, The graph convolution structure branch uses graph convolution units to segment the feature map of the input branch according to a set number of segmentation regions, thereby obtaining enhanced feature maps at different granularity levels. Finally, the enhanced feature maps at different granularity levels are fused to obtain the multi-granularity fused feature map output by the current branch.

4. The rapid damage assessment method based on remote sensing video target detection and tracking as described in claim 1, characterized in that, The multi-target tracker includes a feature extraction part and a tracking part. The feature extraction part takes the processing results of the multi-target detector in adjacent video frames as input, enhances appearance features, and assigns a unique identifier to the bounding box as the detection result of the bounding box. The tracking part continuously tracks the bounding box based on the unique identifier.

5. The rapid damage assessment method based on remote sensing video target detection and tracking as described in claim 4, characterized in that, The feature extraction part takes the output results of the multi-target detector in adjacent video frames as input, concatenates the inputs and inputs them into the low-level feature extractor, and obtains low-level features at different scales through several downsampling processes; Cross-scale appearance features are obtained by fusing low-level features at different scales through a cross-scale block embedding module. The appearance feature extractor processes cross-scale appearance features sequentially using several appearance transformer modules, outputting an appearance-enhanced feature map. Finally, the appearance-enhanced feature map is processed by global average pooling and a linear classifier to assign a unique identifier to the bounding box.

6. The rapid damage assessment method based on remote sensing video target detection and tracking as described in claim 4, characterized in that, The tracking component employs a Hungarian matching algorithm to associate bounding boxes with unique identifiers; it performs motion estimation based on a Kalman filter, predicting the bounding box information of the next video frame based on the bounding box information of the previous video frame in adjacent video frames; and it continuously tracks bounding boxes based on the results of bounding box association and motion estimation, combined with the bounding box detection results.

7. The rapid damage assessment method based on remote sensing video target detection and tracking as described in claim 1, characterized in that, The steps for obtaining target damage assessment results based on bounding box detection and tracking results include: extracting abrupt video frames of bounding boxes from the bounding box detection and tracking results of the video data to be processed; when the bounding box tracking of the target in the abrupt video frame is lost, and the detection result of the bounding box corresponding to the damage is added, the abrupt video frame is identified as the detonation point of the target, and the damage assessment result of the target being hit is obtained; otherwise, the damage assessment result of the target not being hit is obtained.

8. A rapid damage assessment method system based on remote sensing video target detection and tracking, characterized in that, include: The video acquisition module is used to acquire video data containing the target to be processed; The target and damage detection and tracking module is used to input the video data to be processed into the trained detection and tracking model and output the detection and tracking results of the bounding boxes; The damage assessment module is used to obtain the target damage assessment results of the video data to be processed based on the detection and tracking results of the bounding box. The training process of the detection and tracking model includes: Construct a dataset for joint detection and tracking of targets and damage; Construct a detection and tracking model: A multi-object detector based on a multi-scale context block structure optimization processes video frames and outputs a multi-scale fused feature map and at least one bounding box information, wherein the bounding box information includes the bounding box and the label category, and the label category includes target and damage; An appearance-enhanced multi-object tracker takes the output results of the multi-object detector of adjacent video frames as input and outputs the detection and tracking results of the bounding boxes. The detection and tracking model is trained using a dataset for joint target and damage detection and tracking, and the trained model is output.

9. An electronic device, characterized in that, The method includes a memory and a processor, the memory storing program instructions that are executed by the processor, the processor invoking the program instructions to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the method described in any one of claims 1 to 7.

Images