Multi-aircraft tracking method and system based on visible light images

Abstract

The application discloses a kind of multi-aircraft tracking method and system based on visible light image, belong to multi-aircraft tracking, digital image processing technical field, wherein, the method includes: the current input image frame is preprocessed, the image frame feature pre-extraction is extracted to the processed by feature extraction network E-ResNet50, again with previous frame pre-extraction feature fusion, the feature after fusion is input into 4 groups of cascaded context perception coding layer and carries out feature coding;Encoded feature is input into 2 groups of 4 groups of cascaded decoding layer and respectively realizes the detection of current frame image aircraft target and the extension to previous frame aircraft target, the detected current frame aircraft target and extended aircraft target are input into motion appearance correlation module and are associated to realize the tracking task of multi-aircraft target.The method improves the tracking precision of multi-aircraft target, overcomes the inadaptability of conventional multi-target tracking algorithm to multi-aircraft tracking task.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-aircraft tracking and digital image processing technology, and particularly relates to a multi-aircraft tracking method based on visible light images. BACKGROUND

[0002] Airspace reconnaissance surveillance is the main content of guaranteeing national security and people's livelihood security. With the development of modern electronic technology, using reconnaissance means to discover, identify, monitor, track targets and locate targets will undoubtedly play a greater role in future wars. Among them, unmanned reconnaissance aircraft is increasingly valued due to its low cost, high reliability, ability to complete dangerous reconnaissance tasks, small infrared radiation, and difficulty in being discovered. The visible light camera carried by the unmanned reconnaissance aircraft can transmit reconnaissance images back to the ground station in time through high-altitude and low-altitude photography. At the same time, with the rapid development of air traffic, the throughput of civil aviation is increasing, leading to the continuous increase of airway transportation density, and the problem of people's livelihood security caused by civil aviation is increasingly prominent. Therefore, it is particularly important to study the multi-aircraft tracking technology using computer vision and image processing technology to analyze the trajectory of each aircraft, thereby achieving reconnaissance and guaranteeing people's livelihood security.

[0003] The existing aircraft tracking algorithm can be divided into infrared image-based aircraft tracking algorithm and visible light image-based aircraft tracking algorithm according to the data source. Among them, the infrared image-based aircraft tracking algorithm can be further divided into edge information-based, correlation filtering, centroid motion trajectory, Kalman filtering technology and neural network technology-based aircraft tracking algorithm. Due to the disadvantages of low contrast, low signal-to-noise ratio, lack of texture information and high cost of infrared images, there are numerous aircraft tracking algorithms based on visible light images in recent years, such as morphological technology, particle filtering technology, Kalman filtering technology, optical flow method and neural network-based aircraft tracking algorithm. However, the current research on aircraft tracking mainly focuses on single aircraft. Since multi-aircraft tracking has more practical application prospects, the current research on multi-aircraft tracking is in the blank. The current mainstream multi-target tracking mainly focuses on tracking pedestrians and vehicles. According to the different detection methods, it can be divided into multi-target tracker without detection, multi-target tracker based on detection and multi-target tracker with synchronous detection. Due to the particularity of multi-aircraft tracking, such as small target and micro target formation characteristics, completely similar appearance characteristics and even serious image blur problem caused by shaking due to long-distance shooting, the multi-target tracking algorithm cannot be directly applied to the multi-aircraft tracking task.

[0004] Therefore, there is an urgent need for a solution to the inapplicability of the existing multi-target tracking algorithm to the multi-aircraft tracking problem, to provide new algorithm choices for multi-aircraft tracking tasks. SUMMARY

[0005] The application provides a multi-aircraft tracking method and system based on a visible light image to solve the technical problem of inapplicability of existing multi-target tracking algorithms to multi-aircraft tracking.

[0006] An embodiment of the application provides a multi-aircraft tracking method based on a visible light image, comprising the following steps:

[0007] In step S1, a current input image frame is preprocessed.

[0008] In step S2, the preprocessed image frame is input into a feature extraction network E-ResNet50 for preliminary feature extraction to obtain pre-extracted features of the current frame.

[0009] In step S3, the pre-extracted features of the current frame are fused with pre-extracted features of a previous frame.

[0010] In step S4, the fused features are input into four cascaded context-aware encoding layers to obtain encoded features.

[0011] In step S5, the encoded features are input into two groups of four cascaded decoding layers to respectively detect aircraft targets in the current frame image and extend aircraft targets in the previous frame image, to obtain detected aircraft targets and extended aircraft targets.

[0012] In step S6, the detected aircraft targets and the extended aircraft targets are input into a current motion appearance correlation module for correlation of the multi-aircraft to complete multi-aircraft tracking.

[0013] Another embodiment of the application provides a multi-aircraft tracking system based on a visible light image, comprising:

[0014] A preprocessing module is configured to preprocess a current input image frame.

[0015] A feature extraction module is configured to input the preprocessed image frame into a feature extraction network E-ResNet50 for preliminary feature extraction to obtain pre-extracted features of the current frame.

[0016] A fusion module is configured to fuse the pre-extracted features of the current frame with pre-extracted features of a previous frame.

[0017] An encoding module is configured to input the fused features into four cascaded context-aware encoding layers to obtain encoded features.

[0018] A decoding module is configured to input the encoded features into two groups of four cascaded decoding layers to respectively detect aircraft targets in the current frame image and extend aircraft targets in the previous frame image, to obtain detected aircraft targets and extended aircraft targets.

[0019] a tracking module, configured to input the detected aircraft target and the extended aircraft target into a current motion appearance correlation module to complete multi-aircraft correlation for multi-aircraft tracking.

[0020] In still another aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the multi-aircraft tracking method based on visible light images as described in the above embodiment when executing the computer program.

[0021] In still another aspect, an embodiment of the present application provides a non-transitory computer readable storage medium, having a computer program stored thereon, wherein the computer program is executable on a processor to implement the multi-aircraft tracking method based on visible light images as described in the above embodiment.

[0022] The technical scheme of the present application at least achieves the following beneficial technical effects: based on TransTrack, the receptive field enhancement module, the context-aware encoding module, and the motion appearance correlation module are introduced to form a robust multi-aircraft tracking algorithm based on visible light images, the technical problem of inapplicability of existing multi-target tracking algorithms to multi-aircraft tracking is solved, the tracking accuracy for multi-aircraft targets is improved, and new algorithm selection is provided for multi-aircraft tracking tasks.

[0023] Additional aspects and advantages of the present application will be made apparent by the following description and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0024] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0025] Figure 1 is a multi-aircraft tracking method based on visible light images according to an embodiment of the present application;

[0026] Figure 2 is a whole structure diagram of a multi-aircraft tracking method based on visible light images according to an embodiment of the present application;

[0027] Figure 3 is a flowchart of a multi-aircraft tracking system based on visible light images according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0029] The following describes a multi-aircraft tracking method and system based on visible light images according to an embodiment of the present invention with reference to the accompanying drawings. First, the multi-aircraft tracking method based on visible light images according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0030] Figure 1 This is a flowchart of a multi-aircraft tracking method based on visible light images according to an embodiment of the present invention.

[0031] like Figure 1 As shown, this multi-vehicle tracking method based on visible light images includes the following steps:

[0032] In step S1, the current input image frame is preprocessed.

[0033] Specifically, based on TransTrack for the current image frame I to be tracked t All image frames input to the feature extraction network need to undergo preprocessing operations, including: resizing, center cropping, horizontal flipping, and normalization. These preprocessing operations are well known to those skilled in the art, and will not be described in detail in this embodiment of the invention.

[0034] In step S2, the preprocessed image frame is input into the feature extraction network E-ResNet50 for preliminary feature extraction to obtain the pre-extracted features of the current frame.

[0035] Furthermore, in one embodiment of the present invention, the feature extraction network E-ResNet50 includes a cascaded 7×7 convolutional layer, a max pooling layer, four residual blocks, and a receptive field enhancement block. The four residual blocks are each composed of a cascaded 1×1, a 3×3, and a 1×1 convolutional layer. The receptive field enhancement block is composed of two 1×1, two 3×3, and one 7×7 convolutional layers connected in parallel, so as to map the preprocessed image frame into a high-dimensional feature space.

[0036] Specifically, the preprocessed image frame is input into the enhanced feature extraction network E-ResNet50 at the front end to obtain the pre-extracted features F of the current frame. 1,t And store it in the feature library G.

[0037] In step S3, the pre-extracted features of the current frame are fused with the pre-extracted features of the previous frame.

[0038] Specifically, the features F pre-extracted from the previous frame image are extracted from the feature library G. 1,t-1 , will F 1,t With F 1,t-1 The feature map F is obtained by fusion. 1,t,t-1 .

[0039] In step S4, the fused features are input into four cascaded context-aware coding layers to obtain coded features.

[0040] like Figure 2 As shown, further, in one embodiment of the present invention, step S4 specifically includes:

[0041] Step S401: Input the fused features into the first context-aware coding layer CaEL1 to capture features with inter-frame information, and obtain the first coded feature F. CaEL1 Among these, it is necessary to process the fused features using F... 1,t,t-1 The corresponding vector features H are obtained by expanding the image space into the vector space according to the coordinates. t,t-1 Then it is input into the context-aware coding layer CaEL1 to obtain the first coding feature F. CaEL1 .

[0042] Step S402, the first encoded feature F CaEL1 The input is fed into the second context-aware coding layer CaEL2 to obtain the second coding feature F. CaEL2 ;

[0043] Step S403, the second encoded feature F CaEL2 The input is fed into the third context-aware coding layer CaEL3 to obtain the third coding feature F. CaEL3 ;

[0044] Step S404, the third encoded feature F CaEL3 The input is fed into the fourth context-aware coding layer CaEL4 to obtain the fourth coding feature F. CaEL4 .

[0045] Among them, the first context-aware coding layer CaEL1, the second context-aware coding layer CaEL2, the third context-aware coding layer CaEL3, and the fourth context-aware coding layer CaEL4 are structurally composed of two parts, global blocks and local blocks, according to feature attributes. At the end of each context-aware coding layer, the features obtained from the global block and the local block are fused together again as the output of the current context-aware coding layer.

[0046] Specifically, for the input high-dimensional features (i.e., the fused features), they are unfolded from the image space to a two-dimensional vector space according to coordinates. These features, along with the corresponding positional encoding vectors, are then input into a (convolution-normalization-activation) combination operation for feature remapping. The mapped features are then input into a multi-scale deformable attention module for feature encoding. The encoded features are then input into a feedforward network to achieve spatial transformation, enhancing the model's expressive power. This yields global features. These global features are then input into a channel splitting and adjustment module to achieve another transformation from vector space to graph space, converting them into four feature maps of different scales (1 / 8, 1 / 16, 1 / 32, and 1 / 64 of the input image frame size, respectively). Each feature map is then extracted using a combination operation (convolution-normalization). The extracted feature maps are then input back into the channel splitting and adjustment module to achieve a transformation from image space to vector space, resulting in vector features containing local information. These vector features are then fused with the global features to obtain the output of the context-aware encoding layer, which considers both global and local features.

[0047] In step S5, the encoded features are input into two groups of four cascaded decoding layers to detect the aircraft target in the current frame image and the aircraft target in the extended previous frame image, thereby obtaining the detected aircraft target and the extended aircraft target.

[0048] like Figure 2 As shown, further, in one embodiment of the present invention, step S5 specifically includes:

[0049] Step S501, the fourth encoded feature F CaEL4 The inputs are respectively fed into the first decoding layer DL1-1 of the first group and the first decoding layer DL2-1 of the second group to obtain the first decoding feature F. DL1-1 Second-to-second decoding feature F DL2-1 ;

[0050] Step S502, decode the first feature F DL1-1 The input is fed into the second decoding layer DL1-2 of the first group to obtain the first and second decoding features F. DL1-2 At the same time, the second decoding feature F DL2-1 The input is fed into the second decoding layer DL2-2 of the second group to obtain the second binary decoding feature F. DL2-2 ;

[0051] Step S503, decode the first feature F DL1-2 The input is fed into the third decoding layer DL1-3 of the first group to obtain the third decoding feature F. DL1-3 At the same time, the second decoding feature F DL2-2 The input is fed into the third decoding layer DL2-3 of the second group to obtain the second and third decoding features F. DL2-3 ;

[0052] Step S504, decode the third feature F DL1-3 The input is fed into the fourth decoding layer DL1-4 of the first group to obtain the first four-decoding feature F. DL1-4,Dt and the first decoding position B Dt At the same time, the second and third decoding features F DL2-3 The input is fed into the fourth decoding layer DL2-4 of the second group to obtain the second fourth decoding feature F. DL2-4,Et and the second decoding position B Et Wherein, the first decoding position B Dt To detect the aircraft target, the second decoding position B Et To extend the target of the aircraft.

[0053] The decoding layers DL1-1, DL1-2, DL1-3, and DL1-4 of Group 1 and DL2-1, DL2-2, DL2-3, and DL2-4 of Group 2 are all structurally composed of three parts: self-attention, mutual attention, and a feedforward network. For Group 1, the primary function is to detect the aircraft target in the current frame and output the detected target position B in the current frame. Dt and the corresponding feature F DL1-4,Dt The second decoding layer is mainly used to extend the trajectory of the aircraft target in the previous frame to the current frame, and output the position B of the aircraft target in the previous frame in the current frame. Et and the corresponding feature F DL2-4,Et .

[0054] In step S6, the detected aircraft target and the extended aircraft target are input into the current motion appearance association module to perform multi-aircraft association and complete multi-aircraft tracking. The current motion appearance association module is structurally composed of three modules consisting of multi-layer sensing blocks.

[0055] Specifically, the motion information and appearance color information of the detected aircraft target, the extended target bounding box corresponding to the aircraft target, are input into the current motion appearance association module. First, high-dimensional feature mapping is performed to form the features of each aircraft target. Then, the position information B of each aircraft target is combined. Dt B Et The Hungarian matching algorithm is used to correlate the detection location and the extended location to achieve multi-aircraft tracking.

[0056] The following two specific embodiments further illustrate the multi-aircraft tracking method based on visible light images proposed in this invention. Specific Implementation Method 1

[0058] All image frames input to the feature extraction network undergo preprocessing. These preprocessed image frames are then fed into the enhanced feature extraction network E-ResNet50 for feature pre-extraction. E-ResNet50 is constructed by stacking a series of convolutions and max pooling operations, and its specific structural parameters are shown in Table 1. As can be seen from the table, this implementation uses a large 7×7 convolution kernel for feature extraction on the 32x downsampled feature map at the receptive field enhancement module. By balancing efficiency and accuracy, the 7×7 convolution kernel was ultimately selected, abandoning the conventional use of a 3×3 convolution kernel. This maximizes the model's receptive field, enabling the model to perceive the presence of multi-scale objects.

[0059] Table 1

[0060]

[0061] Furthermore, the pre-extracted features F of the current frame 1,t Pre-extracted features F from the previous frame image 1,t-1 Perform fusion and analyze the fused feature map F 1,t,t-1 Expanding from image space to vector space by coordinates specifically includes:

[0062] The fusion of pre-extracted features from two consecutive frames is mainly accomplished through the fusion network shown in Table 2. The input pre-extracted features consist of four parts, namely, the current frame F... 1,t ={F 1,t 8 F 1,t 16 F 1,t 32 F 1,t 64} and the previous frame F 1,t-1 ={F 1,t-1 8 F 1,t-1 16 F 1,t-1 32 F 1,t-1 64 The expression `}` represents the extraction of pre-extracted feature maps with downsampling rates of 8x, 16x, and 32x from the enhanced feature extraction network. These are the residual blocks 2 and 3 in Table 1, as well as the output feature maps of the last layer in the receptive field enhancement module. The 64x downsampled feature map is obtained by further downsampling the current 32x downsampled feature maps using a (1×1 convolution-activation) combination operation. Therefore, the pre-extracted features with the same downsampling rate for two consecutive frames are each input into the fusion layer shown in Table 2, outputting the fused features of the two frames at four downsampling rates. H and W are the dimensions of the input frame.

[0063] Table 2

[0064]

[0065] The pre-extracted features after the above fusion Unfolding from image space to vector space yields the unfolded fused features. The expanded feature vectors from the four scales are concatenated to obtain the final multi-scale fused concatenated vector map. j=[(H×W) / 8+(H×W) / 16+(H×W) / 32+(H×W) / 64].

[0066] Furthermore, the context-aware coding layer CaEL structure is described in [reference needed]. Figure 1 This embodiment uses four CaELs: CaEL1, CaEL2, CaEL3, and CaEL4. Each context-aware coding layer consists of four parts: a multi-scale deformable attention module, a feedforward network, channel splitting and adjustment (inverse), and local feature extraction. Specifically, for the input feature vector... j = [(H×W) / 8 + (H×W) / 16 + (H×W) / 32 + (H×W) / 64]. Firstly, following the Transformer architecture, to compensate for the shortcomings of Transformer permutation invariance, the position corresponding to each pixel is first encoded and input into the context-aware coding layer. This implementation adopts the sine and cosine position coding method in absolute position coding, as detailed below:

[0067]

[0068]

[0069] Where i∈{0,1,2,…}, l is the boundary value of the corresponding dimension, and ε= is a differential factor, which is set to 10 in this embodiment. -6 p = [0, 1, ..., 64] represents the encoding baseline. Different encoding methods are used for different coordinate positions. Therefore, the above formula can calculate the encoding value of the j-th bit in the encoding vector corresponding to position i. At this time, each coordinate point position will be encoded into a fixed-length and distinct position vector. This embodiment uses a 64-dimensional feature length. Since each position consists of two dimensions (x, y), each position will obtain two position encoding vectors. Concatenating the two position encoding vectors in the order of x and y dimensions yields the 128-dimensional position encoding vector corresponding to each coordinate position. Therefore, for an 8x downsampled image, a matrix composed of position encoding vectors is obtained. The matrix consisting of position encoding vectors corresponding to 16x, 32x, and 64x downsampling was obtained in the same manner. Finally, the matrix composed of all the position encoding vectors is concatenated to obtain the final position encoding matrix. j = [(H×W) / 8 + (H×W) / 16 + (H×W) / 32 + (H×W) / 64]. To reduce model complexity, this invention pre-extracts the 256-dimensional feature vector from the input. An adaptive anti-bottleneck mapping is performed, resulting in 128-dimensional vector features for global feature extraction. This adaptive approach involves constructing a set of convolution-batch normalization-activation operations. The mapped vector features are... Combine it with the position encoding matrix The fusion will proceed again, and the fusion method is as follows:

[0070] Ψ(i,j)=H t ′ ,t-1 (i,j)+PE(i,j)(3)

[0071] This is the fused pre-extracted feature matrix after incorporating positional encoding information. Ψ is further input into a multi-scale deformable attention module to obtain the output encoded features as follows.

[0072]

[0073] The multi-scale deformable attention module is based on key-value pairs and queries via query vectors. Here, Q represents the queries required by the multi-scale deformable attention module, V represents the values ​​required by the multi-scale deformable attention module, and both are pre-extracted fusion features Ψ. Q and W V Here, s represents the corresponding attention weights, s is the normalization operation (softmax probability normalization is used in this embodiment), ei = {1, 2, 3, 4} represents the context-aware coding layer index, and K represents the keys required by the multi-scale deformable attention module, specifically corresponding to the sampling points at different positions in the downsampling map. In this embodiment, the sampling points are obtained according to the following steps:

[0074] Since the input feature vector includes four scales: 8x downsampling, 16x downsampling, 32x downsampling, and 64x downsampling, the sampling points are also extracted within these four scales.

[0075] Step 1: Obtain the sampling point location.

[0076] For an 8x downsampled image, its size is H / 8 × W / 8. Divide the horizontal and vertical directions into equal intervals of H / 8 and W / 8 respectively, and take the coordinates of the intersection points as the sampling point locations, resulting in a total of H / 8 × W / 8 sampling points {(x i ,y i )|x i ∈[1,H / 8]∩N + ,y i ∈[1,W / 8]∩N + .

[0077] Step 2: Normalize the sampling point location.

[0078] The sampling point positions obtained from Step 1 are normalized to the range [0,1] as follows:

[0079]

[0080] At this point, we obtain the normalized coordinate matrix corresponding to the 8x downsampled image. Repeat Step 1 to Step 2 to obtain the normalized coordinates of the sampling points in the 16x, 32x, and 64x downsampling images, with the following values ​​respectively. splicing SPN8, SPN 16 SPN 32 SPN 64 This yields the coordinate matrices of all sampling points across four scales. j=[(H×W) / 8+(H×W) / 16+(H×W) / 32+(H×W) / 64].

[0081] Step 3: The sampling point locations for the secondary mapping are as follows:

[0082] SPN′(:,i,2)=SPN(:,2)×sr(i)(6)

[0083] i = {0, 1, 2, 3}, sr = {8, 16, 32, 64} represents the sampling rate. The sampling points at each scale are then mapped back to four different sizes to obtain the positions of the secondary sampling points.

[0084] Furthermore, the feedforward network structure is described in [reference needed]. Figure 2 The specific parameters of the network are shown in Table 3. After being mapped again by the feedforward network, the global features are obtained. Since global features are obtained in vector space, this invention further integrates local features from the image space to compensate for the model's lack of context awareness. (See attached diagram for local feature structure.) Figure 2The specific parameters of its network are shown in Table 3. Specifically, the obtained encoded feature vectors containing global information are input into the channel splitting and adjustment module for feature splitting. This module first splits the input features according to different scales, resulting in four sets of feature vectors: {(H / 8×W / 8)×128, (H / 16×W / 16)×128, (H / 32×W / 32)×128, and (H / 64×W / 64)×128}. These four sets of feature vectors are then remapped back into the image space, resulting in feature map sets {H / 8×W / 8×128, H / 16×W / 16×128, H / 32×W / 32×128, and H / 64×W / 64×128}. These four feature map sets are then input into their respective local feature extractors according to different scales for local feature extraction. The extracted features are then input back into the channel splitting and adjustment module for the reverse operation, resulting in the fusion of the four sets of local features. Finally, the fused local features are... and global features The mixture is fused again to obtain the final output of CaEL1. The fusion method is also carried out in the form shown in Formula 3.

[0085] Table 3

[0086]

[0087]

[0088] At this point, the encoded features obtained from CaEL1 are sequentially input into the subsequent three context-aware encoding modules for further encoding. The specific operation is the same as that of CaEL1, and will not be repeated in this embodiment. The difference is that, for CaEL2, Q2 and V2 in Equation 4 are now F. CaEL1 In CaEL3, Q3 and V3 in Equation 4 are now F. CaEL2 In CaEL4, Q4 and V4 in Equation 4 are now F. CaEL3 .

[0089] Furthermore, the decoding layer is structurally described in [reference needed]. Figure 2 This implementation uses two sets of decoding layers to decode the encoded features, each set consisting of four decoders. The first set (…) Figure 1 Group 1) The decoding layer is mainly used to detect the aircraft target in the current frame, and Group 2 ( Figure 1 Group 2) decoding layer is mainly used to extend the detection of aircraft targets from the previous frame to the current frame. Therefore, for Group 1 decoder, a set of position keys K is input at this time. DL-1 and the corresponding query feature Q DL-1Used to detect aircraft targets in the current frame; for group 2 decoders, the position of the detected target in the previous frame is used as key K. DL-2 and the corresponding feature vector as the query feature Q DL-2 This is used for extending the target range of aircraft. Specifically, each decoder is structurally composed of three parts: a self-attention module, a mutual attention module, and a feedforward network. In this embodiment, the self-attention module is implemented using a multi-head self-attention mechanism, and the mutual attention module is implemented using the same multi-scale deformable attention mechanism as in the encoder. Therefore, this embodiment only explains the multi-head attention mechanism of the self-attention module.

[0090] Multi-head self-attention mechanisms use different "heads" to generate different attention responses to the input query. The output of the multi-head attention mechanism is obtained by fusing the response information from each "head" branch, as follows:

[0091]

[0092] Among them, W Q W K W V Let W be the corresponding attention weight matrix, and τ = {1, 2} and υ = {1, 2, 3, 4} represent the number of decoding layer groups and the decoding layer index, respectively. CAT represents the concatenation operation, and Att represents attention mapping. hi is the index of the number of heads used; this implementation uses 8 heads. For the first decoder in group 1, its... All are query vectors adaptively learned by the model (500 queries q are learned in this implementation), and are simultaneously assigned to both the key and value; hence, this is also called a self-attention mechanism. The purpose is to prevent multiple query vectors from converging to the same target simultaneously, thus eliminating the need for subsequent non-maximum suppression. The decoded features output from the multi-head attention mechanism in group 1 are then used. As Q 1-1 At the same time, the sampling positions learned adaptively are used as Use the output of CaEL4 as The features are further input into the corresponding multi-scale deformable attention network (as in Equation 4) to obtain the decoded features. These features are then input into the corresponding feedforward network at the back end, and finally output as the features decoded by the first decoder DL1-1 in group 1. F in the same manner DL1-1 As MHA in DL1-2 1-2 Required Q 1-2 , will F DL1-2 As MHA in DL1-3 1-3 Required Q 1-3 , will F DL1-3 As MHA in DL1-4 1-4Required Q 1-4 Finally, the detection position information decoded by the Group 1 decoder is obtained. and the corresponding feature vectors

[0093] For the first decoder in group 2, its All are F DL1-4,Dt-1 This refers to the decoding vector corresponding to the detected aircraft target in the previous frame. The decoding features output by the multi-head attention in group 2 are... As Q 2-1 At the same time, the sampling positions learned adaptively are used as Use the output of CaEL4 as The features are further input into the corresponding multi-scale deformable attention network (as in Equation 4) to obtain the decoded features. These features are then input into the corresponding feedforward network at the back end, and finally output as the decoded features of the first decoder DL2-1 in group 2. F in the same manner DL2-1 As MHA in DL2-2 2-2 Required Q 2-2 , will F DL2-2 As MHA in DL2-3 2-3 Required Q 2-3 , will F DL2-3 As MHA in DL2-4 2-4 Required Q 2-4 Finally, the extended position information decoded by the Group 2 decoder is obtained. and the corresponding feature vectors

[0094] Furthermore, the structure of the motion appearance association module is described in [reference needed]. Figure 2 It is constructed by stacking a series of multilayer perceptrons, and the specific parameters of the motion appearance association module are shown in Table 4. Specifically, this implementation model the changes in the bounding box of the aircraft target during its motion to obtain the corresponding motion dimension features; it also models the appearance information corresponding to the center point of the aircraft target to obtain the color dimension features. The motion dimension features and the color dimension features are then concatenated to obtain a feature vector containing both motion and color information. For the input current frame detection position information... and extended information on aircraft targets The input is fed into the motion dimension feature perceptron (Group 1) to obtain the corresponding motion dimension feature vectors. The input is fed into the color dimension feature perceptron (group 2) to obtain the corresponding color dimension feature vectors. The corresponding concatenation yields a feature vector containing both motion and color features. The input is then fed into the third multilayer perceptron to obtain the final encoded feature vector containing motion and color features. At this point, find L Dt With L Et The similarity between the detected aircraft targets and the extended aircraft targets is determined by minimizing the similarity. Simultaneously, the corresponding detected aircraft target B is obtained using the Interchange of Units (IOU). Dt With extended aircraft target B Et The positional similarity between the elements is calculated and added to the total similarity. The specific calculation is as follows:

[0095]

[0096] Where λ is the balance factor, which is set to 0.5 in this implementation, and N is the normalization operation.

[0097] Table 4

[0098]

[0099] Specific Implementation Method Two

[0101] Follow these steps:

[0102] Step 1: Data Preparation Stage

[0103] Due to the limited research on multi-aircraft tracking, most studies currently focus on single-aircraft tracking, resulting in a lack of usable datasets for multi-aircraft tracking. To address this issue, we independently collected and labeled the first dataset for multi-aircraft tracking, naming it HIT-MATD. Its details are as follows:

[0104] 1) This dataset contains 32 video sequences, totaling 16,505 frames, with a resolution of 450×450;

[0105] 2) This dataset contains 44,963 labeled aircraft targets and 144 flight paths;

[0106] 3) This dataset contains common and challenging scenarios during flight, such as severe contrail cloud obstruction, tracking of small aircraft targets, tracking of similar aircraft targets, and foggy scenarios.

[0107] All training and testing data used in this embodiment are from HIT-MATD. Nineteen video sequences were selected for training, and thirteen video sequences were selected for testing. During training and testing, each frame was enlarged to a size of 768×768.

[0108] Step 2, Training Phase

[0109] Repeat steps 21 and 22 for the training process of the training network.

[0110] Step 21: Initialize network training parameters and hyperparameters. In this implementation, the maximum number of iterations (Iteration) is set to 50; the batch size is set to BatchSize = 3; and the learning rate (lr) is set to 2 × 10⁻⁶ to achieve faster convergence during training. -4 The learning rate is then decayed to 2 × 10 at Iteration = 35. -5 The AdamW algorithm was introduced as the optimizer, with a weight decay coefficient L2 = 0.0001 and momentum = 0.9. All network parameters were initialized according to the Xavier method. The loss function chosen was L1 Loss.

[0111] Step 22: Use the image obtained in step S1 as input to train the network and begin training. The training process is iterative. In each iteration, the loss value of the forward propagation is calculated first, and then all parameters of the network are updated in reverse with the goal of minimizing the loss value.

[0112] Step 3, Testing Phase

[0113] Repeat steps 31 and 32 for the testing phase.

[0114] Step 31: Repeat step S1 for the test image;

[0115] Step 32: Input the test image frame obtained in Step 31 into the training network trained in Steps 21 and 22, and output the tracking result of the target in the test image frame.

[0116] The experimental results are shown in Table 5. In the table, R, C, and M represent the addition of the receptive field enhancement module, context-aware coding module, and motion appearance association module proposed in this embodiment of the invention, respectively. TransTrack_MS indicates that multi-scale training was used during training, while all other models did not use multi-scale training. Furthermore, TransTrack_R indicates that the baseline algorithm TransTrack was trained according to the reference... Figure 2 A receptive field enhancement module was added in the form of [module name], and the remaining models follow the same pattern. Meanwhile, the evaluation metrics used are all well-known to those skilled in the art, and will not be elaborated upon in the embodiments of this invention. As shown in Table 5:

[0117] 1. The algorithm (TranTracMAT) proposed in this invention achieved optimal results in 5 out of 8 evaluation metrics, accounting for 62.5%;

[0118] 2. The metrics FP and MOTAL were significantly improved. Quantitatively, TransTrack_MS was improved by 89.7% and 28.7% respectively compared to multi-scale training.

[0119] 3. The metrics Recall and Prcn have been improved. Quantitatively, TransTrack_MS has been improved by 14.2% and 10.9% respectively compared to multi-scale training.

[0120] Meanwhile, the multi-aircraft tracker (TranTracMAT) proposed in this embodiment of the invention also achieves real-time performance (~12 FPS, NVIDIA-A100). Therefore, the multi-aircraft tracker proposed in this embodiment of the invention has greater advantages in both tracking performance and tracking efficiency, providing a new algorithm option for multi-aircraft tracking tasks.

[0121] Table 5

[0122]

[0123] In summary, the multi-vehicle tracking method based on visible light images proposed in this embodiment of the invention, by introducing a receptive field enhancement module, a context-aware coding module, and a motion appearance association module on top of TransTrack, forms a robust multi-vehicle tracking algorithm based on visible light images. This solves the technical problem of the inapplicability of existing multi-target tracking algorithms to multi-vehicle tracking problems, improves the tracking accuracy for multi-vehicle targets, and provides a new algorithm option for multi-vehicle tracking tasks.

[0124] Next, with reference to the accompanying drawings, a multi-aircraft tracking system based on visible light images according to an embodiment of the present invention is described.

[0125] Figure 3 This is a schematic diagram of the structure of a multi-aircraft tracking system based on visible light images according to an embodiment of the present invention.

[0126] like Figure 3 As shown, the system 10 includes: a preprocessing module 101, a feature extraction module 102, a fusion module 103, an encoding module 104, a decoding module 105, and a tracking module 106.

[0127] The system comprises the following modules: a preprocessing module 101 preprocesses the current input image frame; a feature extraction module 102 inputs the preprocessed image frame into an E-ResNet50 feature extraction network for preliminary feature extraction, yielding pre-extracted features for the current frame; a fusion module 103 fuses the pre-extracted features of the current frame with those of the previous frame; an encoding module 104 inputs the fused features into four cascaded context-aware encoding layers to obtain encoded features; a decoding module 105 inputs the encoded features into two sets of four cascaded decoding layers to detect aircraft targets in the current frame and extend them to the previous frame, yielding detected and extended aircraft targets; and a tracking module 106 inputs the detected and extended aircraft targets into a current motion appearance association module for multi-aircraft association and tracking.

[0128] It should be noted that the foregoing explanation of the multi-aircraft tracking method embodiment based on visible light images also applies to the system of this embodiment, and will not be repeated here.

[0129] The multi-vehicle tracking system based on visible light images proposed in this embodiment of the invention, by introducing a receptive field enhancement module, a context-aware coding module, and a motion appearance association module on the basis of TransTrack, forms a robust multi-vehicle tracking algorithm based on visible light images. This solves the technical problem of the inapplicability of existing multi-target tracking algorithms to multi-vehicle tracking problems, improves the tracking accuracy of multi-vehicle targets, and provides a new algorithm option for multi-vehicle tracking tasks.

[0130] To implement the above embodiments, the present invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multi-aircraft tracking method based on visible light images as described in the foregoing embodiments.

[0131] To implement the above embodiments, the present invention also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multi-aircraft tracking method based on visible light images as described in the foregoing embodiments.

[0132] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0133] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0134] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.

[0135] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0136] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0137] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0138] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0139] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A multi-aircraft tracking method based on visible light images, characterized in that, Includes the following steps: Step S1: Preprocess the current input image frame; Step S2: Input the preprocessed image frame into the feature extraction network E-ResNet50 for preliminary feature extraction to obtain the pre-extracted features of the current frame; Step S3: Fuse the pre-extracted features of the current frame with the pre-extracted features of the previous frame; Step S4: Input the fused features into four cascaded context-aware coding layers to obtain the coded features; Step S5: Input the encoded features into two groups of four cascaded decoding layers to detect the aircraft target in the current frame image and the aircraft target in the extended previous frame image, thereby obtaining the detected aircraft target and the extended aircraft target. Step S6: Input the detected aircraft target and the extended aircraft target into the current motion appearance association module to perform multi-aircraft association and complete multi-aircraft tracking; The feature extraction network E-ResNet50 includes a cascaded 7×7 convolutional layer, a max pooling layer, four residual blocks, and a receptive field enhancement block. Each of the four residual blocks consists of a cascaded 1×1, a 3×3, and a 1×1 convolutional layer. The receptive field enhancement block consists of a parallel convolutional layer consisting of two 1×1, two 3×3, and one 7×7 convolutional layers, in order to map the preprocessed image frame into a high-dimensional feature space. Step S6 specifically includes: The motion information and appearance color information of the target bounding boxes corresponding to the detected aircraft target and the extended aircraft target are input into the current motion appearance association module. First, high-dimensional feature mapping is performed to form the features of each aircraft target. Then, the position information B of each aircraft target is combined. Dt B Et The Hungarian matching algorithm is used to associate the detection position and the extended position to achieve multi-aircraft tracking; among them, the motion appearance association module is a module composed of three sets of multi-layer sensing blocks.

2. The multi-vehicle tracking method based on visible light images according to claim 1, characterized in that, Step S4 specifically includes: Step S401: Input the fused features into the first context-aware coding layer CaEL1 to capture features with inter-frame information, and obtain the first coded features. F CaEL1 ; Step S402, the first encoded feature F CaEL1 The input is fed into the second context-aware coding layer CaEL2 to obtain the second encoded features. F CaEL2 ; Step S403, the second encoded feature F CaEL2 The input is fed into the third context-aware coding layer CaEL3 to obtain the third coding feature. F CaEL3 ; Step S404, the third encoding feature F CaEL3 The input is fed into the fourth context-aware coding layer CaEL4 to obtain the fourth coding feature. F CaEL4 .

3. The multi-vehicle tracking method based on visible light images according to claim 2, characterized in that, The first context-aware coding layer CaEL1, the second context-aware coding layer CaEL2, the third context-aware coding layer CaEL3, and the fourth context-aware coding layer CaEL4 are structurally composed of two parts, global blocks and local blocks, according to feature attributes. At the end of each context-aware coding layer, the features obtained from the global block and the local block are fused together again as the output of the current context-aware coding layer.

4. The multi-vehicle tracking method based on visible light images according to claim 1, characterized in that, Step S5 specifically includes: Step S501, the fourth encoded feature F CaEL4 The inputs are respectively fed into the first decoding layer DL1-1 of the first group and the first decoding layer DL2-1 of the second group to obtain the first decoding feature. F DL1-1 Second-to-second decoding features F DL2-1 ; Step S502, the first decoded feature F DL1-1 The input is fed into the second decoding layer DL1-2 of the first group to obtain the first and second decoding features. F DL1-2 At the same time, the second decoding feature F DL2-1 The input is fed into the second decoding layer DL2-2 of the second group to obtain the second binary decoding feature. F DL2-2 ; Step S503, the first decoded feature F DL1-2 The input is fed into the third decoding layer DL1-3 of the first group to obtain the third decoding feature. F DL1-3 At the same time, the second decoding feature F DL2-2 The input is fed into the third decoding layer DL2-3 of the second group to obtain the second and third decoding features. F DL2-3 ; Step S504, the third decoding feature F DL1-3 The input is fed into the fourth decoding layer DL1-4 of the first group to obtain the first four decoding features. F DL1-4,Dt and the first decoding position B Dt At the same time, the second and third decoding features F DL2-3 The input is fed into the fourth decoding layer DL2-4 of the second group to obtain the second fourth decoding feature. F DL2-4,Et and the second decoding position B Et Wherein, the first decoding position B Dt For the detected aircraft target, the second decoding position B Et The extended aircraft target.

5. The multi-vehicle tracking method based on visible light images according to claim 4, characterized in that, The first decoding layer DL1-1, the second decoding layer DL1-2, the third decoding layer DL1-3, and the fourth decoding layer DL1-4 in the first group, and the first decoding layer DL2-1, the second decoding layer DL2-2, the third decoding layer DL2-3, and the fourth decoding layer DL2-4 in the second group, are all structurally composed of three parts: self-attention, mutual attention, and feedforward network.

6. A multi-aircraft tracking system based on visible light images, characterized in that, include: The preprocessing module is used to preprocess the current input image frame; The feature extraction module is used to input the preprocessed image frame into the feature extraction network E-ResNet50 for preliminary feature extraction to obtain the pre-extracted features of the current frame; The fusion module is used to fuse the pre-extracted features of the current frame with the pre-extracted features of the previous frame; The encoding module is used to input the fused features into four cascaded context-aware encoding layers to obtain encoded features; The decoding module is used to input the encoded features into two groups of four cascaded decoding layers to detect the aircraft target in the current frame image and the aircraft target in the extended previous frame image, so as to obtain the detected aircraft target and the extended aircraft target. The tracking module is used to input the detected aircraft target and the extended aircraft target into the current motion appearance association module to perform multi-aircraft association and complete multi-aircraft tracking. The feature extraction network E-ResNet50 includes a cascaded 7×7 convolutional layer, a max pooling layer, four residual blocks, and a receptive field enhancement block. Each of the four residual blocks consists of a cascaded 1×1, a 3×3, and a 1×1 convolutional layer. The receptive field enhancement block consists of a parallel convolutional layer consisting of two 1×1, two 3×3, and one 7×7 convolutional layers, in order to map the preprocessed image frame into a high-dimensional feature space. The specific operations of the tracking module include: The motion information and appearance color information of the target bounding boxes corresponding to the detected aircraft target and the extended aircraft target are input into the current motion appearance association module. First, high-dimensional feature mapping is performed to form the features of each aircraft target. Then, the position information B of each aircraft target is combined. Dt B Et The Hungarian matching algorithm is used to associate the detection position and the extended position to achieve multi-aircraft tracking; among them, the motion appearance association module is a module composed of three sets of multi-layer sensing blocks.

7. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multi-aircraft tracking method based on visible light images as described in any one of claims 1-5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-aircraft tracking method based on visible light images as described in any one of claims 1-5.

Images