Remote small target detection and tracking system based on multi-scale feature fusion and deep learning

Abstract

The invention relates to the technical field of small target detection and tracking, in particular to a remote small target detection and tracking system based on multi-scale feature fusion and deep learning. The method comprises the following steps: a feature extraction unit extracts multi-scale features from an input image frame by using a deep convolutional neural network, and generates a multi-scale feature map; the feature fusion unit fuses the generated multi-scale feature maps through a triangular feature method; the target detection unit uses a convolution adaptation framework to detect and position a target according to the multi-scale feature map; the target tracking unit keeps tracking of the target through a predictive tracking method. According to the method, the deep convolutional neural network is used for extracting the multi-scale features, and the feature fusion technology is combined, so that the feature information of the target under different scales can be effectively captured. Due to the generation of the multi-scale feature map, the system not only can identify the rough contour of the target, but also can capture the fine features of the target, which is especially important for the small target in the distance.

Description

Technical Field

[0001] The present invention relates to the technical field of small target detection and tracking, and in particular to a distant small target detection and tracking system based on multi-scale feature fusion and deep learning. Background Art

[0002] In images taken from a distance, the target may become very small, making it difficult for traditional methods to extract effective features for identification. Due to the small pixel-level information of small targets, traditional feature extraction methods often cannot obtain enough information for reliable detection; single-scale feature representation may not be enough to capture all relevant information of the target, especially when the target size varies greatly. For small targets at a distance, relying only on deep features may ignore some important details; targets may be easily lost due to rapid motion, occlusion or complex background, and traditional tracking algorithms may fail in these cases because feature representation is not robust enough. Therefore, a small target detection and tracking system based on multi-scale feature fusion and deep learning is designed. Summary of the invention

[0003] The purpose of the present invention is to provide a distant small target detection and tracking system based on multi-scale feature fusion and deep learning, so as to solve the problems of small targets being difficult to detect, insufficient feature representation capability and easy target loss raised in the above background technology.

[0004] To achieve the above object, the present invention aims to provide a distant small target detection and tracking system based on multi-scale feature fusion and deep learning, comprising: A feature extraction unit, wherein the feature extraction unit extracts multi-scale features from an input image frame using a deep convolutional neural network and generates a multi-scale feature map; A feature fusion unit, wherein the feature fusion unit fuses the generated multi-scale feature maps through a triangular feature method; An object detection unit, wherein the object detection unit detects and locates objects according to a multi-scale feature map using a convolutional adaptation framework; A target tracking unit is used to track the target using a predictive tracking method.

[0005] As a further improvement of the technical solution, the feature extraction unit uses a deep convolutional neural network to extract multi-scale features from the input image frame, including the following steps: S1.1, preprocessing the input image frame; S1.2, the image frame passes through a series of convolutional layers, which use a set of filters to scan the entire image frame. The filters capture local features in the image frame; S1.3, each convolution operation is followed by a nonlinear activation function; S1.4, using the maximum pooling method in the pooling layer to reduce the size of the feature map while retaining the most important information; S1.5. Generate multi-scale feature maps by adjusting the combination of convolutional layers and pooling layers.

[0006] As a further improvement of the technical solution, in S1.4, the maximum pooling method is specifically: ; in, Indicates the output multi-scale feature map at position The value of Represents the multi-scale feature map of the input at position The value at Represents the position relative to the output multi-scale feature map The horizontal offset in the input multi-scale feature map; Represents the position relative to the output feature map The vertical offset in the input feature map; Indicates the height of the pooling window; Indicates the width of the pooling window.

[0007] As a further improvement of the technical solution, the feature fusion unit fuses the generated multi-scale feature maps through a triangular feature method, including the following steps: S2.1, preprocess the multi-scale feature map, and during the detection process, adaptively adjust the detection threshold according to the changes in different lighting conditions; S2.2, restore the higher-level multi-scale feature map to the lower-level size through transposed convolution, introduce a control gradient coefficient in the transposed convolution operation, consider the problem of incomplete edge information and consistency of feature maps, and use information filling method and region verification criteria to further optimize the transposed convolution operation; S2.3, fusing the transposed convolutional multi-scale feature map with the multi-scale feature map of the lower level by element-by-element addition method; S2.4. Apply the convolution operation again on the fused multi-scale feature map to further optimize the feature representation.

[0008] As a further improvement of the technical solution, in S2.2, the transposed convolution is specifically: ; The control gradient coefficient introduced in the transposed convolution operation is: ; The information filling method and region checking criterion are used to further optimize the transposed convolution operation as follows: ; ; ; in, Indicates the output multi-scale feature map at position The value of A multi-scale feature map representing the input; Indicates that the filter is in position The value of represents the step length; Represents the amount of padding, used to add zero padding around the input multi-scale feature map; Indicates the offset in the x-axis direction; Indicates the offset in the y-axis direction; represents the control gradient coefficient; Indicates that the multi-scale feature map output after the control gradient coefficient is introduced is at position The value of It means that the multi-scale feature map is output at the position after the information filling method and the region verification criterion are used. The value of Indicates channel; Represents the value of a pixel in the feature map after filling; Indicates the information filling method; Indicates The attention weight of each channel; Represents the value of a pixel in the feature map obtained by the region verification criterion; Represents the value of a pixel in the original feature map.

[0009] As a further improvement of the technical solution, the target detection unit uses a convolution adaptation framework to detect and locate the target according to the multi-scale feature map, including the following steps: S3.1. Run the region proposal network on the fused multi-scale feature map to generate candidate regions for each scale, and add a scale adaptation module on the multi-scale feature map. S3.2. For each candidate region, crop the feature maps of all scales, and convert the feature maps of all scales into feature vectors of fixed size through the feature region alignment method; S3.3, the feature vector processed by the feature region alignment method is further processed by a fully connected layer for object classification and bounding box regression, using focal loss to solve the problem of class imbalance; S3.4, output the probability that each candidate region belongs to each category, and fine-tune the bounding box of each candidate region; S3.5. Perform an overlap scoring algorithm on bounding boxes of the same category, remove overlapping bounding boxes, adjust the suppression strength in the overlap scoring according to the importance of the target, introduce smoothing processing in the overlap scoring algorithm, and further optimize the overlap scoring algorithm by considering the importance of different targets and the degree of occlusion of the candidate area when multiple targets coexist; S3.6. Output the final detection result, which includes the category label of each detected target and the location of its bounding box.

[0010] As a further improvement of the technical solution, in S3.2, the feature maps of all scales are converted into feature vectors of fixed size by a feature region alignment method, including the following steps: S3.21, determining the position of each candidate region on the feature map; S3.22, each candidate region is mapped back to a rectangular region on the feature map, and the rectangular region is divided into several small blocks; S3.23, for each pixel point in each sub-area, perform bilinear interpolation to obtain the accurate feature value at that position; S3.24, calculating the average of all pixel values ​​obtained by bilinear interpolation in each sub-region to obtain a feature value of the sub-region, and applying an additional convolution layer based on the feature value of the sub-region; S3.25. Standardize the eigenvalues ​​of each sub-region and combine the eigenvalues ​​of all sub-regions to form a eigenvector of a fixed size.

[0011] As a further improvement of the technical solution, in S3.5, the overlap scoring algorithm is specifically: ; Introducing smoothing into the overlap scoring algorithm is: ; Considering the importance of different targets and the degree of occlusion of candidate regions when multiple targets coexist, the overlap score algorithm is further optimized as follows: ; in, Indicates The raw scores of the bounding boxes; It indicates the updated score after correcting the original score; Indicates The updated score after smoothing is introduced into the bounding box; represents the initial attenuation factor; represents the final attenuation factor; Represents the parameter that controls the smoothness of the attenuation factor; Represents a bounding box and The intersection-and-union ratio between them; represents the score of the previous bounding box; Indicates the updated score after considering the importance of different targets and the degree of occlusion of the candidate area when multiple targets coexist; Indicates the relative importance factor between objectives; Represents the occlusion score compensation factor.

[0012] As a further improvement of the technical solution, the target tracking unit keeps tracking the target through a predictive tracking method, including the following steps: S4.1. For each candidate region, extract features from the multi-scale feature map of the current image frame and compare them with the feature representation of the target in the initialization image frame. Use the Euclidean distance to evaluate the matching degree between the candidate region and the target feature. S4.2. Calculate the matching score for each candidate area. The higher the score, the better the match with the target. Select the candidate area with the highest score as the new position of the target in the current image frame, update the position information of the target, and use this position information to determine the search area of ​​the next frame. Introduce an occlusion detection mechanism. When it is detected that the target is occluded, temporarily relax the threshold k of the matching score, and record the distribution of the matching score over a period of time. Dynamically adjust the score threshold k of the current image frame according to the distribution of historical scores. S4.3, using the predictive tracking method to predict the position and velocity of the target in the current image frame, and using the best matching result obtained from the feature matching to update the state estimate of the predictive tracking method; S4.4, if no candidate region with sufficient matching degree is found in the current image frame, it is considered that the target may be temporarily lost, and the information of the historical frame is used to relocate the target; S4.5. When the target no longer needs to be tracked, stop the tracking process.

[0013] As a further improvement of the technical solution, in S4.3, the prediction and tracking method is specifically: Status prediction: ; Status Update: ; in, Indicates the time step The predicted state vector of represents the state transfer matrix; Indicates the time step state estimation; represents the control input matrix; Indicates the time step The control input vector of Indicates the time step The updated state estimate of Indicates the time step The Kalman gain matrix of Indicates the time step The observation vector of represents the observation matrix.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. In this small target detection and tracking system based on multi-scale feature fusion and deep learning, by using deep convolutional neural networks to extract multi-scale features and combining feature fusion technology, it can effectively capture the feature information of the target at different scales. The generation of multi-scale feature maps allows the system to not only identify the general outline of the target, but also capture the subtle features of the target, which is especially important for small targets at a distance. In addition, the triangular feature method integrates shallow detail information and deep semantic information, enhancing the robustness of the model for target detection under different environments and lighting conditions, thereby improving the accuracy and reliability of detection.

[0015] 2. This small target detection and tracking system based on multi-scale feature fusion and deep learning uses a predictive tracking method to maintain stable tracking of the target between consecutive frames. By comparing the target features in the current frame and the initialization frame, the system can accurately locate the target position, and maintain the continuity of tracking by dynamically adjusting the matching score threshold when the target is occluded or partially occluded. The predictive tracking method combines the feature matching results to update the state estimate, so that the target can be effectively tracked even when the target moves quickly or there is complex background interference, greatly improving the tracking performance and practicality of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is the overall flow chart of the present invention; The meaning of each number in the figure is: 1. Feature extraction unit; 2. Feature fusion unit; 3. Target detection unit; 4. Target tracking unit. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Example: See Figure 1 As shown in the figure, a distant small target detection and tracking system based on multi-scale feature fusion and deep learning is provided, including: The feature extraction unit 1 uses a deep convolutional neural network to extract multi-scale features from the input image frame and generate a multi-scale feature map; In this embodiment, the multi-scale features include deep-level features (high-level semantic information and position information such as the overall shape and category of the target) and shallow-level features (local features such as the edge and texture of the target). In a deep convolutional neural network (CNN), each layer of convolution operation generates a new set of feature maps, which usually have different resolutions as the depth of the network increases. In CNN, image information is captured at different scales by using convolution kernels of different sizes or adjusting the step size, adding pooling layers, etc.

[0019] Among them, the design of the convolution layer enables the network to focus on the local area of ​​the input data, thereby capturing the local features in the image (such as edges, textures). At the same time, through the operation of the pooling layer, the deep convolutional neural network can also achieve a certain degree of invariance to the position, that is, within a certain range, the position change of the target will not significantly affect its feature representation; as the depth of the network increases, each layer will learn higher-level abstract features. The shallow layer may learn simple texture and edge information, while the deep layer can learn more complex shapes and object-level features. This hierarchical feature representation helps improve the model's ability to understand complex patterns; the filters (or kernels) in the convolutional layer are reused on the entire input image, which means that the same parameters can be used to detect the same features at any position in the image. This parameter sharing mechanism not only reduces the number of parameters that need to be learned, but also improves the generalization ability of the model; using a deep convolutional neural network to extract multi-scale features from the input image frame includes the following steps: S1.1, preprocessing the input image frame, including operations such as scaling, normalization, and noise reduction; S1.2, the image frame passes through a series of convolutional layers, which use a set of filters (also called kernels) to scan the entire image frame. The filters capture local features in the image frame. These filters can capture edges, textures, and other low-level features; S1.3. Each convolution operation is followed by a nonlinear activation function (such as ReLU) so that the model can learn nonlinear mapping relationships. S1.4. Use the maximum pooling method in the pooling layer to reduce the size of the feature map while retaining the most important information, which helps reduce the amount of computation and increase robustness to input changes; Furthermore, the maximum pooling operation reduces the size of the feature map by applying a smaller window (such as 2x2) to the feature map and selecting the maximum value in each window as the output, which directly reduces the amount of data that needs to be processed by the subsequent network layer, thereby reducing the computational cost and memory usage; the maximum pooling operation achieves invariance to the translation of the input image to some extent, because even if some features in the input feature map are displaced in a small range, the maximum pooling can still capture the main information of these features, which enables the model to maintain stable performance when facing small changes in the input image; the maximum pooling selects the maximum value in the local area, which actually emphasizes the most important or most representative feature information in the area, while ignoring some less important details. This helps the model focus on the most critical parts, thereby improving the accuracy of recognition and classification; the specific maximum pooling method is: ; in, Indicates the output multi-scale feature map at position The value of Represents the multi-scale feature map of the input at position The value at Represents the position relative to the output multi-scale feature map The horizontal offset in the input multi-scale feature map; Represents the position relative to the output feature map The vertical offset in the input feature map; Indicates the height of the pooling window; Represents the width of the pooling window, that is, the size of the local area considered when performing the maximum pooling operation; S1.5. Generate multi-scale feature maps by adjusting the combination of convolutional layers and pooling layers, and use convolutional layers with smaller steps or pooling layers with smaller windows to obtain larger feature maps.

[0020] The feature fusion unit 2 fuses the generated multi-scale feature maps through the triangular feature method, so that the model can use shallow semantic information and deep position information at the same time; In this embodiment, the triangular feature fusion method can effectively combine the advantages of feature maps of different scales, so that the fused feature map can not only retain the detail information in the shallow features, but also utilize the semantic information provided by the deep features. In this way, the model can obtain a more comprehensive and rich feature representation, thereby improving the accuracy of detection and tracking; by fusing features at different levels, the model can better cope with the challenges brought by complex backgrounds and lighting changes, and the fusion of multi-scale features helps the model maintain a high detection rate and tracking stability under different conditions; for small targets in the distance, traditional single-scale features may be difficult to provide sufficient information for reliable detection, and multi-scale feature fusion can make up for this deficiency. By combining multiple layers of feature information, the sensitivity to small targets is enhanced, thereby improving the detection accuracy of small targets; fusing the generated multi-scale feature maps through the triangular feature method includes the following steps: S2.1. Preprocess the multi-scale feature map, such as using 1×1 convolution to adjust the dimension of the feature map to keep it consistent. During the detection process, the detection threshold is adaptively adjusted according to the changes in the target size under different lighting conditions or different scenes to improve the robustness and accuracy of the detection. S2.2, restore the higher-level multi-scale feature map to the lower-level size through transposed convolution, introduce a control gradient coefficient in the transposed convolution operation, consider the problem of incomplete edge information and consistency of feature maps, and use information filling method and region verification criteria to further optimize the transposed convolution operation; Furthermore, the spatial resolution of higher-level feature maps is usually low due to multiple downsampling operations (such as pooling, convolution with a step size greater than 1, etc.). The spatial resolution of these features can be restored through transposed convolution, thereby retaining more position information; transposed convolution can help combine high-level feature maps with low-level feature maps, so that the model can simultaneously utilize high-level semantic information and low-level detail information. This information fusion can improve the recognition accuracy and robustness of the model; transposed convolution can be used to generate feature maps of different scales, which helps the model to be more flexible in dealing with targets of different sizes, especially in object detection tasks, where features of different scales are very useful for detecting objects of different sizes; transposed convolution is specifically: ; By introducing the control gradient coefficient, the information transfer from the higher-level feature map to the lower-level feature map can be better controlled. The purpose of this is to ensure that in the process of feature fusion, important information can be fully retained and utilized, while non-critical information can be appropriately suppressed or filtered; when processing small targets at a distance, there may be a variety of interference factors in the image (such as complex background, illumination changes, etc.). By controlling the gradient coefficient, the model's resistance to these interference factors can be enhanced, thereby improving the robustness of detection and tracking; by optimizing the feature fusion method, the model's ability to represent features can be enhanced, so that the model can more accurately identify and track small targets at a distance, especially when the target size changes greatly; the control gradient coefficient introduced in the transposed convolution operation is: ; Through the information filling method, the missing information in the edge area can be supplemented, so as to better retain the edge details, which is particularly important for detecting small targets in the distance, because the edge information of these targets is often the key to distinguishing them from the background. The complete processing of edge information can improve the robustness of the model in the face of complex situations such as occlusion and illumination changes, making the model more stable during detection and tracking. The information filling method helps to provide more detailed feature information near the target boundary, thereby improving the accuracy of target detection, especially when locating small targets in the distance; when performing transposed convolution, if it is not controlled, it may cause information distortion in the feature map. The use of regional kernel alignment can ensure the consistency of information between feature maps during upsampling, avoiding detection errors caused by information loss or distortion. Through the regional verification criterion, the correspondence between feature maps of different scales can be adjusted more accurately, thereby optimizing the effect of feature fusion, which helps the model to better utilize multi-scale features and improve the accuracy of detection and tracking; the operation of transposed convolution is further optimized by the information filling method and the regional verification criterion as follows: ; ; ; ; in, Indicates the output multi-scale feature map at position The value of Represents the multi-scale feature map of the input, that is, the multi-scale feature map of the higher level; Indicates that the filter is in position The value of It represents the step size, which determines the step size of the filter sliding on the input multi-scale feature map; Represents the amount of padding, which is used to add zero padding around the input multi-scale feature map to keep the output size consistent; Indicates the offset in the x-axis direction; Indicates the offset in the y-axis direction; represents the control gradient coefficient; Indicates that the multi-scale feature map output after the control gradient coefficient is introduced is at position The value of It means that the multi-scale feature map is output at the position after the information filling method and the region verification criterion are used. The value of Indicates channel; Represents the value of a pixel in the feature map after padding. The padding is to maintain the integrity of the information at the boundary. Represents the information filling method, which is used to process edge information and ensure that boundary information is not lost; Indicates The attention weight of each channel; Represents the value of a pixel in the feature map obtained by the region verification criterion; Represents the value of a pixel in the original feature map; represents the activation function; Represents the weight matrix of the first layer of the fully connected network; Represents the weight matrix of the second layer fully connected network; The bias vector of the first fully connected network layer; The bias vector of the second fully connected network layer; Indicates Global average pooling result of channels; S2.3, the multi-scale feature map after the transposed convolution is fused with the multi-scale feature map of the lower level by element-by-element addition method, so that the feature map of the higher level contains the detail information from the bottom layer and the semantic information of the high level; S2.4. Apply the convolution operation again on the fused multi-scale feature map to further optimize the feature representation.

[0021] The target detection unit 3 detects and locates the target based on the multi-scale feature map using a convolutional adaptation framework; In this embodiment, the convolution adaptation framework can better handle target detection in complex backgrounds. Through the comprehensive judgment of multi-scale feature maps, the impact of background interference on the detection results can be reduced. It allows parallel processing of candidate areas on multi-scale feature maps, which helps to speed up the detection speed; by adding a scale adaptation module to the multi-scale feature map, the system can dynamically adjust the scale of the feature map according to the actual size of the target. This means that no matter how the size of the target changes, the system can better match the characteristics of the target, thereby improving the detection accuracy; using the convolution adaptation framework to detect and locate the target according to the multi-scale feature map, includes the following steps: S3.1. Run the region proposal network (RPN) on the fused multi-scale feature map to generate candidate regions for each scale. The RPN generates anchors of different sizes for feature maps of different scales, so that objects of different sizes can be matched. A scale adaptation module is added to the multi-scale feature map, which can dynamically adjust the scale of the feature map according to the actual size of the object, so as to better match objects of different sizes. S3.2. For each candidate region, crop the feature maps of all scales, and convert the feature maps of all scales into feature vectors of fixed size through the feature region alignment method; Among them, the feature maps of all scales are converted into feature vectors of fixed size through the feature region alignment method, including the following steps: S3.21, determine the position of each candidate region on the feature map, RPN generates multiple anchor points for each position, and adjusts the positions of these anchor points according to the offsets obtained by optimizing the loss function to form a preliminary candidate region; S3.22. Each candidate region is mapped back to a rectangular region on the feature map, and this rectangular region is divided into several small blocks. The number of these small blocks is usually fixed (such as 7x7), which depends on the input size required by the subsequent network layer. Each small block corresponds to a sub-region on the feature map; S3.23, for each pixel in each sub-region, bilinear interpolation is performed to obtain the accurate feature value at the position, where the bilinear interpolation utilizes the values ​​of the four surrounding pixels and performs a weighted average based on the distances between these pixels and the desired position; S3.24, calculating the average of all pixel values ​​obtained by bilinear interpolation in each sub-region to obtain the feature value of the sub-region, and applying additional convolutional layers based on the feature value of the sub-region to further refine and enhance the feature representation; The basic idea of ​​bilinear interpolation is to estimate the pixel value at any position within a two-dimensional grid by using the values ​​of four known neighboring pixels. S3.25, normalize the feature values ​​of each sub-region to eliminate the scale differences that may exist between different feature maps, and combine the feature values ​​of all sub-regions to form a feature vector of fixed size; S3.3, the feature vector processed by the feature region alignment method is further processed by the fully connected layer for target classification and bounding box regression. The focal loss is used to solve the problem of class imbalance, which can improve the detection accuracy of small targets; Among them, in the target detection task, especially for the detection of small targets in the distance, the background class (i.e., the area without the target) is far more than the foreground class (i.e., the area with the target). This class imbalance will cause the model to be overly biased towards the background class during the training process, resulting in poor detection of small targets. The focal loss effectively solves the class imbalance problem by introducing weighting factors and focusing factors, and improves the performance of the model in processing the task of detecting small targets in the distance. S3.4. Output the probability that each candidate region belongs to each category, and fine-tune the bounding box of each candidate region to more accurately frame the target, where the category refers to the category label to which the detected target belongs; S3.5. Perform an overlap scoring algorithm on bounding boxes of the same category, remove overlapping bounding boxes, retain only the most likely detection results, adjust the suppression strength in the overlap scoring according to the importance of the target (for example, based on the size, position or category of the target), introduce smoothing processing in the overlap scoring algorithm, and further optimize the overlap scoring algorithm by considering the importance of different targets and the degree of occlusion of the candidate area when multiple targets coexist; Furthermore, during the object detection process, especially when using sliding windows or candidate regions (such as candidate boxes generated by RPN), multiple overlapping bounding boxes may be generated. The overlap score algorithm can help us select the most appropriate one from these overlapping boxes; ensure that each detected object is marked only once and represented by only one bounding box, so as to avoid the situation where the same object is marked multiple times; after removing the overlapping boxes through the overlap score algorithm, it can ensure that the remaining bounding boxes have a higher confidence score and their positions are more accurate; because the overlap score algorithm will suppress some overlapping prediction boxes, it can reduce the occurrence of false positives, that is, reduce the situation where the background is mistaken for the target; the overlap score algorithm is specifically: ; By introducing smoothing, the final bounding box can be selected more accurately, avoiding inaccurate bounding box selection caused by hard thresholds; smoothing can reduce bounding box selection errors caused by factors such as target size changes or occlusion, making the system more robust in complex environments; through smoothing, the system can better handle the overlap between bounding boxes, thereby improving the accuracy of the detection results, especially when dealing with small targets in the distance. This accuracy improvement is particularly important; introducing smoothing in the overlap scoring algorithm is: ; In the case of multiple targets coexisting, it is necessary to distinguish the importance of different targets and correctly handle the overlapping relationship between them to ensure that each target can be accurately detected and located. By considering the importance of the target, those falsely detected bounding boxes can be better excluded, thereby reducing false positives. In different application scenarios, the importance of the target may be different. By optimizing the algorithm, these changes can be better adapted to improve the generalization ability of the system; in the presence of occlusion, the system needs to be able to correctly handle the occluded area to avoid mistaking the occluded part as part of the target, thereby improving the robustness of detection. By considering the degree of occlusion, the target caused by occlusion can be avoided from being mistaken for the background, thereby reducing false negatives. By considering the degree of occlusion, the target bounding box can be more accurately located, especially in the case of partial occlusion. By introducing the occlusion score compensation factor, the influence of the occluded area can be compensated, making the system more robust when dealing with occlusion; by considering the importance and occlusion degree of different targets, the bounding box that is finally retained can be selected more reasonably to reduce the possibility of false detection and missed detection; in the case of multiple targets coexisting, the importance of different targets and the degree of occlusion of the candidate area, the overlap score algorithm is further optimized as follows: ; ; ; in, Indicates The raw scores of the bounding boxes; It indicates the updated score after correcting the original score; Indicates The updated score after smoothing is introduced into the bounding box; represents the initial attenuation factor; represents the final attenuation factor; Represents the parameter that controls the smoothness of the attenuation factor; Represents a bounding box and The intersection-and-union ratio between them; Represents the score of the previous bounding box, which is used to calculate the adaptive score update; Indicates the updated score after considering the importance of different targets and the degree of occlusion of the candidate area when multiple targets coexist; Indicates the relative importance factor between goals, used to distinguish the importance of different goals; Represents the occlusion score compensation factor, which is used to compensate for the impact of the occluded area; S3.6. Output the final detection result, which includes the category label of each detected target and the location of its bounding box.

[0022] The target tracking unit 4 keeps tracking the target by a predictive tracking method; In this embodiment, the target is tracked by a prediction tracking method, including the following steps: S4.1. For each candidate region, extract features from the multi-scale feature map of the current image frame and compare them with the feature representation of the target in the initialization image frame. Use the Euclidean distance to evaluate the matching degree between the candidate region and the target feature. Among them, the basic principle of Euclidean distance is to measure the matching degree between two feature vectors by calculating the distance between them; S4.2. Calculate the matching score for each candidate region. The higher the score, the closer it matches the target. Select the candidate region with the highest score as the new position of the target in the current image frame. Update the target's position information (such as coordinates, size, etc.). Use this position information to determine the search area for the next frame. Introduce an occlusion detection mechanism. When it is detected that the target is occluded, temporarily relax the threshold k of the matching score. Record the distribution of the matching score over a period of time. Dynamically adjust the score threshold k of the current image frame based on the distribution of historical scores. S4.3, using the predictive tracking method to predict the position and velocity of the target in the current image frame, and using the best matching result obtained from the feature matching to update the state estimate of the predictive tracking method; Among them, the predictive tracking method can predict the position and speed of the target in the next frame based on the information of the current frame. Even if the target is temporarily occluded or moves quickly, the target can be tracked by prediction to reduce the probability of target loss. By predicting the future position of the target, a smoother trajectory can be generated to avoid tracking interruptions caused by sudden movement. When the target is partially or completely occluded, the predictive tracking method can continue to track the target based on historical information and prediction models, thereby improving the robustness in the case of occlusion. For fast-moving targets, traditional tracking methods based on appearance features may be difficult to cope with. The predictive tracking method can better adapt to fast-moving situations by predicting the future position of the target. The specific predictive tracking method is as follows: Status prediction: ; Status Update: ; in, Indicates the time step The predicted state vector of represents the state transfer matrix; Indicates the time step state estimation; represents the control input matrix; Indicates the time step The control input vector of Indicates the time step The updated state estimate of Indicates the time step The Kalman gain matrix, which determines the influence of the observations on the state estimate; Indicates the time step The observation vector includes the target position information extracted from the image; Represents the observation matrix, which maps the state space to the observation space, that is, describes how to extract observation values ​​from the state vector; S4.4, if no candidate region with sufficient matching degree is found in the current image frame, it is considered that the target may be temporarily lost, and the target is relocated using the information of the historical frame or the pre-set search area; S4.5. When the target no longer needs to be tracked, stop the tracking process.

[0023] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only preferred examples of the present invention, and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A distant small target detection and tracking system based on multi-scale feature fusion and deep learning, characterized by: include: A feature extraction unit (1), wherein the feature extraction unit (1) uses a deep convolutional neural network to extract multi-scale features from an input image frame and generate a multi-scale feature map; A feature fusion unit (2), wherein the feature fusion unit (2) fuses the generated multi-scale feature maps using a triangular feature method; A target detection unit (3), wherein the target detection unit (3) detects and locates the target based on the multi-scale feature map using a convolution adaptation framework; A target tracking unit (4), wherein the target tracking unit (4) keeps tracking the target by using a predictive tracking method.

2. The distant small target detection and tracking system based on multi-scale feature fusion and deep learning according to claim 1, characterized in that: The feature extraction unit (1) uses a deep convolutional neural network to extract multi-scale features from an input image frame, comprising the following steps: S1.1, preprocessing the input image frame; S1.2, the image frame passes through a series of convolutional layers, which use a set of filters to scan the entire image frame. The filters capture local features in the image frame; S1.3, each convolution operation is followed by a nonlinear activation function; S1.4, using the maximum pooling method in the pooling layer to reduce the size of the feature map while retaining the most important information; S1.

5. Generate multi-scale feature maps by adjusting the combination of convolutional layers and pooling layers.

3. The distant small target detection and tracking system based on multi-scale feature fusion and deep learning according to claim 2, characterized in that: In S1.4, the maximum pooling method is specifically: ； in, Indicates the output multi-scale feature map at position The value of Represents the multi-scale feature map of the input at position The value at Represents the position relative to the output multi-scale feature map The horizontal offset in the input multi-scale feature map; Represents the position relative to the output feature map The vertical offset in the input feature map; Indicates the height of the pooling window; Indicates the width of the pooling window.

4. The distant small target detection and tracking system based on multi-scale feature fusion and deep learning according to claim 1, characterized in that: The feature fusion unit (2) fuses the generated multi-scale feature maps using a triangular feature method, comprising the following steps: S2.1, preprocess the multi-scale feature map, and during the detection process, adaptively adjust the detection threshold according to the changes in different lighting conditions; S2.2, restore the higher-level multi-scale feature map to the lower-level size through transposed convolution, introduce a control gradient coefficient in the transposed convolution operation, consider the problem of incomplete edge information and consistency of feature maps, and use information filling method and region verification criteria to further optimize the transposed convolution operation; S2.3, fusing the transposed convolutional multi-scale feature map with the multi-scale feature map of the lower level by element-by-element addition method; S2.

4. Apply the convolution operation again on the fused multi-scale feature map to further optimize the feature representation.

5. The distant small target detection and tracking system based on multi-scale feature fusion and deep learning according to claim 4, characterized in that: In S2.2, the transposed convolution is specifically: ； The control gradient coefficient introduced in the transposed convolution operation is: ； The information filling method and region checking criterion are used to further optimize the transposed convolution operation as follows: ； ； ； in, Indicates the output multi-scale feature map at position The value of A multi-scale feature map representing the input; Indicates that the filter is in position The value of represents the step length; Represents the amount of padding, used to add zero padding around the input multi-scale feature map; Indicates the offset in the x-axis direction; Indicates the offset in the y-axis direction; represents the control gradient coefficient; Indicates that the multi-scale feature map output after the control gradient coefficient is introduced is at position The value of It means that the multi-scale feature map is output at the position after the information filling method and the region verification criterion are used. The value of Indicates channel; Represents the value of a pixel in the feature map after filling; Indicates the information filling method; Indicates The attention weight of each channel; Represents the value of a pixel in the feature map obtained by the region verification criterion; Represents the value of a pixel in the original feature map.

6. The distant small target detection and tracking system based on multi-scale feature fusion and deep learning according to claim 1, characterized in that: The target detection unit (3) detects and locates the target according to the multi-scale feature map using a convolution adaptation framework, including the following steps: S3.

1. Run the region proposal network on the fused multi-scale feature map to generate candidate regions for each scale, and add a scale adaptation module on the multi-scale feature map. S3.

2. For each candidate region, crop the feature maps of all scales, and convert the feature maps of all scales into feature vectors of fixed size through the feature region alignment method; S3.3, the feature vector processed by the feature region alignment method is further processed by a fully connected layer for object classification and bounding box regression, using focal loss to solve the problem of class imbalance; S3.4, output the probability that each candidate region belongs to each category, and fine-tune the bounding box of each candidate region; S3.

5. Perform an overlap scoring algorithm on bounding boxes of the same category, remove overlapping bounding boxes, adjust the suppression strength in the overlap scoring according to the importance of the target, introduce smoothing processing in the overlap scoring algorithm, and further optimize the overlap scoring algorithm by considering the importance of different targets and the degree of occlusion of the candidate area when multiple targets coexist; S3.

6. Output the final detection result, which includes the category label of each detected target and the location of its bounding box.

7. The distant small target detection and tracking system based on multi-scale feature fusion and deep learning according to claim 6, characterized in that: In S3.2, the feature maps of all scales are converted into feature vectors of fixed size by a feature region alignment method, including the following steps: S3.21, determining the position of each candidate region on the feature map; S3.22, each candidate region is mapped back to a rectangular region on the feature map, and the rectangular region is divided into several small blocks; S3.23, for each pixel point in each sub-area, perform bilinear interpolation to obtain the accurate feature value at that position; S3.24, calculating the average of all pixel values ​​obtained by bilinear interpolation in each sub-region to obtain a feature value of the sub-region, and applying an additional convolution layer based on the feature value of the sub-region; S3.

25. Standardize the eigenvalues ​​of each sub-region and combine the eigenvalues ​​of all sub-regions to form a eigenvector of a fixed size.

8. The distant small target detection and tracking system based on multi-scale feature fusion and deep learning according to claim 6, characterized in that: In S3.5, the overlap scoring algorithm is specifically: ； Introducing smoothing into the overlap scoring algorithm is: ； Considering the importance of different targets and the degree of occlusion of candidate regions when multiple targets coexist, the overlap score algorithm is further optimized as follows: ； in, Indicates The raw scores of the bounding boxes; It indicates the updated score after correcting the original score; Indicates The updated score after smoothing is introduced into the bounding box; represents the initial attenuation factor; represents the final attenuation factor; Represents the parameter that controls the smoothness of the attenuation factor; Represents a bounding box and The intersection-and-union ratio between them; represents the score of the previous bounding box; Indicates the updated score after considering the importance of different targets and the degree of occlusion of the candidate area when multiple targets coexist; Indicates the relative importance factor between objectives; Represents the occlusion score compensation factor.

9. The distant small target detection and tracking system based on multi-scale feature fusion and deep learning according to claim 1, characterized in that: The target tracking unit (4) keeps tracking the target by using a predictive tracking method, comprising the following steps: S4.

1. For each candidate region, extract features from the multi-scale feature map of the current image frame and compare them with the feature representation of the target in the initialization image frame. Use the Euclidean distance to evaluate the matching degree between the candidate region and the target feature. S4.

2. Calculate the matching score for each candidate area. The higher the score, the closer it matches the target. Select the candidate area with the highest score as the new position of the target in the current image frame, update the position information of the target, and use this position information to determine the search area of ​​the next frame. Introduce an occlusion detection mechanism. When it is detected that the target is occluded, temporarily relax the threshold k of the matching score, and record the distribution of the matching score over a period of time. Dynamically adjust the score threshold k of the current image frame according to the distribution of historical scores. S4.3, using the predictive tracking method to predict the position and velocity of the target in the current image frame, and using the best matching result obtained from the feature matching to update the state estimate of the predictive tracking method; S4.4, if no candidate region with sufficient matching degree is found in the current image frame, it is considered that the target may be temporarily lost, and the information of the historical frame is used to relocate the target; S4.

5. When the target no longer needs to be tracked, stop the tracking process.

10. The distant small target detection and tracking system based on multi-scale feature fusion and deep learning according to claim 9, characterized in that: In S4.3, the prediction and tracking method is specifically: Status prediction: ； Status Update: ； in, Indicates the time step The predicted state vector of represents the state transfer matrix; Indicates the time step state estimation; represents the control input matrix; Indicates the time step The control input vector of Indicates the time step The updated state estimate of Indicates the time step The Kalman gain matrix of Indicates the time step The observation vector of represents the observation matrix.

Images