Infrared weak and small target real-time detection method and device

Abstract

The application provides an infrared weak and small target real-time detection method and device, wherein the method comprises the following steps: collecting a sequence of infrared images of an infrared weak and small target, inputting the sequence of infrared images into a preset lightweight perception detection network frame by frame, and generating a target detection candidate area; acquiring a detection result of a front specific frame image in the sequence of infrared images generated by the preset lightweight perception detection network, inputting the detection result into a preset time domain trajectory prediction module, and acquiring a target prediction position after the time domain trajectory prediction module performs correlation candidate target motion trajectory, trajectory screening and trajectory prediction operations on a time domain input data set; and after the target prediction position is associated and matched with the target detection candidate area, a real-time detection result of the infrared weak and small target is acquired. The problem that lightweight and precision are difficult to maintain a balance in infrared weak and small target detection is solved.

Description

TECHNICAL FIELD

[0001] The present document relates to the field of real-time detection of infrared dim small targets, and particularly relates to a real-time detection method and device for infrared dim small targets. BACKGROUND

[0002] Low-altitude, slow-speed, and small targets such as unmanned aerial vehicles and new guided ammunition have shown great performance in combat. An infrared detection and early warning system is the first choice for dealing with the above-mentioned low, slow, and small targets. However, in a battlefield environment, the above-mentioned targets have the characteristics of low signal-to-noise ratio (small imaging size, low brightness, and large noise), small amount of information (features such as texture and shape disappear), and complex and changeable background. How to quickly and effectively detect these infrared dim small targets and then take effective countermeasures has been a difficulty and hotspot in the fields of detection and early warning, infrared guidance, and homeland security.

[0003] Traditional infrared dim small target detection methods mainly include three categories of filtering, human visual system (HVS), and low-rank sparse recovery. These three types of algorithms mainly detect and analyze the characteristics of the scene to artificially design target features for detection. They are suitable for specific scenes, but have poor robustness in complex scenes. In recent years, with the rapid development of deep learning methods, target detection technology based on convolutional neural networks (CNN) has shown strong feature extraction and generalization capabilities. Two-stage target detection algorithms based on candidate regions such as RCNN and Faster-RCNN and single-stage target detection algorithms based on regression such as SSD and YOLO have emerged. These algorithms have achieved good detection results on various visible light target datasets and have been widely used in various target detection tasks.

[0004] In view of the excellent performance of deep learning based target detection algorithm in visible light target detection, people began to introduce it into infrared dim small target detection. Li et al. introduced attention mechanism in the channel and spatial dimension of each residual block based on YOLOv3 to realize infrared dim small target detection. Zhang et al. proposed a low-altitude infrared dim small target detection algorithm based on fully convolutional regression network, which can accurately detect small targets in complex motion background. Xi et al. proposed a sparse skip connection block to build the backbone of the detection model, enhance the response of infrared dim small target and suppress the background response. Wang et al. proposed a coarse-to-fine internal attention perception network for infrared small target detection, which can accurately detect infrared dim small target and suppress various false alarms. Li Haijun et al. improved the detection ability of infrared dim small ship target by introducing coordinate attention mechanism and adaptive spatial feature fusion measurement. Zhao Xiaofeng et al. combined global perception mechanism based on YOLOv3 to realize accurate detection of ground infrared target. However, the above researches mainly focus on how to improve the detection accuracy of infrared dim small target, ignoring the problems of network parameter redundancy, slow detection speed and so on.

[0005] In actual military application, while considering the detection accuracy, the constraints of embedded platform device deployment also need to be considered, and the requirement of system response speed also needs to be met. Therefore, it is particularly important to maintain high precision of infrared dim small target detection while realizing model lightweight and real-time detection. YOLO-tiny series, MobileDets series and the like are lightweight single-stage target detection models, which have the advantages of small parameter quantity and fast speed, but the model has insufficient deep semantic information extraction and the detection accuracy is not ideal. SUMMARY

[0006] The present application provides an infrared dim small target real-time detection method and device, which introduces linear operation, changes the conventional convolution method, greatly reduces the network parameter quantity, designs a bidirectional feature fusion module based on context perception, enriches the global feature information, and improves the model feature extraction capability. Further, the target time domain motion feature is utilized to realize target position prediction and false alarm filtering, effectively solving the problem that lightweight and precision are difficult to maintain balance.

[0007] The present application provides an infrared dim small target real-time detection method, characterized in that it comprises:

[0008] S1, sequentially inputting infrared images of an infrared dim small target into a preset lightweight perception detection network frame by frame, extracting target spatial features through the lightweight perception detection network, and generating a target detection candidate area;

[0009] S2, obtaining a detection result generated by a preset lightweight perception detection network from a previous specific frame image in a sequence of infrared images, and recording it as a time domain input data set;

[0010] S3, input the time domain input data set into a preset time domain trajectory prediction module, and the time domain trajectory prediction module obtains a target prediction position after performing an associated candidate target motion trajectory, trajectory screening and trajectory prediction operation on the time domain input data set;

[0011] S4, after associating and matching the target prediction position with the target detection candidate area, a real-time detection result of the infrared dim small target is obtained.

[0012] The application provides an infrared dim small target real-time detection device, comprising:

[0013] A perception detection network module is configured to input a sequence of infrared images frame by frame into a preset lightweight perception detection network, extract target spatial features through the lightweight perception detection network, and generate a target detection candidate area;

[0014] A time domain input data set generation module is configured to obtain a detection result of a front specific frame image in a sequence of infrared images generated through a preset lightweight perception detection network, and mark the detection result as a time domain input data set;

[0015] A time domain trajectory prediction module is configured to input the time domain input data set into a preset time domain trajectory prediction module, and the time domain trajectory prediction module obtains a target prediction position after performing an associated candidate target motion trajectory, trajectory screening and trajectory prediction operation on the time domain input data set;

[0016] A target matching module is configured to obtain a real-time detection result after associating and matching the target prediction position with the target detection candidate area.

[0017] The preset lightweight perception detection network changes the conventional convolution mode, greatly reduces the network parameter quantity, the preset time domain trajectory prediction module is used, a bidirectional feature fusion module based on context perception is designed, global feature information is enriched, the model feature extraction capability is improved, the target time domain motion features are used by the preset time domain trajectory prediction module, target position prediction and false alarm filtering are realized, and the problem that the balance between light weight and precision is difficult to maintain is effectively solved. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the one or more embodiments of the present specification or the prior art, the drawings needed to be used in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present specification, and those skilled in the art can also obtain other drawings according to these drawings without paying creative labor.

[0019] Figure 1 A flowchart of an infrared dim small target real-time detection method according to an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a lightweight perception detection network according to an embodiment of the present application;

[0021] Figure 3 A schematic diagram of a Ghost module convolution process according to an embodiment of the present application;

[0022] Figure 4 A schematic diagram of a Ghost bottleneck structure according to an embodiment of the present application;

[0023] Figure 5 A schematic diagram of multi-scale fusion according to an embodiment of the present application;

[0024] Figure 5 (a) A schematic diagram of an FPN according to an embodiment of the present application;

[0025] Figure 5 (b) A schematic diagram of a CAM-PANet according to an embodiment of the present application;

[0026] Figure 6 A schematic diagram of a CAM module according to an embodiment of the present application;

[0027] Figure 7 A schematic diagram of a time domain trajectory prediction module according to an embodiment of the present application;

[0028] Figure 8 An example infrared data set according to an embodiment of the present application;

[0029] Figure 8 (a) An example infrared data set of a sky background according to an embodiment of the present application;

[0030] Figure 8 (b) An example infrared data set of a complex ground background according to an embodiment of the present application;

[0031] Figure 9 A comparison schematic diagram of weak and small target detection in a sky background according to an embodiment of the present application;

[0032] Figure 9 (a) A labeled contrast diagram in a sky background according to an embodiment of the present application;

[0033] Figure 9 (b) A schematic diagram of an algorithm provided by an embodiment of the present application in a sky background according to an embodiment of the present application;

[0034] Figure 9 (c) A schematic diagram of YOLOv4 in a sky background according to an embodiment of the present application;

[0035] Figure 9 (d) A schematic diagram of YOLOv4-tiny in a sky background according to an embodiment of the present application;

[0036] Figure 10 Contrast diagram for weak and small target detection of complex ground background of embodiments of the present application;

[0037] Figure 10 (a) Labeling diagram for complex ground background of embodiments of the present application;

[0038] Figure 10 (b) Diagram for algorithm provided by embodiments of the present application under complex ground background of embodiments of the present application;

[0039] Figure 10 (c) Diagram for YOLOv4 of complex ground background of embodiments of the present application;

[0040] Figure 10 (d) Diagram for YOLOv4-tiny of complex ground background of embodiments of the present application;

[0041] Figure 11 P-R curve diagram for ablation experiment of embodiments of the present application;

[0042] Figure 11 (a) P-R curve diagram for YOLOv4-tiny of embodiments of the present application;

[0043] Figure 11 (b) P-R curve diagram for YOLOv4-tiny+GhostNet of embodiments of the present application;

[0044] Figure 11 (c) P-R curve diagram for GhostNet+CAM-PANet of embodiments of the present application;

[0045] Figure 11 (d) P-R curve diagram for GhostNet+CAM-PANet+Track of embodiments of the present application. DETAILED DESCRIPTION

[0046] In order to enable persons skilled in the art to better understand the technical solutions in one or more embodiments of the present specification, the technical solutions in one or more embodiments of the present specification will be described clearly and completely in conjunction with the drawings in one or more embodiments of the present specification. Obviously, the described embodiments are only a part of the embodiments of the present specification, not all the embodiments. Based on one or more embodiments of the present specification, all other embodiments obtained by persons skilled in the art without creative labor should be within the protection scope of the present document.

[0047] Method embodiment

[0048] The embodiment of the application provides a real-time detection method for an infrared weak small target, Figure 1 A flow chart of a real-time detection method for an infrared weak small target according to the embodiment of the application is shown in the figure, Figure 1 The real-time detection method for the infrared weak small target according to the embodiment of the application specifically comprises the following steps:

[0049] S1, sequentially collecting infrared images of the infrared weak small target and inputting the infrared images into a preset lightweight perception detection network frame by frame, extracting target spatial features through the lightweight perception detection network, and generating a target detection candidate area;

[0050] The lightweight perception detection network Ghost-yolo-tiny is based on the YOLOv4-tiny architecture, and the lightweight perception detection network mainly comprises a Backbone backbone network, a Neck neck network and a Head head prediction network, as shown in the figure, Figure 2 The Backbone backbone network adopts a lightweight network module GhostNet, extracts different levels of feature information, and generates multi-layer feature maps; the Neck neck network constructs a bidirectional feature fusion module based on context perception, enhances the model's ability to perceive global context information, and introduces a bidirectional feature aggregation path parallel to upsampling and downsampling, fully utilizes shallow fine-grained feature information, and improves the utilization rate of target features; the Head target prediction network sets three conventional detection scales to two output feature scales, 52x52 and 104x104, which are more suitable for weak small target detection, thereby improving the detection accuracy and speed of the infrared weak small target.

[0051] The Backbone backbone network and the Neck neck network will be described in detail as follows:

[0052] 1. Construction of the backbone network

[0053] When a conventional convolution is used to extract infrared image features, a large number of gray feature maps are generated, and the gray scale color domain has small distinguishability, so that a large number of redundant feature maps appear, and the existence of the redundant feature maps is limited for improving the model performance, but the use of the conventional convolution to generate the redundant feature maps greatly increases the model calculation amount and reduces the detection speed. To construct a lightweight model, the GhostNet is used as the backbone feature extraction network, the way of generating feature maps by the conventional convolution is changed, redundant feature information is obtained with lower parameter amount, the model calculation amount is reduced, and the feature extraction effect of the model is ensured. The convolution process of the Ghost module is shown in the figure, Figure 3 The first step is to obtain a condensed feature layer from the input feature layer through a standard convolution Conv, the second step is to obtain a similar feature layer of the condensed feature layer through a linear operation, and the third step is to obtain a final output feature layer by Concat splicing the condensed feature layer and the similar feature layer.

[0054] Assuming that the input feature layer is h x w x c in , the output feature layer is h' x w' x c out , and the convolution kernel size is k x k, the input feature layer is divided into n parts.

[0055] The calculation amount of the conventional convolution processing is:

[0056] n x h' x w' x c x k x k

[0057] Formula 1;

[0058] The calculation amount of the Ghost module processing is:

[0059]

[0060] The calculation amount of the Ghost module is divided into two parts: standard convolution and depth convolution. By comparing the calculation amounts of the conventional convolution and the Ghost module, it can be seen that the compression rate of the Ghost model is n, and the calculation amount is obviously reduced. The stacking of two Ghost modules forms a Ghost bottleneck structure. According to the strides of 1 and 2, there are two cases, as shown in FIG. 1, which is a schematic diagram of the Ghost bottleneck structure. The first Ghost module is used to expand the channel number, and the second Ghost module is used to reduce the channel number, so as to match the residual edge with the input channel number. Finally, the Ghost bottleneck is used to build the GhostNet network, and the parameter table of the GhostNet network is shown in Table 1. Figure 4

[0061] Table 1 Parameter table of the GhostNet network

[0062]

[0063] 2. Construction of bidirectional feature fusion based on context perception

[0064] The multi-scale feature fusion strategy can effectively improve the small target detection precision by fusing different levels of features. The feature pyramid structure FPN is a classic feature fusion method, Figure 5 (a) is a schematic diagram of the FPN structure; through a top-down upsampling path, deep features are fully utilized to integrate deep semantic information into shallow feature layers. However, due to the extremely unobvious outline and texture features of the infrared small target, the target features are seriously lost after a large number of down-sampling of the deep feature layer, and there are problems of low feature utilization and poor fusion effect in the fusion process. In order to preserve multi-scale information and enhance the expression ability of features, the embodiment of the present application proposes a bidirectional feature fusion module CAM-PANet based on context perception, as shown in Figure 5 ​(b)As shown, the network's feature extraction capability is improved by enriching context information and enhancing the use of fine-grained feature information.

[0065] The input of the CAM-PANet module comes from the multi-scale feature maps extracted by the backbone network. In the backbone network, four scales of feature maps {C2, C3, C4, C5} are generated, including 208x208, 104x104, 52x52, and 26x26. However, the noise information in the C2 feature map is complex, so {C3, C4, C5} are selected as the input effective feature layer. The effective feature layer is first processed by upsampling operation to obtain {P3, P4, P5}, which integrates deep feature information into shallow layers and enriches the semantic information of each layer. Then, the network's context perception ability is strengthened. The C5 feature layer is input into the CAM context perception module to obtain context information of different scales, which is then integrated into P5 to make the output feature maps {P3, P4, P5} fully contain global context information. In addition, inspired by the PAN network, a bidirectional feature fusion structure is designed. On the basis of top-down, a bottom-up path is added. N3 is reduced to the same scale as N4 by downsampling, and the shallow fine-grained features of N3 are integrated into N4 by addition operation, thereby effectively improving the utilization of high-resolution information and making up for the lack of shallow information in deep feature layers. Overall, the CAM-PANet structure enables each scale of feature map to have strong context perception ability and fine-grained feature information, which enhances the feature fusion effect.

[0066] The CAM module uses context information of different levels to solve the problem of missing local features. The structure diagram of the CAM module is shown in Figure 6 The input feature layer (HxWxC) is processed by four parallel empty convolutions with sizes of 2, 4, 6, and 8 to obtain four branch features {f1, f2, f3, f4} containing different scale context information. Then, the branch features are fused using the Concat splicing method, and the size of the fused feature map T is HxWx4C. A 1x1 convolution is used to reduce the dimension of the feature map to C channels, and a 3x3 convolution is used to map the reduced feature to the corresponding branch, obtaining the feature The weight matrix of each branch is shown in equation 3.

[0067]

[0068] where i represents the i-th branch, W is the branch weight, δ is the activation function, and W is the branch weight, δ is the activation function,

[0069]

[0070]

[0071] S2, acquire the detection result of a certain frame image in a sequence infrared image through a preset lightweight perception detection network, and record it as a time domain input data set;

[0072] S3, input the time domain input data set into a preset time domain trajectory prediction module, acquire a target prediction position after the time domain trajectory prediction module performs an associated candidate target motion trajectory, trajectory screening and trajectory prediction operation on the time domain input data set;

[0073] Since the weak and small target in the infrared background has a small degree of distinction from the background noise, a large number of false alarms exist in the initial detection result. Based on this, the time domain trajectory prediction module is constructed, the time sequence correlation between the sequence images is utilized to associate the candidate target trajectory, the false alarm trajectory in the initial detection result is eliminated according to the target motion trajectory feature, and finally the target position of the subsequent frame is predicted, so that the algorithm complexity is reduced. Figure 7 FIG. 1 is a schematic diagram of the time domain trajectory prediction module of the embodiment of the present application; according to Figure 7 It can be known that the time domain trajectory prediction module specifically comprises:

[0074] S31, associate the target trajectory.

[0075] The real target has motion continuity, and the target position in the adjacent two frames will not be mutated. The position of the target in the Kth frame is (x, y), and the position of the target in the K+1th frame should be in the neighborhood range with (x, y) as the center. If there is a target in the neighborhood range, the target is associated, and if there is no target, there is no target to be associated in the frame.

[0076] S32, trajectory screening

[0077] The main difference between the real target and the false alarm noise motion trajectory T is that the motion of the real target has continuity, the motion direction and speed will not be mutated, and the motion trajectory is relatively smooth; the appearance of the noise often has randomness and independence, and the position of the same noise point is basically unchanged. The trajectory screening is realized by utilizing the target motion direction, target position feature and target trajectory length three features in the embodiment of the present application, and the calculation formulas of the features are as follows.

[0078]

[0079] Wherein, (x i ,y i ) represents the target centroid coordinates on the ith trajectory T i , n represents the number of targets, and the calculated β is the trajectory motion direction angle.

[0080] d i =(x i -xi-1 ) 2 +(y i -y i-1 ) 2

[0081] Formula 6;

[0082]

[0083] Among them, (x i ,y i ) represents the position of the target's centroid, d i Let f be the target inter-frame moving pixels, and f be the mean square error.

[0084]

[0085] Where L represents the total number of associated frames, which is taken as L=15 here, and K is the length of the associated trajectory.

[0086] S33, Trajectory Prediction

[0087] Based on the above target motion trajectory features, trajectory filtering is performed to retain the next true target motion trajectory. In subsequent frames, the Kalman filter method is used to predict the trajectory and predict the target position in the next frame.

[0088] S4. After associating and matching the predicted target location with the target detection candidate area, obtain the real-time detection result of the infrared weak target. Step S4 specifically includes:

[0089] To eliminate false alarms in subsequent frames, the predicted target location is matched with the target detection candidate region location. If the target detection location is within the neighborhood of the target detection candidate region, the two are matched, the candidate region location is retained, and other target detection candidate regions that deviate from the predicted location are filtered out. If the predicted location cannot be matched with any target detection candidate region, a target is missed. In this case, the target predicted location information is supplemented to improve the target detection accuracy.

[0090] The following specific example verifies the real-time infrared target detection method proposed in this invention. The steps are as follows:

[0091] 1. Construct a dataset of small infrared targets under different backgrounds.

[0092] To fully verify the effectiveness of the proposed algorithm in detecting weak infrared targets under different backgrounds, this embodiment of the invention selects infrared image sequences under two backgrounds: sky and complex ground, as the experimental dataset, such as... Figure 8 As shown. Among them, Figure 8 (a) The background infrared data of the sky is a video of a UAV target flying at a distance of about 800 meters, which was actually captured by the research group. The image sequence consists of 1427 frames.Figure 8 (b) For the image data of complex ground background, the public infrared dim target data set is selected, and a total of 3126 frames are included. The data set constructed by the above-mentioned 4553 frames of images is trained according to the ratio of training set: test set: verification set = 8:1:1.

[0093] 2. Evaluation index and training parameter

[0094] In order to objectively evaluate the performance of the algorithm, the precision (P), recall (R), average precision value (mAP) and model size are used as evaluation indexes, and the comparative experiments of the algorithm of the embodiment of the present application and several mainstream algorithms are carried out. At the same time, through the ablation experiment, the improvement effect of each module on the detection performance is further verified.

[0095] During the training of the network model, the Adam optimizer is used to dynamically optimize the network training parameters, the initial learning rate is set to 0.001, the batch size is 8, the backbone network is frozen for 50 epochs; then the backbone feature extraction network is unfrozen for training, the learning rate is set to 0.0001, the batch size is 16, and the training is completed after 150 epochs. The specific configuration content of the computer used in the experiment is shown in Table 2.

[0096] Table 2 Experimental environment configuration

[0097]

[0098] 3. Infrared dim target detection experiment under sky background

[0099] Figure 9 is the detection result of the model of the embodiment of the present application, the YOLOv4 model and the YOLOv4-tiny model on the dim target under the sky background in the infrared data set, Figure 9 (a) The mark control chart under the sky background of the embodiment of the present application; Figure 9 (b) The schematic diagram of the algorithm provided by the embodiment of the present application under the sky background of the embodiment of the present application; Figure 9 (c) The schematic diagram of YOLOv4 under the sky background of the embodiment of the present application; Figure 9 (d) The schematic diagram of YOLOv4-tiny under the sky background of the embodiment of the present application; in the figure, there is only one real target, as Figure 9 (a) shown in the mark control chart. There are fewer false alarms under the sky background, but there are mistakes and omissions in the detection results of YOLOv4 and YOLOv4-tiny, while in the detection results of the model of the embodiment of the present application, the target can be accurately detected, and the detection precision is high.

[0100] 4. Infrared dim target detection experiment under complex ground background

[0101] In order to further verify the robustness and compatibility of the algorithm of the embodiment of the present application, the infrared dim target detection will be carried out under the background of the complex ground in the aerial view. Figure 10 is the detection result of the model of the embodiment of the present application and the YOLOv4 and YOLOv4-tiny model on the dim target under the background of the infrared data set of the complex ground. Figure 10 (a) is a marked control diagram of the complex ground background of the embodiment of the present application;

[0102] Figure 10 (b) is a schematic diagram of the algorithm provided by the embodiment of the present application for the complex ground background of the embodiment of the present application; Figure 10 (c) is a schematic diagram of YOLOv4 for the complex ground background of the embodiment of the present application; Figure 10 (d) is a schematic diagram of YOLOv4-tiny for the complex ground background of the embodiment of the present application;

[0103] In the following three scenes, multiple detection results appear in the detection results of YOLOv4 and YOLOv4-tiny, and the false alarm rate is high. For example, in the first row, YOLOv4 detects two targets, and it is found by comparing with the control diagram that one of them is a false alarm, while the model of the embodiment of the present application can realize accurate detection of the real target. Similarly, in the second and third rows, the model of the embodiment of the present application has higher detection accuracy and more accurate positioning, effectively eliminating false alarms. It shows that the target detection model of the embodiment of the present application still has higher detection ability under the complex background, the algorithm robustness is stronger, and it can adapt to the change of complex scene.

[0104] 5. Detection effect comparison and analysis

[0105] The experimental results of the comparison between the algorithm of the embodiment of the present application and the target detection algorithm are shown in Table 3. The mAP value of the algorithm of the embodiment of the present application for infrared dim target detection is 81.46%, which is 44.07%, 8.98%, 10.26% and 42.76% higher than that of the mainstream single-stage detection algorithms SSD, YOLOv4, YOLOv5 and YOLOv7 respectively, which shows that the model of the embodiment of the present application has obvious improvement in detection accuracy. The model parameter quantity measures whether a model is lightweight, and the model parameter quantity of the model of the embodiment of the present application is 12.84MB, which is 11.98MB, 9.57MB, 2.65MB and 1.76MB lower than that of the lightweight models YOLOv3-tiny, YOLOv4-tiny, Efficientdet and MobileDets respectively. The frame number of the model of the embodiment of the present application is 58.3FPS, and when the frame number reaches more than 30FPS, it can be judged that the model reaches the real-time detection standard. According to the obtained experimental data, it can be shown that the model is a lightweight model, and has high infrared dim target detection performance, and can meet the real-time detection requirement.

[0106] Table 3 Comparison experimental results of different target detection models

[0107]

[0108] 6 Ablation experiment

[0109] In order to analyze the performance influence of each module in the algorithm of the embodiment of the present application on infrared dim target detection, an ablation experiment is performed, and the experimental results are shown in Table 4. The first group of experiments is the result of the benchmark model YOLOv4-tiny, and the mAP value is 49.74% at this time. The second group of experiments is the detection result after combining the time domain trajectory prediction module with the benchmark model, and the mAP value increases by 10.15%, which shows that the time sequence trajectory prediction module designed in the embodiment of the present application has obvious false alarm target filtering effect. The third group of experiments is to replace the backbone network of the benchmark model with the GhostNet model, at this time the parameter quantity is effectively reduced by 8.09MB, and the mAP value increases by 10.05%, which shows that the GhostNet model has stronger feature extraction ability for infrared dim targets. The fourth group of experiments introduces the bidirectional feature fusion module based on context perception on the basis of the third group of experiments, at this time the model detection recall rate, recall rate and detection accuracy are obviously improved, and the mAP value can reach 70.27%, which shows that the module can effectively utilize feature information and improve the utilization rate of infrared feature information. The fifth group of experiments increases the time domain trajectory prediction module, and the model detection accuracy is further improved, and the Precision, Recall and AP are 88.63%, 85.18% and 81.46% respectively.

[0110] Figure 11 The P-R curve visualization results of the four groups of ablation experiments are shown in Figure 2.Figure 11 (a) is the P-R curve diagram of YOLOv4-tiny of the embodiment of the application; Figure 11 (b) is the P-R curve diagram of YOLOv4-tiny+GhostNet of the embodiment of the application; Figure 11 (c) is the P-R curve diagram of GhostNet+CAM-PANet of the embodiment of the application; Figure 11 (d) is the P-R curve diagram of GhostNet+CAM-PANet+Track of the embodiment of the application. The area surrounded by Recall and Precision in the training process increases continuously, that is, the mAP value is effectively improved, and it can be seen that the performance of each module of the algorithm of the embodiment of the application on infrared dim small target detection is obviously improved.

[0111] Table 4 ablation experiment

[0112]

[0113] By adopting the embodiment of the application, the following beneficial effects are achieved:

[0114] The infrared dim small target detection model of the light-weight time domain perception network reduces the redundant parameter quantity by changing the feature extraction mode of the backbone network, constructs a bidirectional feature fusion module based on context perception, uses different size hole convolution to extract the context features on the last effective layer, increases the utilization degree of shallow layer fine-grained information, and supplements to the feature fusion, so as to strengthen the feature extraction capability of the model on the small target and improve the detection capability of the model; the time domain trajectory prediction module is designed, the target motion trajectory information is used to further eliminate the detection false alarm, and the noise interference is reduced. Under the sky and complex ground background, the experimental results show that compared with the current mainstream target detection algorithm, the algorithm provided by the embodiment of the application is obviously improved in accuracy, recall rate and precision rate and other performances, has real-time detection performance, and the research results can provide technical support for solving the real-time and efficient detection problem of infrared detection field targets.

[0115] Device embodiment

[0116] The application provides an infrared dim small target real-time detection device, which comprises:

[0117] The perception detection network module is used for inputting the sequence infrared images frame by frame into a preset light-weight perception detection network, extracting target spatial features through the light-weight perception detection network, and generating a target detection candidate area;

[0118] The time domain input data set generation module is used for obtaining the detection result of the front specific frame image in the sequence infrared image generated through the preset light-weight perception detection network, and recording as a time domain input data set;

[0119] The time domain trajectory prediction module is configured to input the time domain input data set into a preset time domain trajectory prediction module, and the time domain trajectory prediction module is configured to perform association candidate target motion trajectory, trajectory screening and trajectory prediction operations on the time domain input data set to obtain a target prediction position;

[0120] The target matching module is configured to perform association matching between the target prediction position and the target detection candidate region to obtain a real-time detection result.

[0121] The perception detection network module specifically comprises a backbone network module, a neck network module and a head network module;

[0122] The backbone network module is specifically configured to perform convolution operation on the GhostNet as a backbone feature extraction network to generate feature maps of each scale;

[0123] The neck network module is specifically configured to introduce a CAM enhanced context perception module to construct a context perception bidirectional feature fusion module CAM-PANet, so that each scale feature map has context perception ability and fine-grained feature information through the context perception bidirectional feature fusion module CAM-PANet;

[0124] The backbone network module specifically comprises:

[0125] The sequence infrared image is taken as an input feature layer to obtain a condensed feature layer through standard convolution Conv;

[0126] A similar feature layer of the condensed feature layer is obtained through linear operation;

[0127] The condensed feature layer and the similar feature layer are concatenated to obtain a final output feature layer;

[0128] The neck network specifically comprises:

[0129] The input feature layer HxWxC is processed through four parallel hole convolutions with sizes of 2, 4, 6 and 8 to obtain four branch features {f1, f2, f3, f4} containing different scale context information;

[0130] The branch features are fused through Concat splicing, and the fused feature map T has a size of HxWx4C;

[0131] The feature map is reduced to a channel number of C through 1x1 convolution, and the reduced feature is mapped through 3x3 convolution to obtain a feature The weight matrix of each branch is obtained through formula 1;

[0132]

[0133] Finally, the four branch features obtained are weighted feature fusion, and the final output global context feature R is obtained through formula 2,

[0134]

[0135] Wherein, i represents the i-th branch, W is the branch weight, and δ is the activation function, is the branch mapping matrix.

[0136] The time domain trajectory prediction module specifically comprises:

[0137] The associated target trajectory module is used to judge whether the position in the K+1 frame in the sequence infrared image is in the neighborhood range of the position in the K frame. If it is in the neighborhood range, the target association is performed, otherwise the target association is not performed.

[0138] The trajectory screening module is used to realize trajectory screening by using three aspects of features of weak small target motion direction, target position feature and target trajectory length, specifically comprising: obtaining the weak small target motion direction through formula 3, obtaining the target position feature through formula 4 and formula 5, and obtaining the target trajectory length through formula 6.

[0139]

[0140] Wherein, (x i ,y i ) represents the target centroid coordinates on the i-th trajectory T i , n represents the number of targets, and β is the trajectory motion direction angle.

[0141] d i =(x i -x i-1 ) 2 +(y i -y i-1 ) 2

[0142] Formula 4;

[0143]

[0144] Wherein, (x i ,y i ) represents the target centroid position, d i is the target interframe moving pixel, and f is the mean square error.

[0145]

[0146] Wherein, L represents the total number of frames of association, and K is the length of the associated trajectory.

[0147] The trajectory prediction module is configured to predict the position of the small target in the next frame by using a Kalman filter method.

[0148] The target matching module is configured to:

[0149] The target prediction position is matched with the target detection candidate region. If the target detection position is in the neighborhood of the target detection candidate region, the two are matched, the position of the target detection candidate region is reserved, and the target detection candidate region deviating from the prediction position is filtered out. If the target prediction position cannot be matched with all target detection candidate regions, the target prediction position information is completed.

[0150] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A real-time detection method for infrared dim small targets, characterized in that, The application relates to a method for detecting infrared weak small targets, and the method comprises the following steps: S1, sequentially inputting infrared images of infrared weak small targets into a preset lightweight perception detection network frame by frame, extracting target spatial features through the lightweight perception detection network, and generating a target detection candidate area; S2, acquiring a detection result of a front specific frame image in the sequence infrared images through the preset lightweight perception detection network, and recording the detection result as a time domain input data set; S3, inputting the time domain input data set into a preset time domain trajectory prediction module, acquiring a target prediction position after the time domain trajectory prediction module performs an associated candidate target motion trajectory, trajectory screening and trajectory prediction operation on the time domain input data set; S4, after associating and matching the target prediction position and the target detection candidate area, acquiring a real-time detection result of the infrared weak small target; The construction steps of the preset lightweight perception detection network in the step S1 specifically comprise the following steps: establishing a backbone network, establishing a neck network and establishing a head network; The establishment of the backbone network specifically comprises the following steps: taking a GhostNet as a backbone feature extraction network to perform a convolution operation and generate scale feature maps; The establishment of the neck network specifically comprises the following steps: introducing a CAM to enhance a context perception module, constructing a context perception bidirectional feature fusion module CAM-PANet, and enabling the scale feature maps to have context perception capability and fine-grained feature information through the context perception bidirectional feature fusion module CAM-PANet; The construction steps of the preset time domain trajectory prediction module in the step S3 specifically comprise the following steps: S31, judging whether a position in a K+1 frame in the sequence infrared images is in a neighborhood range of a position in a K frame, if the position is in the neighborhood range, target association is performed, otherwise, target association is not performed; S32, realizing trajectory screening by using three features of a weak small target motion direction, a target position feature and a target trajectory length; S33, performing trajectory prediction by using a Kalman filter method to predict a weak small target position in a next frame; The step S32 specifically comprises the following steps: a weak small target motion direction is acquired through a formula 3, a target position feature is acquired through a formula 4 and a formula 5, and a target trajectory length is acquired through a formula 6; Equation 3; in, Indicates the first i Trajectory The coordinates of the centroid of the target on the graph, where n represents the number of targets. The direction angle of the trajectory motion; Formula 4: Formula 5: wherein, denotes the target centroid position, is the target inter-frame moving pixel, f is the mean square error; Formula 6: wherein, L represents the total number of frames associated, K is the length of the trajectory to which it is associated.

2. The method of claim 1, wherein, The convolution operation of taking the GhostNet as the backbone feature extraction network to generate the scale feature maps specifically comprises the following steps: taking the sequence infrared images as an input feature layer to obtain a condensed feature layer through standard convolution Conv; obtaining a similar feature layer of the condensed feature layer through linear operation; performing Concat splicing on the condensed feature layer and the similar feature layer to obtain a final output feature layer.

3. The method of claim 1, wherein, The step of introducing the CAM to enhance the context perception module specifically comprises the following steps: The input feature layer is processed by four parallel hole convolutions with sizes of 2, 4, 6, and 8 to obtain four branch features containing context information at different scales The input feature layer is processed by four parallel hole convolutions with sizes of 2, 4, 6, and 8 to obtain four branch features containing context information at different scales ; The branch features are fused by using a Concat splicing manner, and the fused feature map The size is ; Use Convolution reduces the feature map to the channel number , and a 3x3 convolution is used to map the reduced features to the corresponding branch, obtaining the feature The weight matrix of each branch is obtained by formula 1; Formula 1 ; Finally, the four branch features are weighted and fused to obtain the final output global context feature through formula 2 , Formula 2: wherein, represents the branch, is a branch weight, is an activation function, is a branch mapping matrix.

4. The method of claim 1, wherein, The step S4 specifically comprises the following steps: after associating and matching the target prediction position and the target detection candidate area, if the target detection position is in a neighborhood range of the target detection candidate area, the target detection candidate area position is reserved and other target detection candidate areas deviating from the prediction position are filtered out, if the target prediction position cannot be matched with all the target detection candidate areas, the target prediction position information is completed.

5. An infrared dim small target real-time detection device, characterized in that, The method comprises the following steps: a perception detection network module is used to input a sequence of infrared images frame by frame into a preset lightweight perception detection network, extract target spatial features through the lightweight perception detection network, and generate a target detection candidate area; a time domain input data set generation module is used to obtain a detection result of a front specific frame image in the sequence of infrared images generated through the preset lightweight perception detection network, which is recorded as a time domain input data set; a time domain trajectory prediction module is used to input the time domain input data set into a preset time domain trajectory prediction module, and obtain a target predicted position after the time domain trajectory prediction module performs an associated candidate target motion trajectory, trajectory screening and trajectory prediction operation on the time domain input data set; a target matching module is used to obtain a real-time detection result after the target predicted position is associated and matched with the target detection candidate area; the perception detection network module specifically comprises a backbone network module, a neck network module and a head network module; the backbone network module is specifically used to perform convolution operation through a GhostNet as a backbone feature extraction network to generate feature maps of each scale; the neck network module is specifically used to introduce a CAM enhanced context perception module to construct a context perception bidirectional feature fusion module CAM-PANet, so that the feature maps of each scale have context perception ability and fine-grained feature information through the context perception bidirectional feature fusion module CAM-PANet; the time domain trajectory prediction module specifically comprises: an associated target trajectory module is used to judge whether a position in a K+1 frame of the sequence of infrared images is in a neighborhood range of a position in a K frame, if it is in the neighborhood range, target association is performed, otherwise target association is not performed; a trajectory screening module is used to realize trajectory screening by using three features of weak target motion direction, target position feature and target trajectory length, specifically including obtaining the weak target motion direction through formula 3, obtaining the target position feature through formula 4 and formula 5, and obtaining the target trajectory length through formula 6; Equation 3; wherein, represents the target centroid coordinates on the nth i track, n represents the number of targets, is the track motion direction angle; Formula 4; Equation 5; wherein, denotes the target centroid position, is the target inter-frame moving pixel, f is the mean square error; Equation 6; wherein, L represents the total number of frames associated, K is the length of the trajectory associated; a trajectory prediction module is used to perform trajectory prediction by using a Kalman filter method to predict a weak target position in a next frame.

6. The apparatus of claim 5, wherein, the backbone network module specifically comprises: the sequence of infrared images is taken as an input feature layer to obtain a condensed feature layer through standard convolution Conv; a similar feature layer of the condensed feature layer is obtained through linear operation; the condensed feature layer and the similar feature layer are concatenated to obtain a final output feature layer; the neck network specifically comprises: The input feature layer is processed by four parallel hole convolutions with sizes of 2, 4, 6, and 8 to obtain four branch features containing context information at different scales The input feature layer is processed by four parallel hole convolutions with sizes of 2, 4, 6, and 8 to obtain four branch features containing context information at different scales ; The branch features are fused by using a Concat splicing manner, and the fused feature map The size is ; Using Convolution reduces the feature map to the channel number , and a 3x3 convolution is used to map the reduced features to the corresponding branch, obtaining features The weight matrix of each branch is obtained by formula 1; Formula 1 ; Finally, the four branch features are weighted and fused to obtain the final output global context feature through formula 2 , Formula 2: wherein, represents the branch, is a branch weight, is an activation function, is a branch mapping matrix.

7. The apparatus of claim 5, wherein, the target matching module is specifically used to: associate and match the target predicted position with the target detection candidate area, if the target detection position is in the neighborhood range of the target detection candidate area, the two can be matched, the target detection candidate area position is retained, and other target detection candidate areas deviating from the predicted position are filtered out; if the target predicted position cannot be matched with all target detection candidate areas, the target predicted position information is completed.

Images