Dynamic small target sensing method based on detection tracking cooperation and motion clue enhancement

Abstract

The invention provides a dynamic small target sensing method based on detection tracking cooperation and motion clue enhancement, and relates to the field of small target detection sensing in a complex environment. The dynamic small target sensing method comprises the following steps: acquiring continuous multi-frame grayscale images of a dynamic small target and tracking information of the dynamic small target, wherein the continuous multi-frame grayscale images comprise motion clues of the dynamic small target; and inputting the continuous multi-frame grayscale image and the tracking information into a target detection network, and carrying out feature extraction and feature fusion on the dynamic small target by adopting the target detection network to obtain a detection result of the dynamic small target, and the motion clues and tracking information in the continuous multi-frame grayscale images are used for enhancing the feature extraction capability of the target detection network on the dynamic small target.

Description

Technical Field

[0001] The present invention relates to the field of small target detection and perception in complex environments, and in particular to a dynamic small target perception method and device based on detection and tracking coordination and motion cue enhancement. Background Art

[0002] The widespread popularity of drones has led to frequent abuse and "illegal flying", which not only poses a potential threat to public safety, but also poses a severe challenge to airspace management. Faced with this problem, how to efficiently and accurately detect and perceive aerial drones, a typical small dynamic target, has become a research hotspot in academia and industry. Infrared thermal imaging, as a day-and-night continuous passive detection imaging technology, has become one of the important means of drone target detection due to its advantage of not being restricted by lighting conditions. However, infrared imaging can only display temperature differences, lacks color information and texture details, and drone targets are usually small and easily interfered by factors such as thermal crosstalk, dynamic background, and motion occlusion. These have brought huge challenges to the accurate perception of dynamic small targets such as drones.

[0003] The industry has conducted a lot of research on this issue. However, existing methods mostly focus on target tracking tasks, usually assuming that the target has been detected and requiring manual template setting. This method that relies on prior information is difficult to adapt to the needs of initial target detection in actual scenarios. At the same time, due to changes in the scale and perspective of drone imaging during the observation process, as well as the frequent appearance / disappearance of targets, long-term reliable tracking is difficult to achieve. To this end, researchers have proposed some collaborative methods that fuse detection and tracking, such as confidence-based post-processing fusion or detection and tracking switching based on evidence theory. However, these methods usually require additional design of fusion conditions or switching rules, which increases system complexity and parameter adjustment requirements, while affecting computational efficiency.

[0004] On the other hand, target detection is the basis for realizing autonomous perception of drones, and small target detection has always been a difficult point in the field of target detection, especially in infrared imaging with little texture information and low contrast. In complex backgrounds, infrared targets are difficult to distinguish from the background. Conventional single-frame detection methods have difficulty extracting effective features of small targets and are prone to missed detection or false detection. For this reason, multi-frame detection methods have received much attention by extracting spatiotemporal motion information. Such methods either rely on optical flow or association networks for feature association, but are limited by single-frame feature extraction and are susceptible to interference when optical flow is inaccurate; or explicitly extract motion areas through image difference and other methods to guide the detection network to focus on the target. However, infrared images often make image registration difficult due to insufficient texture and significant background dynamic changes, which affects the effect of optical flow or difference methods. Summary of the invention

[0005] In view of this, the present invention provides a dynamic small target perception method and device based on detection and tracking collaboration and motion cue enhancement.

[0006] On the one hand, the present invention provides a dynamic small target perception method based on detection and tracking collaboration and motion clue enhancement, including: obtaining continuous multi-frame grayscale images of the dynamic small target and tracking information of the dynamic small target, the continuous multi-frame grayscale images contain motion clues of the dynamic small target; based on the continuous multi-frame grayscale images and tracking information, using a target detection network to extract and fuse features of the dynamic small target to obtain detection results of the dynamic small target, wherein the motion clues and tracking information in the continuous multi-frame grayscale images are used to enhance the feature extraction capability of the target detection network for the dynamic small target.

[0007] According to an embodiment of the present invention, based on continuous multiple frames of grayscale images and tracking information, a target detection network is used to perform feature extraction and feature fusion on dynamic small targets, including: allocating continuous multiple frames of grayscale images to different color channels, so that motion cues are converted into pixel differences in different color channels of continuous multiple frames of grayscale images, and the pixel differences are used to enhance the target detection network's perception of dynamic small targets; converting tracking information into spatial attention weights, and the spatial attention weights are used to guide the target detection network to focus on potential target areas of dynamic small targets; based on pixel differences and spatial attention weights, a target detection network is used to perform feature extraction and feature fusion on dynamic small targets.

[0008] According to an embodiment of the present invention, tracking information is converted into spatial attention weights, including: converting the tracking information into a mask, wherein the potential target area on the mask has a first value, the area outside the potential target area has a second value, and the first value is not equal to the second value; based on the first value and the second value, converting the mask into a spatial attention weight.

[0009] According to an embodiment of the present invention, based on pixel differences and spatial attention weights, a target detection network is used to perform feature extraction and feature fusion on dynamic small targets, including: after converting tracking information into spatial attention weights, the target detection network is used to extract different layers of features of the dynamic small targets; and attention feature fusion processing is performed on different layers of features to achieve adaptive fusion of different layers of features.

[0010] According to an embodiment of the present invention, attention feature fusion processing is performed on features at different layers, including: using maximum pooling to calculate channel attention weights and spatial attention weights for each layer of features at different layers in turn to obtain weighted features; and using a feature pyramid network and a path aggregation network to perform multi-scale feature fusion on the weighted features.

[0011] According to an embodiment of the present invention, continuous multiple frames of grayscale images of a dynamic small target and tracking information of the dynamic small target are obtained, including: preprocessing the collected continuous multiple frames of grayscale images to reduce the impact of background changes on target detection and tracking; determining tracking information from the preprocessed continuous multiple frames of grayscale images.

[0012] According to an embodiment of the present invention, preprocessing is performed on the collected continuous multiple frames of grayscale images, including: determining background changes in the continuous multiple frames of grayscale images, and registering and aligning the continuous multiple frames of grayscale images when background changes occur.

[0013] According to an embodiment of the present invention, tracking information is determined from a plurality of preprocessed continuous grayscale image frames, including: using a lightweight tracking algorithm based on a twin network, and determining tracking information from a current frame image in the plurality of preprocessed continuous grayscale image frames through a target tracking network.

[0014] According to an embodiment of the present invention, after obtaining the detection result of the dynamic small target, the dynamic small target perception method also includes: filtering the detection result to output the perception result of the dynamic small target, and automatically initializing or updating the template required for target tracking.

[0015] On the other hand, the present invention also provides a dynamic small target perception device based on detection and tracking coordination and motion clue enhancement, including: an acquisition module, used to obtain continuous multi-frame grayscale images of dynamic small targets and tracking information of dynamic small targets, the continuous multi-frame grayscale images contain motion clues of dynamic small targets; a detection module, used to extract and fuse features of dynamic small targets based on continuous multi-frame grayscale images and tracking information, using a target detection network to obtain detection results of dynamic small targets, wherein the motion clues and tracking information in the continuous multi-frame grayscale images are used to enhance the feature extraction capability of the target detection network for dynamic small targets.

[0016] The dynamic small target perception method and device based on detection tracking coordination and motion clue enhancement provided by the embodiment of the present invention integrates the motion clues and tracking information of the dynamic small target into the detection network, guides the detection network to focus on the potential target area, and enables the target detection network to better understand the motion pattern and context information of the dynamic small target, thereby improving the accuracy of detection. The method and device proposed by the present invention realize simple and efficient detection tracking coordination and motion clue utilization, effectively improving the perception effect of dynamic small targets such as drones in infrared videos under complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0018] Figure 1A flowchart of a dynamic small target perception method according to an embodiment of the present invention is schematically shown;

[0019] Figure 2 A schematic diagram of a target detection network according to an embodiment of the present invention is shown;

[0020] Figure 3 A schematic diagram of a framework diagram of a dynamic small target perception method according to an embodiment of the present invention is shown;

[0021] Figure 4 A flowchart of a dynamic small target perception method according to an embodiment of the present invention is schematically shown;

[0022] Figure 5 A schematic diagram showing a correlation diagram of detection results according to an embodiment of the present invention is shown;

[0023] Figure 6 A comparison diagram of detection and tracking coordination results according to an embodiment of the present invention is schematically shown;

[0024] Figure 7 A block diagram of a dynamic small target sensing device according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0025] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0026] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0027] In the description of the present invention, “several” means one or more, “more” means more than two, “greater than”, “less than”, “exceed”, etc. are understood as not including the number itself, and “above”, “below”, “within”, etc. are understood as including the number itself.

[0028] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0029] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0030] Figure 1 The flowchart of the dynamic small target perception method according to an embodiment of the present invention is schematically shown.

[0031] like Figure 1 As shown, the dynamic small target perception method based on detection tracking collaboration and motion cue enhancement may include steps S110 to S120.

[0032] In step S110, a plurality of continuous grayscale images of a small dynamic target and tracking information of the small dynamic target are acquired, wherein the plurality of continuous grayscale images contain motion clues of the small dynamic target.

[0033] In step S120, based on the continuous multi-frame grayscale images and tracking information, a target detection network is used to perform feature extraction and feature fusion on the dynamic small target to obtain the detection result of the dynamic small target, wherein the motion clues and tracking information in the continuous multi-frame grayscale images are used to enhance the feature extraction capability of the target detection network for the dynamic small target.

[0034] Dynamic small targets can be, for example, low-altitude aircraft such as drones, and their flight backgrounds are usually more complex. The grayscale image can be a single-channel grayscale image of infrared imaging, or it can be a grayscale image converted from an RGB multi-channel image. In other words, RGB multi-channel images can also be applicable to the dynamic small target perception method provided by the present invention. Continuous multi-frame grayscale images can include the latest current frame and adjacent multi-frame historical frames to form a multi-frame image window. As an example, the window size can be an integer greater than 3, such as 10, and can also be adjusted according to actual needs. The present invention does not make specific limitations. Motion clues can reflect characteristic information such as position changes and speed changes of dynamic small targets. Combined with the tracking information of dynamic small targets, the feature extraction capability of the target detection network to extract infrared dynamic small targets under complex backgrounds can be enhanced, thereby obtaining more accurate detection and tracking collaborative results.

[0035] In some embodiments, the above step S120 may include: allocating multiple consecutive frames of grayscale images to different color channels, so that motion clues are converted into pixel differences in different color channels of the multiple consecutive frames of grayscale images, and the pixel differences are used to enhance the target detection network's perception of dynamic small targets; converting tracking information into spatial attention weights, and the spatial attention weights are used to guide the target detection network to focus on potential target areas of dynamic small targets; based on pixel differences and spatial attention weights, using the target detection network to extract and fuse features of dynamic small targets.

[0036] Figure 2The structure diagram of the target detection network according to an embodiment of the present invention is schematically shown. Figure 2 As shown in the figure, the target detection network can include a motion cue enhancement branch and a position prior guidance branch. The motion cue enhancement branch can be used to distribute multiple frames of infrared grayscale images to different color channels, and use the pixel differences between channels to enhance the saliency of moving targets, thereby avoiding the infrared small target features being obliterated by the background during the convolution process, and no additional image difference or optical flow information is required. The position prior guidance branch can generate spatial attention weights for the tracking results and guide the detection network to focus on potential target areas. This branch can efficiently fuse the tracking results, avoid post-processing in the parallel structure of detection and tracking or conditional judgment in the switching structure, and verify the tracking results at the same time to reduce false detections.

[0037] In some embodiments, converting the tracking information into the spatial attention weight may include: converting the tracking information into a mask, wherein the potential target area on the mask has a first value, and the area outside the potential target area has a second value, and the first value is not equal to the second value; converting the mask into the spatial attention weight according to the first value and the second value. For example, the first value may be 1, and the second value may be 0. Figure 2 As shown, the mask-generated spatial attention weight can be applied to the shallow network of the feature extraction branch. The generation method of the spatial attention weight can be expressed as:

[0038]

[0039] Among them, mask represents mask, Weight mask represents the spatial attention weight of the mask, Conv(·) represents the convolution processing, and Sigmoid(·) represents the Sigmoid activation function processing.

[0040] In some embodiments, after the tracking information is converted into spatial attention weights, the target detection network can be used to extract different layer features of the dynamic small target, and the attention feature fusion processing can be performed on the different layer features to achieve adaptive fusion of different layer features. Figure 2 As shown in the figure, multiple convolutional blocks (CNN Block) can be used to extract deep, middle, shallow and other layer features of dynamic small targets, and adaptive fusion of features can be achieved through a specially designed attention fusion module (AFFM). The calculation method of the detection network attention fusion module can be similar to the convolutional block attention module (CBAM), which first calculates the channel attention and then the spatial attention.

[0041] In some embodiments, the channel attention weight and the spatial attention weight can be calculated in turn using maximum pooling for each layer of features at different layers to obtain weighted features, and then a feature pyramid network and a path aggregation network can be used to perform multi-scale feature fusion on the weighted features.

[0042] In view of the fact that the target area in the infrared small target scene usually has a high significance, the present invention improves the attention weight calculation method of CBAM, such as Figure 2 As shown, maximum pooling can be used to calculate channel and spatial attention weights, which can be expressed as:

[0043]

[0044] Among them, Weight channel Represents the channel attention weight, Weight spatial represents the spatial attention weight, MaxPool (·) represents the maximum pooling process, and FC (·) represents the fully connected layer process.

[0045] like Figure 2 As shown in the figure, the backbone and detection head of the target detection network can adopt a lightweight and flexible YOLOv8 network structure. The motion cue enhancement branch and the position prior guidance branch can share the same network backbone, expand the receptive field by increasing the number of convolutional layers to extract feature maps at different levels, and use the residual structure to ensure the effective transmission of feature flow. Multi-scale feature fusion can adopt feature pyramid networks and path aggregation networks to effectively fuse features at different levels through bottom-up and top-down information flow, so that the feature maps of each scale contain rich semantic information and detail information, thereby enhancing the detection ability of small targets. Finally, the target detection output is completed through decoupled target position regression and category classification.

[0046] Figure 3 A schematic diagram of a framework diagram of a dynamic small target perception method according to an embodiment of the present invention is shown; Figure 4 The flowchart of the dynamic small target perception method according to an embodiment of the present invention is schematically shown.

[0047] like Figure 3 and Figure 4 As shown, in some embodiments, before executing the above step S110, the collected continuous multiple frames of images may be preprocessed. That is, the dynamic small target perception method may also include: preprocessing the collected continuous multiple frames of grayscale images to reduce the impact of background changes on target detection and tracking; and determining tracking information from the preprocessed continuous multiple frames of grayscale images.

[0048] like Figure 3As shown, the preprocessing includes background change judgment and image registration and alignment. The preprocessing of the collected continuous multiple frames of grayscale images may include: judging the background changes in the continuous multiple frames of grayscale images, and registering and aligning the continuous multiple frames of grayscale images when the background changes occur. By judging the background changes and registering and aligning the images, the influence of the background changes on the target detection and tracking can be reduced.

[0049] The background change judgment may adopt the frame difference method, for example, to calculate the absolute difference between the current frame and the previous frame image. If the proportion of pixels with a difference value greater than a certain threshold to all pixels is large, it is considered that the background has changed.

[0050] When the background changes, the sparse optical flow tracking method can be used to achieve fast and efficient image registration and alignment. First, the Shi-Tomasi key corner points are extracted, and then the key points are tracked using the sparse optical flow algorithm based on the Lucas-Kanade method, and the affine transformation matrix is ​​calculated. The transformation matrix is ​​used to align the two frames of images, and a more accurate search area is provided for the target tracking network. Image alignment can be used only for the current frame and the previous frame to control the amount of calculation, and in order to ensure the robustness and rationality of the alignment, the number and distribution of key points can be set, and the amplitude of the translation and rotation of the change matrix can be limited according to actual conditions.

[0051] like Figure 3 and Figure 4 As shown, the tracking information can be determined from the preprocessed continuous multi-frame grayscale images. A lightweight tracking algorithm based on a twin network can be used to determine the tracking information from the current frame image of the preprocessed continuous multi-frame grayscale images through the target tracking network.

[0052] It should be noted that the multi-frame grayscale images are only used for the motion enhancement branch of the target detection network. The input of the position guidance branch of the target detection network and the target tracking network is the current frame image in the multi-frame image. The lightweight tracking algorithm can use NanoTrack, which uses the correlation between the template features and the search area features to locate the target. By designing an efficient network structure, it can achieve fast target tracking. It uses a pixel-by-pixel correlation convolution method to calculate the correlation:

[0053]

[0054] in is the pixel index after the template feature is flattened, y, x are the pixel coordinates of the input feature map, Kernel[c,i] is the value of the i-th pixel of the c-th channel of the convolution kernel formed by the template feature map. X[c,y,x] is the value of the search area feature map at the c-th channel position (y, x).

[0055] The NanoTrack tracking algorithm can be used to ensure the real-time performance of the program. In some embodiments, other tracking methods can also be used. The target tracking network does not need a manual target template. The method proposed in the embodiment of the present invention can automatically set and update the template after the target detection network detects a stable target. At the same time, the target tracking result will be used as a position prior to guide the target detection network to focus on the potential target area.

[0056] like Figure 2 , Figure 3 and Figure 4 As shown in Figure 1, the pre-processed continuous multi-frame and target tracking results can be input into the designed dual-path fusion multi-scale target detection network to achieve the fusion of motion information and tracking information. The processing process of the target detection network has been explained in the previous article and will not be repeated here.

[0057] Continue reading Figure 3 and Figure 4 After obtaining the detection result of the dynamic small target, the dynamic small target perception method also includes: filtering the detection result to output the perception result of the dynamic small target, and automatically initializing or updating the template required for target tracking.

[0058] In this embodiment, the detection result filtering is to match and associate the detection results between consecutive frames, and use the spatiotemporal continuity of the target to filter and output the most likely target. In order to solve the problem of difficulty in associating detection results when the background changes or the target moves quickly, a method for calculating the intersection over union (IoU) for moving scenes is proposed. This method not only considers the spatial overlap between targets, but also integrates factors such as target size change and center distance to improve the robustness of association in complex scenes.

[0059] Figure 5 The following schematically shows a correlation diagram of detection results according to an embodiment of the present invention. Figure 5 As shown, bounding box A and bounding box B represent different shapes, sizes and center positions of the target. The height and width of bounding box A are h1 and w1 respectively, and the height and width of bounding box B are h2 and w2 respectively. The distance between the centers of bounding box A and bounding box B is d, and the distance between the two diagonals is c. Figure 5 , the IoU calculation method can be as follows:

[0060]

[0061] Among them, A ctr , B ctr Represent the center points of bounding box A and bounding box B respectively, represents the Euclidean distance. The above formula also takes into account the shape size D shape and center distance Dcenter Two factors, among which the shape size includes the length-width product and aspect ratio consistency between the two bounding boxes A and B, that is, the consistency of their size and shape, and normalizes them; the center distance is calculated by dividing the center distance by the circumscribed bounding boxes of the two bounding boxes, which can normalize the distance to 0-1 and avoid the problem that the conventional IoU distance calculation is always 0 when the two bounding boxes do not overlap; α is the proportional weight adjustment factor, for example, it can be set to 0.5, which can be adjusted according to actual needs.

[0062] According to the characteristics of a single target tracking task, we can optimize it based on the Bytetrack framework, use the confidence to weight the trajectory length to select the best target for output, and use the filtered output to initialize the tracking template. At the same time, since changes in the target scale or perspective will affect the tracking effect, the tracking template is updated at regular intervals when there is a stable target. The detection result filtering builds a bridge between target detection and tracking tasks, allowing them to work together and thus improve the overall perception effect.

[0063] Based on the above embodiments, the following can be formed: Figure 3 The dynamic small target perception algorithm framework shown in the figure mainly includes four parts: continuous multi-frame preprocessing, target tracking based on twin network, dual-path fusion multi-scale target detection, and detection result filtering output. Among them, continuous multi-frame preprocessing is to judge the background change situation, so as to adaptively select the frames input to the target detection network. When the background changes and the texture is rich, the sparse optical flow tracking method is used for image alignment to reduce the impact of background changes on target detection and tracking. The target tracking adopts a lightweight tracking algorithm based on the twin network to perform efficient local area target search and matching, and uses the coordinate transformation matrix obtained by image alignment to correct the search area. The tracking result is used as a position prior to guide the target detection network to pay more attention to the potential target area. The two feature extraction branches of the dual-path fusion target detection network use multi-frame motion clues and tracking position priors to improve the extraction ability of infrared small target features under complex backgrounds, and then use the attention mechanism to adaptively fuse dual-path features, and improve the detection ability of targets of different scales through multi-scale feature structures. Finally, through the target filtering method, according to the size, shape and motion trajectory continuity of the target, the false detection target is filtered out, the most likely target is output, and the target template required by the target tracking network is automatically initialized or updated, building a bridge for the coordination of target detection and tracking tasks, and realizing the improvement of target perception performance.

[0064] The above embodiment aims at the needs of sensing dynamic small targets such as drones in complex environments. By converting the target tracking results into a position attention-guided target detection network, the motion clues are converted into multi-channel pixel differences to enhance the saliency of the motion area, without the need for explicit motion information input such as image difference or optical flow that motion clue extraction relies on. A filtering method based on the spatiotemporal continuity of the target is used to establish a bridge between the detection and tracking tasks, and a dual-path fusion multi-scale target detection network is used to achieve simple and efficient detection and tracking coordination and motion clue utilization. The method proposed in this embodiment effectively improves the autonomous perception capability of infrared small targets under complex backgrounds, and provides a feasible solution for the perception of dynamic small targets in complex environments.

[0065] Example

[0066] This embodiment uses the public infrared drone video dataset Anti-UAV410 to verify the effectiveness of the proposed method. Anti-UAV410 covers various scenes in the wild, such as forests, mountains, and lakes. Small targets and dynamic backgrounds with clutter pose challenges to the perception of drones. The experimental computer is equipped with an Intel i7 processor, 32 GB RAM, and an NVIDIA Geforce RTX 3090 GPU. The input image size is set to 640×640, and the target detection network uses the Adam optimizer with a learning momentum of 0.937 and an initial learning rate of 0.01. The model is trained for 120 iterations and the batch size is 16. The evaluation criteria for the processing results use two commonly used indicators: target tracking score SA Score and target detection accuracy AP.

[0067] The motion-enhanced collaborative detection and tracking in this example mainly includes the following steps:

[0068] Step A: Collect multiple frames of infrared image data and perform preprocessing, which includes background change judgment and image registration and alignment to reduce the impact of background changes on target detection and tracking. The frame difference method is used to judge the background change, and the absolute difference between the current frame and the previous frame is calculated. If the proportion of pixels with a difference greater than a certain threshold to all pixels is large, it is considered that the background has changed. When the background changes, the sparse optical flow tracking method is used to achieve fast and efficient image registration and alignment, and the affine transformation matrix is ​​used to provide a more accurate search area for the target tracking network.

[0069] Step B: NanoTrack, a lightweight tracking algorithm based on a twin network, is used to perform efficient local area target search and matching. The target tracking network does not require a manual target template, but automatically sets and updates the template after the target detection network detects a stable target. At the same time, the target tracking result will be used as a position prior to guide the target detection network to focus on the potential target area.

[0070] Step C: The pre-processed continuous multi-frames and target tracking results are input into the dual-path fusion multi-scale target detection network to achieve the fusion of motion information and tracking information. The detection network first uses dual-path feature extraction to enhance the feature extraction capability of infrared small targets under complex backgrounds using motion clues and tracking results respectively, and then realizes adaptive fusion of features through a specially designed attention fusion module, and combines the multi-scale feature structure to improve the detection capability of targets of different scales.

[0071] Step D: Filter the detection results using the spatiotemporal continuity of the target to output the optimal target. Based on the Bytetrack framework, we optimize the single target tracking task based on the characteristics of the task, use the confidence level to weight the trajectory length to select the best target for output, and use the filtered output to initialize or update the tracking template. This builds a bridge between target detection and tracking tasks, allowing them to work together and improve the overall perception effect.

[0072] Repeat the above steps until the task is completed.

[0073] Table 1. Ablation experiment results

[0074]

[0075] Figure 6 The comparison diagram of the detection and tracking cooperation results according to the embodiment of the present invention is schematically shown, and Table 1 is the ablation experiment results. Figure 6 As shown in Table 1. It can be seen that due to the small size of the drone target and the influence of factors such as thermal crossover and rapid movement, the target detection effect is limited, and local tracking is also difficult to work. Once the tracking error occurs, it is easy to fall into the local optimum, so the effect is very poor. Since most drone targets are moving, motion guidance plays a greater role and can significantly improve the accuracy of target detection. Target filtering can select the detection target that is most likely to be the real target according to the spatiotemporal continuity of the target. Collaborative target detection and tracking can simultaneously improve the effect of target detection and tracking, and realize the collaborative promotion of target detection and tracking. During the operation, the initial position of the target is first obtained through target detection, and then the target position is continuously tracked through target tracking, providing position guidance for the target detection network, and improving the detection effect under occlusion, thermal crossover and other conditions. This verifies the effectiveness of the proposed target tracking framework, and at the same time shows that even a local tracker with low accuracy itself is still effective for collaborative tasks. The method of the present invention can provide support for drone countermeasures, and does not require manual setting of the initial position of the target, which meets the needs of practical applications.

[0076] Based on the dynamic small target sensing method of the above embodiment, the present invention also provides a corresponding dynamic small target sensing device. The dynamic small target sensing device has the same or similar technical features and beneficial effects as the dynamic small target sensing method, which will not be described in detail here.

[0077] Figure 7 A block diagram of a dynamic small target sensing device according to an embodiment of the present invention is schematically shown.

[0078] like Figure 7 As shown, the dynamic small target perception device 700 may include an acquisition module 710 and a detection module 720 .

[0079] The acquisition module 710 can be used to acquire continuous multi-frame grayscale images of a dynamic small target and tracking information of the dynamic small target, wherein the continuous multi-frame grayscale images contain motion clues of the dynamic small target. In some embodiments, the acquisition module 710 can be used to perform the above step S110, which will not be repeated here.

[0080] The detection module 720 can be used to extract and fuse features of a dynamic small target using a target detection network based on continuous multi-frame grayscale images and tracking information to obtain a detection result of a dynamic small target, wherein the motion clues and tracking information in the continuous multi-frame grayscale images are used to enhance the feature extraction capability of the target detection network for dynamic small targets. In some embodiments, the detection module 720 can be used to perform the above step S120, which will not be repeated here.

[0081] In some embodiments, the detection module 720 can also be used to assign multiple consecutive grayscale frames to different color channels, so that motion clues are converted into pixel differences in different color channels of multiple consecutive grayscale frames, and the pixel differences are used to enhance the target detection network's perception of dynamic small targets; the tracking information is converted into spatial attention weights, and the spatial attention weights are used to guide the target detection network to focus on the potential target area of ​​dynamic small targets; based on pixel differences and spatial attention weights, the target detection network is used to extract and fuse features of dynamic small targets.

[0082] In some embodiments, the detection module 720 can also be used to convert the tracking information into a mask, where the potential target area on the mask has a first value, the area outside the potential target area has a second value, and the first value is not equal to the second value; based on the first value and the second value, the mask is converted into a spatial attention weight.

[0083] In some embodiments, the detection module 720 can also be used to convert the tracking information into spatial attention weights, and then use the target detection network to extract different layer features of dynamic small targets; perform attention feature fusion processing on different layer features to achieve adaptive fusion of different layer features.

[0084] In some embodiments, the detection module 720 can also be used to use maximum pooling to calculate channel attention weights and spatial attention weights for each layer of features in different layers in turn to obtain weighted features; and use a feature pyramid network and a path aggregation network to perform multi-scale feature fusion on the weighted features.

[0085] In some embodiments, the acquisition module 710 can also be used to preprocess the collected continuous multiple frames of grayscale images to reduce the impact of background changes on target detection and tracking; and determine tracking information from the preprocessed continuous multiple frames of grayscale images.

[0086] In some embodiments, the acquisition module 710 may also be used to determine background changes in a plurality of consecutive grayscale image frames, and to perform registration and alignment on the plurality of consecutive grayscale image frames when background changes occur.

[0087] In some embodiments, the acquisition module 710 can also be used to adopt a lightweight tracking algorithm based on a twin network to determine tracking information from a current frame image of a preprocessed continuous multi-frame grayscale image through a target tracking network.

[0088] In some embodiments, the dynamic small target sensing device 700 may include a filtering module. The filtering module may be used to filter the detection result to output the perception result of the dynamic small target, and automatically initialize or update the template required for target tracking.

[0089] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A dynamic small target perception method based on detection and tracking collaboration and motion cue enhancement, characterized in that: include: Acquire a continuous multi-frame grayscale image of a small dynamic target and tracking information of the small dynamic target, wherein the continuous multi-frame grayscale image contains motion clues of the small dynamic target; Based on the continuous multi-frame grayscale images and the tracking information, a target detection network is used to perform feature extraction and feature fusion on the dynamic small target to obtain the detection result of the dynamic small target, wherein the motion clues and the tracking information in the continuous multi-frame grayscale images are used to enhance the feature extraction capability of the target detection network for the dynamic small target.

2. The dynamic small target perception method according to claim 1, characterized in that: The method of extracting and fusing features of the dynamic small target using a target detection network based on the continuous multiple frames of grayscale images and the tracking information includes: Assigning the continuous multiple frames of grayscale images to different color channels, so that the motion clues are converted into pixel differences of the continuous multiple frames of grayscale images in the different color channels, and the pixel differences are used to enhance the perception ability of the target detection network to the dynamic small target; Converting the tracking information into spatial attention weights, where the spatial attention weights are used to guide the target detection network to focus on the potential target area of ​​the dynamic small target; Based on the pixel difference and the spatial attention weight, the target detection network is used to perform feature extraction and feature fusion on the dynamic small target.

3. The dynamic small target perception method according to claim 2, characterized in that: The converting the tracking information into a spatial attention weight comprises: Converting the tracking information into a mask, wherein the potential target area on the mask has a first value, and an area outside the potential target area has a second value, and the first value is not equal to the second value; The mask is converted into the spatial attention weight according to the first value and the second value.

4. The dynamic small target perception method according to claim 2, characterized in that: The method of extracting and fusing features of the dynamic small target using the target detection network based on the pixel difference and the spatial attention weight includes: After converting the tracking information into spatial attention weights, the target detection network is used to extract different layer features of the dynamic small target; Attention feature fusion processing is performed on the features of the different layers to achieve adaptive fusion of the features of the different layers.

5. The dynamic small target perception method according to claim 4, characterized in that: The performing attention feature fusion processing on the features of different layers includes: For each layer of features of the different layers, the channel attention weight and the spatial attention weight are calculated by using maximum pooling in turn to obtain weighted features; A feature pyramid network and a path aggregation network are used to perform multi-scale feature fusion on the weighted features.

6. The dynamic small target perception method according to claim 1, characterized in that: The step of acquiring a continuous multi-frame grayscale image of a dynamic small target and tracking information of the dynamic small target includes: Preprocessing the collected continuous multiple frames of grayscale images to reduce the impact of background changes on target detection and tracking; The tracking information is determined from the preprocessed continuous multiple frames of grayscale images.

7. The dynamic small target perception method according to claim 6, characterized in that: The preprocessing of the collected continuous multiple frames of grayscale images includes: The background change in the continuous multiple frames of grayscale images is determined, and when the background change occurs, the continuous multiple frames of grayscale images are registered and aligned.

8. The dynamic small target perception method according to claim 6, characterized in that: Determining the tracking information from the preprocessed continuous multiple frames of grayscale images includes: A lightweight tracking algorithm based on a twin network is adopted, and the tracking information is determined from the current frame image in the preprocessed continuous multi-frame grayscale image through a target tracking network.

9. The dynamic small target perception method according to claim 1 or 6, characterized in that: After obtaining the detection result of the dynamic small target, the dynamic small target perception method further includes: The detection result is filtered to output the perception result of the dynamic small target, and the template required for target tracking is automatically initialized or updated.

10. A dynamic small target perception device based on detection and tracking coordination and motion clue enhancement, characterized in that: include: An acquisition module, used for acquiring a continuous multi-frame grayscale image of a small dynamic target and tracking information of the small dynamic target, wherein the continuous multi-frame grayscale image contains motion clues of the small dynamic target; A detection module is used to perform feature extraction and feature fusion on the dynamic small target using a target detection network based on the continuous multi-frame grayscale images and the tracking information to obtain a detection result of the dynamic small target, wherein the motion clues and the tracking information in the continuous multi-frame grayscale images are used to enhance the feature extraction capability of the target detection network for the dynamic small target.

Images