An infrared small target tracking method based on dynamic convolution kernel
The infrared small target tracking method using dynamic convolution kernels extracts infrared small target features using multi-layer dynamic convolution modules, solving the tracking problem of infrared small targets in complex backgrounds and achieving higher tracking accuracy and adaptability.
Patent Information
- Application Number
- CN202310828952.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-07
- Publication Date
- 2026-01-02
- Estimated Expiration
- 2043-07-07
AI Technical Summary
Existing infrared small target tracking algorithms suffer from performance degradation in complex backgrounds and lack universality, especially when the infrared image signal-to-noise ratio is low, making it difficult to effectively extract and track weak targets.
A tracking method based on dynamic convolution kernels is adopted. By mining the features of the template and search region through multi-layer dynamic convolution modules, dynamic convolution kernels are generated for the template using dynamic convolution modules to extract strong features and adapt to weak targets in complex backgrounds.
It improves the accuracy and adaptability of infrared small target tracking, enhances feature representation capabilities, and improves tracking performance in complex backgrounds.
Smart Images
Figure CN116805327B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to infrared target tracking technology, in particular to an infrared small target tracking method based on a dynamic convolution kernel. BACKGROUND
[0002] Infrared small target tracking is a basic but challenging research task in the field of computer vision, and the purpose is to select the target of interest in the initial frame of the video and predict the position and shape of the target in the subsequent frame. Due to the advantages of long action distance, good adaptability to weather and background, strong anti-interference ability, and all-weather work of the infrared imaging system, infrared target tracking plays an important role in security warning, military and other fields.
[0003] The infrared target is often in a complex environment such as the ground background, the ground-air interface background, and the sky background. When the environment of the target is complex, the infrared small target is difficult to identify, and similar interference is easy to occur. Moreover, the infrared image has low signal-to-noise ratio, and the target is often submerged in a large amount of noise, which is easy to cause tracking drift.
[0004] Traditional infrared target tracking algorithms mainly include image filtering-based algorithms, feature matching-based algorithms, and target and background classification-based algorithms. When the infrared image has low signal-to-noise ratio, the background is complex, and there are similar target interference (false alarm), the performance of the algorithm will decrease significantly.
[0005] The existing methods enhance the local contrast in the pre-processing stage, such as using Gaussian curvature filtering, IHBF, LOG filtering and other methods to enhance the contrast between the target and the background, so as to extract strong features for tracking. However, they are suitable for the application of a specific type of infrared small target, and lack universality. The neural network-based method can obtain deep features suitable for multiple complex scenes by training a large number of samples. However, the current method mainly uses the backbone network (such as the residual connection network ResNet) in the image field to extract the features of the infrared small target, and lacks adjustment for the characteristics of the infrared small target problem. SUMMARY
[0006] The technical problem to be solved by the present application is to provide an infrared small target tracking method based on a dynamic convolution kernel to reduce the difficulty of extracting infrared small targets in view of the deficiencies of the prior art.
[0007] To solve the above technical problems, the technical solution adopted by the present application is: an infrared small target tracking method based on a dynamic convolution kernel, comprising the following steps:
[0008] S1, acquiring an infrared video frame sequence containing a small target;
[0009] S2, select a target in a first image in an infrared video frame sequence, expand N times of the side length from the center of the target, and reshape into a square as a template, wherein the template comprises appearance information of the target and a local surrounding scene of the target; expand 2N times of the side length from the center coordinate of the target in the i-th image in the infrared video frame sequence, and reshape into a square as a search area; i>1;
[0010] S3, input the template and the search area into a tracking network based on a dynamic convolution module to perform matching, and output the target in a single frame image.
[0011] S4, expand 2N times of the side length from the center coordinate of the target in the single frame image, and reshape into a square as a new search area; input the template and the new search area into the tracking network based on the dynamic convolution module to perform matching, and output the target in the single frame image, until all frames have been tracked.
[0012] In view of the difficulty in extracting features of an infrared small target, the tracking method based on the dynamic convolution provided by the application uses a plurality of dynamic convolution modules to mine more key information in template and search area features, and reduces the difficulty in extracting features of the infrared small target.
[0013] The tracking network based on the dynamic convolution module comprises at least one dynamic convolution module connected in series, and all the dynamic convolution modules are connected with a backbone network of a Siamese tracker.
[0014] The application introduces the dynamic convolution into the field of infrared small target tracking, and proposes a method of mapping template features into dynamic convolution kernels, so that the feature expression capability is effectively improved.
[0015] The number of the dynamic convolution modules is greater than or equal to 1.
[0016] The dynamic convolution module comprises a convolution unit formed by cascading a first convolution layer and a plurality of second convolution layers; the input of the first convolution layer is a template feature, and the input of the convolution unit is a search area feature; the features output by the first convolution layer and the convolution unit are cascaded and input into a third convolution layer; the third convolution layer maps the cascaded features into a dynamic convolution kernel, the dynamic convolution kernel is subjected to a convolution operation with the search area feature to obtain a response map; and the response map and the search area feature are cascaded to obtain the output of the dynamic convolution module.
[0017] The Siamese tracker comprises a backbone network and a similarity calculation part; the backbone network is used for extracting features of a template and a search area, and the similarity calculation part obtains a sub-area similar to the template in the search area, and the sub-area is an output target.
[0018] The backbone network comprises two feature extraction modules for extracting features of the template and the search area respectively; the feature extraction module is a ResNet50 network, and a fourth stage of the ResNet50 network is a final output.
[0019] A convolution stride of a fourth stage downsampling unit of the ResNet50 network is 1, so as to obtain a larger feature resolution.
[0020] A 3*3 convolution of the fourth stage is replaced by a dilated convolution with a stride of 2, so as to increase a receptive field.
[0021] A sub-area similar to the template in the search area is obtained through cross-correlation calculation.
[0022] Compared with the prior art, the present application has the beneficial effects that:
[0023] 1. The present application introduces dynamic convolution into the field of infrared small target tracking, effectively improving the feature expression ability.
[0024] 2. The present application uses a multi-layer dynamic convolution module to mine more critical information in the features of the template and the search area, solving the problem of difficulty in extracting features of infrared small targets. DETAILED DESCRIPTION
[0025] Figure 1 A tracking network structure diagram based on a dynamic convolution kernel of an embodiment of the present application;
[0026] Figure 2 A process of dynamic convolution kernel convolution of an embodiment of the present application;
[0027] Figure 3 A dynamic convolution module schematic diagram of an embodiment of the present application;
[0028] Figure 4 A Siamese tracker structure diagram of an embodiment of the present application. DETAILED DESCRIPTION
[0029] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.
[0030] In this document, the terms "first," "second," and other similar words are not intended to imply any order, quantity, or importance, but are merely used to distinguish different elements. The terms "one," "a," and other similar words are not intended to indicate the existence of only one of the stated things, but rather that the description refers only to one of the stated things, which may have one or more. The terms "comprising," "including," and other similar words are intended to indicate a logical relationship, not a spatial relationship. For example, "A includes B" means that logically B belongs to A, not that spatially B is located inside A. Furthermore, the meanings of the terms "comprising," "including," and other similar words should be considered open-ended, not closed. For example, "A includes B" means that B belongs to A, but B does not necessarily constitute all of A; A may also include other elements such as C, D, and E.
[0031] This invention provides an infrared small target tracking method based on dynamic convolution kernels, specifically including the following steps:
[0032] Step 1: Input an infrared video sequence containing small targets;
[0033] Step 2: Given the ground truth of the target of interest in the first frame image (which includes coordinate values and the length and width of the image) or select the target in the first frame image, expand the target by twice the side length from the center of the target and reshape it into a square as a template, which includes the appearance information of the target and its local surrounding scene;
[0034] Step 3: The second frame image is expanded by four times the side length with respect to the center coordinates of the target and reshaped into a square as the search area. The search area usually covers the possible range of movement of the target.
[0035] Step 4: Input the template and search region into the tracking network based on dynamic convolution kernels for matching, and output the target in a single frame image;
[0036] Step 5: Expand the target center coordinates obtained from the previous step by four times the side length and reshape it into a square as a new search region. Input the template and the new search region into a tracking network based on dynamic convolution kernels for matching, and output the target in a single frame image.
[0037] For the remaining frames, repeat steps three through five above until all frames have been tracked.
[0038] The tracking network based on the dynamic convolution kernel designed in the embodiment of the application comprises one or more dynamic convolution modules of the same structure but not sharing parameters and a Siamese tracker. When an infrared video sequence containing a small target is input, a template image containing the target and a search area image are determined, and then the images are input into the tracking network based on the dynamic convolution kernel for tracking. The images are first input into the backbone network part of the Siamese tracker, and the search area features and the template features are output; each dynamic convolution module takes the template features and the search area features processed in the previous step as input and takes the new search area features processed as output. The dynamic convolution module can mine more valuable parts in the template and search area features and extract more effective features. After the effective features are extracted through one or more dynamic convolution modules, the template features and the search area features are input into the similarity calculation part of the Siamese tracker for tracking.
[0039] The process of using the dynamic convolution kernel for convolution in the embodiment of the application can be represented as:
[0040]
[0041] wherein K represents a convolution kernel with a size of k*k. Iin and Iout represent input and output images respectively. i and j are coordinates in the image, and u and v are coordinates in each Ki,j. These pixel-by-pixel convolution kernels perform a weighted summation operation on the nearby images.
[0042] The dynamic convolution module designed in the embodiment of the application is as shown in Figure 3 First, the template features are reduced in dimension through a 1*1 convolution kernel, and the search area features are further extracted through convolution operation of 3*3, 2*2 and 3*3 convolution kernels; then, the two parts of features are concatenated and mapped into a dynamic convolution kernel through a 3*3 convolution kernel, which is conducive to mining the parts that need to be paid attention to in the search area from the template. The search area and the response map predicted by the dynamic convolution kernel are concatenated, and the size is adjusted to obtain a new search area. The dynamic convolution module can be stacked to enhance the feature extraction capability and improve the tracking effect.
[0043] The Siamese tracker adopted in the embodiment of the application comprises a backbone network and a similarity calculation part. The backbone network extracts the features of the template and the search area, the similarity calculation part obtains a sub-region similar to the template in the search area, and the similarity estimation is obtained by calculating the cross-correlation.
[0044] The embodiment of the present application takes the template image and the search region image as the input of the backbone network. The embodiment of the present application uses a modified version of ResNet50 for feature extraction. The first stage of ResNet50 is composed of a 7*7 convolution layer with a stride of 2, plus a 3*3 maximum pooling layer with a stride of 2, and the second, third and fourth stages are composed of several three-layer bottleneck blocks. The convolution stride of the down-sampling unit in the fourth stage is changed from 2 to 1 to obtain a larger feature resolution. The 3*3 convolution in the fourth stage is modified to a dilated convolution with a stride of 2 to increase the receptive field. The last stage of ResNet50 is removed in the embodiment of the present application, and the output of the fourth stage is taken as the final output.
[0045] In the embodiment of the present application, the similarity calculation is realized through a cross-correlation operation, that is, the template features are taken as the convolution kernel, and the search region is subjected to a convolution operation to calculate the similarity of each position to obtain a score map, wherein the position with the highest score corresponds to the target position.
[0046] Embodiment 2
[0047] The embodiment 2 of the present application provides a terminal device corresponding to the above-mentioned embodiment 1. The terminal device can be a processing device for a client, such as a mobile phone, a notebook computer, a tablet computer, a desktop computer, etc., to execute the method of the above-mentioned embodiment.
[0048] The terminal device of the embodiment includes a memory, a processor and a computer program stored in the memory; the processor executes the computer program in the memory to realize the steps of the method of the above-mentioned embodiment 1.
[0049] In some implementations, the memory can be a high-speed random access memory (RAM), and can also include a non-volatile memory, such as at least one disk memory.
[0050] In other implementations, the processor can be a central processing unit (CPU), a digital signal processor (DSP) or various types of general-purpose processors, which are not limited here.
[0051] Embodiment 3
[0052] The embodiment 3 of the present application provides a computer readable storage medium corresponding to the above-mentioned embodiment 1, which stores a computer program / instruction. When the computer program / instruction is executed by the processor, the steps of the method of the above-mentioned embodiment 1 are realized.
[0053] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof.
[0054] It is apparent that a person having ordinary skill in the art can be able to conceive variations and modifications of the present application after the essential characteristics of the present application are ascertained. The scope of the present application should, therefore, be construed as not limited to the preferred embodiments set forth herein but to encompass all such changes and modifications as fall within the scope of the present application.
[0055] The present application is described with reference to the accompanying drawings, which show embodiments of the present application. The drawings described are intended to be illustrative, and not restrictive. Several embodiments are described below with reference to the accompanying drawings. The embodiments are intended to be illustrative, and not restrictive. The present application is described with reference to the accompanying drawings, which show embodiments of the present application. The drawings described are intended to be illustrative, and not restrictive. Several embodiments are described below with reference to the accompanying drawings. The embodiments are intended to be illustrative, and not restrictive. Figure 1 one or more processes and / or blocks Figure 1 means for performing the functions specified in the one or more processes and / or blocks.
[0056] These computer program instructions can also be loaded into a computer or other programmable data processing devices to cause one or more processes and / or functions specified in the flow diagrams and / or block diagrams to be executed on the computer or other programmable data processing devices. These computer program instructions can also be stored in a computer readable storage medium that can guide the computer or other programmable data processing devices to operate in a specific manner, so that the instructions stored in the computer readable storage medium produce the computer implemented process when the instructions are executed on the computer or other programmable data processing devices. Figure 1 one or more processes and / or blocks Figure 1 one or more processes and / or blocks.
[0057] Although the preferred embodiments of the present application have been described, those skilled in the art will be able to make additional changes and modifications to these embodiments once they grasp the fundamental inventive concept. Accordingly, it is intended to include all such changes and modifications in the scope of the present application as set forth in the claims.
[0058] Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.
Claims
1. A method for tracking small infrared targets based on dynamic convolution kernels, characterized in that, Includes the following steps: S1. Obtain an infrared video frame sequence containing small targets; S2. Select the target in the first frame of the infrared video frame sequence, expand the target center by N times the side length, and reshape it into a square as a template. The template includes the appearance information of the target and the local surrounding scene of the target. Expand the i-th frame of the infrared video frame sequence by 2N times the side length with the target center coordinates as the center, and reshape it into a square as the search area; i > 1. S3. Input the template and search region into the tracking network based on the dynamic convolution module for matching, and output the target in a single frame image; S4. Expand the target center coordinates in the single frame image by 2N times the side length and reshape it into a square as a new search area. Input the template and the new search area into the tracking network based on the dynamic convolution module for matching, and output the target in the single frame image until all frames have been tracked. The tracking network based on dynamic convolution modules includes at least one cascaded dynamic convolution module, and all dynamic convolution modules are connected to the backbone network of the Siamese tracker. The dynamic convolution module includes a convolutional unit consisting of a first convolutional layer and multiple cascaded second convolutional layers. The input to the first convolutional layer is template features, and the input to the convolutional unit is search region features. The features output by the first convolutional layer and the convolutional unit are concatenated and then input to a third convolutional layer. The third convolutional layer maps the concatenated features to a dynamic convolutional kernel. The dynamic convolutional kernel performs a convolution operation with the search region features to obtain a response map. The response map is concatenated with the search region features to obtain the output of the dynamic convolution module. The Siamese tracker includes a backbone network and a similarity calculation part; the backbone network is used to extract features of the template and the search region, and the similarity calculation part obtains a sub-region similar to the template within the search region, which is the output target.
2. The infrared small target tracking method based on dynamic convolution kernel according to claim 1, characterized in that, The number of dynamic convolutional modules is greater than or equal to 1.
3. The infrared small target tracking method based on dynamic convolution kernel according to claim 1, characterized in that, The backbone network includes two feature extraction modules with shared parameters, which are used to extract features of the template and features of the search region, respectively; the feature extraction module is a ResNet50 network, and the fourth stage of the ResNet50 network is the final output.
4. The infrared small target tracking method based on dynamic convolution kernel according to claim 3, characterized in that, The convolution stride of the fourth-stage downsampling unit in the ResNet50 network is 1.
5. The infrared small target tracking method based on dynamic convolution kernel according to claim 3, characterized in that, The 3×3 convolutions in the fourth stage of the ResNet50 network are replaced with dilated convolutions with a stride of 2.
6. The infrared small target tracking method based on dynamic convolution kernel according to claim 3, characterized in that, The sub-regions similar to the template within the search area are obtained by cross-correlation calculation.