Infrared target tracking method, device, equipment and storage medium

Abstract

The disclosed embodiments relate to an infrared target tracking method, device, equipment and storage medium. The method comprises: determining a search image block in a target infrared image frame, and inputting it into a feature extraction model to determine a search feature; the feature extraction model comprises a visual transformer unit and a lightweight feature extraction unit with a bottleneck structure; the model parameters of the visual transformer unit are obtained by pre-training based on a visible light image sample set, and its model parameters are frozen during the model training process; the model parameters of the feature extraction unit are obtained by training based on an infrared image sample set during the training process of the feature extraction model; target tracking processing is performed based on the template features of the template image of the target object to be tracked and the search features, and the target position information of the target object in the target infrared image frame is determined. In this way, the cost and time consumption of model training are greatly reduced, and the accuracy of feature extraction of infrared images is improved, thereby improving the accuracy of infrared target tracking.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an infrared target tracking method, device, equipment and storage medium. Background Art

[0002] Infrared target tracking refers to the process of continuously tracking the target object using thermal infrared data obtained by infrared sensors. The general process is to obtain the position information of the target object to be tracked in the first infrared image frame, then predict the position of the target object in subsequent infrared image frames by modeling the target object and combining it with video context information.

[0003] At present, the mainstream infrared target tracking solution is implemented through a machine learning model. The machine learning model requires a large number of infrared image samples with the target object position marked for training. In this way, if the number of infrared image samples is insufficient, the model accuracy will be low due to insufficient training of the machine learning model, which will in turn cause the problem of poor accuracy of the target object position in infrared target tracking. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides an infrared target tracking method, device, equipment and storage medium.

[0005] In a first aspect, an embodiment of the present disclosure provides an infrared target tracking method, the method comprising:

[0006] Acquire a target infrared image frame, and determine a search image block in the target infrared image frame;

[0007] The search image block is input into a feature extraction model to determine the search feature; wherein the feature extraction model includes a first encoder, the first encoder includes a visual transformer unit and a lightweight feature extraction unit with a bottleneck structure; the model parameters of the visual transformer unit are obtained by pre-training based on a visible light image sample set, and the model parameters of the visual transformer unit are frozen during the training process of the feature extraction model; the model parameters of the feature extraction unit are obtained by training based on an infrared image sample set during the training process of the feature extraction model;

[0008] Target tracking processing is performed based on the template features of the template image of the target object to be tracked and the search features to determine the target position information of the target object in the target infrared image frame; wherein the template image is an image block containing a reference of the target object; and the template features are obtained by inputting the template image into the feature extraction model.

[0009] In a second aspect, an embodiment of the present disclosure further provides an infrared target tracking device, the device comprising:

[0010] A search image block determination module, used to acquire a target infrared image frame and determine a search image block in the target infrared image frame;

[0011] A search feature determination module is used to input the search image block into a feature extraction model to determine the search feature; wherein the feature extraction model includes a first encoder, the first encoder includes a visual transformer unit and a lightweight feature extraction unit with a bottleneck structure; the model parameters of the visual transformer unit are obtained by pre-training based on a visible light image sample set, and the model parameters of the visual transformer unit are frozen during the training process of the feature extraction model; the model parameters of the feature extraction unit are obtained by training based on an infrared image sample set during the training process of the feature extraction model;

[0012] A target position information determination module is used to perform target tracking processing based on the template features of the template image of the target object to be tracked and the search features, and determine the target position information of the target object in the target infrared image frame; wherein the template image is an image block containing a reference of the target object; and the template features are obtained by inputting the template image into the feature extraction model.

[0013] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0014] Processor and memory;

[0015] The processor is used to execute the infrared target tracking method described in any embodiment of the present disclosure by calling the program or instruction stored in the memory.

[0016] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instructions, wherein the program or the instructions enable a computer to execute the infrared target tracking method described in any embodiment of the present disclosure.

[0017] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product, which is used to execute the infrared target tracking method described in any embodiment of the present disclosure.

[0018] The infrared target tracking solution provided by the embodiment of the present disclosure, on the one hand, takes the visual transformer unit obtained by training the visible light image sample set as the basic structure, and introduces the feature extraction unit with a lightweight bottleneck structure to form a feature extraction model suitable for target tracking of infrared images, so that a large amount of image processing knowledge learned in its pre-training process can be inherited by freezing the model parameters of the visual transformer unit; at the same time, the feature extraction unit with fewer model parameters is trained by the infrared image sample set, so that it can focus on learning the infrared image features that are crucial to infrared tracking, so as to achieve the effect of fine-tuning the visual transformer unit, so that it can better adapt to the characteristics of infrared images; the feature extraction model obtained by such training establishes a bridge between visible light images and infrared images, so that the model can smoothly transition between the two images, which not only reduces the demand for infrared image sample sets, greatly reduces the cost and time of model training, but also improves the accuracy of feature extraction of infrared images, thereby improving the accuracy of infrared target tracking. On the other hand, designing the model structure of the feature extraction unit as a lightweight bottleneck structure can limit the amount of newly introduced model parameters by reducing the intermediate dimensions and reduce the amount of feature extraction calculations, thereby improving the calculation speed of infrared target tracking and reducing the demand for computing resources for infrared target tracking. This makes the infrared target tracking algorithm applicable to resource-constrained platforms such as drones or satellite platforms, thereby improving the universality of infrared target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Figure 1 A flowchart of an infrared target tracking method provided by an embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of a bottleneck structure feature extraction unit provided in an embodiment of the present disclosure;

[0023] Figure 3 A structural diagram of a feature extraction model provided in an embodiment of the present disclosure;

[0024] Figure 4 A schematic diagram of the structure of another feature extraction model provided in an embodiment of the present disclosure;

[0025] Figure 5 A structural diagram of another feature extraction model provided in an embodiment of the present disclosure;

[0026] Figure 6 A schematic diagram of the structure of an infrared target tracking device provided by an embodiment of the present disclosure;

[0027] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described in detail below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0030] At present, one solution for infrared target tracking is to use a large number of infrared image sample sets labeled with target objects to train deep learning models. This not only requires a lot of manpower and time to collect a large number of infrared image samples, but also consumes more computing resources and training time due to the large number of model parameters of the deep learning model, resulting in low efficiency in infrared target tracking. In addition, if the number of infrared image samples is insufficient, the training degree of the deep learning model will be insufficient, resulting in low model accuracy and poor accuracy of the position of the target object tracked by infrared.

[0031] Based on the above situation, an embodiment of the present disclosure provides a target infrared tracking solution, which introduces a lightweight feature extraction unit with a bottleneck structure to fine-tune the vision transformer (VisionTransformer, Vit) model obtained by training a visible light image sample set to form a feature extraction model, and train it with a small number of infrared image samples to quickly obtain a feature extraction model suitable for infrared images with high model accuracy. The feature extraction model is then used to perform feature extraction and target tracking on a series of infrared images, thereby improving the accuracy and efficiency of infrared target tracking.

[0032] Figure 1This is a flow chart of an infrared target tracking method provided by an embodiment of the present disclosure. The infrared target tracking method is mainly applicable to scenarios where infrared images are used to track target objects or predict motion trajectories. The infrared target tracking method provided by an embodiment of the present disclosure can be performed by an infrared target tracking device, which can be implemented by software and / or hardware. The device can be integrated in an electronic device with certain computing capabilities, such as a laptop computer, desktop computer, personal workstation, server, drone, or a processor built into a satellite platform.

[0033] See also Figure 1 , the infrared target tracking method specifically includes:

[0034] S 11 0. Acquire a target infrared image frame, and determine a search image block in the target infrared image frame.

[0035] The target infrared image frame is an image frame that needs to be processed for infrared target tracking, and is an image frame other than the first frame in the video collected by the infrared sensor. The search image block is an image block / image area that may contain the target object to be tracked.

[0036] Specifically, the electronic device extracts the first infrared image frame from the video collected by the infrared sensor, or reads the first infrared image frame from the storage space storing the data collected by the infrared sensor. Then, the first infrared image frame is subjected to target object recognition processing, such as manually marking the target object, or detecting the target object using a target detection algorithm, etc., to obtain the image block where the target object is located in the first infrared image frame as the initial template image for target object tracking. The template image will be used as reference information for target object detection in subsequent infrared image frames. Afterwards, the processing of S110 can be performed on any subsequent infrared image frame.

[0037] In specific implementation, the electronic device obtains a target infrared image frame from the infrared sensor or the storage space, and then determines a search image block from the target infrared image frame for detecting the target object.

[0038] In some embodiments, if the computing resources of the electronic device are sufficient, the entire image of the target infrared image frame may be directly used as the search image block.

[0039] In other embodiments, if the computing resources of the electronic device are limited, then in order to reduce the amount of computing of subsequent image feature processing, the electronic device may extract an image block of a partial area from the target infrared image frame as a search image block. That is, the area range of the search image block is smaller than the area range of the entire frame image of the target infrared image frame. In this way, determining the search image block in the target infrared image frame in S110 includes: determining the search image block from the target infrared image frame based on a template image containing the target object.

[0040] Specifically, considering that the template image is a reference image block containing the target object and the position movement of the target object in several consecutive infrared image frames will not be too large, in this embodiment, the search image block can be determined from the target infrared image frame based on the regional position of the template image in the infrared image frame.

[0041] In one example, the electronic device can determine the area corresponding to the template image from the target infrared image frame, then appropriately expand the area to a certain range, and use the image block covered by the expanded area as the search image block to ensure that the target object can still be captured when it moves or deforms slightly.

[0042] In another example, the electronic device can predict the possible position of the target object in the target infrared image frame based on the moving speed and direction of the target object between consecutive historical infrared image frames using a template image, and determine the size and position of the area to search for the target object based on the possible position, and use the image block at that position and within the area as the search image block.

[0043] The above-mentioned historical infrared image frame is an infrared image frame obtained before the target infrared image frame, and is before the target infrared image frame in terms of image acquisition time.

[0044] S120. Input the search image block into a feature extraction model to determine the search features; the feature extraction model includes a first encoder, the first encoder includes a visual transformer unit and a lightweight feature extraction unit with a bottleneck structure; the model parameters of the visual transformer unit are obtained by pre-training based on a visible light image sample set, and the model parameters of the visual transformer unit are frozen during the training process of the feature extraction model; the model parameters of the feature extraction unit are obtained by training based on an infrared image sample set during the training process of the feature extraction model.

[0045] The search feature is an image feature extracted from the search image block. The feature extraction model is a deep learning model pre-trained in the embodiment of the present disclosure for extracting image features from infrared images. The visual transformer unit is composed of the main part of the Vit model, which at least includes an image segmentation module, a position encoding module and a transformer encoder based on a self-attention mechanism.

[0046] The bottleneck structure is a model structure in which the number of channels in the middle layer is less than the number of channels in the layers before and after it. Lightweight means that the model has few parameters. The feature extraction unit is a machine learning model with feature extraction function. For example, see Figure 2 , the feature extraction unit in the embodiment of the present disclosure includes a first fully connected layer 210, a nonlinear activation layer 220, and a second connected layer 230 connected in series. The first fully connected layer 210 performs downsampling to reduce the dimension, and the second fully connected layer 230 performs upsampling to increase the dimension, so that the dimension / channel number of the nonlinear activation layer 220 in the middle can be reduced, thereby reducing the number of parameters in the feature extraction unit and reducing the amount of calculation in the feature extraction process to a certain extent.

[0047] The visible light image sample set refers to a set of training samples consisting of visible light images, which are compatible with the input requirements of the visual transformer unit. The infrared image sample set refers to a set of training samples consisting of infrared images, which are at least annotated with the target object specified in the infrared target tracking task.

[0048] Specifically, according to the above description, infrared target tracking in the related art requires retraining of the deep learning model, which has high training cost and takes a long time. Therefore, the embodiment of the present disclosure introduces the visual transformer Vit model obtained by pre-training of the visible light image sample set, and uses its main structure as the main structure of the feature extraction model, that is, the visual transformer unit, and introduces a feature extraction unit with a bottleneck structure design. Then, considering that the pre-trained Vit model has learned a lot of image processing knowledge, and this image processing knowledge can be compatible with the image processing of infrared images to a certain extent. Therefore, in order to reduce the amount of training, the embodiment of the present disclosure freezes the trained model parameters of the visual transformer unit during the training process of the feature extraction model to inherit the above-mentioned image processing knowledge. At the same time, in view of the difference in image representation between infrared images and visible light images, for example, compared with visible light images, infrared images lack rich textures, colors, and fine details, etc., in order to improve the applicability of the feature extraction model to infrared images, a small number of infrared image sample sets are used to fine-tune the model parameters in the additional feature extraction unit, so as to capture the unique features of the infrared image that are crucial to infrared target tracking, such as hot spots and temperature changes, etc., through the small number of model parameters additionally introduced in the feature extraction unit. The feature extraction model obtained in this way not only has low training cost and fast training speed, but also is more suitable for infrared images on the basis of ensuring model accuracy, thereby improving its applicability to infrared target tracking.

[0049] In the actual infrared target tracking process, the electronic device inputs the obtained search image block into the feature extraction model obtained above, and after being processed by the model, the search feature can be output.

[0050] It should be noted that the specific model structure of the feature extraction model can be found in the detailed description of the subsequent embodiments.

[0051] S130 , performing target tracking processing based on the template features and the search features of the template image of the target object to be tracked, and determining the target position information of the target object in the target infrared image frame.

[0052] The template image is a reference image block containing the target object. The template image can be an initial template image determined from the first infrared image frame, or a template image updated in a subsequent tracking process. The template feature is an image feature extracted from the template image, which can be obtained by inputting the template image into a feature extraction model. The target position information is information about the position of the target object in the target infrared image frame, for example, the area position of the bounding rectangle of the target object in the image, or the position of the outer edge of the target object in the image, etc.

[0053] Specifically, after obtaining the template image, the electronic device can input it into the above-mentioned feature extraction model to obtain the corresponding template features. The template features can include texture features, shape features, heat distribution features, etc. related to the target object, which are reference information for tracking the target object. Then, after obtaining the search features, the electronic device can match / identify the search features according to the template features to determine the target position information where the target object is located in the search image block.

[0054] In one example, the electronic device may perform sliding window feature matching on the template feature and the search feature, and determine the region position of the image region within a sliding window range with the best matching result as the target position information.

[0055] In another example, the electronic device may input both the template features and the search features into a pre-trained machine learning model with a target tracking function, and after being processed by the machine learning model, output the area location of the target object in the search image block as the target location information.

[0056] The target infrared tracking method of each of the above embodiments of the present disclosure can obtain a target infrared image frame and determine a search image block therein; input the search image block into a feature extraction model to determine a search feature; the feature extraction model includes a first encoder, and the first encoder includes a visual transformer unit and a lightweight feature extraction unit with a bottleneck structure; the model parameters of the visual transformer unit are obtained by pre-training based on a visible light image sample set, and the model parameters of the visual transformer unit are frozen during the training process of the feature extraction model; the model parameters of the feature extraction unit are obtained by training based on the infrared image sample set during the training process of the feature extraction model; target tracking processing is performed based on the template features obtained by inputting the template image of the target object to be tracked into the feature extraction model and the search features to determine the target position information of the target object in the target infrared image frame; on the one hand, the visual transformer unit obtained by training with the visible light image sample set is used to obtain the target position information of the target object in the target infrared image frame; on the other hand, the visual transformer unit obtained by training with the visible light image sample set is used to obtain the target position information of the target object in the target infrared image frame; on the other hand, the visual transformer unit obtained by pre-training ... model parameters of the visual transformer unit. The visual transformer unit is used as the basic structure, and a lightweight bottleneck structure feature extraction unit is introduced to form a feature extraction model suitable for target tracking of infrared images. In this way, a large amount of image processing knowledge learned in the pre-training process can be inherited by freezing the model parameters of the visual transformer unit; at the same time, the feature extraction unit with fewer model parameters is trained through the infrared image sample set, so that it can focus on learning the infrared image features that are crucial to infrared tracking, so as to achieve the fine-tuning effect of the visual transformer unit, so that it can better adapt to the characteristics of infrared images; the feature extraction model trained in this way establishes a bridge between visible light images and infrared images, so that the model can smoothly transition between the two images, which not only reduces the demand for infrared image sample sets, greatly reduces the cost and time of model training, but also improves the accuracy of feature extraction of infrared images, thereby improving the accuracy of infrared target tracking. On the other hand, designing the model structure of the feature extraction unit as a lightweight bottleneck structure can limit the amount of newly introduced model parameters by reducing the intermediate dimensions and reduce the amount of feature extraction calculations, thereby improving the calculation speed of infrared target tracking and reducing the demand for computing resources for infrared target tracking. This makes the infrared target tracking algorithm applicable to resource-constrained platforms such as drones or satellite platforms, thereby improving the universality of infrared target tracking.

[0057] In some embodiments, based on the above technical solution, the infrared target tracking method also includes: if the preset template update condition is met, based on the historical position information of the target object in the previous adjacent infrared image frame, a historical feature point set is extracted from the previous adjacent infrared image frame; based on the historical feature point set, feature point matching is performed on the target infrared image frame to determine the matching feature point set of the target object in the target infrared image frame; and the template image is updated based on the matching feature point set.

[0058] The template update condition is a pre-set condition that triggers the execution of the image template update. In one example, the template update condition may be the number of frames of the image that has been subjected to the infrared target tracking process, such as one frame or several frames. In another example, the template update condition may be a critical value (i.e., a preset difference threshold) of the difference between the position information of the target object obtained from two adjacent or multiple adjacent infrared image frames.

[0059] The previous adjacent infrared image frame is an image frame immediately before the target infrared image frame in order of acquisition time. The historical position information is the position information of the target object obtained by tracking in the historical infrared image frame. The historical feature point set is a set of feature points extracted from the image area where the target object is located in the historical infrared image frame.

[0060] Specifically, the electronic device monitors the variables corresponding to the template update condition. If it is determined that the template update condition is met, then based on the historical position information of the target object in the previous adjacent infrared image frame, feature points are extracted from the previous adjacent infrared image frame to obtain a historical feature point set. Then, the target infrared image frame is subjected to feature point search and matching processing using the historical feature point set to obtain a feature point set in the target infrared image frame that matches the historical feature point set as a matching feature point set. Based on the assumption that the historical position information of the target object in the previous adjacent infrared image frame is accurate, the matching feature point set can be considered to be the accurate regional position of the target object in the target infrared image. At this time, the image block within the regional range corresponding to the matching feature point set can be used as a new template image to complete the update of the template image.

[0061] In some embodiments, on the basis of the template update condition being a preset difference threshold, the above-mentioned target update condition includes: the regional difference between the target position information in the target infrared image frame and the historical position information in the previous adjacent infrared image frame is greater than the preset difference threshold. At this time, the above-mentioned template update processing can be triggered. In this way, the redundant calculation loss caused by frequent updates of the template image can be reduced to a certain extent, and the template image can be updated more timely on demand, thereby improving the accuracy of the template image, and further improving the accuracy of infrared target tracking.

[0062] In other embodiments, on the basis of the template update condition being a preset difference threshold, the above-mentioned target update condition is reached including: the regional differences between a continuous preset number of adjacent historical position information are all greater than the preset difference threshold. For example, when the electronic device determines that the regional difference between the target position information in the target infrared image frame and the historical position information in the previous adjacent infrared image frame is greater than the preset difference threshold, it continues to judge whether the regional difference between the historical position information of the two adjacent infrared image frames before the target infrared image frame is greater than the preset difference threshold, and successively calculates whether the regional difference between the two historical position information is greater than the preset difference threshold until the number of judgments reaches the preset number. Moreover, when the preset number of regional differences are all greater than the preset difference threshold, it is considered that the target update condition is reached, and the above-mentioned template update processing is triggered. In this way, the redundant calculation loss of the template image update can be further reduced, while taking into account the timeliness of the template image update, improving the accuracy of the template image, and further improving the accuracy of infrared target tracking.

[0063] Next, combine Figure 3 to Figure 5 , the model structure of the feature extraction model is explained in detail.

[0064] In some embodiments, the feature extraction model includes a visual transformer unit connected in parallel with the feature extraction unit.

[0065] Figure 3 Schematic diagram of a model structure of a feature extraction model provided by an embodiment of the present disclosure. Figure 3 , the feature extraction model includes a first encoder, the first encoder includes a visual transformer unit 310 on the left and a feature extraction unit 320 on the right. The visual transformer unit 310 includes the main part of the Vit model, which includes layer normalization 311, multi-head self-attention mechanism 312, fusion 313, layer normalization 314, multi-layer perceptron 315 and fusion 316 in sequence. The feature extraction unit 320 is in a parallel connection relationship with the layer normalization 314 and the multi-layer perceptron 315. The feature extraction unit 320 includes a first fully connected layer 321, a nonlinear activation layer 322 and a second fully connected layer 323.

[0066] Through the above-mentioned feature extraction model, the search image block is processed by layer normalization 311, multi-head self-attention mechanism 312 and fusion 313 in the visual transformer unit 310, and then the intermediate feature is output. After that, the intermediate feature is processed by two branches. One branch is to input the intermediate feature into layer normalization 314 and multi-layer perceptron 315 to extract image features similar to visible light; the other branch is to input the intermediate feature into the feature extraction unit 320 to extract infrared-specific features. The extraction results of the two branches are input into fusion 316 for summary processing to obtain the search feature.

[0067] In some embodiments, the feature extraction model includes a visual transformer unit connected in series with the feature extraction unit.

[0068] Figure 4 is a schematic diagram of the model structure of another feature extraction model provided by an embodiment of the present disclosure. Figure 4 , the feature extraction model includes a first encoder, and the first encoder includes a visual transformer unit 410 and a feature extraction unit 420. The visual transformer unit 410 includes the main part of the Vit model, which includes layer normalization 411, multi-head self-attention mechanism 412, fusion 413, layer normalization 414, multi-layer perceptron 415 and fusion 416 in sequence. The feature extraction unit 420 is connected in series before or after the multi-layer perceptron 415. The feature extraction unit 420 includes a first fully connected layer 421, a nonlinear activation layer 422 and a second fully connected layer 423.

[0069] If the feature extraction unit 420 is connected in series before the multi-layer perceptron 415, then, through the above-mentioned feature extraction model, the search image block is processed by the layer normalization 411, the multi-head self-attention mechanism 412, the fusion 413 and the layer normalization 414 in the visual transformer unit 410 in sequence, and the intermediate feature is output. After that, the intermediate feature is input into the feature extraction unit 420, and after being processed by the first fully connected layer 421, the nonlinear activation layer 422 and the second fully connected layer 423, the result containing the infrared image-specific features is output. The result is then continuously input into the multi-layer perceptron 415 and the fusion 416 in sequence to obtain the search feature.

[0070] If the feature extraction unit 420 is connected to the multi-layer perceptron 415 in series, then, through the above-mentioned feature extraction model, the search image block is processed by the layer normalization 411, the multi-head self-attention mechanism 412, the fusion 413, the layer normalization 414 and the multi-layer perceptron 415 in the visual transformer unit 410 in sequence, and the intermediate feature is output. After that, the intermediate feature is input into the feature extraction unit 420, and after being processed by the first fully connected layer 421, the nonlinear activation layer 422 and the second fully connected layer 423, the result containing the features similar to the visible light image and the specific features of the infrared image is output. Then the result is further input into the fusion 416 to obtain the search feature.

[0071] In some embodiments, the first encoder further comprises a scaling factor. Figure 3 , the scaling factor 324 is connected in series after the second fully connected layer 323 in the feature extraction unit 320. Similarly, Figure 4The feature extraction model shown may also include a scaling factor 424, which is connected in series after the second fully connected layer 423 in the feature extraction unit 420. The scaling factor is a hyperparameter, which is set during the model training process and is used to adjust the extracted infrared features to adapt to different infrared target tracking tasks.

[0072] The above-mentioned scaling factor is determined based on the degree of characterization of the infrared features of the infrared tracking task and / or the data distribution characteristics of the infrared image sample set. In one example, when the infrared tracking task requires a stronger feature representation capability, a scaling factor with a relatively large value can be set. For example, when the target object is small or its moving speed is fast, it is required to extract more infrared-specific features, and a larger scaling factor can be set. In another example, if the data distribution of the infrared image sample set is unstable, then a suitable scaling factor can be selected by cross-validation or other methods.

[0073] By setting the above-mentioned scaling factor, the infrared image-specific features extracted by the feature extraction unit can be more adapted to the requirements of the infrared target tracking task, thereby improving the flexibility of the feature extraction model.

[0074] In some embodiments, the feature extraction model also includes a second encoder and a third encoder, the second encoder includes at least a first number of convolutional layers, the third encoder includes at least a second number of convolutional layers, and the first number is less than the second number; and the first encoder, the second encoder and the third encoder are connected in parallel.

[0075] Specifically, during the target tracking process, the scale of the target object may change as the distance between it and the infrared sensor changes, which brings the challenge of scale change to infrared target tracking. In order to solve the problem of scale change, in the embodiment of the present disclosure, feature extraction of different scales can be performed in the feature extraction stage to extract image features at different scales, which on the one hand increases the extracted infrared information, and on the other hand adapts to the scale change of the target object, further improves the accuracy of the search feature, and further improves the accuracy of infrared target tracking.

[0076] See also Figure 5, the feature extraction model includes the first encoder 510, the second encoder 520 and the third encoder 530 described in the above embodiments. The second encoder 520 includes at least a first number of convolutional layers, and the third encoder 530 includes at least a second number of convolutional layers, and the first number is less than the second number. In this way, the first encoder 510 can obtain features that incorporate global semantic information. The second encoder 520 can focus on extracting local details in the search image block, such as edge features, texture features, and color features, which have high resolution and rich location information. The third encoder 530 can focus on extracting more abstract semantic information in the search image block, such as the shape of the target object, partial combinations, etc., which has low resolution but broader contextual information.

[0077] Furthermore, in order to avoid interference and information loss among the features extracted by the three encoders, the first encoder 510, the second encoder 520 and the third encoder 530 are connected in parallel in the embodiment of the present disclosure.

[0078] Based on the above embodiment, the features output by the first encoder 510, the features output by the second encoder 520, and the features output by the third encoder 530 can be fused first, and then the fused features can be input into a machine learning model with target object tracking function to perform target tracking processing.

[0079] Specifically, see Figure 5 , the scale of the features output by the first encoder 510 is smaller than the scales of the second encoder and the third encoder. Therefore, in order to better integrate and summarize the above-mentioned various features, the output features of the first encoder 510 can be upscaled by upsampling and normalization 540 to obtain features of the same scale, and then the features are input together with the features input by the second encoder 520 and the features output by the third encoder 530 into the fusion 550 module for feature aggregation and connection to obtain the search features.

[0080] On the basis of the above, combined with Figure 5 It shows that the feature extraction model can determine the search feature from the search image block through the following steps A to C:

[0081] Step A: input the search image block into the first encoder 510, the second encoder 520 and the third encoder 530 respectively, and after being processed by the corresponding encoders, output corresponding extracted features, namely the first feature, the second feature and the third feature. The first feature integrates global semantic information, the second feature contains high-resolution detail information in the search image block, and the third feature contains low-resolution context information in the search image block.

[0082] Step B: perform upsampling and normalization processing on the first feature using upsampling and normalization 540, and output a corresponding feature, namely, a fourth feature.

[0083] Step C: input the second feature, the third feature and the fourth feature into the fusion 550 module for fusion processing to obtain a search feature that summarizes the second feature, the third feature and the fourth feature.

[0084] The search features obtained in the above manner can integrate information at different scales, effectively overcoming the lack of information in infrared images compared to visible light images, allowing infrared target tracking algorithms to accurately identify and track target objects at multiple scales, even when the appearance of the target object changes due to occlusion, lighting changes, or motion blur.

[0085] Figure 6 The following is a schematic diagram of the structure of an infrared target tracking device provided by an embodiment of the present disclosure. Figure 6 As shown, the infrared target tracking device 600 includes:

[0086] A search image block determination module 610 is used to obtain a target infrared image frame and determine a search image block in the target infrared image frame;

[0087] The search feature determination module 620 is used to input the search image block into the feature extraction model to determine the search feature; wherein the feature extraction model includes a first encoder, the first encoder includes a visual transformer unit and a lightweight feature extraction unit with a bottleneck structure; the model parameters of the visual transformer unit are obtained by pre-training based on a visible light image sample set, and the model parameters of the visual transformer unit are frozen during the training process of the feature extraction model; the model parameters of the feature extraction unit are obtained by training based on an infrared image sample set during the training process of the feature extraction model;

[0088] The target position information determination module 630 is used to perform target tracking processing based on the template features and search features of the template image of the target object to be tracked, and determine the target position information of the target object in the target infrared image frame; wherein the template image is an image block containing a reference of the target object; the template features are obtained by inputting the template image into a feature extraction model.

[0089] The infrared target tracking device provided by the embodiment of the present disclosure, on the one hand, takes the visual transformer unit obtained by training the visible light image sample set as the basic structure, and introduces the feature extraction unit with a lightweight bottleneck structure to form a feature extraction model suitable for target tracking of infrared images, so that a large amount of image processing knowledge learned in its pre-training process can be inherited by freezing the model parameters of the visual transformer unit; at the same time, the feature extraction unit with fewer model parameters is trained by the infrared image sample set, so that it can focus on learning the infrared image features that are crucial to infrared tracking, so as to achieve the effect of fine-tuning the visual transformer unit, so that it can better adapt to the characteristics of infrared images; the feature extraction model obtained by such training establishes a bridge between visible light images and infrared images, so that the model can smoothly transition between the two images, which not only reduces the demand for infrared image sample sets, greatly reduces the cost and time of model training, but also improves the accuracy of feature extraction of infrared images, thereby improving the accuracy of infrared target tracking. On the other hand, designing the model structure of the feature extraction unit as a lightweight bottleneck structure can limit the amount of newly introduced model parameters by reducing the intermediate dimensions and reduce the amount of feature extraction calculations, thereby improving the calculation speed of infrared target tracking and reducing the demand for computing resources for infrared target tracking. This makes the infrared target tracking algorithm applicable to resource-constrained platforms such as drones or satellite platforms, thereby improving the universality of infrared target tracking.

[0090] In some embodiments, the feature extraction unit includes a first fully connected layer, a non-linear activation layer, and a second connected layer connected in series.

[0091] In some embodiments, the first encoder further includes a scaling factor, which is determined based on the degree of characterization of the infrared features by the infrared tracking task and / or the data distribution characteristics of the infrared image sample set;

[0092] The scaling factor is connected in series after the feature extraction unit.

[0093] In some embodiments, the feature extraction unit is connected in parallel with at least the multi-layer perceptron in the vision transformer unit.

[0094] In some embodiments, the feature extraction unit is connected in series before or after the multi-layer perceptron in the vision transformer unit.

[0095] In some embodiments, the feature extraction model further includes a second encoder and a third encoder, and the first encoder, the second encoder, and the third encoder are connected in parallel;

[0096] The second encoder includes at least a first number of convolutional layers, the third encoder includes at least a second number of convolutional layers, and the first number is smaller than the second number.

[0097] Furthermore, the feature extraction model in the search feature determination module 620 determines the search feature from the search image block in the following manner:

[0098] The search image block is input into a first encoder, a second encoder and a third encoder respectively, and a first feature, a second feature and a third feature are output; wherein the first feature integrates global semantic information, the second feature contains high-resolution detail information in the search image block, and the third feature contains low-resolution context information in the search image block;

[0099] Performing upsampling and normalization processing on the first feature to generate a fourth feature;

[0100] The second feature, the third feature and the fourth feature are integrated to generate a search feature.

[0101] In some embodiments, the search image block determination module 610 is specifically configured to:

[0102] Based on the template image, a search image block is determined from the target infrared image frame; wherein the area range of the search image block is smaller than the area range of the target infrared image frame.

[0103] In some embodiments, the infrared target tracking device 600 further includes a template image updating module for:

[0104] After performing target tracking processing based on the template features and search features of the template image and determining the target position information of the target object in the target infrared image frame, if a preset template update condition is met, a historical feature point set is extracted from the previous adjacent infrared image frame based on the historical position information of the target object in the previous adjacent infrared image frame; wherein the previous adjacent infrared image frame is an image frame immediately before the target infrared image frame in order of acquisition time;

[0105] Based on the historical feature point set, feature point matching is performed on the target infrared image frame to determine the matching feature point set of the target object in the target infrared image frame;

[0106] Update the template image based on the set of matching feature points.

[0107] Furthermore, the template image updating module is specifically used for:

[0108] Meeting the preset template update condition includes: the area difference between the target location information and the historical location information is greater than a preset difference threshold.

[0109] The infrared target tracking device provided in the embodiments of the present disclosure can execute the infrared target tracking method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0110] It is worth noting that in the embodiment of the above-mentioned infrared target tracking device, the various modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the various functional modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present disclosure.

[0111] Figure 7 The structure diagram of an electronic device provided by the embodiment of the present disclosure is shown in FIG. Figure 7 As shown, the electronic device 700 includes one or more processors 701 and a memory 702 .

[0112] The processor 701 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 700 to perform desired functions.

[0113] The memory 702 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 701 may run the program instructions to implement the infrared target tracking method of the embodiment of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0114] In one example, the electronic device 700 may further include: an input device 703 and an output device 707, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). The input device 703 may include, for example, a keyboard, a mouse, etc. The output device 707 may output various information to the outside, including determined distance information, direction information, etc. The output device 707 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0115] Of course, to simplify, Figure 7 Only some of the components related to the present disclosure in the electronic device 700 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 700 may also include any other appropriate components.

[0116] In addition to the above-mentioned method and device, the embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, and when the computer program instructions are executed by a processor, the processor executes the infrared target tracking method provided by the embodiment of the present disclosure.

[0117] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0118] In addition, the embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the infrared target tracking method provided by the embodiment of the present disclosure.

[0119] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0120] It should be noted that the terms used in the present disclosure are only for describing specific embodiments, rather than limiting the scope of the present application. As shown in the present disclosure specification and claims, unless the context clearly indicates an exception, the words "one", "a", "a kind of" and / or "the" do not specifically refer to the singular, but may also include the plural. The term "and / or" includes any and all combinations of one or more related listed items. Relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method or device including the elements.

[0121] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An infrared target tracking method, characterized in that: include: Acquire a target infrared image frame, and determine a search image block in the target infrared image frame; The search image block is input into a feature extraction model to determine the search feature; wherein the feature extraction model includes a first encoder, the first encoder includes a visual transformer unit and a lightweight feature extraction unit with a bottleneck structure; the model parameters of the visual transformer unit are obtained by pre-training based on a visible light image sample set, and the model parameters of the visual transformer unit are frozen during the training process of the feature extraction model, so that the model parameters of the visual transformer unit are fixed and not trained during the training process; the model parameters of the feature extraction unit are obtained by training based on an infrared image sample set during the training process of the feature extraction model; Target tracking processing is performed based on the template features of the template image of the target object to be tracked and the search features to determine the target position information of the target object in the target infrared image frame; wherein the template image is an image block containing a reference of the target object; and the template features are obtained by inputting the template image into the feature extraction model.

2. The method according to claim 1, characterized in that The first encoder further comprises a scaling factor, which is determined based on the degree of characterization of the infrared features by the infrared tracking task and / or the data distribution characteristics of the infrared image sample set; The scaling factor is connected in series after the feature extraction unit.

3. The method according to claim 1 or 2, characterized in that: The feature extraction unit is connected in parallel with at least the multi-layer perceptron in the visual transformer unit.

4. The method according to claim 1 or 2, characterized in that: The feature extraction unit is connected in series before or after the multi-layer perceptron in the visual transformer unit.

5. The method according to claim 1, characterized in that The feature extraction model further includes a second encoder and a third encoder, and the first encoder, the second encoder and the third encoder are connected in parallel; The second encoder includes at least a first number of convolutional layers, the third encoder includes at least a second number of convolutional layers, and the first number is smaller than the second number.

6. The method according to claim 5, characterized in that The feature extraction model determines the search feature from the search image block in the following manner: Input the search image block into the first encoder, the second encoder and the third encoder respectively, and output a first feature, a second feature and a third feature; wherein the first feature integrates global semantic information, the second feature contains high-resolution detail information in the search image block, and the third feature contains low-resolution context information in the search image block; Performing upsampling and normalization processing on the first feature to generate a fourth feature; The second feature, the third feature and the fourth feature are integrated to generate the search feature.

7. The method according to claim 1, characterized in that After performing target tracking processing based on the template feature of the template image and the search feature to determine the target position information of the target object in the target infrared image frame, the method further includes: If the preset template update condition is met, then based on the historical position information of the target object in the previous adjacent infrared image frame, a historical feature point set is extracted from the previous adjacent infrared image frame; wherein the previous adjacent infrared image frame is an image frame immediately before the target infrared image frame in order of acquisition time; Based on the historical feature point set, performing feature point matching on the target infrared image frame to determine a matching feature point set of the target object in the target infrared image frame; The template image is updated based on the set of matching feature points.

8. An infrared target tracking device, characterized in that: include: A search image block determination module, used to acquire a target infrared image frame and determine a search image block in the target infrared image frame; A search feature determination module is used to input the search image block into a feature extraction model to determine the search feature; wherein the feature extraction model includes a first encoder, the first encoder includes a visual transformer unit and a lightweight feature extraction unit with a bottleneck structure; the model parameters of the visual transformer unit are obtained by pre-training based on a visible light image sample set, and the model parameters of the visual transformer unit are frozen during the training process of the feature extraction model to fix the model parameters of the visual transformer unit during the training process without training; the model parameters of the feature extraction unit are obtained by training based on an infrared image sample set during the training process of the feature extraction model; A target position information determination module is used to perform target tracking processing based on the template features of the template image of the target object to be tracked and the search features, and determine the target position information of the target object in the target infrared image frame; wherein the template image is an image block containing a reference of the target object; and the template features are obtained by inputting the template image into the feature extraction model.

9. An electronic device, characterized in that: The electronic device comprises: Processor and memory; The processor is used to execute the infrared target tracking method as claimed in any one of claims 1 to 7 by calling the program or instruction stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute the infrared target tracking method according to any one of claims 1 to 7.

Images