A method, apparatus, and device for tracking specific target objects in low-resolution images in complex scenes.

Abstract

The present application belongs to the technical field of image processing, and in particular to a method, device and equipment for tracking a specific target object in a low-resolution image in a complex scene. First, an image containing a target object to be tracked is obtained as a template frame, and a sequence of images in which the target object is to be tracked is obtained. Then, a de-fogging and sharpening module is used to obtain high-resolution sharpened images corresponding to the template frame and each image frame in the sequence of images. A target object tracking module is then used to obtain the position coordinates of the target object in the high-resolution sharpened image corresponding to the template frame, which matches the target object in the high-resolution sharpened image. This enables tracking of the target object in the sequence of images in the presence of smoke interference and low resolution.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method for tracking specific target objects in low-resolution images in complex scenes. Background Technology

[0002] In complex natural scenes in the wild, vision-based tracking of specific moving targets is limited by the following factors: (1) smoke interference; (2) insufficient image resolution of low-cost imaging equipment, and the target is easily blurred in the image when shooting moving targets; (3) the target object has a large scale range. Among them, the first two factors make the target object in the image or video blurry and difficult to distinguish, and the third factor makes it difficult for the tracking algorithm to adaptively detect and track objects at multiple scales. Summary of the Invention

[0003] The present invention aims to provide a method, apparatus and device for tracking specific target objects in low-resolution images in complex scenes, and to solve the technical problem of difficulty in tracking specific target objects in low-resolution images in complex scenes in the prior art.

[0004] To solve the above-mentioned technical problems, the first aspect of the present invention is:

[0005] A method for tracking specific target objects in low-resolution images in complex scenes is provided, including:

[0006] Acquire an image containing the target object to be tracked as a template frame;

[0007] The first high-resolution image corresponding to the template frame is obtained through the desmoke and sharpening module;

[0008] Specify the target object in the first high-resolution sharpened image;

[0009] The image sequence for target object tracking is acquired, and the second high-resolution sharpened image corresponding to each image frame in the image sequence is obtained through the smoke removal and sharpening module.

[0010] The target object tracking module obtains the position coordinates of the target object that matches the target object specified in the first high-resolution image in each of the second high-resolution sharpened images.

[0011] Preferably, obtaining the first high-resolution sharpened image corresponding to the template frame through the desmoking and sharpening module includes:

[0012] Two sub-tasks are achieved by cascading two sub-networks, namely a first sub-network and a second sub-network. The first sub-network removes smoke interference in the template frame by fusing multi-scale images, and the second sub-network sharpens the low-resolution image through an encoder-decoder network structure, ultimately obtaining the first high-resolution sharpened image.

[0013] Preferably, the first sub-network removes smoke interference in the template frame through multi-scale image fusion, including:

[0014] Let the first sub-network be The template frame is a low-resolution image contaminated by smoke. Using deep convolutional network models to process images Multiple upsampling and downsampling processes are performed to extract image features and obtain a feature map.

[0015] The acquired feature maps are fused to capture the image. The local and global features are fused to obtain the feature map. .

[0016] Preferably, during training, the smoke removal and sharpening module acquires a real image containing the target object to be tracked and free from smoke interference as the first comparison image, and then compares the fused feature map. Perform convolution operations to obtain intermediate predicted feature maps The intermediate predicted feature map is calculated using the first loss function. The degree of difference between the image and the first comparison image.

[0017] Preferably, the smoke removal and sharpening module incorporates a first discriminator during the training process. Through the first discriminator Determine the first sub-network The output image is either a generated image without smoke interference or a real image without smoke interference, and the first sub-network is driven to generate a more realistic image through adversarial loss.

[0018] Preferably, the second sub-network achieves image sharpening of low resolution through an encoder-decoder network structure, including:

[0019] Let the second sub-network be The feature map processed by the first sub-network is upsampled to obtain a feature map with the same size as the first high-resolution sharpened image. ;

[0020] Then, an encoder-decoder network structure is used to process the feature maps. Further encoding and decoding are performed to obtain the first high-resolution sharpened image.

[0021] Preferably, during training, the smoke removal and sharpening module acquires a high-resolution, clear real image containing the target object to be tracked as a second comparison image, and performs a convolution operation on the first high-resolution sharpened image to obtain a predicted high-resolution image. The predicted high-resolution image is calculated using a second loss function. The degree of difference compared to the second comparison image.

[0022] Preferably, during the training process, the smoke removal and sharpening module introduces a second discriminator. Through the second discriminator Determine the second sub-network The output image is either a generated high-resolution clear image or a real high-resolution clear image, and the second sub-network is driven to generate a more realistic image through adversarial loss.

[0023] Preferably, obtaining the position coordinates of the target object matching the target object specified in the first high-resolution image in each of the second high-resolution sharpened images by the target object tracking module includes:

[0024] Let the target object specified in the first high-resolution image be the target object template Z, and let the current second high-resolution image be the image to be tracked X;

[0025] The target object tracking module uses a lightweight network model as the backbone to extract image features of the target object template Z and the image to be tracked X;

[0026] The target object tracking module uses a multi-scale feature fusion unit to fuse image features of different scales extracted from the target object template Z and the image to be tracked X;

[0027] The target object tracking module uses a cross-attention fusion unit to implicitly calculate the correlation between the target object template Z and the image X to be tracked through a single-branch structure, thereby detecting the position of the target object in the image X to be tracked.

[0028] Preferably, the target object tracking module uses a multi-scale feature fusion unit to fuse image features of different scales extracted from the target object template Z and the image to be tracked X, including:

[0029] The multi-scale feature fusion unit divides the input image features into n groups, passes them through n groups of separable convolutional networks of different depths to learn image features at n scales, then restores the image feature dimensions through tensor concatenation, and finally uses an n×n convolutional kernel to fuse the n groups of multi-scale features and inputs them into the cross-attention fusion unit.

[0030] Preferably, the target object tracking module uses a cross-attention fusion unit to implicitly calculate the correlation between the target object template Z and the image X to be tracked through a single-branch structure, including:

[0031] The image features of the target object template Z are processed by the multi-scale feature fusion unit to obtain the target object template feature map. The image features of the image to be tracked, X, are processed by the multi-scale feature fusion unit to obtain the feature map of the image to be tracked. ;

[0032] The cross-attention fusion unit processes the target object template feature map transmitted from the corresponding multi-scale feature fusion unit. and feature map of the image to be tracked By fusing the data, the feature map of the image to be tracked is implicitly obtained. The position of each pixel in the target object template feature map The correlation.

[0033] Preferably, the target object tracking module uses two detection branches for prediction, namely the first detection branch. Second detection branch The first detection branch The second detection branch is used to output the probability that each pixel location in the image X to be tracked contains the target object. Used to output the size of the target object detected in the image X to be tracked.

[0034] Preferably, during the training process, the target object tracking module incorporates a third loss function and a fourth loss function for supervised training, and calculates the first detection branch using the third loss function and the fourth loss function respectively. Second detection branch The degree of difference between the output result and the actual result.

[0035] Preferably, the acquisition of an image containing the target object to be tracked is done using a low-resolution camera; the image sequence is obtained by taking pictures of the target object using the same camera in a dynamic environment.

[0036] A second aspect of the present invention is to provide an apparatus for tracking a specific target object in a low-resolution image in a complex scene, comprising:

[0037] The desmoke and sharpening module is used to obtain the first high-resolution sharpened image corresponding to the template frame; and to obtain the second high-resolution sharpened image corresponding to each image frame in the image sequence to be tracked.

[0038] The target object tracking module is used to obtain the position coordinates of a target object that matches the target object specified in the first high-resolution image in each of the second high-resolution sharpened images.

[0039] Preferably, the smoke removal and sharpening module includes a first sub-network and a second sub-network. The two sub-tasks are achieved by cascading these two sub-networks. The first sub-network removes smoke interference in the template frame by fusing multi-scale images, and the second sub-network sharpens the low-resolution image through an encoder-decoder network structure, finally obtaining the first high-resolution sharpened image.

[0040] Preferably, the target object tracking module uses a lightweight network model as the backbone to extract image features of the target object template Z and the image to be tracked X;

[0041] The target object tracking module includes a multi-scale feature fusion unit and a cross-attention fusion unit. The multi-scale feature fusion unit fuses image features of different scales extracted from the target object template Z and the image to be tracked X. The cross-attention fusion unit implicitly calculates the correlation between the target object template Z and the image to be tracked X through a single-branch structure, thereby detecting the position of the target object in the image to be tracked X.

[0042] Preferably, the target object tracking module includes a first detection branch and a second detection branch. The first detection branch is used to output the probability that each pixel position in the image to be tracked contains a target object, and the second detection branch is used to output the size of the target object detected in the image to be tracked X.

[0043] The third aspect of the present invention is:

[0044] A device for tracking a specific target object in a low-resolution image in a complex scene is provided, comprising at least one processor and a memory communicatively connected to the processor, the memory storing instructions executable by the processor, the instructions being executed by the processor to enable the processor to perform the method described in the first aspect of the present invention.

[0045] Compared with existing technologies, the beneficial effects of this invention are as follows: It removes smoke interference from the template frame and each image in the image sequence to be tracked by fusing multi-scale images, and achieves clarity of low-resolution images through an encoder-decoder network structure, ultimately obtaining clearer images of the template frame and each image in the image sequence to be tracked, thus laying the groundwork for accurate tracking of the target object in the template frame within the image sequence; it uses a lightweight network model as the backbone to extract image features from the target object template and the image to be tracked, and performs multi-scale feature fusion, enabling the model to identify target objects with large size spans in the image. Then, it calculates the correlation between the target object template and the image to be detected through cross-attention fusion, thereby quickly detecting the position of the target object in any image frame and achieving tracking of the target object in the image sequence. Attached Figure Description

[0046] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0047] Figure 1 This is a flowchart of an embodiment of the method for tracking specific target objects in low-resolution images in complex scenes according to the present invention.

[0048] Figure 2 This is a tracking schematic diagram of an embodiment of the method for tracking specific target objects in low-resolution images in complex scenes according to the present invention.

[0049] Figure 3 This is a flowchart illustrating the workflow of the smoke removal and sharpening module in one embodiment of the method for tracking specific target objects in low-resolution images under complex scenes according to the present invention.

[0050] Figure 4 This is a schematic diagram of the architecture of the smoke removal and sharpening module in one embodiment of the method for tracking specific target objects in low-resolution images in complex scenes according to the present invention.

[0051] Figure 5 This is a flowchart illustrating the workflow of the target object tracking module in one embodiment of the method for tracking specific target objects in low-resolution images under complex scenes according to the present invention.

[0052] Figure 6 This is a schematic diagram of the target object tracking module in one embodiment of the method for tracking specific target objects in low-resolution images in complex scenes according to the present invention.

[0053] Figure 7 This is a structural block diagram of an embodiment of the apparatus for tracking specific target objects in low-resolution images in complex scenes according to the present invention.

[0054] Figure 8 This is a structural block diagram of the smoke removal and sharpening module in one embodiment of the apparatus for tracking specific target objects in low-resolution images in complex scenes according to the present invention.

[0055] Figure 9 This is a structural block diagram of the target object tracking module in one embodiment of the apparatus for tracking specific target objects in low-resolution images in complex scenes according to the present invention.

[0056] Figure 10 This is a structural block diagram of the first detection branch in an embodiment of the apparatus for tracking specific target objects in low-resolution images in complex scenes according to the present invention.

[0057] Figure 11 This is a block diagram of an embodiment of a device for tracking specific target objects in low-resolution images in complex scenes according to the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] In one embodiment, a method for tracking specific target objects in low-resolution images in complex scenes is provided. This method can track dynamic target objects such as vehicles traveling on the ground and flying objects in the air. In dynamic scenes, each frame of the video stream containing the target object may have smoke interference and low resolution and unclear phenomena.

[0060] like Figure 1 As shown, the method for tracking a specific target object in a low-resolution image in a complex scene includes the following steps:

[0061] S100: Obtain an image containing the target object to be tracked as a template frame.

[0062] S200: Obtain the first high-resolution sharpened image corresponding to the template frame through the smoke removal and sharpening module.

[0063] The template frame in step S100 is a low-resolution image captured by a low-cost low-resolution camera. The image contains the target object to be tracked. The smoke interference in the image is removed and the image resolution is improved by the desmoke and sharpening module to obtain a clear image.

[0064] S300, Specify the target object in the first high-resolution sharpened image.

[0065] The target object can be specified in the first high-resolution image by manually selecting it on the electronic device using an external device.

[0066] S400: Obtain the image sequence for target object tracking, and obtain the second high-resolution sharpened image corresponding to each image frame in the image sequence through the desmoke and sharpening module.

[0067] The image sequence here is obtained by capturing the target object in a dynamic environment using the same camera that captures the template frame. The acquired image sequence is a video stream, which contains multiple frames of images with the same specifications as the template frame, so as to track the image sequence based on the target object in the template frame.

[0068] S500: The target object is obtained in each of the second high-resolution sharpening images by the target object tracking module. The target object matches the target object specified in the first high-resolution sharpening image.

[0069] By obtaining the position coordinates of the target object that matches the target object specified in the first high-resolution image in each of the second high-resolution images, it is possible to track the target object in the video stream.

[0070] See Figure 2 The method for tracking specific target objects in low-resolution images in complex scenes performs desmoking and sharpening processing on template frames, and also performs desmoking and sharpening processing on each image frame in the image sequence that needs to be tracked. This reduces the tracking difficulty and improves the tracking accuracy when the target object tracking module performs tracking processing in the subsequent process.

[0071] Furthermore, in one embodiment, combined with Figure 3 As shown, in the method for tracking specific target objects in low-resolution images in complex scenes, the first high-resolution sharpened image corresponding to the template frame is obtained through the desmoking and sharpening module. This is achieved by cascading two sub-networks to accomplish two sub-tasks. These two sub-networks are the first sub-network and the second sub-network. Second sub-network First sub-network Smoke interference in the template frame is removed by fusing multi-scale images to obtain feature maps. Second sub-network Feature maps are obtained through an encoder-decoder network structure. The low-resolution image is sharpened to obtain the first high-resolution sharpened image.

[0072] Figure 4This is a schematic diagram of the architecture of the smoke removal and sharpening module in one embodiment of the method for tracking specific target objects in low-resolution images in complex scenes according to the present invention. Figure 4 In the diagram, the dashed box represents the model where the target object tracking module is deployed; the other parts are only used during the model's training phase. Figure 4 As shown, the first sub-network Second sub-network The specific working method is as follows:

[0073] First Sub-network Suppose the input low-resolution template frame contaminated by smoke is an image. First, classic deep convolutional network models such as ResNet18 and VGGNet-19 are used to extract... Considering the low image resolution and small image size of the template frame, only three levels of downsampling are used to ensure image feature resolution. Secondly, upsampling is performed starting from the last layer of image features obtained from downsampling, and then fused with the image features of the previous level. Here, a simple addition method is used during fusion to improve inference speed. This invention fuses the second, third, and fourth level image features obtained from downsampling to capture the local and global features of the template frame image. The resulting feature image is defined as a feature map. .

[0074] Second Subnetwork First, the feature map Upsampling is performed to obtain feature maps with the same size as the required high-resolution image. Secondly, classic encoder-decoder network structures, such as UNet and SegNet, are used to process the input feature maps. Further encoding and decoding are performed to obtain the first high-resolution, clear image.

[0075] Furthermore, in one embodiment, combined with Figure 4 As shown, the desmoothing and sharpening module used in the method for tracking specific target objects in low-resolution images in complex scenes is formed through training. During training, the first sub-network is adjusted using a loss function. Second sub-network The difference between the predicted image and the real image is calculated and fed back to improve the prediction accuracy of the desmog and sharpening module, so that the predicted image is as close as possible to the real image.

[0076] For the first subnetwork First, a real image containing the target object to be tracked and free from smoke interference is acquired as the first comparison image. The first comparison image is captured using the same camera as the template frame in a smoke-free scene. The feature map obtained after fusion is then compared. Perform convolution operations to obtain intermediate predicted feature maps (Image after desmoking), calculate intermediate predicted feature maps using the first loss function (loss function 1). The degree of difference compared to the first comparison image.

[0077] Similarly, as shown in 4, for the second sub-network During training, the smoke removal and sharpening module first acquires a high-resolution, sharp real image containing the target object to be tracked as a second comparison image. This second comparison image is captured by a high-resolution camera in a scene without smoke interference. The first high-resolution sharpening image is then convolved to obtain the predicted high-resolution image. (Predicted high-resolution image), the predicted high-resolution image is calculated using the second loss function (loss function 2). The degree of difference compared to the second comparison image.

[0078] Equation (1) is the formula for the loss function. As shown in Equation (1), the final loss function used by the desmoking and sharpening module is defined as the sum of the first loss function and the second loss function. Both the first loss function and the second loss function adopt the minimum mean square error loss. The training of the desmoking and sharpening module model is implemented by the gradient backpropagation algorithm. The model parameters are corrected by the feedback of the loss function. The image generated by the model is driven by loss detection to be as close as possible to the corresponding real image.

[0079] , (1)

[0080] Furthermore, in one embodiment, to obtain results closer to reality, adversarial loss is used to supervise the training of the desmoking and sharpening module models. Two discriminators are introduced here, implemented using a small convolutional classifier network such as AlexNet. These two discriminators are denoted as the first discriminator. Second discriminator The discriminator's role is to determine whether the output image is a generated sharp image or a real sharp image, and it drives the first sub-network through adversarial loss. With the second sub-network Generate more realistic images.

[0081] Equation (2) below is for introducing the first discriminator. Second discriminator The formula for the adversarial loss function is given in equation (2) below. Indicates through the first sub-network This model predicts low-resolution images free from smoke interference. Indicates through the second sub-network The high-resolution image predicted by this model, and the final adversarial loss function used by the desmoothing and sharpening module, are defined as the first subnetwork. With the second sub-network The training of the desmoking and sharpening module models is achieved by summing the adversarial loss functions of each module. The parameters of the model are corrected by the feedback of the adversarial loss function, and the model generates images that are as close as possible to the corresponding real images through loss adversarial analysis.

[0082]

[0083] The desmoking and sharpening module used to obtain the second high-resolution sharpened image corresponding to each image frame in the image sequence is the same as the desmoking and sharpening module used to obtain the first high-resolution sharpened image, and will not be described again here.

[0084] Furthermore, in one embodiment, such as Figure 5 As shown, the method for tracking a specific target object in a low-resolution image in a complex scene involves obtaining the position coordinates of the target object that matches the target object specified in the first high-resolution image in each second high-resolution sharpened image through the target object tracking module.

[0085] S510, The target object specified in the first high-resolution image is denoted as the target object template Z, and the current second high-resolution image is denoted as the image to be tracked X.

[0086] S520 The target object tracking module uses a lightweight network model as the backbone to extract image features of the target object template Z and the image to be tracked X.

[0087] The target object tracking module can use a lightweight network model such as ShuffleNetV2, which can improve the inference speed of the model.

[0088] S530, the target object tracking module uses a multi-scale feature fusion unit to fuse image features of different scales extracted from the target object template Z and the image to be tracked X.

[0089] The purpose of the multi-scale feature fusion unit in fusing features from different scales is to enable the target object tracking module to identify target objects with a large size range in an image. For example... Figure 6As shown, the multi-scale feature fusion unit uses grouped convolution and depthwise separable convolution to reduce computation and improve model inference speed. On the other hand, it fuses multi-scale image features to adapt to target objects with large size variations. This unit divides the input image features into three groups and passes them through three separable convolutional networks of different depths to learn the image features at three scales. Then, the feature dimensions are restored by tensor concatenation, and a 3x3 convolutional kernel is used to fuse the three groups of multi-scale image features and input them into the next stage network.

[0090] S540 The target object tracking module uses a cross-attention fusion unit to implicitly calculate the correlation between the target object template Z and the image to be tracked X through a single-branch structure, thereby detecting the position of the target object in the image to be tracked X.

[0091] Combination Figure 6 As shown, the cross-attention fusion unit, based on the attention mechanism commonly used in the transformer model, fuses the image features of the target object template Z with the image features of the image to be tracked X, implicitly obtaining the correlation between each pixel position in the image to be tracked X and the target object, and then fusing the correlation result with the original image features. The working mechanism in the transformer model is as follows, let this cross-attention fusion unit be the... Layered network (this target object tracking module has two branch networks, each with a cross-attention fusion unit and a multi-scale feature fusion unit), target object template With the image to be tracked The input image features of this layer are set to... and Using tensor dimension operations to and Adjusted to one-dimensional features, then passed through three fully connected layers Two sets of features were obtained respectively. and Combined with the following equation (3), the correlation between the target object template Z and the image X to be tracked is calculated using two sets of features:

[0092]

[0093] By performing dimensional changes on the correlation expressed in the implicit form (3) above, the correlation with... and Consistent dimensions, and fused with the input image features according to the following formula (4):

[0094]

[0095] In the process of object detection, the classic Siamese network model uses a two-branch structure to calculate the correlation between the target object template and the image to be detected. This type of method is computationally intensive and slow. However, the cross-attention fusion unit used in this embodiment of the invention implicitly calculates the correlation between the target object template and the image to be detected through a single-branch structure, thereby enabling the rapid detection of the position of the target object in any image frame.

[0096] Furthermore, in one embodiment, such as Figure 6 As shown, the target object tracking module in the method for tracking specific target objects in low-resolution images in complex scenes uses two detection branches for prediction. These two detection branches are the first detection branch and the second detection branch. Second detection branch ,like Figure 6 The target object tracking module includes network layers such as Conv3x3-BN-RELU, MaxPooling, ShuffleNetV2 convolutional group 1, multi-scale feature fusion unit, cross-attention fusion unit, MaxPooling, ShuffleNetV2 convolutional group 2, multi-scale feature fusion unit, and cross-attention fusion unit. Among them, the first MaxPooling, ShuffleNetV2 convolutional group 1, multi-scale feature fusion unit, and cross-attention fusion unit constitute the first detection branch. The components include MaxPooling, ShuffleNetV2 convolutional group 2, multi-scale feature fusion unit, and cross-attention fusion unit, which form the second detection branch. Conv3x3-BN-RELU is a component used to extract, normalize, and activate the input image data.

[0097] The first detection branch above The second detection branch is used to output the probability that each pixel location in the image X to be tracked contains the target object. , used to output the size of the target object detected in the image X to be tracked.

[0098] Furthermore, in one embodiment, the target object tracking module incorporates a third loss function and a fourth loss function for supervised training during the training process, and calculates the first detection branch using the third loss function and the fourth loss function respectively. Second detection branch The degree of difference between the output result and the actual result, the first detection branch Second detection branch The final loss is the sum of the third and fourth loss functions, as shown in equation (5):

[0099]

[0100]

[0101] In the above formula, the hyperparameter Adjusting the weighting between the two losses; The features output by the last layer of the model; To train the ground truth value (1 - belongs to the target object, 0 - does not belong to the target object) and the model prediction value (0-1 probability value) for any pixel position in the image belonging to the target object. To train the image so that any pixel location corresponds to the true size and predicted size of the target object; the target object location... Let (x, y) represent the offset of the center point of the target object to be predicted, and (w, h) represent the offset of the bounding box size of the target object to be predicted. Classification loss. Using the softmax loss function, location regression loss Use the smoothL1 loss function.

[0102] In one embodiment, a device is provided for tracking a specific target object in a low-resolution image in a complex scene, such as... Figure 7 As shown, the device 600 includes a smoke removal and desmearing module 610 and a target object tracking module 620.

[0103] First, an image containing the target object to be tracked is acquired as a template frame, and the image sequence for target object tracking is acquired. The image data of the template frame and the image data of the image sequence are then input into the desmog and sharpening module 610.

[0104] The desmoke and sharpening module 610 is used to obtain a first high-resolution sharpened image corresponding to the template frame; and to obtain a second high-resolution sharpened image corresponding to each image frame in the image sequence.

[0105] The target object is specified as the target object template in the first high-resolution image, and the image data of the target object template and the second high-resolution image corresponding to each image frame in the image sequence are input into the target object tracking module 620.

[0106] The target object tracking module 620 is used to obtain the position coordinates of the target object that matches the target object template specified in the first high-resolution image in each of the second high-resolution sharpening images.

[0107] After the target object tracking module 620 obtains the position coordinates of the target object that matches the target object template in each of the second high-resolution sharpened images, it stores these position coordinates and applies a mark, and then plays the marked second high-resolution sharpened images in sequence, thus realizing the tracking of the target object.

[0108] Furthermore, in one embodiment, such as Figure 8 As shown, the smoke removal and sharpening module 610 in the device for tracking specific target objects in low-resolution images in complex scenes includes a first sub-network 611 and a second sub-network 612, so as to achieve two sub-tasks through the cascading of these two sub-networks. The first sub-network 611 removes smoke interference in the template frame by fusing multi-scale images, and the second sub-network 612 sharpens the low-resolution image through an encoder-decoder network structure, finally obtaining a first high-resolution sharpened image.

[0109] For example, taking the processing of the template frame by the desmoking and sharpening module to obtain the first high-resolution sharpened image as an example, the first sub-network 611 can use classic deep convolutional network models such as ResNet18 and VGGNet-19 to extract the image features of the template frame. Considering that the image resolution and image size of the template frame are low, only three levels of downsampling are used to ensure the image feature resolution. Then, the first sub-network 611 fuses the sampled image features. For example, it can start upsampling from the last layer of image features obtained by downsampling and fuse it with the image features of the previous level. Here, a simple summation method is used to improve the inference speed. The second, third, and fourth level image features obtained by downsampling are fused to capture the local and global features of the template frame. The resulting feature image is defined as the feature map. Next, the second sub-network 612 will process the feature maps. Upsampling is performed to obtain feature maps with the same size as the required high-resolution image. Then, the second sub-network 612 can use a classic encoder-decoder network structure, such as UNet or SegNet, to process the input feature map. Further encoding and decoding are performed to obtain the first high-resolution, clear image.

[0110] Furthermore, in one embodiment, the target object tracking module 620 in the device for tracking specific target objects in low-resolution images in complex scenes uses a lightweight network model as the backbone to extract image features of the target object template Z and the image to be tracked X. The lightweight network model used can be ShuffleNetV2, which can improve the model inference speed.

[0111] The target object tracking module 620 includes a multi-scale feature fusion unit and a cross-attention fusion unit. The multi-scale feature fusion unit fuses image features of different scales extracted from the target object template Z and the image to be tracked X. The cross-attention fusion unit implicitly calculates the correlation between the target object template Z and the image to be tracked X through a single-branch structure, thereby detecting the position of the target object in the image to be tracked X.

[0112] Furthermore, such as Figure 9 As shown, the target object tracking module 620 in the device for tracking specific target objects in low-resolution images in complex scenes includes a first detection branch 621 and a second detection branch 622. Both the first detection branch 621 and the second detection branch 622 are equipped with a multi-scale feature fusion unit and a cross-attention fusion unit. The first detection branch 621 is used to output the probability that each pixel position in the image X to be tracked contains a target object, and the second detection branch 622 is used to output the size of the target object detected in the image X to be tracked.

[0113] Combination Figure 10 As shown, taking the first detection branch 621 as an example, it uses grouped convolution and depthwise separable convolution through the multi-scale feature fusion unit 6211. On the one hand, it reduces the amount of computation and improves the inference speed of the model. On the other hand, it fuses multi-scale image features to adapt to target objects with large size changes. Specifically, in this unit, the input image features are divided into three groups and passed through three groups of separable convolutional networks of different depths to learn the image features of three scales. Then, the feature dimension is restored by tensor concatenation. Then, a 3x3 convolutional kernel is used to fuse the three groups of multi-scale image features and input them to the cross-attention fusion unit 6212. The cross-attention fusion unit 6212 is based on the attention mechanism commonly used in the transformer model. It fuses the image features of the target object template Z with the image features of the image to be tracked X, implicitly obtaining the correlation between each pixel position in the image to be tracked X and the target object. Then, the correlation result is fused with the original image features.

[0114] In this device for tracking specific target objects in low-resolution images in complex scenarios, each functional module or unit is a corresponding program code module that implements the relevant function. This program code module can be provided to a computer, handheld electronic device, or cloud server for execution to achieve the tracking of specific target objects.

[0115] In one embodiment, a device is provided for tracking specific target objects in low-resolution images of complex scenes, such as... Figure 11 As shown, the device 700 includes a processor 710 and a memory 720 communicatively connected to the processor 710. The memory 720 stores instructions that can be executed by the processor. The instructions are executed by the processor to enable the processor 710 to perform some or all of the steps in the methods in the above embodiments.

[0116] Device 700 is an electronic device. Processor 710 can be a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processor (DSP), or other processors that meet the requirements. Memory 720 includes internal and external memory, where the main memory includes Read-Only Memory (ROM) and Random Access Memory (RAM). Device 700 also includes a bus 730, an I / O interface 740, and peripheral units 750. Peripheral units 750 are communicatively connected to bus 730 via I / O interface 740. Processor 710 and memory 720 are also communicatively connected to bus 730, enabling communication between memory 720, peripheral units 750, and processor 710. Peripheral units 750 may include a keyboard, mouse, communication module, display, etc.

[0117] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for tracking a specific target object in a low resolution image under a complex scene, characterized in that, The method comprises: acquiring an image containing a target object to be tracked as a template frame; obtaining a first high-resolution sharpened image corresponding to the template frame through a dehazing and sharpening module; specifying a target object in the first high-resolution sharpened image; acquiring an image sequence in which target object tracking is required, and obtaining a second high-resolution sharpened image corresponding to each image frame in the image sequence through the dehazing and sharpening module; obtaining the position coordinates of a target object matching the target object specified in the first high-resolution sharpened image in each second high-resolution sharpened image through a target object tracking module; the obtaining of the first high-resolution sharpened image corresponding to the template frame through the dehazing and sharpening module comprises: two sub-tasks are realized through the cascade of two sub-networks, the two sub-networks are a first sub-network and a second sub-network, the first sub-network removes smoke interference in the template frame through the fusion of multi-scale images, and the second sub-network realizes the sharpening of a low-resolution image through an encoder-decoder network structure, and finally the first high-resolution sharpened image is obtained; the removal of the smoke interference in the template frame through the fusion of multi-scale images by the first sub-network comprises: The first sub-network is , the template frame is a low-resolution image polluted by smoke , the image is subjected to multiple upsampling and downsampling using a deep convolutional network model to extract image features to obtain a feature map; fusing the obtained feature maps, capturing local and global features of the image, and obtaining a feature map after fusion fusing the obtained feature maps, capturing local and global features of the image, and obtaining a feature map after fusion fusing the obtained feature maps, capturing local and global features of the image, and obtaining a feature map after fusion The smoke removing and sharpening module obtains a real image containing a target object to be tracked and without smoke interference as a first contrast image in a training process, and calculates a difference degree between the intermediate prediction feature map and the first contrast image through a first loss function The convolution operation is performed to obtain an intermediate prediction feature map The difference degree between the intermediate prediction feature map and the first contrast image is calculated through a first loss function ​ The dehazing and clarification module introduces a first discriminator in a training process , judges whether the image output by the first sub-network is a generated image without smoke interference or a real image without smoke interference through the first discriminator , and drives the first sub-network to generate a more realistic image in the form of an adversarial loss the realization of the sharpening of the low-resolution image through the encoder-decoder network structure by the second sub-network comprises: The second sub-network is The feature map processed by the first sub-network is up-sampled to obtain a feature map consistent with the size of the first high-resolution sharpened image ; The feature maps are then further encoded and decoded using an encoder-decoder network structure to obtain the first high-resolution sharpened image. The feature maps are then further encoded and decoded using an encoder-decoder network structure to obtain the first high-resolution sharpened image. The dehazing and sharpening module obtains a high-resolution clear real image containing a target object to be tracked as a second comparison image in a training process, and performs a convolution operation on the first high-resolution sharpened image to obtain a predicted high-resolution image The difference between the predicted high-resolution image and the second comparison image is calculated by a second loss function ​ The dehazing and sharpening module introduces a second discriminator in a training process , judges whether an image output by the second subnetwork is a generated high-resolution sharpened image or a real high-resolution sharpened image through the second discriminator , and drives the second subnetwork to generate a more realistic image in the form of an adversarial loss the obtaining of the position coordinates of the target object matching the target object specified in the first high-resolution sharpened image in each second high-resolution sharpened image through the target object tracking module comprises: the target object specified in the first high-resolution sharpened image is denoted as a target object template Z, and the current second high-resolution sharpened image is denoted as a tracking image X; the target object tracking module uses a lightweight network model as a backbone network to extract image features of the target object template Z and the tracking image X; the target object tracking module uses a multi-scale feature fusion unit to fuse image features of different scales extracted from the target object template Z and the tracking image X; the target object tracking module uses a cross-attention fusion unit to implicitly calculate the correlation of the target object template Z and the tracking image X through a single-branch structure, so as to detect the position of the target object in the tracking image X; the target object tracking module uses a multi-scale feature fusion unit to fuse image features of different scales extracted from the target object template Z and the tracking image X comprises: the multi-scale feature fusion unit divides the input image features into n groups, and learns n-scale image features through n groups of separable convolutional networks of different depths, then restores the image feature dimension through a tensor splicing method, and then uses an n*n convolution kernel to fuse the n groups of multi-scale features and inputs them into the cross-attention fusion unit; the target object tracking module uses a cross-attention fusion unit to implicitly calculate the correlation of the target object template Z and the tracking image X through a single-branch structure comprises: The image features of the target object template Z are obtained as target object template feature maps after passing through the multi-scale feature fusion unit The image features of the image to be tracked X are obtained as image to be tracked feature maps after passing through the multi-scale feature fusion unit ; The cross-attention fusion unit fuses the target object template feature map transmitted from the corresponding multi-scale feature fusion unit and an image feature map to be tracked to obtain the correlation between each pixel position in the image feature map to be tracked and the target object template feature map implicitly.

2. The method for tracking a specific target object in a low-resolution image in a complex scene according to claim 1, wherein, The target object tracking module uses two detection branches for prediction, the two detection branches are respectively a first detection branch and a second detection branch , the first detection branch is used to output the probability that each pixel position in the image to be tracked X contains a target object, and the second detection branch is used to output the size of the target object detected in the image to be tracked X; The target object tracking module introduces a third loss function and a fourth loss function to supervise the training in the training process, and the first detection branch and the second detection branch output results and the difference degree of the true results are calculated through the third loss function and the fourth loss function respectively.

3. The method for tracking a specific target object in a low resolution image under a complex scene according to claim 1, wherein, The image containing the target object to be tracked is captured by using a low-resolution camera; and the image sequence is captured by using the same camera in a dynamic environment.

4. An apparatus for tracking a specific target object in a low resolution image in a complex scene, the apparatus comprising: a target object detector configured to detect a target object in the low resolution image; a target object tracker configured to track the target object in the low resolution image; and a target object classifier configured to classify the target object in the low resolution image. The method comprises: a dehazing and sharpening module for obtaining a first high-resolution sharpened image corresponding to the template frame; and for obtaining second high-resolution sharpened images corresponding to each image frame in the image sequence to be tracked; a target object tracking module for obtaining the position coordinates of the target object matching the target object specified in the first high-resolution sharpened image in each second high-resolution sharpened image. The dehazing and sharpening module comprises a first sub-network and a second sub-network, and the two sub-tasks are realized by cascading the two sub-networks. The first sub-network removes the smoke interference in the template frame by fusing multi-scale images, and the second sub-network realizes the sharpening of low-resolution images by using an encoder-decoder network structure, thereby obtaining the first high-resolution sharpened image. The target object tracking module uses a lightweight network model as the backbone network to extract the image features of the target object template Z and the image to be tracked X. In the target object tracking module, a multi-scale feature fusion unit and a cross-attention fusion unit are provided. The multi-scale feature fusion unit fuses the image features of different scales extracted from the target object template Z and the image to be tracked X, and the cross-attention fusion unit implicitly calculates the correlation between the target object template Z and the image to be tracked X through a single-branch structure, thereby detecting the position of the target object in the image to be tracked X.

5. The apparatus for tracking a specific target object in a low resolution image under a complex scene according to claim 4, wherein, The target object tracking module comprises a first detection branch and a second detection branch. The first detection branch is used to output the probability that each pixel position in the image to be tracked X contains a target object, and the second detection branch is used to output the size of the target object detected in the image to be tracked X.

6. An apparatus for tracking a specific target object in a low resolution image in a complex scene, the apparatus comprising: a target object detector configured to detect a target object in the low resolution image; a target object tracker configured to track the target object in the low resolution image; and a target object classifier configured to classify the target object in the low resolution image. The method comprises at least one processor and a memory connected to the processor, wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor to enable the processor to execute the method of any one of claims 1 to 3.

Images