A method, apparatus and device for precise guidance of dynamic target objects

Abstract

This invention belongs to the field of image processing technology, and in particular to a method, apparatus, and device for precise guidance of dynamic target objects. The method determines an initial target object pattern in an initial template frame containing the target object and acquires a dynamic target object pattern. Then, based on the initial target object pattern and the dynamic target object pattern, the target object is identified in the corresponding image frames of the image sequence. In this way, when identifying and locating the target object in each image frame of the image sequence, there is a relatively accurate reference template, which is beneficial to accurately identify the target object. This solves the problem that the correlation between the template pattern and the appearance features of the target object in the image frame to be tracked weakens due to changes in the scale and viewpoint of the target object during the tracking process. Furthermore, the twin network tracking model used in this method proposes a multi-scale feature dual fusion mechanism, which improves the model's ability to represent multi-scale target objects by fusing multi-scale features.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method for precise guidance of dynamic target objects. Background Technology

[0002] Image-based guidance systems are subject to numerous complexities when tracking dynamic objects. Because the object is in motion, its scale and viewpoint change drastically. First, the object appears small in the image at longer distances, but its size increases significantly as the distance decreases. Second, the object presents different viewpoints in the image as the viewing angle dynamically changes. These factors pose challenges to the identification and localization of image-based objects of interest, making accurate tracking even more difficult. Summary of the Invention

[0003] The present invention aims to provide a method, apparatus and device for precise guidance of dynamic target objects, and to solve the technical problem of difficulty in accurately tracking dynamic target objects in the prior art.

[0004] To solve the above-mentioned technical problems, the first aspect of the present invention is:

[0005] A method for precise guidance of dynamic target objects is provided, including:

[0006] Obtain the initial template frame of the target object;

[0007] Acquire image sequences of dynamic target objects;

[0008] The initial target object pattern is determined in the initial template frame;

[0009] A dynamic target object pattern is acquired, wherein, in the initial stage of tracking the image sequence, the initial target object pattern is used as the dynamic target object pattern, and in subsequent tracking stages, the target object identified in the image frame of the previous moment in the image sequence is used as the dynamic target object pattern.

[0010] Based on the initial target object pattern and the dynamic target object pattern, the target object is identified in the image frame corresponding to the image sequence.

[0011] Preferably, identifying the target object in the image frame corresponding to the image sequence includes:

[0012] The target object is identified in each image frame of the image sequence using a twin network tracking model.

[0013] Preferably, the twin network tracking model includes a first multi-scale feature fusion subnetwork, a second multi-scale feature fusion subnetwork, and a target tracking subnetwork, wherein the first multi-scale feature fusion subnetwork and the second multi-scale feature fusion subnetwork share weights;

[0014] Using Z 0 The initial target object pattern is represented using Z. D The dynamic target object pattern is represented using X. t This represents the image frame to be tracked at time t in the image sequence;

[0015] The step of identifying target objects in each image frame of the image sequence using a twin network tracking model includes:

[0016] The initial target object image Z is processed by the first multi-scale feature fusion subnetwork. 0 Image features f are obtained by multi-scale extraction and fusion. 0 The dynamic target object pattern Z is processed through the first multi-scale feature fusion sub-network. D Image features f are obtained by multi-scale extraction and fusion. D ;

[0017] The image frame X is processed by the second multi-scale feature fusion subnetwork. t Image features f are obtained by multi-scale extraction and fusion. x ;

[0018] Based on the image feature f 0 Image features f D Image features f X The image frame X is identified through the target tracking sub-network. t The location of the target object.

[0019] Preferably, the step of identifying the image frame X through the target tracking sub-network is... t The location of the target object includes:

[0020] The image features f are calculated using the target tracking subnetwork. 0 Image features f D Image features f X Correlation:

[0021] Cor(f 0 f X )=f 0 *f X ;Cor(f D f x )=f D *f X, where * indicates point-to-point correlation calculation;

[0022] The image feature f 0 Image features f X and the image feature f D Image features f X The two correlation tensors are fused and a 1x1 convolution operation is used for feature dimensionality reduction:

[0023] f = Conv1 × 1([Cor(f) 0 f X ); Cor(f D f x )]), where [] represents the image feature f obtained by concatenating tensors on the feature channels;

[0024] Using image features f as input, a classification head and a regression head are used to predict image frame X. t The confidence score of the target object at position (x,y) is calculated, and the target object in image frame X is regressed. t The specific dimensions are as follows.

[0025] Preferably, the step of identifying the image frame X through the target tracking sub-network is... t The location of the target object also includes: obtaining image frame X based on the prediction results of the target tracking sub-network. t The location of the target object with the highest confidence is used to update the current dynamic target object pattern Z with the region bounded by that location. D This serves as the dynamic target object pattern for the next moment's image frame tracking phase.

[0026] Preferably, the initial target object pattern Z is processed by the first multi-scale feature fusion sub-network. 0 Image features f are obtained by multi-scale extraction and fusion. 0 include:

[0027] Image feature extraction and fusion are performed using multiple lightweight models and multi-scale feature fusion module A to obtain multiple feature maps at different scales. The input of the next lightweight model and multi-scale feature fusion module A is the feature map obtained by the previous lightweight model and multi-scale feature fusion module A.

[0028] The multi-scale feature fusion module B is used to fuse feature maps of different scales based on a cross-scale feature fusion strategy. During the fusion process, large-scale feature maps are upsampled and fused with small-scale feature maps to enable the identification of small-sized objects in small-scale feature maps and large-scale objects in large-scale feature maps.

[0029] Preferably, the multi-scale feature fusion module A fuses image features extracted by the corresponding lightweight model based on a depthwise separable grouped convolution strategy, specifically including:

[0030] The input image features are divided into n groups and passed through n separable convolutional kernels of different depths to learn image features at n scales. Then, the feature dimensions are restored by tensor concatenation. Finally, an n*n convolutional kernel is used to fuse the n groups of multi-scale features and input them into the next level network.

[0031] Preferably, the step of using the multi-scale feature fusion module B to fuse feature maps of different scales based on a cross-scale feature fusion strategy includes:

[0032] The m multi-scale feature fusion modules A output m feature maps, which are respectively feature maps. Figure 1 ,feature Figure 2 ,feature Figure 3 …feature map m, by removing features Figure 1 The feature maps outside the original feature maps are recovered by upsampling to the feature maps. Figure 1 Feature maps of uniform size, for features Figure 1 Then, a 1x1 convolution kernel is used to transform the feature channels, and the m feature maps after convolution transformation are fused by feature concatenation to obtain the final feature map used for target tracking.

[0033] A second aspect of the present invention is to provide an apparatus for precisely guiding a dynamic target object, comprising:

[0034] First multi-scale feature fusion sub-network: used to process the initial target object pattern Z... 0 Image features f are obtained by multi-scale extraction and fusion. 0 And the dynamic target object pattern Z D Image features f are obtained by multi-scale extraction and fusion. D The initial target object pattern Z 0 To determine in the initial template frame of the acquired target object; for the dynamic target object pattern Z D In the initial stage of tracking the image sequence, the initial target object pattern is used as the dynamic target object pattern Z. D In the subsequent tracking phase, the target object identified in the image frame of the previous time step in the image sequence is used as the dynamic target object pattern Z. D ;

[0035] Second multi-scale feature fusion subnetwork: used for image frame X t Image features f are obtained by multi-scale extraction and fusion. X The image frame X tThis represents the image frame to be tracked at time t in the acquired image sequence of the dynamic target object;

[0036] Target tracking subnetwork, used for tracking based on the image features f 0 Image features f D Image features f X The image frame X is identified through the target tracking sub-network. t The location of the target object.

[0037] Preferably, the target tracking sub-network includes:

[0038] Correlation calculation unit: used to calculate the image features f respectively. 0 Image features f D Image features f X Correlation:

[0039] Cor(f 0 f X )=f 0 *f X ;Cor(f D f X )=f D *f X , where * indicates point-to-point correlation calculation;

[0040] Dimensionality reduction unit: This unit reduces the image features f... 0 Image features f X and the image feature f D Image features f X The two correlation tensors are fused and a 1x1 convolution operation is used for feature dimensionality reduction:

[0041] f = Conv1 × 1([Cor(f) 0 f X ); Cor(f D f X )]), where [] represents the image feature f obtained by concatenating tensors on the feature channels;

[0042] Classification and Regression Unit: Used to predict image frame X using image features f as input and a classification head and a regression head. t The confidence score of the target object at position (x,y) is calculated, and the target object in image frame X is regressed. t The specific dimensions are as follows.

[0043] Preferably, the target tracking sub-network further includes:

[0044] Dynamic target object pattern update unit: used to obtain image frame X based on the prediction results of the classification and regression unit.t The location of the target object with the highest confidence is used to update the current dynamic target object pattern Z with the region bounded by that location. D This serves as the dynamic target object pattern for the next moment's image frame tracking phase.

[0045] Preferably, the first multi-scale feature fusion sub-network includes:

[0046] Multi-scale feature map output unit: The multi-scale feature map output unit includes a lightweight model and a multi-scale feature fusion module A, which is used to extract and fuse image features to obtain multiple feature maps of different scales. The input of the next multi-scale feature map output unit is the feature map obtained by the previous multi-scale feature map output unit.

[0047] Multi-scale feature fusion module B: It is used to fuse feature maps of different scales based on cross-scale feature fusion strategy. During the fusion process, the large-scale feature map is upsampled and fused with the small-scale feature map so that small-sized objects can be identified in the small-scale feature map and large-scale objects can be identified in the large-scale feature map.

[0048] The third aspect of the present invention is:

[0049] A device for precisely guiding a dynamic target object is provided, comprising at least one processor and a memory communicatively connected to the processor, the memory storing instructions executable by the processor to enable the processor to perform any of the methods described above.

[0050] Compared with existing technologies, the beneficial effects of this invention are as follows: An initial target object pattern is determined in an initial template frame containing the target object, and a dynamic target object pattern is obtained. Specifically, in the initial stage of tracking the image sequence, the initial target object pattern is used as the dynamic target object pattern. In subsequent tracking stages, the target object identified in the previous image frame in the image sequence is used as the dynamic target object pattern. Then, based on the initial and dynamic target object patterns, the target object is identified in the corresponding image frames of the image sequence. This provides a relatively accurate reference template for identifying and locating the target object in each image frame of the image sequence, which is beneficial for accurate target object identification. It also solves the problem of weakened correlation between the template pattern and the apparent features of the target object in the image frame being tracked due to changes in the scale and viewpoint of the target object during tracking. The tracking performance for dynamically changing target objects is improved by dynamically updating the target object pattern. Furthermore, the Siamese network tracking model used in this method proposes a multi-scale feature dual-fusion mechanism. By fusing multi-scale features, the model's ability to represent multi-scale target objects is improved, thereby enhancing the target object identification ability, accuracy, and speed. Attached Figure Description

[0051] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0052] Figure 1 This is a flowchart of an embodiment of the method for precise guidance of dynamic target objects according to the present invention.

[0053] Figure 2 This is a tracking diagram of an embodiment of the method for precise guidance of dynamic target objects according to the present invention.

[0054] Figure 3 This is a schematic diagram illustrating the working principle of a twin network tracking model in one embodiment of the method for precise guidance of dynamic target objects according to the present invention.

[0055] Figure 4 This is a schematic diagram illustrating the working principle of the first multi-scale feature fusion subnetwork in one embodiment of the method for precise guidance of dynamic target objects according to the present invention.

[0056] Figure 5 This is a schematic diagram illustrating the working principle of the multi-scale feature fusion module B in the first multi-scale feature fusion sub-network of an embodiment of the method for precise guidance of dynamic target objects according to the present invention.

[0057] Figure 6 This is a schematic diagram of the architecture of an embodiment of the device for precise guidance of dynamic target objects according to the present invention.

[0058] Figure 7 This is a structural block diagram of the target tracking sub-network in one embodiment of the device for precise guidance of dynamic target objects according to the present invention.

[0059] Figure 8 This is a structural block diagram of the first multi-scale feature fusion subnetwork in one embodiment of the device for precise guidance of dynamic target objects according to the present invention.

[0060] Figure 9 This is a structural block diagram of a multi-scale feature map output unit in one embodiment of the device for precise guidance of dynamic target objects according to the present invention.

[0061] Figure 10 This is a block diagram of an embodiment of the device for precise guidance of dynamic target objects according to the present invention. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] In one embodiment, a method for precise guidance of dynamic target objects is provided. This method can track dynamic target objects such as vehicles traveling on the ground and flying objects in the air, especially when the size and viewing angle of the target object change during the movement, so as to achieve accurate tracking.

[0064] like Figure 1 As shown, the method for precise guidance of dynamic target objects includes the following steps:

[0065] S100: Obtain the initial template frame of the target object.

[0066] S200: Obtain the image sequence of the dynamic target object.

[0067] S300. Determine the initial target object pattern in the initial template frame.

[0068] In the above steps, a pre-captured image is first used as the initial template frame, which contains an image of the target object. The imaging device continuously captures images of the moving target object to obtain a video stream, and the image sequence contained in the video stream is the image to be tracked. Then, the user manually frames the target object to be tracked in the initial template frame as a reference for identifying the target object in the image sequence.

[0069] S400: Obtain the dynamic target object pattern.

[0070] In the initial stage of tracking the image sequence, the initial target object pattern is used as the dynamic target object pattern. In the subsequent tracking stages, the target object identified in the image frame of the previous moment in the image sequence is used as the dynamic target object pattern.

[0071] For example, when identifying the target object in the first image frame of an image sequence, there is only the initial target object pattern. Therefore, the initial target object pattern is used as a reference for the dynamic target object pattern.

[0072] In the tracking phase of image frames after the first image frame of the image sequence, since the target object in the moving image frame has been identified, the target object identified in the previous image frame in the image sequence can be used as the dynamic target object pattern.

[0073] S500: Based on the initial target object pattern and the dynamic target object pattern, identify the target object in the image frame corresponding to the image sequence.

[0074] In the image sequence of a video stream of a moving target object, each image frame is ordered chronologically, combined with... Figure 2 As shown, the size of the target object in the image changes significantly with the distance between the imaging device and the target object, resulting in a large difference between the scale of the target object in the current image frame to be tracked and that in the initial template frame. Furthermore, the viewing angle of the target object in the captured image frame also changes significantly with the imaging angle of the imaging device, causing a large difference in appearance between the target object in the current image frame to be tracked and that in the initial template frame. This method uses both the initial target object pattern and the dynamic target object pattern as references, which is beneficial for accurately identifying the target object in each image frame of the image sequence.

[0075] Furthermore, in one embodiment, identifying the target object in the image frames corresponding to the image sequence includes: identifying the target object in each image frame of the image sequence using a Siamese network tracking model.

[0076] The twin network tracking model consists of two neural networks with identical weights and architectures. These two neural networks analyze the template pattern and the image frame to be tracked, and finally, a sub-network identifies the target object in the image frame to be tracked based on the analysis results of the template pattern and the image frame to be tracked.

[0077] In one embodiment, the Siamese network tracking model includes a first multi-scale feature fusion subnetwork, a second multi-scale feature fusion subnetwork, and a target tracking subnetwork. The first multi-scale feature fusion subnetwork and the second multi-scale feature fusion subnetwork share weights. Figure 3 As shown, the first multi-scale feature fusion subnetwork ( Figure 3 The upper-middle multi-scale feature fusion CNN), the second multi-scale feature fusion sub-network ( Figure 3 The multi-scale feature fusion CNN (multi-scale feature fusion CNN) and the target tracking sub-network are coupled. The first multi-scale feature fusion sub-network extracts and fuses multi-scale features of the initial target object pattern and the dynamic target object pattern determined in the initial template frame. The second multi-scale feature fusion sub-network extracts and fuses multi-scale features of the current image frame to be tracked. The first and second multi-scale feature fusion sub-networks send the fused feature maps to the target tracking sub-network, which then identifies the target object in the current image frame to be tracked.

[0078] Without loss of generality, use Z 0 Represents the initial target object pattern, using Z. D To represent a dynamic target object pattern, use X.t Let t represent the image frame to be tracked in the image sequence. Then, the target object identification in each image frame of the image sequence using the Siamese network tracking model includes:

[0079] Step 1: Process the initial target object pattern Z using the first multi-scale feature fusion sub-network. 0 Image features f are obtained by multi-scale extraction and fusion. 0 The dynamic target object pattern Z is processed through the first multi-scale feature fusion sub-network. D Image features f are obtained by multi-scale extraction and fusion. D The image frame X is processed through a second multi-scale feature fusion subnetwork. t Image features f are obtained by multi-scale extraction and fusion. X ;

[0080] Step 2: Based on image features f 0 Image features f D Image features f X Image frame X is identified through a target tracking subnetwork. t The location of the target object.

[0081] Step 2 here specifically includes:

[0082] Step 2.1: Calculate image features f using the target tracking sub-network. 0 Image features f D Image features f X The correlation can be calculated using the following formula:

[0083] Cor(f 0 f X )=f 0 *f X ;Cor(f D f x )=f D *f X , where * indicates point-to-point correlation calculation.

[0084] Step 2.2: Extract image features f 0 Image features f X and image features f D Image features f X The two correlation tensors are fused and a 1x1 convolution operation is used for feature dimensionality reduction. The dimensionality reduction process can be described by the following formula:

[0085] f = Conv1 × 1([Cor(f) 0 f X ); Cor(f Df X )]), where [] represents the image feature f obtained by splicing tensors on the feature channels.

[0086] Step 2.3: Using image features f as input, predict image frame X using the classification head and regression head. t The confidence score of the target object at position (x,y) is calculated, and the target object in image frame X is regressed. t The specific dimensions are as follows.

[0087] The classification head and regression head are implemented through corresponding convolutional networks. The classification head is used to determine the category of the selected target object, so as to judge whether the selected target object has a high correlation with the real target object. The regression head is used to fine-tune the position of the selected region, so that the selected region can correctly select the target object in the image to be tracked after fine-tuning.

[0088] Step 2.4: Based on the prediction results of the target tracking subnetwork, obtain image frame X. t The location of the target object with the highest confidence is used to update the current dynamic target object pattern Z with the region bounded by that location. D This serves as the dynamic target object pattern for the next moment's image frame tracking phase.

[0089] Considering that in the initial stage of image sequence tracking, only the initial target object pattern provided by the user is available, the initial target object pattern is used to create a dynamic target object pattern at time t=1, and the dynamic target object pattern is dynamically updated starting at time t>1. Figure 3 The dashed line indicates that the dynamic target object pattern has been updated.

[0090] The loss function of the twin network tracking model in this method is designed as follows during the training phase:

[0091] We use softmax loss to define the classification loss and IOU loss to define the bounding box (target object size) regression loss. in, Let l represent the probability that the i-th position belongs to the target object. * This represents the ground truth value of the bounding box where a target object exists at position i. This represents the predicted bounding box value of the target object at position i obtained by the model, and IOU represents the area intersection-union ratio between two bounding boxes. Finally, the training of the model is driven by the two loss functions. The optimization of the twin network tracking model is achieved by gradient descent through gradient backpropagation, and optimizers such as SGD and Adam are used to achieve rapid convergence of model parameters.

[0092] Furthermore, in one embodiment, the initial target object pattern Z is processed by the first multi-scale feature fusion sub-network as described above. 0 Image features f are obtained by multi-scale extraction and fusion. 0 include:

[0093] Combination Figure 4 As shown, the first multi-scale feature fusion sub-network uses the efficient lightweight model ShuffleNetV2 as the backbone network architecture for image feature extraction. In order to improve the tracking performance of the first multi-scale feature fusion sub-network model for multi-scale target objects, a multi-scale feature dual fusion mechanism is proposed here. That is, the fusion of multi-scale image features is realized through multi-scale feature fusion module A and multi-scale feature fusion module B, thereby improving the representation ability of multi-scale target objects.

[0094] In classic deep neural network models such as VGGNet and ResNet, each convolutional group consists of a series of stacked 3x3 convolutions. Therefore, the feature scale output by each convolutional group is relatively singular. This paper proposes a depth-separable grouped convolutional strategy (multi-scale feature fusion module A) to fuse features of multiple scales within the corresponding ShuffleNetV2 convolutional group. Secondly, considering that the discriminative power of small objects in deep feature maps (large-scale features) deteriorates during feature map downsampling in deep neural network models, a cross-scale feature fusion strategy (multi-scale feature fusion module B) is proposed. This strategy involves upsampling large-scale feature maps and fusing them with small-scale feature maps, aiming to identify small objects in small-scale feature maps and large-scale objects in large-scale feature maps.

[0095] exist Figure 4 In this process, three lightweight models and a multi-scale feature fusion module A are used to extract and fuse image features, resulting in three feature maps at different scales. The input to the next lightweight model and the multi-scale feature fusion module A is the feature map obtained from the previous lightweight model and the multi-scale feature fusion module A. Then, combined with... Figure 5 As shown, the multi-scale feature fusion module B is used to fuse feature maps of different scales based on a cross-scale feature fusion strategy.

[0096] Combination Figure 4 As shown, the multi-scale feature fusion module A uses grouped convolution and depthwise separable convolution. This reduces computational cost and improves model inference speed, while also fusing multi-scale features to adapt to targets with large size variations. Figure 4In the middle, the multi-scale feature fusion module A divides the input features into three groups and passes them through three groups of separable convolutions of different depths to learn image features at three scales. Then, the feature dimensions are restored by tensor concatenation, and a 3x3 convolution kernel is used to fuse the three groups of multi-scale features and input them into the next level network.

[0097] Combination Figure 5 As shown, the multi-scale feature fusion module B uses upsampling operations, such as transposed convolution, to upsample large-scale features to a certain size and fuse them with small-scale feature maps. Figure 5 In the middle, the multi-scale feature fusion module B fuses features from three scales, i.e., features Figure 1 (H1xW1), Features Figure 2 (H2xW2), characteristics Figure 3 (H3xW3), by using features Figure 2 With features Figure 3 Use transposed convolution (or linear interpolation) to restore the feature set. Figure 1 Features of consistent size, for features Figure 1 The feature channels are transformed using a 1x1 convolution kernel, and then the three feature maps are fused by feature concatenation to obtain the final feature map used for target tracking.

[0098] Since the second multi-scale feature fusion subnetwork and the first multi-scale feature fusion subnetwork are two symmetrical neural networks with the same working mechanism, the working principle of the second multi-scale feature fusion subnetwork will not be elaborated here.

[0099] In one embodiment, a device is provided for precisely guiding a dynamic target object, such as... Figure 6 As shown, the device 600 includes a first multi-scale feature fusion sub-network 610, a second multi-scale feature fusion sub-network 620, and a target tracking sub-network 630. The first multi-scale feature fusion sub-network 610 and the second multi-scale feature fusion sub-network 620 share weights, and the first multi-scale feature fusion sub-network 610, the second multi-scale feature fusion sub-network 620, and the target tracking sub-network 630 are coupled together.

[0100] The initial target object image Z is processed by the first multi-scale feature fusion subnetwork 610. 0 Image features f are obtained by multi-scale extraction and fusion. 0 And the dynamic target object pattern Z D Image features f are obtained by multi-scale extraction and fusion. D Initial target object pattern Z 0 To determine in the initial template frame of the acquired target object; for dynamic target object pattern Z DIn the initial stage of tracking the image sequence, the initial target object pattern is used as the dynamic target object pattern Z. D In the subsequent tracking phase, the target object identified in the image frame of the previous time step in the image sequence is used as the dynamic target object pattern Z. D .

[0101] The second multi-scale feature fusion subnetwork 620 is used to analyze image frame X. t Image features f are obtained by multi-scale extraction and fusion. X Image frame X t This represents the image frame to be tracked at time t in the acquired image sequence of the dynamic target object.

[0102] Based on image features, the target tracking subnetwork 630 is used. 0 Image features f D Image features f X Identify image frame X t The location of the target object.

[0103] like Figure 7 As shown, in one embodiment, the target tracking sub-network 630 specifically includes a correlation calculation unit 631, a dimensionality reduction unit 632, a classification and regression unit 633, and a dynamic target object pattern update unit 634, wherein:

[0104] Correlation calculation unit 631 is used to calculate image features f based on the following formula. 0 Image features f D Correlation of image feature f:

[0105] Cor(f 0 f X ) = f 0 *f X ;Cor(f D f X )=f D *f X , where * indicates point-to-point correlation calculation.

[0106] Dimensionality reduction unit 632 is used to reduce image features f based on the following formula 0 Image features f X and image features f D Image features f X The two correlation tensors are fused and a 1x1 convolution operation is used for feature dimensionality reduction:

[0107] f = Conv1 × 1([Cor(f) 0 f X ); Cor(f D fX )]), where [] represents the image feature f obtained by splicing tensors on the feature channels.

[0108] The classification and regression unit 633 is used to predict image frame X using image features f as input and a classification head and a regression head. t The confidence score of the target object at position (x,y) is calculated, and the target object in image frame X is regressed. t The specific dimensions are as follows.

[0109] The dynamic target object pattern update unit 634 is used to obtain image frame X based on the prediction results of the classification and regression unit 633. t The location of the target object with the highest confidence is used to update the current dynamic target object pattern Z with the region bounded by that location. D This serves as the dynamic target object pattern for the next moment's image frame tracking phase.

[0110] Furthermore, in one embodiment, such as Figure 8 As shown, the first multi-scale feature fusion sub-network includes three sets of multi-scale feature map output units 611 and a multi-scale feature fusion module B 612. Of course, an image feature input unit is also provided before the first set of multi-scale feature map output units 611.

[0111] like Figure 9 As shown, the multi-scale feature map output unit 611 includes a lightweight model 6111 and a multi-scale feature fusion module A 6112. The lightweight model adopts the ShuffleNetV2 model for image feature extraction, and the multi-scale feature fusion module A 6112 is used to fuse the extracted image features. After processing by the three sets of multi-scale feature map output units 611, three feature maps of different scales are obtained. During the processing of the three sets of multi-scale feature map output units 611, the input of the next set of multi-scale feature map output units 611 is the feature map obtained by the previous set of multi-scale feature map output units 611.

[0112] Specifically, in combination Figure 4 As shown, the multi-scale feature fusion module A divides the input features into three groups and passes them through three groups of separable convolutions of different depths to learn image features at three scales. Then, the feature dimensions are restored by tensor concatenation, and a 3x3 convolution kernel is used to fuse the three groups of multi-scale features and input them into the next level network.

[0113] The multi-scale feature fusion module B 612 is used to fuse feature maps of different scales based on a cross-scale feature fusion strategy. During the fusion process, large-scale feature maps are upsampled and fused with small-scale feature maps to enable the identification of small-sized objects in small-scale feature maps and large-scale objects in large-scale feature maps.

[0114] Specifically, in combination Figure 5 As shown, the multi-scale feature fusion module B uses upsampling operations, such as transposed convolution, to upsample large-scale features to a certain size and fuse them with small-scale feature maps. Figure 5 In the middle, the multi-scale feature fusion module B fuses features from three scales, i.e., features Figure 1 (H1xW1), Features Figure 2 (H2xW2), characteristics Figure 3 (H3xW3), by using features Figure 2 With features Figure 3 Use transposed convolution (or linear interpolation) to restore the feature set. Figure 1 Features of consistent size, for features Figure 1 The feature channels are transformed using a 1x1 convolution kernel, and then the three feature maps are fused by feature concatenation to obtain the final feature map used for target tracking.

[0115] The principle of the second multi-scale feature fusion subnetwork 620 is the same as that of the first multi-scale feature fusion subnetwork 610, and will not be repeated here.

[0116] Each functional module or unit in this device for precisely guiding dynamic target objects is a corresponding program code module that implements the relevant function. This program code module can be provided to a computer, handheld electronic device, or cloud server for execution to achieve the tracking of a specific target object.

[0117] In one embodiment, a device is provided for precise guidance of a dynamic target object, such as... Figure 10 As shown, the device 700 includes a processor 710 and a memory 720 communicatively connected to the processor 710. The memory 720 stores instructions executable by the processor. These instructions are executed by the processor to enable the processor 710 to perform some or all of the steps in the methods described in the above embodiments. The device 700 is an electronic device, and the processor 710 can be a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), or other processors that meet the requirements. The memory 720 includes inner and outer memory, wherein the inner memory includes read-only memory (ROM) and random access memory (RAM). The device 700 also includes a bus 730, an I / O interface 740, and peripheral units 750. The peripheral units 750 are communicatively connected to the bus 730 via the I / O interface 740. The processor 710 and the memory 720 are also communicatively connected to the bus 730, enabling communication between the memory 720, the peripheral units 750, and the processor 710. The peripheral units 750 may include a keyboard, mouse, communication module, display, etc.

[0118] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for precise guidance of dynamic target objects, characterized in that, include: To obtain the initial template frame of the target object, a pre-captured image is used as the initial template frame, which contains the image of the target object. Acquire image sequences of dynamic target objects; The initial target object pattern is determined in the initial template frame; A dynamic target object pattern is acquired, wherein, in the initial stage of tracking the image sequence, the initial target object pattern is used as the dynamic target object pattern, and in subsequent tracking stages, the target object identified in the image frame of the previous moment in the image sequence is used as the dynamic target object pattern. Based on the initial target object pattern and the dynamic target object pattern, the target object is identified in the image frame corresponding to the image sequence; The step of identifying the target object in the image frame corresponding to the image sequence includes: The target object is identified in each image frame of the image sequence using a twin network tracking model; The twin network tracking model includes a first multi-scale feature fusion subnetwork, a second multi-scale feature fusion subnetwork, and a target tracking subnetwork, wherein the first multi-scale feature fusion subnetwork and the second multi-scale feature fusion subnetwork share weights; Using Z 0 The initial target object pattern is represented using Z. D The dynamic target object pattern is represented using X. t This represents the image frame to be tracked at time t in the image sequence; The step of identifying target objects in each image frame of the image sequence using a twin network tracking model includes: The initial target object image Z is processed by the first multi-scale feature fusion subnetwork. 0 Image features f are obtained by multi-scale extraction and fusion. 0 The dynamic target object pattern Z is processed through the first multi-scale feature fusion sub-network. D Image features f are obtained by multi-scale extraction and fusion. D ; The image frame X is processed by the second multi-scale feature fusion subnetwork. t Image features f are obtained by multi-scale extraction and fusion. X ; Based on the image feature f 0 Image features f D Image features f X The image frame X is identified through the target tracking sub-network. t The location of the target object; The image frame X is identified through the target tracking subnetwork. t The location of the target object includes: The image features f are calculated using the target tracking subnetwork. 0 Image features f D Image features f X Correlation: Cor(f 0 ,f X )=f 0 *f X ;Cor(f D ,f X )=f D *f X , where * indicates point-to-point correlation calculation; The image feature f 0 Image features f X and the image feature f D Image features f X The two correlation tensors are fused and a 1x1 convolution operation is used for feature dimensionality reduction: f = Conv1 × 1([Cor(f) 0 ,f X ); Cor(f D ,f X )]), where [] represents the image feature f obtained by concatenating tensors on the feature channels; Using image features f as input, a classification head and a regression head are used to predict image frame X. t The confidence score of the target object at position (x,y) is calculated, and the target object in image frame X is regressed. t The specific dimensions are as follows; The image frame X is identified through the target tracking subnetwork. t The location of the target object also includes: obtaining image frame X based on the prediction results of the target tracking sub-network. t The location of the target object with the highest confidence is used to update the current dynamic target object pattern Z with the region bounded by that location. D And serve as the dynamic target object pattern for the next moment's image frame tracking phase; The initial target object image Z is processed through the first multi-scale feature fusion subnetwork. 0 Image features f are obtained by multi-scale extraction and fusion. 0 include: Image feature extraction and fusion are performed using multiple lightweight models and multi-scale feature fusion module A to obtain multiple feature maps at different scales. The input of the next lightweight model and multi-scale feature fusion module A is the feature map obtained by the previous lightweight model and multi-scale feature fusion module A. The multi-scale feature fusion module B is used to fuse feature maps of different scales based on a cross-scale feature fusion strategy. During the fusion process, the large-scale feature map is upsampled and fused with the small-scale feature map so that small-sized objects can be identified in the small-scale feature map and large-scale objects can be identified in the large-scale feature map. The multi-scale feature fusion module A fuses image features extracted by the corresponding lightweight model based on a depthwise separable grouped convolution strategy, specifically including: The input image features are divided into n groups and passed through n separable convolutional kernels of different depths to learn image features at n scales. Then, the feature dimensions are restored by tensor concatenation. Finally, an n*n convolutional kernel is used to fuse the n groups of multi-scale features and input them into the next level network. The use of multi-scale feature fusion module B to fuse feature maps of different scales based on a cross-scale feature fusion strategy includes: The m multi-scale feature fusion modules A output m feature maps, namely feature map 1, feature map 2, feature map 3... feature map m. The feature maps other than feature map 1 are restored to feature map with the same size as feature map 1 by upsampling. For feature map 1, a 1x1 convolution kernel is used to transform the feature channels. Then, the m feature maps after convolution transformation are fused by feature concatenation to obtain the final feature map for target tracking.

2. A device for precise guidance of dynamic target objects, characterized in that, include: First multi-scale feature fusion sub-network: used to process the initial target object pattern Z... 0 Image features f are obtained by multi-scale extraction and fusion. 0 And the dynamic target object pattern Z D Image features f are obtained by multi-scale extraction and fusion. D The initial target object pattern Z 0 To determine in the initial template frame of the acquired target object; for the dynamic target object pattern Z D In the initial stage of tracking the image sequence, the initial target object pattern is used as the dynamic target object pattern Z. D In the subsequent tracking phase, the target object identified in the image frame of the previous time step in the image sequence is used as the dynamic target object pattern Z. D ; Second multi-scale feature fusion subnetwork: used for image frame X t Image features f are obtained by multi-scale extraction and fusion. X The image frame X t This represents the image frame to be tracked at time t in the acquired image sequence of the dynamic target object; Target tracking subnetwork, used for tracking based on the image features f 0 Image features f D Image features f X The image frame X is identified through the target tracking sub-network. t The location of the target object; The target tracking subnetwork includes: Correlation calculation unit: used to calculate the image features f respectively. 0 Image features f D Image features f X Correlation: Cor(f 0 ,f X )=f 0 *f X ;Cor(f D ,f X )=f D *f X , where * indicates point-to-point correlation calculation; Dimensionality reduction unit: This unit reduces the image features f... 0 Image features f X and the image feature f D Image features f X The two correlation tensors are fused and a 1x1 convolution operation is used for feature dimensionality reduction: f = Conv1 × 1([Cor(f) 0 ,f X ); Cor(f D ,f X )]), where [] represents the image feature f obtained by concatenating tensors on the feature channels; Classification and Regression Unit: Used to predict image frame X using image features f as input and a classification head and a regression head. t The confidence score of the target object at position (x,y) is calculated, and the target object in image frame X is regressed. t The specific dimensions are as follows; The target tracking subnetwork also includes: Dynamic target object pattern update unit: used to obtain image frame X based on the prediction results of the classification and regression unit. t The location of the target object with the highest confidence is used to update the current dynamic target object pattern Z with the region bounded by that location. D And serve as the dynamic target object pattern for the next moment's image frame tracking phase; The first multi-scale feature fusion subnetwork includes: Multi-scale feature map output unit: The multi-scale feature map output unit includes a lightweight model and a multi-scale feature fusion module A, which is used to extract and fuse image features to obtain multiple feature maps of different scales. The input of the next multi-scale feature map output unit is the feature map obtained by the previous multi-scale feature map output unit. Multi-scale feature fusion module B: It is used to fuse feature maps of different scales based on cross-scale feature fusion strategy. During the fusion process, the large-scale feature map is upsampled and fused with the small-scale feature map so that small-sized objects can be identified in the small-scale feature map and large-scale objects can be identified in the large-scale feature map.

3. A device for precise guidance of dynamic target objects, characterized in that, The method includes at least one processor and a memory communicatively connected to the processor, the memory storing instructions executable by the processor to enable the processor to perform the method of claim 1.

Images