Adaptive twin network target tracking method and device, medium and equipment

Abstract

The invention provides a self-adaptive twin network target tracking method and device, a medium and equipment, and relates to the technical field of target tracking, and the method comprises the steps: firstly, obtaining a to-be-tracked image group, and then carrying out the enhancement processing of a search image and a template image comprising a preset target object; thirdly, utilizing a twin network feature extraction module in the trained target tracking model to perform feature extraction on the enhanced template image and the search image, and calculating the similarity between a target object feature vector and a search image feature vector to determine whether a to-be-tracked target object exists in the search image; and if the target is not found, repeating the processing steps by using the adjacent image as a search image until the target object to be tracked is successfully found, and determining the position information of the target object to be tracked by using the anchor-frame-free module so as to determine the prediction frame position of the target object to be tracked in the search image. According to the embodiment of the invention, the accuracy and real-time performance of target tracking are improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of target tracking, and in particular to an adaptive twin network target tracking method, device, medium and equipment. Background Art

[0002] As a key technology in the field of artificial intelligence, target tracking is driven by technologies such as the Internet of Things, big data, and cloud computing. Its application scope is constantly expanding, and its technical requirements are also increasing. Related target tracking methods such as Kalman filtering and particle filtering are widely used in specific scenarios, but they are sensitive to complex backgrounds and occlusions, and their generalization ability is limited.

[0003] With the rise of deep learning methods, target tracking technology has made significant progress. However, some target tracking algorithms proposed in related technologies require intensive region extraction of the entire image to generate multiple predefined anchor frames during the target object annotation frame generation stage; then, target tracking is completed by performing feature calculations on the image. This may cause the existing target tracking algorithms to have high computational complexity and low tracking accuracy when facing complex scenes. Summary of the invention

[0004] The disclosed embodiments at least provide an adaptive twin network target tracking method, apparatus, medium and equipment, which improves the accuracy of target tracking by utilizing the twin network feature extraction module to extract richer feature information of the image, and utilizes the anchor-free frame module to determine the position information of the key points of the image and the predicted frame position, without generating redundant predefined anchor frames, thereby reducing the computational complexity and improving the real-time performance of target tracking.

[0005] The present disclosure provides an adaptive twin network target tracking method, including:

[0006] Step 1: obtaining a group of images to be tracked, the group of images to be tracked comprising a template image and a search image, the template image comprising a preset target object; and performing image enhancement processing on the template image and the search image to obtain a group of enhanced images to be tracked; the group of enhanced images to be tracked comprising a template enhanced image and a search enhanced image;

[0007] Step 2: Based on the twin network feature extraction module in the trained target tracking model, feature extraction is performed on the template enhanced image to obtain a template enhanced image feature vector; the template enhanced image feature vector includes a preset target object feature vector; and based on the twin network feature extraction module, feature extraction is performed on the search enhanced image to obtain a search enhanced image feature vector; the target tracking model also includes an anchor-free frame module;

[0008] Step 3: Calculate the similarity value between the preset target object feature vector and the search enhanced image feature vector based on the mutual convolution method, and determine whether there is a target object to be tracked in the search enhanced image according to the similarity value;

[0009] Step 4: When the target object to be tracked exists in the search enhanced image, determine the position information of the target object to be tracked in the search enhanced image based on the anchor-free frame module; and determine the predicted frame position of the target object to be tracked in the search enhanced image based on the position information; when the target object to be tracked does not exist in the search enhanced image, obtain an adjacent image for the search image, and repeat steps 1 to 3 above using the adjacent image as a new search image until the target object to be tracked is found in the search enhanced image.

[0010] The present disclosure provides an adaptive twin network target tracking device, including:

[0011] The image acquisition module is used to execute step 1: acquire a group of images to be tracked, the group of images to be tracked includes a template image and a search image, the template image includes a preset target object; and perform image enhancement processing on the template image and the search image to obtain a group of enhanced images to be tracked; the group of enhanced images to be tracked includes a template enhanced image and a search enhanced image;

[0012] A feature extraction module is used to perform step 2: based on the twin network feature extraction module in the trained target tracking model, feature extraction is performed on the template enhanced image to obtain a template enhanced image feature vector; the template enhanced image feature vector includes a preset target object feature vector; and based on the twin network feature extraction module, feature extraction is performed on the search enhanced image to obtain a search enhanced image feature vector; the target tracking model also includes an anchor-free frame module;

[0013] The object judgment module is used to execute step 3: calculate the similarity value between the preset target object feature vector and the search enhanced image feature vector based on the mutual convolution method, and judge whether there is a target object to be tracked in the search enhanced image according to the similarity value;

[0014] A position determination module is used to execute step 4: when the target object to be tracked exists in the search enhanced image, determine the position information of the target object to be tracked in the search enhanced image based on the anchor-free frame module; and determine the predicted frame position of the target object to be tracked in the search enhanced image based on the position information; when the target object to be tracked does not exist in the search enhanced image, obtain an adjacent image for the search image, and repeat steps 1 to 3 above using the adjacent image as a new search image until the target object to be tracked is found in the search enhanced image.

[0015] An embodiment of the present disclosure provides a computer device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory through the bus, and when the machine-readable instructions are executed by the processor, the adaptive twin network target tracking method described in any possible implementation manner described above is performed.

[0016] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the adaptive twin network target tracking method as described in any possible implementation manner described above is implemented.

[0017] The adaptive twin network target tracking method, apparatus, medium and equipment provided in the embodiments of the present invention improve the accuracy of target tracking by utilizing the twin network feature extraction module to extract richer feature information of the image, and utilize the anchor-free frame module to determine the position information of the key points of the image and the position of the prediction frame, without the need to generate redundant predefined anchor frames, thereby reducing the computational complexity and improving the real-time performance of target tracking.

[0018] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required to be cited in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0020] Figure 1A flowchart of an adaptive twin network target tracking method provided by an embodiment of the present disclosure is shown;

[0021] Figure 2 A flowchart of a similarity value calculation method in an adaptive twin network target tracking method provided by an embodiment of the present disclosure is shown;

[0022] Figure 3 A flowchart of a method for training a target tracking model in the adaptive twin network target tracking method provided by an embodiment of the present disclosure is shown;

[0023] Figure 4 A schematic diagram of the structure of an adaptive twin network target tracking device provided by an embodiment of the present disclosure is shown;

[0024] Figure 5 A schematic diagram of the structure of another adaptive twin network target tracking device provided by an embodiment of the present disclosure is shown;

[0025] Figure 6 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0027] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0028] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0029] As a key technology in the field of artificial intelligence, target tracking is driven by technologies such as the Internet of Things, big data, and cloud computing. Its application scope is constantly expanding, and its technical requirements are also increasing. Related target tracking methods such as Kalman filtering and particle filtering are widely used in specific scenarios, but they are sensitive to complex backgrounds and occlusions, and their generalization ability is limited.

[0030] According to the research, it is proposed to apply deep learning methods to target tracking technology in related technologies, and significant progress has been made. However, there are still some problems with the target tracking algorithm that applies deep learning methods. For example, the SORT algorithm based on deep learning combined with a lightweight target detector (such as YOLO) achieves efficient target tracking. However, this method relies on the extraction of simple features, which limits the ability to capture complex target features. Furthermore, the introduction of deep neural networks in DeepSORT improves the tracking accuracy in complex scenes, but its computational complexity and real-time problems remain to be solved. In addition, the ATOM algorithm improves the target feature representation through adaptive target-oriented metric learning, but it introduces complex background modeling and model update mechanisms, which reduces real-time performance. The FairMOT algorithm improves the detection and tracking capabilities of complex scenes through joint training, but increases the computational complexity of the algorithm. Therefore, although many new algorithms such as SiamFC and SiamRPN combine the advantages of deep learning and traditional methods, there are still challenges in accuracy, robustness and real-time performance when facing complex scenarios, and further optimization is needed.

[0031] Based on the above research, an adaptive twin network target tracking method, device, medium and equipment are provided in the embodiments of the present invention. First, the search image and the template image including the preset target object are enhanced; then, the twin network feature extraction module in the trained target tracking model is used to extract features of the enhanced template image and the search image, and the similarity between the feature vector of the target object and the feature vector of the search image is calculated to determine whether there is a target object to be tracked in the search image; if the target is not found, the adjacent image will be used as the search image to repeat the above processing steps until the target object to be tracked is successfully found, and its position information is determined using the anchor-free frame module, and then the predicted frame position of the target object to be tracked in the search image is determined.

[0032] In the disclosed embodiment, the accuracy of target tracking is improved by utilizing the twin network feature extraction module to extract richer feature information of the image, and the anchor-free frame module is used to determine the position information of the key points of the image and the prediction frame position, without the need to generate redundant predefined anchor frames, thereby reducing the computational complexity and improving the real-time performance of target tracking.

[0033] To facilitate understanding of this embodiment, the execution subject of the adaptive twin network target tracking method provided by the embodiment of the present disclosure is first introduced in detail. The execution subject of the adaptive twin network target tracking method provided by the embodiment of the present disclosure is a computer device. The computer device can be a server. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data and artificial intelligence platforms.

[0034] The following is a detailed description of the adaptive twin network target tracking method provided by the embodiment of the present application in conjunction with the accompanying drawings. Figure 1 As shown, it is a flowchart of an adaptive twin network target tracking method provided by an embodiment of the present disclosure, and the method includes the following S101-S105:

[0035] S101, obtaining a group of images to be tracked, wherein the group of images to be tracked includes a template image and a search image, wherein the template image includes a preset target object; and performing image enhancement processing on the template image and the search image to obtain a group of enhanced images to be tracked; the group of enhanced images to be tracked includes a template enhanced image and a search enhanced image.

[0036] It is understandable that the template image and the search image belong to the same video clip. Among them, the template image is a specific frame selected manually from the video, which contains the target person to be tracked; the search image is a frame image captured from the same video segment at different frame rates or time intervals, which is used to find and track the target object selected in the template image. Here, in the disclosed embodiment, the template image also includes a part of background information, which can provide necessary contextual information for model tracking, help understand the changes and positions of the target object in different backgrounds, so as to reduce false detection and enhance the stability of tracking, which is conducive to improving the robustness and accuracy of subsequent model tracking.

[0037] For example, after the template image and the search image are obtained, image enhancement processing can be performed on them to obtain a template enhanced image and a search enhanced image. Specific image enhancement processing methods may include but are not limited to: color enhancement, contrast adjustment, sharpening, denoising, etc. These processing methods can be selected and combined according to specific applications to maximize the visual effect and information expression ability of the image.

[0038] S102, based on the twin network feature extraction module in the trained target tracking model, feature extraction is performed on the template enhanced image to obtain a template enhanced image feature vector; the template enhanced image feature vector includes a preset target object feature vector; and based on the twin network feature extraction module, feature extraction is performed on the search enhanced image to obtain a search enhanced image feature vector.

[0039] Specifically, the target tracking model includes a twin network feature extraction module and an anchor-free frame module; the twin network feature extraction module includes a local feature extraction module, a global feature extraction module, and a feature fusion module, which are mainly used to extract features from images; the anchor-free frame module includes a multi-keypoint extraction module and a multi-keypoint prediction module, which are mainly used to determine the predicted frame position of the target object to be tracked in the search image. Here, the training method of the target tracking model will be described in detail later.

[0040] It is understandable that the twin network in the twin network feature extraction module refers to a neural network structure that processes input data through two parallel branches. The two branches usually share the same weights and architecture to extract and compare input features. In response to the problem of insufficient feature extraction in the backbone network of the twin network in the related art, the embodiment of the present disclosure introduces a large-core convolution attention mechanism into the backbone network of the existing twin network, thereby increasing the receptive field and the range of feature extraction. In this way, the twin network with the addition of large-core convolution can more effectively track targets in complex backgrounds and dynamic scenes because it can accurately identify the target object and ignore irrelevant background.

[0041] Specifically, when the twin network feature extraction module is used to extract features from the template enhanced image and the search enhanced image, the following (a) to (f) may be included:

[0042] (a) extracting local features from the template-enhanced image based on the local feature extraction module to obtain a local feature vector of the template-enhanced image;

[0043] (b) performing global feature extraction on the template enhanced image based on the global feature extraction module to obtain a global feature vector of the template enhanced image;

[0044] (c) performing feature fusion processing on the local feature vector of the template enhanced image and the global feature vector of the template enhanced image based on the feature fusion module to obtain the feature vector of the template enhanced image;

[0045] (d) extracting local features from the search enhanced image based on the local feature extraction module to obtain a local feature vector of the search enhanced image;

[0046] (e) performing global feature extraction on the search enhanced image based on the global feature extraction module to obtain a global feature vector of the search enhanced image;

[0047] (f) Based on the feature fusion module, feature fusion processing is performed on the local feature vector of the search enhanced image and the global feature vector of the search enhanced image to obtain the feature vector of the search enhanced image.

[0048] Here, the large kernel convolution attention is to separate the standard large convolution into local convolution, global convolution and fusion convolution (i.e. local feature extraction module, global feature extraction module and feature fusion module). For example, a standard convolution kernel of size K×K is decomposed into one A local convolution, a (2d-1)×(2d-1) global convolution and a 1×1 fused convolution, where d is the dilation rate, can achieve a wide receptive field while reducing the amount of computation by utilizing three convolutions.

[0049] Among them, the local feature extraction module is used to perform convolution on the channel dimension of the input data to obtain the local feature vector of the image. These local feature vectors capture detailed information of specific areas in the image, such as edges, textures or other local structures; the global feature extraction module captures a larger receptive field by introducing a dilation rate, that is, spanning a farther input area, and extracting global feature vectors. These global feature vectors contain the image semantics and background information of the image as a whole, which can help the model better understand the overall content and background environment of the image. After obtaining the local feature vector and global feature vector of the image, the feature fusion module is used to adaptively learn and selectively focus on the important information in the input and fuse them to obtain a comprehensive image feature vector. In this way, by integrating the local and global information of the image, the model can selectively focus on the important information in the image features to improve the performance and generalization ability of the model.

[0050] S103, calculating a similarity value between the preset target object feature vector and the search enhanced image feature vector based on a mutual convolution method, and judging whether there is a target object to be tracked in the search enhanced image according to the similarity value.

[0051] It can be understood that the mutual convolution method is a method for calculating the similarity between the feature vector of the target object and the feature vector of the search enhanced image. Figure 2 As shown, when calculating the similarity value between the preset target object feature vector and the search enhanced image feature vector to determine whether there is a target object to be tracked in the search enhanced image, the following steps S1031 to S1032 are included:

[0052] S1031, calculating the similarity between the preset target object feature vector and the search enhanced image feature vector based on the mutual convolution formula, and obtaining a similarity response map corresponding to the search enhanced image.

[0053] Here, the mutual convolution operation is to perform a convolution operation on the feature vector of the target object and the feature vector of the search enhanced image, by calculating the element-by-element product between their feature maps and then accumulating these products to calculate the similarity between the two feature vectors. The mutual convolution formula is as follows:

[0054] C(i,j)=Σ m,n F1(m,n)·F2(m+i,n+j);

[0055] Among them, C(i,j) represents the similarity of the search enhanced image F2 at (i,j), F1(m,n) represents the element value at the position (m,n) in the preset target object feature vector F1; F2(m+i,n+j) represents the element value at the corresponding position (m+i,n+j) in the window with (i,j) as the offset on the search enhanced image F2.

[0056] It can be understood that after the similarity between the preset target object feature vector and the search enhanced image feature vector is calculated, a similarity response map is generated. In the similarity response map, each pixel value represents the similarity between the position of each pixel point in the search enhanced image and the preset target object, which is used to determine whether there is a target object to be tracked in the search enhanced image.

[0057] S1032: Taking the maximum similarity value in the similarity response graph as the similarity value, and determining whether the similarity value is greater than a preset value, so as to determine whether there is a target object to be tracked in the search enhanced image.

[0058] Specifically, after obtaining the similarity response map, the maximum similarity value in the similarity response map is obtained as the similarity value, and the similarity value is compared with the preset value. If the similarity value is greater than the preset value, it means that the search enhanced image does contain a target object to be tracked that is the same or similar to the preset target object, and the position corresponding to the similarity value is the estimated position of the target object in the search image. On the other hand, if the similarity value in the response map is not greater than the preset value, or the similarity value is very low, it means that there is no target object to be tracked in the search enhanced image. Here, the preset value can be set according to the actual situation, such as 0.6, 0.8, etc., and is not specifically limited here.

[0059] In this way, the interconvolution method can quantify the similarity between the preset target object feature vector and the search enhanced image feature vector in the feature space, thereby determining whether the current frame image (search image) contains the target object to be tracked.

[0060] S104, when the target object to be tracked exists in the search enhanced image, determine the position information of the target object to be tracked in the search enhanced image based on the anchor-free frame module; and determine the predicted frame position of the target object to be tracked in the search enhanced image based on the position information; when the target object to be tracked does not exist in the search enhanced image, obtain an adjacent image for the search image, and repeat the above steps S101 to S103 using the adjacent image as a new search image until the target object to be tracked is found in the search enhanced image.

[0061] It is understandable that, when there is a target object to be tracked in the search enhanced image (i.e., the similarity value is greater than a preset value), the position information of the target object to be tracked in the search enhanced image is determined based on the multi-keypoint extraction module. Specifically, the following (1) to (2) may be included:

[0062] (1) according to the location of the target pixel corresponding to the similarity value, extracting multiple key points at the location of the target pixel using two sets of deformable convolutions based on the multi-key point extraction module;

[0063] (2) determining a key point set based on the multiple key points, and determining the position information of the target object to be tracked in the search enhanced image based on the anchor-free frame module and the key point set.

[0064] Here, when extracting key points at the locations of target pixels corresponding to similarity values, the present disclosure utilizes two sets of deformable convolutions to adaptively learn and search to enhance the key point information of the target object to be tracked in the image. These key points are pixels with obvious grayscale changes in the area where the target pixels are located in the image. The point set of multiple key points obtained can often describe the outline of an object and distinguish them from the background. Among them, a set of deformable convolutions can learn 9 key points. By splicing the results of two sets of deformable convolutions, multiple key points about the target object to be tracked can be obtained. At the same time, compared with the key point acquisition results obtained using only one set of deformable convolutions, the present disclosure is more obvious when tracking some targets with rich textures or complex target edge shapes, and can better describe the target object to be tracked to improve the accuracy of target tracking.

[0065] For example, after obtaining the key point set, since the positions of some key points in the key point set may deviate from the target object to be tracked, which will affect the subsequent positioning and tracking of the target object to be tracked, when determining the position information of the target object to be tracked in the search enhanced image based on the multi-key point prediction module and the key point set, the following (I) to (III) may be included:

[0066] (I) for each key point in the key point set, based on the multiple key point prediction module, performing regression prediction on a first position corresponding to the key point; and based on the first position and the multiple key point prediction module, performing regression prediction on a second position corresponding to the key point;

[0067] (II) for each key point in the key point set, performing weighted calculation on the first position and the second position corresponding to the key point based on a preset weight coefficient to obtain coordinate information of the key point;

[0068] (III) Determining the position information of the target object to be tracked in the search enhanced image based on the coordinate information of each key point in the key point set.

[0069] It is understandable that the regression prediction of the position of each key point can be based on a linear regression model, a polynomial regression model or a more complex regression model, such as support vector regression (SVR) or neural network regression, which is not specifically limited here. The position predicted by the first regression is taken as the first position, and then the second regression prediction is continued at the first position to obtain the second position, and then the first position and the second position are weighted based on the preset weight coefficient to obtain the coordinate information of the key point. Here, the value of the preset weight coefficient can be expressed as: the weight value corresponding to the first position is set to 0.8, and the weight value corresponding to the second position is set to 0.2; it can also be set to other values, which are not specifically limited here.

[0070] For example, after obtaining the coordinate information of each key point in the key point set, the minimum horizontal coordinate information, the maximum horizontal coordinate information, the minimum vertical coordinate information, and the maximum vertical coordinate information in the coordinate information are first determined, and then the coordinate information of the prediction box of the target object to be tracked is determined based on the minimum horizontal coordinate information, the maximum horizontal coordinate information, the minimum vertical coordinate information, and the maximum vertical coordinate information. That is, the coordinate of the upper left corner of the prediction box position can be expressed as (x min ,y max ), the coordinates of its lower right corner can be expressed as (x max ,y min); and according to the coordinate information of the prediction box, determine the prediction box position of the target object to be tracked in the search enhanced image. In this way, determining the prediction box position of the target object to be tracked according to the minimum and maximum coordinate information in the key point set can effectively improve the accuracy of target tracking and the real-time performance of the system, and is particularly suitable for application scenarios such as video surveillance and autonomous driving that require efficient target positioning.

[0071] It can be understood that when there is no target object to be tracked in the search enhanced image (i.e., the similarity value is not greater than a preset value), an adjacent image for the search image is obtained, where the adjacent image is represented by an image captured after skipping several frames at a frame rate, and the above steps S101 to S103 are repeated using the adjacent image as a new search image until the target object to be tracked is found in the search enhanced image.

[0072] The adaptive twin network target tracking method, apparatus, medium and equipment provided in the embodiments of the present invention improve the accuracy of target tracking by utilizing the twin network feature extraction module to extract richer feature information of the image, and utilize the anchor-free frame module to determine the position information of the key points of the image and the position of the prediction frame, without the need to generate redundant predefined anchor frames, thereby reducing the computational complexity and improving the real-time performance of target tracking.

[0073] The following is a detailed description of the training method of the above target tracking model, see Figure 3 FIG. 1 is a flowchart of a method for training a target tracking model provided by an embodiment of the present disclosure, and the method includes the following S301 to S306:

[0074] S301, obtaining a training sample set and a target tracking model to be trained; the training sample set includes multiple groups of training sample subsets, the training sample subsets include template sample images and search sample images; the template sample images include sample target objects.

[0075] Here, the training sample set disclosed in this paper uses the DET dataset and the VOT2019 dataset. Among them, the DET dataset covers a variety of complex image scenes, such as indoor, outdoor, woodland and city, and contains a variety of target types, such as people, cars, animals, etc., and has complex lighting conditions, weather changes, target occlusion and rotation and other characteristics; this diversity and complexity helps to improve the generalization ability of the model in various real-world scenarios. The VOT2019 dataset is a benchmark dataset widely used for visual target tracking, containing a variety of video clips, covering various scenes, target types, scale changes, occlusion and dynamic changes; these video clips are collected in real time, reflecting the complexity and challenges in actual application scenarios, and provide an important benchmark for evaluating the performance of models in dynamic and changing environments.

[0076] S302, for each group of training sample subsets, feature extraction is performed on the template sample image based on the twin network feature extraction module to obtain a template sample image feature vector, wherein the template sample image feature vector includes a sample target object feature vector; and feature extraction is performed on the search sample image based on the twin network feature extraction module to obtain a search sample image feature vector.

[0077] Here, the specific scheme for extracting features of the image based on the twin network feature extraction module has been described in detail in the above step S102 and will not be repeated here.

[0078] S303: Calculate the sample similarity value between the feature vector of the sample target object and the feature vector of the search sample image based on the mutual convolution method to determine whether there is a sample target object to be tracked in the search sample image.

[0079] Here, the specific scheme of calculating the similarity between vectors based on the mutual convolution method to determine whether there is a sample target object to be tracked has been described in detail in the above step S103, and will not be repeated here.

[0080] S304, when there is a sample target object to be tracked in the search sample image, determine the position information of the sample target object to be tracked in the search sample image based on the anchor-free frame module, and determine the sample prediction frame position of the sample target object to be tracked in the search sample image based on the position information.

[0081] Here, the specific scheme for determining the position information and predicting the frame position based on the anchor-free frame module has been described in detail in the above step S104 and will not be repeated here.

[0082] S305, calculating loss data of the sample prediction box position and the label object box based on the Smooth L1 loss function, and adjusting the target tracking model to be trained based on the loss data.

[0083] Among them, the Smooth L1 loss function is a commonly used loss function in deep learning, especially in target tracking tasks. Compared with the traditional mean square error loss function, it has better robustness when dealing with outliers. The Smooth L1 loss function is used to calculate the loss data of the sample prediction box position and the label object box, and the target tracking model to be trained is adjusted according to the loss data.

[0084] In some other embodiments, the loss function may also include a structural loss function, a perceptual loss function, etc., which are not specifically limited here.

[0085] At the same time, an adaptive weight coefficient update module is also proposed in the embodiment of the present disclosure, that is, after the initial weight coefficient value is given, in the weighted calculation process of the model key point regression prediction, after each iteration or after a certain number of iterations, the nn.parameter function (used to define learnable parameters in deep learning frameworks such as PyTorch) is used to treat the weight coefficient as part of the model parameter, and these weight coefficients are automatically adjusted through the back propagation algorithm. Specifically, the weight coefficient can be updated by optimization algorithms such as gradient descent to minimize the loss value. In this way, the accuracy and robustness of key point prediction can be improved, and the cost of manual intervention and parameter adjustment can also be reduced.

[0086] S306, continuing to train the target tracking model based on the adjusted target tracking model to be trained and the training sample set until the training result meets the preset requirements, thereby obtaining the trained target tracking model.

[0087] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0088] Based on the same inventive concept, an adaptive twin network target tracking device corresponding to the adaptive twin network target tracking method is also provided in an embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned adaptive twin network target tracking method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0089] Reference Figure 4 As shown, it is a schematic diagram of an adaptive twin network target tracking device 400 provided in an embodiment of the present disclosure, and the device includes:

[0090] The image acquisition module 401 is used to execute step 1: acquire a group of images to be tracked, the group of images to be tracked includes a template image and a search image, the template image includes a preset target object; and perform image enhancement processing on the template image and the search image to obtain a group of enhanced images to be tracked; the group of enhanced images to be tracked includes a template enhanced image and a search enhanced image;

[0091] The feature extraction module 402 is used to execute step 2: based on the twin network feature extraction module in the trained target tracking model, feature extraction is performed on the template enhanced image to obtain a template enhanced image feature vector; the template enhanced image feature vector includes a preset target object feature vector; and based on the twin network feature extraction module, feature extraction is performed on the search enhanced image to obtain a search enhanced image feature vector; the target tracking model also includes an anchor-free frame module;

[0092] The object determination module 403 is used to execute step 3: calculate the similarity value between the preset target object feature vector and the search enhanced image feature vector based on the mutual convolution method, and determine whether there is a target object to be tracked in the search enhanced image according to the similarity value;

[0093] The position determination module 404 is used to execute step 4: when the target object to be tracked exists in the search enhanced image, determine the position information of the target object to be tracked in the search enhanced image based on the anchor-free frame module; and determine the predicted frame position of the target object to be tracked in the search enhanced image based on the position information; when the target object to be tracked does not exist in the search enhanced image, obtain an adjacent image for the search image, and repeat steps 1 to 3 above using the adjacent image as a new search image until the target object to be tracked is found in the search enhanced image.

[0094] In some possible embodiments, the twin network feature extraction module includes a local feature extraction module, a global feature extraction module and a feature fusion module; the feature extraction module 402 is specifically used for:

[0095] Performing local feature extraction on the template enhanced image based on the local feature extraction module to obtain a local feature vector of the template enhanced image;

[0096] Performing global feature extraction on the template enhanced image based on the global feature extraction module to obtain a global feature vector of the template enhanced image;

[0097] Based on the feature fusion module, feature fusion processing is performed on the local feature vector of the template enhanced image and the global feature vector of the template enhanced image to obtain the feature vector of the template enhanced image;

[0098] Accordingly, the feature extraction module 402 is specifically used for:

[0099] Performing local feature extraction on the search enhanced image based on the local feature extraction module to obtain a local feature vector of the search enhanced image;

[0100] Performing global feature extraction on the search enhanced image based on the global feature extraction module to obtain a global feature vector of the search enhanced image;

[0101] Based on the feature fusion module, feature fusion processing is performed on the local feature vector of the search enhanced image and the global feature vector of the search enhanced image to obtain the feature vector of the search enhanced image.

[0102] In some possible embodiments, the object determination module 403 is specifically used to:

[0103] Calculate the similarity between the feature vector of the preset target object and the feature vector of the search enhanced image based on the mutual convolution formula to obtain a similarity response map corresponding to the search enhanced image; the similarity response map includes the similarity between the position of each pixel point in the search enhanced image and the preset target object;

[0104] Taking the maximum similarity value in the similarity response graph as the similarity value, and judging whether the similarity value is greater than a preset value, so as to judge whether there is a target object to be tracked in the search enhanced image;

[0105] The interconvolution formula includes:

[0106] C(i,j)=Σ m,n F1(m,n)·F2(m+i,n+j);

[0107] Among them, C(i,j) represents the similarity of the search enhanced image F2 at (i,j), F1(m,n) represents the element value at the position (m,n) in the preset target object feature vector F1; F2(m+i,n+j) represents the element value at the corresponding position (m+i,n+j) in the window with (i,j) as the offset on the search enhanced image F2.

[0108] In some possible embodiments, the anchor-free frame module includes a multi-keypoint extraction module; the position determination module 404 is specifically used for:

[0109] According to the location of the target pixel corresponding to the similarity value, extracting multiple key points at the location of the target pixel using two sets of deformable convolutions based on the multi-key point extraction module;

[0110] A key point set is determined based on the multiple key points, and position information of the target object to be tracked in the search enhanced image is determined based on the anchor-free frame module and the key point set.

[0111] In some possible embodiments, the anchor-free frame module further includes a multi-keypoint prediction module; and the position determination module is specifically used for:

[0112] For each key point in the key point set, based on the multiple key point prediction module, a first position corresponding to the key point is regressed and predicted; and based on the first position and the multiple key point prediction module, a second position corresponding to the key point is regressed and predicted;

[0113] For each key point in the key point set, weighted calculation is performed on the first position and the second position corresponding to the key point based on a preset weight coefficient to obtain coordinate information of the key point;

[0114] The position information of the target object to be tracked in the search enhanced image is determined based on the coordinate information of each key point in the key point set.

[0115] In some possible embodiments, the location determination module 404 is further configured to:

[0116] According to the coordinate information of each key point in the key point set, the minimum horizontal coordinate information, the maximum horizontal coordinate information, the minimum vertical coordinate information, and the maximum vertical coordinate information in the coordinate information are determined; based on the minimum horizontal coordinate information, the maximum horizontal coordinate information, the minimum vertical coordinate information, and the maximum vertical coordinate information, the coordinate information of the prediction box of the target object to be tracked is determined; and according to the coordinate information of the prediction box, the prediction box position of the target object to be tracked in the search enhanced image is determined.

[0117] In some possible embodiments, reference Figure 5 As shown, the device 400 also includes:

[0118] The tracking model preparation module 405 is used to obtain a training sample set and a target tracking model to be trained; the training sample set includes multiple sets of training sample subsets, the training sample subsets include template sample images and search sample images; the template sample images include sample target objects;

[0119] The sample feature extraction module 406 is used to extract features of the template sample image based on the twin network feature extraction module for each set of training sample subsets to obtain a template sample image feature vector, wherein the template sample image feature vector includes a sample target object feature vector; and to extract features of the search sample image based on the twin network feature extraction module to obtain a search sample image feature vector;

[0120] The sample object determination module 407 is used to calculate the sample similarity value between the sample target object feature vector and the search sample image feature vector based on the mutual convolution method to determine whether there is a sample target object to be tracked in the search sample image;

[0121] A prediction frame position determination module 408 is used to determine the position information of the sample target object to be tracked in the search sample image based on the anchor-free frame module when there is a sample target object to be tracked in the search sample image, and determine the sample prediction frame position of the sample target object to be tracked in the search sample image based on the position information;

[0122] A model loss calculation module 409, used to calculate the loss data of the sample prediction box position and the label object box based on the Smooth L1 loss function, and adjust the target tracking model to be trained based on the loss data;

[0123] The tracking model training module 410 is used to continue training the target tracking model based on the adjusted target tracking model to be trained and the training sample set until the training result meets the preset requirements, thereby obtaining the trained target tracking model.

[0124] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Figure 6 , which is a schematic diagram of the structure of a computer device 600 provided in an embodiment of the present disclosure, including a processor 601, a memory 602, and a bus 603. The memory 602 is used to store execution instructions, including a memory 6021 and an external memory 6022; the memory 6021 is also called an internal memory, which is used to temporarily store the operation data in the processor 601 and the data exchanged with the external memory 6022 such as a hard disk. The processor 601 exchanges data with the external memory 6022 through the memory 6021.

[0125] In the embodiment of the present application, the memory 602 is specifically used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 601. That is, when the computer device 600 is running, the processor 601 communicates with the memory 602 through the bus 603, so that the processor 601 executes the application code stored in the memory 602, and then executes the method described in any of the above embodiments.

[0126] Among them, the memory 602 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0127] Processor 601 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0128] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the computer device 600. In other embodiments of the present application, the computer device 600 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0129] The present disclosure also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the adaptive twin network target tracking method described in the above method embodiment are executed. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0130] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the adaptive twin network target tracking method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0131] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0132] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0133] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0135] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0136] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An adaptive twin network target tracking method, characterized in that: include: Step 1: obtaining a group of images to be tracked, wherein the group of images to be tracked includes a template image and a search image, and the template image includes a preset target object; and performing image enhancement processing on the template image and the search image to obtain an enhanced image group to be tracked; The enhanced image group to be tracked includes a template enhanced image and a search enhanced image; Step 2: Based on the twin network feature extraction module in the trained target tracking model, feature extraction is performed on the template enhanced image to obtain a template enhanced image feature vector; the template enhanced image feature vector includes a preset target object feature vector; and based on the twin network feature extraction module, feature extraction is performed on the search enhanced image to obtain a search enhanced image feature vector; the target tracking model also includes an anchor-free frame module; Step 3: Calculate the similarity value between the preset target object feature vector and the search enhanced image feature vector based on the mutual convolution method, and determine whether there is a target object to be tracked in the search enhanced image according to the similarity value; Step 4: when the target object to be tracked exists in the search enhanced image, determine the position information of the target object to be tracked in the search enhanced image based on the anchor-free frame module; and determine the predicted frame position of the target object to be tracked in the search enhanced image based on the position information; When the target object to be tracked does not exist in the search enhanced image, an adjacent image for the search image is obtained, and the adjacent image is used as a new search image to repeat steps 1 to 3 until the target object to be tracked is found in the search enhanced image.

2. The method according to claim 1, characterized in that The twin network feature extraction module includes a local feature extraction module, a global feature extraction module and a feature fusion module; the twin network feature extraction module based on the trained target tracking model performs feature extraction on the template enhanced image, including: Performing local feature extraction on the template enhanced image based on the local feature extraction module to obtain a local feature vector of the template enhanced image; Performing global feature extraction on the template enhanced image based on the global feature extraction module to obtain a global feature vector of the template enhanced image; Based on the feature fusion module, feature fusion processing is performed on the local feature vector of the template enhanced image and the global feature vector of the template enhanced image to obtain the feature vector of the template enhanced image; Accordingly, the feature extraction of the search enhanced image based on the twin network feature extraction module includes: Performing local feature extraction on the search enhanced image based on the local feature extraction module to obtain a local feature vector of the search enhanced image; Performing global feature extraction on the search enhanced image based on the global feature extraction module to obtain a global feature vector of the search enhanced image; Based on the feature fusion module, feature fusion processing is performed on the local feature vector of the search enhanced image and the global feature vector of the search enhanced image to obtain the feature vector of the search enhanced image.

3. The method according to claim 1, characterized in that The method of calculating the similarity value between the preset target object feature vector and the search enhanced image feature vector based on the interconvolution method, and judging whether there is a target object to be tracked in the search enhanced image according to the similarity value, comprises: Calculate the similarity between the feature vector of the preset target object and the feature vector of the search enhanced image based on the mutual convolution formula to obtain a similarity response map corresponding to the search enhanced image; the similarity response map includes the similarity between the position of each pixel point in the search enhanced image and the preset target object; Taking the maximum similarity value in the similarity response graph as the similarity value, and judging whether the similarity value is greater than a preset value, so as to judge whether there is a target object to be tracked in the search enhanced image; The interconvolution formula includes: C(i,j)=∑ m,n F1(m,n)·F2(m+i,n+j); Among them, C(i,j) represents the similarity of the search enhanced image F2 at (i,j), F1(m,n) represents the element value at the position (m,n) in the preset target object feature vector F1; F2(m+i,n+j) represents the element value at the corresponding position (m+i,n+j) in the window with (i,j) as the offset on the search enhanced image F2.

4. The method according to claim 3, characterized in that The anchor-free frame module includes a multi-keypoint extraction module; the determining of the position information of the target object to be tracked in the search enhanced image based on the anchor-free frame module includes: According to the location of the target pixel corresponding to the similarity value, extracting multiple key points at the location of the target pixel using two sets of deformable convolutions based on the multi-key point extraction module; A key point set is determined based on the multiple key points, and position information of the target object to be tracked in the search enhanced image is determined based on the anchor-free frame module and the key point set.

5. The method according to claim 4, characterized in that The anchor-free frame module further includes a multi-keypoint prediction module; and determining the position information of the target object to be tracked in the search enhanced image based on the anchor-free frame module and the key point set includes: For each key point in the key point set, based on the multiple key point prediction module, a first position corresponding to the key point is regressed and predicted; and based on the first position and the multiple key point prediction module, a second position corresponding to the key point is regressed and predicted; For each key point in the key point set, weighted calculation is performed on the first position and the second position corresponding to the key point based on a preset weight coefficient to obtain coordinate information of the key point; The position information of the target object to be tracked in the search enhanced image is determined based on the coordinate information of each key point in the key point set.

6. The method according to claim 5, characterized in that The step of determining the predicted box position of the target object to be tracked in the search enhanced image based on the position information comprises: According to the coordinate information of each key point in the key point set, the minimum horizontal coordinate information, the maximum horizontal coordinate information, the minimum vertical coordinate information, and the maximum vertical coordinate information in the coordinate information are determined; based on the minimum horizontal coordinate information, the maximum horizontal coordinate information, the minimum vertical coordinate information, and the maximum vertical coordinate information, the coordinate information of the prediction box of the target object to be tracked is determined; and according to the coordinate information of the prediction box, the prediction box position of the target object to be tracked in the search enhanced image is determined.

7. The method according to any one of claims 1 to 6, characterized in that: The target tracking model is trained by the following steps: Acquire a training sample set and a target tracking model to be trained; the training sample set includes multiple groups of training sample subsets, the training sample subsets include template sample images and search sample images; the template sample images include sample target objects; For each set of training sample subsets, feature extraction is performed on the template sample image based on the twin network feature extraction module to obtain a template sample image feature vector, wherein the template sample image feature vector includes a sample target object feature vector; and feature extraction is performed on the search sample image based on the twin network feature extraction module to obtain a search sample image feature vector; Calculating a sample similarity value between the feature vector of the sample target object and the feature vector of the search sample image based on a mutual convolution method to determine whether there is a sample target object to be tracked in the search sample image; In the case where there is a sample target object to be tracked in the search sample image, determining the position information of the sample target object to be tracked in the search sample image based on the anchor-free frame module, and determining the sample prediction frame position of the sample target object to be tracked in the search sample image based on the position information; Calculating loss data of the sample prediction box position and the label object box based on the Smooth L1 loss function, and adjusting the target tracking model to be trained based on the loss data; The target tracking model is continuously trained based on the adjusted target tracking model to be trained and the training sample set until the training result meets the preset requirements, thereby obtaining the trained target tracking model.

8. An adaptive twin network target tracking device, characterized in that: include: The image acquisition module is used to execute step 1: acquire a group of images to be tracked, wherein the group of images to be tracked includes a template image and a search image, wherein the template image includes a preset target object; and perform image enhancement processing on the template image and the search image to obtain an enhanced image group to be tracked; The enhanced image group to be tracked includes a template enhanced image and a search enhanced image; A feature extraction module is used to perform step 2: based on the twin network feature extraction module in the trained target tracking model, feature extraction is performed on the template enhanced image to obtain a template enhanced image feature vector; the template enhanced image feature vector includes a preset target object feature vector; and based on the twin network feature extraction module, feature extraction is performed on the search enhanced image to obtain a search enhanced image feature vector; the target tracking model also includes an anchor-free frame module; The object judgment module is used to execute step 3: calculate the similarity value between the preset target object feature vector and the search enhanced image feature vector based on the mutual convolution method, and judge whether there is a target object to be tracked in the search enhanced image according to the similarity value; a position determination module, configured to execute step 4: determining, when the target object to be tracked exists in the search enhanced image, the position information of the target object to be tracked in the search enhanced image based on the anchor-free frame module; and determining the predicted frame position of the target object to be tracked in the search enhanced image based on the position information; When the target object to be tracked does not exist in the search enhanced image, an adjacent image for the search image is obtained, and the adjacent image is used as a new search image to repeat steps 1 to 3 until the target object to be tracked is found in the search enhanced image.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images