Adaptive twin network target tracking method, device, medium and equipment

Abstract

The present disclosure provides a kind of adaptive twin network target tracking method, device, medium and equipment, it is related to target tracking technical field, the method comprises: first, obtain the image group to be tracked, secondly, search image and the template image including preset target object are enhanced processing;Then, using the twin network feature extraction module in the target tracking model trained, the template image and search image after enhancement are carried out feature extraction, and the similarity between target object feature vector and search image feature vector is calculated, to determine whether the target object to be tracked exists in search image;If the target is not found, the adjacent image will be used as search image to repeat the above processing steps, until successfully find the target object to be tracked, and the position information is determined using anchor frameless module, to determine the predicted frame position of the target object to be tracked in search image.The present disclosure embodiment improves the accuracy and real-time performance of target tracking.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of target tracking, in particular, to a self-adaptive twin network target tracking method, device, medium and equipment. BACKGROUND

[0002] As a key technology in the field of artificial intelligence, target tracking is driven by technologies such as the Internet of Things, big data and cloud computing, and its application scope is expanding, and its technical requirements are also increasing. Related target tracking methods such as Kalman filtering and particle filtering are widely used in specific scenarios, but they are sensitive to complex backgrounds and occlusions, and have limited generalization ability.

[0003] With the rise of deep learning methods, target tracking technology has made significant progress. However, some target tracking algorithms proposed in related technologies require dense region extraction of the entire image during the target object bounding box generation stage, generating multiple predefined anchor boxes; then, the target tracking is completed by performing feature calculation on the image. In this way, the existing target tracking algorithm may have the problem of high computational complexity, and its tracking accuracy is low when facing complex scenes. SUMMARY

[0004] The embodiments of the present disclosure at least provide a self-adaptive twin network target tracking method, device, medium and equipment, which extracts more rich feature information of the image by using a twin network feature extraction module, improves the accuracy of target tracking, and determines the position information of the image key points and the prediction box position by using an anchor-free box module, without generating redundant predefined anchor boxes, thereby reducing the computational complexity and improving the real-time performance of target tracking.

[0005] The embodiments of the present disclosure provide a self-adaptive twin network target tracking method, comprising:

[0006] Step 1: obtaining a target tracking image group, the target tracking image group comprising a template image and a search image, the template image comprising a preset target object; and performing image enhancement processing on the template image and the search image to obtain a target tracking enhanced image group; the target tracking enhanced image group comprising a template enhanced image and a search enhanced image;

[0007] Step 2: performing feature extraction on the template enhanced image based on a twin network feature extraction module in a trained target tracking model to obtain a template enhanced image feature vector; the template enhanced image feature vector comprising a preset target object feature vector; and performing feature extraction on the search enhanced image based on the twin network feature extraction module to obtain a search enhanced image feature vector; the target tracking model further comprising an anchor-free box module;

[0008] Step 3: calculating a similarity value between the preset target object feature vector and the search enhanced image feature vector based on a cross-convolution method, and determining whether the search enhanced image contains the target object to be tracked according to the similarity value;

[0009] Step 4: in the case that the search enhanced image contains the target object to be tracked, determining the position information of the target object to be tracked in the search enhanced image based on the anchor-free box module, and determining the predicted box position of the target object to be tracked in the search enhanced image based on the position information; in the case that the search enhanced image does not contain the target object to be tracked, obtaining a neighboring image of the search image, and repeating steps 1-3 above with the neighboring image as a new search image until the target object to be tracked is found in the search enhanced image.

[0010] The embodiment of the present disclosure provides a self-adaptive twin network target tracking device, which comprises:

[0011] An image acquisition module is configured to perform step 1: acquiring a target image group, wherein the target image group comprises a template image and a search image, the template image comprises a preset target object, and the template image and the search image are subjected to image enhancement processing to obtain a target enhanced image group, wherein the target enhanced image group comprises a template enhanced image and a search enhanced image.

[0012] A feature extraction module is configured to perform step 2: extracting features of the template enhanced image based on a twin network feature extraction module in a trained target tracking model to obtain a template enhanced image feature vector, wherein the template enhanced image feature vector comprises a preset target object feature vector; and extracting features of the search enhanced image based on the twin network feature extraction module to obtain a search enhanced image feature vector; the target tracking model further comprises an anchor-free box module.

[0013] An object judgment module is configured to perform step 3: calculating a similarity value between the preset target object feature vector and the search enhanced image feature vector based on a cross-convolution method, and determining whether the search enhanced image contains the target object to be tracked according to the similarity value.

[0014] The position determination module is configured to perform step 4: in a case where the target object to be tracked exists in the search enhanced image, determining position information of the target object to be tracked in the search enhanced image based on the anchor-free box module; and determining a predicted box position of the target object to be tracked in the search enhanced image based on the position information; in a case where the target object to be tracked does not exist in the search enhanced image, obtaining a neighboring image of the search image, and repeating steps 1-3 above with the neighboring image as a new search image until the target object to be tracked is found in the search enhanced image.

[0015] The disclosure provides a computer device, comprising a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the computer device is running, the processor and the memory communicate through the bus, the machine readable instructions are executed by the processor to perform the adaptive twin network target tracking method as described in any possible implementation manner.

[0016] The disclosure provides a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is run by the processor to realize the adaptive twin network target tracking method as described in any possible implementation manner.

[0017] The adaptive twin network target tracking method, device, medium and equipment provided in the disclosure can extract more rich feature information of an image by using a twin network feature extraction module, improve the accuracy of target tracking, determine the position information and the predicted box position of key points of an image by using an anchor-free box module, do not need to generate a redundant predefined anchor box, reduce the calculation complexity, and thus improve the real-time performance of target tracking.

[0018] In order to make the above-mentioned objectives, features and advantages of the disclosure more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the disclosure, the drawings referred to in the embodiments will be briefly introduced below, the drawings herein are incorporated into the specification and form a part of the specification, the drawings show the embodiments consistent with the disclosure, and are used to illustrate the technical solutions of the disclosure together with the specification. It should be understood that the following drawings only show some embodiments of the disclosure, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0020] Figure 1A flow chart of a self-adaptive twin network target tracking method is shown.

[0021] Figure 2 A flow chart of a similarity value calculation method in the self-adaptive twin network target tracking method is shown.

[0022] Figure 3 A flow chart of a target tracking model training method in the self-adaptive twin network target tracking method is shown.

[0023] Figure 4 A structural schematic diagram of a self-adaptive twin network target tracking device is shown.

[0024] Figure 5 A structural schematic diagram of another self-adaptive twin network target tracking device is shown.

[0025] Figure 6 A structural schematic diagram of a computer device is shown. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the following will be combined with the accompanying drawings for the embodiments of the present disclosure to make a clear and complete description of the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure and not all the embodiments. The components of the embodiments of the present disclosure described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed present disclosure, but only represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0027] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0028] The term "and / or" in this document is only to describe an association relationship, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" in this document means any one of the plurality or any combination of at least two of the plurality, for example, including at least one of A, B and C can mean including any one or more elements selected from the set consisting of A, B and C.

[0029] Target tracking, as a key technology in the field of artificial intelligence, is driven by technologies such as the Internet of Things, big data, and cloud computing, and its application scope is expanding. The related target tracking methods such as Kalman filtering and particle filtering are widely used in specific scenarios, but they are sensitive to complex backgrounds and occlusions, and have limited generalization ability.

[0030] It is found through research that the deep learning method is applied to the target tracking technology in the related technology, and significant progress has been made. However, there are still some problems in the target tracking algorithm using deep learning method, for example, the SORT algorithm based on deep learning combines a lightweight target detector (such as YOLO) to achieve efficient target tracking. However, this method limits the ability to capture complex target features due to the reliance on simple feature extraction. Further, the DeepSORT introduces a deep neural network to improve the tracking accuracy in complex scenes, but the computational complexity and real-time problem still need to be solved. In addition, the ATOM algorithm improves the target feature representation through adaptive target-oriented metric learning, but it introduces a complex background modeling and model updating mechanism, which reduces the real-time performance. The FairMOT algorithm improves the detection and tracking ability in complex scenes through joint training, but increases the computational complexity of the algorithm. Therefore, although many new algorithms such as SiamFC and SiamRPN combine the advantages of deep learning and traditional methods, there are still challenges in accuracy, robustness, and real-time performance when facing complex scenarios, and further optimization is needed.

[0031] Based on the above research, an adaptive twin network target tracking method, device, medium and equipment are provided in the embodiments of the present disclosure. First, the search image and the template image including the preset target object are enhanced. Then, the twin network feature extraction module in the trained target tracking model is used to extract the features of the enhanced template image and the search image, and calculate the similarity between the target object feature vector and the search image feature vector to determine whether the target object to be tracked exists in the search image. If the target is not found, the above processing steps will be repeated using the adjacent image as the search image until the target object to be tracked is successfully found, and the anchor-free module is used to determine the position information of the target object to be tracked, and then determine the predicted box position of the target object to be tracked in the search image.

[0032] In the embodiments of the present disclosure, the twin network feature extraction module is used to extract more rich feature information of the image, which improves the accuracy of target tracking, and the anchor-free module is used to determine the position information of the image key points and the predicted box position, without generating redundant pre-defined anchor boxes, which reduces the computational complexity and improves the real-time performance of target tracking.

[0033] To facilitate the understanding of the present embodiment, first, the execution subject of the adaptive twin network target tracking method provided by the present embodiment is described in detail. The execution subject of the adaptive twin network target tracking method provided by the present embodiment is a computer device. The computer device can be a server. The server can be a physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud storage, big data and artificial intelligence platform, etc.

[0034] The adaptive twin network target tracking method provided by the present embodiment will be described in detail below with reference to the accompanying drawings. Referring to FIG. 1, which is a flowchart of an adaptive twin network target tracking method provided by the present embodiment, the method comprises the following S101-S105: Figure 1

[0035] S101, obtaining a to-be-tracked image group, the to-be-tracked image group comprising a template image and a search image, the template image comprising a preset target object; and performing image enhancement processing on the template image and the search image to obtain a to-be-tracked enhanced image group; the to-be-tracked enhanced image group comprising a template enhanced image and a search enhanced image.

[0036] It can be understood that the template image and the search image belong to the same video segment. The template image is a specific frame artificially selected from the video, which contains the target person to be tracked; the search image is a frame image taken at different frame rates or time intervals from the same video segment, which is used to find and track the target object selected in the template image. Here, in the present embodiment, the template image also includes a part of background information, which can provide necessary context information for model tracking, help to understand the changes and positions of the target object in different backgrounds, reduce false detection and enhance the stability of tracking, and is conducive to improving the robustness and accuracy of subsequent model tracking.

[0037] Exemplarily, after obtaining the template image and the search image, image enhancement processing can be performed thereon to obtain the template enhanced image and the search enhanced image. The specific image enhancement processing method can include but is not limited to color enhancement, contrast adjustment, sharpening, noise reduction, etc. These processing methods can be selected and combined according to specific applications to maximize the visual effect and information expression ability of the image.

[0038] ​S102, feature extraction is performed on the template enhanced image based on a twin network feature extraction module in the trained target tracking model, to obtain a template enhanced image feature vector; the template enhanced image feature vector includes a preset target object feature vector; and feature extraction is performed on the search enhanced image based on the twin network feature extraction module, to obtain a search enhanced image feature vector.

[0039] Specifically, the target tracking model includes a twin network feature extraction module and an anchor-free module; the twin network feature extraction module includes a local feature extraction module, a global feature extraction module and a feature fusion module, and is mainly used for feature extraction processing of an image; the anchor-free module includes a multi-keypoint extraction module and a multi-keypoint prediction module, and is mainly used for determining a predicted bounding box position of a target object to be tracked existing in a search image. Here, the training method of the target tracking model will be described in detail later.

[0040] It can be understood that the twin network in the twin network feature extraction module refers to a neural network structure that processes input data through two parallel branches, and the two branches usually share the same weights and architecture for extracting and comparing the features of the input. In view of the insufficient feature extraction of the backbone network of the related art twin sub-network, the present embodiment introduces a large kernel convolution attention mechanism into the backbone network of the existing twin network, thereby increasing the receptive field and the range of feature extraction. In this way, the twin network with large kernel convolution can more effectively track the target in a complex background and dynamic scene, because it can accurately identify the target object and ignore the irrelevant background.

[0041] Specifically, when the template enhanced image and the search enhanced image are feature-extracted by the twin network feature extraction module, the following (a) to (f) can be included:

[0042] (a) performing local feature extraction on the template enhanced image based on the local feature extraction module to obtain a template enhanced image local feature vector;

[0043] (b) performing global feature extraction on the template enhanced image based on the global feature extraction module to obtain a template enhanced image global feature vector;

[0044] (c) performing feature fusion processing on the template enhanced image local feature vector and the template enhanced image global feature vector based on the feature fusion module to obtain the template enhanced image feature vector;

[0045] (d) performing local feature extraction on the search enhanced image based on the local feature extraction module to obtain a search enhanced image local feature vector;

[0046] (e) Based on the global feature extraction module, global feature extraction is performed on the search enhancement image to obtain the global feature vector of the search enhancement image;

[0047] (f) Based on the feature fusion module, the local feature vector and the global feature vector of the search-enhanced image are fused to obtain the feature vector of the search-enhanced image.

[0048] Here, large kernel convolutional attention decomposes the standard large convolution into local convolution, global convolution, and fusion convolution (i.e., local feature extraction module, global feature extraction module, and feature fusion module). For example, a standard K×K convolutional kernel is decomposed into a single kernel. The three convolutions—a local convolution, a (2d-1)×(2d-1) global convolution, and a 1×1 fusion convolution, where d is the dilation rate—can achieve a wide receptive field while reducing computational cost.

[0049] The local feature extraction module performs convolution along the channel dimension of the input data to obtain local feature vectors of the image. These local feature vectors capture detailed information about specific regions in the image, such as edges, textures, or other local structures. The global feature extraction module introduces dilation to capture a larger receptive field, i.e., spanning a more distant input region, and extracts global feature vectors. These global feature vectors contain overall image semantics and background information, helping the model better understand the overall content and background environment of the image. After obtaining the local and global feature vectors, the feature fusion module adaptively learns and selectively focuses on important information in the input and fuses them to obtain a comprehensive image feature vector. In this way, by integrating the local and global information of the image, the model can selectively focus on important information in the image features, thereby improving the model's performance and generalization ability.

[0050] S103, calculate the similarity value between the feature vector of the preset target object and the feature vector of the search enhancement image based on the cross-convolution method, and determine whether there is a target object to be tracked in the search enhancement image based on the similarity value.

[0051] Understandably, the cross-convolution method is a way to calculate the similarity between the feature vector of the target object and the feature vector of the search-enhanced image. Specifically, refer to... Figure 2 As shown, when calculating the similarity value between the feature vector of the preset target object and the feature vector of the search enhanced image to determine whether there is a target object to be tracked in the search enhanced image, the following steps are included: S1031 to S1032:

[0052] S1031, calculate the similarity between the preset target object feature vector and the search enhanced image feature vector based on a cross convolution formula, to obtain a similarity response map corresponding to the search enhanced image.

[0053] Here, the cross convolution operation is a convolution operation on the feature vector of the target object and the feature vector of the search enhanced image, the similarity between the two feature vectors is calculated by calculating the element-by-element product between their feature mappings and then accumulating the products, and the cross convolution formula is as follows:

[0054] C(i,j) =∑ m,n F1(m,n)·F2(m+i,n+j);

[0055] Wherein, C(i,j) represents the similarity of the search enhanced image F2 at (i,j), F1(m,n) represents the element value of the preset target object feature vector F1 at (m,n); F2(m+i,n+j) represents the element value at the corresponding position (m+i,n+j) in the window with the offset (i,j) on the search enhanced image F2.

[0056] It can be understood that after the similarity between the preset target object feature vector and the search enhanced image feature vector is calculated, a similarity response map is generated, and each pixel value in the similarity response map represents the similarity between the position of each pixel point in the search enhanced image and the preset target object, which is used to determine whether the target object to be tracked exists in the search enhanced image.

[0057] S1032, take the maximum value of the similarity in the similarity response map as the similarity value, and determine whether the similarity value is greater than a preset value to determine whether the target object to be tracked exists in the search enhanced image.

[0058] Specifically, after obtaining the similarity response map, the maximum value of the similarity in the similarity response map is obtained as the similarity value, and the similarity value is compared with the preset value. If the similarity value is greater than the preset value, it means that the search enhanced image indeed contains the target object to be tracked which is the same as or similar to the preset target object, and the position corresponding to the similarity value is the estimated position of the target object in the search image. On the other hand, if the similarity value in the response map is not greater than the preset value, or the value of the similarity is very low, it means that the target object to be tracked does not exist in the search enhanced image. Here, the preset value can be set according to the actual situation, for example, 0.6, 0.8, etc., which is not limited here.

[0059] Thus, the cross-convolution method can quantify the similarity between the preset target object feature vector and the search enhanced image feature vector in the feature space, so as to determine whether the target object to be tracked is contained in the current frame image (search image).

[0060] S104, in the case where the target object to be tracked exists in the search enhanced image, determining the position information of the target object to be tracked in the search enhanced image based on the anchor-free box module; and determining the predicted box position of the target object to be tracked in the search enhanced image based on the position information; in the case where the target object to be tracked does not exist in the search enhanced image, obtaining a neighboring image corresponding to the search image, and repeating the above steps S101-S103 with the neighboring image as a new search image until the target object to be tracked is found in the search enhanced image.

[0061] It can be understood that, in the case where the target object to be tracked exists in the search enhanced image (i.e., the similarity value is greater than the preset value), the position information of the target object to be tracked in the search enhanced image is determined based on the multi-key point extraction module. Specifically, it can include the following (1)-(2):

[0062] (1) Based on the position of the target pixel point corresponding to the similarity value, the multi-key point extraction module uses two groups of deformable convolution to extract a plurality of key points at the position of the target pixel point;

[0063] (2) determining a key point set based on the plurality of key points, and determining the position information of the target object to be tracked in the search enhanced image based on the anchor-free box module and the key point set.

[0064] Here, when extracting key points at the position of the target pixel point corresponding to the similarity value, the present disclosure uses two groups of deformable convolution to adaptively learn the key point information of the target object to be tracked in the search enhanced image. These key points are pixel points with obvious gray scale changes in the region where the target pixel point is located. The point set of the obtained multi-key points can often describe the outline of an object and distinguish them from the background. One group of deformable convolution can learn 9 key points. By splicing the results of the two groups of deformable convolution, the multi-key points of the target object to be tracked can be obtained. Compared with the key point acquisition result of using only one group of deformable convolution, the present disclosure is more obvious when tracking some targets with rich texture or complex target edge shape, and can better describe the target object to be tracked to improve the accuracy of target tracking.

[0065] Exemplarily, after obtaining the key point set, since the positions of some key points in the key point set may deviate from the to-be-tracked target object, which will affect the subsequent positioning and tracking of the to-be-tracked target object, when determining the position information of the to-be-tracked target object in the search enhanced image based on the multi-key point prediction module and the key point set, the following (I)-(III) can be included:

[0066] (I) For each key point in the key point set, based on the multi-key point prediction module, a first position corresponding to the key point is predicted by regression; and based on the first position and the multi-key point prediction module, a second position corresponding to the key point is predicted by regression;

[0067] (II) For each key point in the key point set, the first position and the second position corresponding to the key point are weighted calculated based on a preset weight coefficient to obtain coordinate information of the key point;

[0068] (III) Based on the coordinate information of each key point in the key point set, the position information of the to-be-tracked target object in the search enhanced image is determined.

[0069] It can be understood that the regression prediction of the position of each key point can be based on a linear regression model, a polynomial regression model or a more complex regression model, such as support vector regression (SVR) or neural network regression, which is not specifically limited here. The position predicted by the first regression prediction is taken as the first position, and then the second regression prediction is continued based on the first position to obtain the second position, and then the first position and the second position are weighted calculated based on a preset weight coefficient to obtain the coordinate information of the key point. Here, the value of the preset weight coefficient can be represented as: the weight value corresponding to the first position is set to 0.8, and the weight value corresponding to the second position is set to 0.2; it can also be set to other values, which are not specifically limited here.

[0070] Exemplarily, after obtaining the coordinate information of each key point in the key point set, the minimum horizontal coordinate information, the maximum horizontal coordinate information, the minimum vertical coordinate information and the maximum vertical coordinate information in the coordinate information are determined, and then based on the minimum horizontal coordinate information, the maximum horizontal coordinate information, the minimum vertical coordinate information and the maximum vertical coordinate information, the coordinate information of the prediction box of the to-be-tracked target object is determined, i.e. the top-left corner coordinate of the prediction box position can be represented as (x min ,y max ), and the bottom-right corner coordinate can be represented as (x max ,y min) and determine the prediction box position of the target object to be tracked in the search enhanced image according to the coordinate information of the prediction box. In this way, the prediction box position of the target object to be tracked is determined according to the minimum and maximum coordinate information in the key point set, which can effectively improve the accuracy of target tracking and the real-time performance of the system, and is particularly suitable for application scenarios such as video monitoring and automatic driving that require efficient target positioning.

[0071] It can be understood that, in the case that the target object to be tracked does not exist in the search enhanced image (i.e., the similarity value is not greater than the preset value), an adjacent image for the search image is obtained, where the adjacent image is an image collected by skipping a plurality of frames according to a frame rate, and the above steps S101-S103 are repeated with the adjacent image as a new search image until the target object to be tracked is found in the search enhanced image.

[0072] The adaptive twin network target tracking method, device, medium and equipment provided in the embodiments of the present disclosure improve the accuracy of target tracking by extracting more rich feature information of the image using the twin network feature extraction module, and determine the position information of the image key points and the prediction box position using the anchor-free box module, without generating redundant pre-defined anchor boxes, thereby reducing the computational complexity and improving the real-time performance of target tracking.

[0073] The training method of the target tracking model will be described in detail below. Referring to FIG. 3, Figure 3 The method includes the following S301-S306:

[0074] S301, obtain a training sample set and a target tracking model to be trained; the training sample set includes a plurality of training sample subsets, and each training sample subset includes a template sample image and a search sample image; and the template sample image includes a sample target object.

[0075] Here, the training sample set of the present disclosure uses the DET dataset and the VOT2019 dataset. The DET dataset covers a variety of complex image scenes, such as indoor, outdoor, forest and city, and contains a variety of target types, such as people, vehicles, animals, etc., and has complex lighting conditions, weather changes, target occlusion and rotation, etc. Such diversity and complexity help to improve the generalization ability of the model in various real-world scenarios. The VOT2019 dataset is a widely used benchmark dataset for visual target tracking, which contains a variety of video clips, covering various scenes, target types, scale changes, occlusions and dynamic changes. These video clips are collected in real time, reflecting the complexity and challenges in real-world scenarios, and providing an important benchmark for evaluating the performance of the model in dynamic and changing environments.

[0076] S302, for each training sample subset, based on the twin network feature extraction module, the template sample image is extracted to obtain the template sample image feature vector, and the template sample image feature vector includes the sample target object feature vector; and based on the twin network feature extraction module, the search sample image is extracted to obtain the search sample image feature vector.

[0077] Here, the specific scheme of feature extraction of the image based on the twin network feature extraction module has been described in detail in the above step S102, which will not be repeated here.

[0078] S303, based on the mutual convolution method, the sample similarity value between the sample target object feature vector and the search sample image feature vector is calculated to determine whether the search sample image exists the sample target object to be tracked.

[0079] Here, the specific scheme of calculating the similarity between vectors to determine whether there is a sample target object to be tracked based on the mutual convolution method has been described in detail in the above step S103, which will not be repeated here.

[0080] S304, in the case that the search sample image exists the sample target object to be tracked, the position information of the sample target object to be tracked in the search sample image is determined based on the anchor-free module, and the sample prediction box position of the sample target object to be tracked in the search sample image is determined based on the position information.

[0081] Here, the specific scheme of determining the position information and the prediction box position based on the anchor-free module has been described in detail in the above step S104, which will not be repeated here.

[0082] S305, based on the Smooth L1 loss function, the loss data of the sample prediction box position and the label object box is calculated, and the loss data is used to adjust the target tracking model to be trained.

[0083] Among them, the Smooth L1 loss function is a commonly used loss function in deep learning, especially in target tracking tasks. It has better robustness in handling outliers than traditional mean square error loss function. The loss data of the sample prediction box position and the label object box is calculated by the Smooth L1 loss function, and the target tracking model to be trained is adjusted according to the loss data.

[0084] In some other embodiments, the loss function can also include a structure loss function, a perception loss function, etc., which is not limited here.

[0085] Meanwhile, in the embodiments of the present disclosure, an adaptive weight coefficient updating module is also proposed, that is, after the initial weight coefficient value is given, in the model key point regression prediction weighted calculation process, the weight coefficient is regarded as part of the model parameters by using the nn. parameter function (used to define the learnable parameters in the deep learning framework such as PyTorch) after each iteration or after every certain number of iterations, and the weight coefficient is automatically adjusted by the back propagation algorithm. Specifically, the weight coefficient can be updated by the optimization algorithm such as gradient descent to minimize the loss value. In this way, the accuracy and robustness of the key point prediction can be improved, and the manual intervention and parameter tuning cost can also be reduced.

[0086] S306, continue to train the target tracking model based on the adjusted target tracking model to be trained and the training sample set until the training result meets the preset requirement, and obtain the trained target tracking model.

[0087] Those skilled in the art can understand that the writing order of each step in the above-mentioned method of the specific embodiment does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0088] Based on the same inventive concept, the embodiments of the present disclosure also provide an adaptive twin network target tracking device corresponding to the adaptive twin network target tracking method. Since the principle of solving problems in the device of the embodiments of the present disclosure is similar to the above-mentioned adaptive twin network target tracking method of the embodiments of the present disclosure, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described here.

[0089] Referring to Figure 4 Fig. 4 shows a schematic diagram of an adaptive twin network target tracking device 400 provided by the embodiments of the present disclosure. The device comprises:

[0090] An image acquisition module 401 is configured to perform step 1: acquiring a to-be-tracked image group, wherein the to-be-tracked image group comprises a template image and a search image, and the template image comprises a preset target object; and performing image enhancement processing on the template image and the search image to obtain a to-be-tracked enhanced image group; the to-be-tracked enhanced image group comprises a template enhanced image and a search enhanced image.

[0091] A feature extraction module 402 is configured to perform step 2: performing feature extraction on the template enhanced image based on a twin network feature extraction module in a trained target tracking model to obtain a template enhanced image feature vector; the template enhanced image feature vector comprises a preset target object feature vector; and performing feature extraction on the search enhanced image based on the twin network feature extraction module to obtain a search enhanced image feature vector; the target tracking model further comprises an anchor-free module.

[0092] the object judging module 403 is configured to perform step 3: calculating a similarity value between the preset target object feature vector and the search enhanced image feature vector based on a cross convolution method, and judging whether the search enhanced image contains the target object to be tracked according to the similarity value;

[0093] the position determining module 404 is configured to perform step 4: in the case that the search enhanced image contains the target object to be tracked, determining position information of the target object to be tracked in the search enhanced image based on the anchor-free box module, and determining a predicted box position of the target object to be tracked in the search enhanced image based on the position information; in the case that the search enhanced image does not contain the target object to be tracked, acquiring a neighboring image of the search image, and repeating the above steps 1-3 with the neighboring image as a new search image until the target object to be tracked is found in the search enhanced image.

[0094] In some possible embodiments, the twin network feature extraction module includes a local feature extraction module, a global feature extraction module and a feature fusion module; and the feature extraction module 402 is specifically configured to:

[0095] perform local feature extraction on the template enhanced image based on the local feature extraction module to obtain a template enhanced image local feature vector;

[0096] perform global feature extraction on the template enhanced image based on the global feature extraction module to obtain a template enhanced image global feature vector;

[0097] perform feature fusion processing on the template enhanced image local feature vector and the template enhanced image global feature vector based on the feature fusion module to obtain the template enhanced image feature vector;

[0098] Correspondingly, the feature extraction module 402 is specifically configured to:

[0099] perform local feature extraction on the search enhanced image based on the local feature extraction module to obtain a search enhanced image local feature vector;

[0100] perform global feature extraction on the search enhanced image based on the global feature extraction module to obtain a search enhanced image global feature vector;

[0101] perform feature fusion processing on the search enhanced image local feature vector and the search enhanced image global feature vector based on the feature fusion module to obtain the search enhanced image feature vector.

[0102] In some possible embodiments, the object judgment module 403 is specifically configured to:

[0103] calculate a similarity between the preset target object feature vector and the search enhanced image feature vector based on a cross-convolution formula, to obtain a similarity response map corresponding to the search enhanced image; the similarity response map includes similarities between positions of each pixel point in the search enhanced image and the preset target object;

[0104] take a maximum value of the similarities in the similarity response map as the similarity numerical value, and determine whether the similarity numerical value is greater than a preset numerical value, to determine whether the search enhanced image includes the target object to be tracked;

[0105] The cross-convolution formula includes:

[0106] C(i,j) = Σ m,n F1(m,n)·F2(m+i,n+j);

[0107] wherein C(i,j) represents a similarity of the search enhanced image F2 at (i,j), F1(m,n) represents an element value of the preset target object feature vector F1 at (m,n), and F2(m+i,n+j) represents an element value of the search enhanced image F2 at the corresponding position (m+i,n+j) in a window with an offset of (i,j).

[0108] In some possible embodiments, the anchor-free box module includes a multi-keypoint extraction module; and the position determination module 404 is specifically configured to:

[0109] extract, based on the multi-keypoint extraction module, a plurality of key points at a position of a target pixel point corresponding to the similarity numerical value, by using two groups of deformable convolution;

[0110] determine a key point set based on the plurality of key points, and determine position information of the target object to be tracked in the search enhanced image based on the anchor-free box module and the key point set.

[0111] In some possible embodiments, the anchor-free box module further includes a multi-keypoint prediction module; and the position determination module is specifically configured to:

[0112] for each key point in the key point set, perform regression prediction on a first position corresponding to the key point based on the multi-keypoint prediction module, and perform regression prediction on a second position corresponding to the key point based on the first position and the multi-keypoint prediction module;

[0113] For each key point in the key point set, a first position and a second position corresponding to the key point are weighted and calculated based on a preset weight coefficient, to obtain coordinate information of the key point.

[0114] Based on the coordinate information of each key point in the key point set, position information of the target object to be tracked in the search enhanced image is determined.

[0115] In some possible embodiments, the position determination module 404 is further configured to:

[0116] Based on the coordinate information of each key point in the key point set, minimum horizontal coordinate information, maximum horizontal coordinate information, minimum vertical coordinate information, and maximum vertical coordinate information in the coordinate information are determined, the coordinate information of the prediction box of the target object to be tracked is determined based on the minimum horizontal coordinate information, the maximum horizontal coordinate information, the minimum vertical coordinate information, and the maximum vertical coordinate information, and a prediction box position of the target object to be tracked in the search enhanced image is determined according to the coordinate information of the prediction box.

[0117] In some possible embodiments, referring to Figure 5 As shown in the figure, the apparatus 400 further includes:

[0118] A tracking model preparation module 405 is configured to acquire a training sample set and a target tracking model to be trained, the training sample set includes multiple training sample subsets, each training sample subset includes a template sample image and a search sample image, and the template sample image includes a sample target object.

[0119] A sample feature extraction module 406 is configured to, for each training sample subset, perform feature extraction on the template sample image based on a Siamese network feature extraction module to obtain a template sample image feature vector, the template sample image feature vector including a sample target object feature vector, and perform feature extraction on the search sample image based on the Siamese network feature extraction module to obtain a search sample image feature vector.

[0120] A sample object judgment module 407 is configured to calculate a sample similarity value between the sample target object feature vector and the search sample image feature vector based on a cross-convolution method, to determine whether a target sample object to be tracked exists in the search sample image.

[0121] A prediction box position determination module 408 is configured to, in a case where the target sample object to be tracked exists in the search sample image, determine position information of the target sample object to be tracked in the search sample image based on an anchor-free box module, and determine a sample prediction box position of the target sample object to be tracked in the search sample image based on the position information.

[0122] The model loss calculation module 409 is configured to calculate loss data of the sample predicted frame position and the label object frame based on a Smooth L1 loss function, and adjust the target tracking model to be trained based on the loss data.

[0123] The tracking model training module 410 is configured to continue training the target tracking model based on the adjusted target tracking model to be trained and the training sample set until the training result meets a preset requirement, and obtain the trained target tracking model.

[0124] Based on the same technical concept, the embodiments of the present disclosure also provide a computer device. Referring to FIG. 6, Figure 6 As shown in FIG. 6, the computer device 600 provided by the embodiments of the present disclosure includes a processor 601, a memory 602, and a bus 603. The memory 602 is configured to store execution instructions, including an internal memory 6021 and an external memory 6022. The internal memory 6021 is also called an internal storage, and is used to temporarily store operation data in the processor 601 and exchange data with the external memory 6022 such as a hard disk. The processor 601 exchanges data with the external memory 6022 through the internal memory 6021.

[0125] In the embodiments of the present application, the memory 602 is specifically configured to store application program codes for executing the schemes of the present application, and is controlled to execute by the processor 601. That is, when the computer device 600 is running, the processor 601 and the memory 602 communicate through the bus 603, so that the processor 601 executes the application program codes stored in the memory 602, and further executes the methods described in any of the preceding embodiments.

[0126] The memory 602 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), and the like.

[0127] The processor 601 can be an integrated circuit chip having a processing capability for signals. The processor described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; or can be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The methods, steps and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed by the processor. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor or the like.

[0128] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the computer device 600. In some other embodiments of the present application, the computer device 600 can include more or fewer components than those illustrated, or combine certain components, or split certain components, or different arrangement of components. The illustrated components can be implemented in hardware, software or a combination of software and hardware.

[0129] The embodiments of the present disclosure further provide a computer readable storage medium, which stores a computer program. When the computer program is run by a processor, the steps of the adaptive twin network target tracking method described in the above method embodiments are executed. The storage medium can be a volatile or non-volatile computer readable storage medium.

[0130] The embodiments of the present disclosure further provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the adaptive twin network target tracking method described in the above method embodiments. For details, refer to the above method embodiments, which will not be described here.

[0131] The computer program product can be specifically implemented by hardware, software or a combination thereof. In one optional embodiment, the computer program product is specifically embodied as a computer storage medium. In another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0132] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here. In several embodiments provided in the present disclosure, it should be understood that the disclosed system and method can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division, and there can be another division in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, devices or units, and can be electrical, mechanical or other forms.

[0133] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0134] In addition, the functional units in each embodiment of the present disclosure can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0135] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present disclosure essentially or the part of the prior art or the part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the various embodiments of the present disclosure. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various program code storage media.

[0136] Finally, it should be noted that the above-described embodiments are merely specific embodiments of the present disclosure, used to illustrate the technical solutions of the present disclosure, and are not intended to limit the present disclosure. The protection scope of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easy changes to the technical solutions described in the foregoing embodiments, or easily think of changes or equivalent replacements for some of the technical features; and these modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. An adaptive twin network target tracking method, characterized in that, include: Step 1: Obtain the image group to be tracked, which includes a template image and a search image, wherein the template image includes a preset target object; The template image and the search image are then subjected to image enhancement processing to obtain a group of enhanced images to be tracked; The group of enhanced images to be tracked includes template enhanced images and search enhanced images; Step 2: Based on the Siamese network feature extraction module in the trained target tracking model, feature extraction is performed on the template-enhanced image to obtain a template-enhanced image feature vector; the template-enhanced image feature vector includes a preset target object feature vector; and based on the Siamese network feature extraction module, feature extraction is performed on the search-enhanced image to obtain a search-enhanced image feature vector; the target tracking model also includes an anchorless module; Step 3: Calculate the similarity value between the feature vector of the preset target object and the feature vector of the search-enhanced image based on the cross-convolution method, and determine whether there is a target object to be tracked in the search-enhanced image based on the similarity value; Step 4: If the target object to be tracked exists in the search augmented image, determine the set of key points about the search augmented image based on the anchorless box module, and determine the position information of the target object to be tracked in the search augmented image based on the anchorless box module and the set of key points; and determine the predicted box position of the target object to be tracked in the search augmented image based on the position information. If the target object to be tracked is not found in the enhanced search image, obtain a neighboring image for the search image, and repeat steps 1 to 3 above with the neighboring image as a new search image until the target object to be tracked is found in the enhanced search image; The anchorless bounding box module includes a multi-keypoint prediction module; the step of determining the position information of the target object to be tracked in the search augmented image based on the anchorless bounding box module and the keypoint set includes: For each key point in the set of key points, regression prediction is performed on the first position corresponding to the key point based on the multi-key point prediction module; and regression prediction is performed on the second position corresponding to the key point based on the first position and the multi-key point prediction module. For each key point in the set of key points, the first position and the second position corresponding to the key point are weighted and calculated based on a preset weight coefficient to obtain the coordinate information of the key point; The location information of the target object to be tracked in the search enhancement image is determined based on the coordinate information of each key point in the key point set.

2. The method according to claim 1, characterized in that, The Siamese network feature extraction module includes a local feature extraction module, a global feature extraction module, and a feature fusion module; the Siamese network feature extraction module based on the trained target tracking model extracts features from the template-enhanced image, including: Based on the local feature extraction module, local feature extraction is performed on the template-enhanced image to obtain the local feature vector of the template-enhanced image; Based on the global feature extraction module, global feature extraction is performed on the template-enhanced image to obtain the global feature vector of the template-enhanced image; Based on the feature fusion module, feature fusion processing is performed on the local feature vector and the global feature vector of the template-enhanced image to obtain the feature vector of the template-enhanced image. Accordingly, the feature extraction of the search enhancement image based on the Siamese network feature extraction module includes: Based on the local feature extraction module, local feature extraction is performed on the search enhancement image to obtain the local feature vector of the search enhancement image; Based on the global feature extraction module, global feature extraction is performed on the search-enhanced image to obtain the global feature vector of the search-enhanced image; The feature fusion module performs feature fusion processing on the local feature vector and the global feature vector of the search-enhanced image to obtain the feature vector of the search-enhanced image.

3. The method according to claim 1, characterized in that, The step of calculating the similarity value between the feature vector of the preset target object and the feature vector of the search-enhanced image based on the cross-convolution method, and determining whether there is a target object to be tracked in the search-enhanced image based on the similarity value, includes: The similarity between the feature vector of the preset target object and the feature vector of the search-enhanced image is calculated based on the cross-convolution formula to obtain a similarity response map corresponding to the search-enhanced image; the similarity response map includes the similarity between the position of each pixel in the search-enhanced image and the preset target object; The maximum similarity value in the similarity response graph is used as the similarity value, and it is determined whether the similarity value is greater than a preset value, so as to determine whether there is a target object to be tracked in the search enhancement image; The cross-convolution formula includes: C(i,j)=∑ m,n F1(m,n)·F2(m+i,n+j); Where C(i,j) represents the similarity of the search-enhanced image F2 at (i,j), F1(m,n) represents the element value at (m,n) in the preset target object feature vector F1, and F2(m+i,n+j) represents the element value at (m+i,n+j) in the search-enhanced image F2 within the window with offset (i,j).

4. The method according to claim 3, characterized in that, The anchorless frame module includes a multi-keypoint extraction module; determining the set of keypoints for the search-enhanced image includes: Based on the location of the target pixel corresponding to the similarity value, the multi-keypoint extraction module uses two sets of deformable convolutions to extract multiple keypoints at the location of the target pixel; and, based on the multiple keypoints, determines the keypoint set.

5. The method according to claim 4, characterized in that, Determining the predicted bounding box position of the target object to be tracked in the enhanced search image based on the location information includes: Based on the coordinate information of each key point in the key point set, the minimum x-coordinate, maximum x-coordinate, minimum y-coordinate, and maximum y-coordinate are determined. Based on the minimum x-coordinate, maximum x-coordinate, minimum y-coordinate, and maximum y-coordinate, the coordinate information of the predicted bounding box of the target object to be tracked is determined. And based on the coordinate information of the predicted bounding box, the position of the predicted bounding box of the target object to be tracked in the search augmented image is determined.

6. The method according to any one of claims 1-5, characterized in that, The target tracking model is trained through the following steps: Obtain a training sample set and a target tracking model to be trained; the training sample set includes multiple training sample subsets, each subset including template sample images and search sample images; the template sample images include sample target objects; For each training sample subset, feature extraction is performed on the template sample image based on the Siamese network feature extraction module to obtain the template sample image feature vector, which includes the feature vector of the sample target object; and feature extraction is performed on the search sample image based on the Siamese network feature extraction module to obtain the search sample image feature vector. The sample similarity value between the feature vector of the target object and the feature vector of the search sample image is calculated based on the cross-convolution method to determine whether there is a target object to be tracked in the search sample image; If a target object to be tracked exists in the search sample image, the position information of the target object to be tracked in the search sample image is determined based on the anchorless box module, and the position of the sample prediction box of the target object to be tracked in the search sample image is determined based on the position information. The loss data between the sample prediction box position and the label object box is calculated based on the Smooth L1 loss function, and the target tracking model to be trained is adjusted based on the loss data. Based on the adjusted target tracking model to be trained and the training sample set, the target tracking model is trained again until the training result meets the preset requirements, and the trained target tracking model is obtained.

7. An adaptive twin network target tracking device, characterized in that, include: The image acquisition module is used to perform step 1: acquire a group of images to be tracked, the group of images to be tracked includes a template image and a search image, the template image includes a preset target object; and perform image enhancement processing on the template image and the search image to obtain an enhanced group of images to be tracked. The group of enhanced images to be tracked includes template enhanced images and search enhanced images; The feature extraction module is used to perform step 2: extracting features from the template-enhanced image based on the Siamese network feature extraction module in the trained target tracking model to obtain a template-enhanced image feature vector; the template-enhanced image feature vector includes a preset target object feature vector; and extracting features from the search-enhanced image based on the Siamese network feature extraction module to obtain a search-enhanced image feature vector; the target tracking model also includes an anchorless module; The object determination module is used to perform step 3: calculate the similarity value between the feature vector of the preset target object and the feature vector of the search enhancement image based on the cross-convolution method, and determine whether there is a target object to be tracked in the search enhancement image based on the similarity value; The location determination module is used to perform step 4: when the target object to be tracked exists in the search augmented image, it determines the set of key points about the search augmented image based on the anchorless frame module, and determines the location information of the target object to be tracked in the search augmented image based on the anchorless frame module and the set of key points; and determines the predicted bounding box position of the target object to be tracked in the search augmented image based on the location information. If the target object to be tracked is not found in the enhanced search image, obtain a neighboring image for the search image, and repeat steps 1 to 3 above with the neighboring image as a new search image until the target object to be tracked is found in the enhanced search image; The anchorless module includes a multi-keypoint prediction module; the position determination module is specifically used for: For each key point in the set of key points, regression prediction is performed on the first position corresponding to the key point based on the multi-key point prediction module; and regression prediction is performed on the second position corresponding to the key point based on the first position and the multi-key point prediction module. For each key point in the set of key points, the first position and the second position corresponding to the key point are weighted and calculated based on a preset weight coefficient to obtain the coordinate information of the key point; The location information of the target object to be tracked in the search enhancement image is determined based on the coordinate information of each key point in the key point set.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

9. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

Images