A target tracking method based on graph attention fusion network

Abstract

The present disclosure relates to the field of image processing technology, and in particular to a target tracking method based on a graph attention fusion network. The method comprises: using a target backbone network in a target twin network to obtain a first template image feature corresponding to a first template image and a search image feature of each search image in a single target tracking data set; using a feature fusion network based on a graph attention network in the target twin network to perform spatial feature fusion processing and channel feature fusion processing on the first template image feature and the search image feature of each search image to obtain a fusion feature; inputting the fusion feature into a target tracking head network in the target twin network to obtain a first tracking result corresponding to a first target to be tracked in the first template image. The present disclosure is used to improve the resolution capability of the twin network for the target to be tracked, so as to achieve robust tracking of the tracker under background interference, improve the accuracy of target recognition, and improve the accuracy of target tracking.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a target tracking method based on a graph attention fusion network. Background Art

[0002] With the development of science and technology, video single-target tracking, as a fundamental research topic in computer vision, is the foundation for tasks such as navigation and guidance, video surveillance, and scene understanding, and can be widely applied in various fields. For example, a twin network target tracking algorithm can be used for target search. However, feature fusion is performed only in the spatial dimension, resulting in fewer channels for effectively activating the fused template image features and search image features, and weaker object boundary resolution. This affects the tracker's robustness when background interference is present, resulting in poor target tracking results. Summary of the Invention

[0003] This disclosure provides a target tracking method based on a graph attention fusion network, which can improve the resolution capability of the twin network for the target to be tracked, achieve robust tracking of the tracker under background interference, improve the accuracy of target recognition, and improve the accuracy of target tracking. The technical solutions of this disclosure are as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a target tracking method based on a graph attention fusion network is provided, comprising:

[0005] Using the target backbone network in the target twin network, obtain the first template image features corresponding to the first template image and the search image features of each search image in the single target tracking data set;

[0006] Using a feature fusion network based on a graph attention network in the target twin network, performing spatial feature fusion processing and channel feature fusion processing on the first template image features and the search image features of each search image to obtain fused features;

[0007] The fusion feature is input into the target tracking head network in the target twin network to obtain a first tracking result corresponding to the first target to be tracked in the first template image, wherein the first tracking result includes a first classification result corresponding to the first target to be tracked and a first regression result corresponding to the first target to be tracked.

[0008] According to some embodiments, the step of using a feature fusion network based on a graph attention network in the target twin network to perform spatial feature fusion processing and channel feature fusion processing on the first template image features and the search image features of each search image to obtain fused features includes:

[0009] Using the target spatial feature fusion network based on the graph attention network in the target twin network, the first template image feature and the search image feature of each search image are fused to obtain a first spatial fusion feature;

[0010] The target channel feature fusion network based on the graph attention network in the target twin network is used to enhance and fuse the first spatial fusion feature, obtain the first channel fusion feature, and use the first channel fusion feature as the fusion feature.

[0011] According to some embodiments, fusing the first template image feature with the search image features of each search image to obtain a first spatial fusion feature includes:

[0012] Obtaining a first node set corresponding to the first template image feature, wherein each pixel of the first template image feature is a node;

[0013] Obtaining a second node set corresponding to the search image feature of each search image, wherein each pixel of the search image feature of each search image is a node;

[0014] constructing a bipartite graph based on the first node set and the second node set, wherein the bipartite graph is used to indicate a correspondence between the first template image feature and the search image features of each search image;

[0015] Obtaining scores between the first node and the second node based on the bipartite graph, a first eigenvector corresponding to a first node in the first node set, and a second eigenvector corresponding to a second node in the second node set, and regularizing the scores using a Softmax function to obtain weight information, wherein the first node is any node in the first node set, and the second node is any node in the second node set;

[0016] Obtaining an aggregate feature corresponding to the first node based on at least one weight information transmitted to the first node by all nodes in the first subgraph of the bipartite graph, a second eigenvector corresponding to each node in the second node set, and a linear transformation matrix;

[0017] Fusing the aggregated feature with the first node feature of the first node to obtain a splicing feature;

[0018] Perform spatial feature fusion on at least one splicing feature corresponding to the first node set to obtain a first spatial fusion feature.

[0019] According to some embodiments, performing enhancement processing and fusion processing on the first spatial fusion feature to obtain a first channel fusion feature includes:

[0020] Generate a random graph using the small-world model;

[0021] Encoding the channel features corresponding to the first spatial fusion features to obtain second node features;

[0022] Decomposing the second node feature according to the channel dimension information to obtain the decomposed node feature;

[0023] Sorting the disassembled node features according to a preset order to obtain a third node feature of the random graph;

[0024] Performing a graph attention convolution operation on the third node feature to obtain a fourth node feature;

[0025] Performing feature extraction and value assignment on the fourth node feature, and performing splicing processing in the order corresponding to the channel dimensions to obtain channel fused features;

[0026] The channel-fused features are decoded to obtain first channel-fused features.

[0027] According to some embodiments, the initial Siamese network includes an initial backbone network, an initial spatial feature fusion network based on a graph attention network, an initial channel feature fusion network based on a graph attention network, and an initial tracking head network, and the method further includes:

[0028] Acquire a training data set, wherein the training data set includes at least one training sample image;

[0029] Using the initial backbone network, initially acquiring a second search image feature corresponding to the second search image and a training sample image feature corresponding to each training sample image in the training data set;

[0030] Using the initial spatial feature fusion network based on the graph attention network to fuse the second search image feature and the training sample image to obtain a first spatial fusion feature;

[0031] Using the initial channel feature fusion network based on the graph attention network to enhance and fuse the first spatial fusion features to obtain second channel fusion features;

[0032] Inputting the second channel fusion feature into the initial tracking head network to obtain a second tracking result corresponding to a second target to be tracked in the second search image, wherein the second tracking result includes a second classification result and a second regression result corresponding to the second target to be tracked;

[0033] Obtaining a first error between the second classification result and a third classification result and a second error between the second regression result and a third regression result, wherein the third classification result is a classification result obtained by labeling the second target to be tracked, and the third regression result is a regression result obtained by labeling the second target to be tracked;

[0034] The model parameters of the initial twin network are adjusted according to the first error and the second error until the initial twin network meets the network training requirements, and the target twin network is obtained.

[0035] According to some embodiments, obtaining a first error between the second classification result and the third classification result and a second error between the second regression result and the third regression result includes:

[0036] Using a binary cross entropy loss function to obtain a first error between the second classification result and the third classification result;

[0037] An intersection-over-union loss function is used to obtain a second error between the second regression result and the third regression result.

[0038] According to some embodiments, adjusting the model parameters of the initial twin network according to the first error and the second error until the initial twin network meets the network training requirements and the target twin network is obtained includes:

[0039] Obtaining a total error according to a first weight corresponding to the first error, the first error, a second weight corresponding to the second error, and the second error;

[0040] The total error is back-propagated using the stochastic gradient descent method, and the model parameters of the initial twin network are adjusted until the initial twin network meets the network training requirements, and the target twin network is obtained.

[0041] According to a second aspect of an embodiment of the present disclosure, a target tracking device based on a graph attention fusion network is provided, comprising:

[0042] An image acquisition unit, configured to acquire, using a target backbone network in a target twin network, a first template image feature corresponding to the first template image and a search image feature of each search image in a single target tracking data set;

[0043] a feature fusion unit, configured to use a feature fusion network based on a graph attention network in the target twin network to perform spatial feature fusion processing and channel feature fusion processing on the first template image features and the search image features of each search image to obtain fused features;

[0044] A target tracking unit is used to input the fusion feature into the target tracking head network in the target twin network to obtain a first tracking result corresponding to the first target to be tracked in the first template image, wherein the first tracking result includes a first classification result corresponding to the first target to be tracked and a first regression result corresponding to the first target to be tracked.

[0045] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0046] processor;

[0047] a memory for storing instructions executable by the processor;

[0048] The processor is configured to execute the instructions to implement the target tracking method based on the graph attention fusion network described in any one of the aforementioned aspects.

[0049] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the target tracking method based on a graph attention fusion network described in any one of the aforementioned aspects.

[0050] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the method described in any one of the aforementioned aspects when executed by a processor.

[0051] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0052] In some or related embodiments, by adopting the target backbone network in the target twin network, the first template image features corresponding to the first template image and the search image features of each search image in the single target tracking data set are obtained; the feature fusion network based on the graph attention network in the target twin network is used to perform spatial feature fusion processing and channel feature fusion processing on the first template image features and the search image features of each search image to obtain fusion features; the fusion features are input into the target tracking head network in the target twin network to obtain the first tracking result corresponding to the first target to be tracked in the first template image, wherein the first tracking result includes the first classification result corresponding to the first target to be tracked and the first regression result corresponding to the first target to be tracked. Therefore, the topological relationship of the channel dimension can be increased, and channel feature fusion can be added on the basis of spatial feature fusion, so as to effectively enhance and fuse the template image features and the search image features in the channel dimension. This can reduce the situation where feature fusion only in the spatial dimension leads to fewer channels with effective activation of features after the fusion of template image features and search image features, and the ability to distinguish object boundaries is weak. This improves the resolution ability of the twin network for the target to be tracked, realizes robust tracking of the tracker under background interference, improves the accuracy of target recognition, and improves the accuracy of target tracking.

[0053] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0055] Figure 1 This is a flowchart of a first target tracking method based on a graph attention fusion network provided by an embodiment of the present disclosure;

[0056] Figure 2 is a flowchart of a second target tracking method based on a graph attention fusion network provided by an embodiment of the present disclosure;

[0057] Figure 3 This is a twin network structure diagram based on a graph attention fusion network provided by an embodiment of the present disclosure;

[0058] Figure 4 This is a flow chart of a spatial feature fusion network based on graph attention provided by an embodiment of the present disclosure;

[0059] Figure 5 This is a flow chart of a channel feature fusion network based on graph attention provided by an embodiment of the present disclosure;

[0060] Figure 6 2 is a comparative example diagram of a second target tracking method based on a graph attention fusion network provided by an embodiment of the present disclosure;

[0061] Figure 7 is a schematic diagram illustrating an example of an electronic device according to an exemplary embodiment;

[0062] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0063] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0064] The disclosed embodiments provide a target tracking method, device, electronic device, and storage medium based on a graph attention fusion network. In some embodiments, the term "target tracking method based on a graph attention fusion network" is interchangeable with the terms "information processing method" and "communication method"; the term "target tracking device based on a graph attention fusion network" is interchangeable with the terms "information processing device" and "communication device"; and the term "information processing system" and "communication system" are interchangeable.

[0065] The embodiments of the present disclosure are not exhaustive and are merely illustrative of some embodiments, and are not intended to be a specific limitation on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a certain embodiment can also be implemented as an independent embodiment, and the order of the steps in a certain embodiment can be arbitrarily exchanged. In addition, the optional implementation methods in a certain embodiment can be arbitrarily combined; in addition, the embodiments can be arbitrarily combined. For example, some or all steps of different embodiments can be arbitrarily combined, and a certain embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0066] In each embodiment of the present disclosure, unless otherwise specified or provided for by logic, the terms and / or descriptions between the embodiments are consistent and can be referenced by each other. The technical features in different embodiments can be combined to form a new embodiment based on their inherent logical relationships.

[0067] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure.

[0068] In the embodiments of the present disclosure, unless otherwise specified, elements expressed in the singular, such as "a", "an", "the", "above", "said", "the", "the", etc., may mean "one and only one", or "one or more", "at least one", etc. For example, when using articles such as "a", "an", "the" in English in translation, the noun following the article may be understood as a singular expression or a plural expression.

[0069] In the embodiments of the present disclosure, “plurality” refers to two or more.

[0070] In some embodiments, the terms “at least one,” “one or more,” “a plurality of,” “multiple,” etc. may be used interchangeably.

[0071] The prefixes such as "first" and "second" in the embodiments of the present disclosure are only used to distinguish different description objects and do not constitute any restriction on the position, order, priority, quantity or content of the description objects. For the statement of the description object, please refer to the description in the context of the claims or embodiments, and no unnecessary restriction should be constituted due to the use of prefixes. For example, if the description object is a "field", the ordinal number before the "field" in the "first field" and the "second field" does not limit the position or order between the "fields". "First" and "second" do not limit whether the "fields" they modify are in the same message, nor do they limit the order of the "first field" and the "second field". For another example, if the description object is a "level", the ordinal number before the "level" in the "first level" and the "second level" does not limit the priority between the "levels". For another example, the number of description objects is not limited by the ordinal number and can be one or more. Taking "first device" as an example, the number of "devices" can be one or more. In addition, the objects modified by different prefixes can be the same or different. For example, if the description object is "device", then the "first device" and the "second device" can be the same device or different devices, and their types can be the same or different; for another example, if the description object is "information", then the "first information" and the "second information" can be the same information or different information, and their contents can be the same or different.

[0072] In some embodiments, "terminal" or "terminal device" may be referred to as "user equipment (UE)", "user terminal", "mobile station (MS)", "mobile terminal (MT)", subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless electronic device (wireless communication device), remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, etc.

[0073] In some embodiments, data, information, etc. may be obtained with the user's consent.

[0074] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0075] According to some embodiments, the task of video single target tracking is: for a video sequence, first initialize the tracker according to the position of the target in the initial frame, then extract the target features and establish a target model, adopt a certain tracking strategy in subsequent frames, estimate the position of the target in the current frame based on the target model, and finally use the current position to update the target model and continue tracking the next frame.

[0076] The twin network target tracking algorithm uses a neural network trained on a dataset to learn more expressive features and applies these features to target location within the search area. The twin network for single target tracking takes two images as input: a template image of the target to be tracked and a slightly larger image of the search area. The output is the predicted classification of the target category within the search area and the bounding box regression result. Therefore, the network needs to learn from the data how to distinguish the presence of a target in the search image and, if so, identify it with a bounding box.

[0077] The structure of the twin network is divided into three parts: the backbone network, which is generally modified from the network structure that has achieved excellent performance in image classification tasks, and is used to extract template image features and search image features from the template image and search image; the spatial feature fusion network, which is used to further enhance and fuse the template image features and search image features in the spatial dimension to obtain fusion features; the tracking head network, for example, can use some convolutional layers or multi-layer perceptrons to obtain classification results and regression results.

[0078] Using the same backbone network and tracking head network, the graph neural network-based tracker constructs a bipartite graph between the feature pixels of the template image and the feature pixels of the search image. Based on this bipartite graph, the graph neural network is used to design a spatial feature fusion network to enhance and fuse the template image features and search image features in the spatial dimension. However, feature fusion performed only in the spatial dimension ignores the correlation of features in the channel dimension. As a result, the fused template and search image features have fewer effectively activated channels and weaker ability to resolve object boundaries, thus affecting the tracker's robustness in the presence of background interference.

[0079] Figure 1 This is a flowchart of the first target tracking method based on graph attention fusion network provided by the embodiment of the present disclosure. Figure 1 As shown in FIG, the target tracking method based on the graph attention fusion network can be used to track the target in the scenario of background interference during the single target tracking process in the video, including the following steps:

[0080] In step S11, the target backbone network in the target twin network is used to obtain the first template image features corresponding to the first template image and the search image features of each search image in the single target tracking data set;

[0081] According to some embodiments, the execution subject of the embodiments of the present disclosure may be, for example, an electronic device. The electronic device does not specifically refer to a fixed device. For example, when the device identification of an electronic device changes, the electronic device may also change accordingly. For example, when the device structure of an electronic device changes, the electronic device may also change accordingly. Among them, the execution subject of the embodiments of the present disclosure may also be, for example, a server. The server may be, for example, a single server or a server cluster, and the embodiments of the present disclosure do not limit this.

[0082] In some embodiments, the twin network is a neural network composed of two or more sub-networks with the same structure and shared weights, which is used to measure the similarity between input samples. The target twin network can be, for example, a trained network that can be used for image tracking. The target twin network does not specifically refer to a fixed network. For example, when the sub-networks in the target twin network change, the target twin network can also change accordingly. Among them, the target twin network can, for example, include a target backbone network, a feature fusion network based on a graph attention network, and a target tracking head network.

[0083] According to some embodiments, a target backbone network can be used, for example, to extract features from an input image. The target backbone network is not specifically a fixed network. For example, the target backbone network can correspond to a template image. For example, template images of different sizes or different pixels can correspond to different backbone networks.

[0084] In some embodiments, the first template image may be, for example, an image including the target to be tracked. The first template image does not specifically refer to a fixed image. For example, when the image content corresponding to the first template image changes, the first template image may also change accordingly. For example, when the acquisition time point or reception time point corresponding to the first template image changes, the first template image may also change accordingly. The first in the first template image is used to distinguish it from the other template images.

[0085] According to some embodiments, the first template image feature refers to a feature corresponding to the first template image, and may be obtained by extracting image features from the first template image. The first of the first template image features is used to distinguish it from the remaining template image features and does not specifically refer to a fixed feature. For example, if the feature extraction method corresponding to the first template image feature changes, the first template image feature may also change accordingly.

[0086] According to some embodiments, a single target tracking data set may be, for example, a collection of at least one single target tracking data set. The single target tracking data set does not specifically refer to a fixed set. For example, when a data item in the single target tracking data set changes, the single target tracking data set may also change accordingly. For example, when the amount of data corresponding to the single target tracking data set changes, the single target tracking data set may also change accordingly. The single target tracking data set may also be referred to as a single target search image set, which is not limited in the presently disclosed embodiments.

[0087] In some embodiments, the search image may be, for example, an image for which it is to be determined whether it includes a target to be tracked.

[0088] In some embodiments, a target backbone network in a target twin network is used to obtain first template image features corresponding to the first template image and search image features of each search image in the single target tracking dataset. The order of the first template image features and the search image features is not limited.

[0089] In step S12, a feature fusion network based on a graph attention network in the target twin network is used to perform spatial feature fusion processing and channel feature fusion processing on the first template image features and the search image features of each search image to obtain fused features;

[0090] According to some embodiments, the feature fusion network based on the graph attention network can be, for example, a feature fusion network that is trained and can be used for feature fusion. The feature fusion network based on the graph attention network can, for example, include a spatial feature fusion network based on the graph attention network and a channel feature fusion network based on the graph attention network. The feature fusion network based on the graph attention network does not specifically refer to a fixed network. For example, when the parameters in the feature fusion network based on the graph attention network change, the feature fusion network based on the graph attention network can also change accordingly.

[0091] In some embodiments, the spatial feature fusion processing may be, for example, an enhancement and fusion processing of the first template image features and the search image features of each search image in the spatial dimension.

[0092] According to some embodiments, the channel feature fusion processing may be, for example, an enhancement and fusion processing of the first template image features and the search image features of each search image in the channel dimension.

[0093] In some embodiments, the fused feature may be, for example, a feature obtained by fusing the first template image feature with the search image features of each search image. The fused feature is not specifically a fixed feature. For example, when the feature fusion network based on the graph attention network changes, the fused feature may also change accordingly.

[0094] In some embodiments, a feature fusion network based on a graph attention network in the target twin network can be used to perform spatial feature fusion processing and channel feature fusion processing on the first template image features and the search image features of each search image to obtain fused features.

[0095] In step S13, the fused features are input into the target tracking head network in the target twin network to obtain a first tracking result corresponding to the first target to be tracked in the first template image, wherein the first tracking result includes a first classification result corresponding to the first target to be tracked and a first regression result corresponding to the first target to be tracked.

[0096] According to some embodiments, the target tracking head network in the target twin network is used to obtain classification results and regression results. Among them, the first tracking result may include, for example, a first classification result and a first regression result. The first in the first classification result is used to distinguish it from the other classification results and does not specifically refer to a fixed result. For example, when the target twin network changes, the first classification result may also change accordingly.

[0097] In some embodiments, the first target to be tracked may be, for example, a target to be tracked in the first template image. The first target to be tracked does not specifically refer to a fixed target. The first target to be tracked may be, for example, determined according to a target setting instruction. The first target to be tracked does not specifically refer to a fixed target.

[0098] In some embodiments, the fused features are input into the target tracking head network in the target twin network to obtain a first tracking result corresponding to the first target to be tracked in the first template image, wherein the first tracking result includes a first classification result corresponding to the first target to be tracked and a first regression result corresponding to the first target to be tracked.

[0099] In some or related embodiments, by adopting the target backbone network in the target twin network, the first template image features corresponding to the first template image and the search image features of each search image in the single target tracking data set are obtained; the feature fusion network based on the graph attention network in the target twin network is adopted to perform spatial feature fusion processing and channel feature fusion processing on the first template image features and the search image features of each search image to obtain fusion features; the fusion features are input into the target tracking head network in the target twin network to obtain the first tracking result corresponding to the first target to be tracked in the first template image, wherein the first tracking result includes the first classification result corresponding to the first target to be tracked and the first regression result corresponding to the first target to be tracked. Therefore, the topological relationship of the channel dimension can be increased, and channel feature fusion can be added on the basis of spatial feature fusion, so as to effectively enhance and fuse the template image features and the search image features in the channel dimension. This can reduce the situation where feature fusion only in the spatial dimension leads to fewer channels with effective activation of features after the fusion of template image features and search image features, and the ability to distinguish object boundaries is weak. This improves the resolution ability of the twin network for the target to be tracked, realizes robust tracking of the tracker under background interference, improves the accuracy of target recognition, and improves the accuracy of target tracking.

[0100] Figure 2 This is a flowchart of a second target tracking method based on a graph attention fusion network provided by an embodiment of the present disclosure. Figure 2 As shown in the figure, the target tracking method based on the graph attention fusion network can be used in the target tracking scenario, including the following steps:

[0101] In step S21, the target backbone network in the target twin network is used to obtain the first template image features corresponding to the first template image and the search image features of each search image in the single target tracking data set;

[0102] Among them, the relevant description is as above and will not be repeated here.

[0103] According to some embodiments, the initial Siamese network includes an initial backbone network, an initial spatial feature fusion network based on a graph attention network, an initial channel feature fusion network based on a graph attention network, and an initial tracking head network, and the method further includes:

[0104] Acquire a training data set, wherein the training data set includes at least one training sample image;

[0105] Using an initial backbone network, initially acquiring a second search image feature corresponding to the second search image and a training sample image feature corresponding to each training sample image in the training data set;

[0106] The initial spatial feature fusion network based on the graph attention network is used to fuse the second search image features and the training sample image to obtain the first spatial fusion feature;

[0107] The initial channel feature fusion network based on the graph attention network is used to enhance and fuse the first spatial fusion features to obtain the second channel fusion features;

[0108] Inputting the second channel fusion feature into the initial tracking head network to obtain a second tracking result corresponding to the second target to be tracked in the second search image, wherein the second tracking result includes a second classification result and a second regression result corresponding to the second target to be tracked;

[0109] Obtaining a first error between the second classification result and the third classification result and a second error between the second regression result and the third regression result, wherein the third classification result is a classification result obtained by labeling the second target to be tracked, and the third regression result is a regression result obtained by labeling the second target to be tracked;

[0110] The model parameters of the initial twin network are adjusted based on the first and second errors until the initial twin network meets the network training requirements and the target twin network is obtained. Therefore, parameter adjustment can be performed based on multiple errors to improve the accuracy of obtaining the target twin network and the accuracy of target retrieval.

[0111] According to some embodiments, obtaining a first error between the second classification result and the third classification result and a second error between the second regression result and the third regression result includes:

[0112] A binary cross entropy loss function is used to obtain the first error between the second classification result and the third classification result;

[0113] The second error between the second regression result and the third regression result is obtained by using the intersection-over-union loss function. Therefore, different errors can be obtained according to different loss functions, which can improve the accuracy of error acquisition.

[0114] According to some embodiments, adjusting the model parameters of the initial twin network according to the first error and the second error until the initial twin network meets the network training requirements and the target twin network is obtained includes:

[0115] Obtaining a total error according to a first weight corresponding to the first error, the first error, a second weight corresponding to the second error, and the second error;

[0116] The stochastic gradient descent method is used to backpropagate the total error and adjust the model parameters of the initial twin network until the initial twin network meets the network training requirements and the target twin network is obtained.

[0117] According to some embodiments, Figure 3 This is a twin network structure diagram based on a graph attention fusion network provided by an embodiment of the present disclosure, such as Figure 3 As shown in the figure, the twin network based on the graph attention fusion network includes a backbone network, a feature fusion network based on the graph attention network, and a tracking head network. The backbone network, a network structure that can achieve excellent performance in image classification tasks, is used to extract template image features and search image features from the template image and search image; the feature fusion network based on the graph attention network is used to enhance and fuse the template image features and search image features to obtain fused features; the tracking head network can use some convolutional layers or multi-layer perceptrons to obtain classification and regression results. The feature fusion network based on the graph attention network consists of a spatial feature fusion network based on the graph attention network and a channel feature fusion network based on the graph attention network. The spatial feature fusion network based on the graph attention network enhances and fuses the template image features and search image features in the spatial dimension, while the channel feature fusion network based on the graph attention network enhances and fuses the template image features and search image features in the channel dimension.

[0118] According to some embodiments, Figure 5 The Siamese network based on the graph attention fusion network shown in the figure can include:

[0119] Step 1: Select the MobileNetV2 network that has achieved good performance in image classification tasks as the backbone network of the initial twin network in this embodiment, denoted as The last fully connected layer of the MobileNetV2 network is removed to meet the requirements of the single target tracking task. At the same time, in order to make the stride of the backbone network 8, the operations after the last downsampling convolution layer in the network are removed.

[0120] The template image extracted from the training set of the single target tracking dataset and search images Input two MobileNetV2 networks with the same structure and shared parameters for feature extraction, and output the template image features extracted by MobileNetV2. and search image features , its formula can be shown as formula (1):

[0121] (1)

[0122] in, For a size of Tensor of and For height and width, is the number of feature channels; For a size of Tensor of and For height and width.

[0123] Step 2: Transform the template image features and search image features Input the fusion network based on the graph attention mechanism, and output the related operation features after enhancing and fusing the two. , the relevant operation network is recorded as , that is, its formula can be shown as formula (2):

[0124] (2)

[0125] in, For a size of Tensor of and For height and width, is the number of feature channels.

[0126] Step 3: Correlation operation features Input tracking head network, denoted as , output the prediction results of the entire twin network for the target to be tracked in the search image, including the classification results And the regression results .

[0127]

[0128] in, For a size of Tensor of For a size of Tensor of and For height and width.

[0129] Step 4: Track the classification results predicted by the head network And the regression results The actual classification results of the target to be tracked in the search image And the regression results Perform error calculation.

[0130] The error calculation of the classification result uses the binary cross entropy loss function (Binary Cross Entropy Loss), and the error calculation of the regression result uses the intersection over union loss function (Intersetion Over Union Loss). The former is denoted as , the latter is recorded as , then the total error The calculation of can be shown as formula (3):

[0131] (3)

[0132] in, and are two weight factors, both of which are set to 1 in this disclosure.

[0133] Using the Stochastic Gradient Descent method to backpropagate the calculated error can optimize the parameters in the entire twin network.

[0134] Step 5: Continuously extract template images and search images from the training set of the single target tracking dataset and input them into the twin network based on the diversion attention mechanism. Repeat steps 1 to 4 until the error is less than the threshold of 0.01 set in this disclosure. The twin network is considered to have converged, that is, the target twin network is obtained.

[0135] According to some embodiments, the technical solution of the embodiments of the present disclosure can be run on a single 3090 graphics card, for example, to improve the robustness of the single target tracking method, especially the ability to adapt to interference in the background of the target.

[0136] In step S22, the target spatial feature fusion network based on the graph attention network in the target twin network is used to fuse the first template image features and the search image features of each search image to obtain the first spatial fusion feature;

[0137] Among them, the relevant description is as above and will not be repeated here.

[0138] According to some embodiments, fusing the first template image feature with the search image features of each search image to obtain a first spatial fusion feature includes:

[0139] Obtain a first node set corresponding to the first template image feature, wherein each pixel of the first template image feature is a node;

[0140] Obtaining a second node set corresponding to the search image feature of each search image, wherein each pixel of the search image feature of each search image is a node;

[0141] constructing a bipartite graph based on the first node set and the second node set, wherein the bipartite graph is used to indicate a correspondence between the first template image feature and the search image feature of each search image;

[0142] Obtaining a score between the first node and the second node according to the bipartite graph, a first eigenvector corresponding to the first node in the first node set, and a second eigenvector corresponding to the second node in the second node set, and regularizing the score using a Softmax function to obtain weight information, where the first node is any node in the first node set, and the second node is any node in the second node set;

[0143] Obtaining an aggregate feature corresponding to the first node based on at least one weight information transmitted to the first node by all nodes in the first subgraph of the bipartite graph, a second eigenvector corresponding to each node in the second node set, and a linear transformation matrix;

[0144] Fusing the aggregated feature with the first node feature of the first node to obtain a splicing feature;

[0145] Perform spatial feature fusion on at least one splicing feature corresponding to the first node set to obtain a first spatial fusion feature.

[0146] According to some embodiments, Figure 4 This is a flow chart of a spatial feature fusion network based on graph attention provided by an embodiment of the present disclosure. Figure 4 As shown, the template image features and search image features Input the Spatial Fusion Network based on graph attention, denoted as , and the enhanced and fused image features are obtained, which are recorded as The formula is expressed as formula (4):

[0147] (4)

[0148] Specifically, these may include:

[0149] A. Construct a bipartite graph:

[0150] The template image features and search image features Each pixel is regarded as a node, and It is a feature The set of all nodes on It is a feature The set of all nodes above.

[0151] Using a complete bipartite graph To construct the correspondence between the template image features and the search image features, where:

[0152] .

[0153] For example, you can define Two subgraphs of ,in Refers to the empty set;

[0154] B. Applying graph attention mechanism: Using graph attention mechanism to enhance and fuse template image features and search image features.

[0155] For each ,make Representation node and nodes The correlation score between , its formula can be shown as formula (5) for example:

[0156] (5)

[0157] in, Node and nodes The eigenvector of Represents the number of feature channels, score Proportional to the similarity between the features of two nodes.

[0158] Using the inner product between features as the similarity measure, the node features are first linearly transformed, and then the inner product between the transformed feature vectors is taken to calculate the correlation score. The formula can be shown as formula (6):

[0159] (6)

[0160] in and is a linear transformation matrix and a learnable parameter.

[0161] According to some embodiments, f is an intermediate variable that can represent h x and h z The degree of matching, The normalized weight can be further calculated based on this matching degree.

[0162] Use the Softmax function to Regularization, its formula can be shown as formula (7):

[0163] (7)

[0164] in, Slave nodes The perspective of the tracker to the node How much attention to devote.

[0165] Using All nodes in the The The attention of nodes, computing nodes The aggregate representation of , its formula can be shown as formula (8):

[0166] (8)

[0167] in is the linear transformation matrix, which is a learnable parameter. is the attention weight.

[0168] Finally, the aggregated features and node features are combined Fusion, to obtain a more powerful feature based on target information, can be shown as formula (9):

[0169] (9)

[0170] in Represents a vector concatenation operation.

[0171] For any node calculate , and finally complete the calculation of the graph attention mechanism, and obtain the features after spatial feature fusion using the graph attention mechanism , recorded as spatial fusion features.

[0172] In step S23, the target channel feature fusion network based on the graph attention network in the target twin network is used to enhance and fuse the first spatial fusion feature to obtain the first channel fusion feature, and the first channel fusion feature is used as the fusion feature;

[0173] Among them, the relevant description is as above and will not be repeated here.

[0174] According to some embodiments, performing enhancement processing and fusion processing on the first spatial fusion feature to obtain the first channel fusion feature includes:

[0175] Generate a random graph using the small-world model;

[0176] Encode the channel feature corresponding to the first spatial fusion feature to obtain the second node feature;

[0177] Decompose the second node feature according to the channel dimension information to obtain the decomposed node feature;

[0178] Sort the disassembled node features according to a preset order to obtain the third node feature of the random graph;

[0179] Perform graph attention convolution operation on the third node feature to obtain the fourth node feature;

[0180] The fourth node feature is subjected to feature extraction and value assignment, and is concatenated in the order corresponding to the channel dimensions to obtain the channel fused feature;

[0181] The channel-fused features are decoded to obtain the first channel fusion features. Using the Watts-Strogatz small-world model, a random graph can be generated to establish the topological connection between the feature channels, and a graph neural network can be used to learn the fusion of features in the channel dimension. Therefore, by applying the Watts-Strogatz random graph and the graph attention network in the graph neural network, a channel feature fusion module based on the graph attention network can be designed. Together with the spatial feature fusion module based on the graph attention network, a fusion network based on the graph attention network can be formed to enhance and fuse the template image features and the search image features. This allows the single target tracker to better distinguish the target and background areas in the search image by learning the topological relationship between the template image pixels and the search image pixels, and adapt to background interference during the tracking process, such as low-light backgrounds or backgrounds with similar objects, thereby improving the robustness of the tracker.

[0182] According to some embodiments, Figure 5 This is a flow chart of a channel feature fusion network based on graph attention provided by an embodiment of the present disclosure. Figure 5 As shown, the spatial fusion features Input the channel feature fusion network based on graph attention (Channel Fusion Network), denoted as , the image features after further enhancement and fusion are obtained, recorded as The formula can be expressed as, for example, formula (10):

[0183] (10)

[0184] According to some embodiments, a specific implementation of a channel feature fusion network based on graph attention may include:

[0185] A. Generate a random graph using the Watts-Strogatz small-world model:

[0186] Set the number of random graph nodes to ,Right now When generating the random graph, the average degree of the graph can be set to 2, for example, and the parameter in the Watts-Strogatz model is Set to 1.

[0187] B. Feature encoding:

[0188] Will The channel features of Figure 5 Node features on:

[0189] First Perform deformation operation and transform into , and then use the two parameter learnable linear transformation matrix and Perform feature encoding, and its formula can be shown as formula (11):

[0190] (11)

[0191] in, is the broadcast product, Sum for broadcast.

[0192] Parameter Matrix ,therefore ; parameter matrix ,therefore , is a four-dimensional tensor.

[0193] in, is the channel dimension, is the dimension of the channel feature. Among them, the Watts-Strogatz random graph pre-generated in step A has nodes, and the dimension of node features is .

[0194] C. Feature analysis:

[0195] Will Decomposing by channel dimension, the formula can be shown as formula (12):

[0196] (12)

[0197] Features obtained by disassembly Sequencing as a node feature of Watts-Strogatz random graph , the process of assignment is as follows Figure 5 As shown, its formula can be shown as formula (13):

[0198] (13)

[0199] in It is an assignment operation.

[0200] D. Feature transformation:

[0201] The graph attention mechanism is used to perform a graph attention convolution operation to fuse the channel features. The process is as follows: Figure 5As shown, the output of feature transformation is finally obtained, and its formula can be shown as formula (14):

[0202] (14)

[0203] E. Feature stitching:

[0204] Take out the node features on the graph and assign them to the new channel features , the process is as follows Figure 5 As shown, its formula can be shown as formula (15):

[0205] (15)

[0206] in .

[0207] The channel feature fusion result is obtained by splicing in the original order of the channel dimensions. The formula can be shown as formula (16):

[0208] (16)

[0209] This is a 4D tensor.

[0210] F. Feature decoding:

[0211] Will Decode the channel features to obtain channel fusion features .

[0212] Use a two-parameter learnable linear transformation matrix and Perform feature decoding, and its formula can be shown as formula (17):

[0213] (17)

[0214] in, is the broadcast product operation, It is a broadcast summing mechanism.

[0215] Parameter Matrix ,therefore ; parameter matrix ,therefore , is a three-dimensional tensor.

[0216] According to some embodiments, the spatial feature fusion processing based on graph attention can be performed first, and then the channel feature fusion processing based on graph attention can be performed, and the features obtained by the channel feature fusion processing based on graph attention can be used as the fusion feature. Output is performed, and its formula can be shown as formula (18):

[0217] (18)

[0218] in, It is an assignment operation.

[0219] In step S24, the fused features are input into the target tracking head network in the target twin network to obtain a first tracking result corresponding to the first target to be tracked in the first template image, wherein the first tracking result includes a first classification result corresponding to the first target to be tracked and a first regression result corresponding to the first target to be tracked.

[0220] Among them, the relevant description is as above and will not be repeated here.

[0221] According to some embodiments, for example, when identifying a bus, Figure 6 A comparative schematic diagram of target tracking based on a graph attention fusion network without adopting an embodiment of the present disclosure (before deformation) and target tracking based on a graph attention fusion network with adopting an embodiment of the present disclosure (after deformation) is shown.

[0222] In some or related embodiments, a target spatial feature fusion network based on a graph attention network in a target twin network is used to fuse the first template image features and the search image features of each search image to obtain a first spatial fusion feature; a target channel feature fusion network based on a graph attention network in a target twin network is used to enhance and fuse the first spatial fusion feature to obtain a first channel fusion feature, and the first channel fusion feature is used as a fusion feature. Therefore, the template image features and the search image features can be fused in the spatial dimension and then in the channel dimension, so that the network has a stronger ability to distinguish the boundaries of objects and realizes robust tracking of the tracker under background interference.

[0223] A block diagram of a target tracking device based on a graph attention fusion network is shown according to an exemplary embodiment. Figure 7 , the apparatus 700 comprises:

[0224] An image acquisition unit 701 is configured to use a target backbone network in a target twin network to acquire a first template image feature corresponding to a first template image and a search image feature of each search image in a single target tracking data set;

[0225] A feature fusion unit 702 is configured to use a feature fusion network based on a graph attention network in the target twin network to perform spatial feature fusion processing and channel feature fusion processing on the first template image features and the search image features of each search image to obtain fused features;

[0226] The target tracking unit 703 is used to input the fused features into the target tracking head network in the target twin network to obtain the first tracking result corresponding to the first target to be tracked in the first template image, wherein the first tracking result includes the first classification result corresponding to the first target to be tracked and the first regression result corresponding to the first target to be tracked.

[0227] According to some embodiments, the feature fusion unit 702 is configured to use a feature fusion network based on a graph attention network in the target twin network to perform spatial feature fusion processing and channel feature fusion processing on the first template image features and the search image features of each search image, and to obtain fused features, specifically for:

[0228] The target spatial feature fusion network based on the graph attention network in the target twin network is used to fuse the first template image features and the search image features of each search image to obtain the first spatial fusion feature;

[0229] The target channel feature fusion network based on the graph attention network in the target twin network is used to enhance and fuse the first spatial fusion feature, obtain the first channel fusion feature, and use the first channel fusion feature as the fusion feature.

[0230] According to some embodiments, the feature fusion unit 702 is configured to fuse the first template image feature with the search image features of each search image to obtain the first spatial fusion feature, specifically to:

[0231] Obtain a first node set corresponding to the first template image feature, wherein each pixel of the first template image feature is a node;

[0232] Obtaining a second node set corresponding to the search image feature of each search image, wherein each pixel of the search image feature of each search image is a node;

[0233] constructing a bipartite graph based on the first node set and the second node set, wherein the bipartite graph is used to indicate a correspondence between the first template image feature and the search image feature of each search image;

[0234] Obtaining a score between the first node and the second node according to the bipartite graph, a first eigenvector corresponding to the first node in the first node set, and a second eigenvector corresponding to the second node in the second node set, and regularizing the score using a Softmax function to obtain weight information, where the first node is any node in the first node set, and the second node is any node in the second node set;

[0235] Obtaining an aggregate feature corresponding to the first node based on at least one weight information transmitted to the first node by all nodes in the first subgraph of the bipartite graph, a second eigenvector corresponding to each node in the second node set, and a linear transformation matrix;

[0236] Fusing the aggregated feature with the first node feature of the first node to obtain a splicing feature;

[0237] Perform spatial feature fusion on at least one splicing feature corresponding to the first node set to obtain a first spatial fusion feature.

[0238] According to some embodiments, the feature fusion unit 702 is configured to perform enhancement processing and fusion processing on the first spatial fusion feature, and to obtain the first channel fusion feature, specifically to:

[0239] Generate a random graph using the small-world model;

[0240] Encode the channel feature corresponding to the first spatial fusion feature to obtain the second node feature;

[0241] Decompose the second node feature according to the channel dimension information to obtain the decomposed node feature;

[0242] Sort the disassembled node features according to a preset order to obtain the third node feature of the random graph;

[0243] Perform graph attention convolution operation on the third node feature to obtain the fourth node feature;

[0244] The fourth node feature is subjected to feature extraction and value assignment, and is concatenated in the order corresponding to the channel dimensions to obtain the channel fused feature;

[0245] The channel fused features are decoded to obtain the first channel fused features.

[0246] According to some embodiments, the initial Siamese network includes an initial backbone network, an initial spatial feature fusion network based on a graph attention network, an initial channel feature fusion network based on a graph attention network, and an initial tracking head network, and the image acquisition unit 701 is further used to:

[0247] Acquire a training data set, wherein the training data set includes at least one training sample image;

[0248] Using an initial backbone network, initially acquiring a second search image feature corresponding to the second search image and a training sample image feature corresponding to each training sample image in the training data set;

[0249] The initial spatial feature fusion network based on the graph attention network is used to fuse the second search image features and the training sample image to obtain the first spatial fusion feature;

[0250] The initial channel feature fusion network based on the graph attention network is used to enhance and fuse the first spatial fusion features to obtain the second channel fusion features;

[0251] Inputting the second channel fusion feature into the initial tracking head network to obtain a second tracking result corresponding to the second target to be tracked in the second search image, wherein the second tracking result includes a second classification result and a second regression result corresponding to the second target to be tracked;

[0252] Obtaining a first error between the second classification result and the third classification result and a second error between the second regression result and the third regression result, wherein the third classification result is a classification result obtained by labeling the second target to be tracked, and the third regression result is a regression result obtained by labeling the second target to be tracked;

[0253] The model parameters of the initial twin network are adjusted according to the first error and the second error until the initial twin network meets the network training requirements and the target twin network is obtained.

[0254] According to some embodiments, the image acquisition unit 701, when used to acquire the first error between the second classification result and the third classification result and the second error between the second regression result and the third regression result, is specifically configured to:

[0255] A binary cross entropy loss function is used to obtain the first error between the second classification result and the third classification result;

[0256] The intersection-over-union loss function is used to obtain the second error between the second regression result and the third regression result.

[0257] According to some embodiments, the image acquisition unit 701 is configured to adjust the model parameters of the initial twin network according to the first error and the second error until the initial twin network meets the network training requirements. When the target twin network is obtained, the image acquisition unit 701 is specifically configured to:

[0258] Obtaining a total error according to a first weight corresponding to the first error, the first error, a second weight corresponding to the second error, and the second error;

[0259] The stochastic gradient descent method is used to backpropagate the total error and adjust the model parameters of the initial twin network until the initial twin network meets the network training requirements and the target twin network is obtained.

[0260] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0261] In some or related embodiments, an image acquisition unit is used to adopt the target backbone network in the target twin network to obtain the first template image features corresponding to the first template image and the search image features of each search image in the single target tracking data set; a feature fusion unit is used to adopt the feature fusion network based on the graph attention network in the target twin network to perform spatial feature fusion processing and channel feature fusion processing on the first template image features and the search image features of each search image to obtain fused features; a target tracking unit is used to input the fused features into the target tracking head network in the target twin network to obtain the first tracking result corresponding to the first target to be tracked in the first template image, wherein the first tracking result includes the first classification result corresponding to the first target to be tracked and the first regression result corresponding to the first target to be tracked. Therefore, the topological relationship of the channel dimension can be increased, and channel feature fusion can be added on the basis of spatial feature fusion, so as to effectively enhance and fuse the template image features and the search image features in the channel dimension. This can reduce the situation where feature fusion only in the spatial dimension leads to fewer channels with effective activation of features after the fusion of template image features and search image features, and the ability to distinguish object boundaries is weak. This improves the resolution ability of the twin network for the target to be tracked, realizes robust tracking of the tracker under background interference, improves the accuracy of target recognition, and improves the accuracy of target tracking.

[0262] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device 800 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0263] like Figure 8As shown, electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. RAM 803 can also store various programs and data required for the operation of electronic device 800. Computing unit 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to bus 804.

[0264] Multiple components in the electronic device 800 are connected to the I / O interface 805, including an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0265] The computing unit 801 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above. For example, in some embodiments, the above-described methods may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed onto the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the above-described methods may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the above-described methods by any other suitable means (e.g., via firmware).

[0266] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0267] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0268] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0269] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0270] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0271] A computer system may include a client and a server. The client and server are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical servers and VPS services ("Virtual Private Servers" or "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0272] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0273] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A target tracking method based on graph attention fusion network, characterized in that: include: Using the target backbone network in the target twin network, obtain the first template image features corresponding to the first template image and the search image features of each search image in the single target tracking data set; Using the target spatial feature fusion network based on the graph attention network in the target twin network, the first template image feature and the search image feature of each search image are fused to obtain a first spatial fusion feature; Using the target channel feature fusion network based on the graph attention network in the target twin network, the first spatial fusion feature is enhanced and fused to obtain a first channel fusion feature, and the first channel fusion feature is used as the fusion feature; Inputting the fused features into the target tracking head network in the target twin network to obtain a first tracking result corresponding to the first target to be tracked in the first template image, wherein the first tracking result includes a first classification result corresponding to the first target to be tracked and a first regression result corresponding to the first target to be tracked; The performing enhancement processing and fusion processing on the first spatial fusion feature to obtain a first channel fusion feature includes: Generate a random graph using the small-world model; Encoding the channel features corresponding to the first spatial fusion features to obtain second node features; Decomposing the second node feature according to the channel dimension information to obtain the decomposed node feature; Sorting the disassembled node features according to a preset order to obtain a third node feature of the random graph; Performing a graph attention convolution operation on the third node feature to obtain a fourth node feature; Performing feature extraction and value assignment on the fourth node feature, and performing splicing processing in the order corresponding to the channel dimensions to obtain channel fused features; The channel-fused features are decoded to obtain first channel-fused features.

2. The method according to claim 1, characterized in that The fusing the first template image feature and the search image features of each search image to obtain a first spatial fusion feature includes: Obtaining a first node set corresponding to the first template image feature, wherein each pixel of the first template image feature is a node; Obtaining a second node set corresponding to the search image feature of each search image, wherein each pixel of the search image feature of each search image is a node; constructing a bipartite graph based on the first node set and the second node set, wherein the bipartite graph is used to indicate a correspondence between the first template image feature and the search image features of each search image; Obtaining scores between the first node and the second node based on the bipartite graph, a first eigenvector corresponding to a first node in the first node set, and a second eigenvector corresponding to a second node in the second node set, and regularizing the scores using a Softmax function to obtain weight information, wherein the first node is any node in the first node set, and the second node is any node in the second node set; Obtaining an aggregate feature corresponding to the first node based on at least one weight information transmitted to the first node by all nodes in the first subgraph of the bipartite graph, a second eigenvector corresponding to each node in the second node set, and a linear transformation matrix; Fusing the aggregated feature with the first node feature of the first node to obtain a splicing feature; Perform spatial feature fusion on at least one splicing feature corresponding to the first node set to obtain a first spatial fusion feature.

3. The method according to claim 1, characterized in that in, The initial Siamese network includes an initial backbone network, an initial spatial feature fusion network based on a graph attention network, an initial channel feature fusion network based on a graph attention network, and an initial tracking head network. The method further includes: Acquire a training data set, wherein the training data set includes at least one training sample image; Using the initial backbone network, initially acquiring a second search image feature corresponding to the second search image and a training sample image feature corresponding to each training sample image in the training data set; Using the initial spatial feature fusion network based on the graph attention network to fuse the second search image feature and the training sample image to obtain a first spatial fusion feature; Using the initial channel feature fusion network based on the graph attention network to enhance and fuse the first spatial fusion features to obtain second channel fusion features; Inputting the second channel fusion feature into the initial tracking head network to obtain a second tracking result corresponding to a second target to be tracked in the second search image, wherein the second tracking result includes a second classification result and a second regression result corresponding to the second target to be tracked; Obtaining a first error between the second classification result and a third classification result and a second error between the second regression result and a third regression result, wherein the third classification result is a classification result obtained by labeling the second target to be tracked, and the third regression result is a regression result obtained by labeling the second target to be tracked; The model parameters of the initial twin network are adjusted according to the first error and the second error until the initial twin network meets the network training requirements, and the target twin network is obtained.

4. The method according to claim 3, characterized in that The obtaining of a first error between the second classification result and the third classification result and a second error between the second regression result and the third regression result includes: Using a binary cross entropy loss function to obtain a first error between the second classification result and the third classification result; An intersection-over-union loss function is used to obtain a second error between the second regression result and the third regression result.

5. The method according to claim 3 or 4, characterized in that The adjusting the model parameters of the initial twin network according to the first error and the second error until the initial twin network meets the network training requirements and the target twin network is obtained includes: Obtaining a total error according to a first weight corresponding to the first error, the first error, a second weight corresponding to the second error, and the second error; The total error is back-propagated using the stochastic gradient descent method, and the model parameters of the initial twin network are adjusted until the initial twin network meets the network training requirements, and the target twin network is obtained.

6. A target tracking device based on a graph attention fusion network, characterized in that: include: An image acquisition unit, configured to acquire, using a target backbone network in a target twin network, a first template image feature corresponding to the first template image and a search image feature of each search image in a single target tracking data set; a feature fusion unit, configured to use a target space feature fusion network based on a graph attention network in the target twin network to fuse the first template image features and the search image features of each search image to obtain a first spatial fusion feature; Using the target channel feature fusion network based on the graph attention network in the target twin network, the first spatial fusion feature is enhanced and fused to obtain a first channel fusion feature, and the first channel fusion feature is used as the fusion feature; a target tracking unit, configured to input the fused features into a target tracking head network in the target twin network, and obtain a first tracking result corresponding to a first target to be tracked in the first template image, wherein the first tracking result includes a first classification result corresponding to the first target to be tracked and a first regression result corresponding to the first target to be tracked; The feature fusion unit is used to perform enhancement processing and fusion processing on the first spatial fusion feature, and when obtaining the first channel fusion feature, specifically to: Generate a random graph using the small-world model; Encoding the channel features corresponding to the first spatial fusion features to obtain second node features; Decomposing the second node feature according to the channel dimension information to obtain the decomposed node feature; Sorting the disassembled node features according to a preset order to obtain a third node feature of the random graph; Performing a graph attention convolution operation on the third node feature to obtain a fourth node feature; Performing feature extraction and value assignment on the fourth node feature, and performing splicing processing in the order corresponding to the channel dimensions to obtain channel fused features; The channel-fused features are decoded to obtain first channel-fused features.

7. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the target tracking method based on the graph attention fusion network as described in any one of claims 1 to 5.

8. A storage medium storing instructions, characterized in that: When the instruction is executed on an electronic device, the electronic device executes the target tracking method based on a graph attention fusion network as described in any one of claims 1 to 5.

Images