Embedded real-time single object tracking method based on feature clustering twin network

Abstract

The application provides an embedded real-time single-target tracking method based on a feature clustering twin network, relates to the cross technical field of electronic information and artificial intelligence, and comprises the following steps: template image features and search image features of a video sequence are extracted through a light feature extraction network; after feature enhancement, the template image features and the search image features are cross-fused through a feature clustering correlation operation network to obtain correlation operation features; a tracking head network is used to track the classification results of the single target category to be tracked and the regression results of the bounding box obtained by the correlation operation features, and a converged correlation operation network and a feature extraction network are trained to constitute an embedded real-time single-target tracking method based on a feature clustering twin network deployed to an embedded platform. Thus, the template image features and the search image features are effectively enhanced and fused through the feature clustering twin network, and real-time high-precision tracking of the embedded single target is realized.

Description

Technical Field

[0001] The present invention relates to the interdisciplinary technical field of electronic information and artificial intelligence, and in particular to an embedded real-time single target tracking method based on a feature clustering twin network. Background Art

[0002] Single-target video tracking, a fundamental research topic in computer vision, underpins tasks such as navigation and guidance, video surveillance, and scene understanding, and has broad application across a wide range of fields. Related technologies typically implement single-target video tracking based on twin networks. However, twin networks cannot meet the requirements for high-precision single-target tracking or real-time tracking. Therefore, a more reliable, embedded, real-time single-target tracking method based on a feature-clustering twin network is urgently needed. Summary of the Invention

[0003] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0004] To this end, the first purpose of the present invention is to propose an embedded real-time single target tracking method based on feature clustering twin network, which effectively enhances and fuses template image features and search image features through feature clustering twin network to achieve real-time and high-precision tracking of embedded single targets.

[0005] The second object of the present invention is to propose an embedded real-time single target tracking device based on feature clustering twin network.

[0006] A third object of the present invention is to provide an electronic device.

[0007] A fourth object of the present invention is to provide a non-transitory computer-readable storage medium storing computer instructions.

[0008] To achieve the above objectives, the first embodiment of the present invention proposes an embedded real-time single target tracking method based on a feature clustering twin network, the method comprising:

[0009] A lightweight feature extraction network is used to extract features from the template image and the search image respectively, obtaining the template image features and the search image features. The template image and the search image are obtained by cropping the video sequence of the single target to be tracked frame by frame.

[0010] Through a preset feature clustering correlation operation network, the template image features and the search image features are respectively enhanced and then cross-fused to obtain correlation operation features, wherein the correlation operation network includes a dual-branch feature self-enhancement module based on feature clustering constructed by a multi-layer neural network for image feature clustering and enhancement, and a dual-branch feature cross-fusion module based on feature clustering constructed by a neural network for image feature clustering and cross-fusion;

[0011] A tracking head network based on network tracking is selected to track the correlation operation features to obtain a prediction result of the correlation operation network for the single target to be tracked in the search image, wherein the prediction result includes a classification result of the category of the single target to be tracked and a regression result of the bounding box;

[0012] performing error calculation on the classification result and regression result and the true classification result and true regression result of the single target to be tracked in the search image; when the error value is greater than or equal to a set threshold, performing backpropagation on the calculated error value and adjusting the network parameters in the correlation operation network and the feature extraction network;

[0013] Until the error value corresponding to the adjustment of the network parameters is less than the set threshold, a converged correlation operation network and feature extraction network are obtained;

[0014] The correlation operation network and feature extraction network are composed of a feature clustering twin network and deployed on an embedded platform for embedded real-time single target tracking.

[0015] To achieve the above objectives, the second embodiment of the present invention proposes an embedded real-time single target tracking device based on a feature clustering twin network, the device comprising:

[0016] The feature extraction module is used to select a lightweight feature extraction network to extract features from the template image and the search image respectively, thereby obtaining the template image features and the search image features. The template image and the search image are obtained by cropping the video sequence of the single target to be tracked frame by frame.

[0017] A feature enhancement and cross-fusion module, configured to perform feature enhancement and cross-fusion on the template image features and the search image features respectively through a preset feature clustering correlation operation network, to obtain correlation operation features, wherein the correlation operation network includes a dual-branch feature self-enhancement module based on feature clustering constructed by a multi-layer neural network for image feature clustering and enhancement, and a dual-branch feature cross-fusion module based on feature clustering constructed by a neural network for image feature clustering and cross-fusion;

[0018] a prediction module, configured to track the correlation operation features by selecting a tracking head network based on network tracking to obtain a prediction result of the correlation operation network for the single target to be tracked in the search image, wherein the prediction result includes a classification result of the category of the single target to be tracked and a regression result of the bounding box;

[0019] a calculation module, configured to perform error calculation on the classification result and regression result and the true classification result and true regression result of the single target to be tracked in the search image; and when the error value is greater than or equal to a set threshold, backpropagate the calculated error value and adjust the network parameters in the correlation operation network and the feature extraction network;

[0020] An adjustment module is used to obtain a converged correlation operation network and a feature extraction network until the corresponding error value after adjusting the network parameters is less than a set threshold;

[0021] The tracking module is used to deploy the feature clustering twin network composed of the correlation operation network and the feature extraction network to the embedded platform for embedded real-time single target tracking.

[0022] To achieve the above-mentioned purpose, the third aspect embodiment of the present invention proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect.

[0023] In order to achieve the above-mentioned objectives, an embodiment of the fourth aspect of the present invention proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0024] The embodiments of the present invention provide an embedded real-time single-target tracking method, device, electronic device, and storage medium based on a feature clustering twin network. The method extracts template image features and search image features from a video sequence through a lightweight feature extraction network. The method then performs feature cross-fusion on the template image features and search image features through a feature clustering correlation operation network to obtain correlation operation features. The tracking head network tracks the classification results of the single target category to be tracked and the regression results of the bounding box obtained by tracking the correlation operation features. The method then trains a converged correlation operation network and feature extraction network to form a method based on a feature clustering twin network that is deployed on an embedded platform for embedded real-time single-target tracking. Thus, the template image features and search image features are effectively enhanced and fused through the feature clustering twin network, achieving real-time, high-precision tracking of an embedded single target.

[0025] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0027] Figure 1 A schematic flow chart of an embedded real-time single target tracking method based on a feature clustering twin network provided by an embodiment of the present invention;

[0028] Figure 2 A structural diagram of a dual-branch feature self-enhancement module based on feature clustering provided by an embodiment of the present invention;

[0029] Figure 3 A structural diagram of a dual-branch feature cross-fusion module based on feature clustering provided by an embodiment of the present invention;

[0030] Figure 4 A twin network structure diagram based on feature clustering provided by an embodiment of the present invention;

[0031] Figure 5 A structural schematic diagram of an embedded real-time single target tracking device based on a feature clustering twin network provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0033] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of the present invention comply with the relevant provisions of relevant laws and regulations.

[0034] The following describes an embedded real-time single target tracking method, device, electronic device and storage medium based on a feature clustering twin network according to an embodiment of the present invention with reference to the accompanying drawings.

[0035] Figure 1 A flow chart of an embedded real-time single target tracking method based on a feature clustering twin network provided in an embodiment of the present invention.

[0036] like Figure 1 As shown, the method includes the following steps:

[0037] Step 101: Select a lightweight feature extraction network to extract features from the template image and the search image respectively to obtain template image features and search image features, wherein the template image and the search image are obtained by cropping the video sequence of the single target to be tracked frame by frame.

[0038] In some possible implementations, the convolutional neural network AlexNet, which has achieved excellent performance in image classification tasks, is selected as the lightweight feature extraction network in the present invention. The AlexNet network is modified for the single-target tracking task. A template image and a search image extracted from a training set of a single-target tracking dataset are input into two AlexNet networks with identical structures and shared parameters. The output is the template image features and the search image features extracted by the AlexNet network.

[0039] Specifically, when the AlexNet network is selected as the feature extraction network in the present invention, it is recorded as The final fully connected layer of the AlexNet network is removed to meet the requirements of the single target tracking task. At the same time, in order to make the stride of the feature extraction network 8, the last downsampling convolution layer and subsequent operations in the network are removed.

[0040] The template image will be extracted from the training set of the single target tracking dataset and search images Input two AlexNet networks with the same structure and shared parameters for feature extraction, and output the template image features extracted by the AlexNet network. and search image features ,Right now:

[0041]

[0042] in, For a size of Tensor of and are height and width, c is the number of feature channels; For a size of Tensor of and For height and width.

[0043] Step 102, respectively, the template image features and search image features are enhanced by a preset feature clustering correlation operation network, and then cross-fused to obtain correlation operation features, wherein the correlation operation network includes a feature clustering based double-branch feature self-enhancement module constructed by a neural network for image feature clustering and enhancement, and a feature clustering based double-branch feature cross-fusion module constructed by a neural network for image feature clustering and cross-fusion.

[0044] Specifically, the template image features and the search image features are input into the feature clustering based correlation operation network, and the correlation operation features enhanced and fused are output. That is:

[0045] wherein, is a tensor with a size of , and are the height and width, is the number of feature channels.

[0046] In some possible embodiments, the template image features and the search image features are enhanced by a preset feature clustering correlation operation network, and then cross-fused to obtain correlation operation features, including: inputting the template image features into a feature clustering based double-branch feature self-enhancement module to obtain self-enhanced first template image features, and inputting the search image features into the same feature clustering based double-branch feature self-enhancement module to obtain self-enhanced first search image features; simultaneously inputting the first template image features and the first search image features into a feature clustering based double-branch feature cross-fusion module to obtain cross-fused second template image features and cross-fused second search image features; and taking the second template image features and the second search image features as correlation operation features, which can effectively enhance and fuse the template image features and the search image features.

[0047] Step 103, track the correlation operation features by selecting a tracking head network based on network tracking to obtain a prediction result of a single target to be tracked in the search image by the correlation operation network, wherein the prediction result includes a classification result of the single target to be tracked and a regression result of a bounding box.

[0048] In some possible embodiments, the correlation operation features are input into the tracking head network, denoted as , and a prediction result of a single target to be tracked in the search image is output, including a classification result and a regression result , .

[0049] in, For a size of Tensor of One size is a tensor of , and For height and width.

[0050] Figure 2 A structural diagram of a dual-branch feature self-enhancement module based on feature clustering provided by an embodiment of the present invention is shown as follows: Figure 2 As shown, feature extraction is performed on a template image (e.g., a single-person or single-vehicle video frame) and a search image (e.g., a multi-person or multi-vehicle video frame) to obtain template image features and search image features, and a first template image feature point set (obtained by two-dimensional position encoding of the template image features) is subjected to aggregating and diverging operations according to a set template image feature clustering center, and then processed by a multi-layer perceptron to obtain a self-enhanced first template image feature, and a first search image feature point set (obtained by two-dimensional position encoding of the template image features) is subjected to aggregating and diverging operations according to a search image feature clustering center, and then processed by a multi-layer perceptron to obtain a self-enhanced first search image feature.

[0051] Optionally, the template image features are input into a dual-branch feature self-enhancement module based on feature clustering to obtain a self-enhanced first template image feature, and the search image features are input into the same dual-branch feature self-enhancement module based on feature clustering to obtain a self-enhanced first search image feature, including: the dual-branch feature self-enhancement module based on feature clustering splits the template image features into multiple template image feature shapes (two-dimensional position encoding) by pixels, assigns a two-dimensional coordinate to each template image feature shape, and regards each template image feature shape as a point on a two-dimensional plane to form a first template image feature point set; the dual-branch feature self-enhancement module based on feature clustering splits the search image features into multiple first search image feature shapes by pixels to form a first template image feature point set. A two-dimensional coordinate is assigned to each first search image feature symbol in the same manner as the first template image feature symbol, and each first search image feature symbol is regarded as a point on the same two-dimensional plane as the first template image feature symbol to form a first search image feature point set; a point in the first template image feature point set is selected as the template image feature clustering center, and agglomeration and divergence operations are performed on the first template image feature point set, and then a self-enhanced first template image feature is obtained after processing by a multi-layer perceptron; a point in the first search image feature point set is selected as the search image feature clustering center, and agglomeration and divergence operations are performed on the first search image feature point set, and then a self-enhanced first search image feature is obtained after processing by a multi-layer perceptron.

[0052] Specifically, the template image features Input the dual-branch feature self-enhancing module based on feature clustering (Self-Enhancing Module), denoted as , get the first template image feature, recorded as ; will search for image features Input the same dual-branch feature self-enhancement module based on feature clustering , get the first search image feature, recorded as , the superscript k indicates that it is the kth dual-branch feature self-enhancement module. The formula is expressed as: .

[0053] The specific implementation of the dual-branch feature self-enhancement module based on feature clustering is as follows:

[0054] 1. If the dual-branch feature self-enhancement module based on feature clustering is the first module, that is, k=1, then the input template image feature and search image features Perform two-dimensional position encoding, that is, divide the template image features into the first template image feature shape and the search image features into the first search image feature shape according to the pixels:

[0055]

[0056] in, One size A tensor of , representing the mth first template image feature shape symbol, the first template image feature shape symbol has a total of indivual, and is the height and width of the template image feature; Is a size of A tensor representing the nth first search image feature shape, which has a total of indivual, and is the height and width of the search image feature.

[0057] According to the position of each first template image feature symbol in the original template image feature, a two-dimensional vector is obtained by encoding. Take the pixel from row i and column j, and its assigned two-dimensional vector yes:

[0058]

[0059] Similarly, according to the position of each first search image feature symbol in the original search image feature, a two-dimensional vector is encoded. Take the pixel from row i and column j, and its assigned two-dimensional vector yes:

[0060]

[0061] The two-dimensional vector is incorporated as a feature into each first template image feature shape or first search image feature shape to obtain a new feature:

[0062]

[0063] in, is a tensor concatenation operation, Is a size of Tensor of , Also a size of Tensor of .

[0064] Then use the fully connected layer to restore the new feature number c+2 to the number of original template image features c:

[0065]

[0066] in, It is a fully connected layer with an input dimension of c+2 and an output dimension of c. It is an assignment operation.

[0067] The above process completes the two-dimensional position encoding of the first template image feature shape and the first search image feature shape. The feature shape after two-dimensional position encoding is regarded as a point on a two-dimensional plane, and the first template image feature point set and the first search image feature point set can be obtained and output. These two point sets are and .

[0068] If the dual-branch feature self-enhancement module based on feature clustering is not the first module, that is, , then the input template image features and search image features The pixels are divided into the first template image feature shape and the second search image feature shape, which are directly output as the first template image feature point set and the first search image feature point set without two-dimensional position encoding.

[0069] 2. Select the first template image feature point to concentrate on one point As the template image feature clustering center, the first template image feature point set is first clustered:

[0070]

[0071] in, is a function, and is a learnable scalar, is the mth point in the template feature point set, and is the height and width of the template image feature. measure Cluster center point with template image features The similarity between them is calculated by inner product operation in the present invention. Plus a learnable fully connected layer express:

[0072]

[0073] Then the aggregated features Based on similarity Spread out:

[0074]

[0075] in, and is a learnable scalar.

[0076] Finally, a multi-layer perceptron deal with:

[0077]

[0078] in, It is an assignment operation.

[0079] The new template image feature point set after processing Reconvert to template image features pixel by pixel , which is used as the first template image feature Output.

[0080] 3. Select the first search image feature point to focus on As the search image feature clustering center, the search image feature point set is first clustered:

[0081]

[0082] in, yes function, and is a learnable scalar, is the nth point in the first search feature point set, and is the height and width of the search image feature. measure and search for image feature cluster centers The similarity between them is calculated by inner product operation in the present invention. Plus a learnable fully connected layer express:

[0083]

[0084] Then the aggregated features Based on similarity Spread out:

[0085]

[0086] in, and is a learnable scalar.

[0087] Finally, a multi-layer perceptron deal with:

[0088]

[0089] in, It is an assignment operation.

[0090] The new search image feature point set after processing Re-transform the search for image features by pixel , which is used as the first search image feature Output.

[0091] Figure 3 A structural diagram of a dual-branch feature cross-fusion module based on feature clustering provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, feature extraction is performed on a template image (e.g., a single-person or single-vehicle video frame) and a search image (e.g., a multi-person or multi-vehicle video frame) to obtain template image features and search image features, and a clustering operation and a diverging operation are performed on a second template image feature point set (obtained by performing two-dimensional position encoding on the first template image features) according to a set template image feature clustering center, and then processed by a multi-layer perceptron to obtain a cross-fused second template image feature, and a clustering operation and a diverging operation are performed on a second search image feature point set (obtained by performing two-dimensional position encoding on the first template image features) according to the search image feature clustering center, and then processed by a multi-layer perceptron to obtain a cross-fused second search image feature.

[0092] Optionally, the first template image feature and the first search image feature are simultaneously input into a two-branch feature cross-fusion module based on feature clustering to obtain a second template image feature of feature cross-fusion and a second search image feature of feature cross-fusion, including: the two-branch feature cross-fusion module based on feature clustering splits the first template image feature into multiple second template image feature shapes by pixel, forming a group of two-dimensional coordinate points, which constitute the second template image feature point set; the two-branch feature cross-fusion module based on feature clustering splits the first search image feature into multiple search image feature shapes by pixel, forming another group of two-dimensional coordinate points, which constitute the second search image feature point set; selects a point in the second search image feature point set as the template image feature clustering center, performs aggregation and divergence operations on the second template image feature point set, and then obtains the cross-fused second template image feature after processing by a multi-layer perceptron; selects a point in the second template image feature point set as the search image feature clustering center, performs aggregation and divergence operations on the second search image feature point set, and then obtains the cross-fused second search image feature after processing by a multi-layer perceptron.

[0093] Specifically, the first template image feature Input the dual-branch feature cross-fusion module (Cross-Fusing Module) based on feature clustering, denoted as , get the second template image feature, recorded as ; The first search image feature Input the same dual-branch feature cross-fusion module based on feature clustering , get the second search image feature, recorded as , the superscript k indicates the kth dual-branch feature cross-fusion module. The formula is: .

[0094] The specific implementation of the dual-branch feature cross-fusion module based on feature clustering is as follows:

[0095] 1. Input the first template image feature Divide the image into the second template image feature shape by pixel, divide and first search image feature Search for the second image feature:

[0096]

[0097] in, One size A tensor of , representing the mth second template image feature shape symbol, the second template image feature shape symbol has a total of indivual, and is the height and width of the first template image feature; Is a size of A tensor representing the nth second search image feature shape, which has a total of indivual, and The height and width of the first search image feature.

[0098] The second template image feature shape and the second search image feature shape are regarded as point sets on a two-dimensional plane, and the second template image feature point set and the second search image feature point set can be obtained and output. These two point sets are and .

[0099] 2. Select the second template image feature point to concentrate on one point As the template image feature clustering center, the second template image feature point set is first clustered:

[0100]

[0101] in, yes function, and is a learnable scalar, is the mth point in the second template feature point set, and are the height and width of the first template image feature. measure The center point of the first template image feature cluster The similarity between them is calculated by inner product operation in the present invention. Plus a learnable fully connected layer express:

[0102]

[0103] Then the aggregated features Based on similarity Spread out:

[0104]

[0105] in, and is a learnable scalar.

[0106] Finally, a multi-layer perceptron deal with:

[0107]

[0108] in, It is an assignment operation.

[0109] The new second template image feature point set after processing Reconvert pixel by pixel back to the first template image feature , which is used as the second template image feature Output.

[0110] 3. Select the second search image feature point to focus on As the search image feature clustering center, the second search image feature point set is first clustered:

[0111]

[0112] in, yes function, and is a learnable scalar, is the nth point in the second search feature point set, and The height and width of the first search image feature. measure and search for image feature cluster centers The similarity between them is calculated by inner product operation in the present invention. Plus a learnable fully connected layer express:

[0113]

[0114] Then the aggregated features Based on similarity Spread out:

[0115]

[0116] in, and is a learnable scalar.

[0117] Finally, a multi-layer perceptron deal with:

[0118]

[0119] in, It is an assignment operation.

[0120] The new second search image feature point set after processing Re-transform the search for image features by pixel , which is used as the second search image feature Output.

[0121] In addition, the present invention can cross-fuse the second search image features As a related operation feature To output, , It is an assignment operation.

[0122] In step 104, the classification result and regression result are compared with the true classification result and true regression result of the single target to be tracked in the search image. When the error value is greater than or equal to the set threshold, the calculated error value is back-propagated to adjust the network parameters in the relevant operation network and the feature extraction network.

[0123] In some possible implementations, the classification results predicted by the tracking head network are And the regression results The actual classification result of the single target to be tracked in the search image And the regression results Perform error calculation.

[0124] The error calculation of the classification result uses the binary cross entropy loss function (Binary Cross Entropy Loss), and the error calculation of the regression result uses the intersection over union loss function (IOULoss). The former is denoted as , the latter is recorded as , then the error loss is calculated as follows:

[0125]

[0126] in, and are two weight factors, both of which are set to 1 in the present invention.

[0127] The calculated error value is back-propagated using the Stochastic Gradient Descent (SGD) method to optimize the network parameters in the relevant operation network and feature extraction network.

[0128] Step 105 , until the corresponding error value after adjusting the network parameters is less than the set threshold, a converged correlation operation network and feature extraction network are obtained.

[0129] Optionally, the threshold value may be set to 0.01, but is not limited thereto. When the error value is less than 0.01, it is considered that the correlation operation network and the feature extraction network have converged.

[0130] Step 106: The correlation operation network and the feature extraction network are combined into a feature clustering twin network and deployed on an embedded platform to perform embedded real-time single target tracking.

[0131] Optionally, the feature clustering-based twin network can be deployed to the Horizon development hardware platform to test the real-time performance of single target tracking on the embedded platform. High-precision real-time single target tracking at 30 frames per second can be achieved.

[0132] The embedded real-time single-target tracking method based on the feature clustering twin network of the embodiment of the present invention extracts the template image features and search image features of the video sequence through a lightweight feature extraction network; through the feature clustering correlation operation network, the template image features and search image features are respectively enhanced and then cross-fused to obtain the correlation operation features; the classification results of the single target category to be tracked and the regression results of the bounding box are obtained by tracking the correlation operation features through the tracking head network, and a converged correlation operation network and feature extraction network are trained to form a method based on the feature clustering twin network deployed on the embedded platform for embedded real-time single-target tracking. Thus, the template image features and search image features are effectively enhanced and fused through the feature clustering twin network, realizing real-time and high-precision tracking of embedded single targets.

[0133] Figure 4A twin network structure diagram based on feature clustering provided by an embodiment of the present invention is used to execute an embedded real-time single-target tracking method based on a twin network of feature clustering. Specifically, through a feature extraction network, feature extraction is performed on the template image and the search image respectively to obtain the template image features and the search image features; through a preset feature clustering related operation network (a multi-layer dual-branch feature self-enhancement module based on feature clustering and a dual-branch feature cross-fusion module based on feature clustering), the template image features and the search image features are enhanced respectively. ( ), ( ) and then perform feature cross fusion ( ), ( ) to obtain relevant operation features; then the tracking head network tracks the relevant operation features to obtain the prediction results of the relevant operation network for the single target to be tracked in the search image, where the prediction results include the classification results of the single target category to be tracked and the regression results of the bounding box. Then, the converged relevant operation network and feature extraction network are trained as a feature clustering-based twin network and deployed on the embedded platform for real-time high-precision tracking.

[0134] In order to implement the above embodiment, the present invention also proposes an embedded real-time single target tracking device based on feature clustering twin network.

[0135] Figure 5 A structural schematic diagram of an embedded real-time single target tracking device based on a feature clustering twin network provided in an embodiment of the present invention.

[0136] like Figure 5 As shown, the embedded real-time single target tracking device 50 based on feature clustering twin network includes: a feature extraction module 51, a feature enhancement and cross-fusion module 52, a prediction module 53, a calculation module 54, an adjustment module 55, and a tracking module 56.

[0137] A feature extraction module 51 is configured to select a lightweight feature extraction network to perform feature extraction on the template image and the search image, respectively, to obtain template image features and search image features, wherein the template image and the search image are obtained by cropping the video sequence of the single target to be tracked frame by frame;

[0138] A feature enhancement and cross-fusion module 52 is configured to perform feature enhancement and cross-fusion on the template image features and the search image features respectively through a preset feature clustering correlation operation network, to obtain correlation operation features, wherein the correlation operation network includes a dual-branch feature self-enhancement module based on feature clustering constructed by a multi-layer neural network for image feature clustering and enhancement, and a dual-branch feature cross-fusion module based on feature clustering constructed by a neural network for image feature clustering and cross-fusion;

[0139] a prediction module 53 configured to track the correlation operation features by selecting a tracking head network based on network tracking to obtain a prediction result of the correlation operation network for the single target to be tracked in the search image, wherein the prediction result includes a classification result of the category of the single target to be tracked and a regression result of the bounding box;

[0140] a calculation module 54 for performing error calculation between the classification result and the regression result and the true classification result and the true regression result of the single target to be tracked in the search image; and when the error value is greater than or equal to a set threshold, performing backpropagation on the calculated error value to adjust the network parameters in the correlation operation network and the feature extraction network;

[0141] An adjustment module 55 is configured to obtain a converged correlation operation network and a feature extraction network until the corresponding error value after adjusting the network parameters is less than a set threshold;

[0142] The tracking module 56 is used to deploy the feature clustering twin network composed of the correlation operation network and the feature extraction network to the embedded platform for embedded real-time single target tracking.

[0143] Furthermore, in a possible implementation of the embodiment of the present invention, the feature enhancement and cross-fusion module 52 includes:

[0144] a feature enhancement unit, configured to input the template image features into a dual-branch feature self-enhancement module based on feature clustering to obtain a self-enhanced first template image feature, and input the search image features into the same dual-branch feature self-enhancement module based on feature clustering to obtain a self-enhanced first search image feature;

[0145] a feature cross-fusion unit, configured to simultaneously input the first template image feature and the first search image feature into a dual-branch feature cross-fusion module based on feature clustering, to obtain a second template image feature subjected to feature cross-fusion and a second search image feature subjected to feature cross-fusion;

[0146] An output unit is configured to use the second template image feature and the second search image feature as correlation operation features.

[0147] Furthermore, in a possible implementation of the embodiment of the present invention, the feature enhancement unit is specifically configured to:

[0148] The dual-branch feature self-enhancement module based on feature clustering splits the template image features into multiple first template image feature glyphs by pixel, assigns a two-dimensional coordinate to each first template image feature glyph, and regards each first template image feature glyph as a point on a two-dimensional plane to form a first template image feature point set;

[0149] A dual-branch feature self-enhancement module based on feature clustering splits the search image features into a plurality of first search image feature glyphs by pixel, assigns a two-dimensional coordinate to each first search image feature glyph in the same manner as the first template image feature glyph, and regards each first search image feature glyph as a point on the same two-dimensional plane as the first template image feature glyph to form a first search image feature point set;

[0150] Selecting a point in the first template image feature point set as the template image feature clustering center, performing aggregation and divergence operations on the first template image feature point set, and then processing it through a multi-layer perceptron to obtain a self-enhanced first template image feature;

[0151] A point in the first search image feature point set is selected as the search image feature clustering center, and a clustering operation and a divergence operation are performed on the first search image feature point set. After being processed by the multi-layer perceptron, a self-enhanced first search image feature is obtained.

[0152] Furthermore, in a possible implementation of the embodiment of the present invention, the feature cross-fusion unit is specifically configured to:

[0153] The dual-branch feature cross-fusion module based on feature clustering splits the first template image feature into multiple second template image feature shapes by pixel, forming a set of two-dimensional coordinate points, which constitute the second template image feature point set;

[0154] The dual-branch feature cross-fusion module based on feature clustering splits the first search image feature into multiple search image feature glyphs by pixel, forming another set of two-dimensional coordinate points, which constitute the second search image feature point set;

[0155] Selecting a point in the second search image feature point set as a template image feature clustering center, performing agglomeration and divergence operations on the second template image feature point set, and then processing the second template image feature through a multi-layer perceptron to obtain a cross-fused second template image feature;

[0156] A point in the second template image feature point set is selected as the search image feature clustering center, and agglomeration and divergence operations are performed on the second search image feature point set. After being processed by a multi-layer perceptron, a cross-fused second search image feature is obtained.

[0157] It should be noted that the above explanation of the method embodiment is also applicable to the device of this embodiment and will not be repeated here.

[0158] The embedded real-time single-target tracking device based on the feature clustering twin network of the embodiment of the present invention extracts the template image features and search image features of the video sequence through a lightweight feature extraction network; through the feature clustering correlation operation network, the template image features and search image features are enhanced and then cross-fused to obtain correlation operation features; the classification results of the single target category to be tracked and the regression results of the bounding box are obtained by tracking the correlation operation features through the tracking head network, and a converged correlation operation network and feature extraction network are trained to form a method based on the feature clustering twin network deployed on the embedded platform for embedded real-time single-target tracking. As a result, the template image features and search image features are effectively enhanced and fused through the feature clustering twin network, realizing real-time and high-precision tracking of embedded single targets.

[0159] In order to implement the above embodiment, the present invention further provides an electronic device, including:

[0160] at least one processor; and

[0161] a memory communicatively connected to the at least one processor; wherein,

[0162] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the aforementioned method.

[0163] In order to implement the above embodiment, the present invention further proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable the computer to execute the above method.

[0164] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0165] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0166] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0167] The logic and / or steps represented in flow diagrams or otherwise described herein, for example, can be considered as a sequence of instructions to implement logic functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this specification, a "computer-readable medium" can be any means that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a machine-readable storage device (e.g., magnetic, optical or other) a machine-readable storage diskette (e.g., floppy disk, optical disk, etc.), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), optical fibers, and a portable compact disc read-only memory (CDROM). Further, the computer-readable medium can even be paper or other suitable medium upon which the program is printed, as the program can be electronically captured, for example via the optical scanner of a device or other electronic capture device, and then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and stored in a computer memory.

[0168] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. As such, in some embodiments, the steps or methods can be implemented in a combination of hardware and software. If implemented in hardware, as in another embodiment, any of the above techniques can be implemented with or without the use of a programmable data processing apparatus, using any of the following technologies: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0169] Those of skill in the art would understand that the steps or methods carried out in the above-described embodiments can be carried out by program instructions executed by a processor, and that the program instructions can be stored in a computer readable storage medium. The program instructions, when executed by the processor, can cause the processor to carry out the steps or methods of the embodiments.

[0170] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0171] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and are not to be construed as limiting the present invention. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. An embedded real-time single target tracking method based on feature clustering twin network, characterized in that: The method comprises: A lightweight feature extraction network is selected to extract features from the template image and the search image respectively, obtaining template image features and search image features. The template image and the search image are obtained by cropping the video sequence of the single target to be tracked frame by frame. The lightweight feature extraction network is an AlexNet network with the fully connected layer removed and the step size adjusted to 8. Inputting the template image features into a dual-branch feature self-enhancement module based on feature clustering, the dual-branch feature self-enhancement module based on feature clustering splits the template image features into a plurality of first template image feature glyphs by pixel, assigns a two-dimensional coordinate to each first template image feature glyph, and regards each first template image feature glyph as a point on a two-dimensional plane to form a first template image feature point set; A dual-branch feature self-enhancement module based on feature clustering splits the search image features into a plurality of first search image feature glyphs by pixel, assigns a two-dimensional coordinate to each first search image feature glyph in the same manner as the first template image feature glyph, and regards each first search image feature glyph as a point on the same two-dimensional plane as the first template image feature glyph to form a first search image feature point set; Selecting a point in the first template image feature point set as the template image feature clustering center, performing aggregation and divergence operations on the first template image feature point set, and then processing it through a multi-layer perceptron to obtain a self-enhanced first template image feature; Selecting a point in the first search image feature point set as a search image feature clustering center, performing aggregating and diverging operations on the first search image feature point set, and then processing the first search image feature set through the multi-layer perceptron to obtain a self-enhanced first search image feature; Inputting the first template image feature and the first search image feature into a dual-branch feature cross-fusion module based on feature clustering at the same time to obtain a second template image feature of feature cross-fusion and a second search image feature of feature cross-fusion; using the second template image feature and the second search image feature as correlation operation features; A tracking head network based on network tracking is selected to track the correlation operation features to obtain a prediction result of the correlation operation network for a single target to be tracked in a search image, wherein the prediction result includes a classification result of the category of the single target to be tracked and a regression result of a bounding box. The correlation operation network includes a dual-branch feature self-enhancement module based on feature clustering constructed by a multi-layer neural network for performing image feature clustering and enhancement, and a dual-branch feature cross-fusion module based on feature clustering constructed by a neural network for performing image feature clustering and cross-fusion. performing error calculation on the classification result and regression result and the true classification result and true regression result of the single target to be tracked in the search image; when the error value is greater than or equal to a set threshold, performing backpropagation on the calculated error value and adjusting the network parameters in the correlation operation network and the feature extraction network; Until the error value corresponding to the adjustment of the network parameters is less than the set threshold, a converged correlation operation network and feature extraction network are obtained; The correlation operation network and feature extraction network are composed of a feature clustering twin network and deployed on an embedded platform for embedded real-time single target tracking.

2. The method according to claim 1, characterized in that The step of simultaneously inputting the first template image feature and the first search image feature into a dual-branch feature cross-fusion module based on feature clustering to obtain a second template image feature and a second search image feature by cross-fusion, includes: The dual-branch feature cross-fusion module based on feature clustering splits the first template image feature into multiple second template image feature shapes by pixel, forming a set of two-dimensional coordinate points, which constitute the second template image feature point set; The dual-branch feature cross-fusion module based on feature clustering splits the first search image feature into multiple search image feature glyphs by pixel, forming another set of two-dimensional coordinate points, which constitute the second search image feature point set; Selecting a point in the second search image feature point set as a template image feature clustering center, performing agglomeration and divergence operations on the second template image feature point set, and then processing the second template image feature through a multi-layer perceptron to obtain a cross-fused second template image feature; A point in the second template image feature point set is selected as the search image feature clustering center, and agglomeration and divergence operations are performed on the second search image feature point set. After being processed by a multi-layer perceptron, a cross-fused second search image feature is obtained.

3. An embedded real-time single target tracking device based on feature clustering twin network, characterized in that: The device comprises: A feature extraction module is configured to select a lightweight feature extraction network to extract features from the template image and the search image, respectively, to obtain template image features and search image features. The template image and the search image are obtained by cropping the video sequence of the single target to be tracked frame by frame. The lightweight feature extraction network is an AlexNet network with the fully connected layer removed and the step size adjusted to 8. A feature enhancement and cross-fusion module, configured to perform feature enhancement and cross-fusion on the template image features and the search image features respectively through a preset feature clustering correlation operation network, to obtain correlation operation features, wherein the correlation operation network includes a dual-branch feature self-enhancement module based on feature clustering constructed by a multi-layer neural network for image feature clustering and enhancement, and a dual-branch feature cross-fusion module based on feature clustering constructed by a neural network for image feature clustering and cross-fusion; a prediction module, configured to track the correlation operation features by selecting a tracking head network based on network tracking to obtain a prediction result of the correlation operation network for the single target to be tracked in the search image, wherein the prediction result includes a classification result of the category of the single target to be tracked and a regression result of the bounding box; a calculation module, configured to perform error calculation on the classification result and regression result and the true classification result and true regression result of the single target to be tracked in the search image; and when the error value is greater than or equal to a set threshold, backpropagate the calculated error value and adjust the network parameters in the correlation operation network and the feature extraction network; An adjustment module is used to obtain a converged correlation operation network and a feature extraction network until the corresponding error value after adjusting the network parameters is less than a set threshold; The tracking module is used to deploy the feature clustering twin network composed of the correlation operation network and the feature extraction network to the embedded platform for embedded real-time single target tracking; The feature enhancement and cross-fusion module includes: a feature enhancement unit, configured to input the template image features into a dual-branch feature self-enhancement module based on feature clustering, wherein the dual-branch feature self-enhancement module based on feature clustering splits the template image features into a plurality of first template image feature glyphs by pixel, assigns a two-dimensional coordinate to each first template image feature glyph, and regards each first template image feature glyph as a point on a two-dimensional plane to form a first template image feature point set; A dual-branch feature self-enhancement module based on feature clustering splits the search image features into a plurality of first search image feature glyphs by pixel, assigns a two-dimensional coordinate to each first search image feature glyph in the same manner as the first template image feature glyph, and regards each first search image feature glyph as a point on the same two-dimensional plane as the first template image feature glyph to form a first search image feature point set; Selecting a point in the first template image feature point set as the template image feature clustering center, performing aggregation and divergence operations on the first template image feature point set, and then processing it through a multi-layer perceptron to obtain a self-enhanced first template image feature; Selecting a point in the first search image feature point set as a search image feature clustering center, performing aggregating and diverging operations on the first search image feature point set, and then processing the first search image feature set through the multi-layer perceptron to obtain a self-enhanced first search image feature; a feature cross-fusion unit, configured to simultaneously input the first template image feature and the first search image feature into a dual-branch feature cross-fusion module based on feature clustering, to obtain a second template image feature subjected to feature cross-fusion and a second search image feature subjected to feature cross-fusion; The output unit is configured to use the second template image feature and the second search image feature as correlation operation features.

4. The device according to claim 3, characterized in that The feature cross-fusion unit is specifically used to: The dual-branch feature cross-fusion module based on feature clustering splits the first template image feature into multiple second template image feature shapes by pixel, forming a set of two-dimensional coordinate points, which constitute the second template image feature point set; The dual-branch feature cross-fusion module based on feature clustering splits the first search image feature into multiple search image feature glyphs by pixel, forming another set of two-dimensional coordinate points, which constitute the second search image feature point set; Selecting a point in the second search image feature point set as a template image feature clustering center, performing agglomeration and divergence operations on the second template image feature point set, and then processing the second template image feature through a multi-layer perceptron to obtain a cross-fused second template image feature; A point in the second template image feature point set is selected as the search image feature clustering center, and agglomeration and divergence operations are performed on the second search image feature point set. After being processed by a multi-layer perceptron, a cross-fused second search image feature is obtained.

5. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 2.

6. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-2.

Images