Target motion tracking method based on brain-like memory network

Abstract

The application discloses a target motion tracking method based on a brain-like memory network and belongs to the technical field of target tracking. Technical solution points are as follows: after a feature branch output by a backbone feature extraction network VGG16, a brain-like long-term memory network is constructed. The brain-like long-term memory network is composed of a long short-term memory network and an attention driving module. First, a long-term memory mechanism is constructed to explore target appearance changes from two dimensions of time and space and effectively maintain target information. Meanwhile, the attention driving module is used to strengthen information related to the target in memory and reduce the dependence of a tracker on scene information, so that a brain-like long-term memory model is obtained, the reliability of a response graph is improved, accurate target positioning is realized, the accuracy of a current tracking result is evaluated through a designed threshold, the stability of memory updating is ensured, the robustness of the model to target changes is improved, and therefore the adaptability of a tracking algorithm to challenges such as target deformation is improved.

Description

Technical Field

[0001] This invention relates to the field of target tracking technology, and in particular to a target motion tracking method based on a brain-like memory network. Background Technology

[0002] Visual object tracking is a task that involves locating and scale-estimating any object in a video sequence given a target state in the initial frame. Object tracking is widely used in computer vision tasks such as autonomous driving and video surveillance. Although some trackers have demonstrated excellent tracking performance in recent years, visual object tracking remains an important research topic in computer vision due to challenges such as drastic deformation, lighting variations, and occlusion.

[0003] In recent years, memory mechanisms have attracted attention in tracking tasks due to their ability to utilize target information, thus providing trackers with higher robustness. Although these trackers strive to extract target information from memory mechanisms, tracking failures still occur in complex environments. This is mainly because the memory content is not mined, resulting in limited memory representation capabilities. Furthermore, during tracking, the target state changes constantly over time; without effective and timely updates to the memory content, the tracker will be unable to accurately predict the target's location. When drastic deformations and cluttered backgrounds occur, simple memory storage mechanisms merely preserve previously observed target states, failing to mine the inherently invariant attributes of the target within the memory. Therefore, they cannot effectively distinguish the target from its surrounding background information, leading to tracking drift or even tracking failure.

[0004] Therefore, it is necessary to propose a target tracking method that can mine the essential attributes of the target, reduce the tracker's dependence on scene information, and effectively highlight the target's brain-like long-term memory. Summary of the Invention

[0005] To address the shortcomings of the aforementioned background technologies, this invention proposes a target motion tracking method based on a brain-like memory network to improve the robustness of the tracking algorithm. This method solves the problem that traditional memory-related tracking algorithms cannot obtain the essential memory of the target and are unable to effectively improve the tracking adaptability to changes in the target's appearance.

[0006] The technical solution of this invention is implemented as follows:

[0007] A target motion tracking method based on brain-like memory networks includes the following steps:

[0008] S1. Based on the initial frame image and the target's initial true position information, the VGG16 backbone network is used to extract the target's feature information, and the long-term memory network is trained online to store the target's true state in the brain-like long-term memory network.

[0009] S2. Read the next frame of the image and extract the features of the current image using the VGG16 backbone network;

[0010] S3. Input the current frame feature information into the brain-like long-term memory network, combine it with the previously memorized content, and use the attention-driven module to mine the target information and predict the target location;

[0011] S4. Determine whether the confidence level of the current tracking result is greater than the stability threshold. If yes, proceed to step S5; otherwise, return to step S2 until the video sequence ends.

[0012] S5. Extract the target image from the tracking results, use the VGG16 backbone network to extract target feature information, write the extracted target features into memory, and use the target feature information in memory to update the brain-like long-term tracking network online.

[0013] Furthermore, in step S3, the attention-driven module performs the target information mining process as follows:

[0014]

[0015] in Represents dot product. Represents the channel space attention mechanism, s t The memory knowledge output by the forgetting gate, u t c represents the memory content of the target in the current input. t This is a memory state, used to store long-term memory information about previous targets.

[0016] Furthermore, in step S3, note the following regarding the target memory mining process in the driving module:

[0017] S3.1. Merge the two parts of memory to obtain a comprehensive memory. The merged memory information is represented as follows:

[0018] F = s t +u t

[0019] S3.2. Mining target memory content from the channels, the channel attention representation is as follows:

[0020] M c (F)=σ(MLP(Avgpool(F))+MLP(Maxpool(F)))

[0021] Where F represents the fused memory information, Avgpool() represents global average pooling, Maxpool() represents global max pooling, MLP represents the final channel attention feature map generated in a shared multilayer perceptron network, and σ represents the Sigmoid function.

[0022] S3.3, Then obtain the memory information obtained through channel attention mining:

[0023]

[0024] Where F′ represents the memory information enhanced by channel attention, and M... c (F) is the channel attention feature map;

[0025] S3.4 Next, we will explore the target memory content from a spatial dimension. Spatial attention is represented as follows:

[0026] M s (F′)=σ(f 7×7 ([Avgpool(F′); Maxpool(F′)]))

[0027] Among them, f 7×7 For a convolution operation of size 7, M s (F′) is the spatial attention feature map;

[0028] S3.5 Finally, the spatial attention feature map M s Multiplying (F′) by the memory information F′ enhanced by channel attention completes the exploration of the memory content by the attention-driven module, yielding the final output.

[0029]

[0030] Furthermore, in step S5, the loss function during online training is:

[0031]

[0032] Where exp(Y i ) represents the value of the soft label, indicating the difficulty level of the sample, β i is the gradient coordination factor, N is the number of samples, and di is the difference between the output and the label.

[0033] Compared with the prior art, the beneficial effects of the present invention are:

[0034] This invention utilizes an attention-driven module and previously memorized content to mine feature information that expresses the essential attributes of the target. It explores changes in the target's appearance from both temporal and spatial dimensions, reducing the tracker's dependence on scene information, improving the reliability of the response map, and achieving accurate target localization. Then, a designed threshold is used to evaluate the accuracy of the current tracking results, ensuring the stability of memory updates and improving the model's robustness to target changes, thereby enhancing the tracking algorithm's adaptability to challenges such as target deformation. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0036] Figure 1 This is a flowchart of the target motion tracking method based on a brain-like memory network according to the present invention;

[0037] Figure 2 This is a schematic diagram comparing the effects of the brain-like memory network and the long short-term memory network of the present invention; the first row shows the effect of the long short-term memory network, and the second row shows the effect of the brain-like memory network.

[0038] Figure 3 This is a schematic diagram comparing the present invention with other algorithms on the OTB-50 and OTB-100 datasets;

[0039] Figure 4 This is a schematic diagram illustrating the tracking effect of this invention compared to other algorithms. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0041] like Figure 1 As shown, a target motion tracking method based on a brain-like memory network includes the following steps:

[0042] S1. Based on the initial frame image and the target's initial true position information, the VGG16 backbone network is used to extract the target's feature information, and the long-term memory network is trained online to store the target's true state in the brain-like long-term memory network.

[0043] S2. Read the next frame of the image and extract the features of the current image using the VGG16 backbone network;

[0044] S3. Input the current frame feature information into the brain-like long-term memory network, combine it with the previously memorized content, and use the attention-driven module to mine the target information and predict the target location;

[0045] The attention-driven module's process for mining target information is as follows:

[0046]

[0047] in Represents dot product. Represents the channel space attention mechanism, s tThe memory knowledge output by the forgetting gate, u t c represents the memory content of the target in the current input. t This is a memory state used to store long-term memory information about previous targets;

[0048] Note that the target memory mining process in the driver module is as follows:

[0049] S3.1. Merge the two parts of memory to obtain a comprehensive memory. The merged memory information is represented as follows:

[0050] F = s t +u t

[0051] S3.2. Mining target memory content from the channels, the channel attention representation is as follows:

[0052] M c (F)=σ(MLP(Avgpool(F))+MLP(Maxpool(F)))

[0053] Where F represents the fused memory information, Avgpool() represents global average pooling, Maxpool() represents global max pooling, MLP represents the final channel attention feature map generated in a shared multilayer perceptron network, and σ represents the Sigmoid function.

[0054] S3.3, Then obtain the memory information obtained through channel attention mining:

[0055]

[0056] Where F′ represents the memory information enhanced by channel attention, and M... c (F) is the channel attention feature map;

[0057] S3.4 Next, we will explore the target memory content from a spatial dimension. Spatial attention is represented as follows:

[0058] M s (F′)=σ(f 7×7 ([Avgpool(F′); Maxpool(F′)]))

[0059] Among them, f 7×7 For a convolution operation of size 7, M s (F′) is the spatial attention feature map;

[0060] S3.5 Finally, the spatial attention feature map M s Multiplying (F′) by the memory information F′ enhanced by channel attention completes the exploration of the memory content by the attention-driven module, yielding the final output.

[0061]

[0062] S4. Determine whether the confidence level of the current tracking result is greater than the stability threshold. If yes, proceed to step S5; otherwise, return to step S2 until the video sequence ends.

[0063] S5. Extract the target image from the tracking results, use the VGG16 backbone network to extract target feature information, write the extracted target features into the memory, and use the target feature information in the memory to update the brain-like long-term tracking network online.

[0064] The loss function during online training is:

[0065]

[0066] Where exp(Y i ) represents the value of the soft label, indicating the difficulty level of the sample, β i The gradient coordination factor describes the distribution of samples under different gradient norms. N is the number of samples, and di is the difference between the output and the label. The weighted gradient coordination loss function can balance the proportion of different samples according to the gradient norm, reducing overfitting of simple negative samples. Finally, the brain-like memory network can maintain the memory of the target's state at different times through online training.

[0067] The experiments of this invention were conducted using Python 3.6 on a computer with an Intel i7-9700K CPU (3.60GHz), 16GB RAM, and an NVIDIA Quadro RTX 4000.

[0068] like Figure 2 As shown, this invention, through a brain-like long-term memory network, can obtain more essential memory content, highlight target expression, and more effectively cope with deformation challenges. Figure 3 As shown, the target tracking algorithm (ours) based on a brain-like memory network provided in this invention demonstrates superior performance in terms of accuracy compared to other advanced target tracking algorithms (CFNet, fDSST, SiamFC, SRDCF, Staple) on the OTB-50 and OTB-100 datasets. Figure 4As shown in the figure, the target tracking algorithm based on a neuromorphic memory network provided in this invention and various excellent algorithms are demonstrated in some frames, namely four videos: Bird1, DragonBaby, Human3, and Skating1. In Bird1, there are interference objects similar to the target and severe deformation of the target during flight. In DragonBaby, the target is blurred due to its rapid movement. In the Human3 scene, the target is obscured by other passersby and faces the challenge of interference from similar targets. In Skating1, the background is cluttered, and there is similar interference from other dancers, as well as target deformation caused by changes in lighting. This poses a great challenge to the tracker and may even cause some algorithms to fail. Figure 4 As can be seen from the results, the method proposed in this invention achieves better tracking performance.

[0069] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for object motion tracking based on a brain-like memory network, characterized in that, Includes the following steps: S1. Based on the initial frame image and the target's initial true position information, extract the target's feature information using the VGG16 backbone network, train the brain-like memory network online, and store the target's true state in the brain-like memory network. S2. Read the next frame of the image and extract the features of the current image using the VGG16 backbone network; S3. Input the current frame feature information into the brain-like memory network, combine it with the target's true state stored in the brain-like memory network in step S1, and use the attention-driven module to mine the target information and predict the target's location. S4. Determine whether the confidence level of the current tracking result is greater than the stability threshold. If yes, proceed to step S5; otherwise, return to step S2 until the video sequence ends. S5. Extract the target image from the tracking results, use the VGG16 backbone network to extract target feature information, write the extracted target features into the memory, and use the target feature information in the memory to update the brain-like memory network online. In step S3, the attention-driven module performs the target information mining process as follows: wherein represents a point-wise multiplication, represents a channel spatial attention mechanism, s t represents the memory knowledge output by the forget gate, u t is the memory content in the current input that represents the target, c t is the memory state, which stores the long-term memory of previous target information.

2. The target motion tracking method based on brain-like memory networks according to claim 1, characterized in that, In step S3, an alternative attention-driven module is used, in which the target memory mining process is as follows: S3.

1. Merge the two parts of memory to obtain a comprehensive memory. The merged memory information is represented as follows: F = s t +u t wherein s t represents the memory knowledge output by the forgetting gate, u t is the memory content representing the target in the current input; S3.

2. Mining target memory content from the channels, the channel attention representation is as follows: M c (F) = σ(MLP(Avgpool(F)) + MLP(Maxpool(F))) Where F represents the fused memory information, Avgpool() represents global average pooling, Maxpool() represents global max pooling, MLP represents the final channel attention feature map generated in a shared multilayer perceptron network, and σ represents the Sigmoid function. S3.3, Then obtain the memory information obtained through channel attention mining: wherein F' is the memory information reinforced by channel attention, M c (F) is the channel attention feature map; S3.4 Next, we will explore the target memory content from a spatial dimension. Spatial attention is represented as follows: M s (F′) = σ(f 7×7 ([Avgpool(F′) ; Maxpool(F′)])) wherein f 7×7 is a convolution operation of size 7, M s (F′) is a spatial attention feature map; S3.5 Finally, the spatial attention feature map M s Multiplying (F′) by the memory information F′ enhanced by channel attention completes the exploration of the memory content by the attention-driven module, yielding the final output.

3. The target motion tracking method based on brain-like memory networks according to claim 1, characterized in that, In step S5, the loss function during online training is: where exp(Y i ) is the value of the soft label, representing the difficulty of the sample, β i is the gradient coordination factor, N is the number of samples, and d i is the difference between the output and the label.

Images