Scale-adaptive active visual tracking method

Abstract

The invention discloses a scale-adaptive active visual tracking method, and aims to enhance the tracking capability of an active tracker under the condition of target scale change and improve the tracking performance. According to the technical scheme, a scale-adaptive active visual tracking system composed of a data acquisition module, a data preprocessing module, an active visual tracking teacher module and an active visual tracking student module is constructed; and training the active visual tracking teacher module by adopting a reinforcement learning mode, training the active visual tracking student module by adopting a supervised learning mode, and obtaining the trained scale adaptive active visual tracking system. And the trained scale adaptive active visual tracking system tracks a target. According to the method, the conventional-scale training, the small-scale training and the large-scale training are respectively carried out in the training, so that the multi-scale features of the visual observation image can be extracted, the multi-scale features are adaptively fused, and the tracking performance of a tracker under the condition that the target scale changes and interferents exist in the environment is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of active visual tracking, and in particular to a scale-adaptive active visual tracking method. Background Art

[0002] Visual object tracking is a key technology in the field of computer vision. It refers to the process of tracking one or more objects in a continuous sequence of images. This process involves detecting the object of interest in a video frame and continuously locating the position of the object in subsequent frames. This is crucial for understanding video content, intelligent monitoring, augmented reality and other fields. Visual object tracking can be divided into two categories according to whether the camera motion needs to be controlled: passive visual tracking and active visual tracking. The task of passive visual tracking is to predict the positioning frame of the target in subsequent frames given the target positioning frame in the initial frame, without considering the camera motion control problem; unlike passive visual tracking, active visual tracking not only considers locating the target in the video sequence, but also involves controlling the movement of the camera to actively follow the target so that the target is always located in the center of the camera field of view with a suitable size, that is, in the ideal position. Active visual tracking includes two subtasks: visual perception and motion control. The visual perception task is responsible for extracting features from the input visual observation image and detecting and identifying the target, while the motion control task outputs the camera action based on the perception results.

[0003] Active visual tracking methods can be divided into two categories: traditional two-stage methods and end-to-end methods based on deep reinforcement learning. The traditional two-stage method divides active visual tracking into two modules: passive visual tracking and controller. The passive visual tracking module is used to obtain the positioning frame of the target from the input visual observation image, and the controller module outputs the camera's action signal according to the difference between the position of the positioning frame and the ideal position. The two-stage method has the following limitations: the passive visual tracking module usually adopts the existing passive visual tracking algorithm, and training these algorithms requires a large number of manually annotated data sets, which is time-consuming and labor-intensive; the intermediate results output by the passive visual tracking module may not be suitable for action decision-making; the joint debugging between the passive visual tracking and controller modules is difficult. With the continuous development of deep reinforcement learning technology in recent years and its success in many control decision-making fields such as robot motion control, the end-to-end method based on deep reinforcement learning has made great progress and has become a research hotspot.

[0004] Reinforcement learning has powerful decision-making capabilities. Through continuous interaction and trial and error between the agent (including the tracker, target, and distractor) and the environment, the agent's action strategy is optimized according to the rewards fed back by the environment, with the goal of maximizing the cumulative rewards. The end-to-end method based on deep reinforcement learning combines the decision-making ability of reinforcement learning with the perception ability of deep learning. It uses an end-to-end neural network to directly map the visual observation image to the camera's action signal, that is, the tracker's policy network, and continuously optimizes this policy network during the interaction between the tracker and the environment. The end-to-end method based on deep reinforcement learning has the advantages of high computational efficiency and good generalization performance, and has achieved results that surpass the two-stage method. Therefore, it has become an important research direction in the field of active visual tracking.

[0005] Luo et al. first proposed an active visual tracking method based on deep reinforcement learning in "End-to-end Active Object Tracking via Reinforcement Learning" (see Luo W, Sun P, Zhong F, et al. End-to-end active object tracking via reinforcement learning [C]. International conference on machine learning. 2018: 3286–3295.). This method uses a convolutional long short-term memory (Conv-LSTM) network to input video image frames and output camera movements. The entire network is trained by the reinforcement learning A3C (Asynchronous Advantage Actor-Critic, A3C) algorithm during the interaction between the tracker and the environment. During the training process, the target moves along a pre-set route. However, the movement of the target along a fixed route will make its trajectory too single, which will cause the tracker to overfit the specific target motion trajectory, resulting in tracking failure when encountering targets with other motion trajectories. To this end, the paper "AD-VAT: An asymmetric dueling mechanism for learning visual active tracking, Visual active tracking based on asymmetric adversarial mechanism" (see the paper Zhong F, Sun P, Luo W, et al. AD-VAT: An asymmetric dueling mechanism for learning visual active tracking [C]. International Conference on Learning Representations. 2018.) proposed an asymmetric adversarial mechanism based on adversarial reinforcement learning, which models both the tracker and the target as learnable agents, and forms a competitive relationship between the two: the tracker learns to track the target, while the target tries to get rid of the tracker. This adversarial mechanism is implemented by defining opposite reward functions for the tracker and the target. In order to enable the target to learn the optimal strategy to stay away from the tracker, the target is defined as being able to perceive the observations and actions of the tracker, and its network inputs not only its own visual observations, but also the visual observations and action signals of the tracker. During the training process, the tracker and the target optimize their strategies to each other, and the tracker can always compete with a target of appropriate difficulty.In order to solve the problem of visual confusion and occlusion of targets caused by distractors, a hybrid cooperative competitive multi-agent game framework is proposed in the paper "Towards distraction-robust active visual tracking" (see Zhong F, Sun P, Luo W, et al. Towards distraction-robust active visual tracking [C]. International Conference on Machine Learning. 2021: 12782-12792.), in which the tracker, target and distractors are modeled as learnable agents. This multi-agent game framework is implemented by defining the reward functions of the tracker, target and distractor respectively. Let the subscript i∈{1,2,3} represent the tracker, target and distractor respectively. The ideal position of the target is at a distance D directly in front of the tracker. The geometric relationship between the tracker and other agents (i.e., the target and distractor) is defined as the position difference d(i) between the other agents and the ideal position of the target. d(i) represents the position difference between the i-th agent and the ideal position of the target. The reward function of the tracker is defined as r. 1 =1-d(2), d(2) represents the position difference between the actual position of the target and the ideal position of the target. The reward function of the target is defined as r 2 =-r 1 The reward function for the distractor is defined as r 3 =r 2 -d(3), d(3) represents the position difference between the distractor and the target's ideal position. In the game, the target and the distractor cooperate with each other to form a team (called the opponent) to fight against the tracker. Specifically, the tracker follows the target and maintains a specific relative posture with it to minimize the difference between the target and the ideal position; the target escapes the tracker and moves as far away from the ideal position as possible; the distractor cooperates with the target to help the target escape the tracker by causing visual interference to the tracker or blocking the target. Since it is inefficient to directly use reinforcement learning to train multiple agents with visual input, a cross-modal teacher-student learning method is proposed in this paper, and the training process is divided into a teacher training stage and a student training stage. In the teacher training stage, the relative posture of the agent is used to train the meta-strategy to obtain the meta-tracker (teacher tracker), meta-target and meta-distractor. Since the relative posture is a low-dimensional real state information, it is easy to train a near-optimal strategy. In the student training stage, the visual tracker is supervised by sampling the strategies of the meta-target and meta-distractor and using the action labels provided by the teacher tracker to optimize its strategy network to obtain the student tracker.

[0006] Although existing end-to-end methods based on deep reinforcement learning have made significant progress in active visual tracking methods, they still face the challenging problem of target scale changes during the tracking process. In active visual tracking, since both the tracker and the target are constantly moving, the scale of the target in the tracker's visual observation may change significantly. For example, when the target suddenly moves backward, its scale in the tracker's camera field of view will increase significantly; conversely, when the target accelerates forward, its scale will decrease. This scale change is particularly prominent when there are interference objects in the environment (such as other people or vehicles in the tracking scene), because multiple objects of different scales may appear in the tracker's field of view at the same time, resulting in scale interference problems. At present, existing active visual tracking methods basically focus on handling conventional scale target tracking problems, without considering the scale change problem and its impact, and therefore no effective response solutions have been proposed.

[0007] Therefore, how to solve the scale change problem, especially when there are interferences in the environment, is a difficult problem that needs to be solved urgently in the field of active visual tracking. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide a scale-adaptive active visual tracking method for the problem of target scale change in active visual tracking, which can enhance the tracking ability of the active tracker when the target scale changes, improve the tracking performance of the active tracker (including three indicators: Accumulated Reward (AR), Episode Length (EL), and Success Rate (SR)), and especially improve the tracking performance when there are interferences in the environment.

[0009] The present invention comprises the following steps:

[0010] The first step is to construct a scale-adaptive active visual tracking system, which is composed of a data acquisition module, a data preprocessing module, an active visual tracking teacher module, and an active visual tracking student module.

[0011] The data acquisition module is connected to the camera of the tracker, the posture sensors of all agents (tracker, target, and interferer), and the data preprocessing module. It acquires two-dimensional visual observation images from the camera of the tracker and acquires the posture data of the tracker, target, and interferer from the posture sensors of the tracker, target, and interferer. During training, the data acquisition module sends the visual observation images acquired from the camera of the tracker and the posture data acquired from the posture sensors of the tracker, target, and interferer to the data preprocessing module; when actually tracking the target, the data acquisition module only sends the visual observation images acquired from the camera of the tracker to the data preprocessing module.

[0012] The data preprocessing module is connected to the data acquisition module, the active visual tracking teacher module, and the active visual tracking student module. It receives the visual observation image and posture data from the data acquisition module, preprocesses the visual observation image and posture data, and obtains the preprocessed visual observation image and preprocessed posture data. During training, the data preprocessing module sends the preprocessed posture data to the active visual tracking teacher module, and sends the preprocessed visual observation image to the active visual tracking student module. During actual tracking, the data preprocessing module only sends the preprocessed visual observation image to the active visual tracking student module.

[0013] The function of the active visual tracking teacher module is to assist the active visual tracking student module. It only works during training and does not participate in the actual tracking. The active visual tracking teacher module is connected to the data preprocessing module and the active visual tracking student module, and consists of a meta-tracker network, a meta-target network, a meta-disturbance network, a reward function calculation module, and a first loss function calculation module. The meta-tracker network consists of a first bidirectional gated recurrent unit, a first long short-term memory unit, a first fully connected layer, and a second fully connected layer; the first bidirectional gated recurrent unit encodes the preprocessed posture data received from the data preprocessing module to obtain the posture features of the tracker, and sends the posture features of the tracker to the first long short-term memory unit; the first long short-term memory unit extracts the temporal features of the posture features of the tracker received from the first bidirectional gated recurrent unit to obtain the temporal posture features of the tracker, and sends the temporal posture features of the tracker to the first fully connected layer and the second fully connected layer; the first fully connected layer performs action prediction on the temporal posture features of the tracker received from the first long short-term memory unit to obtain the action probability of the tracker. When training the active visual tracking teacher module, the first fully connected layer sends the action probability of the tracker to the meta-target network, the meta-interference network, and the first loss function calculation module. When training the active visual tracking student module, the first fully connected layer sends the action probability of the tracker to the meta-target network, the meta-interference network, and the active visual tracking student module (in Figure 1 The dotted lines in the figure indicate that the first fully connected layer sends the action probability of the tracker to the meta-target network, the meta-interference network, the first loss function calculation module, and the second loss function calculation module of the active visual tracking student module); the second fully connected layer estimates the value of the temporal posture features of the tracker received from the first long short-term memory unit to obtain the state value of the tracker. When training the active visual tracking teacher module, the second fully connected layer sends the state value of the tracker to the first loss function calculation module.

[0014] The meta-target network consists of a second bidirectional gated recurrent unit, a second long short-term memory unit, a third fully connected layer, and a fourth fully connected layer; the second bidirectional gated recurrent unit encodes the preprocessed posture data received from the data preprocessing module to obtain the posture features of the target, and sends the posture features of the target to the second long short-term memory unit; the second long short-term memory unit extracts the temporal features of the posture features of the target received from the second bidirectional gated recurrent unit and the action probability of the tracker received from the meta-tracker network to obtain the temporal posture features of the target, and sends the temporal posture features of the target to the third fully connected layer and the fourth fully connected layer; the third fully connected layer performs action prediction on the temporal posture features of the target received from the second long short-term memory unit to obtain the action probability of the target. When training the active visual tracking teacher module, the third fully connected layer sends the action probability of the target to the first loss function calculation module; the fourth fully connected layer estimates the value of the temporal posture features of the target received from the second long short-term memory unit to obtain the state value of the target. When training the active visual tracking teacher module, the fourth fully connected layer sends the state value of the target to the first loss function calculation module.

[0015] The meta-interference network consists of a third bidirectional gated recurrent unit, a third long short-term memory unit, a fifth fully connected layer, and a sixth fully connected layer; the third bidirectional gated recurrent unit encodes the preprocessed posture data received from the data preprocessing module to obtain the posture features of the interference, and sends the posture features of the interference to the third long short-term memory unit; the third long short-term memory unit extracts the temporal features of the posture features of the interference received from the third bidirectional gated recurrent unit and the action probability of the tracker received from the meta-tracker network to obtain the temporal posture features of the interference, and sends the temporal posture features of the interference to the third long short-term memory unit. to the fifth fully connected layer and the sixth fully connected layer; the fifth fully connected layer performs action prediction on the temporal position features of the distractor received from the third long short-term memory unit to obtain the action probability of the distractor. When training the active visual tracking teacher module, the fifth fully connected layer sends the action probability of the distractor to the first loss function calculation module; the sixth fully connected layer performs value estimation on the temporal position features of the distractor received from the third long short-term memory unit to obtain the state value of the distractor. When training the active visual tracking teacher module, the sixth fully connected layer sends the state value of the distractor to the first loss function calculation module.

[0016] The reward function calculation module is connected to the data preprocessing module and the first loss function calculation module. When training the active visual tracking teacher module, the reward function calculation module calculates the rewards of the tracker, the target, and the distractor according to the preprocessed posture data received from the data preprocessing module, and sends the rewards of the tracker, the target, and the distractor to the first loss function calculation module.

[0017] The first loss function calculation module is connected to the reward function calculation module, the meta-tracker network, the meta-target network, and the meta-interference network. When training the active visual tracking teacher module, the first loss function calculation module receives the rewards of the tracker, the target, and the interference from the reward function calculation module, receives the action probability of the tracker and the state value of the tracker from the meta-tracker network, receives the action probability of the target and the state value of the target from the meta-target network, receives the action probability of the interference and the state value of the interference from the meta-interference network, calculates the temporal difference error loss and policy gradient loss of the tracker according to the reward, action probability, and state value of the tracker, calculates the temporal difference error loss and policy gradient loss of the target according to the reward, action probability, and state value of the target, and calculates the temporal difference error loss and policy gradient loss of the interference according to the reward, action probability, and state value of the interference. When training the active visual tracking teacher module, the active visual tracking teacher module optimizes the meta-tracker network, the meta-target network, and the meta-interference network by minimizing the temporal difference error loss and policy gradient loss of the tracker, the target, and the interference obtained by the first loss function calculation module.

[0018] The active visual tracking student module is connected to the data preprocessing module and the active visual tracking teacher module, and is composed of a visual tracker network and a second loss function calculation module. The visual tracker network is composed of a convolutional backbone network, a multi-scale convolutional network branch, a scale attention module, an adaptive fusion module, a recurrent attention module, and the eighth fully connected layer.

[0019] The convolution backbone network is connected to the data preprocessing module, the multi-scale convolutional network branch and the scale attention module. It performs feature extraction on the preprocessed visual observation image received from the data preprocessing module to obtain visual observation image features, and sends the visual observation image features to the multi-scale convolutional network branch and the scale attention module.

[0020] The number of branches of the multi-scale convolutional network branch is 4, which consists of a first-scale convolutional network branch, a second-scale convolutional network branch, a third-scale convolutional network branch, and a fourth-scale convolutional network branch. The first-scale convolutional network branch extracts features from visual observation image features to obtain first-scale visual observation image features, and sends the first-scale visual observation image features to the adaptive fusion module; the second-scale convolutional network branch extracts features from visual observation image features to obtain second-scale visual observation image features, and sends the second-scale visual observation image features to the adaptive fusion module; the third-scale convolutional network branch extracts features from visual observation image features to obtain third-scale visual observation image features, and sends the third-scale visual observation image features to the adaptive fusion module; the fourth-scale convolutional network branch performs dimensionality reduction and feature extraction on the visual observation image features received from the convolutional backbone network to obtain fourth-scale visual observation image features, and sends the fourth-scale visual observation image features to the adaptive fusion module; the first-scale convolutional network branch It consists of a first ordinary convolution layer with a convolution kernel size of 1×1 and a first dilated convolution layer with a convolution kernel size of 3×3 and a dilation rate of 3, with a receptive field size of 7; the second-scale convolution network branch consists of a second ordinary convolution layer with a convolution kernel size of 1×1 and a second dilated convolution layer with a convolution kernel size of 3×3 and a dilation rate of 2, with a receptive field size of 5; the third-scale convolution network branch consists of a third ordinary convolution layer with a convolution kernel size of 1×1 and a third dilated convolution layer with a convolution kernel size of 3×3 and a dilation rate of 1, with a receptive field size of 3; the fourth-scale convolution network branch consists of a fourth ordinary convolution layer with a convolution kernel size of 1×1, with a receptive field size of 1; the first-scale visual observation image features, the second-scale visual observation image features, the third-scale visual observation image features, and the fourth-scale visual observation image features constitute multi-scale visual observation image features. The multi-scale convolution network branch can extract features of visual observation images in multiple scale ranges (corresponding to receptive fields of different sizes), so it can extract features of targets with varying scales.

[0021] The scale attention module consists of a convolutional long short-term memory unit, a global average pooling layer, a seventh fully connected layer, and a sigmoid activation function layer; the convolutional long short-term memory unit extracts temporal features from the visual observation image features received from the convolutional backbone network to obtain temporal features of the visual observation image, and sends the temporal features of the visual observation image to the global average pooling layer; the global average pooling layer compresses the temporal features of the visual observation image received from the convolutional long short-term memory unit in the spatial dimension to obtain a first feature vector whose length is equal to the number of channels of the temporal features of the visual observation image, and sends the first feature vector to the seventh fully connected layer; the seventh fully connected layer compresses the first feature vector received from the global average pooling layer to obtain a second feature vector whose length is equal to the number of branches of the multi-scale convolutional network (4), and sends the second feature vector to the sigmoid activation function layer; the sigmoid activation function layer scales the value of the second feature vector received from the seventh fully connected layer to the range of [0,1] to obtain the weights of the multi-scale visual observation image features, and sends the weights of the multi-scale visual observation image features to the adaptive fusion module. The scale attention module can adaptively calculate the weights of multi-scale visual observation image features according to the scale of the target in the visual observation image. The visual observation image features of the scale containing more target features are assigned larger weights, and the visual observation image features of the scale containing fewer target features are assigned smaller weights.

[0022] The adaptive fusion module is connected to the multi-scale convolutional network branch, the scale attention module, and the recurrent attention module, and is composed of a weighted splicing layer and a fifth ordinary convolutional layer with a convolution kernel size of 1×1; the weighted splicing layer uses the weights of the multi-scale visual observation image features received from the scale attention module to weightedly splice the first scale visual observation image features, the second scale visual observation image features, the third scale visual observation image features, and the fourth scale visual observation image features received from the multi-scale convolutional network branch to obtain the weighted spliced ​​multi-scale visual observation image features, and sends the weighted spliced ​​multi-scale visual observation image features to the fifth ordinary convolutional layer; the fifth ordinary convolutional layer reduces the channel dimension of the weighted spliced ​​multi-scale visual observation image features received from the weighted splicing layer to obtain the adaptively fused multi-scale visual observation image features, and sends the adaptively fused multi-scale visual observation image features to the recurrent attention module. The adaptive fusion module uses the weights of the multi-scale visual observation image features to weightedly fuse the multi-scale visual observation image features, thereby adaptively extracting features from targets with varying scales.

[0023] The recurrent attention module is connected to the adaptive fusion module and the eighth fully connected layer, performs temporal feature extraction and weighting on the adaptively fused multi-scale visual observation image features received from the adaptive fusion module, obtains adaptive temporal scale features, and sends the adaptive temporal scale features to the eighth fully connected layer.

[0024] The eighth fully connected layer performs action prediction on the adaptive temporal scale features received from the recurrent attention module to obtain the action probability of the tracker. When training the active visual tracking student module, the eighth fully connected layer sends the action probability of the tracker to the second loss function calculation module.

[0025] The second loss function calculation module is connected to the eighth fully connected layer and the meta tracker network. When training the active visual tracking student module, the second loss function calculation module calculates the KL divergence loss for the action probability of the tracker received from the eighth fully connected layer and the action probability of the tracker received from the meta tracker network of the active visual tracking teacher module. When training the active visual tracking student module, the active visual tracking student module optimizes the visual tracker network by minimizing the KL divergence loss obtained by the second loss function calculation module.

[0026] The second step is to train the active visual tracking teacher module and the active visual tracking student module in the scale-adaptive active visual tracking system. The training method of the active visual tracking teacher module is a reinforcement learning method, and the training method of the active visual tracking student module is a supervised learning method.

[0027] 2.1 The active visual tracking teacher module is trained by using reinforcement learning. The method is to use adversarial reinforcement learning to perform multi-scale and multi-stage training on the meta-tracker network, meta-target network and meta-distractor network during the interaction between the tracker, target, distractor and environment. The reward functions of the target and distractor in each stage are different, corresponding to different scales of training, specifically:

[0028] 2.1.1 The first phase of regular scale training of the active tracking teacher module is carried out by using reinforcement learning. The method is:

[0029] 2.1.1.1 Set the number of training steps T to a positive integer greater than 1000000, preferably T = 2000000, and initialize the number of trained steps t = 1.

[0030] 2.1.1.2 Set the target position entropy reward coefficient a to a constant greater than 0.5, preferably a=0.8, set the regularization factor α to a constant less than 1, preferably α=0.005; set the reward attenuation factor γ to a constant greater than 0.5, preferably γ=0.9; set the environment horizontal grid division number n to a positive integer, preferably n=10.

[0031] 2.1.1.3 Set the number of steps used to calculate the target position entropy reward N = min(t, 50);

[0032] 2.1.1.4 The data acquisition module obtains the pose data of the tth step from the pose sensors of the tracker, target, and interference, and sends the pose data of the tth step to the data preprocessing module. The pose data of the tth step includes the position coordinates of the tracker and orientation angle The target's location coordinates and orientation angle The coordinates of the distractor's location and orientation angle

[0033] 2.1.1.5 The data preprocessing module uses the first preprocessing method to preprocess the posture data of the tth step received from the data acquisition module to obtain the preprocessed posture data of the tth step, and sends the preprocessed posture data of the tth step to the active visual tracking teacher module. The first preprocessing method is:

[0034] 2.1.1.5.1 Calculate the relative distance between the target's actual position and the tracker Relative Angle Relative orientation angle The relative distance of the actual position of the interference object relative to the tracker Relative Angle Relative orientation angle

[0035] 2.1.1.5.2 Set the relative distance ρ of the target ideal position relative to the tracker * =250cm, relative angle θ * =0, relative orientation angle ω * =0, indicating that the ideal position of the target is 250cm in front of the tracker and facing the same direction as the tracker; Set the maximum distance of the tracker's field of view ρ max =750cm and maximum angle θ max =90°.

[0036] 2.1.1.5.3 Calculate the normalized relative distance, normalized relative angle, and normalized relative orientation angle of the actual position of the target and the actual position of the interference object:

[0037] 2.1.1.5.3.1 Calculate the normalized relative distance of the target's actual position Normalized relative distance to the actual position of the distractor

[0038] 2.1.1.5.3.2 Calculate the relative angle of the target's actual position The sine value of and cosine Get the relative angle of the normalized representation of the actual position of the target;

[0039] 2.1.1.5.3.3 Calculate the relative orientation angle of the target's actual position The sine value of and cosine value cos Obtain the relative orientation angle of the normalized representation of the actual position of the target;

[0040] 2.1.1.5.3.4 Calculate the relative angle of the actual position of the interference object The sine value of and cosine value cos Get the relative angle of the normalized representation of the actual position of the interference object;

[0041] 2.1.1.5.3.5 Calculate the relative orientation angle of the actual position of the interference object The sine value of and cosine value cos The relative orientation angle of the normalized representation of the actual position of the interference object is obtained.

[0042] 2.1.1.5.4 Set the tracker's position coordinates The target's location coordinates The coordinates of the distractor's location The relative distance to the actual position of the target Relative Angle Relative orientation angle The relative distance of the actual location of the distractor Relative Angle Relative orientation angle Normalized relative distance to the target's actual position The relative angle sin of the normalized representation of the actual position of the target cos The relative orientation angle sin of the normalized representation of the actual position of the target cos Normalized relative distance to the actual position of the distractor The relative angle sin of the normalized representation of the actual position of the distractor cos The relative orientation angle sin of the normalized representation of the actual position of the interference object cos The preprocessed pose data that constitutes the tth step.

[0043] 2.1.1.6 The reward function calculation module in the active visual tracking teacher module calculates the tracker’s reward based on the preprocessed pose data of step t

[0044]

[0045] 2.1.1.7 The reward function calculation module in the active visual tracking teacher module divides the training environment into I grids, I = n × n, and uses the frequency calculation method to calculate the set of target position coordinates in the past N steps based on the preprocessed pose data in the past t steps (the set of target position coordinates is ) appears in each grid with a frequency p1,…,p i ,…,p I , the method is:

[0046] 2.1.1.7.1 Initialize variable i = 1;

[0047] 2.1.1.7.2 Calculate the frequency p of the set of target position coordinates in the past N steps appearing in the i-th grid i :

[0048]

[0049] Where N i is the number of positions in the i-th grid in the set of target position coordinates in the past N steps, N i <N。

[0050] 2.1.1.7.3 Let i=i+1, if i≤I, go to 3.1.1.7.2; if i>I, get p1,…,p i ,…,p I , go to 2.1.1.8.

[0051] 2.1.1.8 The reward function calculation module in the active vision teacher module calculates the position entropy reward loc_entropy of the target:

[0052]

[0053] 2.1.1.9 The reward function calculation module in the active visual tracking teacher module calculates the reward of the target based on the preprocessed pose data of step t

[0054]

[0055] 2.1.1.10 The reward function calculation module in the active visual tracking teacher module calculates the reward of the distractor based on the preprocessed pose data of step t

[0056]

[0057] 2.1.1.11 The reward function calculation module in the active visual tracking teacher module converts the tracker's reward Rewards for goals Distractor Rewards Sent to the first loss function calculation module in the active visual tracking teacher module.

[0058] 2.1.1.12 The meta-tracker network, meta-target network, and meta-disturbance network in the active visual tracking teacher module use estimation methods to perform action prediction and value estimation on the preprocessed pose data, respectively, to obtain the state value V of the tracker. 1 (s t ) and the tracker’s action probability π 1 (a t |s t ), the state value V of the target 2 (s t ) and the target action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ) and the action probability π of the distractor 3 (a t |s t ), the method is:

[0059] 2.1.1.12.1 Let the normalized relative distance of the actual position of the target in the preprocessed pose data of step t be Normalized relative angle sin cos Normalized relative orientation angle sin cos Normalized relative distance to the actual position of the distractor Normalized relative angle sin cos Normalized relative orientation angle sin cos The environmental state that constitutes step t

[0060] 2.1.1.12.2 The First Bidirectional Gated Recurrent Unit Pair in the Meta-Tracker Network tEncode to obtain the posture features of the tracker, and send the posture features of the tracker to the first long short-term memory unit; the first long short-term memory unit extracts the temporal features of the posture features of the tracker received from the first bidirectional gated recurrent unit to obtain the temporal posture features of the tracker, and sends the temporal posture features of the tracker to the first fully connected layer and the second fully connected layer; the first fully connected layer predicts the action of the temporal posture features of the tracker received from the first long short-term memory unit to obtain the action probability π of the tracker 1 (a t |s t ), the first fully connected layer sends the action probability of the tracker to the meta-target network and the meta-interference network; the second fully connected layer estimates the value of the temporal posture features of the tracker received from the first long short-term memory unit to obtain the state value V of the tracker 1 (s t ).

[0061] 2.1.1.12.3 Second Bidirectional Gated Recurrent Unit Pair s in Meta-Target Network t Encode to obtain the target's posture features, and send the target's posture features to the second long short-term memory unit; the second long short-term memory unit encodes the target's posture features received from the second bidirectional gated recurrent unit and the tracker's action probability π received from the meta-tracker network 1 (a t |s t ) extracts temporal features to obtain the temporal position and posture features of the target, and sends the temporal position and posture features of the target to the third fully connected layer and the fourth fully connected layer; the third fully connected layer performs action prediction on the temporal position and posture features of the target received from the second long short-term memory unit to obtain the action probability π of the target 2 (a t |s t ); The fourth fully connected layer estimates the value of the temporal position features of the target received from the second long short-term memory unit to obtain the state value V of the target 2 (s t ).

[0062] 2.1.1.12.4 The third bidirectional gated recurrent unit pair s in the perturbation network t Encode to obtain the pose feature of the interferer, and send the pose feature of the interferer to the third long short-term memory unit; the third long short-term memory unit encodes the pose feature of the interferer received from the third bidirectional gated recurrent unit and the action probability π of the tracker received from the meta-tracker network 1 (a t |s t) to extract temporal features, obtain temporal position features of the interferer, and send the temporal position features of the interferer to the fifth fully connected layer and the sixth fully connected layer; the fifth fully connected layer performs action prediction on the temporal position features of the interferer received from the third long short-term memory unit to obtain the action probability π of the interferer 3 (a t |s t ); The sixth fully connected layer estimates the value of the temporal position features of the interference object received from the third long short-term memory unit to obtain the state value V of the interference object 3 (s t ). t is a random variable representing the action of the agent, and its value range is the action range of the agent (the action range of the tracker, target, and interferer are all 7 actions: forward, backward, left, right, forward and left, forward and right, stop), π k (a t |s t ) is a conditional probability, indicating that in state s t Probability distribution of the 7 actions in the action range of the agent, k∈{1,2,3}, 1 represents the tracker, 2 represents the target, and 3 represents the distractor.

[0063] 2.1.1.13 The meta-tracker network in the active visual tracking teacher module converts the state value V of the tracker 1 (s t ) and action probability π 1 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the tracker is 1 (a t |s t ) is sent to the tracker. The meta-target network converts the target state value V 2 (s t ) and action probability π 2 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the target action probability π 2 (a t |s t ) is sent to the target. The meta-interference network sends the state value V of the distractor 3 (s t ) and action probability π 3 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the interference object is 3 (at |s t ) is sent to the interferer.

[0064] 2.1.1.14 If t = 1, go to 2.1.1.16; otherwise: the first loss function calculation module in the active visual tracking teacher module calculates the reward of the tracker received from the reward function calculation module Rewards for goals Distractor Rewards and the state value V of the tracker received from the meta-tracker network, meta-target network, and meta-distractor network 1 (s t ), the action probability π of the tracker 1 (a t |s t ), the state value V of the target 2 (s t ), the target action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ), the action probability of the distractor π 3 (a t |s t ) Calculate the time difference error loss of the tracker Policy gradient loss for tracker Target temporal difference error loss Policy gradient loss of the target Temporal Differential Error Loss of Interferers Policy Gradient Loss for Distractors

[0065]

[0066] where logπ k (a t |s t ) is π k (a t |s t ), H(π k (·|s t )) is the entropy regularization term,

[0067] 2.1.1.15 The active visual tracking teacher module uses the Adam optimization algorithm (see the document "Kingma DP, BaJ.Adam:A method for stochastic optimization[J].arXiv preprint arXiv:1412.6980,2014", Kingma's paper: Adam, a stochastic optimization method) to minimize and To update the parameters of the meta-tracker network, use the Adam optimization algorithm to minimize and To update the parameters of the meta-target network, the Adam optimization algorithm is used to minimize and to update the parameters of the meta-distractor network.

[0068] 2.1.1.16 Tracker According to the tracker's action probability π 1 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the target is based on the target’s action probability π 2 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the interference object is based on the action probability π of the interference object. 3 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking the action to interact with the environment.

[0069] 2.1.1.17 Let t = t + 1. If t ≤ T, go to 2.1.1.18; if t > T, go to 2.1.1.19.

[0070] 2.1.1.18 If t%10000=0: save the parameters of the meta-target network and the meta-interference network to the opponent model pool, go to 2.1.1.3, and continue training using the data of the next time step; otherwise: directly go to 2.1.1.3 and continue training using the data of the next time step.

[0071] 2.1.1.19 Save the parameters of the meta-tracker network, meta-target network, and meta-distractor network, and go to 2.1.2.

[0072] 2.1.2 After the first stage of regular scale training, the meta-tracker network has the ability to track regular scale targets, but still lacks the ability to track small and large scale targets. Therefore, based on the first stage of training, the active visual tracking teacher module is further trained on a second stage of small scale using reinforcement learning. The method is:

[0073] 2.1.2.1 Meta-tracker network, meta-target network, meta-distractor network Load the parameters of the meta-tracker network, meta-target network, and meta-distractor network saved at the end of the first stage of training in 2.1.1.19.

[0074] 2.1.2.2 Set the number of training steps T to a positive integer greater than 100,000, preferably T = 1,000,000, and initialize the number of trained steps t = 1;

[0075] 2.1.2.3 Set the target position entropy reward coefficient a to a constant greater than 0.5, preferably a = 0.8; set the regularization factor α to a constant less than 1, preferably α = 0.005; set the reward attenuation factor γ to a constant greater than 0.5, preferably γ = 0.9; set the number of environmental horizontal grid divisions n to a positive integer, preferably n = 10; set the threshold distance ρ of the target away from the tracker s 100~1000cm, preferably ρ s =500cm.

[0076] 2.1.2.4 Set the number of steps used to calculate the target position entropy reward to N = min(t, 50).

[0077] 2.1.2.5 The data acquisition module obtains the posture data of the tth step from the posture sensors of the tracker, target, and interference object, and sends the posture data of the tth step to the data preprocessing module.

[0078] 2.1.2.6 The data preprocessing module adopts the first preprocessing method in 2.1.1.5 to preprocess the posture data of the tth step received from the data acquisition module to obtain the preprocessed posture data of the tth step, and sends the preprocessed posture data of the tth step to the active visual tracking teacher module.

[0079] 2.1.2.7 The reward function calculation module in the active visual tracking teacher module calculates the reward of the tracker according to the formula (7) in 2.1.1.6 based on the preprocessed pose data of step t

[0080] 2.1.2.8 The reward function calculation module in the active visual tracking teacher module divides the training environment into I grids, and uses the frequency calculation method described in 2.1.1.7 to calculate the frequency p1,…,p of the set of target position coordinates in the past N steps in each grid based on the preprocessed pose data of the past t steps. i ,…,p I .

[0081] 2.1.2.9 The reward function calculation module in the active vision teacher module calculates the target's position entropy reward loc_entropy according to formula (9) in 2.1.1.8.

[0082] 2.1.2.10 The reward function calculation module in the active visual tracking teacher module calculates the reward of the target based on the preprocessed pose data of step t

[0083]

[0084] 2.1.2.11 The reward function calculation module in the active visual tracking teacher module calculates the reward of the distractor based on the preprocessed pose data of step t

[0085]

[0086] 2.1.2.12 The reward function calculation module in the active visual tracking teacher module converts the tracker's reward Rewards for goals Distractor Rewards Sent to the first loss function calculation module in the active visual tracking teacher module.

[0087] 2.1.2.13 The meta-tracker network, meta-target network, and meta-interference network in the active visual tracking teacher module use the estimation method described in 2.1.1.12 to perform action prediction and value estimation on the preprocessed pose data, respectively, to obtain the state value V of the tracker 1 (s t ) and the tracker’s action probability π 1 (a t |s t ), the state value V of the target 2 (s t ) and the target action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ) and the action probability π of the distractor 3 (a t |s t).

[0088] 2.1.2.14 The meta-tracker network in the active visual tracking teacher module converts the state value V of the tracker 1 (s t ) and action probability π 1 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the tracker is 1 (a t |s t ) is sent to the tracker. The meta-target network converts the target state value V 2 (s t ) and action probability π 2 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the target action probability π 2 (a t |s t ) is sent to the target. The meta-interference network sends the state value V of the distractor 3 (s t ) and action probability π 3 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the interference object is 3 (a t |s t ) is sent to the interferer.

[0089] 2.1.2.15 If t = 1, go to 2.1.2.17; otherwise, the first loss function calculation module in the active visual tracking teacher module calculates the reward of the tracker received from the reward function calculation module according to formula (12) and formula (13) in 2.1.1.14. Rewards for goals Distractor Rewards and the state value V of the tracker received from the meta-tracker network, meta-target network, and meta-distractor network 1 (s t ), the action probability π of the tracker 1 (a t |s t ), the state value V of the target 2 (s t ), the target action probability π 2 (a t |s t ), the state value V of the interference 3 (st ), the action probability of the distractor π 3 (a t |s t ) Calculate the time difference error loss of the tracker Policy gradient loss for tracker Target temporal difference error loss Policy gradient loss of the target Temporal Differential Error Loss of Interferers Policy Gradient Loss for Distractors

[0090] 2.1.2.16 Active visual tracking teacher module uses Adam optimization algorithm to minimize and To update the parameters of the meta-tracker network, use the Adam optimization algorithm to minimize and To update the parameters of the meta-target network, the Adam optimization algorithm is used to minimize and to update the parameters of the meta-distractor network.

[0091] 2.1.2.17 Tracker According to the tracker's action probability π 1 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the target is based on the target’s action probability π 2 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the interference object is based on the action probability π of the interference object. 3 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking the action to interact with the environment.

[0092] 2.1.2.18 Let t = t + 1. If t ≤ T, go to 2.1.2.19; if t > T, go to 2.1.2.20.

[0093] 2.1.2.19 If t%10000=0: save the parameters of the meta-target network and the meta-interference network to the opponent model pool, and go to 2.1.2.4 to continue training using the data of the next time step; otherwise: directly go to 2.1.2.4 to continue training using the data of the next time step.

[0094] 2.1.2.20 Save the parameters of the meta-tracker network, meta-target network, and meta-distractor network, and go to 2.1.3.

[0095] 2.1.3 After the first stage of regular scale training and the second stage of small scale training, the meta-tracker network has the ability to track regular scale and small scale targets, but still lacks the ability to track large scale targets. Therefore, based on the second stage training, reinforcement learning is used to further perform the third stage large scale training on the active visual tracking teacher module.

[0096] 2.1.3.1 Meta-tracker network, meta-target network, meta-distractor network Load the parameters of the meta-tracker network, meta-target network, and meta-distractor network saved at the end of the second stage of training in 2.1.2.20.

[0097] 2.1.3.2 Set the number of training steps T to a positive integer greater than 100,000, preferably T = 1,000,000, and initialize the number of trained steps t = 1;

[0098] 2.1.3.3 Set the target position entropy reward coefficient a to a constant greater than 0.5, preferably a = 0.8; set the regularization factor α to a constant less than 1, preferably α = 0.005; set the reward attenuation factor γ to a constant greater than 0.5, preferably γ = 0.9; set the number of environmental horizontal grid divisions n to a positive integer, preferably n = 10; set the threshold distance ρl of the target close to the tracker to 100 ~ 1000cm, preferably ρ l =150cm.

[0099] 2.1.3.4 Set the number of steps used to calculate the target position entropy reward to N = min(t, 50).

[0100] 2.1.3.5 The data acquisition module obtains the posture data of the tth step from the posture sensors of the tracker, target, and interference object, and sends the posture data of the tth step to the data preprocessing module.

[0101] 2.1.3.6 The data preprocessing module adopts the first preprocessing method in 2.1.1.5 to preprocess the posture data of the tth step received from the data acquisition module to obtain the preprocessed posture data of the tth step, and sends the preprocessed posture data of the tth step to the active visual tracking teacher module.

[0102] 2.1.3.7 The reward function calculation module in the active visual tracking teacher module calculates the reward of the tracker according to the formula (7) in 2.1.1.6 based on the preprocessed pose data of the tth step

[0103] 2.1.3.8 The reward function calculation module in the active visual tracking teacher module divides the training environment into I grids, and uses the frequency calculation method described in 2.1.1.7 to calculate the frequency p1,…,p of the set of target position coordinates in the past N steps in each grid based on the preprocessed pose data of the past t steps. i ,…,p I .

[0104] 2.1.3.9 The reward function calculation module in the active vision teacher module calculates the target's position entropy reward loc_entropy according to formula (9) in 2.1.1.8.

[0105] 2.1.3.10 The reward function calculation module in the active visual tracking teacher module calculates the reward of the target based on the preprocessed pose data of step t

[0106]

[0107] 2.1.3.11 The reward function calculation module in the active visual tracking teacher module calculates the reward of the distractor based on the preprocessed pose data of step t

[0108]

[0109] 2.1.3.12 The reward function calculation module in the active visual tracking teacher module converts the tracker's reward Rewards for goals Distractor Rewards Sent to the first loss function calculation module in the active visual tracking teacher module.

[0110] 2.1.3.13 The meta-tracker network, meta-target network, and meta-interference network in the active visual tracking teacher module use the estimation method described in step 2.1.1.12 to perform action prediction and value estimation on the preprocessed pose data, respectively, to obtain the state value V of the tracker 1 (s t ) and the tracker’s action probability π 1 (a t |s t ), the state value V of the target 2 (s t ) and the target action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ) and the action probability π of the distractor 3 (a t |s t).

[0111] 2.1.3.14 The meta-tracker network in the active visual tracking teacher module converts the state value V of the tracker 1 (s t ) and action probability π 1 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the tracker is 1 (a t |s t ) is sent to the tracker. The meta-target network converts the target state value V 2 (s t ) and action probability π 2 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the target action probability π 2 (a t |s t ) is sent to the target. The meta-interference network sends the state value V of the distractor 3 (s t ) and action probability π 3 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the interference object is 3 (a t |s t ) is sent to the interferer.

[0112] 2.1.3.15 If t = 1, go to 2.1.3.17; otherwise, the first loss function calculation module in the active visual tracking teacher module calculates the reward of the tracker received from the reward function calculation module according to formula (12) and formula (13) in 2.1.1.14. Rewards for goals Distractor Rewards and the state value V of the tracker received from the meta-tracker network, meta-target network, and meta-distractor network 1 (s t ), the action probability π of the tracker 1 (a t |s t ), the state value V of the target 2 (s t ), the target action probability π 2 (a t |s t ), the state value V of the interference 3 (st ), the action probability of the distractor π 3 (a t |s t ) Calculate the time difference error loss of the tracker Policy gradient loss for tracker Target temporal difference error loss Policy gradient loss of the target Temporal Differential Error Loss of Interferers Policy Gradient Loss for Distractors

[0113] 2.1.3.16 Active visual tracking teacher module uses Adam optimization algorithm to minimize and To update the parameters of the meta-tracker network, use the Adam optimization algorithm to minimize and To update the parameters of the meta-target network, the Adam optimization algorithm is used to minimize and to update the parameters of the meta-distractor network.

[0114] 2.1.3.17 Tracker According to the tracker's action probability π 1 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the target is based on the target’s action probability π 2 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the interference object is based on the action probability π of the interference object. 3 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking the action to interact with the environment.

[0115] 2.1.3.18 Let t = t + 1. If t ≤ T, go to 2.1.3.19. If t > T, go to 2.1.4.

[0116] 2.1.3.19 If t%10000=0, save the parameters of the meta-target network and the meta-interference network to the opponent model pool, and go to 2.1.3.4 to continue training using the data of the next time step; otherwise: directly go to 2.1.3.4 to continue training using the data of the next time step.

[0117] 2.1.4 Active Visual Tracking Teacher Module Save the parameters of the meta-tracker network, go to 2.2.

[0118] 2.2 During the interaction between the tracker and the target, the interference, and the environment, the meta-tracker network is further fine-tuned by using reinforcement learning. Only the parameters of the meta-tracker network are optimized. The method is as follows:

[0119] 2.2.1 Meta-tracker network in the active visual tracking teacher module Load the parameters of the meta-tracker network saved in step 2.1.4.

[0120] 2.2.2 Set the number of training steps T = 1000000 and initialize the number of trained steps t = 1.

[0121] 2.2.3 Randomly select the parameters of the meta-target network from the opponent model pool and load them into the meta-target network; randomly select the parameters of the meta-interferer network from the opponent model pool and load them into the meta-interferer network.

[0122] 2.2.4 Fix the parameters of the meta-target network and meta-disturbance network (only optimize the parameters of the meta-tracker network during the interaction between the tracker and the target, distractors, and environment).

[0123] 2.2.5 The data acquisition module obtains the posture data of the tth step from the posture sensors of the tracker, target, and interference object, and sends the posture data of the tth step to the data preprocessing module.

[0124] 2.2.6 The data preprocessing module adopts the first preprocessing method in 2.1.1.5 to preprocess the posture data of the tth step received from the data acquisition module to obtain the preprocessed posture data of the tth step, and sends the preprocessed posture data of the tth step to the active visual tracking teacher module.

[0125] 2.2.7 The reward function calculation module in the active visual tracking teacher module calculates the reward of the tracker according to the formula (7) in 2.1.1.6 based on the preprocessed pose data of the tth step

[0126] 2.2.8 The reward function calculation module in the active visual tracking teacher module converts the tracker's reward Sent to the first loss function calculation module in the active visual tracking teacher module.

[0127] 2.2.9 The meta-tracker network, meta-target network, and meta-interference network in the active visual tracking teacher module use the estimation method described in step 2.1.1.12 to perform action prediction and value estimation on the preprocessed pose data, respectively, to obtain the state value V of the tracker. 1 (s t ) and the tracker’s action probability π 1 (a t |s t ), the state value V of the target 2(s t ) and the target action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ) and the action probability π of the distractor 3 (a t |s t ).

[0128] 2.2.10 The meta-tracker network in the active visual tracking teacher module converts the state value V of the tracker 1 (s t ) and action probability π 1 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the tracker is 1 (a t |s t ) is sent to the tracker. The meta-target network converts the target action probability π 2 (a t |s t ) is sent to the target. The meta-interference network sends the action probability π of the distractor 3 (a t |s t ) is sent to the interferer.

[0129] 2.2.11 If t = 1, go to 2.2.13; otherwise, the first loss function calculation module in the active visual tracking teacher module calculates the reward of the tracker received from the reward function calculation module according to formula (12) and formula (13) in step 2.1.1.14. and the state value V of the tracker received from the meta-tracker network 1 (s t ), the action probability π of the tracker 1 (a t |s t ) Calculate the time difference error loss of the tracker Policy gradient loss for tracker

[0130] 2.2.12 Active visual tracking teacher module uses Adam optimization algorithm to minimize and to update the parameters of the meta-tracker network.

[0131] 2.2.13 Tracker According to the tracker's action probability π 1 (a t |s t) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the target is based on the target’s action probability π 2 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the interference object is based on the action probability π of the interference object. 3 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking the action to interact with the environment.

[0132] 2.2.14 Let t = t + 1. If t ≤ T, go to 2.2.15; if t > T, go to 2.2.16.

[0133] 2.2.15 If t%5000=0, go to 2.2.3 to train the active visual tracking teacher module loaded with the parameters of the new meta-target network and meta-distractor network; otherwise, go to 2.2.5 to continue training using the data of the next time step.

[0134] 2.2.16 The active visual tracking teacher module saves the parameters of the meta-tracking network, go to 2.3.

[0135] 2.3 The active visual tracking student module is trained using supervised learning. The method is as follows:

[0136] 2.3.1 Meta-tracker network in the active visual tracking teacher module Load the parameters of the meta-tracker network saved in step 2.2.16.

[0137] 2.3.2 Fixed parameters of the meta-tracker network.

[0138] 2.3.3 Set the number of training steps T to a positive integer greater than 100,000, preferably T = 2,000,000, and initialize the number of trained steps t = 1.

[0139] 2.3.4 Randomly select the parameters of the meta-target network from the opponent model pool and load them into the meta-target network; randomly select the parameters of the meta-interferer network from the opponent model pool and load them into the meta-interferer network.

[0140] 2.3.5 Fix the parameters of the meta-target network and meta-distractor network.

[0141] 2.3.6 Data acquisition module obtains the visual observation image o of step t from the camera of the tracker t , obtain the pose data of the tth step from the pose sensors of the tracker, target, and interference object, and send the visual observation image and pose data of the tth step to the data preprocessing module.

[0142] 2.3.7 The data preprocessing module adopts the first preprocessing method in step 2.1.1.5 to preprocess the posture data of the tth step received from the data acquisition module, obtains the preprocessed posture data of the tth step, and sends the preprocessed posture data of the tth step to the active visual tracking teacher module.

[0143] 2.3.8 The data preprocessing module uses the second preprocessing method to process the visual observation image o of the tth step received from the data acquisition module. t Perform preprocessing to obtain the preprocessed visual observation image o′ of the tth step t , the preprocessed visual observation image o′ of step t t Sent to the active visual tracking student module. The second preprocessing method is:

[0144] 2.3.8.1 o t The width is scaled to 320 pixels and the height is scaled to 240 pixels to obtain the scaled visual observation image.

[0145] 2.3.8.2 Normalize the pixel values ​​of the scaled visual observation image to the interval [0,1] to obtain the normalized visual observation image of the tth step.

[0146] 2.3.8.3 Randomly flip the normalized visual observation image of the t-th step horizontally to obtain the preprocessed visual observation image o′t′ of the t-th step.

[0147] 2.3.9 The meta-tracker network, meta-target network, and meta-interference network in the active visual tracking teacher module use the estimation method described in 2.1.1.12 to perform action prediction and value estimation on the preprocessed pose data, respectively, to obtain the state value V of the tracker. 1 (s t ) and action probability π 1 (a t |s t ), the state value V of the target 2 (s t ) and action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ) and action probability π 3 (a t |s t ). The meta-tracker network transforms the tracker’s action probability π 1 (a t |s t ) is sent to the second loss function calculation module in the active visual tracking student module, and the meta-target network converts the target action probability π 2 (at |s t ) is sent to the target, and the meta-interference network sends the action probability π of the distractor 3 (a t |s t ) is sent to the interferer.

[0148] 2.3.10 Active Visual Tracking The visual tracker network in the student module uses an action prediction method to preprocess the visual observation image o′ t Perform action prediction and obtain the action probability π of the tracker 1* (a t |o′ t ), the method is:

[0149] 2.3.10.1 Convolutional backbone in the visual tracker network for the preprocessed visual observation image o′ t Perform feature extraction to obtain visual observation image features, and send the visual observation image features to the multi-scale convolutional network branch and the scale attention module.

[0150] 2.3.10.2 The first scale convolutional network branch in the multi-scale convolutional network branch extracts features from visual observation image features to obtain first scale visual observation image features, wherein the first common convolutional layer reduces the dimension of the visual observation image features received from the convolutional backbone network in the channel dimension to obtain the first visual observation image features after dimension reduction, and sends the first visual observation image features after dimension reduction to the first dilated convolutional layer; the first dilated convolutional layer extracts features from the first visual observation image features after dimension reduction received from the first common convolutional layer to obtain first scale visual observation image features, and sends the first scale visual observation image features to the adaptive fusion module; the second scale convolutional network branch in the multi-scale convolutional network branch extracts features from visual observation image features to obtain second scale visual observation image features, wherein the second common convolutional layer reduces the dimension of the visual observation image features received from the convolutional backbone network in the channel dimension to obtain the second visual observation image features after dimension reduction, and sends the second visual observation image features after dimension reduction to the second dilated convolutional layer; the second dilated convolutional layer extracts features from the first common convolutional layer after dimension reduction received from the second common convolutional layer The third-scale convolutional network branch in the multi-scale convolutional network branch extracts the visual observation image features to obtain the third-scale visual observation image features, wherein the third ordinary convolutional layer reduces the dimension of the visual observation image features received from the convolutional backbone network in the channel dimension to obtain the third visual observation image features after the dimension reduction, and sends the third visual observation image features after the dimension reduction to the third expanded convolutional layer; the third expanded convolutional layer extracts the features of the third visual observation image after the dimension reduction received from the third ordinary convolutional layer to obtain the third-scale visual observation image features, and sends the third-scale visual observation image features to the adaptive fusion module; the fourth ordinary convolutional layer in the fourth-scale convolutional network branch in the multi-scale convolutional network branch reduces the dimension and extracts the features of the visual observation image received from the convolutional backbone network in the channel dimension to obtain the fourth-scale visual observation image features, and sends the fourth-scale visual observation image features to the adaptive fusion module;

[0151] 2.3.10.3 The convolutional long short-term memory unit in the scale attention module extracts temporal features from the visual observation image features received from the convolutional backbone network to obtain temporal features of the visual observation image, and sends the temporal features of the visual observation image to the global average pooling layer; the global average pooling layer compresses the temporal features of the visual observation image received from the convolutional long short-term memory unit in the spatial dimension to obtain a first feature vector whose length is equal to the number of channels of the temporal features of the visual observation image, and sends the first feature vector to the seventh fully connected layer; the seventh fully connected layer compresses the first feature vector received from the global average pooling layer to obtain a second feature vector whose length is equal to the number of branches of the multi-scale convolutional network (4), and sends the second feature vector to the sigmoid activation function layer; the sigmoid activation function layer scales the value of the second feature vector received from the seventh fully connected layer to the range of [0,1] to obtain the weights of the multi-scale visual observation image features, and sends the weights of the multi-scale visual observation image features to the adaptive fusion module.

[0152] 2.3.10.4 The weighted splicing layer in the adaptive fusion module uses the weights of the multi-scale visual observation image features received from the scale attention module to weightedly splice the first-scale visual observation image features, second-scale visual observation image features, third-scale visual observation image features, and fourth-scale visual observation image features received from the multi-scale convolutional network branch to obtain weighted spliced ​​multi-scale visual observation image features, and send the weighted spliced ​​multi-scale visual observation image features to the fifth ordinary convolutional layer; the fifth ordinary convolutional layer performs dimensionality reduction on the weighted spliced ​​multi-scale visual observation image features received from the weighted splicing layer in the channel dimension to obtain adaptively fused multi-scale visual observation image features, and send the adaptively fused multi-scale visual observation image features to the recurrent attention module.

[0153] 2.3.10.5 The recurrent attention module extracts and weights the temporal features of the adaptively fused multi-scale visual observation image features received from the adaptive fusion module to obtain adaptive temporal scale features, and sends the adaptive temporal scale features to the eighth fully connected layer.

[0154] 2.3.10.6 The eighth fully connected layer performs action prediction on the adaptive temporal scale features received from the recurrent attention module to obtain the tracker’s action probability π 1* (a t |o′ t ).

[0155] 2.3.11 Active Visual Tracking The visual tracker network in the student module (here refers to the eighth fully connected layer in the visual tracker network) converts the tracker's action probability π 1* (a t |o′ t) is sent to the second loss function calculation module and tracker.

[0156] 2.3.12 The second loss function calculation module in the active visual tracking student module is based on the tracker action probability π received from the visual tracker network 1* (a t |o′ t ) and the tracker’s action probability π received from the meta-tracker network in the active visual tracking teacher module 1 (a t |s t ) Calculate the KL divergence loss D KL (π 1* (a t |o′ t )|π 1 (a t |s t )):

[0157]

[0158] 2.3.13 Active visual tracking student module uses Adam optimization algorithm to minimize D KL (π 1* (a t |o′ t )|π 1 (a t |s t )) to update the parameters of the visual tracker network.

[0159] 2.3.14 Tracker According to the tracker's action probability π 1* (a t |o′ t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the target is based on the target’s action probability π 2 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the interference object is based on the action probability π of the interference object. 3 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking the action to interact with the environment.

[0160] 2.3.15 Let t = t + 1. If t ≤ T, go to 2.3.16; if t > T, go to 2.3.17.

[0161] 2.3.16 If t%5000=0, go to 2.3.4 to train the active visual tracking teacher module loaded with the parameters of the new meta-target network and meta-distractor network; otherwise go to 2.3.6 and continue training using the data of the next time step.

[0162] 2.3.17 Save the parameters of the visual tracker network and go to step 3.

[0163] In the third step, the active visual tracking student module of the scale-adaptive active visual tracking system loads the parameters of the visual tracker network saved in step 2.3.17 to obtain the trained scale-adaptive active visual tracking system.

[0164] In the fourth step, the trained scale-adaptive active visual tracking system tracks the target by:

[0165] 4.1 The data acquisition module obtains the real-time visual observation image o from the tracker's camera and sends o to the data preprocessing module.

[0166] 4.2 The data preprocessing module receives the real-time visual observation image o, and uses the second preprocessing method described in step 2.3.8 to preprocess the real-time visual observation image o to obtain the preprocessed real-time visual observation image o′, and sends o′ to the active visual tracking student module.

[0167] 4.3 Active visual tracking The student module receives the preprocessed real-time visual observation image o′, and the visual tracker network uses the action prediction method described in step 2.3.10 to predict the action of the preprocessed real-time visual observation image o′ to obtain the tracker's action probability π 1* (a′|o′). a′ is a random variable representing the action of the agent, and its value range is the action range of the agent.

[0168] 4.4 Visual Tracker Network The tracker action probability π 1* (a′|o′) is sent to the tracker.

[0169] 4.5 Tracker According to the tracker's action probability π 1* The conditional probability represented by (a′|o′) selects an action from the 7 actions in the action range and takes the action to track the target.

[0170] The following beneficial effects can be achieved by using a scale-adaptive active visual tracking method provided by the present invention:

[0171] (1) The first step of the present invention is to construct a scale-adaptive active visual tracking system. By using multi-scale convolutional network branches with different receptive field sizes and scale attention modules in the visual tracker network of the active visual tracking student module, the multi-scale features of the visual observation image can be extracted and the multi-scale features can be adaptively fused to enhance the tracking performance of the tracker when the target scale changes.

[0172] (2) The second step of the active visual tracking teacher module of the present invention further enhances the tracking performance of the tracker in a scale-changing environment by performing regular scale, small scale, and large scale training in a multi-scale and multi-stage training.

[0173] (3) The training of the active visual tracking teacher module in the second step of the present invention enhances the target's exploration of the environment and the diversity of its motion patterns by adding a position entropy reward to the target's reward function, thereby improving the tracking effect of the tracker when there are interferences in the environment.

[0174] (4) In the fourth step of the present invention, the trained scale-adaptive active visual tracking system is used to track the target, thereby improving the tracking performance of the tracker, that is, the cumulative reward AR, the screen length EL, and the tracking success rate SR are all improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0175] Figure 1 This is the logical structure diagram of the scale-adaptive active visual tracking system constructed in the first step;

[0176] Figure 2 It is an overall flow chart of the scale-adaptive active visual tracking method of the present invention. DETAILED DESCRIPTION

[0177] The specific implementation modes of the present invention will be further described in detail below in conjunction with the accompanying drawings.

[0178] like Figure 1 As shown, one embodiment of the present invention includes the following steps:

[0179] The first step is to construct a scale-adaptive active visual tracking system, which is composed of a data acquisition module, a data preprocessing module, an active visual tracking teacher module, and an active visual tracking student module.

[0180] The data acquisition module is connected to the camera of the tracker, the posture sensors of all intelligent agents, and the data preprocessing module. It acquires two-dimensional visual observation images from the camera of the tracker and acquires the posture data of the tracker, target, and interferers from the posture sensors of the tracker, target, and interferers. During training, the data acquisition module sends the visual observation images acquired from the camera of the tracker and the posture data acquired from the posture sensors of the tracker, target, and interferers to the data preprocessing module; when actually tracking the target, the data acquisition module only sends the visual observation images acquired from the camera of the tracker to the data preprocessing module.

[0181] The data preprocessing module is connected to the data acquisition module, the active visual tracking teacher module, and the active visual tracking student module. It receives the visual observation image and posture data from the data acquisition module, preprocesses the visual observation image and posture data, and obtains the preprocessed visual observation image and preprocessed posture data. During training, the data preprocessing module sends the preprocessed posture data to the active visual tracking teacher module, and sends the preprocessed visual observation image to the active visual tracking student module. During actual tracking, the data preprocessing module only sends the preprocessed visual observation image to the active visual tracking student module.

[0182] The function of the active visual tracking teacher module is to assist the active visual tracking student module. It only works during training and does not participate in the actual tracking. The active visual tracking teacher module is connected to the data preprocessing module and the active visual tracking student module, and consists of a meta-tracker network, a meta-target network, a meta-disturbance network, a reward function calculation module, and a first loss function calculation module. The meta-tracker network consists of a first bidirectional gated recurrent unit, a first long short-term memory unit, a first fully connected layer, and a second fully connected layer; the first bidirectional gated recurrent unit encodes the preprocessed posture data received from the data preprocessing module to obtain the posture features of the tracker, and sends the posture features of the tracker to the first long short-term memory unit; the first long short-term memory unit extracts the temporal features of the posture features of the tracker received from the first bidirectional gated recurrent unit to obtain the temporal posture features of the tracker, and sends the temporal posture features of the tracker to the first fully connected layer and the second fully connected layer; the first fully connected layer performs action prediction on the temporal posture features of the tracker received from the first long short-term memory unit to obtain the action probability of the tracker. When training the active visual tracking teacher module, the first fully connected layer sends the action probability of the tracker to the meta-target network, the meta-interference network, and the first loss function calculation module. When training the active visual tracking student module, the first fully connected layer sends the action probability of the tracker to the meta-target network, the meta-interference network, and the active visual tracking student module (in Figure 1The dotted lines in the figure indicate that the first fully connected layer sends the action probability of the tracker to the meta-target network, the meta-interference network, the first loss function calculation module, and the second loss function calculation module of the active visual tracking student module); the second fully connected layer estimates the value of the temporal posture features of the tracker received from the first long short-term memory unit to obtain the state value of the tracker. When training the active visual tracking teacher module, the second fully connected layer sends the state value of the tracker to the first loss function calculation module.

[0183] The meta-target network consists of a second bidirectional gated recurrent unit, a second long short-term memory unit, a third fully connected layer, and a fourth fully connected layer; the second bidirectional gated recurrent unit encodes the preprocessed posture data received from the data preprocessing module to obtain the posture features of the target, and sends the posture features of the target to the second long short-term memory unit; the second long short-term memory unit extracts the temporal features of the posture features of the target received from the second bidirectional gated recurrent unit and the action probability of the tracker received from the meta-tracker network to obtain the temporal posture features of the target, and sends the temporal posture features of the target to the third fully connected layer and the fourth fully connected layer; the third fully connected layer performs action prediction on the temporal posture features of the target received from the second long short-term memory unit to obtain the action probability of the target. When training the active visual tracking teacher module, the third fully connected layer sends the action probability of the target to the first loss function calculation module; the fourth fully connected layer estimates the value of the temporal posture features of the target received from the second long short-term memory unit to obtain the state value of the target. When training the active visual tracking teacher module, the fourth fully connected layer sends the state value of the target to the first loss function calculation module.

[0184] The meta-interference network consists of a third bidirectional gated recurrent unit, a third long short-term memory unit, a fifth fully connected layer, and a sixth fully connected layer; the third bidirectional gated recurrent unit encodes the preprocessed posture data received from the data preprocessing module to obtain the posture features of the interference, and sends the posture features of the interference to the third long short-term memory unit; the third long short-term memory unit extracts the temporal features of the posture features of the interference received from the third bidirectional gated recurrent unit and the action probability of the tracker received from the meta-tracker network to obtain the temporal posture features of the interference, and sends the temporal posture features of the interference to the third long short-term memory unit. to the fifth fully connected layer and the sixth fully connected layer; the fifth fully connected layer performs action prediction on the temporal position features of the distractor received from the third long short-term memory unit to obtain the action probability of the distractor. When training the active visual tracking teacher module, the fifth fully connected layer sends the action probability of the distractor to the first loss function calculation module; the sixth fully connected layer performs value estimation on the temporal position features of the distractor received from the third long short-term memory unit to obtain the state value of the distractor. When training the active visual tracking teacher module, the sixth fully connected layer sends the state value of the distractor to the first loss function calculation module.

[0185] The reward function calculation module is connected to the data preprocessing module and the first loss function calculation module. When training the active visual tracking teacher module, the reward function calculation module calculates the rewards of the tracker, the target, and the distractor according to the preprocessed posture data received from the data preprocessing module, and sends the rewards of the tracker, the target, and the distractor to the first loss function calculation module.

[0186] The first loss function calculation module is connected to the reward function calculation module, the meta-tracker network, the meta-target network, and the meta-interference network. When training the active visual tracking teacher module, the first loss function calculation module receives the rewards of the tracker, the target, and the interference from the reward function calculation module, receives the action probability of the tracker and the state value of the tracker from the meta-tracker network, receives the action probability of the target and the state value of the target from the meta-target network, receives the action probability of the interference and the state value of the interference from the meta-interference network, calculates the temporal difference error loss and policy gradient loss of the tracker according to the reward, action probability, and state value of the tracker, calculates the temporal difference error loss and policy gradient loss of the target according to the reward, action probability, and state value of the target, and calculates the temporal difference error loss and policy gradient loss of the interference according to the reward, action probability, and state value of the interference. When training the active visual tracking teacher module, the active visual tracking teacher module optimizes the meta-tracker network, the meta-target network, and the meta-interference network by minimizing the temporal difference error loss and policy gradient loss of the tracker, the target, and the interference obtained by the first loss function calculation module.

[0187] The active visual tracking student module is connected to the data preprocessing module and the active visual tracking teacher module, and is composed of a visual tracker network and a second loss function calculation module. The visual tracker network is composed of a convolutional backbone network, a multi-scale convolutional network branch, a scale attention module, an adaptive fusion module, a recurrent attention module, and the eighth fully connected layer.

[0188] The convolution backbone network is connected to the data preprocessing module, the multi-scale convolutional network branch and the scale attention module. It performs feature extraction on the preprocessed visual observation image received from the data preprocessing module to obtain visual observation image features, and sends the visual observation image features to the multi-scale convolutional network branch and the scale attention module.

[0189] The number of branches of the multi-scale convolutional network branch is 4, which consists of a first-scale convolutional network branch, a second-scale convolutional network branch, a third-scale convolutional network branch, and a fourth-scale convolutional network branch. The first-scale convolutional network branch extracts features from visual observation image features to obtain first-scale visual observation image features, and sends the first-scale visual observation image features to the adaptive fusion module; the second-scale convolutional network branch extracts features from visual observation image features to obtain second-scale visual observation image features, and sends the second-scale visual observation image features to the adaptive fusion module; the third-scale convolutional network branch extracts features from visual observation image features to obtain third-scale visual observation image features, and sends the third-scale visual observation image features to the adaptive fusion module; the fourth-scale convolutional network branch performs dimensionality reduction and feature extraction on the visual observation image features received from the convolutional backbone network to obtain fourth-scale visual observation image features, and sends the fourth-scale visual observation image features to the adaptive fusion module; the first-scale convolutional network branch It consists of a first ordinary convolution layer with a convolution kernel size of 1×1 and a first dilated convolution layer with a convolution kernel size of 3×3 and a dilation rate of 3, with a receptive field size of 7; the second-scale convolution network branch consists of a second ordinary convolution layer with a convolution kernel size of 1×1 and a second dilated convolution layer with a convolution kernel size of 3×3 and a dilation rate of 2, with a receptive field size of 5; the third-scale convolution network branch consists of a third ordinary convolution layer with a convolution kernel size of 1×1 and a third dilated convolution layer with a convolution kernel size of 3×3 and a dilation rate of 1, with a receptive field size of 3; the fourth-scale convolution network branch consists of a fourth ordinary convolution layer with a convolution kernel size of 1×1, with a receptive field size of 1; the first-scale visual observation image features, the second-scale visual observation image features, the third-scale visual observation image features, and the fourth-scale visual observation image features constitute multi-scale visual observation image features. The multi-scale convolution network branch can extract features of visual observation images in multiple scale ranges (corresponding to receptive fields of different sizes), so it can extract features of targets with varying scales.

[0190] The scale attention module consists of a convolutional long short-term memory unit, a global average pooling layer, a seventh fully connected layer, and a sigmoid activation function layer; the convolutional long short-term memory unit extracts temporal features from the visual observation image features received from the convolutional backbone network to obtain temporal features of the visual observation image, and sends the temporal features of the visual observation image to the global average pooling layer; the global average pooling layer compresses the temporal features of the visual observation image received from the convolutional long short-term memory unit in the spatial dimension to obtain a first feature vector whose length is equal to the number of channels of the temporal features of the visual observation image, and sends the first feature vector to the seventh fully connected layer; the seventh fully connected layer compresses the first feature vector received from the global average pooling layer to obtain a second feature vector whose length is equal to the number of branches of the multi-scale convolutional network (4), and sends the second feature vector to the sigmoid activation function layer; the sigmoid activation function layer scales the value of the second feature vector received from the seventh fully connected layer to the range of [0,1] to obtain the weights of the multi-scale visual observation image features, and sends the weights of the multi-scale visual observation image features to the adaptive fusion module. The scale attention module can adaptively calculate the weights of multi-scale visual observation image features according to the scale of the target in the visual observation image. The visual observation image features of the scale containing more target features are assigned larger weights, and the visual observation image features of the scale containing fewer target features are assigned smaller weights.

[0191] The adaptive fusion module is connected to the multi-scale convolutional network branch, the scale attention module, and the recurrent attention module, and is composed of a weighted splicing layer and a fifth ordinary convolutional layer with a convolution kernel size of 1×1; the weighted splicing layer uses the weights of the multi-scale visual observation image features received from the scale attention module to weightedly splice the first scale visual observation image features, the second scale visual observation image features, the third scale visual observation image features, and the fourth scale visual observation image features received from the multi-scale convolutional network branch to obtain the weighted spliced ​​multi-scale visual observation image features, and sends the weighted spliced ​​multi-scale visual observation image features to the fifth ordinary convolutional layer; the fifth ordinary convolutional layer reduces the channel dimension of the weighted spliced ​​multi-scale visual observation image features received from the weighted splicing layer to obtain the adaptively fused multi-scale visual observation image features, and sends the adaptively fused multi-scale visual observation image features to the recurrent attention module. The adaptive fusion module uses the weights of the multi-scale visual observation image features to weightedly fuse the multi-scale visual observation image features, thereby adaptively extracting features from targets with varying scales.

[0192] The recurrent attention module is connected to the adaptive fusion module and the eighth fully connected layer, performs temporal feature extraction and weighting on the adaptively fused multi-scale visual observation image features received from the adaptive fusion module, obtains adaptive temporal scale features, and sends the adaptive temporal scale features to the eighth fully connected layer.

[0193] The eighth fully connected layer performs action prediction on the adaptive temporal scale features received from the recurrent attention module to obtain the action probability of the tracker. When training the active visual tracking student module, the eighth fully connected layer sends the action probability of the tracker to the second loss function calculation module.

[0194] The second loss function calculation module is connected to the eighth fully connected layer and the meta tracker network. When training the active visual tracking student module, the second loss function calculation module calculates the KL divergence loss for the action probability of the tracker received from the eighth fully connected layer and the action probability of the tracker received from the meta tracker network of the active visual tracking teacher module. When training the active visual tracking student module, the active visual tracking student module optimizes the visual tracker network by minimizing the KL divergence loss obtained by the second loss function calculation module.

[0195] The second step is to train the active visual tracking teacher module and the active visual tracking student module in the scale-adaptive active visual tracking system. The training method of the active visual tracking teacher module is a reinforcement learning method, and the training method of the active visual tracking student module is a supervised learning method.

[0196] 2.1 The active visual tracking teacher module is trained by using reinforcement learning. The method is to use adversarial reinforcement learning to perform multi-scale and multi-stage training on the meta-tracker network, meta-target network and meta-distractor network during the interaction between the tracker, target, distractor and environment. The reward functions of the target and distractor in each stage are different, corresponding to different scales of training, specifically:

[0197] 2.1.1 The first phase of regular scale training of the active tracking teacher module is carried out by using reinforcement learning. The method is:

[0198] 2.1.1.1 Set the number of training steps T = 2000000 and initialize the number of trained steps t = 1.

[0199] 2.1.1.2 Set the target position entropy reward coefficient a=0.8, set the regularization factor α=0.005; set the reward attenuation factor γ=0.9; set the number of environmental horizontal grid divisions n=10.

[0200] 2.1.1.3 Set the number of steps used to calculate the target position entropy reward N = min(t, 50);

[0201] 2.1.1.4 The data acquisition module obtains the pose data of the tth step from the pose sensors of the tracker, target, and interference, and sends the pose data of the tth step to the data preprocessing module. The pose data of the tth step includes the position coordinates of the tracker and orientation angle The target's location coordinates and orientation angle The coordinates of the distractor's location and orientation angle

[0202] 2.1.1.5 The data preprocessing module uses the first preprocessing method to preprocess the posture data of the tth step received from the data acquisition module to obtain the preprocessed posture data of the tth step, and sends the preprocessed posture data of the tth step to the active visual tracking teacher module. The first preprocessing method is:

[0203] 2.1.1.5.1 Calculate the relative distance between the target's actual position and the tracker Relative Angle Relative orientation angle The relative distance of the actual position of the interference object relative to the tracker Relative Angle Relative orientation angle

[0204]

[0205]

[0206] 2.1.1.5.2 Set the relative distance ρ of the target ideal position relative to the tracker * =250cm, relative angle θ * =0, relative orientation angle ω * =0, indicating that the ideal position of the target is 250cm in front of the tracker and facing the same direction as the tracker; Set the maximum distance of the tracker's field of view ρ max =750cm and maximum angle θ max =90°.

[0207] 2.1.1.5.3 Calculate the normalized relative distance, normalized relative angle, and normalized relative orientation angle of the actual position of the target and the actual position of the interference object:

[0208] 2.1.1.5.3.1 Calculate the normalized relative distance of the target's actual position Normalized relative distance to the actual position of the distractor

[0209] 2.1.1.5.3.2 Calculate the relative angle of the target's actual position The sine value of and cosine value cos Get the relative angle of the normalized representation of the actual position of the target;

[0210] 2.1.1.5.3.3 Calculate the relative orientation angle of the target's actual position The sine value of and cosine value cos Obtain the relative orientation angle of the normalized representation of the actual position of the target;

[0211] 2.1.1.5.3.4 Calculate the relative angle of the actual position of the interference object The sine value of and cosine value cos Get the relative angle of the normalized representation of the actual position of the interference object;

[0212] 2.1.1.5.3.5 Calculate the relative orientation angle of the actual position of the interference object The sine value of and cosine value cos The relative orientation angle of the normalized representation of the actual position of the interference object is obtained.

[0213] 2.1.1.5.4 Set the tracker's position coordinates The target's location coordinates The coordinates of the distractor's location The relative distance to the actual position of the target Relative Angle Relative orientation angle The relative distance of the actual location of the distractor Relative Angle Relative orientation angle Normalized relative distance to the target's actual position The relative angle sin of the normalized representation of the actual position of the target cos The relative orientation angle sin of the normalized representation of the actual position of the target cos Normalized relative distance to the actual position of the distractor The relative angle sin of the normalized representation of the actual position of the distractor cos The relative orientation angle sin of the normalized representation of the actual position of the interference object cos The preprocessed pose data that constitutes the tth step.

[0214] 2.1.1.6 The reward function calculation module in the active visual tracking teacher module calculates the tracker’s reward based on the preprocessed pose data of step t

[0215]

[0216] 2.1.1.7 The reward function calculation module in the active visual tracking teacher module divides the training environment into I grids, I = n × n, and uses the frequency calculation method to calculate the set of target position coordinates in the past N steps based on the preprocessed pose data in the past t steps (the set of target position coordinates is ) appears in each grid with a frequency p1,…,p i ,…,p I , the method is:

[0217] 2.1.1.7.1 Initialize variable i = 1;

[0218] 2.1.1.7.2 Calculate the frequency p of the set of target position coordinates in the past N steps appearing in the i-th grid i :

[0219]

[0220] Where N i is the number of positions in the i-th grid in the set of target position coordinates in the past N steps, N i <N。

[0221] 2.1.1.7.3 Let i=i+1, if i≤I, go to 3.1.1.7.2; if i>I, get p1,…,p i ,…,p I , go to 2.1.1.8.

[0222] 2.1.1.8 The reward function calculation module in the active vision teacher module calculates the position entropy reward loc_entropy of the target:

[0223]

[0224] 2.1.1.9 The reward function calculation module in the active visual tracking teacher module calculates the reward of the target based on the preprocessed pose data of step t

[0225] 2.1.1.10 The reward function calculation module in the active visual tracking teacher module calculates the reward of the distractor based on the preprocessed pose data of step t

[0226]

[0227] 2.1.1.11 The reward function calculation module in the active visual tracking teacher module converts the tracker's reward Rewards for goals Distractor Rewards Sent to the first loss function calculation module in the active visual tracking teacher module.

[0228] 2.1.1.12 The meta-tracker network, meta-target network, and meta-disturbance network in the active visual tracking teacher module use estimation methods to perform action prediction and value estimation on the preprocessed pose data, respectively, to obtain the state value V of the tracker. 1 (s t ) and the tracker’s action probability π 1 (a t |s t ), the state value V of the target 2 (s t ) and the target action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ) and the action probability π of the distractor 3 (a t |s t ), the method is:

[0229] 2.1.1.12.1 Let the normalized relative distance of the actual position of the target in the preprocessed pose data of step t be Normalized relative angle sin cos Normalized relative orientation angle sin cos Normalized relative distance to the actual position of the distractor Normalized relative angle sin cos Normalized relative orientation angle sin cos The environmental state that constitutes step t

[0230] 2.1.1.12.2 The First Bidirectional Gated Recurrent Unit Pair in the Meta-Tracker Network tEncode to obtain the posture features of the tracker, and send the posture features of the tracker to the first long short-term memory unit; the first long short-term memory unit extracts the temporal features of the posture features of the tracker received from the first bidirectional gated recurrent unit to obtain the temporal posture features of the tracker, and sends the temporal posture features of the tracker to the first fully connected layer and the second fully connected layer; the first fully connected layer predicts the action of the temporal posture features of the tracker received from the first long short-term memory unit to obtain the action probability π of the tracker 1 (a t |s t ), the first fully connected layer sends the action probability of the tracker to the meta-target network and the meta-interference network; the second fully connected layer estimates the value of the temporal posture features of the tracker received from the first long short-term memory unit to obtain the state value V of the tracker 1 (s t ).

[0231] 2.1.1.12.3 Second Bidirectional Gated Recurrent Unit Pair s in Meta-Target Network t Encode to obtain the target's posture features, and send the target's posture features to the second long short-term memory unit; the second long short-term memory unit encodes the target's posture features received from the second bidirectional gated recurrent unit and the tracker's action probability π received from the meta-tracker network 1 (a t |s t ) extracts temporal features to obtain the temporal position and posture features of the target, and sends the temporal position and posture features of the target to the third fully connected layer and the fourth fully connected layer; the third fully connected layer performs action prediction on the temporal position and posture features of the target received from the second long short-term memory unit to obtain the action probability π of the target 2 (a t |s t ); The fourth fully connected layer estimates the value of the temporal position features of the target received from the second long short-term memory unit to obtain the state value V of the target 2 (s t ).

[0232] 2.1.1.12.4 The third bidirectional gated recurrent unit pair s in the perturbation network t Encode to obtain the pose feature of the interferer, and send the pose feature of the interferer to the third long short-term memory unit; the third long short-term memory unit encodes the pose feature of the interferer received from the third bidirectional gated recurrent unit and the action probability π of the tracker received from the meta-tracker network 1 (a t |s t) to extract temporal features, obtain temporal position features of the interferer, and send the temporal position features of the interferer to the fifth fully connected layer and the sixth fully connected layer; the fifth fully connected layer performs action prediction on the temporal position features of the interferer received from the third long short-term memory unit to obtain the action probability π of the interferer 3 (a t |s t ); The sixth fully connected layer estimates the value of the temporal position features of the interference object received from the third long short-term memory unit to obtain the state value V of the interference object 3 (s t ). t is a random variable representing the action of the agent, and its value range is the action range of the agent (the action range of the tracker, target, and interferer are all 7 actions: forward, backward, left, right, forward and left, forward and right, stop), π k (a t |s t ) is a conditional probability, indicating that in state s t The probability distribution of the 7 actions in the action range of the agent, k∈{1,2,3}, 1 represents the tracker, 2 represents the target, and 3 represents the distractor.

[0233] 2.1.1.13 The meta-tracker network in the active visual tracking teacher module converts the state value V of the tracker 1 (s t ) and action probability π 1 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the tracker is 1 (a t |s t ) is sent to the tracker. The meta-target network converts the target state value V 2 (s t ) and action probability π 2 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the target action probability π 2 (a t |s t ) is sent to the target. The meta-interference network sends the state value V of the distractor 3 (s t ) and action probability π 3 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the interference object is 3 (at |s t ) is sent to the interferer.

[0234] 2.1.1.14 If t = 1, go to 2.1.1.16; otherwise: the first loss function calculation module in the active visual tracking teacher module calculates the reward of the tracker received from the reward function calculation module Rewards for goals Distractor Rewards and the state value V of the tracker received from the meta-tracker network, meta-target network, and meta-distractor network 1 (s t ), the action probability π of the tracker 1 (a t |s t ), the state value V of the target 2 (s t ), the target action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ), the action probability of the distractor π 3 (a t |s t ) Calculate the time difference error loss of the tracker Policy gradient loss for tracker Target temporal difference error loss Policy gradient loss of the target Temporal Differential Error Loss of Interferers Policy Gradient Loss for Distractors

[0235]

[0236] where logπ k (a t |s t ) is π k (a t |s t ), H(π k (·|s t )) is the entropy regularization term,

[0237] 2.1.1.15 The active visual tracking teacher module uses the Adam optimization algorithm (see the document "Kingma DP, BaJ.Adam:A method for stochastic optimization[J].arXiv preprint arXiv:1412.6980,2014", Kingma's paper: Adam, a stochastic optimization method) to minimize and To update the parameters of the meta-tracker network, use the Adam optimization algorithm to minimize and To update the parameters of the meta-target network, the Adam optimization algorithm is used to minimize and to update the parameters of the meta-distractor network.

[0238] 2.1.1.16 Tracker According to the tracker's action probability π 1 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the target is based on the target’s action probability π 2 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the interference object is based on the action probability π of the interference object. 3 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking the action to interact with the environment.

[0239] 2.1.1.17 Let t = t + 1. If t ≤ T, go to 2.1.1.18; if t > T, go to 2.1.1.19.

[0240] 2.1.1.18 If t%10000=0: save the parameters of the meta-target network and the meta-interference network to the opponent model pool, go to 2.1.1.3, and continue training using the data of the next time step; otherwise: directly go to 2.1.1.3 and continue training using the data of the next time step.

[0241] 2.1.1.19 Save the parameters of the meta-tracker network, meta-target network, and meta-distractor network, and go to 2.1.2.

[0242] 2.1.2 After the first stage of regular scale training, the meta-tracker network has the ability to track regular scale targets, but still lacks the ability to track small and large scale targets. Therefore, based on the first stage of training, the active visual tracking teacher module is further trained on a second stage of small scale using reinforcement learning. The method is:

[0243] 2.1.2.1 Meta-tracker network, meta-target network, meta-distractor network Load the parameters of the meta-tracker network, meta-target network, and meta-distractor network saved at the end of the first stage of training in 2.1.1.19.

[0244] 2.1.2.2 Set the number of training steps T = 1000000 and initialize the number of trained steps t = 1;

[0245] 2.1.2.3 Set the target position entropy reward coefficient a = 0.8; set the regularization factor α = 0.005; set the reward attenuation factor γ = 0.9; set the number of environmental horizontal grid divisions n = 10; set the threshold distance ρ for the target to be away from the tracker s =500cm.

[0246] 2.1.2.4 Set the number of steps used to calculate the target position entropy reward to N = min(t, 50).

[0247] 2.1.2.5 The data acquisition module obtains the posture data of the tth step from the posture sensors of the tracker, target, and interference object, and sends the posture data of the tth step to the data preprocessing module.

[0248] 2.1.2.6 The data preprocessing module adopts the first preprocessing method in 2.1.1.5 to preprocess the posture data of the tth step received from the data acquisition module to obtain the preprocessed posture data of the tth step, and sends the preprocessed posture data of the tth step to the active visual tracking teacher module.

[0249] 2.1.2.7 The reward function calculation module in the active visual tracking teacher module calculates the reward of the tracker according to the formula (7) in 2.1.1.6 based on the preprocessed pose data of step t

[0250] 2.1.2.8 The reward function calculation module in the active visual tracking teacher module divides the training environment into I grids, and uses the frequency calculation method described in 2.1.1.7 to calculate the frequency p1,…,p of the set of target position coordinates in the past N steps in each grid based on the preprocessed pose data of the past t steps. i ,…,p I .

[0251] 2.1.2.9 The reward function calculation module in the active vision teacher module calculates the target's position entropy reward loc_entropy according to formula (9) in 2.1.1.8.

[0252] 2.1.2.10 The reward function calculation module in the active visual tracking teacher module calculates the reward of the target based on the preprocessed pose data of step t

[0253]

[0254] 2.1.2.11 The reward function calculation module in the active visual tracking teacher module calculates the reward of the distractor based on the preprocessed pose data of step t

[0255]

[0256] 2.1.2.12 The reward function calculation module in the active visual tracking teacher module converts the tracker's reward Rewards for goals Distractor Rewards Sent to the first loss function calculation module in the active visual tracking teacher module.

[0257] 2.1.2.13 The meta-tracker network, meta-target network, and meta-interference network in the active visual tracking teacher module use the estimation method described in 2.1.1.12 to perform action prediction and value estimation on the preprocessed pose data, respectively, to obtain the state value V of the tracker 1 (s t ) and the tracker’s action probability π 1 (a t |s t ), the state value V of the target 2 (s t ) and the target action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ) and the action probability π of the distractor 3 (a t |s t ).

[0258] 2.1.2.14 The meta-tracker network in the active visual tracking teacher module converts the state value V of the tracker 1 (s t ) and action probability π 1 (a t |s t) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the tracker is 1 (a t |s t ) is sent to the tracker. The meta-target network converts the target state value V 2 (s t ) and action probability π 2 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the target action probability π 2 (a t |s t ) is sent to the target. The meta-interference network sends the state value V of the distractor 3 (s t ) and action probability π 3 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the interference object is 3 (a t |s t ) is sent to the interferer.

[0259] 2.1.2.15 If t = 1, go to 2.1.2.17; otherwise, the first loss function calculation module in the active visual tracking teacher module calculates the reward of the tracker received from the reward function calculation module according to formula (12) and formula (13) in 2.1.1.14. Rewards for goals Distractor Rewards and the state value V of the tracker received from the meta-tracker network, meta-target network, and meta-distractor network 1 (s t ), the action probability π of the tracker 1 (a t |s t ), the state value V of the target 2 (s t ), the target action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ), the action probability of the distractor π 3 (a t |s t ) Calculate the time difference error loss of the tracker Policy gradient loss for tracker Target temporal difference error loss Policy gradient loss of the target Temporal Differential Error Loss of Interferers Policy Gradient Loss for Distractors

[0260] 2.1.2.16 Active visual tracking teacher module uses Adam optimization algorithm to minimize and To update the parameters of the meta-tracker network, use the Adam optimization algorithm to minimize and To update the parameters of the meta-target network, the Adam optimization algorithm is used to minimize and to update the parameters of the meta-distractor network.

[0261] 2.1.2.17 Tracker According to the tracker's action probability π 1 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the target is based on the target’s action probability π 2 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the interference object is based on the action probability π of the interference object. 3 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking the action to interact with the environment.

[0262] 2.1.2.18 Let t = t + 1. If t ≤ T, go to 2.1.2.19; if t > T, go to 2.1.2.20.

[0263] 2.1.2.19 If t%10000=0: save the parameters of the meta-target network and the meta-interference network to the opponent model pool, and go to 2.1.2.4 to continue training using the data of the next time step; otherwise: directly go to 2.1.2.4 to continue training using the data of the next time step.

[0264] 2.1.2.20 Save the parameters of the meta-tracker network, meta-target network, and meta-distractor network, and go to 2.1.3.

[0265] 2.1.3 After the first stage of regular scale training and the second stage of small scale training, the meta-tracker network has the ability to track regular scale and small scale targets, but still lacks the ability to track large scale targets. Therefore, based on the second stage training, reinforcement learning is used to further perform the third stage large scale training on the active visual tracking teacher module.

[0266] 2.1.3.1 Meta-tracker network, meta-target network, meta-distractor network Load the parameters of the meta-tracker network, meta-target network, and meta-distractor network saved at the end of the second stage of training in 2.1.2.20.

[0267] 2.1.3.2 Set the number of training steps T = 1000000, and initialize the number of trained steps t = 1;

[0268] 2.1.3.3 Set the target position entropy reward coefficient a = 0.8; set the regularization factor α = 0.005; set the reward attenuation factor γ = 0.9; set the number of environmental horizontal grid divisions n = 10; set the threshold distance ρ for the target to approach the tracker l =150cm.

[0269] 2.1.3.4 Set the number of steps used to calculate the target position entropy reward to N = min(t, 50).

[0270] 2.1.3.5 The data acquisition module obtains the posture data of the tth step from the posture sensors of the tracker, target, and interference object, and sends the posture data of the tth step to the data preprocessing module.

[0271] 2.1.3.6 The data preprocessing module adopts the first preprocessing method in 2.1.1.5 to preprocess the posture data of the tth step received from the data acquisition module to obtain the preprocessed posture data of the tth step, and sends the preprocessed posture data of the tth step to the active visual tracking teacher module.

[0272] 2.1.3.7 The reward function calculation module in the active visual tracking teacher module calculates the reward of the tracker according to the formula (7) in 2.1.1.6 based on the preprocessed pose data of the tth step

[0273] 2.1.3.8 The reward function calculation module in the active visual tracking teacher module divides the training environment into I grids, and uses the frequency calculation method described in 2.1.1.7 to calculate the frequency p1,…,p of the set of target position coordinates in the past N steps in each grid based on the preprocessed pose data of the past t steps. i ,…,p I .

[0274] 2.1.3.9 The reward function calculation module in the active vision teacher module calculates the target's position entropy reward loc_entropy according to formula (9) in 2.1.1.8.

[0275] 2.1.3.10 The reward function calculation module in the active visual tracking teacher module calculates the reward of the target based on the preprocessed pose data of step t

[0276]

[0277] 2.1.3.11 The reward function calculation module in the active visual tracking teacher module calculates the reward of the distractor based on the preprocessed pose data of step t

[0278]

[0279] 2.1.3.12 The reward function calculation module in the active visual tracking teacher module converts the tracker's reward Rewards for goals Distractor Rewards Sent to the first loss function calculation module in the active visual tracking teacher module.

[0280] 2.1.3.13 The meta-tracker network, meta-target network, and meta-interference network in the active visual tracking teacher module use the estimation method described in step 2.1.1.12 to perform action prediction and value estimation on the preprocessed pose data, respectively, to obtain the state value V of the tracker 1 (s t ) and the tracker’s action probability π 1 (a t |s t ), the state value V of the target 2 (s t t) and the target action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ) and the action probability π of the distractor 3 (a t |s t ).

[0281] 2.1.3.14 The meta-tracker network in the active visual tracking teacher module converts the state value V of the tracker 1 (s t ) and action probability π 1 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the tracker is 1 (a t |s t ) is sent to the tracker. The meta-target network converts the target state value V 2 (s t ) and action probability π 2 (at |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the target action probability π 2 (a t |s t ) is sent to the target. The meta-interference network sends the state value V of the distractor 3 (s t ) and action probability π 3 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the interference object is 3 (a t |s t ) is sent to the interferer.

[0282] 2.1.3.15 If t = 1, go to 2.1.3.17; otherwise, the first loss function calculation module in the active visual tracking teacher module calculates the reward of the tracker received from the reward function calculation module according to formula (12) and formula (13) in 2.1.1.14. Rewards for goals Distractor Rewards and the state value V of the tracker received from the meta-tracker network, meta-target network, and meta-distractor network 1 (s t ), the action probability π of the tracker 1 (a t |s t ), the state value V of the target 2 (s t ), the target action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ), the action probability of the distractor π 3 (a t |s t ) Calculate the time difference error loss of the tracker Policy gradient loss for tracker Target temporal difference error loss Policy gradient loss of the target Temporal Differential Error Loss of Interferers Policy Gradient Loss for Distractors

[0283] 2.1.3.16 Active visual tracking teacher module uses Adam optimization algorithm to minimize and To update the parameters of the meta-tracker network, use the Adam optimization algorithm to minimize and To update the parameters of the meta-target network, the Adam optimization algorithm is used to minimize and to update the parameters of the meta-distractor network.

[0284] 2.1.3.17 Tracker According to the tracker's action probability π 1 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the target is based on the target’s action probability π 2 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the interference object is based on the action probability π of the interference object. 3 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking the action to interact with the environment.

[0285] 2.1.3.18 Let t = t + 1. If t ≤ T, go to 2.1.3.19. If t > T, go to 2.1.4.

[0286] 2.1.3.19 If t%10000=0, save the parameters of the meta-target network and the meta-interference network to the opponent model pool, and go to 2.1.3.4 to continue training using the data of the next time step; otherwise: directly go to 2.1.3.4 to continue training using the data of the next time step.

[0287] 2.1.4 Active Visual Tracking Teacher Module Save the parameters of the meta-tracker network, go to 2.2.

[0288] 2.2 During the interaction between the tracker and the target, the interference, and the environment, the meta-tracker network is further fine-tuned by using reinforcement learning. Only the parameters of the meta-tracker network are optimized. The method is as follows:

[0289] 2.2.1 Meta-tracker network in the active visual tracking teacher module Load the parameters of the meta-tracker network saved in step 2.1.4.

[0290] 2.2.2 Set the number of training steps T = 1000000 and initialize the number of trained steps t = 1.

[0291] 2.2.3 Randomly select the parameters of the meta-target network from the opponent model pool and load them into the meta-target network; randomly select the parameters of the meta-interferer network from the opponent model pool and load them into the meta-interferer network.

[0292] 2.2.4 Fix the parameters of the meta-target network and meta-distractor network.

[0293] 2.2.5 The data acquisition module obtains the posture data of the tth step from the posture sensors of the tracker, target, and interference object, and sends the posture data of the tth step to the data preprocessing module.

[0294] 2.2.6 The data preprocessing module adopts the first preprocessing method in 2.1.1.5 to preprocess the posture data of the tth step received from the data acquisition module to obtain the preprocessed posture data of the tth step, and sends the preprocessed posture data of the tth step to the active visual tracking teacher module.

[0295] 2.2.7 The reward function calculation module in the active visual tracking teacher module calculates the reward of the tracker according to the formula (7) in 2.1.1.6 based on the preprocessed pose data of the tth step

[0296] 2.2.8 The reward function calculation module in the active visual tracking teacher module converts the tracker's reward Sent to the first loss function calculation module in the active visual tracking teacher module.

[0297] 2.2.9 The meta-tracker network, meta-target network, and meta-interference network in the active visual tracking teacher module use the estimation method described in step 2.1.1.12 to perform action prediction and value estimation on the preprocessed pose data, respectively, to obtain the state value V of the tracker. 1 (s t ) and the tracker’s action probability π 1 (a t |s t ), the state value V of the target 2 (s t ) and the target action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ) and the action probability π of the distractor 3 (a t |s t ).

[0298] 2.2.10 The meta-tracker network in the active visual tracking teacher module converts the state value V of the tracker 1 (s t ) and action probability π 1 (a t |s t) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the tracker is 1 (a t |s t ) is sent to the tracker. The meta-target network converts the target action probability π 2 (a t |s t ) is sent to the target. The meta-interference network sends the action probability π of the distractor 3 (a t |s t ) is sent to the interferer.

[0299] 2.2.11 If t = 1, go to 2.2.13; otherwise, the first loss function calculation module in the active visual tracking teacher module calculates the reward of the tracker received from the reward function calculation module according to formula (12) and formula (13) in step 2.1.1.14. and the state value V of the tracker received from the meta-tracker network 1 (s t ), the action probability π of the tracker 1 (a t |s t ) Calculate the time difference error loss of the tracker Policy gradient loss for tracker

[0300] 2.2.12 Active visual tracking teacher module uses Adam optimization algorithm to minimize and to update the parameters of the meta-tracker network.

[0301] 2.2.13 Tracker According to the tracker's action probability π 1 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the target is based on the target’s action probability π 2 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the interference object is based on the action probability π of the interference object. 3 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking the action to interact with the environment.

[0302] 2.2.14 Let t = t + 1. If t ≤ t, go to 2.2.15. If t > T, go to 2.2.16.

[0303] 2.2.15 If t%5000=0, go to 2.2.3 to train the active visual tracking teacher module loaded with the parameters of the new meta-target network and meta-distractor network; otherwise, go to 2.2.5 to continue training using the data of the next time step.

[0304] 2.2.16 The active visual tracking teacher module saves the parameters of the meta-tracking network, go to 2.3.

[0305] 2.3 The active visual tracking student module is trained using supervised learning. The method is as follows:

[0306] 2.3.1 Meta-tracker network in the active visual tracking teacher module Load the parameters of the meta-tracker network saved in step 2.2.16.

[0307] 2.3.2 Fixed parameters of the meta-tracker network.

[0308] 2.3.3 Set the number of training steps T = 2000000 and initialize the number of trained steps t = 1.

[0309] 2.3.4 Randomly select the parameters of the meta-target network from the opponent model pool and load them into the meta-target network; randomly select the parameters of the meta-interferer network from the opponent model pool and load them into the meta-interferer network.

[0310] 2.3.5 Fix the parameters of the meta-target network and meta-distractor network.

[0311] 2.3.6 Data acquisition module obtains the visual observation image o of step t from the camera of the tracker t , obtain the pose data of the tth step from the pose sensors of the tracker, target, and interference object, and send the visual observation image and pose data of the tth step to the data preprocessing module.

[0312] 2.3.7 The data preprocessing module adopts the first preprocessing method in step 2.1.1.5 to preprocess the posture data of the tth step received from the data acquisition module, obtains the preprocessed posture data of the tth step, and sends the preprocessed posture data of the tth step to the active visual tracking teacher module.

[0313] 2.3.8 The data preprocessing module uses the second preprocessing method to process the visual observation image o of the tth step received from the data acquisition module. t Perform preprocessing to obtain the preprocessed visual observation image o of step t t ′, the preprocessed visual observation image o′ of step t t Sent to the active visual tracking student module. The second preprocessing method is:

[0314] 2.3.8.1 o tThe width is scaled to 320 pixels and the height is scaled to 240 pixels to obtain the scaled visual observation image.

[0315] 2.3.8.2 Normalize the pixel values ​​of the scaled visual observation image to the interval [0,1] to obtain the normalized visual observation image of the tth step.

[0316] 2.3.8.3 Randomly flip the normalized visual observation image of step t horizontally to obtain the preprocessed visual observation image o′ of step t t .

[0317] 2.3.9 The meta-tracker network, meta-target network, and meta-interference network in the active visual tracking teacher module use the estimation method described in 2.1.1.12 to perform action prediction and value estimation on the preprocessed pose data, respectively, to obtain the state value V of the tracker. 1 (s t ) and action probability π 1 (a t |s t ), the state value V of the target 2 (s t ) and action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ) and action probability π 3 (a t |s t ). The meta-tracker network transforms the tracker’s action probability π 1 (a t |s t ) is sent to the second loss function calculation module in the active visual tracking student module, and the meta-target network converts the target action probability π 2 (a t |s t ) is sent to the target, and the meta-interference network sends the action probability π of the distractor 3 (a t |s t ) is sent to the interferer.

[0318] 2.3.10 Active Visual Tracking The visual tracker network in the student module uses an action prediction method to preprocess the visual observation image o′ t Perform action prediction and obtain the action probability π of the tracker 1* (a t |o′ t ), the method is:

[0319] 2.3.10.1 Convolutional backbone in the visual tracker network for the preprocessed visual observation image o′ t Perform feature extraction to obtain visual observation image features, and send the visual observation image features to the multi-scale convolutional network branch and the scale attention module.

[0320] 2.3.10.2 The first scale convolutional network branch in the multi-scale convolutional network branch extracts features from visual observation image features to obtain first scale visual observation image features, wherein the first common convolutional layer reduces the dimension of the visual observation image features received from the convolutional backbone network in the channel dimension to obtain the first visual observation image features after dimension reduction, and sends the first visual observation image features after dimension reduction to the first dilated convolutional layer; the first dilated convolutional layer extracts features from the first visual observation image features after dimension reduction received from the first common convolutional layer to obtain first scale visual observation image features, and sends the first scale visual observation image features to the adaptive fusion module; the second scale convolutional network branch in the multi-scale convolutional network branch extracts features from visual observation image features to obtain second scale visual observation image features, wherein the second common convolutional layer reduces the dimension of the visual observation image features received from the convolutional backbone network in the channel dimension to obtain the second visual observation image features after dimension reduction, and sends the second visual observation image features after dimension reduction to the second dilated convolutional layer; the second dilated convolutional layer extracts features from the first common convolutional layer after dimension reduction received from the second common convolutional layer The third-scale convolutional network branch in the multi-scale convolutional network branch extracts the visual observation image features to obtain the third-scale visual observation image features, wherein the third ordinary convolutional layer reduces the dimension of the visual observation image features received from the convolutional backbone network in the channel dimension to obtain the third visual observation image features after the dimension reduction, and sends the third visual observation image features after the dimension reduction to the third expanded convolutional layer; the third expanded convolutional layer extracts the features of the third visual observation image after the dimension reduction received from the third ordinary convolutional layer to obtain the third-scale visual observation image features, and sends the third-scale visual observation image features to the adaptive fusion module; the fourth ordinary convolutional layer in the fourth-scale convolutional network branch in the multi-scale convolutional network branch reduces the dimension and extracts the features of the visual observation image received from the convolutional backbone network in the channel dimension to obtain the fourth-scale visual observation image features, and sends the fourth-scale visual observation image features to the adaptive fusion module;

[0321] 2.3.10.3 The convolutional long short-term memory unit in the scale attention module extracts temporal features from the visual observation image features received from the convolutional backbone network to obtain temporal features of the visual observation image, and sends the temporal features of the visual observation image to the global average pooling layer; the global average pooling layer compresses the temporal features of the visual observation image received from the convolutional long short-term memory unit in the spatial dimension to obtain a first feature vector whose length is equal to the number of channels of the temporal features of the visual observation image, and sends the first feature vector to the seventh fully connected layer; the seventh fully connected layer compresses the first feature vector received from the global average pooling layer to obtain a second feature vector whose length is equal to the number of branches of the multi-scale convolutional network (4), and sends the second feature vector to the sigmoid activation function layer; the sigmoid activation function layer scales the value of the second feature vector received from the seventh fully connected layer to the range of [0,1] to obtain the weights of the multi-scale visual observation image features, and sends the weights of the multi-scale visual observation image features to the adaptive fusion module.

[0322] 2.3.10.4 The weighted splicing layer in the adaptive fusion module uses the weights of the multi-scale visual observation image features received from the scale attention module to weightedly splice the first-scale visual observation image features, second-scale visual observation image features, third-scale visual observation image features, and fourth-scale visual observation image features received from the multi-scale convolutional network branch to obtain weighted spliced ​​multi-scale visual observation image features, and send the weighted spliced ​​multi-scale visual observation image features to the fifth ordinary convolutional layer; the fifth ordinary convolutional layer performs dimensionality reduction on the weighted spliced ​​multi-scale visual observation image features received from the weighted splicing layer in the channel dimension to obtain adaptively fused multi-scale visual observation image features, and send the adaptively fused multi-scale visual observation image features to the recurrent attention module.

[0323] 2.3.10.5 The recurrent attention module extracts and weights the temporal features of the adaptively fused multi-scale visual observation image features received from the adaptive fusion module to obtain adaptive temporal scale features, and sends the adaptive temporal scale features to the eighth fully connected layer.

[0324] 2.3.10.6 The eighth fully connected layer performs action prediction on the adaptive temporal scale features received from the recurrent attention module to obtain the tracker’s action probability π 1* (a t |o′ t ).

[0325] 2.3.11 Active Visual Tracking The eighth fully connected layer of the visual tracker network in the student module converts the tracker's action probability π 1* (a t |o′ t) is sent to the second loss function calculation module and tracker.

[0326] 2.3.12 The second loss function calculation module in the active visual tracking student module is based on the tracker action probability π received from the visual tracker network 1* (a t |o′ t ) and the tracker’s action probability π received from the meta-tracker network in the active visual tracking teacher module 1 (a t |s t ) Calculate the KL divergence loss D KL (π 1* (a t t|o′ t )|π 1 (a t |s t )):

[0327]

[0328] 2.3.13 Active visual tracking student module uses Adam optimization algorithm to minimize D KL (π 1* (a t |o′ t )|π 1 (a t |s t )) to update the parameters of the visual tracker network.

[0329] 2.3.14 Tracker According to the tracker's action probability π 1* (a t |o′ t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the target is based on the target’s action probability π 2 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking an action to interact with the environment. At the same time, the interference object is based on the action probability π of the interference object. 3 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking the action to interact with the environment.

[0330] 2.3.15 Let t = t + 1. If t ≤ T, go to 2.3.16; if t > T, go to 2.3.17.

[0331] 2.3.16 If t%5000=0, go to 2.3.4 to train the active visual tracking teacher module loaded with the parameters of the new meta-target network and meta-distractor network; otherwise go to 2.3.6 and continue training using the data of the next time step.

[0332] 2.3.17 Save the parameters of the visual tracker network and go to step 3.

[0333] In the third step, the active visual tracking student module of the scale-adaptive active visual tracking system loads the parameters of the visual tracker network saved in step 2.3.17 to obtain the trained scale-adaptive active visual tracking system.

[0334] In the fourth step, the trained scale-adaptive active visual tracking system tracks the target by:

[0335] 4.1 The data acquisition module obtains the real-time visual observation image o from the tracker's camera and sends o to the data preprocessing module.

[0336] 4.2 The data preprocessing module receives the real-time visual observation image o, and uses the second preprocessing method described in step 2.3.8 to preprocess the real-time visual observation image o to obtain the preprocessed real-time visual observation image o′, and sends o′ to the active visual tracking student module.

[0337] 4.3 Active visual tracking The student module receives the preprocessed real-time visual observation image o′, and the visual tracker network uses the action prediction method described in step 2.3.10 to predict the action of the preprocessed real-time visual observation image o′ to obtain the tracker's action probability π 1* (a′|o′). a′ is a random variable representing the action of the agent, and its value range is the action range of the agent.

[0338] 4.4 Visual Tracker Network The tracker action probability π 1* (a′|o′) is sent to the tracker.

[0339] 4.5 Tracker According to the tracker's action probability π 1* The conditional probability represented by (a′|o′) selects an action from the 7 actions in the action range and takes the action to track the target.

[0340] In order to verify the effect of the above embodiment, the following experiment is performed:

[0341] Three simulation environments based on Unreal Engine, namely Flexible Room, Urban City and Parking Lot, are used to train and test the scale-adaptive active visual tracking system constructed in the first step of the present invention. Flexible Room is a simple room environment for training and testing. The speed of the opponent (target and distractor) is adjusted in different training stages to produce states of different scales. Urban City and Parking Lot are two high-simulation environments used to test the generalization ability of the tracker in unfamiliar environments. Urban City is an urban street environment, and Parking Lot is an underground parking lot environment. In the testing phase, a rule-based navigator is used to control the target and distractor so that the target and distractor move to a randomly selected destination along the nearest route. The maximum speed of the opponent (target and distractor) is set to 1.2 times the maximum speed of the tracker, thereby producing scale changes.

[0342] The evaluation index of active visual tracking is an important criterion for measuring the tracking performance of the tracker. The evaluation indexes used in the existing active visual tracking methods based on deep reinforcement learning (see Luo W, Sun P, Zhong F, et al. End-to-end active object tracking via reinforcement learning [C]. International conference on machine learning. 2018: 3286–3295., Zhong F, Sun P, Luo W, et al. AD-VAT: Anasymmetric dueling mechanism for learning visual active tracking [C]. International Conference on Learning Representations. 2018., Zhong F, Sun P, Luo W, et al. Towards distraction-robust active visual tracking [C]. International Conference on Machine Learning. 2021: 12782–12792.) are followed, including Accumulated Reward (AR), Episode Length (EL), and Success Rate (SR). One scene is a tracking process. The maximum step length of one scene is 500 steps. The maximum reward that the tracker can obtain at each step is 1, so the maximum values ​​of the cumulative reward AR and the scene length EL are both 500. When the target is lost from the tracker's field of view for more than 5 seconds or reaches the maximum scene length, an scene ends. The tracking success rate is the proportion of scenes that reach the maximum scene length (500 steps) in 100 scenes of tracking. The cumulative reward is related to the reward obtained by the tracker at each step and the tracking time (scene length), which is used to evaluate the overall performance and is the main evaluation indicator. The scene length reflects the long-term tracking capability of the tracker. The tracking success rate is used to evaluate the robustness of the tracker. The higher the three indicators, the better. The reward calculation method of the tracker in the present invention is the same as the comparison method, so the cumulative reward AR can reflect the tracking performance of the scale-adaptive active visual tracking method proposed in the present invention compared with the existing method.

[0343] The experiment uses Flexible Room as the training environment and tests the tracking performance of the tracker in three environments: Flexible Room, Urban City, and ParkingLot. The environment is named Nav-x according to the number of distractors x in the environment. The degree of scale interference is controlled by adjusting the number of distractors, and the robustness of the tracker to interference and scale changes is analyzed. In order to verify the effectiveness of the scale-adaptive active visual tracking method proposed in the present invention, the present invention is compared with the two-stage method described in the background technology and the end-to-end method based on deep reinforcement learning. The two-stage method is implemented by a passive visual tracking algorithm and a controller. The passive visual tracking algorithms used in the two-stage method in the comparison method of this experiment include DaSiamRPN (see Zhu Z, Wang Q, Li B, et al. Distractor-aware siamese networks for visual object tracking [C]. Proceedings of the European conference on computer vision (ECCV). 2018: 101-117.), ATOM (see Danelljan M, Bhat G, Khan FS, et al. Atom: Accurate tracking by overlap maximization [C]. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 4660-4669.), DiMP (see Bhat G, Danelljan M, Gool LV, et al. Learning discriminative model prediction for tracking[C].Proceedings of the IEEE / CVF internationalconference on computer vision.2019:6182-6191.), the controller used is PID controller.The end-to-end methods used in this experiment include SARL (see Luo W, Sun P, Zhong F, et al. End-to-end active object tracking via reinforcement learning [C]. International conference on machine learning. 2018: 3286-3295.), SARL+ (see LUO W, SUN P, ZHONG F, et al. End-to-end active object tracking and its real-world deployment via reinforcement learning [J]. IEEE transactions on pattern analysis and machine intelligence, 2019, 42 (6): 1317-1332.), AD-VAT (see Zhong F, Sun P, Luo W, et al. AD-VAT: An asymmetric dueling mechanism for learning visual active tracking [C]. International Conference on Learning Representations. 2018.), and AD-VAT+ (see Zhong F, Sun P, Luo W, et al. Ad-vat+: An asymmetric dueling mechanism for learning and understanding visual active tracking [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 42 (6): 1317-1332.). transactions onpattern analysis and machine intelligence.2019,43(5):1467-1482.), TDR-AVT (see the literature Zhong F, Sun P, Luo W, et al. Towards distraction-robust active visual tracking[C].International Conference on Machine Learning.2021:12782-12792.).

[0344] In order to verify the robustness of the present invention to interference and scale changes, the test was first carried out in a Flexible Room environment containing 1, 2, 3, and 4 interferers, respectively. The experimental results are shown in Table 1. The maximum speed of the target and the interferer is 1.2 times that of the tracker, so the scale change will occur randomly. As can be seen from Table 1, after the scale-adaptive active visual tracking system constructed in the first step of the present invention was trained in the Flexible Room environment, it significantly exceeded all the comparison methods in terms of the three indicators of cumulative reward AR, curtain length EL, and tracking success rate SR, which fully proves that the system uses convolutional network branches with multiple different receptive field sizes to extract the features of objects of multiple different scales. At the same time, based on the scale attention mechanism, it can adaptively and selectively extract the features of the target, so it can not only effectively combat interference, but also adaptively pay attention to the scale changes of the target, thereby significantly improving the tracking performance of the tracker.

[0345] In order to verify the generalization ability of the present invention in a real environment, experiments were conducted in two high-simulation environments, Urban City and Parking Lot, and the experimental results are shown in Table 2. As can be seen from Table 2, the scale-adaptive active visual tracking system of the present invention can still obtain a high cumulative reward in a high-simulation environment, indicating that the system can maintain accurate tracking in a complex environment. The present invention can always maintain a high tracking time, has good long-term tracking ability, and has a high tracking success rate, and has good generalization ability in unknown environments.

[0346] Table 1 Test results of different numbers of distractors (x=1 or 2 or 3 or 4) in the Flexible Room environment

[0347]

[0348]

[0349] Table 2 Test results in a high-simulation environment (number of interferers x = 2 or 4)

[0350]

[0351] Through the above embodiments, it is proved that the scale-adaptive active visual tracking method described in the present invention is more effective in improving the tracking performance of the active tracker than the two-stage method described in the background art and the end-to-end method based on deep reinforcement learning.

[0352] Finally, it should be pointed out that the above content is only used to illustrate the technical solution of the present invention and does not constitute a limitation. Those skilled in the art should understand that the specific implementation methods can be modified or replaced by equivalents without departing from the spirit and scope of the present invention, and these modifications or replacements should be included in the scope of protection of the claims of the present invention.

Claims

1. A scale-adaptive active visual tracking method, characterized in that The following steps are involved: The first step is to build a scale-adaptive active visual tracking system; The scale-adaptive active visual tracking system is composed of a data acquisition module, a data preprocessing module, an active visual tracking teacher module, and an active visual tracking student module; The data acquisition module is connected to the camera of the tracker, the posture sensors of all intelligent agents, namely the tracker, the target, and the interferer, and the data preprocessing module, and acquires a two-dimensional visual observation image from the camera of the tracker, and acquires the posture data of the tracker, the target, and the interferer from the posture sensors of the tracker, the target, and the interferer; during training, the data acquisition module sends the visual observation image acquired from the camera of the tracker and the posture data acquired from the posture sensors of the tracker, the target, and the interferer to the data preprocessing module; When tracking a target, the data acquisition module only sends the visual observation image acquired from the tracker's camera to the data preprocessing module; The data preprocessing module is connected to the data acquisition module, the active visual tracking teacher module, and the active visual tracking student module, receives the visual observation image and posture data from the data acquisition module, preprocesses the visual observation image and posture data, and obtains the preprocessed visual observation image and the preprocessed posture data; during training, the data preprocessing module sends the preprocessed posture data to the active visual tracking teacher module, and sends the preprocessed visual observation image to the active visual tracking student module; when tracking the target, the data preprocessing module only sends the preprocessed visual observation image to the active visual tracking student module; The function of the active visual tracking teacher module is to assist the active visual tracking student module. It only works during training and does not participate in the work when tracking the target. The active visual tracking teacher module is connected to the data preprocessing module and the active visual tracking student module, and is composed of a meta-tracker network, a meta-target network, a meta-disturbance network, a reward function calculation module, and a first loss function calculation module; the meta-tracker network is composed of a first bidirectional gated recurrent unit, a first long short-term memory unit, a first fully connected layer, and a second fully connected layer; the first bidirectional gated recurrent unit encodes the preprocessed posture data received from the data preprocessing module to obtain the posture features of the tracker, and sends the posture features of the tracker to the first long short-term memory unit; the first long short-term memory unit extracts the temporal features of the posture features of the tracker received from the first bidirectional gated recurrent unit to obtain the temporal posture features of the tracker, and sends the temporal posture features of the tracker to the first fully connected layer and the second fully connected layer; the first fully connected layer performs action prediction on the temporal posture features of the tracker received from the first long short-term memory unit to obtain the action probability of the tracker. When training the active visual tracking teacher module, the first fully connected layer sends the action probability of the tracker to the meta-target network, the meta-interference network, and the first loss function calculation module. When training the active visual tracking student module, the first fully connected layer sends the action probability of the tracker to the meta-target network, the meta-interference network, and the active visual tracking student module; the second fully connected layer performs value estimation on the temporal posture features of the tracker received from the first long short-term memory unit to obtain the state value of the tracker. When training the active visual tracking teacher module, the second fully connected layer sends the state value of the tracker to the first loss function calculation module; The meta-target network is composed of a second bidirectional gated recurrent unit, a second long short-term memory unit, a third fully connected layer, and a fourth fully connected layer; the second bidirectional gated recurrent unit encodes the preprocessed posture data received from the data preprocessing module to obtain the posture features of the target, and sends the posture features of the target to the second long short-term memory unit; the second long short-term memory unit extracts the temporal features of the posture features of the target received from the second bidirectional gated recurrent unit and the action probability of the tracker received from the meta-tracker network to obtain the temporal posture features of the target, and sends the temporal posture features of the target to the third fully connected layer and the fourth fully connected layer; the third fully connected layer performs action prediction on the temporal posture features of the target received from the second long short-term memory unit to obtain the action probability of the target. When training the active visual tracking teacher module, the third fully connected layer sends the action probability of the target to the first loss function calculation module; the fourth fully connected layer estimates the value of the temporal posture features of the target received from the second long short-term memory unit to obtain the state value of the target. When training the active visual tracking teacher module, the fourth fully connected layer sends the state value of the target to the first loss function calculation module; The meta-interference network is composed of a third bidirectional gated recurrent unit, a third long short-term memory unit, a fifth fully connected layer, and a sixth fully connected layer; the third bidirectional gated recurrent unit encodes the preprocessed posture data received from the data preprocessing module to obtain the posture features of the interference, and sends the posture features of the interference to the third long short-term memory unit; the third long short-term memory unit extracts the temporal features of the posture features of the interference received from the third bidirectional gated recurrent unit and the action probability of the tracker received from the meta-tracker network to obtain the temporal posture features of the interference, and sends the temporal posture features of the interference to the fifth fully connected layer and the sixth fully connected layer; the fifth fully connected layer performs action prediction on the temporal posture features of the interference received from the third long short-term memory unit to obtain the action probability of the interference. When training the active visual tracking teacher module, the fifth fully connected layer sends the action probability of the interference to the first loss function calculation module; The sixth fully connected layer estimates the value of the temporal position characteristics of the distractor received from the third long short-term memory unit to obtain the state value of the distractor. When training the active visual tracking teacher module, the sixth fully connected layer sends the state value of the distractor to the first loss function calculation module; The reward function calculation module is connected to the data preprocessing module and the first loss function calculation module; when training the active visual tracking teacher module, the reward function calculation module calculates the rewards of the tracker, the target, and the distractor according to the preprocessed posture data received from the data preprocessing module, and sends the rewards of the tracker, the target, and the distractor to the first loss function calculation module; The first loss function calculation module is connected to the reward function calculation module, the meta-tracker network, the meta-target network, and the meta-interference network; when training the active visual tracking teacher module, the first loss function calculation module receives the rewards of the tracker, the target, and the interference from the reward function calculation module, receives the action probability of the tracker and the state value of the tracker from the meta-tracker network, receives the action probability of the target and the state value of the target from the meta-target network, receives the action probability of the interferer and the state value of the interferer from the meta-interference network, calculates the temporal difference error loss and policy gradient loss of the tracker according to the reward, action probability, and state value of the tracker, calculates the temporal difference error loss and policy gradient loss of the target according to the reward, action probability, and state value of the target, and calculates the temporal difference error loss and policy gradient loss of the interferer according to the reward, action probability, and state value of the interferer; when training the active visual tracking teacher module, the active visual tracking teacher module optimizes the meta-tracker network, the meta-target network, and the meta-interference network by minimizing the temporal difference error loss and policy gradient loss of the tracker, the target, and the interference obtained by the first loss function calculation module; The active visual tracking student module is connected to the data preprocessing module and the active visual tracking teacher module, and is composed of a visual tracker network and a second loss function calculation module; the visual tracker network is composed of a convolutional backbone network, a multi-scale convolutional network branch, a scale attention module, an adaptive fusion module, a recurrent attention module, and an eighth fully connected layer; The convolution backbone network is connected to the data preprocessing module, the multi-scale convolution network branch and the scale attention module, performs feature extraction on the preprocessed visual observation image received from the data preprocessing module, obtains visual observation image features, and sends the visual observation image features to the multi-scale convolution network branch and the scale attention module; The number of branches of the multi-scale convolutional network branch is 4, which is composed of a first-scale convolutional network branch, a second-scale convolutional network branch, a third-scale convolutional network branch, and a fourth-scale convolutional network branch; the first-scale convolutional network branch performs feature extraction on the visual observation image features to obtain first-scale visual observation image features, and sends the first-scale visual observation image features to the adaptive fusion module; the second-scale convolutional network branch performs feature extraction on the visual observation image features to obtain second-scale visual observation image features, and sends the second-scale visual observation image features to the adaptive fusion module; the third-scale convolutional network branch performs feature extraction on the visual observation image features to obtain third-scale visual observation image features, and sends the third-scale visual observation image features to the adaptive fusion module; the fourth-scale convolutional network branch performs dimensionality reduction and feature extraction on the visual observation image features received from the convolutional backbone network to obtain fourth-scale visual observation image features, and sends the fourth-scale visual observation image features to the adaptive fusion module; The scale attention module consists of a convolutional long short-term memory unit, a global average pooling layer, a seventh fully connected layer, and a sigmoid activation function layer; the convolutional long short-term memory unit extracts temporal features of the visual observation image features received from the convolutional backbone network to obtain temporal features of the visual observation image, and sends the temporal features of the visual observation image to the global average pooling layer; the global average pooling layer compresses the temporal features of the visual observation image received from the convolutional long short-term memory unit in the spatial dimension to obtain a first feature vector whose length is equal to the number of channels of the temporal features of the visual observation image, and sends the first feature vector to the seventh fully connected layer; the seventh fully connected layer compresses the first feature vector received from the global average pooling layer to obtain a second feature vector whose length is equal to the number of branches of the multi-scale convolutional network, and sends the second feature vector to the sigmoid activation function layer; the sigmoid activation function layer scales the value of the second feature vector received from the seventh fully connected layer to the range of [0,1] to obtain the weights of the multi-scale visual observation image features, and sends the weights of the multi-scale visual observation image features to the adaptive fusion module; The adaptive fusion module is connected to the multi-scale convolutional network branch, the scale attention module, and the recurrent attention module, and is composed of a weighted splicing layer and a fifth ordinary convolutional layer; the weighted splicing layer uses the weights of the multi-scale visual observation image features received from the scale attention module to weightedly splice the first-scale visual observation image features, the second-scale visual observation image features, the third-scale visual observation image features, and the fourth-scale visual observation image features received from the multi-scale convolutional network branch to obtain the weighted spliced ​​multi-scale visual observation image features, and sends the weighted spliced ​​multi-scale visual observation image features to the fifth ordinary convolutional layer; the fifth ordinary convolutional layer performs dimensionality reduction on the weighted spliced ​​multi-scale visual observation image features received from the weighted splicing layer in the channel dimension to obtain adaptively fused multi-scale visual observation image features, and sends the adaptively fused multi-scale visual observation image features to the recurrent attention module; The recurrent attention module is connected to the adaptive fusion module and the eighth fully connected layer, performs temporal feature extraction and weighting on the adaptively fused multi-scale visual observation image features received from the adaptive fusion module, obtains adaptive temporal scale features, and sends the adaptive temporal scale features to the eighth fully connected layer; The eighth fully connected layer performs action prediction on the adaptive temporal scale features received from the recurrent attention module to obtain the action probability of the tracker. When training the active visual tracking student module, the eighth fully connected layer sends the action probability of the tracker to the second loss function calculation module; The second loss function calculation module is connected to the eighth fully connected layer and the meta tracker network. When training the active visual tracking student module, the second loss function calculation module calculates the KL divergence loss for the action probability of the tracker received from the eighth fully connected layer and the action probability of the tracker received from the meta tracker network of the active visual tracking teacher module; when training the active visual tracking student module, the active visual tracking student module optimizes the visual tracker network by minimizing the KL divergence loss obtained by the second loss function calculation module; The second step is to train the active visual tracking teacher module and the active visual tracking student module in the scale-adaptive active visual tracking system; the training method of the active visual tracking teacher module is a reinforcement learning method, and the training method of the active visual tracking student module is a supervised learning method. The specific method is: 2.1 The active visual tracking teacher module is trained by using reinforcement learning. The method is to use adversarial reinforcement learning to perform multi-scale and multi-stage training on the meta-tracker network, meta-target network and meta-distractor network during the interaction between the tracker, target, distractor and environment. The reward functions of the target and distractor in each stage are different, corresponding to different scales of training, specifically: 2.1.1 The first phase of regular scale training of the active tracking teacher module is carried out by using reinforcement learning. The method is: 2.1.1.1 Set the number of training steps T to a positive integer greater than 100,000, and initialize the number of trained steps t = 1; 2.1.1.2 Set the target position entropy reward coefficient a to a constant greater than 0.5; set the regularization factor α to a constant less than 1; set the reward attenuation factor γ to a constant greater than 0.5; set the number of environmental horizontal grid divisions n to a positive integer; 2.1.1.3 Set the number of steps used to calculate the target position entropy reward N = min(t, 50); 2.1.1.4 The data acquisition module obtains the pose data of the tth step from the pose sensors of the tracker, target, and interference object, and sends the pose data of the tth step to the data preprocessing module; the pose data of the tth step includes the position coordinates of the tracker and orientation angle The target's location coordinates and orientation angle The location coordinates of the distractor and orientation angle 2.1.1.5 The data preprocessing module uses the first preprocessing method to preprocess the posture data of the tth step received from the data acquisition module to obtain the preprocessed posture data of the tth step, and sends the preprocessed posture data of the tth step to the active visual tracking teacher module; the first preprocessing method is: 2.1.1.5.1 Calculate the relative distance between the target's actual position and the tracker Relative Angle Relative orientation angle The relative distance of the actual position of the interference object relative to the tracker Relative Angle Relative orientation angle 2.1.1.5.2 Set the relative distance ρ of the target ideal position relative to the tracker * , relative angle θ * =0, relative orientation angle ω * = 0, indicating that the ideal position of the target is located in front of the tracker at a distance ρ * The maximum distance ρ of the tracker's field of view is set to max and the maximum angle θ max ; 2.1.1.5.3 Calculate the normalized relative distance of the target's actual position Normalized relative distance to the actual position of the distractor The relative angle of the normalized representation of the actual position of the target The relative orientation angle of the normalized representation of the actual position of the target The relative angle of the normalized representation of the actual position of the distractor The relative orientation angle of the normalized representation of the actual position of the distractor 2.1.1.5.4 Set the tracker's position coordinates The target's location coordinates The location coordinates of the distractor The relative distance to the actual position of the target Relative Angle Relative orientation angle The relative distance of the actual location of the distractor Relative Angle Relative orientation angle Normalized relative distance to the target's actual position The relative angle of the normalized representation of the actual position of the target The relative orientation angle of the normalized representation of the actual position of the target Normalized relative distance to the actual position of the distractor The relative angle of the normalized representation of the actual position of the distractor The relative orientation angle of the normalized representation of the actual position of the distractor The preprocessed pose data that constitutes the tth step; 2.1.1.6 The reward function calculation module in the active visual tracking teacher module calculates the tracker’s reward based on the preprocessed pose data of step t 2.1.1.7 The reward function calculation module in the active visual tracking teacher module divides the training environment into I grids, I = n × n, and uses the frequency calculation method to calculate the set of target position coordinates in the past N steps based on the preprocessed pose data in the past t steps. The frequencies p1,…,p that appear in each grid i ,…,p I ; 2.1.1.8 The reward function calculation module in the active vision teacher module calculates the position entropy reward loc_entropy of the target: 2.1.1.9 The reward function calculation module in the active visual tracking teacher module calculates the reward of the target based on the preprocessed pose data of step t 2.1.1.10 The reward function calculation module in the active visual tracking teacher module calculates the reward of the distractor based on the preprocessed pose data of step t 2.1.1.11 The reward function calculation module in the active visual tracking teacher module converts the tracker's reward Rewards for goals Distractor Rewards Send to the first loss function calculation module in the active visual tracking teacher module; 2.1.1.12 The meta-tracker network, meta-target network, and meta-disturbance network in the active visual tracking teacher module use estimation methods to perform action prediction and value estimation on the preprocessed pose data, respectively, to obtain the state value V of the tracker. 1 (s t ) and the tracker’s action probability π 1 (a t |s t ), the state value V of the target 2 (s t ) and the target action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ) and the action probability π of the distractor 3 (a t |s t ), the method is: 2.1.1.12.1 Let the normalized relative distance of the actual position of the target in the preprocessed pose data of step t be Normalized relative angle Normalized relative orientation angle Normalized relative distance to the actual position of the distractor Normalized relative angle Normalized relative orientation angle The environmental state that constitutes step t 2.1.1.12.2 The First Bidirectional Gated Recurrent Unit Pair in the Meta-Tracker Network t Encode to obtain the posture features of the tracker, and send the posture features of the tracker to the first long short-term memory unit; the first long short-term memory unit extracts the temporal features of the posture features of the tracker received from the first bidirectional gated recurrent unit to obtain the temporal posture features of the tracker, and sends the temporal posture features of the tracker to the first fully connected layer and the second fully connected layer; the first fully connected layer predicts the action of the temporal posture features of the tracker received from the first long short-term memory unit to obtain the action probability π of the tracker 1 (a t |s t ), the first fully connected layer sends the tracker’s action probability to the meta-target network and the meta-interference network; The second fully connected layer estimates the value of the temporal pose features of the tracker received from the first long short-term memory unit to obtain the state value V of the tracker 1 (s t ); 2.1.1.12.3 Second Bidirectional Gated Recurrent Unit Pair s in Meta-Target Network t Encode to obtain the target's posture features, and send the target's posture features to the second long short-term memory unit; the second long short-term memory unit encodes the target's posture features received from the second bidirectional gated recurrent unit and the tracker's action probability π received from the meta-tracker network 1 (a t |s t ) extracts temporal features to obtain the temporal position and posture features of the target, and sends the temporal position and posture features of the target to the third fully connected layer and the fourth fully connected layer; the third fully connected layer performs action prediction on the temporal position and posture features of the target received from the second long short-term memory unit to obtain the action probability π of the target 2 (a t |s t ); The fourth fully connected layer estimates the value of the temporal position features of the target received from the second long short-term memory unit to obtain the state value V of the target 2 (s t ); 2.1.1.12.4 The third bidirectional gated recurrent unit pair s in the perturbation network t Encode to obtain the pose feature of the interferer, and send the pose feature of the interferer to the third long short-term memory unit; the third long short-term memory unit encodes the pose feature of the interferer received from the third bidirectional gated recurrent unit and the action probability π of the tracker received from the meta-tracker network 1 (a t |s t ) to extract temporal features, obtain temporal position features of the interferer, and send the temporal position features of the interferer to the fifth fully connected layer and the sixth fully connected layer; the fifth fully connected layer performs action prediction on the temporal position features of the interferer received from the third long short-term memory unit to obtain the action probability π of the interferer 3 (a t |s t ); The sixth fully connected layer estimates the value of the temporal position features of the interference object received from the third long short-term memory unit to obtain the state value V of the interference object 3 (s t );where a t is a random variable representing the action of the agent, and its value range is the action range of the agent, π k (a t |s t ) is a conditional probability, indicating that in state s t The probability distribution of actions in the action range of the agent, k∈{1,2,3}, 1 represents the tracker, 2 represents the target, and 3 represents the distractor; 2.1.1.13 The meta-tracker network in the active visual tracking teacher module converts the state value V of the tracker 1 (s t ) and action probability π 1 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the tracker is 1 (a t |s t ) is sent to the tracker; the meta-target network sends the target state value V 2 (s t ) and action probability π 2 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the target action probability π 2 (a t |s t ) is sent to the target; the meta-interference network sends the state value V of the distractor 3 (s t ) and action probability π 3 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the interference object is 3 (a t |s t ) is sent to the interferer; 2.1.1.14If t=1, go to 2.1.1.16; Otherwise: The first loss function calculation module in the active visual tracking teacher module calculates the reward r of the tracker received from the reward function calculation module t 1 , the reward of the target t 2 , Distractor Rewards and the state value V of the tracker received from the meta-tracker network, meta-target network, and meta-distractor network 1 (s t ), the action probability π of the tracker 1 (a t |s t ), the state value V of the target 2 (s t ), the target action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ), the action probability of the distractor π 3 (a t |s t ) Calculate the time difference error loss of the tracker Policy gradient loss for tracker Target temporal difference error loss Policy gradient loss of the target Temporal Differential Error Loss of Interferers Policy Gradient Loss for Distractors where logπ k (a t |s t ) is π k (a t |s t ), H(π k (·|s t )) is the entropy regularization term, 2.1.1.15 Active visual tracking teacher module uses Adam optimization algorithm to minimize and To update the parameters of the meta-tracker network, use the Adam optimization algorithm to minimize and To update the parameters of the meta-target network, the Adam optimization algorithm is used to minimize and To update the parameters of the meta-disturbant network; 2.1.1.16 Tracker According to the tracker's action probability π 1 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking actions to interact with the environment; at the same time, the target is based on the action probability π 2 (a t |s t ) represents the conditional probability of selecting an action from the action range and taking an action to interact with the environment; at the same time, the interferer is based on the action probability π of the interferer. 3 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking the action to interact with the environment; 2.1.1.17 Let t=t+1; if t≤T, go to 2.1.1.18; if t>T, go to 2.1.1.19; 2.1.1.18 If t%10000=0: Save the parameters of the meta-target network and the meta-interference network to the opponent model pool, go to 2.1.1.3, and continue training using the data of the next time step; Otherwise: Go directly to 2.1.1.3 and continue training using the data of the next time step; 2.1.1.19 Save the parameters of the meta-tracker network, meta-target network, and meta-interferer network, and go to 2.1.2; 2.1.2 Based on the first stage training, the active visual tracking teacher module is further trained on a small scale in the second stage using reinforcement learning. The method is: 2.1.2.1 Meta-tracker network, meta-target network, meta-distractor network Load the parameters of the meta-tracker network, meta-target network, and meta-distractor network saved at the end of the first stage of training in 2.1.1.19; 2.1.2.2 Set the number of training steps T to a positive integer greater than 100,000, and initialize the number of trained steps t = 1; 2.1.2.3 Set the target position entropy reward coefficient a to a constant greater than 0.5; set the regularization factor α to a constant less than 1; set the reward attenuation factor γ to a constant greater than 0.5; set the number of environmental horizontal grid divisions n to a positive integer; set the threshold distance ρ for the target to be away from the tracker s 100~1000cm; 2.1.2.4 Set the number of steps used to calculate the target position entropy reward N = min(t, 50); 2.1.2.5 The data acquisition module obtains the posture data of the tth step from the posture sensors of the tracker, target, and interference object, and sends the posture data of the tth step to the data preprocessing module; 2.1.2.6 The data preprocessing module adopts the first preprocessing method in 2.1.1.5 to preprocess the posture data of the tth step received from the data acquisition module to obtain the preprocessed posture data of the tth step, and sends the preprocessed posture data of the tth step to the active visual tracking teacher module; 2.1.2.7 The reward function calculation module in the active visual tracking teacher module calculates the reward of the tracker according to the formula (7) in 2.1.1.6 based on the preprocessed pose data of step t 2.1.2.8 The reward function calculation module in the active visual tracking teacher module divides the training environment into I grids, and uses the frequency calculation method described in 2.1.1.7 to calculate the frequency p1,…,p of the set of target position coordinates in the past N steps in each grid based on the preprocessed pose data of the past t steps. i ,…,p I ; 2.1.2.9 The reward function calculation module in the active vision teacher module calculates the position entropy reward loc_entropy of the target according to formula (9) in 2.1.1.8; 2.1.2.10 The reward function calculation module in the active visual tracking teacher module calculates the reward of the target based on the preprocessed pose data of step t 2.1.2.11 The reward function calculation module in the active visual tracking teacher module calculates the reward of the distractor based on the preprocessed pose data of step t 2.1.2.12 The reward function calculation module in the active visual tracking teacher module converts the tracker's reward Rewards for goals Distractor Rewards Send to the first loss function calculation module in the active visual tracking teacher module; 2.1.2.13 The meta-tracker network, meta-target network, and meta-interference network in the active visual tracking teacher module use the estimation method described in 2.1.1.12 to perform action prediction and value estimation on the preprocessed pose data, respectively, to obtain the state value V of the tracker 1 (s t ) and the tracker’s action probability π 1 (a t |s t ), the state value V of the target 2 (s t ) and the target action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ) and the action probability π of the distractor 3 (a t |s t ); 2.1.2.14 The meta-tracker network in the active visual tracking teacher module converts the state value V of the tracker 1 (s t ) and action probability π 1 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the tracker is 1 (a t |s t ) is sent to the tracker; the meta-target network sends the target state value V 2 (s t ) and action probability π 2 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the target action probability π 2 (a t |s t ) is sent to the target; the meta-interference network sends the state value V of the distractor 3 (s t ) and action probability π 3 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the interference object is 3 (a t |s t ) is sent to the interferer; 2.1.2.15 If t = 1, go to 2.1.2.17; otherwise, the first loss function calculation module in the active visual tracking teacher module calculates the reward of the tracker received from the reward function calculation module according to formula (12) and formula (13) in 2.1.1.

14. Rewards for goals Distractor reward r t 3 and the state value V of the tracker received from the meta-tracker network, meta-target network, and meta-distractor network 1 (s t ), the action probability π of the tracker 1 (a t |s t ), the state value V of the target 2 (s t ), the target action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ), the action probability of the distractor π 3 (a t |s t ) Calculate the time difference error loss of the tracker Policy gradient loss for tracker Target temporal difference error loss Policy gradient loss of the target Temporal Differential Error Loss of Interferers Policy Gradient Loss for Distractors 2.1.2.16 Active visual tracking teacher module uses Adam optimization algorithm to minimize and To update the parameters of the meta-tracker network, use the Adam optimization algorithm to minimize and To update the parameters of the meta-target network, the Adam optimization algorithm is used to minimize and To update the parameters of the meta-disturbant network; 2.1.2.17 Tracker According to the tracker's action probability π 1 (a t |s t ) represents the conditional probability of selecting an action from the action range and taking an action to interact with the environment; at the same time, the target is based on the target’s action probability π 2 (a t |s t ) represents the conditional probability of selecting an action from the action range and taking an action to interact with the environment; at the same time, the interferer is based on the action probability π of the interferer. 3 (a t |s t ) represents the conditional probability of selecting an action from the action range and taking the action to interact with the environment; 2.1.2.18 Let t = t + 1. If t ≤ T, go to 2.1.2.

19. If t > T, go to 2.1.2.

20. 2.1.2.19 If t%10000=0: save the parameters of the meta-target network and the meta-interference network to the opponent model pool, and go to 2.1.2.4 to continue training using the data of the next time step; Otherwise: Go directly to 2.1.2.4 and continue training with the data of the next time step; 2.1.2.20 Save the parameters of the meta-tracker network, meta-target network, and meta-distractor network, and go to 2.1.3; 2.1.3 Based on the training in the second stage, the active visual tracking teacher module is further trained on a large scale in the third stage using reinforcement learning; 2.1.3.1 Meta-tracker network, meta-target network, meta-distractor network Load the parameters of the meta-tracker network, meta-target network, and meta-distractor network saved at the end of the second stage of training in 2.1.2.20; 2.1.3.2 Set the number of training steps T to a positive integer greater than 100,000, and initialize the number of trained steps t = 1; 2.1.3.3 Set the target position entropy reward coefficient a to a constant greater than 0.5; set the regularization factor α to a constant less than 1; set the reward attenuation factor γ to a constant greater than 0.5; set the number of environmental horizontal grid divisions n to a positive integer; the threshold distance ρ of the target close to the tracker l 100~1000cm; 2.1.3.4 Set the number of steps used to calculate the target position entropy reward N = min(t, 50); 2.1.3.5 The data acquisition module obtains the posture data of the tth step from the posture sensors of the tracker, target, and interference object, and sends the posture data of the tth step to the data preprocessing module; 2.1.3.6 The data preprocessing module uses the first preprocessing method in 2.1.1.5 to preprocess the posture data of the tth step received from the data acquisition module to obtain the preprocessed posture data of the tth step, and sends the preprocessed posture data of the tth step to the active visual tracking teacher module; 2.1.3.7 The reward function calculation module in the active visual tracking teacher module calculates the reward of the tracker according to the formula (7) in 2.1.1.6 based on the preprocessed pose data of the tth step 2.1.3.8 The reward function calculation module in the active visual tracking teacher module divides the training environment into I grids, and uses the frequency calculation method described in 2.1.1.7 to calculate the frequency p1,…,p of the set of target position coordinates in the past N steps in each grid based on the preprocessed pose data of the past t steps. i ,…,p I ; 2.1.3.9 The reward function calculation module in the active vision teacher module calculates the position entropy reward loc_entropy of the target according to formula (9) in 2.1.1.8; 2.1.3.10 The reward function calculation module in the active visual tracking teacher module calculates the reward of the target based on the preprocessed pose data of step t 2.1.3.11 The reward function calculation module in the active visual tracking teacher module calculates the reward of the distractor based on the preprocessed pose data of step t 2.1.3.12 The reward function calculation module in the active visual tracking teacher module converts the tracker's reward Rewards for goals Distractor Rewards Send to the first loss function calculation module in the active visual tracking teacher module; 2.1.3.13 The meta-tracker network, meta-target network, and meta-interference network in the active visual tracking teacher module use the estimation method described in step 2.1.1.12 to perform action prediction and value estimation on the preprocessed pose data, respectively, to obtain the state value V of the tracker 1 (s t ) and the tracker’s action probability π 1 (a t |s t ), the state value V of the target 2 (s t ) and the target action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ) and the action probability π of the distractor 3 (a t |s t ); 2.1.3.14 The meta-tracker network in the active visual tracking teacher module converts the state value V of the tracker 1 (s t ) and action probability π 1 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the tracker is 1 (a t |s t ) is sent to the tracker; the meta-target network sends the target state value V 2 (s t ) and action probability π 2 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the target action probability π 2 (a t |s t ) is sent to the target; the meta-interference network sends the state value V of the distractor 3 (s t ) and action probability π 3 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the interference object is 3 (a t |s t ) is sent to the interferer; 2.1.3.15 If t = 1, go to 2.1.3.17; otherwise, the first loss function calculation module in the active visual tracking teacher module calculates the reward of the tracker received from the reward function calculation module according to formula (12) and formula (13) in 2.1.1.

14. Rewards for goals Distractor Rewards and the state value V of the tracker received from the meta-tracker network, meta-target network, and meta-distractor network 1 (s t ), the action probability π of the tracker 1 (a t |s t ), the state value V of the target 2 (s t ), the target action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ), the action probability of the distractor π 3 (a t |s t ) Calculate the time difference error loss of the tracker Policy gradient loss for tracker Target temporal difference error loss Policy gradient loss of the target Temporal Differential Error Loss of Interferers Policy Gradient Loss for Distractors 2.1.3.16 Active visual tracking teacher module uses Adam optimization algorithm to minimize and To update the parameters of the meta-tracker network, use the Adam optimization algorithm to minimize and To update the parameters of the meta-target network, the Adam optimization algorithm is used to minimize and To update the parameters of the meta-disturbant network; 2.1.3.17 Tracker According to the tracker's action probability π 1 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking actions to interact with the environment; at the same time, the target is based on the action probability π 2 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking actions to interact with the environment; at the same time, the interferer is based on the action probability π of the interferer. 3 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking the action to interact with the environment; 2.1.3.18 Let t = t + 1. If t ≤ T, go to 2.1.3.

19. If t > T, go to 2.1.

4. 2.1.3.19 If t%10000=0, save the parameters of the meta-target network and the meta-interference network to the opponent model pool, and go to 2.1.3.4 to continue training using the data of the next time step; Otherwise: Go directly to 2.1.3.4 and continue training with the data of the next time step; 2.1.4 Active Visual Tracking Teacher Module Save the parameters of the meta-tracker network, go to 2.2; 2.2 In the process of the tracker interacting with the target, interference, and environment, the meta-tracker network is further fine-tuned and trained using reinforcement learning to optimize the parameters of the meta-tracker network. The active visual tracking teacher module saves the parameters of the meta-tracker network. 2.3 The active visual tracking student module is trained using supervised learning. The method is as follows: 2.3.1 Meta-tracker network in active visual tracking teacher module Load the parameters of the meta-tracker network saved in step 2.2; 2.3.2 Fixed parameters of the meta-tracker network; 2.3.3 Set the number of training steps T to a positive integer greater than 100,000, and initialize the number of trained steps t = 1; 2.3.4 Randomly select the parameters of the meta-target network from the opponent model pool and load them into the meta-target network; randomly select the parameters of the meta-interferer network from the opponent model pool and load them into the meta-interferer network; 2.3.5 Fix the parameters of the meta-target network and meta-interference network; 2.3.6 Data acquisition module obtains the visual observation image o of step t from the camera of the tracker t , obtain the pose data of the tth step from the pose sensors of the tracker, target, and interference object, and send the visual observation image and pose data of the tth step to the data preprocessing module; 2.3.7 The data preprocessing module uses the first preprocessing method in step 2.1.1.5 to preprocess the posture data of the tth step received from the data acquisition module to obtain the preprocessed posture data of the tth step, and sends the preprocessed posture data of the tth step to the active visual tracking teacher module; 2.3.8 The data preprocessing module uses the second preprocessing method to process the visual observation image o of the tth step received from the data acquisition module. t Perform preprocessing to obtain the preprocessed visual observation image o′ of the tth step t , the preprocessed visual observation image o′ of step t t Sent to the active visual tracking student module; the second preprocessing method is: 2.3.8.1 o t The width is scaled to 320 pixels and the height is scaled to 240 pixels to obtain a scaled visual observation image; 2.3.8.2 Normalize the pixel values ​​of the scaled visual observation image to the interval [0,1] to obtain the normalized visual observation image of step t; 2.3.8.3 Randomly flip the normalized visual observation image of step t horizontally to obtain the preprocessed visual observation image o′ of step t t ; 2.3.9 The meta-tracker network, meta-target network, and meta-interference network in the active visual tracking teacher module use the estimation method described in 2.1.1.12 to perform action prediction and value estimation on the preprocessed pose data, respectively, to obtain the state value V of the tracker. 1 (s t ) and action probability π 1 (a t |s t ), the state value V of the target 2 (s t ) and action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ) and action probability π 3 (a t |s t ), the meta-tracker network converts the tracker’s action probability π 1 (a t |s t ) is sent to the second loss function calculation module in the active visual tracking student module, and the meta-target network converts the target action probability π 2 (a t |s t ) is sent to the target, and the meta-interference network sends the action probability π of the distractor 3 (a t |s t ) is sent to the interferer; 2.3.10 Active Visual Tracking The visual tracker network in the student module uses an action prediction method to preprocess the visual observation image o′ t Perform action prediction and obtain the action probability π of the tracker 1* (a t |o′ t ), the method is: 2.3.10.1 Convolutional backbone in the visual tracker network for the preprocessed visual observation image o′ t Perform feature extraction to obtain visual observation image features, and send the visual observation image features to the multi-scale convolutional network branch and the scale attention module; 2.3.10.2 The first scale convolutional network branch in the multi-scale convolutional network branch extracts features from visual observation image features to obtain first scale visual observation image features, and sends the first scale visual observation image features to the adaptive fusion module; the second scale convolutional network branch in the multi-scale convolutional network branch extracts features from visual observation image features to obtain second scale visual observation image features, and sends the second scale visual observation image features to the adaptive fusion module; the third scale convolutional network branch in the multi-scale convolutional network branch extracts features from visual observation image features to obtain third scale visual observation image features, and sends the third scale visual observation image features to the adaptive fusion module; the fourth ordinary convolutional layer in the fourth scale convolutional network branch in the multi-scale convolutional network branch performs dimensionality reduction and feature extraction on the visual observation image features received from the convolutional backbone network in the channel dimension to obtain fourth scale visual observation image features, and sends the fourth scale visual observation image features to the adaptive fusion module; 2.3.10.3 The convolutional long short-term memory unit in the scale attention module extracts temporal features from the visual observation image features received from the convolutional backbone network to obtain temporal features of the visual observation image, and sends the temporal features of the visual observation image to the global average pooling layer; the global average pooling layer compresses the temporal features of the visual observation image received from the convolutional long short-term memory unit in the spatial dimension to obtain a first feature vector whose length is equal to the number of channels of the temporal features of the visual observation image, and sends the first feature vector to the seventh fully connected layer; the seventh fully connected layer compresses the first feature vector received from the global average pooling layer to obtain a second feature vector whose length is equal to the number of branches of the multi-scale convolutional network, and sends the second feature vector to the sigmoid activation function layer; the sigmoid activation function layer scales the value of the second feature vector received from the seventh fully connected layer to the range of [0,1] to obtain the weights of the multi-scale visual observation image features, and sends the weights of the multi-scale visual observation image features to the adaptive fusion module; 2.3.10.4 The weighted splicing layer in the adaptive fusion module uses the weights of the multi-scale visual observation image features received from the scale attention module to weightedly splice the first-scale visual observation image features, the second-scale visual observation image features, the third-scale visual observation image features, and the fourth-scale visual observation image features received from the multi-scale convolutional network branch to obtain weighted spliced ​​multi-scale visual observation image features, and sends the weighted spliced ​​multi-scale visual observation image features to the fifth ordinary convolution layer; the fifth ordinary convolution layer performs dimensionality reduction on the weighted spliced ​​multi-scale visual observation image features received from the weighted splicing layer in the channel dimension to obtain adaptively fused multi-scale visual observation image features, and sends the adaptively fused multi-scale visual observation image features to the recurrent attention module; 2.3.10.5 The recurrent attention module extracts and weights the time series features of the adaptively fused multi-scale visual observation image features received from the adaptive fusion module to obtain adaptive time series scale features, and sends the adaptive time series scale features to the eighth fully connected layer; 2.3.10.6 The eighth fully connected layer performs action prediction on the adaptive temporal scale features received from the recurrent attention module to obtain the tracker’s action probability π 1* (a t |o′ t ); 2.3.11 Active Visual Tracking The visual tracker network in the student module converts the tracker's action probability π 1* (a t |o′ t ) is sent to the second loss function calculation module and the tracker; 2.3.12 The second loss function calculation module in the active visual tracking student module is based on the tracker action probability π received from the visual tracker network 1* (a t |o′ t ) and the tracker’s action probability π received from the meta-tracker network in the active visual tracking teacher module 1 (a t |s t ) Calculate the KL divergence loss D KL (π 1* (a t |o′ t )|π 1 (a t |s t )): 2.3.13 Active visual tracking student module uses Adam optimization algorithm to minimize D KL (π 1* (a t |o′ t )|π 1 (a t |s t )) to update the parameters of the visual tracker network; 2.3.14 Tracker According to the tracker's action probability π 1* (a t |o′ t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking actions to interact with the environment; at the same time, the target is based on the action probability π 2 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking actions to interact with the environment; at the same time, the interferer is based on the action probability π of the interferer. 3 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking the action to interact with the environment; 2.3.15 Let t = t + 1; if t ≤ T, go to 2.3.16; if t > T, go to 2.3.17; 2.3.16 If t%5000=0, go to 2.3.4 to train the active visual tracking teacher module loaded with the parameters of the new meta-target network and meta-distractor network; otherwise go to 2.3.6 to continue training using the data of the next time step; 2.3.17 Save the parameters of the visual tracker network and go to step 3; Step 3: The active visual tracking student module of the scale-adaptive active visual tracking system loads the parameters of the visual tracker network saved in step 2.3.17 to obtain the trained scale-adaptive active visual tracking system; In the fourth step, the trained scale-adaptive active visual tracking system tracks the target by: 4.1 The data acquisition module obtains the real-time visual observation image o from the camera of the tracker and sends o to the data preprocessing module; 4.2 The data preprocessing module receives the real-time visual observation image o, and preprocesses the real-time visual observation image o using the second preprocessing method described in step 2.3.8 to obtain the preprocessed real-time visual observation image o ′ , will o ′ Sent to the Active Visual Tracking Student module; 4.3 Active visual tracking student module receives the preprocessed real-time visual observation image o ′ The visual tracker network uses the action prediction method described in step 2.3.10 to predict the preprocessed real-time visual observation image o ′ Perform action prediction and obtain the action probability π of the tracker 1* (a ′ |o ′ );a ′ is a random variable representing the action of the agent, and its value range is the action range of the agent; 4.4 Visual Tracker Network The tracker action probability π 1* (a ′ |o ′ ) is sent to the tracker; 4.5 Tracker According to the tracker's action probability π 1* (a ′ |o ′ ) represents the conditional probability of selecting an action from the action range and taking action to track the target.

2. A scale-adaptive active visual tracking method as claimed in claim 1, characterized in that The first-scale convolutional network branch consists of a first ordinary convolutional layer with a convolution kernel size of 1×1 and a first dilated convolutional layer with a convolution kernel size of 3×3 and a dilation rate of 3, and the receptive field size is 7; the second-scale convolutional network branch consists of a second ordinary convolutional layer with a convolution kernel size of 1×1 and a second dilated convolutional layer with a convolution kernel size of 3×3 and a dilation rate of 2, and the receptive field size is 5; the third-scale convolutional network branch consists of a third ordinary convolutional layer with a convolution kernel size of 1×1 and a third dilated convolutional layer with a convolution kernel size of 3×3 and a dilation rate of 1, and the receptive field size is 3; The fourth-scale convolutional network branch consists of the fourth ordinary convolutional layer with a convolution kernel size of 1×1 and a receptive field size of 1.

3. A scale-adaptive active visual tracking method as claimed in claim 1, characterized in that The convolution kernel size of the fifth ordinary convolution layer is 1×1.

4. A scale-adaptive active visual tracking method as claimed in claim 1, characterized in that The number of training steps T in step 2.1.1.1 and step 2.3.3 is set to 2000000, and the number of training steps T in step 2.1.2.2 and step 2.1.3.2 is set to 1000000; In steps 2.1.1.2, 2.1.2.3 and 2.1.3.3, the target position entropy reward coefficient a is set to 0.8; the regularization factor α is set to 0.005; the reward attenuation factor γ is set to 0.9; the number of environmental horizontal grid divisions n is set to 10; 2.1.2.3 The target is far away from the tracker by a threshold distance ρ s Set to 500cm; the threshold distance of the target approaching the tracker in step 2.1.3.3 is set to ρ l =150cm.

5. A scale-adaptive active visual tracking method as claimed in claim 1, characterized in that ρ in step 2.1.1.1 and ρ in step 2.1.1.5.2 * Set to 250cm, the ρ max Set to 750cm, the θ max Set to 90°.

6. A scale-adaptive active visual tracking method as claimed in claim 1, characterized in that 2.1.1.5.3 The normalized relative distance of the actual position of the target Normalized relative distance to the actual position of the distractor The relative angle of the normalized representation of the actual position of the target The relative orientation angle of the normalized representation of the actual position of the target The relative angle of the normalized representation of the actual position of the distractor The relative orientation angle of the normalized representation of the actual position of the distractor The method is: 2.1.1.5.3.1 Calculate the normalized relative distance of the target's actual position Normalized relative distance to the actual position of the distractor 2.1.1.5.3.2 Calculate the relative angle of the target's actual position The sine value of and cosine Get the relative angle of the actual position of the target represented by normalization; 2.1.1.5.3.3 Calculate the relative orientation angle of the target's actual position The sine value of and cosine Obtain the relative orientation angle of the actual position of the target represented by the normalization; 2.1.1.5.3.4 Calculate the relative angle of the actual position of the interference object The sine value of and cosine Get the relative angle of the actual position of the normalized interference object; 2.1.1.5.3.5 Calculate the relative orientation angle of the actual position of the interference object The sine value of and cosine The normalized relative orientation angle of the actual position of the interference object is obtained.

7. A scale-adaptive active visual tracking method as claimed in claim 1, characterized in that The reward function calculation module in the active visual tracking teacher module described in step 2.1.1.7 uses a frequency calculation method to calculate the frequency p1,…,p of the set of target position coordinates in the past N steps in each grid based on the preprocessed pose data of the past t steps. i ,…,p I The method is: 2.1.1.7.1 Initialize variable i = 1; 2.1.1.7.2 Calculate the frequency p of the set of target position coordinates in the past N steps appearing in the i-th grid i : Where N i is the number of positions in the i-th grid in the set of target position coordinates in the past N steps, N i <N; 2.1.1.7.3 Let i=i+1, if i≤I, go to 3.1.1.7.2; if i>I, get p1,…,p i ,…,p I .

8. A scale-adaptive active visual tracking method as claimed in claim 1, characterized in that The action range of the agent in step 2.1.1.12.4 is 7 actions: forward, backward, left, right, forward and left, forward and right, and stop.

9. A scale-adaptive active visual tracking method as claimed in claim 1, characterized in that In step 2.2, the meta-tracker network is further fine-tuned and trained by using reinforcement learning in the process of interaction between the tracker and the target, the interference object, and the environment. The method for optimizing the parameters of the meta-tracker network is as follows: 2.2.1 Meta-tracker network in active visual tracking teacher module Load the parameters of the meta-tracker network saved in step 2.1.4; 2.2.2 Set the number of training steps T = 1000000, and initialize the number of trained steps t = 1; 2.2.3 Randomly select the parameters of the meta-target network from the opponent model pool and load them into the meta-target network; randomly select the parameters of the meta-interferer network from the opponent model pool and load them into the meta-interferer network; 2.2.4 Fix the parameters of the meta-target network and meta-interference network; 2.2.5 The data acquisition module obtains the posture data of the tth step from the posture sensors of the tracker, target, and interference object, and sends the posture data of the tth step to the data preprocessing module; 2.2.6 The data preprocessing module uses the first preprocessing method in 2.1.1.5 to preprocess the posture data of the tth step received from the data acquisition module to obtain the preprocessed posture data of the tth step, and sends the preprocessed posture data of the tth step to the active visual tracking teacher module; 2.2.7 The reward function calculation module in the active visual tracking teacher module calculates the reward of the tracker according to the formula (7) in 2.1.1.6 based on the preprocessed pose data of the tth step 2.2.8 The reward function calculation module in the active visual tracking teacher module converts the tracker's reward Send to the first loss function calculation module in the active visual tracking teacher module; 2.2.9 The meta-tracker network, meta-target network, and meta-interference network in the active visual tracking teacher module use the estimation method described in step 2.1.1.12 to perform action prediction and value estimation on the preprocessed pose data, respectively, to obtain the state value V of the tracker. 1 (s t ) and the tracker’s action probability π 1 (a t |s t ), the state value V of the target 2 (s t ) and the target action probability π 2 (a t |s t ), the state value V of the interference 3 (s t ) and the action probability π of the distractor 3 (a t |s t ); 2.2.10 The meta-tracker network in the active visual tracking teacher module converts the state value V of the tracker 1 (s t ) and action probability π 1 (a t |s t ) is sent to the first loss function calculation module in the active visual tracking teacher module, and the action probability π of the tracker is 1 (a t |s t ) is sent to the tracker; the meta-target network sends the target action probability π 2 (a t |s t ) is sent to the target; the meta-interference network sends the action probability π of the distractor 3 (a t |s t ) is sent to the interferer; 2.2.11 If t = 1, go to 2.2.13; Otherwise, the first loss function calculation module in the active visual tracking teacher module calculates the reward of the tracker received from the reward function calculation module according to formula (12) and formula (13) in step 2.1.1.

14. and the state value V of the tracker received from the meta-tracker network 1 (s t ), the action probability π of the tracker 1 (a t |s t ) Calculate the time difference error loss of the tracker Policy gradient loss for tracker 2.2.12 Active visual tracking teacher module uses Adam optimization algorithm to minimize and to update the parameters of the meta-tracker network; 2.2.13 Tracker According to the tracker's action probability π 1 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking actions to interact with the environment; at the same time, the target is based on the action probability π 2 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking actions to interact with the environment; at the same time, the interferer is based on the action probability π of the interferer. 3 (a t |s t ) represents the conditional probability of selecting an action from the 7 actions in the action range and taking the action to interact with the environment; 2.2.14 Let t = t + 1. If t ≤ T, go to 2.2.

15. If t > T, go to 2.2.

16. 2.2.15 If t%5000=0, go to 2.2.3 to train the active visual tracking teacher module loaded with the parameters of the new meta-target network and meta-distractor network; otherwise, go to 2.2.5 to continue training using the data of the next time step; 2.2.16 The active visual tracking teacher module saves the parameters of the meta-tracker network.

10. A scale-adaptive active visual tracking method as claimed in claim 1, characterized in that 2.3.10.2 The method for extracting features of visual observation image features by the first scale convolutional network branch in the multi-scale convolutional network branch is: the first ordinary convolutional layer performs dimension reduction on the visual observation image features received from the convolutional backbone network in the channel dimension to obtain the first visual observation image features after dimension reduction, and sends the first visual observation image features after dimension reduction to the first dilated convolutional layer; the first dilated convolutional layer performs feature extraction on the first visual observation image features after dimension reduction received from the first ordinary convolutional layer to obtain the first scale visual observation image features, and sends the first scale visual observation image features to the adaptive fusion module; the method for extracting features of visual observation image features by the second scale convolutional network branch in the multi-scale convolutional network branch is: the second ordinary convolutional layer performs dimension reduction on the visual observation image features received from the convolutional backbone network in the channel dimension to obtain the second visual observation image features after dimension reduction, The second visual observation image features after dimensionality reduction are sent to the second dilated convolution layer; the second dilated convolution layer performs feature extraction on the second visual observation image features after dimensionality reduction received from the second ordinary convolution layer to obtain second-scale visual observation image features, and sends the second-scale visual observation image features to the adaptive fusion module; the method for the third-scale convolution network branch in the multi-scale convolution network branch to extract features of the visual observation image features is: the third ordinary convolution layer performs dimensionality reduction on the visual observation image features received from the convolution backbone network in the channel dimension to obtain the third visual observation image features after dimensionality reduction, and sends the third visual observation image features after dimensionality reduction to the third dilated convolution layer; the third dilated convolution layer performs feature extraction on the third visual observation image features after dimensionality reduction received from the third ordinary convolution layer to obtain the third-scale visual observation image features, and sends the third-scale visual observation image features to the adaptive fusion module.

Images