A target tracking method, device and storage medium based on reinforcement learning algorithm SAC
By applying the reinforcement learning algorithm SAC in target tracking, and using actor network and experience pool to optimize action prediction, the drift and high computational cost of the Siamese tracking algorithm are solved, and the tracking accuracy and training speed are improved.
Patent Information
- Application Number
- CN202210240273.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-10
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-03-10
AI Technical Summary
The existing Siamese tracking algorithm based on deep learning is prone to drift when similar objects appear in the tracking image. The training process is computationally expensive and time-cost, the tracking accuracy is low and greatly affected by parameters.
The reinforcement learning algorithm SAC is adopted to obtain video data, determine the search area, and use the pre-trained actor network model to perform feature extraction and action prediction. Combining the experience pool and reward function, network weights are optimized to realize online decision-making and action updates.
Improve the accuracy and training speed of target tracking, reduce the dependence on parameters, and achieve more efficient video sequence tracking.
Smart Images

Figure CN114897930B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of target tracking and relates to a target tracking method, a device and a storage medium based on a reinforcement learning algorithm SAC. Background Art
[0002] As a sub-topic of computer vision, target tracking has always been a hot research field in artificial intelligence. It can be widely used in monitoring, security, autonomous driving and other fields. Target tracking is to track a specific or partial target in a video sequence according to task requirements. In recent years, target tracking research has made great progress, but occlusion, blur, light color change, and deformation in videos are still huge challenges in this field. Therefore, research on target tracking algorithms should be deepened.
[0003] In recent years, the Siamese tracking algorithm based on deep learning has developed rapidly, which uses deep networks to extract image features. The target tracking problem is transformed into using the first frame image to find the area with the maximum similarity in the next video sequence image as the tracking target, which improves the accuracy of the target tracking model to an unprecedented level. At the same time, with the continuous development of reinforcement learning theory, many reinforcement learning applications have emerged in the fields of medical imaging, quantitative trading, and autonomous driving. It regards specific tasks as online Markov decision processes, opening up new solutions in their respective fields.
[0004] However, due to its similarity matching characteristics and offline training method, the current Siamese tracking algorithm based on deep learning will cause the tracker to misjudge the target position and drift when an object similar to the target appears in the tracking image. In addition, the training process is computationally intensive and time-consuming. Existing tracking algorithms based on reinforcement learning, whether discrete or continuous, have low tracking accuracy, slow training convergence, long training time, and are greatly affected by parameters. Summary of the invention
[0005] Objective: To overcome the deficiencies in the prior art, the present invention provides a target tracking method, device and storage medium based on a reinforcement learning algorithm SAC.
[0006] Technical solution: To solve the above technical problems, the technical solution adopted by the present invention is:
[0007] In a first aspect, a target tracking method based on a reinforcement learning algorithm SAC is provided, comprising:
[0008] Get video data;
[0009] Determining a search region location and size in a current frame of video data;
[0010] Determine whether the current frame is the first frame;
[0011] In response to the current frame not being the first frame, obtaining a prediction result of a previous frame of the current frame, cropping an image of the current frame according to the prediction result of the previous frame of the current frame to obtain a cropped image s of the current frame, inputting the cropped image s of the current frame into a pre-trained actor network model (into the actor network computing action a) for feature extraction, and obtaining an output prediction box;
[0012] Track the target according to the prediction frame, and use the prediction frame as the groundTruth of the next frame;
[0013] The training method of the actor network model includes:
[0014] Determine the size and position of the target in the first frame of the video data and set the experience pool parameters;
[0015] Initialize actor, target_actor, critic1, target_critic1, critic2, target_critic2 network parameters;
[0016] In response to the current input frame being the first frame, the actor and target_actor networks are initialized.
[0017] In response to the current frame not being the first frame, the image is cropped in the current frame according to the current frame prediction frame to obtain a prediction frame cropped image s', and the IoU between the current frame prediction frame and the current frame real frame (manually marked) is calculated; according to the IoU, a reward value is calculated by a reward function;
[0018] The prediction result of the previous frame is stored in the experience pool by cropping the image s, action, reward value, and prediction box cropped image s' in the current frame;
[0019] According to the data stored in the experience pool, the action a is calculated through the actor and critic networks. t 、Action a t The logarithm of the probability corresponding to the defined distribution Normal (mu.std) logπ φ (a|s);
[0020] According to the calculated action a t 、Action a t The logarithm of the probability corresponding to the defined distribution Normal (mu.std) logπ φ (a|s), calculate the actor, critic1, and critic2 network losses, and use the reinforcement learning SAC algorithm to update the network weights.
[0021] The setting of the experience pool parameters includes an experience pool capacity X, which indicates that X data can be stored, and each data symbol is: (s, a, r, s'), where s indicates the prediction result of the previous frame and the image is cropped in the current frame, a indicates the action, r indicates the reward value, and s' indicates the prediction box of the current frame and the image is cropped in the current frame.
[0022] The setting of the experience pool parameters includes: setting the experience pool capacity to X according to experience, indicating that X data symbols can be stored: (s, a, r, s'), one data includes: a 1*3*107*107 dimensional image cropped in the current frame, a 1*3 dimensional action, a 1*1 reward value, and a 1*3*107*107 dimensional image cropped after the current frame image takes an action.
[0023] In some embodiments, X is 10,000.
[0024] In some embodiments, the initialization of actor, target_actor, critic1, target_critic1, critic2, target_critic2 network parameters includes: loading the first four layers of network parameters of the vgg-M network pre-trained in imageNet, and using this as the image feature extraction model network, and assigning the actor, critic1, critic2 network parameters to the target_actor, target_critic1, target_critic2 network parameters respectively.
[0025] In some embodiments, in response to the current input frame being the first frame, the actor and target_actor networks are initialized, including: if the frame is the first frame of the video sequence, minimizing the error between the actor and target_actor network outputs and the label, the loss function expression is:
[0026]
[0027] Where μ(s m |φ μ ) is to add Gaussian noise to groundTruth to generate M samples, and the predicted action is output after being processed by the actor network, a m is the label, is the true distance between M samples and groundTruth, μ is the actor network, and m represents the mth data;
[0028] Train the actor and target_actor network parameters using the Adam optimizer.
[0029] In some embodiments, according to the IoU, a reward value is calculated by a reward function, including:
[0030]
[0031] Among them, r represents the reward value, b' is the predicted box, and G is the real box.
[0032] In some embodiments, based on the data stored in the experience pool, the action a is calculated by the actor and critic networks. t 、Action a t The logarithm of the probability corresponding to the defined distribution Normal (mu.std) logπ φ (a|s), including: action a obtained by re-parameterization t =f φ (ε t ;s t ) = pi, and action a t The logarithm of the corresponding probability in the defined distribution Normal (mu.std); logπ φ (a|s) where π φ represents the actor network, and the network parameter is φ;
[0033] a t =f φ (ε t ;s t )=pi The calculation method using the reparameter method includes: sending s taken from the experience pool into the actor network to calculate mu and std, and then calculating the Gaussian distribution with mean mu and variance std: pi_distribution=Normal(mu,std), and then sampling by the reparameter method. The reparameter sampling method is: first sample ε, ε~N(0,1) from a Gaussian normal distribution with a mathematical expectation of 0 and a standard deviation of 1 (Normal(0,1)), and then obtain a through a linear transformation operation. t :a t =mu+std*ε;
[0034] Action a t The logarithm of the probability corresponding to the defined distribution Normal (mu.std) logπ φ (a|s) According to the formula logπ φ (a|s)=log(pi)-∑log(1-tanh 2 (pi)) is calculated; where pi is the action a obtained by the above-mentioned re-parameter method t , where log(pi) is obtained by the Gaussian likelihood function, and the function formula is
[0035] In some embodiments, the actor, critic1, and critic2 network losses are calculated, and the network weights are updated using a reinforcement learning SAC algorithm, including:
[0036] The actor network loss function is defined as:
[0037]
[0038] Among them, Q represents the critic network, θ represents the critic network parameters, and f φ (ε t ;s t ) is the action obtained by re-parameterization, st~D Indicates sampling from the experience pool D st , εt N means sampling from a normal distribution; logπ φ (f φ (ε t ;s t )|s t ) is the logπ φ (a|s), f φ (ε t ;s t ) is a.
[0039] After derivation, the Actor network gradient is calculated as:
[0040]
[0041] The critic network loss is defined as:
[0042]
[0043] Among them (s t ,a t )~D represents sampling data from the experience pool, r(s t ,a t ) is the reward value obtained by sampling, represents the targetCritic network, subscript represents the targetCritic network parameters, s t+1 That is s', a t+1 That is, use s' to input the actor network and calculate it through the above-mentioned re-parameterization method; Indicates that s t+1 ,a t+1 Input the targetCritic network and calculate; log(π φ (at+1 |s t+1 )) is calculated from the above Gaussian likelihood, where a t+1 That is, s' is used to input the actor network, and s is calculated by the above re-parameterization method. t+1 That is s', α is the entropy coefficient, γ is the attenuation coefficient;
[0044] in For target_critic network calculation, take the smaller value of the two for calculation, the formula is:
[0045]
[0046] After derivation, the gradient of the Critic network is:
[0047]
[0048] In a second aspect, the present invention provides a single target tracking device, including a processor and a storage medium;
[0049] The storage medium is used to store instructions;
[0050] The processor is used to operate according to the instructions to execute the steps of the method according to the first aspect.
[0051] In a third aspect, the present invention provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.
[0052] Beneficial effects: The target tracking method and system based on the reinforcement learning algorithm SAC provided by the present invention transform the target tracking problem into an online decision-making problem in the reinforcement learning algorithm, providing a novel research direction for the target tracking field. In addition, the present invention only requires a small amount of data set, fully utilizes existing technologies, and improves training speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 Flow chart of a target tracking method based on a reinforcement learning algorithm SAC according to an embodiment of the present invention;
[0054] Figure 2 FIG. 4 is a network diagram of a target tracking system according to an embodiment of the present invention. DETAILED DESCRIPTION
[0055] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.
[0056] In the description of the present invention, "several" means more than one, "many" means more than two, "greater than", "less than", "exceed", etc. are understood to exclude the number itself, and "above", "below", "within", etc. are understood to include the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.
[0057] In the description of the present invention, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0058] Example 1
[0059] A target tracking method based on a reinforcement learning algorithm SAC, comprising:
[0060] Step S1, manually determine the size and position of the object of interest in the first frame, and set the experience pool parameters;
[0061] Step S2, initialize actor, target_actor, critic1, target_critic1, critic2, target_critic2 network parameters;
[0062] Step S3, randomly select a video sequence, determine the position and size of the search area in the current frame, and judge whether the current input frame is the first frame. If the current input frame is the first frame, initialize the actor and target_actor networks; if the input frame is not the first frame, use the actor network to extract the convolution features of the input frame to obtain the prediction frame;
[0063] Step S4, calculate the IoU between the predicted box and the real box, obtain the reward value according to the reward function, and store the current frame cropped image, action, reward value, and predicted box cropped image into the experience pool;
[0064] Step S5: Select data from the experience pool and calculate a, logπ according to the re-parameter method φ (f φ (ε t ;s t )), for subsequent updates;
[0065] Step S6, calculate the actor, critic1, and critic2 network losses, and use the reinforcement learning SAC algorithm to update the network weights;
[0066] Step S7: Use the predicted frame as the groundTruth of the next frame and return to S3 until the prediction of all frames of the video is completed.
[0067] In some embodiments, step S1 : giving the position and size of the target to be tracked in the first frame.
[0068] Specifically, the position and size of the target to be tracked are given in the first frame, specifically:
[0069] In the first frame of the video, the information of the object of interest is determined by manually or using an object detection algorithm to give the size and position of the object of interest.
[0070] Set the experience pool parameters, specifically:
[0071] According to experience, the experience pool capacity is set to 10000, which means that 10000 data can be stored. The symbol is: (s,a,r,s'), a data contains: a 1*3*107*107 dimensional image cut in the current frame, a 1*3 dimensional action, a 1*1 reward value, and a 1*3*107*107 dimensional image cut after the current frame image takes an action. That is, the result after the network interacts with the video once;
[0072] Step S2, initialize actor, target_actor, critic1, target_critic1, critic2, target_critic2 network parameters.
[0073] Specifically:
[0074] Load the first four layers of the vgg-M network parameters pre-trained on imageNet and use them as the image feature extraction model network, including assigning the actor, critic1, and critic2 network parameters to the target_actor, target_critic1, and target_critic2 network parameters respectively;
[0075] Step S3: randomly select a video sequence, determine the position and size of the search area in the current frame, and determine whether the current input frame is the first frame. If the current input frame is the first frame, initialize the actor and target_actor networks; if the input frame is not the first frame, use the actor network to extract the convolutional features of the input frame and use the convolutional features to obtain the prediction box.
[0076] Randomly select a video sequence, determine the search area position and size in the current frame, and determine whether the current input frame is the first frame. If the current input frame is the first frame, initialize the actor and target_actor networks, specifically:
[0077] If the current input frame is the first frame, initialize the actor and target_actor networks. Specifically, if this frame is the first frame of the video sequence, minimize the error between the actor and target_actor network outputs and labels. The loss function expression is:
[0078]
[0079] In formula (1), μ(s m |φ μ ) is to add Gaussian noise to groundTruth to generate M samples, and the predicted action is output after being processed by the actor network, a m is the label, and is the true distance between M samples and groundTruth. Finally, the actor and target_actor network parameters are initialized through the Adam optimizer;
[0080] If the input frame is not the first frame, the actor network is used to extract the convolutional features of the input frame and the convolutional features are used to obtain the prediction frame, specifically:
[0081] If the input frame is not the first frame, the actor network is used to extract the convolutional features of the input frame and the convolutional features are used to obtain the prediction frame. Specifically, if the picture is not the first frame, no initialization is required. The prediction result of the previous frame is used to crop the image of the current frame and then sent to the actor network to predict the displacement action.
[0082] Step S4, calculate the IoU between the predicted box and the real box, obtain the reward value according to the reward function, and store the current frame cropped image, action, reward value, and predicted box cropped image into the experience pool;
[0083] Calculate the IoU between the predicted box and the true box, and get the reward value according to the reward function, specifically:
[0084] Calculate the IoU (overlap rate) between the predicted box and the real box, and get the reward value according to the reward function. The reward function calculation formula is:
[0085]
[0086] The current frame cropped image, action, reward value, and predicted frame cropped image are stored in the experience pool, specifically:
[0087] The current cropped image (1*3*107*107)s, the action is the output of the actor network (1*3)a, the reward value (obtained by the above reward function, 1*1)r, and the predicted frame cropped image (that is, the new coordinates formed by adding the action based on the current coordinates, 1*3*107*107)s' are stored in the experience pool in a fixed order. If the experience pool is full, it is stored according to the first-in-first-out principle;
[0088] Step S5: Select data from the experience pool and calculate a, logπ according to the re-parameter method φ (f φ (ε t ;s t ), for subsequent updates.
[0089] Specifically:
[0090] When a video sequence repeats the above S3-S4, a batch of data is selected from the experience pool according to formula a t =f φ (ε t ,s t ), a t The selection method is to use Gaussian distribution sampling in the actor network output. In order to solve the problem that the gradient of random variables cannot be returned, a t Use the re-parameter method to obtain it, specifically:
[0091] The s taken from the experience pool is sent to the actor network to calculate mu and std, and then the Gaussian distribution with mean mu and variance std is calculated: pi_distribution = Normal(mu,std), and then sampling is performed by re-parameterization. The re-parameterization sampling method is: first sample ε, ε~N(0,1) from a Gaussian normal distribution with a mathematical expectation of 0 and a standard deviation of 1 (Normal(0,1)), and then obtain a through linear transformation. t :a t =mu+std*ε;
[0092] The logarithm of the probability corresponding to the action at in the defined distribution Normal (mu.std) logπ φ (a|s) According to the formula logπ φ (a|s)=log(pi)-∑log(1-tanh 2 (pi)) is calculated; where pi is the action a obtained by the above-mentioned re-parameter method t , where the log(pi) Gaussian likelihood function is obtained, and the function formula is
[0093] Step S6 uses the reinforcement learning SAC method to calculate the actor, critic1, and critic2 network losses, and uses the gradient to update the network weights.
[0094] Specifically:
[0095] The actor network loss function is defined as:
[0096]
[0097] Q represents the critic network, θ represents the critic network parameters, and f φ (ε t ;s t ) is the action obtained by re-parameterization, st~D Indicates sampling from the experience pool D st , εt ~N means sampling from a normal distribution; logπ φ (f φ (ε t ;s t )|s t ) is the logπ above φ (a|s), f φ (ε t ;s t ) is a.
[0098] After derivation, the Actor network gradient is calculated as:
[0099]
[0100] For the critic network loss, it is defined as:
[0101]
[0102] in Calculate the target_critic network and take the smaller value of the two for calculation
[0103]
[0104] Among them (s t ,a t )~D represents sampling data from the experience pool, r(s t ,a t ) is the reward value obtained by sampling, represents the targetCritic network, subscript represents the targetCritic network parameters, s t+1 That is s', a t+1That is, use s' to input the actor network and calculate it through the above-mentioned re-parameterization method; Indicates that s t+1 ,a t+1 Input the targetCritic network and calculate; log(π φ (a t+1 |s t+1 )) is calculated from the above Gaussian likelihood, where a t+1 That is, s' is used to input the actor network, and s is calculated by the above re-parameterization method. t+1 That is s', α is the entropy coefficient, γ is the attenuation coefficient;
[0105] After derivation, the critic network gradient is:
[0106]
[0107] Example 2
[0108] In a second aspect, this embodiment provides a target tracking device based on a reinforcement learning SAC algorithm, including a processor and a storage medium;
[0109] The storage medium is used to store instructions;
[0110] The processor is used to operate according to the instructions to execute the steps of the method according to embodiment 1.
[0111] Example 3
[0112] In a third aspect, this embodiment provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in Example 1 are implemented.
[0113] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0114] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0115] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0117] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A target tracking method based on reinforcement learning algorithm SAC, characterized in that: include: Get video data; Determining a search region location and size in a current frame of video data; Determine whether the current frame is the first frame; In response to the current frame not being the first frame, obtaining a prediction result of a previous frame of the current frame, cropping an image in the current frame according to the prediction result of the previous frame of the current frame to obtain a cropped image s of the current frame, inputting the cropped image s of the current frame into a pre-trained actor network model for feature extraction, and obtaining an output prediction box; Track the target according to the prediction frame, and use the prediction frame as the groundTruth of the next frame; The training method of the actor network model includes: Determine the size and position of the target in the first frame of the video data and set the experience pool parameters; Initialize actor, target_actor, critic1, target_critic1, critic2, target_critic2 network parameters; In response to the current input frame being the first frame, the actor and target_actor networks are initialized. In response to the current frame not being the first frame, cropping an image in the current frame according to the current frame prediction frame to obtain a prediction frame cropped image s', and calculating an IoU between the current frame prediction frame and the current frame real frame; and obtaining a reward value by calculating a reward function according to the IoU; The prediction result of the previous frame is stored in the experience pool by cropping the image s, action, reward value, and prediction box cropped image s' in the current frame; According to the data stored in the experience pool, the action a is calculated through the actor and critic networks. t 、Action a t The logarithm of the probability corresponding to the defined distribution Normal (mu.std) logπ φ (a|s); According to the calculated action a t 、Action a t The logarithm of the probability corresponding to the defined distribution Normal (mu.std) logπ φ (a|s), calculate the actor, critic1, and critic2 network losses, and use the reinforcement learning SAC algorithm to update the network weights.
2. The target tracking method based on the reinforcement learning algorithm SAC according to claim 1 is characterized in that: in, The setting of the experience pool parameters includes the experience pool capacity X, which indicates that X pieces of data can be stored, and each piece of data is symbolized as (s, a, r, s'), where s indicates the cropped image of the prediction result of the previous frame in the current frame, a indicates the action, r indicates the reward value, and s' indicates the cropped image of the current frame prediction frame in the current frame.
3. The target tracking method based on the reinforcement learning algorithm SAC according to claim 1, characterized in that: The setting of the experience pool parameters includes: setting the experience pool capacity to X according to experience, indicating that X data symbols can be stored: (s, a, r, s'), where one data includes: a 1*3*107*107 dimensional image cropped in the current frame, a 1*3 dimensional action, a 1*1 reward value, and a 1*3*107*107 dimensional image cropped after the current frame image takes an action.
4. The target tracking method based on the reinforcement learning algorithm SAC according to claim 1, characterized in that: The initialization of actor, target_actor, critic1, target_critic1, critic2, target_critic2 network parameters includes: loading the first four layers of network parameters of the vgg-M network pre-trained in imageNet, and using this as the image feature extraction model network, and assigning the actor, critic1, critic2 network parameters to the target_actor, target_critic1, target_critic2 network parameters respectively.
5. The target tracking method based on the reinforcement learning algorithm SAC according to claim 1, characterized in that: In response to the current input frame being the first frame, the actor and target_actor networks are initialized, including: if this frame is the first frame of the video sequence, the error between the actor and target_actor network outputs and the label is minimized. The loss function expression is: Where μ(s m |φ μ ) is to add Gaussian noise to groundTruth to generate M samples, and the predicted action is output after being processed by the actor network, a m is the label, is the true distance between M samples and groundTruth, μ is the actor network, and m represents the mth data; Train the actor and target_actor network parameters using the Adam optimizer.
6. The target tracking method based on the reinforcement learning algorithm SAC according to claim 1, characterized in that: According to IoU, the reward value is calculated by the reward function, including: Among them, r represents the reward value, b' is the predicted box, and G is the real box.
7. The target tracking method based on the reinforcement learning algorithm SAC according to claim 1, characterized in that: According to the data stored in the experience pool, the action a is calculated through the actor and critic networks. t 、Action a t The logarithm of the probability corresponding to the defined distribution Normal (mu.std) logπ φ (a|s), including: action a obtained by re-parameterization t =f φ (ε t ;s t ) = pi, and action a t The logarithm of the corresponding probability in the defined distribution Normal (mu.std); logπ φ (a|s) where π φ represents the actor network, and the network parameter is φ; a t =f φ (ε t ;s t )=pi The calculation method using the reparameter method includes: sending s taken from the experience pool into the actor network to calculate mu and std, and then calculating the Gaussian distribution with mean mu and variance std: pi_distribution=Normal(mu,std), and then sampling by the reparameter method. The reparameter sampling method is: first sample ε, ε~N(0,1) from a Gaussian normal distribution with a mathematical expectation of 0 and a standard deviation of 1 (Normal(0,1)), and then obtain a through a linear transformation operation. t :a t =mu+std*ε; Action a t The logarithm of the probability corresponding to the defined distribution Normal (mu.std) logπ φ (a|s) According to the formula logπ φ (a|s)=log(pi)-∑log(1-tanh 2 (pi)) is calculated; where pi is the action a obtained by the above-mentioned re-parameter method t , where log(pi) is obtained by the Gaussian likelihood function, and the function formula is 8. The target tracking method based on the reinforcement learning algorithm SAC according to claim 7 is characterized in that: Calculate the actor, critic1, and critic2 network losses, and use the reinforcement learning SAC algorithm to update the network weights, including: The actor network loss function is defined as: Among them, Q represents the critic network, θ represents the critic network parameters, and f φ (ε t ;s t ) is the action obtained by re-parameterization, s t ~D means sampling s from the experience pool D t , ε t ~N means sampling from a normal distribution; logπ φ (f φ (ε t ;s t )|s t ) is the logπ φ (a|s), f φ (ε t ;s t ) is a. above; After derivation, the Actor network gradient is calculated as: The critic network loss is defined as: Among them (s t ,a t )~D represents sampling data from the experience pool, r(s t ,a t ) is the reward value obtained by sampling, represents the targetCritic network, subscript represents the targetCritic network parameters, s t+1 That is s', a t+1 That is, use s' to input the actor network and calculate it through the above-mentioned re-parameterization method; Indicates that s t+1 ,a t+1 Input the targetCritic network and calculate; log(π φ (a t+1 |s t+1 )) is calculated from the above Gaussian likelihood, where a t+1 That is, s' is used to input the actor network, and s is calculated by the above-mentioned re-parameterization method. t+1 That is s', α is the entropy coefficient, γ is the attenuation coefficient; in For target_critic network calculation, take the smaller value of the two for calculation, the formula is: After derivation, the gradient of the Critic network is:
9. A single target tracking device, characterized in that: include: including processor and storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 8.
10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.
Citation Information
Patent Citations
Robust target tracking method and system based on hierarchical decision network
CN112802061A
Target tracking identification method and system based on lightweight target tracking network
CN113888597A
Deep cross-correlation learning for object tracking
US20180129906A1
Training action selection neural networks using a differentiable credit function
US20200175364A1
