A target tracking reinforcement learning method based on the target tracking reinforcement learning framework

Abstract

An embodiment of the present invention provides a target tracking reinforcement learning framework, and the method includes: the definition and setting of a tracker, an agent, an executable action space, actions, states, rewards, and an environment in the target tracking reinforcement learning framework; the interaction method between the agent and the environment, the optimization method of the agent control strategy, and the application method of the agent; after the agent is optimized, the agent interacts with the environment, and the tracker motion state information output by the environment is used as the output of the target tracking reinforcement learning method. The tracker motion state information output at each moment is the target track obtained by the reinforcement learning method. An embodiment of the present invention provides a target tracking framework based on reinforcement learning. The given agent can perform self-learning based on a simulated environment or an actual environment, and has the advantages of a wide range of applicability, multiple adaptation scenarios, and low consumption of manpower and material resources.

Description

Technical Field

[0001] The present invention relates to target tracking technology, and more specifically, to a target tracking reinforcement learning method based on a target tracking reinforcement learning framework, which is applicable to single-target or multi-target tracking problems of early warning detection equipment such as radar, remote sensing, optoelectronics, and laser. Background Art

[0002] Target detection is the technical activity of discovering, locating, and tracking targets using early warning and detection equipment such as radar, remote sensing, optoelectronics, and lasers. The results serve as input and basis for further high-level processing, such as attribute identification, threat assessment, and operational decision-making. Target tracking is a crucial step in the target detection process. Its purpose is to connect detection information from the same target at different times and, through smoothing and filtering, generate time-series target state information with a consistent identity, known as the target track, to achieve real-time, continuous, and accurate identification of individual targets.

[0003] Target tracking technology has a long research history, and many high-performance target tracking methods have emerged. However, with the continuous improvement of detection methods, the continuous expansion of detection range, the continuous increase in target types, and the increasing complexity of detection scenarios, current target tracking technology faces multiple challenges. Because the performance of target tracking methods has a decisive impact on the integrity, continuity, and accuracy of target tracks, and is directly related to the accuracy of subsequent target attribute identification, the effectiveness of target threat assessment, and the correctness of combat response decisions, it is necessary to continuously research target tracking methods using new theories and technologies to further improve target tracking performance and solve emerging problems and challenges.

[0004] Currently, statistically based target tracking methods have achieved excellent results in simple or single target tracking scenarios, enabling precise and continuous grasp of the target's position and motion state. However, in complex tracking scenarios such as those involving weak light, dense crowds, and formations, their tracking performance is often unstable, with intermittent performance variations. This makes it difficult to achieve stable and effective tracking results in all situations and across the entire scene. While supervised intelligent target tracking methods based on machine learning and deep learning technologies have addressed this complex tracking problem to some extent, they also face the challenge of constructing training datasets. Over-reliance on training datasets is a common problem with supervised learning methods, and their actual performance is often limited by the size of the training dataset. Achieving acceptable and ideal performance often requires constructing large training datasets. However, constructing training datasets involves sample data collection, preprocessing, and annotation, consuming significant manpower and resources. Consequently, supervised learning methods are considered "labor-intensive" AI methods, often jokingly described as "intelligence is as much labor as there is intelligence; intelligence is artificially constructed." At the same time, target tracking is closely related to factors such as the performance of the specific detection equipment, the type of detection target, and the detection environment. Different types, or even models, of detection equipment often require different training datasets to generate appropriate supervised intelligent target tracking methods. Therefore, overall, using supervised learning methods to solve target tracking problems is extremely costly in terms of both human and material resources, and its widespread application is difficult.

[0005] Unlike supervised learning methods, reinforcement learning methods do not require a training dataset, effectively addressing the pain points of supervised learning and reducing the cost burden of widespread application. In recent years, reinforcement learning has developed rapidly, achieving breakthroughs in multiple industries and fields. Representative achievements include intelligent agents such as AlphaGo, AlphaGo1Zero, AlphaStar, AlphaFold2, and AlphaCode. AlphaGo's Go skills have surpassed the top human professional Go players; AlphaGo Zero, an evolution of AlphaGo, can learn from a blank state; AlphaStar is the first AI system to defeat a professional player in StarCraft II; and AlphaFold2 won the 14th International Protein Structure Prediction Competition. Therefore, for target tracking in various early warning and detection equipment, such as radar, remote sensing, optoelectronics, and lasers, it is necessary to research and develop intelligent target tracking methods based on reinforcement learning techniques that have enhanced performance, wider adaptability, and lower cost. Summary of the Invention

[0006] The purpose of the present invention is to provide a target tracking reinforcement learning method based on a target tracking reinforcement learning framework, providing a reinforcement learning solution to the target tracking problem. The specific technical solution is as follows:

[0007] The tracker in the target tracking reinforcement learning framework is defined as a unit that has the motion characteristics of the tracked target, is controlled by the agent and constrained by the target's motion characteristics, and moves in the detection environment;

[0008] The intelligent agent in the target tracking reinforcement learning framework is defined as the tracker motion decision maker, which controls the tracker motion according to the control strategy. Its input is the state and reward provided by the environment, and its output is the action that the tracker should perform, which is fed back to the environment.

[0009] The executable action space in the target tracking reinforcement learning framework is constructed by the tracker motion controllable parameter variation space, and the specific action is selected from the executable action space;

[0010] The state in the target tracking reinforcement learning framework is the current motion state information of the tracker and the next moment measurement information obtained by the detection device. They are expressed in a unified inertial coordinate system. The measurement information output by the detection device can be composed of multiple measurement points. Each measurement point contains at least the observation time and spatial position information. The tracker motion state information contains at least time, spatial position, motion speed and motion direction information.

[0011] The reward in the target tracking reinforcement learning framework is measured by the distance between the tracker's motion state information and the detection device's measurement information at the same time. The farther the two are, the smaller the reward, and the closer the two are, the larger the reward.

[0012] The environment in the target tracking reinforcement learning framework is constructed by the detection device, the detection environment, and the tracker. Its input is the action, and its output is the state and reward.

[0013] In the target tracking reinforcement learning framework, the agent and the environment can interact continuously. A complete interaction process starts from the initial state of the environment and ends at the final state of the environment, including multiple steps of interaction. In each step of interaction, the agent receives the state and reward output by the environment and outputs the current action decision based on the control strategy. The environment receives the current action output by the agent and outputs the state and reward.

[0014] The optimization method of the intelligent agent in the target tracking reinforcement learning framework is to maximize the cumulative reward, adopt a suitable strategy learning method, and continuously optimize the control strategy of the intelligent agent through the interaction between the intelligent agent and the environment to obtain a target tracking intelligent agent that meets the requirements.

[0015] The application method of the intelligent agent in the target tracking reinforcement learning framework is to use the optimized target tracking intelligent agent to interact with the environment. At each step of interaction, the tracker motion state information output by the environment is used as the output of the target tracking reinforcement learning method. The tracker motion state information output at each moment is the target track obtained by the reinforcement learning method.

[0016] The embodiment of the present invention proposes a target tracking reinforcement learning method based on a target tracking reinforcement learning framework, which can provide a general learning framework for target intelligent tracking methods suitable for radar, remote sensing, optoelectronics, laser and other detection equipment. It does not require the construction of a training data set and can perform self-learning based on a simulated environment or an actual environment to obtain a specific target tracking method. It has the advantages of a wide range of applicability, multiple adaptation scenarios, and low consumption of manpower and material resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is a schematic diagram of a target tracking reinforcement learning method based on a target tracking reinforcement learning framework provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0020] In order to provide a target tracking reinforcement learning method with unified structure and superior performance, the embodiment of the present invention provides a target tracking reinforcement learning method based on the target tracking reinforcement learning framework, such as Figure 1 As shown, the method includes the following steps:

[0021] Step 1: The tracker in the target tracking reinforcement learning framework is defined as a unit that has the motion characteristics of the tracked target, is controlled by the agent and constrained by the target's motion characteristics, and moves in the detection environment;

[0022] Step 1.1: The tracker is a virtual reflection of the real target being tracked. It has the motion characteristics of the tracked target and represents the target being tracked by the target tracking algorithm. The real target being tracked can be a large category of targets, such as ships, air targets, and land targets. The corresponding motion characteristics can be speed range, maximum acceleration, and maximum turning rate. It can also be a more detailed category of targets, or even a specific target, such as an F-16 fighter or a Boeing 747 passenger aircraft. The corresponding motion characteristics can further include cruising speed and minimum turning radius.

[0023] Step 1.2: The tracker is controlled by the agent and constrained by the target's motion characteristics. It moves in the detection environment, and its motion space is consistent with the detection space of the detection device. The tracker's motion is controlled by the agent but cannot exceed the capability boundaries defined by the target's motion characteristics. For example, the tracker's speed cannot exceed the maximum speed specified by the target's motion characteristics, nor can it fall below the minimum speed specified by the target's motion characteristics. The tracker's acceleration cannot exceed the maximum acceleration specified by the target's motion characteristics, and the tracker's turning rate cannot exceed the maximum turning rate specified by the target's motion characteristics.

[0024] Step 1.3: If the tracker's movement exceeds the capability limit defined by the target's motion characteristics under the control of the agent, it can be directly truncated and re-moved according to the capability limit. For example, if the tracker's movement speed exceeds the maximum speed specified by the target's motion characteristics, the tracker's movement speed is directly updated according to the maximum speed specified by the target's motion characteristics;

[0025] Step 2: The intelligent agent in the target tracking reinforcement learning framework is defined as the tracker motion decision maker. It controls the tracker motion according to the control strategy. Its input is the state and reward provided by the environment. The output is the action that the tracker should perform and feedback to the environment.

[0026] Step 2.1: The intelligent agent is defined as the tracker motion decision maker to control the tracker motion;

[0027] Step 2.2: The agent's input is the state and reward provided by the environment. According to the agent's control strategy, the output tracker should perform the action and feed it back to the environment.

[0028] Step 2.3: The core of the agent is the control policy, which is a function whose input is state and reward and whose output is action. According to step 8, the agent can optimize the policy by interacting with the environment and using appropriate policy learning methods.

[0029] Step 3: The executable action space in the target tracking reinforcement learning framework is constructed by the tracker motion controllable parameter variation space, and the specific action is selected from the executable action space;

[0030] Step 3.1: The executable action space in the target tracking reinforcement learning framework is constructed by the space of controllable parameters of the tracker motion, and the specific action output by the agent is selected from the executable action space;

[0031] Step 3.2: For a two-dimensional detection space, the tracker motion controllable parameters can be set to X-direction acceleration and Y-direction acceleration, or to acceleration and heading rate of change. For a three-dimensional detection space, the tracker motion controllable parameters can be set to X-direction acceleration, Y-direction acceleration, and Z-direction acceleration, or to acceleration, heading rate of change, and pitch rate of change.

[0032] Step 3.3: The tracker motion controllable parameter variation space is directly or indirectly defined by the target motion characteristics. In a simple case, it can be assumed that the tracker motion controllable parameters vary independently of each other, and the variation space of each parameter is the boundary defined by the target motion characteristics. Alternatively, the complex relationship between the tracker motion controllable parameters can be considered and a complex space representation can be made based on the target motion characteristics.

[0033] Step 4: The state in the target tracking reinforcement learning framework is the current motion state information of the tracker and the next moment measurement information obtained by the detection device. They are expressed in a unified inertial coordinate system. The measurement information output by the detection device can be composed of multiple measurement points. Each measurement point contains at least the observation time and spatial position information. The tracker motion state information contains at least time, spatial position, motion speed and motion direction information.

[0034] Step 4.1: The state in the target tracking reinforcement learning framework is the current motion state information of the tracker and the next moment measurement information output by the detection device. They are expressed in a unified inertial coordinate system. The detection device refers to the device that can perceive space objects, including radar, remote sensing, optoelectronics, laser, etc.

[0035] Step 4.2: The measurement information output by the detection equipment can be composed of multiple measurement points. Each measurement point contains at least the observation time and spatial position information, where the spatial position is expressed in an inertial coordinate system. For example, the measurement information output by a radar is multiple measurement points obtained during a scanning cycle. Each measurement point has observation time, coordinate position, and even signal strength information.

[0036] Step 4.3: The tracker's motion state information includes at least time, spatial position, motion speed, and motion direction information. The coordinate system representing its spatial position is consistent with the coordinate system representing the detection device's measurement information.

[0037] Step 5: The reward in the target tracking reinforcement learning framework is measured by the distance between the tracker's motion state information and the detection device's measurement information at the same time. The farther the two are, the smaller the reward, and the closer the two are, the larger the reward.

[0038] Step 5.1: The reward in the target tracking reinforcement learning framework is measured by the distance between the tracker's motion state information and the detection device's measurement information at the same time. The farther the two are, the smaller the reward, and the closer the two are, the larger the reward.

[0039] Step 5.2: A feasible reward R calculation method is to first calculate the distance between the tracker's spatial position and each measurement point of the detection device in turn, and then calculate the reward r of the tracker at each measurement point based on this. If the distance between the tracker's spatial position and the detection device's i-th measurement point is greater than the no reward threshold g r , then its reward r i Set to 0, if it is less than the no reward threshold, the reward is set to Finally, the rewards at each measurement point are summed to obtain the total reward at the current moment R = ∑r i , where (x0, y0) is the spatial position of the tracker at the current moment, (x i ,y i ) is the spatial position of the measurement point i, It can be set as the measurement covariance of the detection device measurement point i, or it can be set manually, g r Can be set to It can also be set manually;

[0040] Step 6: The environment in the target tracking reinforcement learning framework is constructed by the detection device, the detection environment, and the tracker. Its input is the action and its output is the state and reward.

[0041] Step 6.1: The environment in the target tracking reinforcement learning framework is jointly constructed by the detection device, the detection environment, and the tracker. The detection environment refers to the space in which the detection device operates, including physical spaces such as land, sea, space, and sky, as well as various targets within that space. For example, for a sea observation radar, the detection environment consists of the marine environment (such as hydrology, meteorology, waves, islands), and various targets at sea, such as ships and submarines, within the radar's detection area.

[0042] Step 6.2: The input to the environment is the action provided by the agent, and the output is the state and reward. The state includes the current motion state information of the tracker and the measurement information of the next moment obtained by the detection device. The reward is calculated according to the reward calculation method based on the current motion state information and the measurement information obtained by the detection device at the current moment.

[0043] Step 6.3: The current motion state information of the tracker is updated according to the target motion equation based on the action received from the environment. For example, for a two-dimensional detection space, the motion state of the tracker at time k-1 is set to time t k-1 、Spatial position (x k-1 ,y k-1 ), movement speed v k-1 and the direction of movement c k-1 , the action is the acceleration in the X direction and Y-direction acceleration Then the motion state information of the tracker at time k is

[0044]

[0045] Step 6.4: Based on the motion state information of the tracker at time k calculated in step 6.3, the motion state information of the tracker at time k can be corrected by using the measurement points around the tracker and a weighted filtering method;

[0046] Step 6.5: The next moment measurement information acquired by the detection device is directly provided by the detection device. In a real environment, it is directly provided by the actual detection device, and in a simulated environment, it is provided by the detection device simulator.

[0047] Step 6.6: In the initial state of the environment, the detection device continuously generates measurement information at two times, t0 and t1. The tracker generates motion state information at time t0. That is, the tracker's initial motion state information is generated based on the measurement information at time t0 and combined with the motion characteristics of the tracked target. The measurement information at time t1 is directly used as the measurement information obtained by the detection device at the next moment. The two are combined to output the initial state of the environment. One feasible way to set the tracker's initial motion state information is to set the time to t0, the spatial position (x0, y0) to be given by the average position of all measurement points at time t0, the motion speed v0 to be generated based on the motion characteristics of the tracked target, and can be set to an intermediate speed between the maximum speed and the minimum speed or the cruising speed, and the motion direction c0 to be randomly generated.

[0048] Step 7: In the target tracking reinforcement learning framework, the agent and the environment can interact continuously. A complete interaction process starts from the initial state of the environment and ends at the final state of the environment, including multiple steps of interaction. In each step of interaction, the agent receives the state and reward output by the environment and outputs the current action decision based on the control strategy. The environment receives the current action output by the agent and outputs the state and reward.

[0049] Step 7.1: In the target tracking reinforcement learning framework, the agent and the environment can interact continuously. A complete interaction process starts from the initial state of the environment and ends at the final state of the environment, including multiple steps of interaction. When the environment is in the initial state, the output state can be calculated and set according to step 6.5, and the output reward can be set to 0.

[0050] Step 7.2: When the target to be tracked in the environment has left the detection device's action space, or the agent's reward is 0 for multiple consecutive times, the environment is considered to have reached a terminal state. At this point, the agent's complete interaction with the environment is complete, and the agent can start interacting again from the initial state of the environment.

[0051] Step 7.3: At each interaction step, the agent receives the state and reward output by the environment and outputs the current action decision according to the agent's control strategy. The environment receives the current action output by the agent and outputs the state and reward according to step 6. The agent's control strategy can be learned and updated according to step 8.

[0052] Step 8: The optimization method of the intelligent agent in the target tracking reinforcement learning framework is to maximize the cumulative reward. By adopting a suitable policy learning method, the intelligent agent control strategy is continuously optimized and learned through the interaction between the intelligent agent and the environment to obtain a target tracking intelligent agent that meets the requirements.

[0053] Step 8.1: The optimization method for the agent in the target tracking reinforcement learning framework is to maximize the cumulative reward. By adopting an appropriate policy learning method, the agent control strategy is continuously optimized and learned through the interaction between the agent and the environment to obtain a target tracking agent that meets the requirements.

[0054] Step 8.2: The agent's interactive environment is constructed according to step 6. This can be the entire real environment, the entire simulated environment, or the simulated environment first and then the real environment.

[0055] Step 8.3: Regarding the policy learning method, since the action space in the target tracking reinforcement learning framework is continuous, the Deep Deterministic Policy Gradient (DDPG) method can be directly used for agent policy learning. Alternatively, the Deep Q-Network (DQN) can be used to learn the agent policy by discretizing the action space.

[0056] Step 8.4: During the strategy learning process, multiple complete interaction processes can be performed until the agent strategy meets the requirements, that is, the target trajectory output in step 9 meets the requirements and can achieve target tracking;

[0057] Step 9: The target tracking reinforcement learning framework uses the optimized target tracking agent to interact with the environment. At each step of the interaction, the tracker motion state information output by the environment is used as the output of the target tracking reinforcement learning method. The tracker motion state information output at each moment is the target track obtained by the reinforcement learning method.

[0058] Step 9.1: The target tracking reinforcement learning framework uses the optimized target tracking agent to interact with the environment. At each step of the interaction, the tracker motion state information output by the environment is used as the output of the target tracking reinforcement learning method. The tracker motion state information output at each moment is the target track obtained by the reinforcement learning method. The tracker motion state information output at the initial state of the environment can be set according to step 6.6.

[0059] Step 9.2: A single agent can only track a single target. When tracking multiple targets, multiple optimized agents and environments can be copied, each environment is set to a different initial state, and then the agent and the environment interact one-to-one to achieve multi-target tracking.

[0060] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0061] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A target tracking reinforcement learning method based on a target tracking reinforcement learning framework, characterized in that: The following steps are involved: Step 1: The tracker in the target tracking reinforcement learning framework is defined as a unit that has the motion characteristics of the tracked target, is controlled by the agent and constrained by the target's motion characteristics, and moves in the detection environment; Step 2: The intelligent agent in the target tracking reinforcement learning framework is defined as the tracker motion decision maker. It controls the tracker motion according to the control strategy. Its input is the state and reward provided by the environment. The output is the action that the tracker should perform and feedback to the environment. Step 3: The executable action space in the target tracking reinforcement learning framework is constructed by the tracker motion controllable parameter variation space, and the specific action is selected from the executable action space; Step 4: The state in the target tracking reinforcement learning framework is the current motion state information of the tracker and the next moment measurement information obtained by the detection device. They are expressed in a unified inertial coordinate system. The measurement information output by the detection device consists of multiple measurement points. Each measurement point contains at least the observation time and spatial position information. The tracker motion state information contains at least time, spatial position, motion speed and motion direction information. Step 5: The reward in the target tracking reinforcement learning framework is measured by the distance between the tracker's motion state information and the detection device's measurement information at the same time. The farther the two are, the smaller the reward, and the closer the two are, the larger the reward. Step 6: The environment in the target tracking reinforcement learning framework is constructed by the detection device, the detection environment, and the tracker. Its input is the action and its output is the state and reward. Step 7: In the target tracking reinforcement learning framework, the agent interacts continuously with the environment. A complete interaction process starts from the initial state of the environment and ends at the final state of the environment, including multiple steps of interaction. In each step of interaction, the agent receives the state and reward output by the environment and outputs the current action decision based on the control strategy. The environment receives the current action output by the agent and outputs the state and reward. Step 8: The optimization method of the intelligent agent in the target tracking reinforcement learning framework is to maximize the cumulative reward. The policy learning method is used to continuously optimize the control strategy of the intelligent agent through the interaction between the intelligent agent and the environment to obtain a target tracking intelligent agent that meets the requirements. Step 9: The target tracking reinforcement learning framework uses the optimized target tracking agent to interact with the environment. At each step of interaction, the tracker motion state information output by the environment is used as the output of the target tracking reinforcement learning method. The tracker motion state information output at each moment is the target track obtained by the reinforcement learning method.

2. A target tracking reinforcement learning method based on a target tracking reinforcement learning framework according to claim 1, characterized in that: The reward in step 5 is calculated as follows: First, the distance between the tracker's spatial position and each measurement point of the detection device is calculated in sequence, and then the reward r of the tracker at each measurement point is calculated based on this. If the distance between the tracker's spatial position and the i-th measurement point of the detection device is greater than the no reward threshold g r , then its reward r i Set to 0, if it is less than the no reward threshold, the reward is set to Finally, the rewards at each measurement point are summed to obtain the total reward at the current moment R = ∑r i , where (x0, y0) is the spatial position of the tracker at the current moment, (x i ,y i ) is the spatial position of the measurement point i; Set to the measurement covariance of the detection equipment measurement point i, or set manually; g r Set to Or set it manually.

3. The target tracking reinforcement learning method based on the target tracking reinforcement learning framework according to claim 1, characterized in that: The strategy learning method used in step 8 is: Directly adopt the deep deterministic policy gradient method for agent policy learning, or use the deep Q learning network for agent policy learning by discretizing the executable action space.

Images