Radar human body action recognition data cost optimization method based on reinforcement learning

Abstract

The invention discloses a radar human body action recognition data cost optimization method based on reinforcement learning, and belongs to the technical field of radar intelligent sensing. According to the method, through reinforcement learning of Markov chain construction of an agent interaction mechanism and a sample field deletion strategy, a data acquisition cost optimization problem is modeled as a sample field deletion optimization problem, and an optimal deletion strategy is learned through maximizing agent accumulation rewards; and the collaborative goal of minimizing the performance loss of the classifier and maximally reducing the acquisition cost is achieved. In the environment-agent interaction process, a dynamic reward scaling mechanism based on real distribution of classification indexes is designed, and accurate regulation and control of a reward function are achieved; meanwhile, an index cache region is constructed, classifier performance fluctuation data corresponding to the same action are reused, and interaction time consumption is remarkably shortened. Compared with the prior art, the method has higher optimization degree of freedom, and excellent human body action recognition and classification performance can be maintained while the radar omnidirectional data acquisition cost is greatly reduced.

Description

Technical Field

[0001] This invention relates to the field of radar intelligent sensing technology, specifically to a data cost optimization method for radar human motion recognition based on reinforcement learning. Background Technology

[0002] Human motion recognition, as a core application of target detection technology, plays an irreplaceable role in key areas such as intelligent monitoring, autonomous driving, intelligent security, and assisted driving. In intelligent monitoring scenarios, accurately identifying abnormal human movements (such as climbing, fighting, and loitering) is the core support for achieving safety warnings and preventing dangerous events. In the field of autonomous driving, real-time perception of pedestrians' walking, running, and crossing of roads is a key prerequisite for ensuring the accuracy of vehicle obstacle avoidance decisions and reducing the incidence of traffic accidents.

[0003] Compared to traditional optical cameras, radar sensors exhibit significant advantages in human motion recognition scenarios: First, radar sensors are not limited by lighting conditions and can still work stably in adverse weather conditions or low-light environments such as darkness, heavy rain, and dense fog, avoiding the recognition failure problem caused by insufficient light in cameras; Second, radar has a certain obstacle penetration capability, and can penetrate walls and obstructions to achieve non-contact perception, adapting to more complex scenario requirements; Third, radar does not need to collect human visual images, but only obtains motion information through electromagnetic wave reflection, which can effectively protect personal privacy and meet relevant privacy protection regulations and social needs.

[0004] However, radar sensors are angle-sensitive, meaning that the radial velocity component changes in different directions for the same action, resulting in a Doppler frequency shift in the spectrum and significant differences between Doppler features (e.g., Figure 1 As shown). To achieve ideal recognition accuracy, the human motion recognition classifier requires radar data from all directions for training, resulting in high data costs (e.g., Figure 2 (As shown). Existing research on reducing data costs mainly uses techniques such as data generation, active learning, and few-shot learning to achieve high recognition performance with a relatively small amount of data. However, they only reduce the amount of data used; omnidirectional data is still required during training, resulting in the radar data acquisition cost remaining unchanged. Therefore, it is essential to study how to reduce acquisition costs while ensuring measurement accuracy.

[0005] Reinforcement learning (RL), an important branch of machine learning, is based on the core idea that an agent continuously interacts with the environment and learns an optimal policy based on reward signals from the environment, in order to maximize long-term cumulative rewards (such as...). Figure 3(As shown). Unlike supervised and unsupervised learning, reinforcement learning does not require a large amount of pre-provided labeled data. Instead, it evolves itself in a dynamic environment through a "trial and error" mechanism, making it particularly suitable for complex problems such as sequential decision-making, resource optimization, and adaptive control. In reinforcement learning frameworks, Markov Decision Processes (MDPs) are typically used for formal description. An MDP consists of a quintuple (...). S , A , P , R , ) constitutes, among which S For state space, A For the action space, Let be the state transition probability. For instant reward function, This is a discount factor used to weigh the importance of current rewards against future rewards. The agent's goal is to find an optimal policy that maximizes the expected long-term cumulative reward from the initial state. To evaluate the effectiveness of a strategy, a state-value function is often introduced. and action value function , respectively representing the state s (or state-action pair) s , a Begin by following the strategy. The expected cumulative return that can be obtained satisfies the Bellman expectation equation:

[0006] The optimal action value function that maximizes the expected long-term cumulative reward. Satisfying the Bellman optimal equation:

[0007] This equation reveals the recursive relationship between the current optimal value and the possible optimal value of the next state, and is the theoretical foundation of many reinforcement learning algorithms. Traditional Q-Learning algorithms approximate the optimal value through iterative updates. Its update rules are as follows:

[0008] in The learning rate is used; however, when the state or action space dimension is too large, traditional tabular Q-learning faces the "curse of dimensionality" problem, making it difficult to store and generalize effectively. The introduction of Deep Q-Network (DQN) is a significant breakthrough in reinforcement learning, utilizing deep neural networks... To parameterize approximation Q Function, i.e. This allows it to handle high-dimensional, continuous state spaces. The core innovation of DQN lies in the introduction of two major mechanisms: Experience Replay and Target Network. Experience Replay stores the agent's interaction experience... s t , a t , r t , s t+1 The data is stored in a replay buffer and randomly sampled during training, breaking the temporal correlation between data and improving data utilization efficiency and learning stability. The target network uses a network with the same structure but with delayed parameter updates (the parameters are denoted as...). This method calculates the Temporal-Difference Target (TD), reducing the number of targets. Q Value and current estimate Q The correlation fluctuations between values ​​stabilized the training process.

[0009] Based on the above, this invention proposes a data cost optimization method for radar human motion recognition based on reinforcement learning. Summary of the Invention

[0010] The purpose of this invention is to propose a data cost optimization method for radar human motion recognition based on reinforcement learning to solve the problems mentioned in the background art. This invention can optimize the data cost of radar human motion recognition. By utilizing the exploration and learning capabilities of reinforcement learning algorithms, it has successfully explored the optimal sample domain combination for classifier metrics under different data domain quantity requirements.

[0011] To achieve the above objectives, the present invention is implemented through the following technical solution: A data cost optimization method for radar human action recognition based on reinforcement learning includes the following steps: S1. Acquire radar omnidirectional data, divide it into training set and test set, and preprocess the training set; S2. Construct a classifier for human motion recognition; S3. Constructing a reinforcement learning environment, specifically including: S31. Determine the state space and action space based on the total number of sample domains. The initial state is a zero tensor with a length equal to the number of sample domains. When a domain is selected, the corresponding element in the state is set to 1, indicating that the corresponding domain has been selected, and then it is removed from the training set. S32. Design the reward function and the mouse buffer for storing ACC; the original reward value is the difference between the metric obtained by training the classifier on the training set after deleting a sample neighborhood and the metric obtained by training the classifier on the training set before deleting the sample neighborhood. S33. Design a reward scaling function, continuously collect the original reward value and calculate the quantile value, and successively scale the original value of the reward function to the interval of -1 to 1 to match the agent's predicted Q value. S4. Construct an agent based on a value function reinforcement learning algorithm, which predicts the value of each action according to the current state and selects the action with the highest value by default; the agent uses DQN, which includes an online network and a target network, wherein the online network is used to directly predict the value of the current action, and the target network is used to predict the future value of the current action; S5. Interact with and update the intelligent agent, specifically including: S51. Execute multiple training rounds. Each training round starts from an initial state. The agent selects an action to delete a sample domain according to the current state. The environment provides feedback rewards and moves to the next state. Experience data is generated and stored in the experience replay pool until the preset number of sample domains deleted is reached. S52. Sample data from the experience replay pool, calculate the loss between the value estimate and the value target, and update the parameters of the online network based on this loss; S53. After each fixed number of training rounds, copy the parameters of the online network to the target network. S6. Optimize the data cost of radar human action recognition by performing a sample domain deletion sequence using the online network weights of the strategy with the highest cumulative reward, and obtain the training set sample domain combination that optimizes the performance of the classifier under the current sample domain number constraint.

[0012] The classifier constructed in S2 uses any one of KimNet, ResNet, or VGGNet neural networks.

[0013] The metric obtained from training the classifier as described in S32 is the classification accuracy.

[0014] The experience data described in S51 is stored in the form of a quadruple (state, action, reward, next_state).

[0015] The loss-based online network parameter update in S52 specifically includes: Based on the empirical data obtained from sampling, the Q-value estimate of the current state-action pair is calculated by the online network. This estimate is then combined with the maximum Q-value of the next state calculated by the target network and the immediate reward to obtain the Q-value target. The parameters of the online network are updated by minimizing the loss between the Q-value estimate and the Q-value target. After each fixed number of training iterations, the parameters of the online network are copied to the target network to update it.

[0016] The strategy with the highest cumulative reward mentioned in S6 specifically refers to recording the cumulative reward obtained at the end of each training round during the training process, and selecting the agent strategy corresponding to the round with the largest cumulative reward value.

[0017] The present invention further protects a computer device, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the instruction, program, code set or instruction set being loaded and executed by the processor to implement the above-mentioned reinforcement learning-based radar human motion recognition data cost optimization method.

[0018] The present invention further protects a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the instruction, program, code set, or instruction set is loaded and executed by a processor to implement the above-mentioned reinforcement learning-based radar human motion recognition data cost optimization method.

[0019] Compared with the prior art, the present invention has the following beneficial effects: This invention proposes a data cost optimization method for radar human action recognition based on reinforcement learning. By constructing a Markov chain using a reinforcement learning agent interaction mechanism and a sample domain deletion strategy, the data acquisition cost optimization problem is modeled as a sample domain deletion optimization problem. The optimal deletion strategy is learned by maximizing the agent's cumulative reward, achieving the synergistic goal of minimizing classifier performance loss and maximizing the reduction of acquisition cost. During the environment-agent interaction process, a dynamic reward scaling mechanism based on the true distribution of classification indicators is designed to achieve precise control of the reward function. Simultaneously, an indicator cache is constructed to reuse classifier performance fluctuation data corresponding to the same action, significantly shortening the interaction time. Compared to data cost optimization strategies based on human prior knowledge, this method, through the autonomous exploration of sample domain deletion schemes by the reinforcement learning agent, possesses higher optimization freedom. It can significantly reduce the radar omnidirectional data acquisition cost while maintaining excellent human action recognition classification performance, providing efficient support for the engineering application of radar intelligent sensing scenarios. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings involved in the embodiments are now briefly described. Obviously, the drawings in the following description are merely illustrative of some embodiments of the present invention. For those skilled in the art, other forms of drawings can be constructed based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram illustrating the differences between Doppler features at different angles mentioned in this invention; Figure 2 This is a schematic diagram illustrating the cost of omnidirectional data acquisition mentioned in this invention; Figure 3 This is a schematic diagram illustrating the interaction between reinforcement learning and the environment mentioned in this invention; Figure 4 Modeling the RL interaction environment for the data cost optimization problem mentioned in this invention; Figure 5 As mentioned in this invention Network diagram Figure 6 This is a schematic diagram of the ResNet18 network structure mentioned in this invention. Detailed Implementation

[0022] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0023] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0024] Example 1: Please see Figure 1 , Figure 1 Each row represents a radar spectrum captured from different angles for a single action. Rows one through six represent: "Walking," "Running," "Jumping," "Boxing," "Crouching Forward," and "Crawling." Figure 1 It can be seen that the intensity of the Doppler effect decreases when the direction of human movement deviates from radial to tangential, especially when the subject moves tangentially (a=90° or 270°), the Doppler characteristics almost disappear. Although the spectra of the same movement at viewpoint a and 360° are extremely similar, subtle differences exist. This is because human movement is not strictly symmetrical, therefore the micro-Doppler spectra obtained from different viewpoints are not entirely identical.

[0025] Please see Figure 2Acquiring omnidirectional data requires the collector to perform the same action in 12 different directions, instead of only facing the radial direction of the radar as in traditional methods, resulting in high acquisition costs.

[0026] To address the aforementioned issues, this invention proposes a data cost optimization method for radar human motion recognition based on reinforcement learning. This method aims to optimize the data cost of radar human motion recognition by exploring radar domain combinations that maximize classifier performance under specific domain quantity requirements. This approach reduces radar data acquisition costs while maintaining high classifier performance.

[0027] The core of this invention lies in utilizing the exploration and learning capabilities of reinforcement learning algorithms to explore the optimal combination of sample domains for classifier metrics under different data domain quantity requirements. Specifically, it includes the following steps: Step 1: Read the radar omnidirectional data and divide it into training and test sets. Then, divide the training set into several sample domains in the form of a certain angle and a certain collector according to different angles and different collectors, while the test set remains unchanged.

[0028] Step 2: Construct the classifier and its loop function. The neural network used for the classifier can be a well-known network such as KimNet, ResNet, or VGGNet to verify the generalization ability of this method on different classifiers.

[0029] Step 3: A reinforcement learning environment is constructed. The state space and action space are determined based on the number of sample domains. A reward function and a mouse buffer for storing ACC (Accuracy of Action) are designed. The initial state is a zero-based tensor with a length equal to the number of sample domains. When a domain is selected, the corresponding element in the state is set to 1, indicating that the domain is selected and removed from the training set. The original reward value is the difference between the metric obtained by training the classifier on the training set after removing the domain and the metric obtained by training the classifier on the training set before removing the domain. Since this value is negative and between -0.1 and 0, it does not conform to the reward design of reinforcement learning. Therefore, a scaling function is designed to continuously collect the original reward value and calculate the quantile value, successively scaling the original reward value to the interval between -1 and 1 to match the agent's predicted Q-value.

[0030] Step 4: Constructing a reinforcement learning agent. This invention employs a value function-based reinforcement learning algorithm, with DQN selected as the agent. It can predict the value of each action based on the current state and defaults to selecting the action with the highest value. It comprises two networks: an online network and a target network. The online network directly predicts the value of the current action, while the target network predicts the future value of the current action.

[0031] Step 5: The interaction begins and the agent is updated. Each epoch executes one episode, the length of which is variable, determined by the number of samples in the neighborhood to be deleted. Within each episode, starting from the initial state, actions are selected sequentially, and state transitions occur until the episode ends, at which point the cumulative reward for that episode is obtained. Each state transition saves an experience quadruple (state, action, reward, next_state) to the experience replay pool and executes the agent update procedure. During agent updates, the pool is first checked for sufficiency, and then a batch of experience is extracted according to preset rules. Based on the state and action from the experience, the estimated value of choosing the current action in the current state is determined, the reward determines its true value, and the next_state determines its future value. Finally, the loss is calculated based on the distance between the estimated value and the sum of the true and future values, and this is fed back to update the online network. The target network is not updated along with the online network; instead, its weights are directly copied from the online network after a certain number of updates.

[0032] Step 6: Optimize the data cost of radar human action recognition. Execute an episode using the online network weights of the epoch with the highest cumulative reward to obtain the training set sample domain combination that maximizes classifier performance within the current sample domain size limit.

[0033] More specifically, including the following technical details 1. Interaction between the sample optimization environment and the reinforcement learning agent To model the data cost optimization process as an environment that can be interacted with by reinforcement agents, this invention designs the data cost optimization process as a process that starts from the complete training set and gradually removes sample neighborhoods that cause minimal performance degradation of the classifier, such as... Figure 4 As shown.

[0034] First, the state space is constructed as follows:

[0035] Where k is the number of sample domains, and in this invention, k = 12 angles. 12 collectors = 144 sample domains; The value is either 0 or 1, where 0 represents that the current sample neighborhood has not been selected, and 1 represents that it has been selected. The initial state is state={0,0,…,0}. The action space is constructed as follows:

[0036] The action space consists of k numbers from 0 to k-1, representing all sample domains. The environment selects actions based on a greedy-exploration strategy, performing a randomness check before each interaction. If a randomness check is performed, an action is randomly selected from the action space; otherwise, the action is selected based on the agent's knowledge, as follows:

[0037] in, The structure of the network is as follows Figure 5 As shown, This represents the agent's estimated value for all possible actions in the current state. The DQN algorithm, by default, selects the index with the highest q-value as the action, as follows:

[0038] After selecting an action, the corresponding sample domain is removed from the training set, and a classifier is trained to obtain the ACC metric. The reward is calculated as follows:

[0039] in, The ACC metric is obtained by training the classifier to remove the training set before the current sample's neighborhood. This invention uses ResNet18 as the classifier, and its network structure is as follows: Figure 6 As shown. Deleting samples typically leads to a decrease in classifier performance, thus causing... Almost all values ​​are negative. The larger the value, the smaller the performance degradation caused by deleting the current sample domain, and the higher the contribution of that domain; conversely, the smaller the value, the lower the contribution of that domain.

[0040] After an agent performs an action and receives a reward, its current state will transition to the next state based on the action value: This refers to `next_state`. At this point, the environment has undergone one state transition, saving the experience quadruple (state, action, reward, next_state) to the experience replay pool, and executing the agent update procedure. The experience of the batch size is extracted, and the estimated Q-value for the corresponding action is calculated as follows:

[0041] in, This represents the agent's estimated value for the current state-action pair. The estimated Q-value for the next state is then calculated as follows:

[0042] in, This represents the estimated value of all actions that can be taken in the next state. (Estimated Q-value) The learning objective is the sum of the actual reward value and the future reward value, as follows:

[0043] in, This is the discount factor. The DQN algorithm by default uses the maximum estimated Q-value of the next state to represent the future reward. The final loss is calculated as follows:

[0044] in, This represents the mean square error loss.

[0045] 2. Metric caching module for accelerating iteration Looking back at the definition of the reward function, a classifier needs to be trained every time a state is obtained and a state transition is performed. This means that the classifier will be trained as many times as the agent interacts during the entire RL training process, resulting in a large amount of training time cost.

[0046] This paper proposes an ACC caching module to reduce the number of times the classifier is repeatedly trained, thereby reducing interaction time. When an environment-agent interaction results in a state-action pair, it checks whether the pair has appeared in the cache. If the number of occurrences, m, does not exceed the repetition factor M, the classifier needs to be trained and the ACC value stored in the cache; otherwise, the average of historical values ​​is directly used, and retraining is unnecessary.

[0047] 3. Adaptive scaling function for reward normalization Since the original reward values ​​are typically between -0.1 and 0, they do not meet the normalization requirements of neural networks. This invention proposes an adaptive reward scaling function. This function dynamically adjusts the scaling boundary to map the original reward values ​​to the target interval [-L, L]. First, a sufficient number of original reward values ​​are collected. When the number of historical samples reaches a preset threshold min_samples, based on percentiles... The upper boundary h and lower boundary l are calculated as follows:

[0048]

[0049] Where H represents the original reward value geometry, and p is the percentile parameter. To prevent scaling failure due to excessively narrow boundaries, a boundary width threshold is set. =1e-06, when Forced Then, based on the current original reward x, its scaling value is calculated as follows:

[0050] This design adapts to changes in reward distribution by dynamically adjusting the boundaries, ensuring that the scaled reward value remains within the effective range while avoiding scaling distortion caused by fixed boundaries.

[0051] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A data cost optimization method for radar human motion recognition based on reinforcement learning, characterized in that, Includes the following steps: S1. Acquire radar omnidirectional data, divide it into training set and test set, and preprocess the training set; S2. Construct a classifier for human motion recognition; S3. Constructing a reinforcement learning environment, specifically including: S31. Determine the state space and action space based on the total number of sample domains. The initial state is a zero tensor with a length equal to the number of sample domains. When a domain is selected, the corresponding element in the state is set to 1, indicating that the corresponding domain has been selected, and then it is removed from the training set. S32. Design the reward function and the indicator cache for storing ACC; the original reward value is the difference between the indicator obtained by training the classifier on the training set after deleting a sample neighborhood and the indicator obtained by training the classifier on the training set before deleting the sample neighborhood. S33. Design a reward scaling function, continuously collect the original value of the reward function and calculate the quantile value, and successively scale the original reward value to the range of -1 to 1 to match the agent's predicted Q value. S4. Construct an agent based on a value function reinforcement learning algorithm, which predicts the value of each action according to the current state and selects the action with the highest value by default; the agent uses DQN, which includes an online network and a target network, wherein the online network is used to directly predict the value of the current action, and the target network is used to predict the future value of the current action; S5. Interact with and update the intelligent agent, specifically including: S51. Execute multiple training rounds. Each training round starts from an initial state. The agent selects an action to delete a sample domain according to the current state. The environment provides feedback rewards and moves to the next state. Experience data is generated and stored in the experience replay pool until the preset number of sample domains deleted is reached. S52. Sample data from the experience replay pool, calculate the loss between the value estimate and the value target, and update the parameters of the online network based on this loss; S53. After each fixed number of training rounds, copy the parameters of the online network to the target network. S6. Optimize the data cost of radar human action recognition by performing a sample domain deletion sequence using the online network weights of the strategy with the highest cumulative reward, and obtain the training set sample domain combination that optimizes the performance of the classifier under the current sample domain number constraint.

2. The method according to claim 1, characterized in that, The classifier constructed in S2 uses any one of KimNet, ResNet, or VGGNet neural networks.

3. The method according to claim 1, characterized in that, The metric obtained from training the classifier as described in S32 is the classification accuracy.

4. The method according to claim 1, characterized in that, The experience data described in S51 is stored in the form of a quadruple (state, action, reward, next_state).

5. The method according to claim 1, characterized in that, The loss-based online network parameter update in S52 specifically includes: Based on the empirical data obtained from sampling, the Q-value estimate of the current state-action pair is calculated by the online network. This estimate is then combined with the maximum Q-value of the next state calculated by the target network and the immediate reward to obtain the Q-value target. The parameters of the online network are updated by minimizing the loss between the Q-value estimate and the Q-value target. After each fixed number of training iterations, the parameters of the online network are copied to the target network to update it.

6. The method according to claim 1, characterized in that, The strategy with the highest cumulative reward mentioned in S6 specifically refers to recording the cumulative reward obtained at the end of each training round during the training process, and selecting the agent strategy corresponding to the round with the largest cumulative reward value.

7. A computer device, characterized in that, The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the instruction, program, code set, or instruction set is loaded and executed by the processor to implement the data cost optimization method for radar human motion recognition based on reinforcement learning as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, at least one program, code set, or instruction set, which is loaded and executed by a processor to implement the data cost optimization method for radar human motion recognition based on reinforcement learning as described in any one of claims 1-6.

Images