A sensor node selection method and system based on hybrid deep reinforcement learning
By combining deep reinforcement learning methods with virtual training and real-world interaction, the computational complexity and real-time performance issues of node selection in underwater sensor networks are addressed, achieving efficient node selection and target tracking.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JIANGSU UNIV OF SCI & TECH
- Filing Date
- 2026-03-18
- Publication Date
- 2026-06-19
AI Technical Summary
In underwater sensor networks, node selection methods are computationally complex and difficult to meet real-time requirements. Furthermore, online reinforcement learning lacks training efficiency and policy robustness in underwater scenarios.
A hybrid deep reinforcement learning approach is adopted, combining offline virtual training with online real interaction. Virtual training samples are generated to augment data and a hybrid deep Q-network (Hybrid-DQN) is constructed to achieve dynamic adaptation of node selection and policy optimization.
It improves training efficiency and convergence speed, reduces the number of activated nodes, meets the requirements of target tracking accuracy and real-time performance, and reduces energy consumption and communication overhead.
Smart Images
Figure CN122241358A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of underwater sensor networks and intelligent information processing, and relates to underwater sensor node selection technology, specifically to a sensor node selection method and system based on hybrid deep reinforcement learning. Background Technology
[0002] Underwater sensor networks can achieve large-scale, long-term tracking of underwater targets through multi-node collaborative sensing. However, due to limitations such as limited communication bandwidth and difficulties in energy replenishment, node failure caused by excessive energy consumption will affect the network's monitoring coverage and tracking continuity. Therefore, dynamically selecting the optimal sensor nodes to reduce network resource consumption while ensuring target tracking accuracy is a crucial technical challenge in this field.
[0003] Existing node selection methods mainly include information-driven heuristic methods and optimization-driven combinatorial optimization methods. Since node selection is essentially a combinatorial optimization problem, the action space grows exponentially with the node size. Traditional exact optimization methods have high computational complexity, making them difficult to meet real-time requirements. Online reinforcement learning can learn strategies through interaction, but underwater scenarios often face challenges such as scarce real-world interaction data, high sampling costs, and rapidly changing environments, leading to insufficient training efficiency and policy robustness. Summary of the Invention
[0004] Purpose of the invention: In order to overcome the shortcomings of the existing technology, a sensor node selection method and system based on hybrid deep reinforcement learning is provided. By organically integrating offline virtual training and online real interaction, sample amplification and dynamic adaptation are achieved, thereby minimizing the number of activated nodes and compressing the node selection decision time to the millisecond level while meeting the target tracking accuracy constraint.
[0005] Technical Solution: To achieve the above objectives, this invention provides a sensor node selection method based on hybrid deep reinforcement learning, comprising the following steps:
[0006] S1: Establish the target state model and observation model, and output the target state estimate and estimation error covariance based on the observation data of the activated node at each real observation time;
[0007] S2: Generate virtual training samples between adjacent real observation times;
[0008] S3: Establish a node selection optimization model based on the target tracking accuracy evaluation index. The node selection optimization model aims to minimize the number of selected nodes while satisfying the tracking accuracy constraint.
[0009] S4: Map the node selection optimization model to a reinforcement learning problem, and construct the state space, action space and reward function;
[0010] S5: Construct a hybrid deep Q network Hybrid-DQN and train Hybrid-DQN based on a hybrid experience replay mechanism that includes virtual training samples and online real interaction samples;
[0011] S6: At each real observation time, based on the activation decision of the trained Hybrid-DQN output node, obtain the observation data of the activated node and perform target state estimation, and output the node selection result.
[0012] Furthermore, in step S1, the target state model is a discrete-time motion model, and the target state includes at least two-dimensional position and velocity components, specifically expressed as follows:
[0013] Suppose a target moves within a region of water depth; its state equation can be expressed as:
[0014]
[0015] in, Let k be the motion state of the target at time k. Let the target be located at time k. The velocity of the target in the x and y directions; For dynamic noise, , Let be the covariance matrix of the state noise;
[0016] When the target is making a uniform turning motion, its state transition matrix is:
[0017]
[0018] Where T is the time interval during the tracking process. Turning speed;
[0019] Covariance matrix of state noise for:
[0020]
[0021] Where q is the intensity of the process noise. This represents the Kronecker product.
[0022] Furthermore, the observation model in step S1 is a nonlinear mapping model from the location of the observation node to range observation, azimuth observation, or a combined range-azimuth observation, specifically expressed as follows:
[0023] Assuming the coordinates of the sensor nodes are fixed and known, the measurement value of node i at time k during the target's motion is expressed as:
[0024]
[0025] in:
[0026]
[0027]
[0028] in, Indicates the position coordinates of node i; Let represent the measurement noise of the distance and angle at node i during the measurement process; assuming the measurement noise is independent and identically distributed, where , , and These are the noise variances of distance and angle during the observation process, respectively.
[0029] Furthermore, the acquisition of target state estimation and estimation error covariance in step S1 includes:
[0030] After obtaining measurement information, each sensor node transmits the information to the fusion center, which then filters and estimates the target. Assuming there are N sensor nodes in total, the observation data obtained by the fusion center at time k is represented in the following form: , , ;
[0031] After acquiring the measurement data, an extended Kalman filter is used to fuse the data to estimate the target position, thereby obtaining the target's motion state during the tracking process. The mean and covariance of the target state estimate are expressed as:
[0032]
[0033] in, This represents the estimated state of the target at time k. Indicates Kalman gain, The Jacobian matrix represents the equation of observation. Let k represent the prior covariance matrix at time k. Let represent the posterior covariance matrix at time k; , and The calculation expressions are as follows:
[0034]
[0035]
[0036] .
[0037] Further, step S2 includes:
[0038] A1: Insert M virtual time steps between adjacent real observation times;
[0039] A2: Predict or sample virtual future states based on prior knowledge of target motion;
[0040] A3: Generating virtual observation data based on the observation model and prior measurement noise;
[0041] A4: Write the virtual future state and virtual observation data into the experience replay pool to construct virtual training samples.
[0042] Furthermore, the target tracking accuracy evaluation index in step S3 includes the posterior Cramer-Rao bound (PCRLB); the tracking accuracy constraint is that the trace, weighted trace, or position dimension error bound of the PCRLB does not exceed a preset threshold.
[0043] Methods for calculating the posterior Cramér-Rao boundary PCRLB:
[0044] The Bayesian Craméro bound provides a lower bound for the estimation of the target state; PCRLB is expressed as:
[0045]
[0046] in, The target state estimate, Let the Bayesian information matrix of the target at time k be represented as:
[0047]
[0048] in, Represents the second-order partial derivative. It is divided into two parts: prior information and measurement information.
[0049]
[0050] in, The Fisher Information Matrix (FIM) represents prior information. The contribution of the measurement data to FIM can be expressed as:
[0051]
[0052] Among them, the first-order partial derivatives of the observation equation The Jacobian matrix represents the measurement equation, while the variance of the measurement noise is represented by the diagonal matrix. Composition; the FIM representation of the measurement data is as follows:
[0053]
[0054] The Jacobian matrix of the measurement equation is calculated as follows:
[0055]
[0056] Independent information contribution of target location x estimation for:
[0057]
[0058] Similarly, the independent information contribution of the target location y estimation The calculation yielded:
[0059]
[0060] Contribution of relevant information to the estimation of target location x and y and The calculations include:
[0061]
[0062] Since the partial derivative with respect to velocity is 0, the FIM representation of the measured data is:
[0063]
[0064] The prior information FIM is given by the state prediction covariance matrix:
[0065]
[0066] In summary, the posterior Cramer-Rao bound for target tracking at time k is obtained as follows:
[0067] .
[0068] Furthermore, in step S4, the node selection optimization model is a combined optimization model, the decision variables are the binary activation variables of each observation node, and the objective function is to minimize the number of activated nodes or their weighted sum.
[0069] The state space includes target position estimation, target velocity estimation, and covariance matrix elements, covariance trace, and equivalent statistics that characterize the uncertainty of the estimation.
[0070] The action space is a binary vector of length N, where N is the number of observation nodes, and each dimension of the vector corresponds to the activation or deactivation of an observation node.
[0071] The reward function includes a precision constraint penalty term and a node cost term. When PCRLB exceeds a preset threshold, the penalty weight of the precision constraint penalty term is increased to encourage Hybrid-DQN to select more effective nodes to meet the precision requirements.
[0072] Furthermore, the method for training Hybrid-DQN in step S5 based on a hybrid experience replay mechanism that includes virtual training samples and online real interaction samples includes:
[0073] In Q-learning, the Q-value is used to represent the expected cumulative reward for taking a certain action in a given state. The Q-value is calculated as follows:
[0074]
[0075] in, For learning rate, The discount factor; this invention uses the current network. and target network During training, the current network interacts with the environment to perform actions and obtains Q-values. The Q-value of the target network is calculated as follows:
[0076]
[0077] The goal of training is to minimize the error between the current network's Q-value and the target Q-value. The network loss can be expressed as:
[0078]
[0079] Update the current network using gradient descent. And use soft updates to adjust the target network:
[0080]
[0081] in, These are soft update coefficients; to avoid the algorithm getting trapped in local optima, random actions are added during the exploration process, namely:
[0082]
[0083] Where p is a randomly generated probability value. It is a value that gradually decreases with the number of training iterations.
[0084] The present invention also provides a sensor node selection system based on hybrid deep reinforcement learning, comprising:
[0085] The model building module is used to build the target state model and the observation model, and outputs the target state estimate and the estimation error covariance based on the observation data of the activated nodes at each real observation time.
[0086] The virtual sample generation module generates virtual training samples between adjacent real observation times;
[0087] The accuracy constraint modeling module establishes a node selection optimization model based on the target tracking accuracy evaluation index. The node selection optimization model aims to minimize the number of selected nodes while satisfying the tracking accuracy constraint.
[0088] The reinforcement learning modeling module maps the node selection optimization model to a reinforcement learning problem, and constructs the state space, action space and reward function.
[0089] The Hybrid Deep Q Network training module constructs a Hybrid Deep Q Network (Hybrid-DQN) and trains it based on a hybrid experience replay mechanism that includes virtual training samples and online real interaction samples.
[0090] The online decision-making module, at each real observation moment, makes activation decisions based on the trained Hybrid-DQN output node, acquires the observation data of the activated node, estimates the target state, and outputs the node selection result.
[0091] Beneficial effects: Compared with the prior art, the present invention has the following advantages:
[0092] 1. Virtual sample augmentation alleviates the scarcity of real data, improving training efficiency and convergence speed;
[0093] 2. Enhance the interpretability and accuracy assurance of the strategy through PCRLB accuracy constraints;
[0094] 3. Hybrid training balances strategy generalization and environmental adaptation;
[0095] 4. While maintaining tracking accuracy, reduce the number of active nodes, lower energy consumption and communication overhead, and meet real-time requirements. Attached Figure Description
[0096] Figure 1 This is a simplified flowchart of the method of the present invention;
[0097] Figure 2 This is a framework diagram of the Hybrid-DQN algorithm;
[0098] Figure 3 A schematic diagram illustrating the deployment of underwater sensor network nodes and target trajectory / filtering estimation;
[0099] Figure 4 The average reward change between the Online-DQN algorithm and the Hybrid-DQN algorithm;
[0100] Figure 5 A diagram showing the comparison of tracking performance of different algorithms (including changes in RMSE and PCRLB over time).
[0101] Figure 6A diagram illustrating how the number of nodes selected for different algorithms changes over time;
[0102] Figure 7 A diagram showing the time consumption for node selection for different algorithms. Detailed Implementation
[0103] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.
[0104] Example 1:
[0105] This embodiment provides a sensor node selection method based on hybrid deep reinforcement learning, applied to target tracking in an underwater sensor network. The underwater sensor network includes N observation nodes and a fusion center. The positions of the observation nodes are known and fixed. When activated, they acquire target range / azimuth or range-azimuth joint observations and upload them to the fusion center. The fusion center performs data fusion, target state estimation, and node selection decisions, and issues activation commands to the observation nodes.
[0106] like Figure 1 As shown, the sensor node selection method includes the following steps:
[0107] S1: Establish the target state model and observation model, and output the target state estimate and estimation error covariance based on the observation data of the activated node at each real observation time;
[0108] The target state model is a discrete-time motion model, and the target state includes at least two-dimensional position and velocity components, as specifically expressed below:
[0109] Suppose a target moves within a region of water depth; its state equation can be expressed as:
[0110]
[0111] in, Let k be the motion state of the target at time k. Let the target be located at time k. The velocity of the target in the x and y directions; For dynamic noise, , Let be the covariance matrix of the state noise;
[0112] When the target is making a uniform turning motion, its state transition matrix is:
[0113]
[0114] Where T is the time interval during the tracking process. Turning speed;
[0115] Covariance matrix of state noise for:
[0116]
[0117] Where q is the intensity of the process noise. This represents the Kronecker product.
[0118] The observation model is a nonlinear mapping model from the location of the observation node to range observations, azimuth observations, or a combination of range and azimuth observations, specifically expressed as follows:
[0119] Underwater sensor networks typically employ multiple underwater acoustic sensors to track targets. Assuming the coordinates of the sensor nodes are fixed and known, the measurement value of node i at time k during the target's movement is expressed as:
[0120]
[0121] in:
[0122]
[0123]
[0124] in, Indicates the position coordinates of node i; Let represent the measurement noise of the distance and angle at node i during the measurement process; assuming the measurement noise is independent and identically distributed, where , , and These are the noise variances of distance and angle during the observation process, respectively.
[0125] The estimation of the target state and the acquisition of the estimation error covariance include:
[0126] After obtaining measurement information, each sensor node transmits the information to the fusion center, which then filters and estimates the target. Assuming there are N sensor nodes in total, the observation data obtained by the fusion center at time k is represented in the following form: , , ;
[0127] After acquiring the measurement data, an extended Kalman filter is used to fuse the data to estimate the target position, thereby obtaining the target's motion state during the tracking process. The mean and covariance of the target state estimate are expressed as:
[0128]
[0129] in, This represents the estimated state of the target at time k. Indicates Kalman gain, The Jacobian matrix represents the equation of observation. Let k represent the prior covariance matrix at time k. Let represent the posterior covariance matrix at time k; , and The calculation expressions are as follows:
[0130]
[0131]
[0132]
[0133] S2: Generate virtual training samples between adjacent real observation times;
[0134] Step S2 includes:
[0135] A1: Insert M virtual time steps between adjacent real observation times;
[0136] A2: Predict or sample virtual future states based on prior knowledge of target motion;
[0137] A3: Generating virtual observation data based on the observation model and prior measurement noise;
[0138] The virtual measurement value can be represented as: , where * indicates that the variable is dummy data.
[0139] A4: Write the virtual future state and virtual observation data into the experience replay pool to construct virtual training samples.
[0140] S3: Establish a node selection optimization model based on the target tracking accuracy evaluation index. The node selection optimization model aims to minimize the number of selected nodes while satisfying the tracking accuracy constraint.
[0141] The target tracking accuracy evaluation index includes the posterior Cramer-Rao bound (PCRLB); the tracking accuracy constraint is that the trace, weighted trace, or position dimension error bound of the PCRLB does not exceed a preset threshold.
[0142] Methods for calculating the posterior Cramér-Rao boundary PCRLB:
[0143] The Bayesian Craméro bound provides a lower bound for the estimation of the target state; PCRLB is expressed as:
[0144]
[0145] in, The target state estimate, The Bayesian information matrix of the target at time k can be represented as:
[0146]
[0147] in, Represents the second-order partial derivative. It can be divided into two parts: prior information and measurement information.
[0148]
[0149] in, The Fisher information matrix (FIM) represents prior information. The contribution of the measurement data to FIM can be expressed as:
[0150]
[0151] Among them, the first-order partial derivatives of the observation equation The Jacobian matrix represents the measurement equation, while the variance of the measurement noise is represented by the diagonal matrix. Composition. The FIM of the measured data can be expressed as:
[0152]
[0153] The Jacobian matrix of the measurement equation can be calculated as follows:
[0154]
[0155] Independent information contribution of target location x estimation for:
[0156]
[0157] Similarly, the independent information contribution of the target location y estimation It can be calculated that:
[0158]
[0159] Contribution of relevant information to the estimation of target location x and y and It can be calculated that:
[0160]
[0161] Because the partial derivative with respect to velocity is 0, the FIM of the measured data can be expressed as:
[0162]
[0163] The prior information FIM is given by the state prediction covariance matrix:
[0164]
[0165] In summary, the posterior Craméroe bound for target tracking at time k can be obtained as follows:
[0166]
[0167] Tracking accuracy constraints: This constraint ensures that the trace of the tracking error covariance does not exceed the threshold η.
[0168] S4: Map the node selection optimization model to a reinforcement learning problem, and construct the state space, action space and reward function;
[0169] The node selection optimization model is a combinatorial optimization model, the decision variables are the binary activation variables of each observation node, and the objective function is to minimize the number of activated nodes or their weighted sum.
[0170] The state space includes target position estimation, target velocity estimation, and covariance matrix elements, covariance trace, and equivalent statistics that characterize the uncertainty of the estimation.
[0171] The action space is a binary vector of length N, where N is the number of observation nodes, and each dimension of the vector corresponds to the activation or deactivation of an observation node.
[0172] The reward function includes a precision constraint penalty term and a node cost term. When PCRLB exceeds a preset threshold, the penalty weight of the precision constraint penalty term is increased to encourage Hybrid-DQN to select more effective nodes to meet the precision requirements, thereby achieving a dynamic balance between precision and resource consumption.
[0173] The reward function expression is:
[0174]
[0175] S5: Construct a hybrid deep Q network Hybrid-DQN and train Hybrid-DQN based on a hybrid experience replay mechanism that includes virtual training samples and online real interaction samples;
[0176] The Hybrid-DQN network architecture employs a classic dual-network structure (current network and target network). Its core innovation lies in the improved training method, which integrates offline virtual training with online real-world interaction through a periodic switching mechanism: based on the target motion prior and observation model, virtual state sequences and virtual observation data are generated between real measurement time steps to construct virtual training samples and write them into the experience replay pool. Simultaneously, online environment interaction samples are retained, forming a hybrid experience replay mechanism. This method effectively expands the training sample size while maintaining the ability to dynamically perceive the real environment, significantly improving training efficiency and policy convergence speed in data-scarce scenarios, and balancing the stability of offline learning with the adaptability of online learning.
[0177] Model initialization and parameter settings:
[0178] 1) Read or set network node coordinates, measurement noise parameters, process noise parameters, communication / energy consumption cost parameters, and accuracy thresholds;
[0179] 2) Construct the state transition matrix F and the process noise covariance Q based on the sampling period T, and construct the observation noise covariance for each node;
[0180] 3) Initialize the filter prior state estimate and covariance, and initialize the Fisher information matrix;
[0181] 4) Initialize the Hybrid-DQN network parameters, target network parameters, experience replay pool, and exploration strategy parameters.
[0182] like Figure 2 As shown, the methods for training Hybrid-DQN based on a hybrid experience replay mechanism that includes virtual training samples and online real interaction samples include:
[0183] The hybrid DQN algorithm consists of two parts: an online DQN algorithm and offline virtual training. The online training process is stabilized by periodically executing offline virtual training, while maintaining the ability to perceive and adapt to changes in the real environment. The hybrid DQN algorithm is used by periodically turning on and off a switch S. The core of the network is the DQN algorithm, and the goal is to find the optimal action selection policy based on the current state. In Q-learning, the Q-value represents the expected cumulative reward for taking a certain action in a given state. The Q-value is calculated as follows:
[0184]
[0185] in, For learning rate, The discount factor; this invention uses the current network. and target network During training, the current network interacts with the environment to perform actions and obtains Q-values. The Q-value of the target network is calculated as follows:
[0186]
[0187] The goal of training is to minimize the error between the current network's Q-value and the target Q-value. The network loss can be expressed as:
[0188]
[0189] Update the current network using gradient descent. And use soft updates to adjust the target network:
[0190]
[0191] in, These are soft update coefficients; to avoid the algorithm getting trapped in local optima, random actions are added during the exploration process, namely:
[0192]
[0193] Where p is a randomly generated probability value. It is a value that gradually decreases with the number of training iterations.
[0194] S6: At each real observation time, based on the activation decision of the trained Hybrid-DQN output node, obtain the observation data of the activated node and perform target state estimation, and output the node selection result.
[0195] S7: The fusion center receives observations from activated nodes, updates state estimates and covariance using methods such as particle filtering, and records relevant experience for subsequent policy fine-tuning.
[0196] To simultaneously meet the constraints of target tracking accuracy and node energy consumption / communication overhead under resource-constrained conditions, this invention generates virtual future states and virtual observation data based on target motion priors and measurement noise priors between adjacent real observation moments, constructing virtual training samples to amplify empirical data. A posterior Cramér-Rao lower bound (PCRLB) or equivalent error bound is derived or calculated as a tracking accuracy evaluation index, establishing a node selection optimization model that minimizes the number of activated nodes while satisfying accuracy constraints. A hybrid deep Q-network (Hybrid-DQN) is constructed within a reinforcement learning framework, employing a hybrid empirical replay mechanism combining offline virtual samples and online real interaction samples for training and online decision-making, thus balancing training efficiency and environmental adaptability. Experiments and analysis show that this invention can reduce node selection computational overhead while maintaining tracking accuracy, compressing node selection decision time to the millisecond level, thus adapting to the dynamic tracking requirements of underwater targets.
[0197] Example 2:
[0198] This embodiment provides a sensor node selection system based on hybrid deep reinforcement learning, including:
[0199] The model building module is used to build the target state model and the observation model, and outputs the target state estimate and the estimation error covariance based on the observation data of the activated nodes at each real observation time.
[0200] The virtual sample generation module generates virtual training samples between adjacent real observation times;
[0201] The accuracy constraint modeling module establishes a node selection optimization model based on the target tracking accuracy evaluation index. The node selection optimization model aims to minimize the number of selected nodes while satisfying the tracking accuracy constraint.
[0202] The reinforcement learning modeling module maps the node selection optimization model to a reinforcement learning problem, and constructs the state space, action space and reward function.
[0203] The Hybrid Deep Q Network training module constructs a Hybrid Deep Q Network (Hybrid-DQN) and trains it based on a hybrid experience replay mechanism that includes virtual training samples and online real interaction samples.
[0204] The online decision-making module, at each real observation moment, makes activation decisions based on the trained Hybrid-DQN output node, acquires the observation data of the activated node, estimates the target state, and outputs the node selection result.
[0205] Example 3:
[0206] To verify the effectiveness and efficacy of the present invention, this embodiment obtained the following results through experiments: Figures 3-7 The performance data is analyzed in detail below:
[0207] like Figure 3 As shown, the distribution of nodes in the sensor network, the trajectory of the target, and the EKF-estimated trajectory are illustrated, and 2000 Monte Carlo experiments were conducted.
[0208] like Figure 4The figure shows the average reward changes of the Online-DQN and Hybrid-DQN algorithms. In the early stages of training, Online-DQN's reward value is low (approximately -700) due to insufficient experience; while Hybrid-DQN, with the expansion of virtual samples, has a higher initial reward value (approximately -500). As training progresses, Hybrid-DQN stabilizes around 250 epochs, with a final reward value of approximately -150. Its convergence speed is significantly faster than Online-DQN (approximately 400 epochs), and its final reward is higher, indicating that it has learned a better node selection strategy.
[0209] like Figure 5 The diagram illustrates the changes in PCRLB and RMSE for four methods. The Hybrid-DQN method proposed in this invention, along with the DP and Greedy methods, all meet the tracking accuracy requirements and significantly outperform Online-DQN. At k=21, 32, and 45, the tracking errors of all four methods decrease, which is due to the target being closer to the sensor node, resulting in more reliable observation data. Ultimately, the tracking errors of each method tend to stabilize, consistent with the trend of PCRLB changes, verifying the effectiveness of the method proposed in this invention.
[0210] like Figure 6 As shown, the number of nodes selected by each method at different tracking times is illustrated. In the initial stage, due to the large tracking error, all methods select more nodes to meet the accuracy requirements; at k=21, 32, and 45, the target is closer to the sensor, and only fewer nodes are needed to complete the tracking. It is worth noting that during k=[8,20], Hybrid-DQN selects the fewest nodes, because the large amount of empirical data generated by virtual prediction allows it to achieve better tracking results with fewer nodes.
[0211] like Figure 7 As shown, the node selection time of the four algorithms is illustrated. The online computation time of Hybrid-DQN is much shorter than that of the DP and Greedy methods, decreasing from seconds to milliseconds (DP approximately 1.1s, Greedy approximately 1.2s, Hybrid-DQN approximately 0.9ms), which greatly improves computational efficiency and meets real-time requirements.
Claims
1. A sensor node selection method based on hybrid deep reinforcement learning, characterized in that, Includes the following steps: S1: Establish the target state model and observation model, and output the target state estimate and estimation error covariance based on the observation data of the activated node at each real observation time; S2: Generate virtual training samples between adjacent real observation times; S3: Establish a node selection optimization model based on the target tracking accuracy evaluation index. The node selection optimization model aims to minimize the number of selected nodes while satisfying the tracking accuracy constraint. S4: Map the node selection optimization model to a reinforcement learning problem, and construct the state space, action space and reward function; S5: Construct a hybrid deep Q network Hybrid-DQN and train Hybrid-DQN based on a hybrid experience replay mechanism that includes virtual training samples and online real interaction samples; S6: At each real observation time, based on the activation decision of the trained Hybrid-DQN output node, obtain the observation data of the activated node and perform target state estimation, and output the node selection result.
2. The sensor node selection method based on hybrid deep reinforcement learning according to claim 1, characterized in that, In step S1, the target state model is a discrete-time motion model, and the target state includes at least two-dimensional position and velocity components, as specifically expressed below: Suppose a target moves within a region of water depth; its state equation can be expressed as: ； in, Let k be the motion state of the target at time k. Let the target be located at time k. The velocity of the target in the x and y directions; For dynamic noise, , Let be the covariance matrix of the state noise; When the target is making a uniform turning motion, its state transition matrix is: ； Where T is the time interval during the tracking process. Turning speed; Covariance matrix of state noise for: ； Where q is the intensity of the process noise. This represents the Kronecker product.
3. The sensor node selection method based on hybrid deep reinforcement learning according to claim 2, characterized in that, The observation model in step S1 is a nonlinear mapping model from the location of the observation node to range observation, azimuth observation, or a combined range-azimuth observation, specifically expressed as follows: Assuming the coordinates of the sensor nodes are fixed and known, the measurement value of node i at time k during the target's motion is expressed as: ； in: ； ； in, Indicates the position coordinates of node i; Let represent the measurement noise of the distance and angle at node i during the measurement process; assuming the measurement noise is independent and identically distributed, where , , and These are the noise variances of distance and angle during the observation process, respectively.
4. The sensor node selection method based on hybrid deep reinforcement learning according to claim 3, characterized in that, The target state estimation and estimation error covariance acquisition in step S1 include: After obtaining measurement information, each sensor node transmits the information to the fusion center, which then filters and estimates the target. Assuming there are N sensor nodes in total, the observation data obtained by the fusion center at time k is represented in the following form: , , ; After acquiring the measurement data, an extended Kalman filter is used to fuse the data to estimate the target position, thereby obtaining the target's motion state during the tracking process. The mean and covariance of the target state estimate are expressed as: ； in, This represents the estimated state of the target at time k. Indicates Kalman gain, The Jacobian matrix represents the equation of observation. Let k represent the prior covariance matrix at time k. Let represent the posterior covariance matrix at time k; , and The calculation expressions are as follows: ； ； 。 5. The sensor node selection method based on hybrid deep reinforcement learning according to claim 4, characterized in that, Step S2 includes: A1: Insert M virtual time steps between adjacent real observation times; A2: Predict or sample virtual future states based on prior knowledge of target motion; A3: Generating virtual observation data based on the observation model and prior measurement noise; A4: Write the virtual future state and virtual observation data into the experience replay pool to construct virtual training samples.
6. The sensor node selection method based on hybrid deep reinforcement learning according to claim 5, characterized in that, In step S3, the target tracking accuracy evaluation index includes the posterior Cramer-Rao bound (PCRLB); the tracking accuracy constraint is that the error bound of the PCRLB trace, weighted trace, or position dimension does not exceed a preset threshold. Methods for calculating the posterior Cramér-Rao boundary PCRLB: The Bayesian Craméro bound provides a lower bound for the estimation of the target state; PCRLB is expressed as: ； in, The target state estimate, Let the Bayesian information matrix of the target at time k be represented as: ； in, Represents the second-order partial derivative. It is divided into two parts: prior information and measurement information. ； in, The Fisher Information Matrix (FIM) represents prior information. The contribution of the measurement data to FIM can be expressed as: ； Among them, the first-order partial derivatives of the observation equation The Jacobian matrix represents the measurement equation, while the variance of the measurement noise is represented by the diagonal matrix. Composition; the FIM representation of the measurement data is as follows: ； The Jacobian matrix of the measurement equation is calculated as follows: ； Independent information contribution of target location x estimation for: ； Similarly, the independent information contribution of the target location y estimation The calculation yielded: ； Contribution of relevant information to the estimation of target location x and y and The calculations include: ； Since the partial derivative with respect to velocity is 0, the FIM representation of the measured data is: ； The prior information FIM is given by the state prediction covariance matrix: ； In summary, the posterior Cramer-Rao bound for target tracking at time k is obtained as follows: 。 7. The sensor node selection method based on hybrid deep reinforcement learning according to claim 6, characterized in that, In step S4, the node selection optimization model is a combined optimization model, the decision variables are the binary activation variables of each observation node, and the objective function is to minimize the number of activated nodes or their weighted sum. The state space includes target position estimation, target velocity estimation, and covariance matrix elements, covariance trace, and equivalent statistics that characterize the uncertainty of the estimation. The action space is a binary vector of length N, where N is the number of observation nodes, and each dimension of the vector corresponds to the activation or deactivation of an observation node. The reward function includes a precision constraint penalty term and a node cost term. When PCRLB exceeds a preset threshold, the penalty weight of the precision constraint penalty term is increased to encourage Hybrid-DQN to select more effective nodes to meet the precision requirements.
8. The sensor node selection method based on hybrid deep reinforcement learning according to claim 7, characterized in that, The reward function expression in step S4 is: 。 9. The sensor node selection method based on hybrid deep reinforcement learning according to claim 8, characterized in that, The method for training Hybrid-DQN in step S5 based on a hybrid experience replay mechanism that includes virtual training samples and online real interaction samples includes: In Q-learning, the Q-value is used to represent the expected cumulative reward for taking a certain action in a given state. The Q-value is calculated as follows: ； in, For learning rate, The discount factor; this invention uses the current network. and target network During training, the current network interacts with the environment to perform actions and obtains Q-values. The Q-value of the target network is calculated as follows: ； The goal of training is to minimize the error between the current network's Q-value and the target Q-value. The network loss can be expressed as: ； Update the current network using gradient descent. And use soft updates to adjust the target network: ； in, These are soft update coefficients; to avoid the algorithm getting trapped in local optima, random actions are added during the exploration process, namely: ； Where p is a randomly generated probability value. It is a value that gradually decreases with the number of training iterations.
10. A sensor node selection system based on hybrid deep reinforcement learning, characterized in that, For implementing the method of claim 1, the system comprises: The model building module is used to build the target state model and the observation model, and outputs the target state estimate and the estimation error covariance based on the observation data of the activated nodes at each real observation time. The virtual sample generation module generates virtual training samples between adjacent real observation times; The accuracy constraint modeling module establishes a node selection optimization model based on the target tracking accuracy evaluation index. The node selection optimization model aims to minimize the number of selected nodes while satisfying the tracking accuracy constraint. The reinforcement learning modeling module maps the node selection optimization model to a reinforcement learning problem, and constructs the state space, action space and reward function. The Hybrid Deep Q Network training module constructs a Hybrid Deep Q Network (Hybrid-DQN) and trains it based on a hybrid experience replay mechanism that includes virtual training samples and online real interaction samples. The online decision-making module, at each real observation moment, makes activation decisions based on the trained Hybrid-DQN output node, acquires the observation data of the activated node, estimates the target state, and outputs the node selection result.