An Optimal Selection Method for Control with Unknown Sensor Noise Covariance

Abstract

An optimal selection method for sensor noise covariance unknown control, which relates to a method for selecting sensors. The method includes designing a linear quadratic state feedback controller based on state measurement values and establishing an optimal sensor selection problem model with unknown sensor noise covariance; analyzing the influence of sensor noise on system performance; transforming the optimal sensor selection problem with unknown sensor noise covariance; and proposing an optimal sensor selection and unknown sensor noise covariance estimation algorithm based on reinforcement learning. The present invention proposes a technology combining linear quadratic control and reinforcement learning to find the sensor that minimizes the expected performance of linear quadratic feedback control and estimate the unknown noise covariance in the case of unknown sensor noise covariance, so as to solve the optimal sensor selection with unknown sensor noise covariance.

Description

Technical Field

[0001] The present invention relates to a method for selecting sensors, and particularly to an optimal selection method for controlling unknown sensor noise covariance. Background Art

[0002] Sensor networks are generally deployed in environments with limited resources. To reduce energy consumption, only one sensor is activated for each measurement data, and finally, an appropriate sensor sequence is selected for data measurement. Based on this background, the optimal sensor scheduling problem needs to find an optimal sensor sequence to minimize the performance index. The premise of the above problem is that the sensor noise covariance is known. If the noise covariance is unknown, the optimal sensor scheduling selection cannot be performed. Secondly, the system of the above problem does not consider feedback control. In a linear quadratic feedback control system, the state feedback controller uses the measured state to stabilize or optimize the system. Therefore, a key problem is how to find a sensor that minimizes the linear quadratic performance index when the sensor noise covariance is unknown, while reducing energy consumption and the impact of noise on the system.

[0003] In recent years, it has become increasingly widespread to combine reinforcement learning methods with control methods to solve control problems. In the multi-armed bandit (MAB) problem, each action is associated with a numerical reward with an unknown expectation, and the agent's goal is to find the action with the maximum expected reward. The agent does not know the value of the expected reward at the beginning of learning, but in the process of continuous learning, it continuously balances the relationship between exploitation and exploration, and finally finds the action with the maximum expected reward. This reinforcement learning tool provides a solution to the problem of optimal sensor selection for linear quadratic control with unknown sensor noise covariance. Summary of the Invention

[0004] The purpose of the present invention is to provide an optimal selection method for controlling unknown sensor noise covariance. When the sensor noise covariance is known, a linear quadratic state feedback controller is designed based on the state measurement value, an optimal sensor selection model with unknown sensor noise covariance is established, and then the influence of sensor noise on the system performance is analyzed; an optimal sensor selection method based on reinforcement learning is proposed.

[0005] The purpose of the present invention is achieved by the following technical solutions:

[0006] An optimal selection method for controlling unknown sensor noise covariance, the method comprising the following steps:

[0007] Step 1, design a linear quadratic state feedback controller based on the state measurement value, and establish an optimal sensor selection problem model with unknown sensor noise covariance;

[0008] Step 2, analyze the influence of sensor noise on the system performance;

[0009] Step 3: Transformation of the optimal sensor selection problem with unknown sensor noise covariance;

[0010] Step 4: Propose an optimal sensor selection and unknown sensor noise covariance estimation algorithm based on reinforcement learning.

[0011] For the optimal selection method for controlling unknown sensor noise covariance, Step 1 is to design a linear quadratic state feedback controller based on the state measurement value and establish an optimal sensor selection problem model with unknown sensor noise covariance:

[0012] Consider the following discrete-time linear system dynamics model:

[0013] x(k + 1) = Ax(k) + Bu(k) (1)

[0014] where x(k) and u(k) represent the state and control input of the system respectively, and A and B are system matrices with appropriate dimensions; the discrete time k ∈ [0, k f , k f is the terminal time; the initial state x(0) ~ N(μ0, φ0) is a Gaussian random variable, μ0 is the mean of the initial state, and φ0 is the covariance of the initial state;

[0015] The measurement output of the sensor will inevitably contain noise. Therefore, if the i-th sensor s is selected at time k i , then the measurement output of the sensor is expressed as:

[0016] y(k) = x(k) + ω i (k) (2)

[0017] where y(k) and ω i (k) represent the measurement output and noise of sensor s i respectively; the noise process {ω i (k)} is an independent Gaussian process with zero mean and unknown covariance Here is a scalar, and I is the identity matrix with appropriate dimensions; it is assumed that for any i = 1,..., N and any k, x(0) and ω i (k) are independent of each other;

[0018] Based on the measurement output of the sensor, design a controller with the following form:

[0019] u(k) = -L(k)y(k) (3)

[0020] where the controller gain is L(k) = [B TP(k + 1)B + R] -1 B T P(k + 1)A, where P(k) is the solution of the discrete Riccati equation, P(k)=[A - BL(k)] T P(k + 1)[A - BL(k)] + L T (k)RL(k)+Q, P(k f ) = F≥0, Q≥0 and R≥0 are matrices with appropriate dimensions;

[0021] After selecting the sensor s i , a controller is constructed using the state values measured at each moment, and the performance of the system is calculated; the desired system performance has the following form:<{

[0022]

[0023] where E[.] represents taking the expectation of the result, F = P(k f )≥0, Q≥0 and R≥0 are matrices with appropriate dimensions;

[0024] In the case where the sensor noise covariance is unknown, the optimal sensor is selected to minimize the desired performance index of the system, and the optimal sensor selection problem model is as follows:

[0025]

[0026] The described optimal selection method for controlling unknown sensor noise covariance, and step 2 is to analyze the influence of sensor noise on the system performance:

[0027] When selecting the sensor s i to measure the state, the influence result of its noise on the system performance is as follows:

[0028]

[0029] where μ₀ and φ₀ are the mean and covariance of the initial state x(0) respectively; S(k)=L(k) T [B T P(k + 1)B + R]L(k), the definitions of P(k) and L(k) are given after (3), and P(0) can be obtained through its iterative formula; is the scalar in the noise covariance of the sensor s i , and tr[·] is the trace of the matrix;

[0030] It can be seen from (6) that since the values of μ₀ and φ₀ are determined, the results of P(0) and S(k) are determined by the system matrix, so the quality of the system performance only depends on the noise covariance of the sensor s i ​Since I is the identity matrix of appropriate dimension, then The smaller it is, the smaller the expected performance index of the system, and the better the system performance.

[0031] The described optimal selection method for unknown sensor noise covariance control, where step 3 is to transform the optimal sensor selection with unknown sensor noise covariance based on the MAB machine problem framework, including the following steps:

[0032] Step 3.1: Analyze the similarities between the optimal sensor selection problem and the MAB machine problem;

[0033] This problem is similar to the optimal sensor selection problem with unknown sensor noise covariance in this method; in the optimal sensor selection problem, the expected performance index of the system cannot be determined due to the unknown sensor noise covariance, but after each sensor selection, the control input is obtained using the measured state data, and the corresponding system performance can be calculated; the goal of the optimal sensor selection problem is to find the sensor that minimizes the expected performance, which is similar to finding the action with the maximum expected reward in the MAB machine problem, except that the goals of the two problems are exactly opposite;

[0034] Step 3.2: Transformation of the optimal sensor selection problem;

[0035] A single sensor selection in the optimal sensor selection problem with unknown sensor noise covariance can be regarded as an action in the MAB machine problem. Introduce A t to represent selecting a sensor at time step t; after applying the sensor to the system, the corresponding system performance index will be obtained. Since the goals of the two problems are opposite, the negative value of the system performance index is used as the numerical reward after sensor selection; select sensor s i at time step t, and its numerical reward is defined as follows

[0036]

[0037] where time step t is different from time instant k. t represents the number of sensor selections, rather than the time instant k of the system dynamic process; t ∈ [1, m], and m (m is a positive integer) is the terminal time step;

[0038] Define the action value q(s i ) of selecting sensor s i as the expected reward for selecting sensor s i at time step t:

[0039] q(s i ) = E[R t |A t = s i (8)

[0040] Among them, E[·|·] represents the expected reward, and E[R t |A t = s i represents the expected reward obtained when the sensor s is selected at the time step t; then maximizing the expected reward is equivalent to minimizing the objective function i ;

[0041]

[0042] Since the expected reward q(s i ) for selecting the sensor s at the time step t is unknown, it needs to be estimated; the following form is used to estimate q(s i ): i ;

[0043]

[0044] Among them, is a random variable, which is 1 if the sensor s is selected at the time step t i , and 0 otherwise. Q t (s i ) calculates the expected reward q(s i ) for selecting the sensor s at the time step t based on the rewards obtained from the previous t - 1 sensor selections. Its estimation method is that up to the time step t - 1 (i.e., in the case where t - 1 sensor selections have been made), the sum of the rewards obtained after selecting the sensor s i is divided by the number of times the sensor s i has been selected; Q0 is the initial action estimate value of all sensors. After the sensor selection, the action estimate value is updated based on the initial value; i ;

[0045] Next, a specific introduction will be given on how to estimate the action value q(s i ) for a sensor s i ; represents the estimated value of the action value for selecting s i after the sensor s i has been selected n - 1 times, and is defined as follows:

[0046]

[0047] Among them, represents the reward value obtained when the sensor s i is selected for the jth time, and n - 1 represents that the sensor s i has been selected n - 1 times;

[0048] Since the storage requirements and computational complexity of Equation (11) increase with the number of selections, a simplified form is chosen as follows. The iterative form of this equation reduces and fixes the computational and storage requirements.

[0049]

[0050] Based on the action estimation value, the optimal sensor selection problem can be transformed into the following problem:

[0051]

[0052] That is, at time step t, select the sensor that minimizes the action estimation value.

[0053] Step 3.3: Use the reinforcement learning method to solve the optimal sensor selection problem.

[0054] When solving the MAB machine problem, a relatively classic method is the action-value method. This method consists of two parts. One is to estimate the action value using the existing reward values, and the other is to select an action based on the obtained action estimation value. This method uses (12) to estimate the action value, and the method of selecting an action adopts the upper confidence bound method. This method needs to explore known experiences and develop unknown actions, that is, consider both known information and the uncertainty of actions. An uncertainty term is added to the original action estimation value, and then action selection is based on the two results. Based on the upper confidence bound method, at time step t, select the sensor that maximizes the two results.

[0055]

[0056] Among them, the first term is the action estimation value, representing known information, that is, select the action with the largest action estimation value; the second term reflects the uncertainty of the action. c > 0 represents the confidence level, ln(·) represents the natural logarithm (logarithm with base e), t is the time step, representing the number of times of action selection, and N t (a) represents the number of times action a is selected in t selections.

[0057] The optimal selection method for controlling unknown sensor noise covariance mentioned above. Step 4 proposes optimal sensor selection based on reinforcement learning and estimation of unknown sensor noise covariance, including the following steps:

[0058] Step 4.1: Estimate the unknown sensor noise covariance.

[0059] Simplify Equation (6). For sensor s i The noise covariance scalar is calculated by the following formula:

[0060]

[0061] where the expected performance index J is unknown;

[0062] According to the result of formula (9), transform (15):

[0063]

[0064] i.e., replace the expected performance J with the expected reward q(s i );

[0065] The expected reward is estimated using the existing rewards. The following result reflects the relationship between the estimated value and the expected reward:

[0066]

[0067] When the number of selections approaches infinity, the estimated value is equivalent to the expected value;

[0068] When n → ∞, the following result holds:

[0069]

[0070] Therefore, use the existing estimated value Estimate the covariance of the noise according to (18); it is a very necessary measure to estimate the covariance of its noise while selecting the optimal sensor;

[0071] Step 4.2, Design of the optimal sensor selection and unknown sensor noise covariance estimation algorithm based on reinforcement learning;

[0072] This method proposes how to select the optimal sensor using the reinforcement learning method to minimize the expected performance index in the case where the covariance of the sensor noise is unknown, and use the obtained reward data to estimate the unknown noise covariance;

[0073] Algorithm 1 Implementation of the optimal sensor selection and unknown sensor noise covariance estimation algorithm based on reinforcement learning:

[0074] 1: Given the system parameter matrices A, B, Q, R, P(k f ) = F, the parameter c, the mean μ0 and covariance φ0 of the initial state x(0); set the terminal time k f and the initial estimate Q0;

[0075] 2: Select the sensor using formula (14) and calculate the reward value R t ;

[0076] 3: For the selected sensor, estimate the current action value using formula (12);

[0077] 4: If the number of selections \(t>m\) (\(m\) is a positive integer), then end the selection process; otherwise, return to step 4.2.2;

[0078] 5: After the selection is ended, first calculate \(P(0)\) and \(S(k)\), and use the obtained According to formula (18), calculate the estimated values of all sensor noise covariance scalars of.

[0079] The advantages and effects of the present invention are:

[0080] Based on the multi-armed bandit (MAB) problem framework in reinforcement learning, the present invention transforms the optimal sensor selection problem of linear quadratic control with unknown sensor noise covariance, and proposes a technology combining linear quadratic control and reinforcement learning. In the case of unknown sensor noise covariance, it is first analyzed that the influence of sensors on the system performance mainly lies in the magnitude of the noise covariance. Then, the reinforcement learning method is used to solve the optimal sensor selection problem with unknown sensor noise covariance, and the unknown noise covariances of all sensors are estimated. The key to solving the problem lies in using the known system performance to estimate the expected performance, and then learning the sensor with the smallest covariance from the known information. Brief Description of the Drawings

[0081] Figure 1 It is the average reward graph obtained by selecting sensors each time under different selection methods in an embodiment of the present invention;

[0082] Figure 2 It is the percentage graph of being able to select the optimal sensor each time under different selection methods in an embodiment of the present invention;

[0083] Figure 3 It is the comparison chart of the estimated values and estimation errors of the sensor noise covariance under different selection methods in an embodiment of the present invention.

[0084] In the figure: \(q(s1)\) is the expected reward when the agent selects sensor \(s1\). In the numerical simulation, \(s1\) has the smallest noise covariance; \(M1\) is the greedy method; \(M2\) is the \(\epsilon\)-greedy method, and its parameter \(\epsilon = 0.05\); \(M3\) is the optimal initial value method, and the initial value is set to \(Q0(s i )=-30\); \(M4\) is the upper confidence bound method, and \(c = 5\) controls the confidence level. Detailed Embodiment

[0085] The present invention will be described in detail below with reference to the embodiments shown in the drawings.

[0086] In an embodiment of the present invention, an optimal sensor selection method for linear quadratic control with unknown sensor noise covariance based on reinforcement learning includes the following steps:

[0087] The step 1 is as follows:

[0088] Consider the following discrete-time linear system dynamics model:

[0089] x(k + 1) = Ax(k) + Bu(k) (1)

[0090] where x(k) and u(k) represent the state and control input of the system respectively, and A and B are system matrices with appropriate dimensions. The discrete time k ∈ [0, k f , k f is the terminal time. The initial state x(0) ~ N(μ0, φ0) is a Gaussian random variable, μ0 is the mean of the initial state, and φ0 is the covariance of the initial state.

[0091] The state of the system cannot be obtained directly, so sensors are needed for measurement. Let the set S = {s1, s2,..., s N} have N sensors, and each sensor can be used to directly measure the state of the system. At time k, only one sensor s i can be selected from the N sensors for state measurement. The measurement output of the sensor will inevitably contain noise. Therefore, if the i-th sensor s i is selected at time k, then the measurement output of the sensor can be expressed as:

[0092] y(k) = x(k) + ω i (k) (2)

[0093] where y(k) and ω i (k) represent the measurement output and noise of the sensor s i respectively. The noise process {ω i (k)} is an independent Gaussian process with zero mean and unknown covariance Here is a scalar, and I is the identity matrix with appropriate dimensions. It is assumed that for any i = 1,..., N and any k, x(0) and ω i (k) are independent of each other.

[0094] Based on the measurement output of the sensor, design a controller of the following form:

[0095] u(k) = -L(k)y(k) (3)

[0096] where the controller gain is L(k) = [B T P(k + 1)B + R] -1 B TThe meaning of P(k + 1)A and L(k) is: assuming there is no noise in the measurement result y(k), it is the optimal feedback control gain obtained in the linear quadratic state feedback problem. Among them, P(k) is the solution of the discrete Riccati equation, and P(k)=[A - BL(k)] T P(k + 1)[A - BL(k)]+L T (k)RL(k)+Q, P(k f ) = F≥0, Q≥0, and R≥0 are matrices with appropriate dimensions.

[0097] After selecting the sensor s i , use the state values measured at each moment to form a controller and calculate the performance of the system. The desired system performance has the following form:

[0098]

[0099] Among them, E[.] represents taking the expectation of the result, F = P(k f )≥0, Q≥0, and R≥0 are matrices with appropriate dimensions.

[0100] In the case where the sensor noise covariance is unknown, select the optimal sensor to minimize the desired performance index of the system. Then the optimal sensor selection problem model is as follows:

[0101]

[0102] The said step 2 is:

[0103] To analyze the influence of the noise in the state measurement value on the system performance, introduce the formula for calculating the quadratic form expectation of a normal random variable:

[0104] E[ξ T Gξ] = μ T Gμ+tr(GW) (6)

[0105] Among them, ξ~N(μ, W) is a normal random variable, with mean μ and covariance matrix W. G≥0 and tr(.) is the trace of the matrix.

[0106] To calculate the result of the system desired performance (4), start calculating from the terminal time k f When k = k f , there is the desired performance of the system at the terminal time

[0107]

[0108] When k = k f - 1, there is the desired performance of the system at time k f - 1

[0109]

[0110] According to (1), (2), and (3), k can be obtained. f System state at time:

[0111] x(k f ) = [A - BL(k f -1)]x(k f -1) - BL(k f -1)ω i (k f -1) (9)

[0112] and k f Control input at time k - 1:

[0113] u(k f -1) = -L(k f -1)x(k f -1) - L(k f -1)ω i (k f -1) (10)

[0114] Substituting (7), (9), and (10) into (8), we get The simplified form of:

[0115]

[0116] where P(k f -1) is defined as follows:

[0117] P(k f -1) = [A - BL(k f -1)] T P(k<000> f )[A - BL(k f -1)] + L T (k f -1)RL(k f -1) + Q (12)

[0118] S(k f -1) is defined as:

[0119] S(k f -1) = L(k f -1) T [B T P(k f )B + R]L(k f -1) (13)

[0120] According to the calculation formula in (6), Further simplify:

[0121]

[0122] Similarly, when k = k f -2, the expected performance of the system at time k f -2 is: For:

[0123]

[0124] And so on, when k = 0, the expected performance of the system is obtained:

[0125]

[0126] where x(0) ~ N(μ0, φ0), which can be calculated with the help of (6).

[0127] When selecting the sensor s i to measure the state, the influence of its noise on the system performance is:

[0128]

[0129] where μ0 and φ0 are the mean and covariance of the initial state x(0) respectively. S(k) = L(k) T [B T P(k + 1)B + R]L(k), the definitions of P(k) and L(k) have been given after (3), and P(0) can be obtained through its iterative formula. is the scalar in the noise covariance of the sensor s i .

[0130] It can be seen from (17) that since the values of μ0 and φ0 are determined and the results of P(0) and S(k) are determined by the system matrix, the quality of the system performance only depends on the noise covariance of the sensor s i Since I is the identity matrix of the appropriate dimension, then the smaller it is, the smaller the system expected performance index is, and the better the system performance is.

[0131] Therefore, based on (1), (2) and (3), the problem of selecting the optimal sensor to minimize the system expected performance index is to select the sensor with the smallest noise covariance to minimize the system expected performance index. The difficulty in solving this problem is that the sensor noise covariance is unknown. How to use the calculable system performance to find the sensor with the smallest noise covariance and estimate the noise covariance of all sensors is the innovation of the present invention.

[0132] ​Step 3 includes the following steps:

[0133] Step 3.1. Analyze the similarities between the optimal sensor selection problem and the MAB machine problem.

[0134] Since the sensor noise covariance is unknown, the expected performance index J cannot be calculated. However, after each selection of a sensor, the feedback control u(k) can be obtained using the measurement value of the sensor, and then the system performance with noise can be calculated. That is, after selecting sensor s i for measurement, the corresponding system performance can be obtained, but the expected system performance cannot be determined.

[0135] In the reinforcement learning method, there is a type of MAB machine problem, which is characterized in that each action has an unknown numerical reward in expectation. That is, after making an action, the agent will obtain the corresponding numerical reward, but the expected value of the numerical reward is unknown; then the goal of the agent is to continuously select different actions and learn from experience to finally find the action with the maximum expected reward.

[0136] This problem is similar to the optimal sensor selection problem of the present invention where the sensor noise covariance is unknown. In the optimal sensor selection problem, the expected performance index of the system cannot be determined due to the unknown sensor noise covariance. However, after each selection of a sensor, the control input can be obtained using the measured state data, and the corresponding system performance can be calculated. The goal of the optimal sensor selection problem is to find the sensor that minimizes the expected performance, which is similar to finding the action with the maximum expected reward in the MAB machine problem, except that the goals of the two problems are exactly opposite.

[0137] Step 3.2. Transformation of the optimal sensor selection problem.

[0138] The MAB machine problem is solved using the reinforcement learning method, which provides a technical basis for the present invention to solve the optimal sensor selection problem with unknown sensor noise covariance. To solve the problem using the reinforcement learning method, it is necessary to transform the optimal sensor selection problem with unknown sensor noise covariance based on the framework of the MAB machine problem.

[0139] One sensor selection in the optimal sensor selection problem with unknown sensor noise covariance can be regarded as an action in the MAB machine problem. Introduce A t to represent selecting a sensor at time step t. After applying the sensor to the system, the corresponding system performance index will be obtained. Since the goals of the two problems are opposite, the negative value of the system performance index is used as the numerical reward after selecting the sensor. Select sensor s i at time step t, and its numerical reward is defined in the following form

[0140]

[0141] Among them, the time step \(t\) is different from the moment \(k\). \(t\) represents the number of times of sensor selection, rather than the moment \(k\) of the system dynamic process. \(t\in[1,m]\), where \(m\) ( \(m\) is a positive integer) is the terminal time step.

[0142] Define the action value \(q(s\) i ) of selecting sensor \(s\) i as the expected reward for selecting sensor \(s\) i at time step \(t\):

[0143] \(q(s\) i ) = E[R t |A t = s i (19)

[0144] Then maximizing the expected reward is equivalent to minimizing the objective function

[0145]

[0146] Since the expected reward \(q(s\) i ) of selecting sensor \(s\) i at time step \(t\) is unknown, it needs to be estimated. The following form is used to estimate \(q(s\) i ):

[0147]

[0148] Among them, is a random variable, which is 1 if sensor \(s\) i is selected at time step \(t\), and 0 otherwise. \(Q\) t (s i ) calculates the expected reward \(q(s\) i ) of selecting sensor \(s\) i at time step \(t\) based on the rewards obtained from the previous \(t - 1\) sensor selections. Its estimation method is that up to time step \(t - 1\) (i.e., in the case where \(t - 1\) sensor selections have been made), the sum of the rewards obtained after selecting sensor \(s\) i is divided by the number of times sensor \(s\) i has been selected. \(Q_0\) is the initial action estimate value of all sensors. After sensor selection, the action estimate value is updated based on the initial value.

[0149] Next, specifically introduce how to estimate the action value \(q(s\) i ) for a sensor \(s\) i . indicates that after sensor \(s\) i has been selected \(n - 1\) times, select \(s\) iThe estimated value of the action value is defined as follows:

[0150]

[0151] Wherein, represents the reward value obtained when the sensor s i is selected for the j-th time, and n - 1 represents that the sensor s i has been selected n - 1 times.

[0152] Since the storage requirement and computational amount of Equation (22) increase with the increase of the number of selections, the following simplified form is selected. The iterative form of this equation reduces and fixes the computational and storage requirements.

[0153]

[0154] Based on the action estimation value, the optimal sensor selection problem can be transformed into the following problem:

[0155]

[0156] That is, at time step t, select the sensor that minimizes the action estimation value.

[0157] Step 3.3. Solve the optimal sensor selection problem using the reinforcement learning method.

[0158] When solving the MAB machine problem, a relatively classic method is the action-value method. This method is divided into two parts. One is to estimate the action value using the existing reward values; the other is to select actions based on the obtained action estimation values. The present invention uses (23) to estimate the action value, and the method for selecting actions adopts the upper confidence bound method. This method explores both the known experience and the unknown actions, that is, it simultaneously considers the known information and the degree of uncertainty of the actions, adds an uncertainty term to the original action estimation value, and then selects actions based on the two results. Based on the upper confidence bound method, at time step t, select the sensor that maximizes the two results

[0159]

[0160] Wherein, the first term is the action estimation value, representing the known information, that is, select the action with the largest action estimation value. The second term reflects the degree of uncertainty of the action. c > 0 represents the confidence level, t is the time step, indicating the number of times of selecting actions, and N t (a) represents the number of times the action a is selected in t selections. For the second term From the perspective of the denominator, if the number of times the action a is selected up to time step t is N t(a) is very large, then the overall result of the second term is small, and the agent will not choose action a but choose other actions with fewer selection times; from the perspective of the numerator, if the number of times action a has been selected up to time step t is N t (a) remains unchanged, then as t increases, the result of the second term increases, and action a has a high probability of being selected. Solving the optimal sensor selection problem with unknown sensor noise covariance based on the reinforcement learning method is the innovation of the present invention.

[0161] The said step 4 includes the following steps:

[0162] Step 4.1, Estimation of unknown sensor noise covariance.

[0163] Simplify formula (17), the noise covariance scalar of sensor s i can be calculated by the following formula: It can be calculated by the following formula:

[0164]

[0165] where the expected performance index J is unknown.

[0166] According to the result of formula (20), (26) can be transformed:

[0167]

[0168] That is, use the expected reward q(s i ) to replace the expected performance J.

[0169] The expected reward can be estimated using the existing rewards. The following results reflect the relationship between the estimated value and the expected reward:

[0170]

[0171] When the number of selection times approaches infinity, the estimated value is equivalent to the expected value.

[0172] When n → ∞, the following results are obtained:

[0173]

[0174] Therefore, the existing estimated value can be used to estimate the covariance of the noise according to (29). Estimating the noise covariance while selecting the optimal sensor is another innovation of the present invention.

[0175] Step 4.2, Design of the optimal sensor selection and unknown sensor noise covariance estimation algorithm based on reinforcement learning.

[0176] In the embodiment of the present invention, Algorithm 1 proposes how to select the optimal sensor by using the reinforcement learning method to minimize the expected performance index when the sensor noise covariance is unknown, and estimates the unknown noise covariance by using the obtained reward data.

[0177] The implementation of Algorithm 1 for the optimal sensor selection and unknown sensor noise covariance estimation algorithm based on reinforcement learning is as follows:

[0178] 1: Given the system parameter matrices A, B, Q, R, P(k f ) = F, parameter c, the mean μ0 and covariance φ0 of the initial state x(0). Set the terminal time k f and the initial estimate Q0;

[0179] 2: Select the sensor by using formula (25) and calculate the reward value R t ;

[0180] 3: For the selected sensor, estimate the current action value by using formula (23);

[0181] 4: If the number of selection times t > m (m is a positive integer), then end the selection process, otherwise return to step 4.2.2;

[0182] 5: After the selection is ended, first calculate P(0) and S(k), and calculate the estimated values of all sensor noise covariance scalars according to formula (29) and the obtained .

[0183] In the embodiment of the present invention, as Figures 1 - 3 shown, in order to more intuitively demonstrate the effectiveness of the optimal sensor selection method for linear quadratic control with unknown sensor noise covariance proposed by the present invention, the method proposed by the present invention is verified by simulation using MATLAB software;

[0184] Numerical simulation example:

[0185] The dynamic model of the discrete-time linear system is:

[0186]

[0187] where, the mean of the initial state x(0) is μ0 = [5 3] T , and the covariance matrix φ0 is a two-dimensional identity matrix. The terminal time k f = 10.

[0188] The system expected performance index is

[0189]

[0190] The number of sensors N = 10. To verify the accuracy of the results, the scalar of the sensor noise covariance is set to For each selection method, m = 1000 sensor selections are performed. Setting 1000 sensor selections as one round, and then 2000 rounds are carried out to obtain the average method performance.

[0191] In addition to the upper confidence bound method, in the numerical simulation, the embodiments of the present invention also compare the performance of three other methods. The first method is the greedy method; the second method is the ε-greedy method with the parameter ε = 0.05; the third method is the optimal initial value method with the initial value set to Q0 = -30, which is the performance index calculated when there is no noise in the system; the fourth method is the upper confidence bound method with c = 5 to control the confidence level.

[0192] The estimation method of the covariance scalar is as follows, taking the third sensor as an example. Using formula (30) and the action estimated value of selecting the third sensor, the noise covariance scalar of the third sensor can be estimated.

[0193]

[0194] In the embodiments of the present invention, M1, M2, M3, and M4 in the three figures respectively correspond to the above four methods. q(s1) is the expected reward of the first sensor, and its noise covariance is the smallest. Figure 1 The average rewards of the four methods in each selection are compared (the average reward refers to that in 2000 rounds, when selecting sensors at the same time step t, there will be 2000 reward values, and the average of these reward values is taken as the average reward obtained by a method at time step t). From Figure 1 It can be seen that the average reward of M4 (the upper confidence bound method) is the closest to q(s1), that is, this method can find the optimal sensor. The average reward of M2 (the optimal initial value method) is the second, and the average reward of M1 (the greedy method) is the lowest. Figure 2 The percentages of the four methods that can select the sensor with the smallest noise covariance in each selection are compared. The percentage of M4 (the upper confidence bound method) in selecting the optimal sensor is the largest. Figure 3 It is a comparison of the estimated noise covariance scalars of all sensors for each method and the estimation accuracy measured by the mean absolute percentage error. The upper confidence bound method has the best performance among all methods, and the estimation error of the noise covariance scalar is the smallest. Through Figure 1 , Figure 2 and Figure 3 , the effectiveness of the optimal sensor selection method for linear quadratic control with unknown sensor noise covariance based on reinforcement learning can be verified.

Claims

1. An optimal selection method for controlling unknown sensor noise covariance, characterized in that The method includes the following steps: Step 1: Design a linear quadratic state feedback controller based on the state measurement value, and establish an optimal sensor selection problem model with unknown sensor noise covariance; Step 2: Analyze the influence of sensor noise on system performance; Step 3: Transformation of the optimal sensor selection problem with unknown sensor noise covariance; Step 4: Propose an optimal sensor selection and unknown sensor noise covariance estimation algorithm based on reinforcement learning, including the following steps: Step 4.1: Estimation of unknown sensor noise covariance; Sensor s i The noise covariance scalar of is calculated by the following formula: where the expected performance index J is unknown; Perform transformation on (14): Replace the expected performance J with the expected reward q(s i ). The expected reward is estimated using the existing reward, and the following results reflect the relationship between the estimated value and the expected reward: When the number of selections approaches infinity, the estimated value is equivalent to the expected value; When n → ∞, the following results are obtained: Therefore, using the existing estimates Estimate the covariance of the noise according to (17); Step 4.2: Algorithm design: 1) Given the system parameter matrix, confidence level, mean μ0 and covariance φ0 of the initial state x(0); set the terminal time k f and the initial estimate Q0; 2) Select sensors using formula (14) and calculate the reward value R t ; 3) For the selected sensor, use the formula to estimate the current action value; 4) If the number of selections t > m, then end the selection process, otherwise return to step 2); 5) After the selection is completed, first calculate P(0) and S(k), and use the obtained Calculate the estimated values of all sensor noise covariance scalars according to formula (17). ​

Images