A trajectory localization method based on deep reinforcement learning for heterogeneous environments

Abstract

The present invention belongs to the field of indoor positioning technology, and specifically relates to a trajectory positioning method based on deep reinforcement learning for heterogeneous environments. The present invention makes full use of the observations in the environment and the historical dynamic information of the intelligent agent itself, and uses the position of the intelligent agent, the RSS vector with device heterogeneity in the environment, and the historical actions of the past n moments as states for action selection. Then, based on the near-field conditions, APs corresponding to a stronger RSS threshold are selected to form a selected APs set, and then the final reward value is calculated according to the size of the set. The position of the intelligent agent is estimated according to the various elements designed in the MDP, and the reward value is used as a learning guide to iteratively train the deep reinforcement learning algorithm based on the experience replay mechanism and the target network. The present invention obtains simulated RSS data with device heterogeneity based on the path loss model. Experimental results show that the method proposed by the present invention can achieve higher positioning accuracy and has a certain robustness in processing heterogeneous RSS data.

Description

Technical Field

[0001] The present invention belongs to the field of indoor positioning technology, and specifically relates to a trajectory positioning method based on deep reinforcement learning for heterogeneous environments. Background Art

[0002] With the advent of the 5G and big data analysis era, people's business demand for location-based services (LBSs) continues to increase, and indoor positioning technology is a key component in LBSs applications. Natural satellite navigation systems, such as the Global Positioning System (GPS), can obtain sufficient positioning accuracy and navigation performance in outdoor environments. However, in indoor scenarios, due to the complex and changeable environment and the obstruction of building materials, a variety of different signal sources (such as WiFi, geomagnetism, and Bluetooth, etc.) can be used for indoor positioning based on different principles. Among them, indoor positioning technology based on wireless received signal strength (RSS) has been widely used because RSS can be easily accessed from different networks.

[0003] In recent years, reinforcement learning-based methods have gained a strong competitive advantage in solving complex decision-making problems such as AlphaGo, and have received widespread attention in many other fields, including indoor positioning. In actual indoor scenarios, we usually encounter state information of high-dimensional environmental observations, and face problems such as large environmental changes and heterogeneity of devices. Traditional reinforcement learning-based positioning methods cannot handle high-dimensional state inputs. Indoor positioning algorithms based on deep reinforcement learning can solve similar problems, but existing methods do not fully utilize the historical dynamic information of environmental observations and the agent itself, and do not consider the impact of device heterogeneity on positioning performance when processing RSS data.

[0004] The paper "Ebrahimi, Dariush, et al." Autonomous UAV trajectory for localizing ground objects: A reinforcement learning approach." IEEE Transactions on Mobile Computing 20.4 (2020): 1312-1324" uses the classic Q-Learning algorithm to localize ground objects with the assistance of a swarm of drones. However, this method cannot handle state inputs in high-dimensional real environments due to the limitations of the reinforcement learning algorithm itself. The paper "Li Y, Hu X, Zhuang Y, et al. Deep Reinforcement Learning (DRL): Another Perspective for Unsupervised Wireless Localization [J]. IEEE Internet of Things Journal, 2020, 7 (7): 6279-6287" uses a deep reinforcement learning algorithm to model the localization problem as a continuous wireless localization process, and proposes a reward setting mechanism for unsupervised wireless localization, which automatically extracts robust landmark data from unlabeled wireless RSS to achieve high-precision localization. However, this method does not make full use of the historical dynamics of the agent itself, which may cause the agent's exploration of the environment to fall into a repetitive cycle. In addition, this method does not consider how to deal with heterogeneous indoor positioning environments. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the above-mentioned technologies and provide a trajectory positioning method based on deep reinforcement learning for heterogeneous environments. Figure 1As shown, the present invention models a simulation environment with a positioning area size of 10m*10m, which is divided into grids of the same size, with each grid point spacing of 1m, and each specific grid point position is identified by an RSS value vector. We place wireless access points (APs) at specific grid points, with the position of the agent at time t-1, the RSS measurement with device heterogeneity received at time t, and the historical action vector obtained by the first-in first-out (FIFO) rule of the agent at the previous n moments as the state. In a training or test trajectory, the agent moves according to the designed direction through the action selected by the algorithm, and the position reached after each step of the action movement is the estimated position at time t. After each move of the agent, the reward function designed in the reinforcement learning Markov decision process (MDP) will feedback the action, and the agent continuously adjusts the learning strategy according to the feedback reward value, so as to finally obtain an optimal and stable behavior strategy through training iteration.

[0006] The technical solution of the present invention is: a trajectory positioning method based on deep reinforcement learning for heterogeneous environments, comprising the following steps:

[0007] S1, divide the area to be located into equally spaced grids and place wireless APs at set locations;

[0008] S2. Establish an RSS fingerprint database with device heterogeneity: Generate RSS measurement data from APs in the environment to be positioned based on the path loss model, and obtain a simulated RSS data set that meets the device heterogeneity characteristics by changing the noise parameters in the model and adding a preset mean disturbance term to the RSS measurement value;

[0009] S3, generate random trajectory data and corresponding RSS trajectory data: randomly generate and obtain the trajectory starting point data of the agent, and the action selection of the agent at time t is also completely random. Each time, the position at time t+1 is obtained according to the action execution rules designed in the action space in the Markov decision process, and at each position of the trajectory, a random one is selected from the pre-established RSS fingerprint database as the RSS value of the position of the trajectory at that time, so as to finally obtain the corresponding RSS trajectory data;

[0010] S4. Design the state space, action space and reward function in the Markov decision process.

[0011] Furthermore, the specific steps of step S4 are:

[0012] S41. Agent: The agent is modeled as the trajectory localization algorithm itself.

[0013] S42, environment: the field with which the agent interacts, that is, the simulation environment of the area to be located mentioned in step S1.

[0014] S43, state space: The estimated position of the agent at time t-1 (when t=1, the starting position on the real trajectory data is used), the RSS vector with device heterogeneity at time t, and the historical action vector of the agent itself are taken as the state.

[0015] S44, Action Space: The action of the intelligent agent is designed to be nine-dimensional. The intelligent agent can move up, down, left, right, and any direction 45° in between, or remain still. The moving step length of the intelligent agent is the unit grid width.

[0016] S45, reward function: based on the near-field conditions, corresponding APs stronger than the RSS threshold are selected to form a set of APs with a selected RSS value, and a final reward value is calculated according to the size of the set.

[0017] S5. Estimate the position of the agent at time t based on the Deep Q-Network (DQN). The state is used as the input of the neural network, and the action selected by the DQN algorithm according to the greedy strategy is the output of the network, from which the reward value can be obtained. For heterogeneous RSS data, we make multiple judgments on the same position point at the same time, and finally execute the action corresponding to the high reward value to estimate the position of the agent at time t.

[0018] S6. The evaluation network in the DQN algorithm is trained based on the experience replay mechanism and the target network.

[0019] Furthermore, the specific training steps of step S6 are:

[0020] S61. Store the experience gained by the agent each time it interacts with the environment into the memory pool. Every time the agent updates its state, it means that a new experience e = [s, a, r, s'] is generated, which can be stored in the memory pool D. Where s and s' are the states before and after the update, a is the action selected by the agent in state s, and r is the feedback reward value obtained by the agent after executing the selected action a.

[0021] S62. When the number of experiences stored in the experience pool D reaches a preset threshold, the DQN algorithm starts to update network parameters such as connection weights and thresholds in the DQN at a fixed frequency to iterate the training of the deep neural network (DNN).

[0022] S63. When training the DQN network, training samples of batch size are uniformly and randomly selected from the experience pool D each time.

[0023] S64, DQN algorithm uses two networks with consistent structure and independent parameters, namely: real-time updated evaluation network and periodically updated target network. For the latter, its network parameters are directly copied from the former after a certain update frequency. For the jth experience e in the batch experience training sample selected in step S6-3, j =[s j ,a j ,r j ,s' j ], the current state s j and action a j As the input of the evaluation network, it is used to calculate the network loss value, and the next moment state s' j The joint reward value r will be used as the input of the target network j Together they are used to calculate the network loss value.

[0024] S65. Calculate the optimal objective function Y of the target network in DQN based on formula (1) i :

[0025]

[0026] Among them, r, s' and a' are the reward value in experience, the state and action at the next moment, respectively, and γ is the decay factor. are the neural network parameters of the target network.

[0027] S66. Use mean square error to calculate the loss function L of the DQN network:

[0028] L=(Y i -Q(s,a;θ) 2 (2)

[0029] Here, θ is the neural network parameter in the evaluation network.

[0030] S67. Execute the gradient descent strategy based on formula (3):

[0031]

[0032] The network parameters are gradually optimized using the back propagation algorithm, where In the experience e=(s,a i ,r,s').

[0033] S7: Process the target with the trained network model to estimate the position of the target. Based on the neural network model trained in the above steps, the trajectory data and the corresponding RSS trajectory data with device heterogeneity characteristics are used as input to estimate the position of the target.

[0034] The beneficial effects of the present invention are as follows: the present invention uses the DQN algorithm in deep reinforcement learning, and uses a small amount of prior knowledge such as the location of APs in the environment and the starting point of the trajectory, and continuously trains based on the designed reward function to obtain the optimal behavior strategy. When processing RSS data with device heterogeneity, it performs multiple judgments and selects actions corresponding to high reward values. In addition, the algorithm makes full use of the observations in the environment and the historical dynamic information of the intelligent agent itself, so that the information in the state space has enough energy to tell us what happened in the past, avoiding the intelligent agent's search trajectory from falling into a repeated cycle. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of the simulation environment of the area to be located established for the algorithm;

[0036] Figure 2 This is a flow chart of the trajectory localization method based on deep reinforcement learning for heterogeneous environments;

[0037] Figure 3 The kernel density distribution of the average positioning error of the trajectories of the background technology method and the method of the present invention under different heterogeneous data;

[0038] Figure 4 The figure shows a comparison of the average trajectory positioning errors of the background technology method and the method of the present invention under different heterogeneous data. DETAILED DESCRIPTION

[0039] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and embodiments:

[0040] like Figure 2 As shown, the present invention mainly comprises the following steps:

[0041] Step 1. Establish a 10m*10m simulation positioning environment. Divide it into grids with equal spacing of 1m. The algorithm sets up 12 wireless APs and places them at the following grid points: (2, 2), (5, 2), (9, 2), (2, 5), (5, 5), (9, 5), (5, 7), (9, 7), (2, 9), (5, 9) and (9, 9). The RSS signals collected at each grid point in the simulation environment are obtained from these 12 APs. Special processing is performed when the distance between the transmitter and the receiver is zero.

[0042] Step 2. Establish an RSS fingerprint database with device heterogeneity based on the path loss model.

[0043] Furthermore, the generation of a simulation data set with device heterogeneity in step 2 can be divided into the following three steps in detail:

[0044] Step 21. Use the path loss model shown in formula (4) to obtain the basic RSS simulation data set:

[0045] r l = -(10n)log 10 (d l )+A, l=1,2,...,L (4)

[0046] Among them, r l is the RSS value transmitted from the lth sensor in dBm, n is the signal propagation constant, which is usually 2 in free space, and d l is the Euclidean distance between the signal source and the lth sensor, and A is the offset RSS reading at a distance of 1 m from the transmitter.

[0047] Step 22. Add simulated noise δ to the RSS data obtained in step 2-1, which can be expressed by formula (5):

[0048]

[0049] Among them, β is the key noise parameter, the unit is dB, and changing the value of β can obtain RSS data sets under different simulation environments. Among them, the smaller the value of β, the greater the noise of the RSS simulation data, and the greater the difference in its value distribution.

[0050] Step 23. Further add a mean disturbance term to the RSS data obtained in step 2-2. The present invention analyzes the distribution differences of the RSS data obtained under different conditions and finally finds that when the values ​​of β are β 1 =2dB, β 2 =10dB and β 3 =100dB, the distribution difference between the RSS data is the largest. Therefore, we add an additional mean value of α to the RSS values ​​obtained under these three conditions. 1 , α 2 and α 3 The disturbance term is used to simulate the distribution difference of RSS data caused by device heterogeneity in real environment.

[0051] Step 3. Generate random trajectory data and corresponding RSS trajectory data. The present invention uses 1000 100-step and 100 100-step trajectory data as training and test samples, respectively.

[0052] Furthermore, the generation of trajectory data in step 3 can be divided into the following two steps:

[0053] Step 31. Generate random agent movement trajectory data. The movement trajectory data of the agent, including the starting point, is completely random, and each position is obtained according to the action space designed in the MDP.

[0054] Step 32. Generate RSS trajectory data corresponding to the random moving trajectory. When generating simulated RSS data with device heterogeneity based on step 2, each grid point collects RSS signals several times. When the agent moves to each trajectory position, it randomly selects one of the collected RSS values ​​as the RSS value corresponding to the position on the trajectory.

[0055] Step 4. Design elements such as the state space, action space, and reward function in the MDP.

[0056] Furthermore, the specific design steps for each element of the MDP in the DQN algorithm in step 4 are:

[0057] Step 41. Agent: The agent is the algorithm itself proposed in the present invention, which can observe and perceive the environment and receive information such as position and measurement in the environment.

[0058] Step 42. Environment: The environment is the field with which the agent interacts. In the positioning algorithm proposed in the present invention, the environment corresponds to the entire indoor environment where the area to be positioned is located, such as Figure 1 shown.

[0059] Step 43. Action space: The action space of the agent is composed of a nine-dimensional vector, including moving up, down, left, right, and a direction in between, as well as staying still. The execution rule of the action is to change the width of one grid point each time in the moving direction step.

[0060] Step 44. State space: The state space dimension in the MDP model corresponds to the input dimension of the DQN algorithm, that is, the number of neurons in the DNN input layer. The state space S in the trajectory localization algorithm proposed in the present invention can be expressed by formula (6):

[0061] S = [x, y, RSS, h] T (6)

[0062] Among them, (x, y) is the position coordinate of the agent at time t-1. When t=1, (x, y) corresponds to the real starting point of the trajectory; the vector RSS is the RSS measurement perceived by the agent at time t, and its dimension is related to the number of wireless APs in the environment; h is a historical action vector containing the most recent n moments in the past. The present invention inputs the above three parts as states into the DQN algorithm network to estimate the position of the agent at time t.

[0063] The historical action vector h is one-hot encoded (each action is represented by a binary vector, the bit corresponding to the action is 1, and the other bits are 0). In the present invention, n=10, that is, It can be seen that the dimension of vector h is higher than the sum of the dimensions of the first two states. Therefore, it can be considered that h has enough energy to tell us what happened in the past, which means that h is useful for preventing the agent's search trajectory from falling into a repetitive cycle. Using historical dynamic information allows the agent to learn from its own history and environment and find the best behavior strategy to accurately locate the target.

[0064] The transformation f of the historical action vector h at time t a (h t ) is updated according to the FIFO rule, which can be expressed as:

[0065] h t+1 =f a (h t ,a t ) (7)

[0066] Among them, h t and h t+1 are the historical action vectors before and after the transition at time t and time t+1, respectively, t is the action chosen and executed by the agent at time t.

[0067] Step 44. Reward function: Utilize the phenomenon of near-field conditions: a very weak RSS measurement value cannot represent a long distance because it may be a very strong RSS value reduced by interference from various factors in the environment; on the contrary, a very strong RSS value can represent a short distance because various unfavorable factors in the environment will only weaken the RSS value instead of enhancing it.

[0068] When the agent receives the RSS measurement value RSS from the jth AP at the i-th grid point i,j Stronger than a pre-set threshold value θ RSS When , it is considered that the agent should be located near the jth AP position, and then according to the actual distance d between the agent and the AP i,j Determine how much reward value to give. We use formula (8) to express the calculation of reward function R:

[0069]

[0070] Among them, vector C represents RSS i,j Stronger than the threshold θ RSS Let ||C|| represent the size of the set. The specific calculation method of function r(·) is shown in formula (9):

[0071]

[0072] In formula (8), when the RSS values ​​received by the agent at the i-th grid point on all APs in the environment are weaker than θ RSS That is, if ||C||=0, the reward value is 0; if there is a RSS value stronger than θ RSS , then d i,j Substitute into formula (9) to calculate; if the RSS collected by the agent at the i-th grid point has more than one value stronger than θ RSS , select the AP corresponding to the maximum RSS value, represents the Euclidean distance between the agent’s position at the i-th grid point and the AP’s position.

[0073] In formula (9), that is, when ||C||≠0, it is necessary to determine the distance d between the agent and the jth AP selected at this time. i,j Is it within the distance threshold θ d If d i,j Less than the distance threshold θ d , then the reward value is the inverse of the distance, otherwise it is the negative of the distance. In order to avoid extreme values ​​when the denominator is zero, a minimum distance d between the agent and the AP is set min .

[0074] Step 5. Estimate the position of the agent based on the MDP elements designed in step 4, obtain the reward value and update the state. Each time the agent interacts with the environment, it will obtain a new experience e = [s, a, r, s'].

[0075] Step 6. Train the evaluation network in the DQN algorithm based on the experience replay mechanism and the target network.

[0076] Furthermore, the specific training steps of step 6 are:

[0077] Step 61. Store each experience e = [s, a, r, s'] obtained in step 5 into the memory pool D;

[0078] Step 62: uniformly randomly select batch-sized training samples from the experience pool D to update the neural network parameters;

[0079] Step 63: Update the evaluation network parameters in the DQN algorithm in real time, and regularly update the parameters of the target network;

[0080] Step 64: Execute the gradient descent strategy based on formula (3) and use the back propagation algorithm to continuously optimize the network parameters.

[0081] Step 7. Use the RSS trajectory data with device heterogeneity as a test sample and input it into the trained evaluation network to obtain the final target positioning result.

[0082] Example

[0083] The present invention uses 100 trajectories of 100 steps as test samples in a 10m*10m grid simulation environment with equal spacing of 1m to verify the effect of the method proposed in the present invention. The DNN model involved in the algorithm contains 4 fully connected layers, and the number of neurons in each layer is the state space dimension, 200, 200 and the action space dimension, that is, the input and output dimensions of the DNN in the DQN algorithm correspond to the dimensions of the state space and action space designed in the MDP respectively.

[0084] The present invention tests RSS data with device heterogeneity and verifies the superiority of the proposed algorithm. The first group of experiments compares the trajectory average positioning error kernel density distribution of the background technology method and the method of the present invention under two different sets of heterogeneous data, such as Figure 3 As shown. We set the noise parameters in the path loss model to β 1 , β 3 , and add the mean value α 1 , α 3 The disturbance term of , to form the first set of heterogeneous data; the noise parameters are β 2 , β 3 , and add the mean value α 2 , α 3 To form a second set of heterogeneous data, we can use the disturbance term of Figure 3 It can be intuitively seen that the positioning performance of the method of the present invention is more stable than that of the background technology method, and the average positioning error of the trajectory of the method of the present invention is smaller than that of the background technology method as a whole, and this rule exists under both sets of heterogeneous data. Quantitatively speaking, the average positioning error of the trajectory under the first set of heterogeneous data is 94% distributed between 1.20 and 1.28m by the method of the present invention, and 92% distributed between 1.20 and 1.90m by the background technology method; the average positioning error of the trajectory under the second set of heterogeneous data is 99% distributed between 1.05 and 1.16m by the method of the present invention, and 89% distributed between 1.05 and 1.42m by the background technology method.

[0085] The second set of experiments compared the average trajectory positioning error of the background technology method and the method of the present invention under RSS data with device heterogeneity. Figure 3The average positioning error of the trajectories of 100 test trajectory samples obtained by the background technology method and the method of the present invention under different noise parameters and mean disturbance terms is plotted. The method of the present invention only uses a small amount of prior knowledge in the environment, while the background technology method uses all label information of all samples for position estimation. Qualitatively, the average positioning error of the trajectory of the method of the present invention under each set of heterogeneous data is basically lower than that of all background technology methods. When conducting quantitative analysis, we select the second set of heterogeneous data corresponding to the maximum value of the average positioning error of the trajectory in the method of the present invention. In this case, the average positioning error of the trajectory of the method of the present invention is 1.242m, while the average positioning errors of the trajectories of the background technology methods 1, 2, and 3 are: 1.475m, 5.077m, and 5.270m, respectively. In summary, the trajectory positioning method based on the DQN algorithm proposed in the present invention can achieve good positioning accuracy by making full use of the observations in the environment and the historical dynamic information of the intelligent agent itself. The method also has certain robustness when facing heterogeneous indoor positioning environments.

Claims

1. A trajectory localization method based on deep reinforcement learning for heterogeneous environments, characterized in that: The following steps are involved: S1, divide the area to be located into equally spaced grids and place wireless APs at set locations; S2. Establish an RSS fingerprint database with device heterogeneity: Generate RSS measurement data from APs in the environment to be positioned based on the path loss model, and obtain a simulated RSS data set that meets the device heterogeneity characteristics by changing the noise parameters in the model and adding a preset mean disturbance term to the RSS measurement value; S3, generate random trajectory data and corresponding RSS trajectory data: randomly generate and obtain the trajectory starting point data of the agent, and the action selection of the agent at time t is also completely random. Each time, the position at time t+1 is obtained according to the action execution rules designed in the action space in the Markov decision process, and at each position of the trajectory, a random one is selected from the pre-established RSS fingerprint database as the RSS value of the position of the trajectory at that time, so as to finally obtain the corresponding RSS trajectory data; S4. Design the state space, action space and reward function in the Markov decision process: S41, Agent: The agent is modeled as the trajectory localization algorithm itself; S42, environment: the area with which the agent interacts, i.e. the area to be located in S1; S43, state space: the estimated position of the agent at time t-1, the RSS vector with device heterogeneity at time t, and the historical action vector of the agent itself are taken as the state; S44, Action Space: The action of the agent is designed to be nine-dimensional. The agent can move in one direction each time: up, down, left, right, upper left, lower left, upper right, lower right, or stand still. The upper left refers to the direction of 45° between the upper and left directions, the lower left refers to the direction of 45° between the lower and left directions, the upper right refers to the direction of 45° between the upper and right directions, and the lower right refers to the direction of 45° between the lower and right directions. The moving step length of the agent is the unit grid width. S45, reward function: select corresponding APs stronger than the RSS threshold based on the near-field condition to form a set of APs with a selected RSS value, and calculate the final reward value according to the size of the set; S5. Estimate the position of the agent at time t based on the deep Q network: With the state as the input of the neural network, the action selected by the DQN algorithm according to the greedy strategy is the output of the network, from which the reward value can be obtained; for heterogeneous RSS data, multiple judgments are made on the same position point at the same time, and finally the action corresponding to the high reward value is executed to estimate the position of the agent at time t; S6. Evaluation network in DQN algorithm based on experience replay mechanism and target network training: S61, storing the experience gained by the agent from each interaction with the environment into a memory pool; Every time the agent updates its state, a new experience e = [s, a, r, s'] is generated, which can be stored in the memory pool D, where s and s' are the states before and after the update, a is the action selected by the agent in state s, and r is the feedback reward value obtained by the agent after executing the selected action a; S62. When the number of experiences stored in the experience pool D reaches a preset threshold, the DQN algorithm starts to update the network parameters in the DQN at a fixed frequency to iterate the training of the deep neural network. S63. When training the DQN network, a batch size of training samples is uniformly and randomly selected from the experience pool D each time; S64. Two networks with consistent structures and independent parameters are used in the DQN algorithm, namely: a real-time updated evaluation network and a regularly updated target network. For the target network, its network parameters are directly copied from the evaluation network after a certain update frequency. For the jth experience e in the batch experience training sample selected in step S63, j =[s j ,a j ,r j ,s' j ], the current state s j and action a j As the input of the evaluation network, it is used to calculate the network loss value, and the next moment state s' j The joint reward value r will be used as the input of the target network j Used together to calculate the network loss value; S65. Calculate the optimal objective function Y of the target network in DQN i : Among them, r, s' and a' are the reward value in experience, the state and action at the next moment, respectively, and γ is the decay factor. are the neural network parameters of the target network; S66. Use mean square error to calculate the loss function L of the DQN network: L=(Y i -Q(s,a;θ)) 2 Where θ is the neural network parameter in the evaluation network; S67, execute the gradient descent strategy: The back propagation algorithm is used to gradually optimize the network parameters, where E (s,ai,r,s') In the experience e=(s,a i ,r,s') under the expected value; S7. Use the trained network model to process the target and estimate the position of the target.

Images