A fingerprint indoor positioning method based on reinforcement learning PPO algorithm

Abstract

The present invention belongs to the field of indoor positioning technology, and specifically relates to a fingerprint indoor positioning method based on a reinforcement learning PPO algorithm. The fingerprint indoor positioning method based on the reinforcement learning PPO algorithm proposed by the present invention allows the model to interact with the environment to generate training data, so as to obtain higher rewards as a drive to update the network. The method of the present invention no longer relies on a labeled fingerprint library, and the provision of positioning services and model updates can be carried out simultaneously, which greatly improves the deployment efficiency of the positioning system. As the environment changes, the method can collect the latest trajectory data to update the network so that it still has good positioning performance in the face of environmental changes.

Description

Technical Field

[0001] The invention belongs to the technical field of indoor positioning, and in particular relates to a fingerprint indoor positioning method based on a reinforcement learning PPO algorithm. Background Art

[0002] With the development of intelligent IoT technology, people's demand for location-based services is growing, both in the field of national defense and in daily life. Location services in indoor environments have attracted more attention in recent years. Indoor positioning technology based on RSS fingerprints has also attracted the attention of many researchers because it is not affected by complex indoor multipath effects and does not require additional sensor equipment.

[0003] The document “M.Altini, D.Brunelli, E.Farella, and L.Benini, “Bluetooth indoor localization with multiple neural networks,” in Proc.5th IEEE Int. Symp. Wireless Pervasive Comput. (ISWPC), 2010, pp.295–300” discloses a fingerprint indoor positioning method based on neural network. Like most fingerprint-based indoor positioning methods, the positioning process needs to be divided into several steps: 1) Offline collection of signal strength RSS at each grid point in a well-divided indoor environment and recording coordinate labels to establish a fingerprint library; 2) Model training phase: Use the collected fingerprint data to train the model parameters; 3) Online positioning phase: According to the RSS fingerprint information collected from the target to be positioned, the position coordinates are output through the trained neural network. This type of method does not need to worry about the multipath effect of wireless signals in complex indoor environments, and because low-cost wireless signals such as Wi-Fi and Bluetooth are usually used, the indoor positioning problem is well solved to a certain extent. However, its shortcomings are also obvious, mainly reflected in the following aspects: 1) Fingerprint collection of the positioning environment is necessary (relying on labeled data): In the offline stage, it is necessary to establish a fingerprint library in the indoor environment to be positioned, which is a very labor-intensive and financially intensive task. In addition, since the environment needs to be divided into grids in advance, the construction of most current fingerprint indoor positioning systems takes a long time before they can be put into use, and these methods cannot be applied well in practice. 2) As the environment changes, the positioning performance of the model will gradually decrease: After the neural network trains the network parameters based on the fingerprints in the offline stage, at this stage, as time goes by, the movement of indoor obstacles, the movement of AP positions, and factors such as temperature and humidity will cause the network effect to gradually decrease. At this time, it is often necessary to retrain the network, or use other algorithms to dynamically update the network to adapt to the new environment. Due to the above problems of this type of method, it is difficult to deploy the fingerprint indoor positioning system in actual application scenarios, and it is not practical. Summary of the invention

[0004] The purpose of the present invention is to solve the defects of several practical applications faced by the indoor positioning method based on RSS fingerprint in the technical background: the need to rely heavily on labeled fingerprint data, the time-consuming process of positioning system deployment, and the inability of positioning model to maintain performance over time, etc., and design a fingerprint indoor positioning method based on reinforcement learning PPO algorithm. The method uses reinforcement learning to achieve positioning and models tracking positioning as a Markov process. The training mechanism of reinforcement learning is to use the data obtained by the agent (Agent) constantly interacting with the environment for network training. After selecting an action (Action) according to the state (State) model of the current environment, the reward value (Reward) for this Action is obtained according to the designed reward function, so as to obtain a higher reward value as a guide to update the network. Since the training of the network only depends on the interaction data between the agent and the environment, these interaction data do not need to be manually collected and labeled, so that the positioning system presents a "self-experimentation, self-growth" mode.

[0005] The present invention is completely different from the traditional RSS fingerprint indoor positioning method in terms of data collection, positioning model training and positioning implementation process. On the one hand, this idea can save time and manpower costs by no longer relying on labeled data, and on the other hand, it allows the model to continue to update the network according to the latest interactive data after a period of use to cope with environmental changes.

[0006] The technical solution of the present invention is:

[0007] A fingerprint indoor positioning method based on reinforcement learning PPO algorithm includes the following steps:

[0008] S1. Install L signal generator APs in the positioning environment and record the installation location coordinates;

[0009] S2. Collect the untagged user continuous motion trajectory information in the positioning environment, record the starting point position coordinates (x0, y0) and regularly record the wireless signal strength RSS fingerprint of the middle point:

[0010] RSSI=[RSS0,RSS1,…,RSS L ] T

[0011] The RSS fingerprint data collected M times from the starting point constitutes a trajectory:

[0012]

[0013] Collect N trajectories to form training data, n = 1, 2, ... N;

[0014] S3. Construct a PPO network. The PPO network includes a policy network and an evaluation network, which are defined as A network and C network respectively. The input of A network is the state of the current agent, including the position coordinates of the agent at the previous moment and the RSS fingerprint information at the current moment. The tth state is defined as:

[0015] State t =[x t-1 ,y t-1 ,RSSI t ]

[0016] The input of the A network is:

[0017] A INPUT =[State t ]

[0018] The output of the A network is the mean and variance of the two physical quantities, the position, the direction and the step length, which are sent to the agent based on the last moment. The agent's final direction and step length are sampled from the normal distribution represented by their respective means and variances. The A network includes two networks with the same structure, one for generating the action instructions of the agent and the other for waiting to be updated, which are denoted as A and A respectively. old ; The C network is used to evaluate the expected value of the action after the agent takes action based on the output of the A network and reaches a new coordinate. The input of the C network is also a state, and the output is a value;

[0019] The PPO network is trained using the training data of step S2, specifically:

[0020] Based on the initial state, the agent inputs the mean variance of the forward direction and step length into the A network, samples the predicted movement direction and step length of the user at the next moment, and lets the agent move in this direction to obtain the new coordinates of the agent (x t+1 ,y t+1 ) and obtain a new State based on the signal strength measurement of the user at the new location t+1 ;

[0021] According to the reward function, the reward value of this step is obtained:

[0022]

[0023] in, Represents the threshold. When the i-th RSS value in the state is greater than When the distance d from the agent to the i-th AP is used t,i The reciprocal of is rewarded;

[0024] Repeat and record the state and reward value of each move. After completing all the steps of a trajectory, trace back the discounted value V of each step according to the PG algorithm process. discount :

[0025] V discount =gamma*V t+1 +r t

[0026] Gamma is the discount rate of the value, which reflects the impact of this step of action on the value of the subsequent steps. It is selected based on the actual situation.

[0027] Update A network:

[0028]

[0029] According to the importance sampling principle of PPO network: As A(s t ,a t ) to form the final cost function. old and θ represent the parameters of the two A networks waiting to be updated and generating data, respectively. θ (a t |s t )and (a t |s t ) is the probability value of selecting the same action in the same scene under their respective parameters. In the PPO algorithm, in order to prevent the distribution gap between the two networks from being too large, the truncation method is used to suppress their gap. The ε in the formula is the truncation parameter, which can be taken as 0.2 according to experience.

[0030] A(s t ,a t )for:

[0031] A(s t ,a t )=gamma*V t+1 +r t -V t

[0032] Update the C network, the cost function of the C network is A(s t ,a t );

[0033] After completing the training of one trajectory, proceed to the next trajectory training until the training is completed and the trained network parameters are saved;

[0034] S4. Real-time tracking and positioning of the user starts from the starting point. According to the real-time collected RSS fingerprint data, the virtual agent and the user are at the same starting point, input into the trained network, and output the movement direction and step length of the agent. Finally, the trajectory of the agent is the predicted movement trajectory of the user.

[0035] The beneficial effect of the present invention is that the method of the present invention does not need to rely on a fingerprint library with labels and does not need to divide the environment into grids in advance, and the cost of system deployment is much lower than that of traditional positioning methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of the positioning system of the present invention.

[0037] Figure 2 It is the A network structure of the reinforcement learning PPO algorithm used in the present invention.

[0038] Figure 3 The C network structure of the reinforcement learning PPO algorithm used in the present invention

[0039] Figure 4 This is the network update process of the reinforcement learning PPO algorithm used in the present invention. DETAILED DESCRIPTION

[0040] The present invention is described in detail below in conjunction with the accompanying drawings:

[0041] like Figure 1 As shown, the main steps of the present invention are as follows:

[0042] Step 1. Install the signal generator AP in the environment and record the installation location coordinates.

[0043] Step 2. Collect the unlabeled user continuous motion trajectory information in the environment, record the starting point position coordinates and regularly record the wireless signal strength RSS fingerprint of the intermediate point.

[0044] Step 3. Use the unlabeled trajectories collected in step 3 to train the reinforcement learning PPO network:

[0045] Step 3-1. The network will let the virtual agent and the user start from the same starting point, try to make each step of the agent coincide with the user's trajectory, and guide the neural network update through the reward obtained at each step.

[0046] Step 3-2. Save the network parameters after training.

[0047] Step 4. Real-time tracking and positioning:

[0048] Step 4-1. The user carries a signal receiver and prepares to set off from the starting point. The virtual agent and the user are at the same starting point.

[0049] Step 4-2. When the user is walking, the RSS information is measured once per unit time and sent to the network output command of the PPO algorithm to direct the movement of the intelligent body.

[0050] Step 4-3. The real-time location of the agent is the real-time location of the user being tracked and located.

[0051] As time goes by and the environment changes, the latest user unlabeled motion trajectories can be collected and the network can be updated with the latest trajectories to ensure that the network always maintains ideal positioning performance.

[0052] Example

[0053] This example uses a 30m×20m laboratory as the indoor positioning environment. Ten Bluetooth beacons are installed as APs in the environment. There is no need to divide the environment into grids. The upper left corner of the indoor environment is used as the coordinate origin. The Bluetooth beacon APs are evenly placed in the room in two columns, with 5 APs in each column, numbered 0, 1, …, 9.

[0054] After the AP base stations are deployed in the environment, you can start collecting unlabeled user trajectory data. The collector carries a mobile phone and walks randomly in the environment from a known starting point for a period of time. During the walking process, the RSS fingerprint of the person's location is collected every 5 seconds. The fingerprint is a 1×9 vector, in which the elements are the signal strength RSS (in dBm) of each AP at this location:

[0055] RSSI = [RSS0,RSS1,…,RSS9] T =[-57.12,-64.54,…,-74.20] T

[0056] The starting point (x0, y0) of this trajectory and the RSS fingerprint data collected 100 times are recorded to form a trajectory. Therefore, a trajectory consists of the following information:

[0057]

[0058] Collecting N trajectories can constitute the data for PPO network training.

[0059] Constructing a PPO network:

[0060] Preprocess the RSS values ​​in the fingerprint data so that they are distributed between 0 and 1, which is conducive to network training.

[0061] Construct and initialize the parameters of the A network (Actor network) in the PPO network, such as Figure 2 , and C network (Critic network), such as Figure 3 . The A network is a policy network, and the input is the current state of the agent. The state is defined as a 1×11 vector, which is the position coordinates of the agent at the previous moment and the RSS fingerprint information at the current moment. The output of the A network is the mean and variance of the two physical quantities, the position at the previous moment, the direction of advance and the step length, which are sent to the agent. The agent's final direction of advance and step length are sampled from the normal distribution represented by their respective means and variances. According to the PPO algorithm, two A networks with the same structure need to be constructed, one network is used to generate the agent's action instructions, and the other is waiting to be updated, denoted as A and A respectively. old .

[0062] State t =[x t-1 ,y t-1 ,RSSI t ]

[0063] A INPUT =[State t ]

[0064] A OUTPUT =[[StepAngle mu ,StepAngle sigma ],[StepSize mu ,StepSize sigma ]]

[0065] The C network is an evaluation network, which is used to evaluate the expected value of the action of the agent to reach a new coordinate after taking action based on the output of the A network. The input of the C network is also a state, and the output is a value:

[0066] C INPUT =[State t+1 ]

[0067] C OUTPUT =[V t+1 ]

[0068] The starting points of the trajectory data for the tracking agent and the user are both placed at (x0, y0).

[0069] like Figure 4 , the network update process is as follows: the trajectory starts to move, the agent is fed into the A network used to generate data according to the initial state to obtain the mean variance of the forward direction and step length, sample to obtain the predicted user's next moment's movement direction and step length, and let the agent move in this direction to obtain the agent's new coordinates (x t+1 ,yt+1 ) and get the new State based on the user's measurement value t+1 .

[0070] According to the reward function, we get the reward value (Reward) for this step:

[0071]

[0072] Among them Indicates a threshold. When the RSS value in the state is greater than When an element is selected, the inverse of the distance from the Agent to this AP is used as the reward.

[0073] Repeat the process and record the State and Reward of each move. After completing all the steps of a trajectory, trace back the discounted value V of each step according to the PG algorithm process:

[0074] V discount =gamma*V t+1 +r t # (2)

[0075] Update A network (PPO2):

[0076]

[0077]

[0078] where θ old and θ represent the parameters of the two A networks waiting to be updated and generating data, respectively, where A(s t ,a t ) represents TD_error in the PG algorithm, which is calculated based on the discount value V in (2):

[0079] A(s t ,a t )=gamma*V t+1 +r t -V t # (4)

[0080] Update C network: The cost function of C network is TD_error, the same as (4).

[0081] Conduct training on a new trajectory.

[0082] Save the trained network parameters and complete the training.

[0083] In the real-time positioning stage, the user starts from the starting point and inputs the RSS fingerprint data collected in real time into the trained A network, which outputs the movement direction and step length of the Agent. The final trajectory of the Agent is the predicted movement trajectory of the user.

[0084] Post-network update correction: Over time, if the positioning performance degrades, the latest unlabeled trajectory data can be collected to further update the data, and positioning and model updates can be performed simultaneously.

[0085] The present invention tracks and locates users in the test site and tests 30 random trajectories (9000 intermediate nodes). The results show that the trajectory matching degree reaches more than 83%, and the average positioning error of the sampling points in the middle of the trajectory is 3.1m.

[0086] Comparing the positioning method in the technical background with the method of the present invention, the method of the present invention and the positioning method in the technical background achieve similar results in positioning accuracy. However, in terms of the positioning process, the method of the present invention does not need to rely on a tagged fingerprint library or divide the environment into grids in advance, and the cost of system deployment is much lower than that of the method in the technical background. This method does not strictly distinguish between the offline stage and the online stage, and the positioning and model growth can be carried out simultaneously. The formation of the positioning model is a process of self-experimentation. Due to this feature, the positioning model can also continue to be updated according to the latest trajectory data after being used for a period of time to maintain satisfactory positioning performance. As shown in Table 1, this is a comparison between the positioning method in the background and the method proposed in the present invention.

[0087] Table 1: Comparison of traditional positioning methods in the background and the method proposed in the present invention

[0088]

[0089]

Claims

1. A fingerprint indoor positioning method based on reinforcement learning PPO algorithm, characterized in that: The following steps are involved: S1. Install L signal generator APs in the positioning environment and record the installation location coordinates; S2. Collect the untagged user continuous motion trajectory information in the positioning environment, record the starting point position coordinates (x0, y0) and regularly record the wireless signal strength RSS fingerprint of the middle point: RSSI=[RSS0,RSS1,...,RSS L ] T The RSS fingerprint data collected M times from the starting point constitutes a trajectory: Collect N trajectories to form training data, n = 1, 2, ... N; S3. Construct a PPO network. The PPO network includes a policy network and an evaluation network, which are defined as A network and C network respectively. The input of A network is the state of the current agent, including the position coordinates of the agent at the previous moment and the RSS fingerprint information at the current moment. The t-th state is defined as t=1, 2, ...L: State t =[x t-1 ,y t-1 ,RSSI t ] The input of the A network is: A INPUT =[State t ] The output of the A network is the mean and variance of the two physical quantities, the position at the last moment, the direction of advance and the step length, which are sent to the agent. The agent's final direction of advance and step length are sampled from the normal distribution represented by their respective means and variances. The A network includes two networks with the same structure, one for generating the action instructions of the agent and the other for waiting to be updated, which are denoted as A and A respectively. old ; The C network is used to evaluate the expected value of the action after the agent takes action based on the output of the A network to reach a new coordinate. The input of the C network is also a state, and the output is a value; The PPO network is trained using the training data of step S2, specifically: The agent inputs the mean variance of the forward direction and step length into the A network according to the user's initial state, samples the predicted movement direction and step length of the user at the next moment, and lets the agent move in this direction to obtain the new coordinates of the agent (x t+1 ,y t+1 ) and get the new State based on the user's measurement value t+1 ; According to the reward function, the reward value of this step is obtained: in, Indicates the threshold. When the RSS value in the state is greater than When the element is , the distance d from the agent to this AP is used t，i The reciprocal of is the reward; Repeat the process and record the status and reward value of each move. After completing all the steps of a trajectory, trace back the discounted value V of each step according to the PG algorithm process: V discount =gamma*V t+1 +r t Among them, gamma is the discount rate of value, which is selected according to the actual situation; Update A network: where θ old and θ represent the parameters of the two A networks waiting to be updated and generating data, respectively. θ (a t |s t )and is the probability value of selecting the same action in the same scene under each parameter, ε is the cutoff parameter, A(s t , a t )for: A(s t ,a t )=gamma*V t+1 +r t -V t Update the C network, the cost function A(s t , a t )for; After completing the training of one trajectory, proceed to the next trajectory training until the training is completed and the trained network parameters are saved; S4. Real-time tracking and positioning of users: Starting from the starting point, based on the real-time collected RSS fingerprint data, the virtual agent and the user are at the same starting point, input into the trained PPO network, and output the movement direction and step length of the agent. Finally, the trajectory of the agent is the predicted movement trajectory of the user.

Images