Indoor positioning and navigation method based on fusion of wi-fi fingerprint information and inertial sensor information

Abstract

An indoor positioning and navigation method based on the fusion of Wi-Fi fingerprint information and inertial sensor information. The method comprises: collecting Wi-Fi fingerprint data of a target device, and inputting the Wi-Fi fingerprint data into a trained fingerprint positioning model to obtain an observed global coordinate value; using an inertial sensor to collect an acceleration signal and an angular velocity signal of the target device, and inputting the acceleration signal and the angular velocity signal into a trained trajectory tracking model to obtain an observed relative displacement amount value, wherein the observed relative displacement amount value reflects a movement trajectory of the target device; and using extended Kalman filtering to fuse the observed global coordinate value and the observed relative displacement amount value to obtain fused position information as a positioning result of the target device. The method can effectively fuse inertial sensor information and received Wi-Fi fingerprint information which are integrated into a smart device, thereby improving the accuracy and robustness of indoor positioning.

Description

Indoor positioning and navigation method based on WiFi fingerprint and inertial sensor information fusion Technical Field

[0001] The present invention relates to the field of computer application technology, and more particularly to an indoor positioning and navigation method based on the fusion of WiFi fingerprint and inertial sensor information. Background Art

[0002] With the development of social and economic levels, people's daily travel is increasingly dependent on positioning services. Indoor positioning technology plays a vital role in emergency evacuation and rescue, smart homes, crowd monitoring, parking lot vehicle positioning, and patient medical guidance. WiFi and inertial sensors (also known as inertial odometry) have rapidly become the mainstream indoor positioning technologies for wearable device platforms due to their simple deployment and low cost.

[0003] In existing technologies, indoor positioning technology can be divided into two categories based on usage conditions: wireless positioning technology that relies on external devices and inertial positioning technology that does not. Wireless positioning technology relies on signals sent by external sources, such as WiFi, Bluetooth, ultra-wideband (UWB), radio frequency identification (RFID), ultrasound, and infrared. The user's terminal then receives and processes the signals, and a specific algorithm calculates the user's current location. Therefore, this type of method requires the deployment of external signal transmission equipment in the indoor environment. Inertial positioning technology based on inertial sensors has high positioning accuracy in the short term, but is easily affected by error accumulation over long periods of time, resulting in a significant reduction in positioning accuracy. Currently, inertial positioning mainly includes three methods: inertial navigation systems, inertial pedestrian dead reckoning, and inertial odometry based on deep learning.

[0004] The advantages and disadvantages of indoor positioning technology using a single method of wireless or inertial sensors are quite obvious, and the fusion of multiple technologies can largely make up for each other's shortcomings, thereby improving positioning accuracy. The commonly used fusion methods currently include extended Kalman filtering and particle filtering. For example, the literature (Poulose A, Han D S. Hybrid indoor localization using IMU sensors and smartphone camera [J]. Sensors, 2019, 19 (23): 5084.) proposed a smartphone camera and inertial sensor fusion positioning system based on linear Kalman filtering. The average positioning error of the system reached 0.069 meters. For another example, the literature (Yu N, Zhan X, Zhao S, et al. A precise dead reckoning algorithm based on Bluetooth and multiple sensors [J]. IEEE Internet of Things Journal, 2017, 5 (1): 336-351) uses a Kalman filter to fuse Bluetooth and multiple inertial sensor data, and the positioning error is within 0.8 meters. However, the use of cameras will bring problems such as high energy consumption and privacy security to smart devices, while wireless positioning technologies such as Bluetooth require the deployment of a large number of additional external devices in indoor environments. Therefore, the integration of WiFi and inertial sensors has obvious advantages.

[0005] While indoor positioning technology plays a vital role in emergency evacuation and rescue, augmented reality, and patient access to healthcare, GPS signals significantly attenuate indoors, making accurate positioning impossible. Consequently, researchers have proposed various approaches based on technologies such as WiFi, inertial sensors, infrared, ultrasound, Bluetooth, RFID, and cameras. These technologies each have their own advantages and disadvantages in different application scenarios, and considerations such as deployment cost, positioning accuracy, stability, and security are influencing their development.

[0006] For example, both ultrasonic and infrared require specific signal receiving equipment, and smartphones lack the necessary hardware, limiting their widespread adoption. Bluetooth, RFID, and cameras are available on smartphones, but Bluetooth and RFID have shorter ranges and require more hardware, while cameras pose privacy and security concerns and high power consumption. Smartphones now include built-in sensors such as gyroscopes, accelerometers, magnetometers, and Wi-Fi signal receiving modules. Given the ubiquity of smartphones and the large number of routers already installed in public buildings, Wi-Fi and inertial sensor positioning offer significant advantages.

[0007] WiFi-based indoor positioning methods mainly include geometric and fingerprint methods. Geometric methods use triangulation based on angle of arrival, time of flight, or time of arrival to achieve positioning. Fingerprint methods are more accurate than geometric methods. Common fingerprint features include RSSI, CSI, and signal-to-noise ratio. Compared with RSSI, CSI is more stable and accurate, but it requires specific signal acquisition software, which limits its application.

[0008] Classic positioning algorithms based on IMU (inertial measurement unit, typically containing three single-axis accelerometers, gyroscopes, and magnetometers) include physics-based (such as IMU double integration) or heuristic-based positioning methods. The IMUs currently installed in smartphones are miniaturized and low-cost, with relatively large measurement errors, and physics-based algorithms will quickly amplify the errors. In addition, in real scenarios, users are not restricted in how they carry their phones, and heuristic-based algorithms have certain limitations. With the development of deep learning, data-driven deep inertial odometry is gaining more and more attention. Deep inertial odometry uses the powerful feature extraction and data fitting capabilities of deep learning to derive results directly from inertial sensor signals, which has obvious advantages over classic algorithms.

[0009] Furthermore, while single indoor positioning technologies have both significant advantages and disadvantages, multimodal sensor fusion can largely offset their respective shortcomings. Fusion of Wi-Fi and IMUs can achieve consistent and more accurate positioning results over long periods of time. However, most current research focuses on integrating classic IMU algorithms (such as PDR) with Wi-Fi positioning, lacking in-depth solutions for integrating inertial odometry with Wi-Fi.

[0010] Summary of the Invention

[0011] The purpose of the present invention is to overcome the above-mentioned shortcomings of the prior art and provide an indoor positioning and navigation method based on the fusion of WiFi fingerprint and inertial sensor information. The method comprises the following steps:

[0012] Collect the WiFi fingerprint data of the target device and input it into the trained fingerprint positioning model to obtain global coordinate observation values;

[0013] Using an inertial sensor to collect acceleration signals and angular velocity signals of the target device, and inputting them into a trained trajectory tracking model to obtain relative displacement observation values, wherein the relative displacement observation values ​​reflect the movement trajectory of the target device;

[0014] The global coordinate observation value and the relative displacement observation value are fused using an extended Kalman filter to obtain fused position information as a positioning result of the target device.

[0015] Compared with the existing technology, the advantage of the present invention is that, in response to the shortcomings of WiFi fingerprint positioning and independent positioning of depth inertial odometer, an indoor positioning and navigation method based on the fusion of WiFi fingerprint and inertial sensor information is proposed. This method can effectively integrate the inertial sensor information integrated in the portable smart device and the received WiFi fingerprint information, thereby improving the accuracy and robustness of indoor positioning.

[0016] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0018] FIG1 is a flow chart of an indoor positioning and navigation method based on WiFi fingerprint and inertial sensor information fusion according to one embodiment of the present invention;

[0019] FIG2 is an RSSI histogram according to one embodiment of the present invention;

[0020] FIG3 is a schematic diagram of AP coverage and AP position estimation according to an embodiment of the present invention;

[0021] FIG4 is a schematic diagram of WiFi fingerprint data enhancement according to an embodiment of the present invention;

[0022] FIG5 is a schematic diagram of the structure of a denoising autoencoder and a CNN regression network according to an embodiment of the present invention;

[0023] FIG6 is a data processing flow chart according to an embodiment of the present invention;

[0024] FIG7 is a schematic diagram of a data segmentation method according to an embodiment of the present invention;

[0025] FIG8 is a schematic diagram of a deep inertial odometry network structure according to an embodiment of the present invention;

[0026] FIG9 is a schematic diagram of the structure of a fusion positioning system according to an embodiment of the present invention;

[0027] FIG10 is a schematic diagram of CDF curves of different algorithms according to an embodiment of the present invention;

[0028] FIG11 is a schematic diagram showing a comparison of trajectories of different algorithms in a certain sample according to an embodiment of the present invention;

[0029] FIG12 is a schematic diagram of evaluating different initial heading offset angles according to an embodiment of the present invention;

[0030] FIG13 is a schematic diagram showing a comparison of trajectories with different initial heading angle deviations according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0032] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0033] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0034] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0035] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0036] This paper uses deep learning methods to study both WiFi fingerprint positioning and deep inertial odometry technologies. It then fuses the results of these two methods using an extended Kalman filter, improving the stability and accuracy of smart device positioning. Smart devices (or target devices) include smartphones and tablets, and this article uses smartphones as an example.

[0037] As shown in FIG1 , the provided indoor positioning and navigation method based on WiFi fingerprint and inertial sensor information fusion includes the following steps:

[0038] Step S110: Acquire a WiFi fingerprint data set and an inertial navigation data set.

[0039] In the following, RSSI is used as fingerprint data (also called WiFi fingerprint data) for example, but it should be understood that CSI or other indicators can also be used as fingerprint data. The RSSI of an AP (access point) is usually between -110dBm and 0dBm.

[0040] For example, the WiFi fingerprint dataset is constructed based on the UJIIndoorLoc dataset, which provides a standard dataset for different WiFi positioning algorithms. It contains fingerprint data of three buildings, and it does not publish test samples. In one embodiment, the training samples are divided into training sets and validation sets in a ratio of 8:2, and all validation samples are used as test sets. The collection interval between the training set and the test set is 10 days. This paper marks this dataset as Dataset1, which is mainly used to verify the performance of the provided WiFi fingerprint positioning algorithm. The obtained WiFi fingerprint dataset contains the coordinates of the location reference point (RP) of the access point coverage area, and the coordinates of the RP are used as labels.

[0041] For example, the inertial navigation dataset uses the RoNIN dataset, which has a total collection time of over 42.7 hours. The data types include inertial sensor data collected by smartphones and real-world 3D motion data output by Tango. The subjects can carry their smartphones as they normally do in daily activities. For security and privacy reasons, RoNIN only releases approximately 50% of the data. This paper labels this dataset Dataset2 and primarily uses it to verify the performance of the provided deep inertial odometry.

[0042] In addition, a local dataset was constructed. For example, WiFi signals and IMU signals were collected in real-world scenarios on a campus. The entire collection process was divided into two phases. In the first phase, only WiFi signals were collected to train the proposed WiFi fingerprint localization network model. The RP position coordinates were provided by a laser rangefinder. A total of 190 RPs were selected, with the collection time at each RP being approximately 25-30 seconds and the sampling frequency set to 1Hz. In the second phase, WiFi and IMU signals were collected simultaneously. The gold standard data came from AREngine (similar to Tango). This dataset is labeled Dataset3 in this article and is primarily used to verify the localization performance of the provided model and the positioning performance of the fusion system.

[0043] Step S120: pre-process the WiFi fingerprint data set and train a fingerprint positioning model.

[0044] After constructing the fingerprint dataset and before inputting the fingerprint data into the fingerprint positioning model, the fingerprint data needs to be further processed so that the model can converge and achieve better positioning accuracy.

[0045] In one embodiment, the WiFi fingerprint dataset preprocessing process includes the following steps:

[0046] Step S121: adding noise to the fingerprint data.

[0047] WiFi signals are subject to random obstruction by the human body and vary over time. This means that the historical data that makes up the WiFi fingerprint library may differ significantly from the data collected during the online phase. Therefore, noise is first added to the WiFi fingerprint to improve the robustness of the model. Examples of noise types include masking noise and Gaussian noise. Masking noise randomly sets a portion of the fingerprint data to -110 dBm to simulate signal variations caused by random human obstruction, with a noise parameter set to 0.3. Gaussian noise, a type of noise that follows a normal distribution, is used to simulate signal variations caused by small fluctuations over time.

[0048] Step S122: normalize the fingerprint data.

[0049] Figure 2 shows the RSSI distribution of a Wi-Fi fingerprint from the lab's fingerprint database. It can be seen that 140 of the 151 APs have no signal strength (which can be forced to -110 dBm). Due to the large number of repeated constant values ​​in the data, the Wi-Fi fingerprint data suffers from data sparsity, which affects model performance.

[0050] In addition, as can be seen from Figure 2, the RSSI value is usually between -110dBm and 0dBm. Directly inputting it into the model will make it difficult for the model to converge. Therefore, it is necessary to normalize the fingerprint data first, which can be expressed as:

[0051] Among them, RSSI ij represents the signal strength value of the i-th AP received at the j-th RP, and Min_RSSI represents the minimum RSSI value in the fingerprint library. After normalizing the data, the distribution of WiFi fingerprint data becomes unbiased and low-variance, which can accelerate model training.

[0052] Step S123: Enhance the fingerprint data based on the AP sorting method.

[0053] To address the data sparsity issue in WiFi fingerprints, we propose a data augmentation method based on AP sorting to achieve globally sparse but locally dense data. This data augmentation method will be demonstrated using Dataset 1 as a basis. Overall, the data augmentation process involves extracting the rough location of APs and reordering the WiFi fingerprint data along a predetermined direction based on this rough location.

[0054] Specifically, according to the signal propagation theory, the signal coverage of each AP is limited, and the fingerprint database has the coordinate information of all RPs; let the i-th AP be AP i , find the AP from the fingerprint database i At which RPs the detection is performed, recorded as:

[0055] in, It means that the kth RP of the i-th AP can be detected and then obtained from the fingerprint database. The corresponding RP coordinate list is recorded as:

[0056] in, It represents the coordinates of the kth RP that can detect the i-th AP, and finally the AP is calculated using formula (4) i The rough position coordinate information is expressed as:

[0057] Figure 3 shows the RP ranges within which four APs can be detected. As can be seen, while some APs have wide coverage, others have more limited coverage, the coordinates of their coverage areas can all be used to describe the approximate location of the APs. Figure 3 shows the approximate location coordinates of all APs calculated using the aforementioned algorithm, which are relatively evenly distributed across the data collection site.

[0058] After calculating the rough coordinates of the AP, the APs are sorted, for example, along the northwest-southeast direction (from the upper left corner to the lower right corner in Figure 3), so that the original WiFi fingerprint becomes data with AP spatial information characteristics. This data enhancement method roughly estimates the AP position by obtaining the reference point (RP) position information covered by the WiFi signal emitted by the access point (AP) in the fingerprint library, and uses this to change the arrangement of the fingerprint data, thereby effectively improving the WiFi positioning accuracy in multi-building scenarios and reducing the complexity of data preprocessing. Furthermore, in order to use two-dimensional convolution to process fingerprint data, in one embodiment, the dimension of the WiFi fingerprint data is changed from 1×m to Figure 4 shows the process of enhancing the original WiFi fingerprint data. As can be seen from Figure 4, two originally similar WiFi fingerprints have higher discrimination after processing, and the originally globally sparse data has become locally dense.

[0059] Step S124 , normalizing the coordinates of the RP used as the label.

[0060] Preferably, the coordinates of the RP as a label also need to be normalized, expressed as:

[0061] Among them, Pos j Indicates the position coordinates of the jth RP (X j , Y j), Max_pos and Min_pos represent the maximum and minimum position coordinates of all RPs, respectively. In addition, for Dataset 1, as can be seen from Figure 3, the entire positioning area is tilted, and the tilt angle can be eliminated by the coordinate rotation method.

[0062] For the fingerprint positioning model, various types of neural network models can be used. For example, considering that the historical data constituting the WiFi fingerprint library may be significantly different from the data collected in the online stage, in one embodiment, the fingerprint positioning model includes a denoising autoencoder and a CNN regression network model. Its overall structure is shown in Figure 5, where Figure 5(a) is a denoising autoencoder, which generally includes an encoder and a decoder. The denoising autoencoder is used to extract robust features from noisy WiFi fingerprints. Its input is 3×m image data consisting of WiFi fingerprints. The encoder includes two convolutional layers and one pooling layer. The encoder compresses the data dimension into The decoder consists of an upsampling layer and a convolutional layer. The decoder reconstructs the features extracted by the encoder into 3×m Wi-Fi fingerprint data. Figure 5(b) shows the structure of the regression network model, which is used to regress Wi-Fi location coordinates. Its input is the feature map extracted by the encoder. The regression network model consists of three convolutional layers, two pooling layers, and a fully connected module. The three convolutional layers are used to extract the spatial features of the Wi-Fi fingerprint image, and the fully connected module is used to regress the location results. The regression network model reflects the nonlinear relationship between Wi-Fi fingerprint data and location coordinates.

[0063] In one embodiment, the parameter settings of the denoising autoencoder and the regression network model (or regression module) are shown in Table 1.

[0064] Table 1: Model detailed parameters

[0065] In Table 1, MLP is a fully connected module composed of a multi-layer perceptron, used to regress the localization results. For example, the fully connected module contains six fully connected layers, with the number of hidden neurons in each layer being 128, 256, 128, 64, 32, and 2, respectively. Except for the last layer, all other fully connected layers are followed by a batch normalization layer and use the tanh activation function to prevent overfitting and introduce nonlinearity into the model. The output dimensions of Table 1 are based on Dataset 2. The output dimensions of Dataset 1 are larger because it contains more APs.

[0066] In summary, to address the problem that WiFi data fluctuates over time and is easily obstructed by human bodies, a regression network model based on a denoising autoencoder and a convolutional neural network is proposed. The denoising autoencoder extracts robust features from noisy WiFi fingerprints through unsupervised training, making the WiFi fingerprint positioning model more noise-resistant.

[0067] Step S130 , preprocessing the inertial navigation data set and training the trajectory tracking model.

[0068] As shown in Figure 6, the preprocessing process of the inertial navigation dataset mainly includes linear interpolation, coordinate transformation, data synchronization, and data segmentation of the inertial sensor data and gold standard data. The RoNIN public dataset has already undergone interpolation and synchronization operations and does not need to be processed again.

[0069] Data interpolation: During the data collection process, the sensor may lose connection briefly, resulting in data loss or abnormal values. In addition, the acquisition frequencies of inertial sensor data and AREngine are 50 Hz and 30 Hz, respectively, while the model input data frequency is 200 Hz, so both types of data need to be interpolated.

[0070] Coordinate transformation: The coordinates of the inertial sensor data need to be rotated multiple times. Assume that the inertial sensor data after data interpolation is [a, ω], where a represents the acceleration signal and ω represents the angular velocity signal. In addition, suppose that the first quaternion output by the rotation vector sensor is q, then [qaq * ,qωq * ] Adjust the initial attitude of the sensor to the geographic coordinate system, and then use the angular velocity data to calculate the quaternion Δq of each sampling point, and then use Δq to further rotate the inertial sensor data into {Δq[qaq * ,qωq * ]Δq *}, which roughly converts the data of each sampling point to the geographic coordinate system. It is worth mentioning that this method of rotating coordinates does not completely rely on the rotation vector sensor. Magnetic materials in indoor environments greatly affect the output value of the rotation vector sensor.

[0071] The data segmentation method is shown in Figure 7, which shows the preprocessed inertial sensor data (IMU) and the gold standard data (ground truth). For the real trajectory, the window length is set to 200 (1 second), t i At point B, t i+200 The coordinate difference between the two points is the average speed on the two-dimensional plane. and These two numbers are the label values. i to t i+200 The inertial sensor data segment between t i-α to t iThe inertial sensor data between α and β is used as the input of the model, where α is set to 200 and β is set to 0. Each segmented sample includes a 400×6 inertial sensor data sequence and a 2×1 average velocity label vector. The inertial sensor data sequence is input into the network model, and the average velocity label vector is used as the reference label of the final average velocity.

[0072] The input of the trajectory tracking model is the preprocessed inertial sensor sequence, and the output is the velocity vector, which is given by formula (6), where the dimensions of a and ω are both n×3, and n is equal to the number of sampling points between point A and point D; data out is an estimate of the model output, Calculated from ground truth trajectory data.

[0073] In one embodiment, the overall structure of the constructed trajectory tracking model (or deep inertial odometry network) is shown in Figure 8. The model contains two branches, each of which contains a one-dimensional convolutional layer, a CBAM (Convolutional Block Attention Module) attention module, and a bidirectional LSTM (BiLSTM). The model input data is preprocessed three-axis acceleration and three-axis angular velocity data with a dimension of b×6×400, where b represents the batch_size size. The data is input into the one-dimensional convolutional layer of the two branches. The two convolution kernels have different sizes, set to 3 and 7 respectively, which can extract local dependencies between variables and short-term features in the time dimension. The number of convolution kernels is set to 64, and the step size is set to 2. The convolution layer is followed by a batch normalization layer (BN), a ReLU layer, and a maximum pooling layer (MaxPool), which can be used to speed up the convergence of the model, introduce nonlinear operations to the model, extract important features, and reduce the dimensionality of the data. The step size of the maximum pooling layer is 2, and the data dimension becomes b×6. 4×100; Then, the CBAM attention module is used on each branch to modify the input features. The CBAM attention module combines the channel attention and spatial attention mechanisms to enhance the useful information in the spatial and channel dimensions and suppress irrelevant features. It multiplies the two feature map information generated by the original input features to suppress irrelevant features; Then, a bidirectional LSTM is used on each branch to extract long-term features in the time dimension. The number of layers is 2, and the number of hidden layers in each layer is set to 64. To prevent overfitting, the dropout is set to 0.4. In addition, the input data dimension needs to be adjusted to b×100×64 first, and the output dimension is b×100×128; Then, the temporal attention module (Temporal The Attention Module (TAM) is used to extract useful features in the time dimension from the many-to-many output of the bidirectional LSTM. The specific approach is to multiply the feature vector of the last time step by the transpose of the entire feature map to obtain a time step value of b×100×1, and then use the softmax function to calculate the weight parameter of each time step. The weight is multiplied by the feature map and accumulated along the time dimension, with an output dimension of b×128; the two branch outputs are spliced ​​together, and the data dimension becomes b×256; finally, three fully connected layers are used as regression calculation modules, with the number of hidden neurons being 512, 512, and 2 respectively. Except for the last layer, each fully connected layer uses a RelU layer and sets the dropout value to 0.5.

[0074] In summary, to address the issues of false and missed gait detections often encountered in traditional Pedestrian Dead Reckoning (PDR) algorithms, a deep inertial odometry network model was proposed. This model uses fixed-window inertial sensor data as input and eliminates the gait detection step required by PDR. The model consists of two branches, each consisting of a one-dimensional convolutional layer and a bidirectional long short-term memory (BiLSTM) network to extract short-term and long-term temporal dependencies within the inertial sensor data sequence. Furthermore, the convolution kernels in the two branches vary in size, enabling the extraction of features at different scales. This model architecture minimizes parameters while ensuring accuracy, making it more suitable for deployment on memory-constrained embedded devices.

[0075] It should be noted that the present invention does not limit the specific structure of the model. For example, different numbers of convolutional layers, pooling layers, fully connected layers, etc., or different activation functions or pooling methods can be used. In addition, the input dimension of the model can also be set according to actual needs.

[0076] In step S140, for the target device, the global coordinate observation value is obtained by using the WiFi fingerprint positioning model, and the relative displacement observation value is obtained by using the trajectory tracking model, and then the positioning result is obtained by fusing them through the extended Kalman filter.

[0077] Figure 9 is a schematic diagram of the fusion positioning process. For the sake of clarity, it may be called a fusion positioning system and described in the form of functional modules, where the WiFi fingerprint positioning module is used to represent fingerprint positioning, the inertial odometer module (or inertial positioning module) is used to represent positioning based on inertial sensor data, and the fusion module is used to represent the output results of the fusion of the WiFi fingerprint positioning module and the inertial odometer module. The fusion module obtains the final positioning result.

[0078] In one embodiment, an extended Kalman filter (EKF) fusion algorithm is used in the fusion module to fuse the global coordinate observation value output by the WiFi fingerprint positioning module and the relative displacement observation value output by the inertial odometer module.

[0079] Specifically, the inertial odometer module based on deep learning can output the average speed in two perpendicular directions on a two-dimensional horizontal plane per unit time. By accumulating the output of this module, the complete walking trajectory can be obtained, but the initial heading of this trajectory is unknown. For example, the first data output by the built-in rotation vector sensor of the smartphone can be used to simply correct the initial heading, but the rotation vector sensor may be affected by magnetic materials or have design defects itself, resulting in a certain angle ψ between the heading and the true heading. The system state vector selected by the present invention includes the pedestrian's position coordinates x, y and the angle ψ: X = [xy ψ] T(7)

[0080] The observation vector is set as: Z=[x wifi y wifi s imu t imu ] T (8)

[0081] Among them, x wifi and y wifi is the coordinate output by the WiFi fingerprint positioning module, and the relative displacement on the two-dimensional plane output by the depth inertial odometer module is accumulated to obtain s imu and t imu , then the system state equation is:

[0082] Among them, W k-1 is the process noise. Then the Jacobian matrix can be expressed as:

[0083] The observation matrix is ​​expressed as:

[0084] So the linearized state equation and observation equation are:

[0085] The process noise of the system W k-1 The corresponding covariance matrix Q and observation noise V k The corresponding covariance matrix can be expressed by the following formula:

[0086] in, and is the variance of WiFi positioning, and is the variance of the relative displacement of the inertial positioning module. The initial value of the state covariance matrix P is not important. After iteration, it will approach the true state. For example, set the state covariance matrix to:

[0087] In summary, considering that IMU-based dead reckoning methods can only predict relatively accurate trajectories, achieving positioning requires an initial heading calibration using a magnetometer. However, magnetometers are susceptible to interference from indoor magnetic materials, leading to calibration failures. To address this issue, the present invention proposes a fusion positioning system based on an extended Kalman filter that is robust to initial heading errors. This system uses the predictions of a fingerprint positioning model and a trajectory tracking model as observations, with the system state set to the pedestrian position and the angular difference between the initial heading of the inertial odometer and the actual heading. This fusion positioning system has been proven to provide accurate positioning results within a local dataset.

[0088] To further verify the performance of the present invention, the Cumulative Probability Distribution Function (CDF) and average positioning error were used for evaluation. The CDF describes the probability that a random variable will take on a value less than or equal to a given value. The x-axis of the CDF curve represents the positioning error, and the y-axis represents the cumulative probability value for each value on the x-axis, that is, the probability that a value is less than or equal to the value on the x-axis. The faster the CDF curve rises, the smaller the positioning error for most samples, and vice versa.

[0089] The deep inertial odometry network model is trained on the RoNIN public dataset; the indoor positioning regression network model of the WiFi fingerprint positioning module is trained on the self-collected dataset; the fusion positioning module uses the self-collected dataset as test data to measure the performance of the fusion positioning system. Figure 10 shows the cumulative distribution function curves of the positioning errors of the WiFi fingerprint positioning module, the inertial odometry module and the EKF fusion module (corresponding to the present invention). As can be seen from Figure 10, the probability of the positioning error of the WiFi fingerprint and the inertial odometry within 2 meters is less than 25%, the probability within 4 meters is less than 60%, and the probability of being greater than 5 meters is more than 30%; the probability of the EKF fusion positioning error within 2 meters is about 40%, the probability within 4 meters is about 85%, and the probability of being greater than 5 meters is within 6%, indicating that the present invention has a higher positioning accuracy, which is much greater than the positioning error of the other two single technologies, and has achieved the best positioning performance.

[0090] Table 2 shows the average, minimum, and maximum positioning errors of the three positioning technologies. Since the initial coordinates of the inertial odometry module and the EKF fusion module are both set to the pedestrian's true coordinates, the minimum positioning error is not calculated. The average positioning error of the EKF fusion positioning method is 2.53 meters, which is approximately 34% and 42% higher than the positioning using the WiFi fingerprint and inertial odometry technologies alone, respectively. Furthermore, the fusion method significantly reduces the maximum positioning error. The average positioning error of the WiFi fingerprint is relatively low, but the minimum and maximum positioning errors of 0.09 and 15.01 meters indicate the instability of WiFi positioning, which is prone to position jumps and discontinuous positioning results. The maximum error of the inertial odometry module is 14.98 meters. This is due to the accumulated error in the trajectory calculation of the inertial odometry and the large deviation in the initial heading calculation using the rotation vector sensor. Figure 11 shows the positioning results calculated by different modules for a data sequence.

[0091] Table 2: Comparison of positioning errors of different algorithms

[0092] As shown in Figure 11, the WiFi fingerprint positioning results are relatively scattered and discontinuous, but are generally distributed around the true trajectory. The trajectory calculated by the inertial odometry module is actually very similar in shape to the true trajectory, demonstrating the high performance of the deep inertial odometry network model. However, due to the large error between the initial heading output by the rotation vector sensor and the true heading, the entire trajectory deviates significantly from the true trajectory, with an average positioning error of approximately 8.8 meters. As shown in Figure 11, the EKF fusion positioning results have a high degree of overlap with the true trajectory, with an average positioning error of approximately 1.6 meters, an 81.8% improvement compared to the inertial odometry. This demonstrates that the EKF fusion system can effectively reduce the impact of initial heading error on positioning performance.

[0093] To further demonstrate the robustness of the EKF fusion system to the initial heading error of the inertial odometry module, the present invention adds different offset angles to the initial heading of the inertial odometry module, ranging from -180 degrees to 180 degrees, with a step size of 10 degrees and a positive direction counterclockwise. Figure 12 shows the experimental results. As shown in Figure 12, the positioning error of the inertial odometry module first decreases and then increases with the increase of the offset angle. The best positioning effect is achieved at an offset angle of approximately -20 degrees, with significant variation in positioning error. The WiFi fingerprint positioning module is unaffected by the offset angle and always maintains a stable value. The EKF fusion module achieves the best positioning effect between offset angles of -140 degrees and 140 degrees, with an overall smaller positioning error than WiFi fingerprint positioning. When the offset angle is greater than 140 degrees or less than -140 degrees, the EKF fusion positioning performance is lower than that of WiFi fingerprint positioning. This is mainly due to the time required for the EKF fusion system to adjust, and the performance loss is mainly in the early stages of positioning. Figure 13 shows a comparison of the trajectories of a data sequence with clockwise heading angle offsets of 30, 60, 120, and 180 degrees. It can be seen that the four trajectories calculated by the EKF fusion system differ only slightly at the beginning of the trajectories, and the trajectories in the later stages are basically the same.

[0094] In summary, the present invention effectively integrates the inertial sensor information integrated in the portable smart device and the received WiFi fingerprint information, improving the accuracy and robustness of indoor positioning. Compared with the existing technology, it has the following main advantages:

[0095] 1) For indoor positioning based on WiFi fingerprints, the coarse location information of access points is extracted to rearrange the WiFi fingerprint data to reduce the impact of the high sparsity of WiFi fingerprints. Furthermore, to address the problem that WiFi data fluctuates over time and is easily obscured by human bodies, a regression network model based on a denoising autoencoder and a convolutional neural network is proposed. This model extracts noise-independent features from noisy WiFi fingerprint data, improving its robustness to noisy data.

[0096] 2) The deep learning-based deep inertial odometry algorithm uses inertial sensor data with a fixed window length as input, avoiding the effects of false and missed gait detections associated with traditional PDR. Furthermore, the model uses two branches with different convolution kernel sizes to extract features at different scales. Each branch contains a convolutional layer and a bidirectional long short-term memory network to extract spatial and temporal information.

[0097] 3) A WiFi fingerprint and inertial odometry fusion positioning method based on extended Kalman filtering is proposed. By setting the pedestrian position based on WiFi fingerprint and the angle difference between the initial heading based on inertial odometry and the true heading as the fusion system state, the influence of the initial heading error of the inertial odometry on the fusion system is effectively eliminated.

[0098] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0099] Computer-readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device.Computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof.More specific examples (non-exhaustive list) of computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof.Computer-readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0100] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0101] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0102] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0103] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0104] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0105] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. An indoor positioning and navigation method based on WiFi fingerprint and inertial sensor information fusion, comprising the following steps: Collect the WiFi fingerprint data of the target device and input it into the trained fingerprint positioning model to obtain the global coordinate observation value; Using an inertial sensor to collect acceleration signals and angular velocity signals of the target device, and inputting them into a trained trajectory tracking model to obtain a relative displacement observation value, wherein the relative displacement observation value reflects the movement trajectory of the target device; The global coordinate observation value and the relative displacement observation value are fused using an extended Kalman filter to obtain fused position information as a positioning result of the target device.

2. The method according to claim 1, characterized in that The fingerprint positioning model includes a denoising encoder and a regression network model, wherein the denoising encoder is provided with an encoder and a decoder, the encoder is used to extract a feature map from WiFi fingerprint data, and the decoder reconstructs the feature map into fingerprint data of the original size; the regression network model uses the feature map extracted by the encoder as input and obtains the fingerprint positioning result through regression calculation.

3. The method according to claim 2, characterized in that The encoder includes two convolutional layers and one pooling layer, and the decoder includes one upsampling layer and one convolutional layer; the regression network model includes three convolutional layers, two pooling layers and a fully connected module, wherein the three convolutional layers are used to extract the spatial features of the WiFi fingerprint data, and the fully connected module is used to perform regression calculation to obtain the fingerprint positioning result, and the fully connected module includes multiple fully connected layers, and except for the last fully connected layer, the other fully connected layers are connected to a batch normalization layer and use a tanh activation function.

4. The method according to claim 1, characterized in that: The trajectory tracking model includes two branches, each of which is provided with a one-dimensional convolution layer, an attention module CBAM of a convolution block, and a bidirectional LSTM. The input data of the trajectory tracking model are three-axis acceleration data and three-axis angular velocity data collected by an inertial sensor. The data is input into the one-dimensional convolution layers of the two branches to extract local dependencies between variables and short-term features in the time dimension. The convolution layer is followed by a batch normalization layer, a ReLU layer, and a maximum pooling layer. On each branch, the attention module CBAM of the convolution block is used to correct the input features. The attention module CBAM of the convolution block combines the channel attention mechanism and the spatial attention mechanism to enhance useful information in the spatial dimension and the channel dimension. And on each branch, the bidirectional LSTM is used to extract long-term features in the time dimension. Then, the time attention module is used to extract useful features in the time dimension from the many-to-many output of the bidirectional LSTM. Then, a multi-layer fully connected layer is used to perform regression calculation to obtain the calculation result of the relative displacement.

5. The method according to claim 1, characterized in that The fingerprint positioning model is trained according to the following steps: Acquire a WiFi fingerprint data set, where the WiFi fingerprint data set includes the location coordinates of a reference point in an access point coverage area, and uses the location coordinates of the reference point as a label; Add noise to the WiFi fingerprint data. The added noise types include masking noise and Gaussian noise. Masking noise is used to randomly set a part of the data to a fixed value to simulate the signal change caused by random occlusion of the target device. Gaussian noise refers to noise that follows a normal distribution. Normalize the WiFi fingerprint data after adding noise; Calculating a rough position of the access point, and reordering the access points along a predetermined direction based on the rough position; The position coordinates of the reference points used as tags are normalized to obtain an enhanced WiFi fingerprint dataset; The fingerprint positioning model is trained using the enhanced WiFi fingerprint dataset.

6. The method according to claim 1, characterized in that The trajectory tracking model is trained according to the following steps: Acquire an inertial navigation data set, wherein the inertial navigation data set includes inertial sensor data and a corresponding 3D motion trajectory, wherein the inertial sensor data includes an acceleration signal and an angular velocity signal; Performing linear interpolation, coordinate conversion and data segmentation on the inertial sensor data to obtain an inertial sensor data sequence; Performing linear interpolation and data segmentation on the 3D motion trajectory to obtain an average velocity label vector; The inertial sensor data sequence is input into the trajectory tracking model, and the average speed label vector is used as a reference label of the average speed for training.

7. The method according to claim 5, characterized in that The rough location of the access point is calculated according to the following steps: Assume that the i-th access point AP in the WiFi fingerprint dataset is AP i , and find out the AP from the WiFi fingerprint dataset i The reference position RP covered by the signal is expressed as: in, Indicates AP i The kth RP covered by the signal; Obtained from the WiFi fingerprint dataset The corresponding RP position coordinate list is expressed as: in, express The location coordinates of AP is calculated according to the following formula i The rough position coordinates of: Among them, Pos i Represents the rough location of the i-th access point AP.

8. The method according to claim 6, characterized in that The fused position information is obtained according to the following steps: Select a state vector that describes the state of the target device, including position coordinates x, y and angle ψ, expressed as: X=[xy ψ] T Set the observation vector, expressed as: Z=[x wifi y wifi s imu t imu ] T Among them, x wifi and wifi is the coordinate obtained using the fingerprint positioning model, s imu and t imu It is obtained by accumulating the relative displacement on the two-dimensional plane output by the trajectory tracking model. The state equation at time k is: Among them, W k-1 is the process noise at time k-1, then the Jacobian matrix is ​​expressed as: The observation matrix is ​​expressed as: The linearized state equation and observation equation are expressed as: Among them, V k is the observation noise, W k-1 The corresponding covariance matrix Q and observation noise V k The corresponding covariance matrix R is expressed as: in, and is the fingerprint localization variance, and is the variance of the relative displacement, and the state covariance matrix P is set to:

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images