Self-correcting dead reckoning system and method based on activity recognition and fusion filtering

Abstract

The application discloses a kind of self-correcting dead reckoning methods based on activity identification and fusion filtering, which is based on dead reckoning system, specifically comprising the following steps: first, the pedestrian data is collected by the data collection module and the data is delivered to the data processing module, then the collected pedestrian data is processed by the data processing module and the processed pedestrian data is delivered to the activity identification module, then the activity identification model is used to identify the pedestrian action based on the data processed by the activity identification module, and the result is delivered to the position estimation module, then the position of the pedestrian is estimated based on the pedestrian action by the position estimation module, and the result is delivered to the position correction module, then the position of the pedestrian is corrected based on the pedestrian position result according to the magnetic intensity sequence by the position correction module, and the result is delivered to the data output module, finally, the corrected pedestrian position result is output by the data output module, the method can effectively improve the accuracy of dead reckoning.

Description

A self-correcting dead reckoning system and method based on activity recognition and fusion filtering Technical Field

[0001] This invention belongs to the field of dead reckoning technology, specifically relating to a self-correcting dead reckoning system and method based on activity recognition and fusion filtering. Background Technology

[0002] Satellite positioning systems (such as BeiDou and GPS) can achieve sub-meter level positioning in outdoor environments, but in indoor settings, they cannot achieve precise positioning and navigation due to phenomena such as refraction and diffraction during signal propagation. Therefore, researchers have proposed various indoor positioning and navigation methods, including those using Wi-Fi, Bluetooth, RFID, UWB, ZigBee, computer vision, infrared, and visible light, achieving good positioning accuracy. However, most of these technologies require hardware deployment and are subject to high requirements in indoor environments. For example, Wi-Fi or Bluetooth-based positioning requires the deployment of multiple signal transmitting devices (Access Points), and its accuracy depends on the density of the collected fingerprint map; computer vision-based positioning requires high stability in the indoor environment and a large number of images for location matching; signal ranging-based positioning methods, such as RFID and UWB, are easily affected by the environment and cannot guarantee positioning accuracy in complex indoor environments. More importantly, their high equipment costs limit their widespread application.

[0003] With the development of wireless communication technology and the miniaturization of sensors, smart devices embedded with various miniature sensors are becoming increasingly widely used. As mobile smart devices become more prevalent, people are placing higher demands on their positioning and navigation capabilities. Currently, smart devices embedded with various sensors are required to perform accurate pedestrian dead reckoning, including step count detection, step length calculation, and dead angle calculation. However, the data collected by these sensors contains a significant amount of environmental noise. This noise-laden data leads to a decrease in the accuracy of pedestrian dead reckoning, and the rate of accuracy decline accelerates with increasing system runtime.

[0004] In summary, indoor positioning technology utilizing MEMS sensor components embedded in commercial smart devices such as smartphones for dead reckoning has garnered widespread attention. This technology eliminates the need for pre-deployed auxiliary equipment. Similarly, the miniaturization of sensor devices allows for the embedding of more and more sensing elements within smart devices, continuously improving their sensing capabilities. However, MEMS sensors inevitably incorporate a significant amount of noisy data during collection. Consequently, most current pedestrian-based dead reckoning methods suffer from large error accumulation, leading to substantial deviations in track estimation. Furthermore, the embedded sensors in smart devices are not professionally calibrated, resulting in measurement results that deviate from actual data. The varying models of built-in sensors across different devices also contribute to lower-than-expected positioning accuracy in practical use. Incorrect data measurement results cause the cumulative error of the dead reckoning algorithm to increase with walking distance and time. While Wi-Fi signal-assisted positioning requires pre-deployed infrastructure, computer vision-based position correction has specific requirements regarding the placement of smart devices and the indoor environment, hindering large-scale deployment. Bang Wu et al. used multiple neural network models to recognize different actions and combined them with dead reckoning algorithms, achieving an average positioning error of only 1.79m in multi-story buildings. However, this method involves multiple neural network models, making it highly complex and unsuitable for operation on smart devices. Furthermore, most current dead reckoning algorithms require pedestrians to hold the smart device horizontally and maintain a constant posture during use, which limits the practicality of this method. Summary of the Invention

[0005] Purpose of the invention: In order to overcome the shortcomings of the existing technology, the present invention provides a self-correcting dead reckoning system and method based on activity recognition and fusion filtering, which can improve the accuracy of track reckoning.

[0006] Technical solution: To achieve the above objectives, the present invention provides a self-correcting dead reckoning system based on activity recognition and fusion filtering. The system specifically includes a data collection module, a data processing module, an activity recognition module, a position prediction module, a position correction module, and a data output module.

[0007] The data collection module includes several MEMS sensors, which collect pedestrian data.

[0008] The data processing module is used to process the collected data.

[0009] The activity recognition module uses an activity recognition model to identify pedestrian actions based on the processed data.

[0010] The position estimation module is used to estimate the pedestrian's position. Specifically, the pedestrian position estimation includes estimating the stride length and calculating the dead angle based on the pedestrian action recognition results.

[0011] The position correction module is used to correct the pedestrian's position based on the magnetic field strength sequence;

[0012] First, pedestrian data is collected by the data collection module and sent to the data processing module. Then, the data processing module processes the collected pedestrian data and sends the processed pedestrian data to the activity recognition module. Next, the activity recognition module uses an activity recognition model to identify pedestrian actions based on the processed data and sends the result to the location prediction module. Then, the location prediction module predicts the pedestrian position based on the pedestrian actions and sends the result to the location correction module. Then, the location correction module corrects the pedestrian position based on the pedestrian position result according to the magnetic intensity sequence and sends the result to the data output module. Finally, the data output module outputs the corrected pedestrian position result.

[0013] Furthermore, the data processing module includes a data denoising module and a data coordinate system transformation module, wherein the data denoising module is used to denoise the collected data; and the data coordinate system transformation module is used to transform the coordinate system of the data.

[0014] Furthermore, based on the processed data, the activity recognition module uses an activity recognition model to identify pedestrian actions. This activity recognition model combines the Attention Mechanism with a CNN-LSTM neural network model. The model uses the cross-entropy function as the multi-class loss function, defined as follows:

[0015] loss = -∑ i (y i *logy p +(1-y i )*log(1-y p )) Formula (13)

[0016] In the above formula (13), i is the current predicted activity instance number, and y i y represents the actual action in the current sequence. p The result of the classifier's prediction action;

[0017] By porting the above activity recognition model to mobile smart devices, pedestrian actions can be recognized in real time on mobile smart devices.

[0018] Furthermore, the position prediction module is used to predict the pedestrian's position, which specifically includes estimating the stride length and calculating the dead angle based on the pedestrian action recognition results;

[0019] Calculating a pedestrian's position angle specifically includes the following steps:

[0020] First, the acceleration output angle is calculated based on the collected acceleration data, the magnetic field strength output angle is calculated based on the collected magnetic field strength data, and the dead angle output by the gyroscope is calculated based on the data collected by the gyroscope.

[0021] Then, the acceleration output angle and the magnetic field strength output angle are both input into the low-pass filter LPF. The acceleration output angle and the magnetic field strength output angle are processed by the low-pass filter to obtain the filtered acceleration output angle and the filtered magnetic field strength output angle. At the same time, the dead angle output by the gyroscope is input into the high-pass filter HPF. The low-frequency components in the gyroscope data are removed by the high-pass filter to obtain the filtered gyroscope output dead angle.

[0022] Finally, the acceleration output angle filtered by the low-pass filter, the magnetic field strength output angle filtered by the low-pass filter, and the gyroscope output dead angle filtered by the high-pass filter are all input into the fusion filter. The fused pedestrian dead angle is calculated by the fusion filter. The formula for the fusion filter is as follows:

[0023] FusedOri=f*Gyr+(1-f)*AccMag Formula (14)

[0024] In the above formula (14), FusedOri is the pedestrian dead angle output by the fusion filter, f is the parameter of the fusion filter, Gyr is the dead angle output by the filtered gyroscope, and AccMag is the dead angle obtained by fusing the filtered acceleration output angle and the filtered magnetic field strength output angle.

[0025] The pedestrian dead angle FusedOri output by the fusion filter obtained through the above steps is the final pedestrian dead angle that needs to be solved.

[0026] Furthermore, the estimation step size specifically includes the following steps:

[0027] First, the number of steps taken by pedestrians is detected using a peak detection method based on dynamic thresholds;

[0028] Then, the pedestrian's walking stride length is calculated using formula (15):

[0029]

[0030] In the above formula (15), S represents the step size, a max and a minThese are the maximum and minimum acceleration values ​​on the Z-axis of the navigation coordinate system, respectively. The step size parameter L is a constant related to the step frequency; generally, the faster the walking speed, the larger the value of L.

[0031] Finally, the pedestrian's current position coordinates are estimated in real time based on the pedestrian's heading angle and step size output by the obtained fusion filter.

[0032] Furthermore, the position correction module is used to correct the pedestrian's position based on the magnetic intensity sequence, specifically including the following steps:

[0033] First, define the pedestrian walking pattern;

[0034] Then, based on the dynamic time planning method of discrete time difference, the magnetic field intensity sequence under different walking states is defined. Specifically, the original magnetic field intensity sequence is converted into a difference sequence of adjacent values ​​through formula (17).

[0035]

[0036] In the above formula (17), m′(i) is the sequence of differences between adjacent values, m i Let i be the i-th value in the magnetic field strength sequence, where i is a natural number;

[0037] Next, based on the difference sequence m′(i) of adjacent values ​​obtained above, the dynamic time programming method is used to determine the similarity of the m′(i) interpolation sequence;

[0038] Finally, the DTW-DTD distance is calculated based on the magnetic field strength sequence collected by the user and the magnetic field strength sequence on the reference path, and the current position coordinates of the pedestrian are replaced with the matching magnetic field data position.

[0039] Furthermore, the self-correcting dead reckoning system based on activity recognition and fusion filtering as described in any one of claims 1 to 6 specifically includes the following steps:

[0040] 1) First, pedestrian data is collected using accelerometers, gyroscopes, and magnetometers;

[0041] 2) Then, the collected data is processed, including data noise reduction and coordinate axis transformation, to improve the effectiveness of the sampled data;

[0042] 3) Next, an activity recognition method based on Attention Mechanism and CNN-LSTM is proposed to accurately identify pedestrian actions;

[0043] 4) Subsequently, data from different types of sensors are fused and filtered to ensure the accuracy of the dead angle calculation during long-term walking. At the same time, the walking distance and direction are estimated based on the actual activities of pedestrians, and the error between the estimated step length and the actual step length is controlled within a reasonable range.

[0044] 5) Finally, a dynamic time planning algorithm based on discrete time difference is proposed to match and compare data collected under different walking modes on the same path.

[0045] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0046] 1. The system of the present invention includes a data processing module, which includes a data noise reduction module. Since the sensor sampling data is subject to noise and cannot be directly applied to activity identification and dead reckoning, the system uses a data noise reduction module to reduce the noise of the collected data so that the original data meets the requirements of dead reckoning, thereby improving the effectiveness of the sampling data.

[0047] 2. The system of the present invention includes a data processing module, which includes a data coordinate system transformation module. Due to problems such as the inconsistency between the device coordinate system and the navigation coordinate system, the sensor sampling data cannot be directly applied to activity recognition and dead reckoning. Therefore, the system uses a data coordinate system transformation module to transform the collected data to make the original data meet the requirements of dead reckoning, thereby improving the effectiveness of the sampling data.

[0048] 3. The activity recognition module in the system of the present invention accurately identifies pedestrian actions based on the processed data using an activity recognition model. This activity recognition model is an activity recognition method based on Attention Mechanism and CNN-LSTM, which has better accuracy than typical methods that use neural networks for activity recognition, thereby improving the accuracy of pedestrian action recognition and fully ensuring the accuracy of dead reckoning.

[0049] 4. The system of the present invention can realize a dead reckoning navigation system. This system is easy to integrate into smart devices, requires no additional infrastructure deployment, is easy to use, and has a wider range of applications.

[0050] 5. The system of this invention has been implemented in large-scale scenarios in both planar and three-dimensional environments. It has been compared with two mainstream dead reckoning methods: one using gyroscope angular velocity integration and the other using acceleration and magnetic field fusion. The comparison results show that the dead reckoning accuracy of this system has been improved by 88.29% and 73.12%, respectively.

[0051] 6. The method of this invention designs a fusion filtering method for multi-sensor data, which reduces the impact of long-term operation and changes in action on dead angle estimation.

[0052] 7. The system of the present invention includes a position correction module, which improves the accuracy of trajectory estimation by calibrating the pedestrian trajectory.

[0053] 8. The method of this invention designs a dynamic time planning algorithm based on discrete time difference to ensure effective dead reckoning. Attached Figure Description

[0054] Figure 1 is a schematic diagram of a self-correcting dead reckoning system module based on activity recognition and fusion filtering.

[0055] Figure 2 shows a schematic diagram of the equipment coordinate system.

[0056] Figure 3 shows a schematic diagram of the navigation coordinate system.

[0057] Figure 4 is a flowchart of the fusion filter structure.

[0058] Figure 5 shows the step detection results of the peak detection method based on dynamic threshold. Detailed Implementation

[0059] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0060] Example 1:

[0061] The self-correcting dead reckoning system based on activity recognition and fusion filtering in this embodiment, as shown in Figure 1, specifically includes a data collection module, a data processing module, an activity recognition module, a position estimation module, a position correction module, and a data output module. This system can achieve high-precision dead reckoning in large-scale scenarios and perform real-time navigation on different types of smart devices. The system uses an activity recognition model ported to the smart device to identify pedestrian activities and estimates the pedestrian walking distance based on the recognition results. Then, it uses fusion filtering to calculate the dead direction, combines the estimated walking distance to calculate the pedestrian position, and finally uses a magnetic field strength sequence matching method to correct the pedestrian position and displays the corrected estimated position coordinates on the system user interface.

[0062] The data collection module includes several MEMS sensors that collect pedestrian data. These MEMS sensors include one or more of gyroscopes, accelerometers, and magnetometers. A gyroscope is a device for measuring the angular motion of an object. By measuring the offset of the gyroscope's dual-axis base point under different motion states, the horizontal, vertical, pitch, acceleration, and heading positions of an object can be determined. The gyroscope is used to provide the current heading angle of the person. The accelerometer is used to measure the linear acceleration of the moving object and is used for step counting, pedestrian activity recognition, and step length estimation. The magnetometer is used to determine the magnitude and direction of the Earth's magnetic field, correcting the pedestrian's current position, reducing the cumulative error of the positioning system, and limiting the overall performance degradation of the system.

[0063] The data processing module is used to process the collected data. The data processing module includes a data denoising module and a data coordinate system transformation module. The data denoising module is used to denoise the collected data; the data coordinate system transformation module is used to transform the coordinate system of the denoised data.

[0064] The aforementioned activity recognition module uses an activity recognition model to identify pedestrian actions based on the processed data.

[0065] The aforementioned position estimation module is used to estimate the position of pedestrians. Specifically, the estimation of pedestrian position includes estimating the step length and calculating the dead angle based on the pedestrian action recognition results.

[0066] The aforementioned position correction module is used to correct pedestrian positions based on magnetic intensity sequences, thereby reducing the cumulative error of the positioning system and limiting the overall performance degradation of the system.

[0067] Dead reckoning is performed using the self-correcting dead reckoning system based on activity recognition and fusion filtering, as described above. Specifically, the system works as follows: First, pedestrian data is collected by the data collection module and sent to the data processing module. Then, the data processing module processes the collected pedestrian data and sends the processed data to the activity recognition module. Next, the activity recognition module uses an activity recognition model to identify pedestrian actions based on the processed data and sends the result to the position prediction module. Then, the position prediction module predicts the pedestrian position based on the pedestrian actions and sends the result to the position correction module. Finally, the position correction module corrects the pedestrian position based on the result and the magnetic field sequence, and sends the result to the data output module. Finally, the data output module outputs the corrected pedestrian position result.

[0068] Example 2:

[0069] The self-correcting dead reckoning system based on activity recognition and fusion filtering in this embodiment is based on Embodiment 1. The data processing module is used to process the collected data. The data processing module includes a data denoising module and a data coordinate system transformation module. The data denoising module is used to denoise the collected data; the data coordinate system transformation module is used to transform the coordinate system of the data.

[0070] The data coordinate system transformation module is used to transform the coordinate system of data, specifically including the following steps:

[0071] It is known that sensor data acquired by smart devices is collected based on the device's own coordinate system. However, the device's posture does not always remain consistent with the data collector's during actual walking. Therefore, the device's own coordinate system is called the device coordinate system, and the coordinate system used for activity classifiers and navigation instructions is called the navigation coordinate system. Figure 2 shows a schematic diagram of the device coordinate system, and Figure 3 shows a schematic diagram of the navigation coordinate system. In Figures 2 and 3, the X-axis points due east, the Y-axis points due north, and the Z-axis points perpendicular to the sky.

[0072] Assume the acceleration data collected by the accelerometer is represented in the device coordinate system as follows:

[0073]

[0074] In the above formula (1), These represent the acceleration data along the X, Y, and Z axes in the device coordinate system, respectively.

[0075] Assume the magnetic field data collected by the magnetometer is represented in the device coordinate system as follows:

[0076]

[0077] In the above formula (2), These represent the magnetic field strength data along the X, Y, and Z axes in the device coordinate system, respectively.

[0078] Assume that during the rotation of the smart device, at time t, the rotation angles of the X-axis, Y-axis, and Z-axis in the device coordinate system are ω, And γ, then the rotation matrix data for the X-axis, Y-axis and Z-axis in the device coordinate system are respectively expressed as:

[0079]

[0080]

[0081]

[0082] In the above formula (3), T ω,t This represents the rotation matrix data of the X-axis in the device coordinate system, in formula (4) above. This represents the rotation matrix data of the Y-axis in the device coordinate system. In the above formula (5), T γ,t This represents the rotation matrix data of the Z-axis in the device coordinate system.

[0083] To provide positioning and navigation services in real-world scenarios, this embodiment uses a data coordinate system transformation module to convert data from the device coordinate system to the navigation coordinate system. Assuming the transformation result in the navigation coordinate system is a transformation matrix T, then:

[0084]

[0085] In the above formula (6), c represents the cosine function and s represents the sinine function.

[0086] Multiply the transformation result matrix T by the acceleration data in the device coordinate system to obtain the acceleration data in the navigation coordinate system. Let the acceleration data in the navigation coordinate system be Acc'.

[0087]

[0088] In the above formula (7), These represent the acceleration data along the X, Y, and Z axes in the navigation coordinate system, respectively.

[0089] Multiplying the transformation result matrix T by the magnetic field data in the device coordinate system yields the magnetic field data in the navigation coordinate system, denoted as M'.

[0090]

[0091] In the above formula (8), These represent the magnetic field data along the X, Y, and Z axes in the navigation coordinate system, respectively.

[0092] Example 3:

[0093] The self-correcting dead reckoning system based on activity recognition and fusion filtering in this embodiment, based on embodiment 2, has a data processing module for processing the collected data. This data processing module includes a data denoising module and a data coordinate system transformation module. The data denoising module is used to denoise the collected data, and the data coordinate system transformation module is used to transform the coordinate system of the denoised data.

[0094] The data noise reduction module is used to perform noise reduction processing on the collected data, specifically including the following steps:

[0095] It is known that when using an accelerometer to collect data, the sampled values ​​are affected by environmental noise. Among these noises, gravitational acceleration has the most significant impact on data sampling. Assuming the smart device is placed flat on a horizontal table, the accelerometer actually collects the gravitational acceleration G. When the smart device is in free fall, the collected data is 0. In this embodiment, to obtain the actual acceleration value, it is necessary to filter out the gravitational acceleration data from the acceleration data. This embodiment uses a high-pass filter to solve this problem, as detailed below:

[0096] This can be achieved using the following formulas (9) and (10):

[0097] g t =a*g t-1 +(1-a)*Acc' t Formula (9)

[0098]

[0099] In the above formula, g t Here is the gravitational acceleration data of the smart device at time t, where 'a' is the high-pass filter parameter and 'g' is the gravitational acceleration. t-1 For the gravitational acceleration data of the smart device at time t-1, Acc' t This refers to the acceleration data of the smart device in the navigation coordinate system at time t. This represents the final acceleration data of the smart device at time t after passing through a high-pass filter.

[0100] In this embodiment, a = 0.8.

[0101] Using the above method, the final acceleration data of the smart device at time t after passing through the high-pass filter is obtained.

[0102] Example 4:

[0103] The self-correcting dead reckoning system based on activity recognition and fusion filtering in this embodiment is based on Embodiment 3.

[0104] It is known that the sensors embedded in different smart devices are affected by their inherent errors and sampling frequency during data sampling. For example, the inherent errors of accelerometers in some common smart devices are shown in Table 1:

[0105] Table 1 Comparison of inherent errors of accelerometers in common smart devices

[0106] Device Model Acc_xAcc_yAcc_zMi10Pro0.0001850.000267-0.005992Mi60.000248-0.003745-0.218308Huawei P400.003538-0.013606-0.218308Honor V200.0026210.005326-0.317217Samsung S100.006388-0.0021460.06709 surface

[0107] In Table 1 above, Acc_x represents the inherent error data of the accelerometer of the smart device on the x-axis, Acc_y represents the inherent error data of the accelerometer of the smart device on the y-axis, and Acc_z represents the inherent error data of the accelerometer of the smart device on the z-axis. As can be seen from Table 1, the inherent errors of sensors in different devices are different. Therefore, it is necessary to collect error data of the smart device separately and remove the inherent errors when collecting the data.

[0108] In this embodiment, the smart device is rotated in a figure-eight motion before the magnetometer provides direction indication. Specifically, the user is required to place the smart device horizontally for h seconds (h ≥ 10) before data collection. The smart device then begins collecting and processing data. Before processing, jitter data collected at the start and end times of the data collection is removed. Specifically, h' seconds of data before and after the h seconds of data collected by the smart device are removed (h' ≤ 2). Then, the average value of all data after removing the h' seconds of data is calculated to determine the inherent error of the smart device's accelerometer. The formula for determining the inherent error of the smart device's accelerometer is as follows:

[0109]

[0110] In the above formula (11), Acc error Acc is the inherent error of the accelerometer in the final stage of intelligent devices. real The acceleration data collected when the smart device is stationary is denoted as n, where n is the total number of samples collected and n′ is the number of samples within h′ seconds.

[0111] Finally, the final acceleration data of the smart device is calculated according to formula (12):

[0112]

[0113] In the above formula (12), This represents the final acceleration data of the smart device at time t. Acc represents the final acceleration data of the smart device at time t after passing through a high-pass filter.error This refers to the inherent error of the accelerometer in the final stage of intelligent devices.

[0114] After processing the accelerometer data from the smart device using the above steps, the processed data is divided into multiple active data instance windows. Each active data instance window consists of a series of sensor data. In this embodiment, the dataset is defined as: Data = {Data...} i ,Labeli}, where Data i For each input action data segment, in,

[0115]

[0116]

[0117]

[0118]

[0119]

[0120] In the above formula, K represents the window size for each input action data segment. In this embodiment, the data window of each activity data instance is set to have 50% data overlap with the previous window to learn the action characteristics of the same action in different time periods. These represent the acceleration data of the smart device along the three axes in the navigation coordinate system. These represent the gyroscope data of the smart device along the three axes in the coordinate system.

[0121] This embodiment addresses the actual walking status of pedestrians using mobile phones for navigation indoors. Since some smart devices lack barometer sensors and cannot directly use sensor data to determine whether someone is going up or down stairs, this embodiment considers five common pedestrian activity states: walking slowly, walking normally, walking quickly, going downstairs, and going upstairs. The corresponding icons for each pedestrian activity state are shown in Table 2.

[0122] Table 2. Signs corresponding to five pedestrian activity states.

[0123] Action Icons and Specific Actions: A1 Pedestrian going downstairs; A2 Pedestrian walking briskly; A3 Pedestrian walking normally; A4 Pedestrian walking slowly; A5 Pedestrian going upstairs. surface

[0124] Action Label i The value is A1, A2, A3, A4, or A5. Label i Action labels corresponding to each action data segment.

[0125] Example 5:

[0126] This embodiment of the self-correcting dead reckoning system based on activity recognition and fusion filtering is based on Embodiment 4. The system includes a data collection module, a data processing module, an activity recognition module, a position prediction module, a position correction module, and a data output module. The activity recognition module uses an activity recognition model to identify pedestrian actions based on the processed data. In this embodiment, the activity recognition model combines Attention Mechanism with a CNN-LSTM neural network model. Based on the high-dimensional data features extracted by the standard CNN network model, Attention Mechanism is used to extract important sub-sequence features, enabling the LSTM neural network model to retain longer effective memory information for more accurate pedestrian activity recognition. This embodiment's activity recognition model employs an Activity Recognition Based on Attention Mechanism and CNN-LSTM (ARAC) method to accurately identify pedestrian actions. The Attention Mechanism method in this activity recognition model can quickly select high-value information from a large amount of data, making it suitable for real-time activity recognition.

[0127] Feature extraction from sensor data is a crucial part of the entire activity recognition process. This embodiment employs a convolutional neural network for feature extraction. In the convolutional layers, 64 1×3 convolutional kernels are used for feature extraction. To prevent overfitting, a dropout layer is used after feature extraction, with a dropout rate of 0.5. Since the action data sequence is time-series data, the model complexity increases with network depth; therefore, this embodiment uses two convolutional layers. After extracting high-dimensional data features from the convolutional layers, the Attention Mechanism uses 64 neurons with Softmax as the activation function to generate weight parameters W. i The convolutional layer multiplies the features element-wise with the data feature set. The higher the feature importance of the output of the convolutional layer, the closer the output of the Attention Mechanism is to 1, and vice versa.

[0128] After obtaining the weighted high-dimensional features, they are used as input units of a Long Short-Term Memory (LSTM) neural network to build a time series prediction model. A Dropout layer is added between the LSTM layer and the Flatten layer to prevent overfitting. Then, the Flatten layer is used to convert the multidimensional output of the LSTM layer into one dimension. Finally, a fully connected (Dense) layer is used as the output layer, and the Softmax function is used as the activation function to output the final classification result.

[0129] This embodiment uses the cross-entropy function as the multi-class classification loss function to minimize the loss result. The loss function is defined as follows:

[0130] loss = -∑ i (y i *logy p +(1-y i )*log(1-y p )) Formula (13)

[0131] In the above formula (13), i is the current predicted activity instance number, and y i y represents the actual action in the current sequence. p This represents the prediction result of the classifier.

[0132] After training the activity recognition model through the above steps, this embodiment will transfer the trained activity recognition model to a mobile smart device, so that the system of this embodiment can recognize pedestrian actions in real time on the mobile smart device.

[0133] Example 6:

[0134] This embodiment of the self-correcting dead reckoning system based on activity recognition and fusion filtering is based on Embodiment 5. The system includes a data collection module, a data processing module, an activity recognition module, a position prediction module, a position correction module, and a data output module. The position prediction module is used to predict pedestrian positions, specifically by estimating step length and calculating dead angles based on pedestrian action recognition results. To accurately estimate pedestrian positions, it is necessary to calculate the pedestrian dead angle and the distance the pedestrian has walked.

[0135] Calculating a pedestrian's position angle specifically includes the following steps:

[0136] It is known that the gyroscope sensors embedded in smart devices are not calibrated and have a very fast response speed in actual use. Therefore, they will incorporate the collected noise data into the integration calculation, causing short-term errors in dead angle estimation. While the dead angle calculated using data from magnetometers and accelerometers may fluctuate significantly in the short term, it accurately reflects changes in dead angle over a long period. Therefore, in this embodiment, the sampled data from the MEMS sensors needs to be fused during the actual trajectory calculation to obtain an accurate pedestrian dead angle. This embodiment uses a fusion filtering method to effectively reduce the impact of environmental noise on pedestrian dead angle estimation. The fusion filtering process is shown in Figure 4.

[0137] First, the acceleration output angle is calculated based on the collected acceleration data, the magnetic field strength output angle is calculated based on the collected magnetic field strength data, and the dead angle output by the gyroscope is calculated based on the data collected by the gyroscope.

[0138] Then, the acceleration output angle and the magnetic field strength output angle are both input into the low-pass filter LPF. The acceleration output angle and the magnetic field strength output angle are processed by the low-pass filter to obtain the filtered acceleration output angle and the filtered magnetic field strength output angle. At the same time, the dead angle output by the gyroscope is input into the high-pass filter HPF. The low-frequency components in the gyroscope data are removed by the high-pass filter to reduce the influence of noise and obtain the filtered gyroscope output dead angle.

[0139] Finally, the acceleration output angle filtered by the low-pass filter, the magnetic field strength output angle filtered by the low-pass filter, and the gyroscope output dead angle filtered by the high-pass filter are all input into the fusion filter. The fused pedestrian dead angle is calculated by the fusion filter. The formula for the fusion filter is as follows:

[0140] FusedOri=f*Gyr+(1-f)*AccMag Formula (14)

[0141] In the above formula (14), FusedOri is the pedestrian dead angle output by the fusion filter, f is the parameter of the fusion filter, Gyr is the dead angle output by the filtered gyroscope, and AccMag is the dead angle obtained by fusing the filtered acceleration output angle and the filtered magnetic field strength output angle.

[0142] AccMag is calculated using the following formula:

[0143]

[0144]

[0145]

[0146]

[0147]

[0148]

[0149]

[0150]

[0151]

[0152]

[0153] M y =AZ*H x -A x *H z

[0154]

[0155] In the above formula, These represent the acceleration data along the X, Y, and Z axes in the navigation coordinate system, respectively. These represent the magnetic field data along the X, Y, and Z axes in the navigation coordinate system, respectively.

[0156] The pedestrian dead angle FusedOri output by the fusion filter obtained through the above steps is the final pedestrian dead angle that needs to be solved.

[0157] Calculating pedestrian walking distance specifically includes the following steps:

[0158] First, the number of steps taken by pedestrians is detected using a peak detection method based on dynamic thresholds, specifically including the following steps:

[0159] It is known that the walking distance of a pedestrian depends on the number of steps detected and the walking pattern corresponding to each step. In this embodiment, a peak detection method based on dynamic thresholds is used to detect the number of steps. When a walking event is detected, the system in this embodiment uses an activity classifier to identify pedestrian activity and estimate the walking distance.

[0160] The first step is to know that the acceleration data collected by the accelerometer of the smart device worn by the pedestrian during walking is presented in the form of a sine wave. In this embodiment, each peak in the sine wave is taken as a step point. By finding the step point, the characteristic value is found by analyzing the waveform characteristics based on the step point.

[0161] The second step is to specify the number of consecutive rises in the acceleration sine wave curve;

[0162] The third step is to stipulate that the difference between the peaks and troughs of the acceleration sine wave curve must be greater than a threshold.

[0163] The fourth step is to stipulate that the distance between adjacent peaks of the acceleration sine wave curve is greater than a threshold.

[0164] In summary, the peaks in the acceleration data curves that meet the above conditions can be identified as step points.

[0165] In this embodiment, a peak is defined as the acceleration data rising twice consecutively in a sinusoidal form, with the peak value exceeding a predetermined threshold for the current gait. The difference between the peak and the adjacent trough is greater than a dynamic threshold S. t A time difference greater than 0.25 seconds is counted as a valid step.

[0166] The fifth step, the peak detection method based on dynamic thresholds, also includes updating the dynamic threshold S. t The steps specifically include: continuously recording the differences between four effective peaks and troughs in the acceleration sine wave curve and calculating their average value as the updated dynamic threshold S. t .

[0167] Figure 5 shows the step detection performance of the peak detection method based on dynamic threshold on a sample mixed gait acceleration data. As can be seen from Figure 5, the peak detection method based on dynamic threshold is more effective in detecting the number of pedestrian steps, further improving the effectiveness of calculating the pedestrian walking distance.

[0168] Then, the pedestrian's walking stride length is calculated using formula (15):

[0169]

[0170] In the above formula (15), S represents the step size, a max and a min These are the maximum and minimum acceleration values ​​on the Z-axis of the navigation coordinate system, respectively. The step size parameter L is a constant related to the step frequency; generally, the faster the walking speed, the larger the value of L.

[0171] Finally, the pedestrian's current position coordinates are estimated in real time based on the pedestrian's heading angle and step size output by the fusion filter. This includes the following steps:

[0172] Let the pedestrian's current position coordinates be (x t ,y t The pedestrian's previous position coordinates are (x... t-1 ,y t-1 The current pedestrian positioning angle output by the fusion filter is θ. t The current step size is S t Then, according to formula (16), the current position coordinates of the pedestrian are calculated as (x t ,y t ),

[0173]

[0174] Example 7:

[0175] This embodiment of the self-correcting dead reckoning system based on activity recognition and fusion filtering is based on Embodiment 6. The self-correcting dead reckoning system based on activity recognition and fusion filtering includes a data collection module, a data processing module, an activity recognition module, a position prediction module, a position correction module, and a data output module. The position correction module is used to correct the pedestrian's position based on the magnetic field strength sequence, reducing the cumulative error of the positioning system and limiting the overall performance degradation. This position correction module does not rely on infrastructure; it only requires a guide to walk along the navigation path using a handheld smart device. During the walk, the position correction module in this embodiment establishes a magnetic field strength fingerprint sequence as a reference path to match the sampled data obtained during use, thereby calibrating the pedestrian's current heading and position. Specifically, it includes the following steps:

[0176] Step 1: Given that pedestrians exhibit different walking patterns, and the collected magnetic field strength sequence data varies depending on these patterns, this step first defines walking patterns. Considering the actual walking state of pedestrians using mobile phones for navigation indoors, and the fact that some smart devices lack barometer sensors to directly determine whether they are going up or down stairs, this embodiment considers five common pedestrian activity states: slow walking, normal walking, fast walking, going downstairs, and going upstairs. The corresponding identifiers for each pedestrian activity state are shown in Table 2.

[0177] Table 2. Signs corresponding to five pedestrian activity states.

[0178] Action Icons and Specific Actions: A1 Pedestrian going downstairs; A2 Pedestrian walking briskly; A3 Pedestrian walking normally; A4 Pedestrian walking slowly; A5 Pedestrian going upstairs. surface

[0179] The second step is to define the magnetic field intensity sequence under different walking states. In this embodiment, the magnetic field intensity sequence under different walking states is defined based on the Dynamic Time Warping based on Discrete Timing Difference (DTW-DTD) method. Specifically, the original magnetic field intensity sequence is converted into a difference sequence of adjacent values ​​through formula (17).

[0180]

[0181] In the above formula (17), m′(i) is the sequence of differences between adjacent values, m i Let i be the i-th value in the magnetic field strength sequence, where i is a natural number.

[0182] The third step is to use the dynamic time warping (DTW) method to determine the similarity of the interpolated sequence of m′(i) based on the difference sequence of adjacent values ​​obtained above. Specifically, the DTW distance between the two variation distribution sequences m′(i) and m′(j) is taken as the distance between the two magnetic field strength sequences, where i and j are both natural numbers.

[0183] Compared to the traditional DTW method, DTW-DTD focuses more on the similarity of sequence change trends. DTW-DTD and DTW were used to compare the similarity of magnetic signal sequences sampled under different walking conditions on the same road segment. The distances between the sequences calculated by the two methods are shown in Table 3. According to the experimental results, after DTW-DTD converts the numerical sequence into a difference sequence, the two curves become more similar. The total distance between the magnetic signal sequences sampled under the two walking modes decreased from 722.09 to 81.53, and the distance between the nearest neighbors decreased from 1.173 to 0.102.

[0184] Table 3. Sequence distances calculated by the traditional DTW method and the DTW-DTD method.

[0185] Method Total Distance Nearest Point Distance DTW 722.09 1.173 DTW-DTD 81.53 0.102 surface

[0186] The fourth step is to calculate the DTW-DTD distance based on the magnetic field strength sequence collected by the user and the magnetic field strength sequence on the reference path, and replace the pedestrian's current position coordinates with the matched magnetic field data location.

[0187] Through the above methods and steps, the system in this embodiment calculates the DTW-DTD distance based on the magnetic field strength sequence collected by the user and the magnetic field strength sequence on the reference path, and replaces the pedestrian's current position coordinates with the matched magnetic field data location, which greatly reduces the cumulative error in the navigation process.

[0188] Example 8:

[0189] The self-correcting dead reckoning method based on activity identification and fusion filtering in this embodiment is based on the self-correcting dead reckoning system based on activity identification and fusion filtering proposed in Embodiment 7. The method specifically includes the following steps:

[0190] 1) First, pedestrian data is collected using accelerometers, gyroscopes, and magnetometers;

[0191] 2) Then, the collected data is processed, including data noise reduction and coordinate axis transformation, to improve the effectiveness of the sampled data;

[0192] 3) Next, an activity recognition method based on Attention Mechanism and CNN-LSTM is proposed to accurately identify pedestrian actions;

[0193] 4) Subsequently, data from different types of sensors are fused and filtered to ensure the accuracy of the dead angle calculation during long-term walking. At the same time, the walking distance and direction are estimated based on the actual activities of pedestrians, and the error between the estimated step length and the actual step length is controlled within a reasonable range.

[0194] 5) Finally, a dynamic time planning algorithm based on discrete time difference is proposed to match and compare data collected under different walking modes on the same path.

[0195] Example 9:

[0196] The self-correcting dead reckoning method based on activity identification and fusion filtering in this embodiment, based on Embodiment 8, includes the following steps:

[0197] The first step involves collecting sensor data through the data collection module of the self-correcting dead reckoning system based on activity recognition and fusion filtering. i Magnetic field strength fingerprint sequence m i And other data, and transfer the sensor data to Data i Magnetic field strength fingerprint sequence m i Input into the data processing module;

[0198] The second step involves converting the input device coordinate system data into navigation coordinate system data using the transformation matrix T in the data processing module of the self-correcting dead reckoning system based on activity recognition and fusion filtering. The navigation coordinate system data is then subjected to noise reduction processing, and the processed data is input into the activity recognition module.

[0199] The third step involves using the activity recognition module in the self-correcting dead reckoning system based on activity recognition and fusion filtering. Based on the processed data, the activity recognition module uses an activity recognition model to identify pedestrian actions and determine pedestrian walking pattern A. i The result of this action is then input into the position prediction module;

[0200] The fourth step involves estimating the pedestrian's position using the position prediction module in the self-correcting dead reckoning system based on activity recognition and fusion filtering. Specifically, this involves estimating the step length and calculating the dead angle based on the pedestrian action recognition results, and includes the following steps:

[0201] Using an activity recognition model to identify pedestrian walking patterns A i ;

[0202] According to walking mode A i Set the corresponding step size parameter L;

[0203] Calculate the pedestrian's walking stride length based on the stride parameter L;

[0204] The pedestrian's heading angle is calculated using the fusion filter formula, which is as follows:

[0205] FusedOri=f*Gyr+(1-f)*AccMag formula (14)

[0206] In the above formula (14), FusedOri is the pedestrian dead angle output by the fusion filter, f is the parameter of the fusion filter, Gyr is the dead angle output by the filtered gyroscope, and AccMag is the dead angle obtained by fusing the filtered acceleration output angle and the filtered magnetic field strength output angle.

[0207] The pedestrian's current position (x) is calculated using the calculated length and heading angle. t ,y t The location is then input into the position correction module.

[0208] The fifth step is to calculate the pedestrian's current position on the magnetic field trajectory data using the DTW-DTD method in the self-correcting dead reckoning system based on activity recognition and fusion filtering, and then correct the pedestrian's actual position based on this position.

[0209] The sixth step is to output the corrected actual position of the pedestrian through the data output module of the self-correcting dead reckoning system based on activity recognition and fusion filtering.

[0210] Example 10:

[0211] The self-correcting dead reckoning method based on activity recognition and fusion filtering in this embodiment is based on embodiment 9, and this embodiment experimentally verifies the method of embodiment 9.

[0212] The experimental equipment and setting are as follows:

[0213] This embodiment adapts the system of the present invention to a mobile smart device. The experimental scenario is 2800m. 2 Multiple volunteers constructed five types of activity behavior datasets in the above scenarios, including a single-story teaching building (planar environment) and a three-story large commercial center (240m×100m).

[0214] Because the sampling frequencies of MEMS sensors in different devices are inconsistent, the sensor data sampling frequency of both the data collection module and the navigation module was set to 20Hz to ensure stable operation on commonly used mobile smart devices. Simultaneously, based on experimental results from volunteers and combined with the gait recognition model, an appropriate step length parameter L was selected to make Seen's step length estimation adaptable to walking distance calculations under mixed gait conditions. After extensive experiments with multiple volunteers, the step length parameter L was set to 0.46 for slow walking, 0.48 for normal walking, and 0.53 for fast walking. The filter parameter f in the fusion process was set to 0.98.

[0215] This embodiment compares three neural networks—Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), and CNN-LSTM—on the publicly available datasets Unimib-SHAR (ADL), WISDM, and the self-built activity dataset of this embodiment. As shown in Table 4, the experimental results demonstrate that the activity recognition method ARAC proposed in this embodiment outperforms the other three on all three datasets.

[0216] Table 4. Comparison of classification accuracy of four neural network models

[0217] Network Model UnimibWISDM In this embodiment, using a self-built active dataset, the percentages for CNN, LSTM, CNN-LSTM, and ARAC are as follows: CNN 92.56%, 93.25%, 94.37%; LSTM 92.34%, 94.71%, 93.91%; CNN-LSTM 95.38%, 96.18%, 96.97%; ARAC 97.16%, 98.13%, 98.75%. surface

[0218] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A self-correcting dead reckoning system based on activity recognition and fusion filtering, characterized in that: The system specifically includes a data collection module, a data processing module, an activity recognition module, a position prediction module, a position correction module, and a data output module. The data collection module includes several MEMS sensors to collect pedestrian data. The data processing module processes the collected data. The activity recognition module uses an activity recognition model to identify pedestrian movements based on the processed data. The position prediction module predicts pedestrian positions, specifically by estimating step length and calculating dead angle based on the pedestrian movement recognition results. The calculation of the pedestrian dead angle includes the following steps: first, calculating the acceleration output angle based on the collected acceleration data; and second, calculating the acceleration output angle based on the collected acceleration data... The magnetic field strength output angle is calculated from the obtained magnetic field strength data, and the dead angle output by the gyroscope is calculated based on the data collected by the gyroscope. Then, both the acceleration output angle and the magnetic field strength output angle are input into a low-pass filter (LPF). The LPF filters the acceleration and magnetic field strength output angles to obtain filtered acceleration and magnetic field strength output angles. Simultaneously, the dead angle output by the gyroscope is input into a high-pass filter (HPF). The HPF removes low-frequency components from the gyroscope data to obtain a filtered dead angle output by the gyroscope. Finally, the acceleration output angle filtered by the low-pass filter, the magnetic field strength output angle filtered by the low-pass filter, and the dead angle output by the gyroscope filtered by the high-pass filter are all input into a fusion filter. The fusion filter calculates the fused pedestrian dead angle. The fusion filter formula is as follows: FusedOri = f Gyr+(1-f) In the above formula (14), FusedOri is the pedestrian dead angle output by the fusion filter, f is the parameter of the fusion filter, Gyr is the dead angle output by the filtered gyroscope, and AccMag is the dead angle obtained by fusing the filtered acceleration output angle and the filtered magnetic field strength output angle; the pedestrian dead angle FusedOri output by the fusion filter obtained through the above steps is the final pedestrian dead angle to be solved; the position correction module is used to correct the pedestrian position according to the magnetic field strength sequence; firstly, pedestrian data is collected by the data collection module and the data is processed. The data is sent to the data processing module, which processes the collected pedestrian data and then sends the processed data to the activity recognition module. The activity recognition module then uses an activity recognition model to identify pedestrian actions based on the processed data and sends the result to the location prediction module. The location prediction module then predicts the pedestrian position based on the pedestrian actions and sends the result to the location correction module. The location correction module then corrects the pedestrian position based on the result and the magnetic intensity sequence and sends the result to the data output module. Finally, the data output module outputs the corrected pedestrian position result.

2. The self-correcting dead reckoning system based on activity recognition and fusion filtering according to claim 1, characterized in that: The data processing module includes a data noise reduction module and a data coordinate system transformation module. The data noise reduction module is used to reduce the noise of the collected data, and the data coordinate system transformation module is used to transform the coordinate system of the data.

3. The self-correcting dead reckoning system based on activity recognition and fusion filtering according to claim 2, characterized in that: The activity recognition module identifies pedestrian actions based on the processed data using an activity recognition model. This activity recognition model combines the Attention Mechanism with a CNN-LSTM neural network model and uses the cross-entropy function as a multi-class loss function, defined as follows: Formula (13) In the above formula (13), This is the current predicted activity instance number. Represents the actual action in the current sequence. The predicted action result represents the classifier; the above activity recognition model is then ported to mobile smart devices, enabling real-time recognition of pedestrian actions on mobile smart devices.

4. The self-correcting dead reckoning system based on activity recognition and fusion filtering according to claim 3, characterized in that: The step length estimation specifically includes the following steps: First, the number of steps taken by the pedestrian is detected by a peak detection method based on dynamic thresholds; then, the step length of the pedestrian is calculated by formula (15): Formula (15); in the above formula (15), Represents step length, and These are the maximum and minimum acceleration values ​​along the Z-axis of the navigation coordinate system, respectively, and the step size parameter. It is a constant related to stride frequency; generally, the faster the walking speed, the higher the frequency. The larger the value, the more accurate the pedestrian's current position coordinates are estimated in real time based on the pedestrian's heading angle and step size output by the fusion filter.

5. The self-correcting dead reckoning system based on activity recognition and fusion filtering according to claim 4, characterized in that: The position correction module is used to correct the pedestrian position based on the magnetic field strength sequence, specifically including the following steps: First, define the pedestrian walking pattern; then, define the magnetic field strength sequence under different walking states based on the dynamic time planning method of discrete time difference, specifically by converting the original magnetic field strength sequence into a difference sequence of adjacent values ​​through formula (17). In the above formula (17), It is a sequence of differences between adjacent values. The first in the magnetic field strength sequence A number, It is a natural number; then, based on the difference sequence of adjacent values ​​obtained above... ,right The interpolation sequence is similar to the reference path using a dynamic time programming method. Finally, the DTW-DTD distance is calculated based on the magnetic field strength sequence collected by the user and the magnetic field strength sequence on the reference path, and the current position coordinates of the pedestrian are replaced with the matching magnetic field data.

6. A self-correcting dead reckoning method based on activity recognition and fusion filtering, characterized in that: The self-correcting dead reckoning system based on activity recognition and fusion filtering according to any one of claims 1 to 5 specifically includes the following steps: 1) First, pedestrian data is collected using accelerometers, gyroscopes, and magnetometers; 2) Then, the collected data is processed, specifically including data denoising and coordinate axis transformation, to improve the effectiveness of the sampled data; 3) Next, an activity recognition method based on Attention Mechanism and CNN-LSTM is proposed to accurately identify pedestrian actions; 4) Subsequently, fusion filtering is performed on data from different types of sensors to ensure the accuracy of dead angle calculation during long-term walking, while estimating the walking distance and direction based on the actual activities of pedestrians, and controlling the error between the estimated step length and the actual step length within a reasonable range; 5) Finally, a dynamic time planning algorithm based on discrete time difference is proposed to match and compare data collected from different walking modes under the same path.

Images