Indoor positioning method and device based on fusion of wifi and vision

Abstract

The application provides an indoor positioning method and device based on Wi-Fi and visual fusion, which comprises the following steps: predicting a first position of a pedestrian according to a received signal strength of a plurality of WiFi devices of a target terminal of the pedestrian by using a random forest algorithm, wherein the random forest algorithm constrains a positioning area of the target terminal; identifying the pedestrian in a video by using YOLOv7, and tracking the pedestrian in the video by using a Deep SORT algorithm to obtain a second position of the pedestrian; and fusing the first position and the second position of the pedestrian by using an adaptive Kalman filter to obtain a final position of the pedestrian. The application significantly reduces the uncertainty of each independent technology by using the adaptive Kalman filter technology (combining the WiFi and visual positioning results, dynamically adjusting the filter parameters and weights), and improves the overall robustness and accuracy of the system.

Description

Technical Field

[0001] The present invention relates to the field of indoor positioning technology, and in particular to an indoor positioning method and device based on Wi-Fi and vision fusion. Background Art

[0002] In today's information age, precise indoor positioning technology has become a key technology in areas such as intelligent navigation, environmental monitoring, emergency rescue, and personal services. Although the Global Positioning System (GPS) provides highly accurate positioning services outdoors, its accuracy is significantly reduced indoors due to signal attenuation and multipath effects. Therefore, outdoor positioning and navigation technology cannot be directly applied indoors. However, with the development and popularization of the Internet of Things (IoT), indoor positioning technology has become indispensable. Improving indoor positioning accuracy has become increasingly urgent, and both academia and the information technology industry have conducted research on this issue. In recent years, WiFi-based and vision-based positioning technologies have attracted widespread attention due to their unique advantages.

[0003] WiFi positioning technology primarily uses the Received Signal Strength Indication (RSSI) signal strength emitted by deployed wireless access points (APs) in indoor environments to estimate a user's location. This technology offers advantages such as low deployment cost, wide coverage, and the lack of additional hardware. However, the main challenge with WiFi positioning is that its accuracy is significantly affected by environmental interference. Factors such as human motion, multipath propagation, and indoor temperature fluctuations can cause signal strength fluctuations, thereby impacting positioning accuracy.

[0004] Visual positioning technology determines location by analyzing image information captured by a camera, offering high positioning accuracy. Visual positioning typically relies on image recognition and machine learning techniques to identify feature points in an image for precise positioning. While visual positioning offers high accuracy within a small area, it significantly decreases in indoor environments with a wider field of view. This is because over large areas, feature points in an image may be difficult to accurately match, leading to increased positioning error. Furthermore, visual positioning requires processing large amounts of image data, requiring high computing resources and exhibiting instability in environments with poor lighting conditions or a restricted field of view. Summary of the Invention

[0005] The present invention provides an indoor positioning method and device based on Wi-Fi and vision fusion, which are used to address the defects of Wi-Fi positioning and vision positioning in the existing technologies. By combining Wi-Fi and vision positioning results through adaptive Kalman filter technology (FAKF), the filter parameters and weights are dynamically adjusted, which significantly reduces the uncertainty of each independent technology and improves the overall robustness and accuracy of the system.

[0006] The present invention provides an indoor positioning method based on WiFi and vision fusion, comprising:

[0007] Predicting a first location of the pedestrian using a random forest algorithm based on received signal strengths of a target terminal of the pedestrian to multiple WiFi devices, wherein the random forest algorithm constrains a positioning area of ​​the target terminal;

[0008] Using YOLOv7 to identify the pedestrian in the video, and using the Deep SORT algorithm to track the pedestrian in the video to obtain a second position of the pedestrian;

[0009] An adaptive Kalman filter is used to fuse the first position and the second position of the pedestrian to obtain a final position of the pedestrian.

[0010] According to an indoor positioning method based on WiFi and vision fusion provided by the present invention, the method predicts the first position of the pedestrian using a random forest algorithm based on the received signal strength of the pedestrian's target terminal to multiple WiFi devices, including:

[0011] Collect the signal strength received by the sample terminal to the WiFi device at multiple locations and establish an offline fingerprint database;

[0012] Predicting the location of the pedestrian based on the online received signal strength of multiple WiFi devices by the pedestrian's target terminal to obtain the pedestrian's current predicted point;

[0013] Searching the offline fingerprint database for a position within a preset offset range of the current prediction point and a received signal strength corresponding to the position;

[0014] Training the random forest algorithm using the position within a preset offset range of the current prediction point and the received signal strength corresponding to the position as a training data set;

[0015] The online received signal strength is input into the trained random forest algorithm to obtain the first position of the pedestrian.

[0016] According to an indoor positioning method based on WiFi and vision fusion provided by the present invention, the random forest algorithm uses mean square error, absolute error or Poisson deviation when calculating node impurity.

[0017] According to an indoor positioning method based on WiFi and vision fusion provided by the present invention, before using the DeepSORT algorithm to track the pedestrian in the video to obtain the second position of the pedestrian, the method further includes:

[0018] Use the external reference matrix to transform the points in the world coordinate system to the camera coordinate system;

[0019] Use the internal reference matrix to project the points in the camera coordinate system onto the imaging plane to obtain pixel coordinates;

[0020] The pixel coordinates acquired by the camera are converted to world coordinates according to the conversion relationship between the world coordinate system and the camera coordinate system, and the conversion relationship between the camera coordinate system and the imaging plane.

[0021] According to an indoor positioning method based on WiFi and vision fusion provided by the present invention, the pixel coordinates obtained by the camera are converted to world coordinates using the following formula:

[0022]

[0023] The internal reference matrix includes the focal length f, the pixel size d x and d y , and the optical center coordinates u0 and v0 of the imaging coordinate system, the external reference matrix contains the rotation matrix R and the translation vector T, (x w ,y w ,z w ) is the world coordinate in the world coordinate system, (u, v) is the pixel coordinate on the imaging plane, Z c The depth or distance of the object in the camera coordinate system.

[0024] According to an indoor positioning method based on WiFi and vision fusion provided by the present invention, the method uses an adaptive Kalman filter to fuse the first position and the second position of the pedestrian to obtain the final position of the pedestrian, including:

[0025] At each time step, the Kalman filter performs a prediction step, estimating the state and covariance matrix for the next time step;

[0026] Updating the Kalman gain using the covariance matrix, and updating the state and the covariance matrix as a whole using the updated Kalman gain and the measurement matrices of the Wi-Fi device and the camera;

[0027] determining residuals between the state and a first position and a second position measured by the Wi-Fi device and the camera, respectively, and updating measurement weights of the Wi-Fi device and the camera when no outliers are determined based on the residuals;

[0028] The final position of the pedestrian is determined according to the measurement weights of the Wi-Fi device and the camera.

[0029] According to an indoor positioning method based on WiFi and vision fusion provided by the present invention, the weight factor of the adaptive Kalman filter is measured by the following formula:

[0030]

[0031] Among them, O(t0,t f ) represents the time from time t0 to t f The weight factor between is the state transition matrix F t At the power of time step t-t0, Tr represents the trace of the matrix, that is, the sum of the diagonal elements of the matrix, H wifi represents the measurement matrix of the WiFi device, H camera represents the measurement matrix of the camera.

[0032] The present invention also provides an indoor positioning device based on WiFi and vision fusion, comprising:

[0033] A first positioning module is configured to predict a first position of the pedestrian based on the received signal strength of a target terminal of the pedestrian to multiple WiFi devices using a random forest algorithm, wherein the random forest algorithm constrains a positioning area of ​​the target terminal;

[0034] a second positioning module, configured to identify the pedestrian in the video using YOLOv7 and track the pedestrian in the video using a Deep SORT algorithm to obtain a second position of the pedestrian;

[0035] The fusion module is used to fuse the first position and the second position of the pedestrian using an adaptive Kalman filter to obtain a final position of the pedestrian.

[0036] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the indoor positioning method based on WiFi and visual fusion as described above is implemented.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the indoor positioning method based on WiFi and vision fusion as described above is implemented.

[0038] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned indoor positioning methods based on WiFi and vision fusion.

[0039] The indoor positioning method and device based on WiFi and vision fusion provided by the present invention use the random forest algorithm in WiFi positioning to process WiFi signal strength data from multiple points, supplemented by regional restrictions, to enhance the adaptability to environmental changes and provide more accurate position estimation. Visual positioning integrates the YOLOv7 object detection algorithm and the DeepSORT object tracking algorithm to process image data collected by the camera for target recognition and tracking. The YOLOv7 algorithm performs well in target detection with high efficiency and precision, while the DeepSORT algorithm further improves the stability and continuity of target tracking. Finally, the results of WiFi positioning and visual positioning are combined through adaptive Kalman filtering technology, effectively reducing the uncertainty associated with each positioning technology, thereby providing more accurate positioning results. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is a flow chart of the indoor positioning method based on WiFi and vision fusion provided by the present invention;

[0042] Figure 2 This is a schematic diagram of the system framework of the indoor positioning method based on WiFi and vision fusion provided by the present invention;

[0043] Figure 3 Schematic diagram of the architecture of the coordinate transformation model in the indoor positioning method based on WiFi and vision fusion provided by the present invention;

[0044] Figure 4 It is a real picture of the indoor environment in the indoor positioning method based on WiFi and vision fusion provided by the present invention;

[0045] Figure 5It is a top-down plan view of the indoor environment in the indoor positioning method based on WiFi and vision fusion provided by the present invention;

[0046] Figure 6 Schematic diagram of the trajectory of three different positioning methods in the indoor positioning method based on WiFi and visual fusion provided by the present invention;

[0047] Figure 7 Schematic diagram of positioning CDF of different positioning methods for three paths in the indoor positioning method based on WiFi and visual fusion provided by the present invention;

[0048] Figure 8 Schematic diagram of the structure of the indoor positioning device based on WiFi and vision fusion provided by the present invention;

[0049] Figure 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0050] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0051] The following combination Figure 1 The present invention describes an indoor positioning method based on WiFi and vision fusion, comprising:

[0052] Step 101: predicting a first location of a pedestrian using a random forest algorithm based on received signal strengths of a target terminal of the pedestrian to multiple WiFi devices, wherein the random forest algorithm constrains a positioning area of ​​the target terminal.

[0053] Step 102: using YOLOv7 to identify the pedestrian in the video, and using Deep Simple Online and Realtime Tracking (Deep SORT) algorithm to track the pedestrian in the video to obtain a second position of the pedestrian;

[0054] Step 103: Using an adaptive Kalman filter, the first position and the second position of the pedestrian are fused to obtain a final position of the pedestrian.

[0055] The system framework diagram of this embodiment is as follows Figure 2As shown in the figure, it mainly includes three core modules: WiFi positioning module, visual positioning module and adaptive Kalman filter (FAKF) data fusion module.

[0056] After analyzing the shortcomings and deficiencies of existing technologies, this embodiment optimizes various technologies through specific improvements. First, regarding the WiFi positioning module, traditional methods are susceptible to path loss in complex environments. This embodiment introduces a random forest algorithm combined with regional constraints, effectively improving the system's adaptability to environmental changes and thus enhancing positioning accuracy. However, a major drawback of WiFi positioning is that it is difficult to maintain a high level of accuracy in environments with large areas and multiple obstacles.

[0057] Secondly, in terms of the visual positioning module, this embodiment takes into account the dependence of traditional visual positioning on ambient lighting and the high computational complexity. It adopts YOLOv7 target detection and Deep SORT target tracking algorithms, which not only improves processing speed but also optimizes performance under dynamic conditions. The Deep SORT algorithm is an extended version of the SORT algorithm, which enhances the ability to detect and re-identify targets under occlusion. However, in visual positioning, it may be difficult to accurately match feature points in the image over a large area, resulting in increased positioning deviation.

[0058] To address the shortcomings of both WiFi and visual positioning, this embodiment uses a fuzzy adaptive Kalman filter (FAKF) data fusion module to combine WiFi and visual positioning results, dynamically adjusting filter parameters and weights. This significantly reduces the uncertainty of each independent technology and improves the overall robustness and accuracy of the system.

[0059] These technical improvements not only enhance the performance of the positioning system, but also reduce costs, simplify the structure, and improve ease of use. They have broad application prospects and practical value.

[0060] Pedestrian detection and tracking are key components of vision-based indoor positioning systems. This embodiment uses the YOLOv7 object detection framework to identify pedestrians in video clips and combines it with the Deep SORT algorithm for tracking. YOLOv7 uses a single-stage detection strategy that divides the input image into a grid and predicts the presence of the target, the bounding box, and the class label of each grid cell. The model architecture utilizes advanced feature extraction networks such as DarkNet, ResNet, or EfficientNet, which are pre-trained to effectively enhance the capture of complex image features. In addition, feature fusion technology with Feature Pyramid Networks (FPN) is also used to achieve robust target detection across scales.

[0061] The YOLOv7 model uses a combination of positioning loss, classification loss, and target confidence loss to improve the prediction accuracy of target location and category. Non-maximum suppression (NMS) is implemented in the later stages of the detection process to process the detection results. NMS uses the intersection over union (IoU) ratio to measure the overlap, thereby eliminating redundant bounding boxes and ensuring the accuracy of the final detection results, as shown in formula (1):

[0062]

[0063] Deep SORT integrates a deep learning model that learns the epigenetic characteristics of the target to improve tracking accuracy in dynamic scenes. It uses Kalman filtering to predict the state of the target in subsequent frames, while the Hungarian algorithm uses the predicted state and observed data to perform data association, and then updates the state based on the association results.

[0064] This embodiment combines YOLOv7 with Deep SORT, effectively improving the accuracy of pedestrian detection and tracking in complex environments, thereby strengthening support for vision-based indoor positioning systems.

[0065] The Kalman filter is an effective data fusion technique widely used in signal processing and control systems. It establishes a dynamic model of the system and uses observational data to continuously update and predict the system state, thereby achieving an optimal estimate of the system state. However, despite its excellent performance in fusing different types of sensor data, its application in complex indoor environments remains challenging. Current research focuses on using the Kalman filter to fuse data from a few sensor types, such as inertial measurement units (IMUs) and GPS, but has not explored combining Wi-Fi signals with visual information for localization.

[0066] This embodiment uses the random forest algorithm in WiFi positioning to process WiFi signal strength data from multiple points, supplemented by regional restrictions, to enhance adaptability to environmental changes and provide more accurate position estimation. Visual positioning integrates the YOLOv7 object detection algorithm and the Deep SORT object tracking algorithm to process image data collected by the camera for target recognition and tracking. The YOLOv7 algorithm performs well in target detection, with high efficiency and accuracy, while the Deep SORT algorithm further improves the stability and continuity of target tracking. Finally, the adaptive Kalman filter technology is used to combine the results of WiFi positioning and visual positioning, effectively reducing the uncertainty associated with each positioning technology, thereby providing more accurate positioning results.

[0067] Based on the above embodiment, the method of predicting the first position of the pedestrian using a random forest algorithm based on the received signal strength of the pedestrian's target terminal to multiple WiFi devices in this embodiment includes:

[0068] Collect the signal strength received by the sample terminal to the WiFi device at multiple locations and establish an offline fingerprint database;

[0069] Predicting the location of the pedestrian based on the online received signal strength of multiple WiFi devices by the pedestrian's target terminal to obtain the pedestrian's current predicted point;

[0070] Searching the offline fingerprint database for a position within a preset offset range of the current prediction point and a received signal strength corresponding to the position;

[0071] Training the random forest algorithm using the position within a preset offset range of the current prediction point and the received signal strength corresponding to the position as a training data set;

[0072] The online received signal strength is input into the trained random forest algorithm to obtain the first position of the pedestrian.

[0073] This embodiment introduces a machine learning-based WiFi indoor positioning method that utilizes the Random Forest Algorithm and enhances location prediction capabilities through regional constraints. This method significantly improves the accuracy and efficiency of indoor positioning.

[0074] First, WiFi data is collected to build an offline fingerprint database. In the positioning phase, the system processes online received signal strength (RSS) data to perform location prediction.

[0075] During the initialization phase, the prediction point is set to the coordinate origin (0,0), and the offset is defined as a constraint on the search area. To reduce initial error, the system ignores the first online RSS data point and processes subsequent RSS data sequentially. For each data point, the algorithm finds the RSS value and corresponding location data within the offset range of the current prediction point from the offline dataset. These data points constitute the training dataset for that time step.

[0076] Online RSS data is fed into a pre-trained regression model, enabling the algorithm to predict the current location. Each prediction is then used as the starting point for the next time step, ensuring the continuity and stability of the prediction. Furthermore, to maintain the real-time accuracy of the training dataset, the algorithm updates the RSS and location information within the restricted area after each prediction.

[0077] Based on the above embodiment, the random forest algorithm described in this embodiment uses mean square error, absolute error or Poisson deviation when calculating node impurity.

[0078] The random forest algorithm with region restriction proposed in this embodiment uses several measurement methods to calculate node impurity, including mean square error (MSE), mean absolute error (MAE), and Poisson deviation. The formula is as follows:

[0079]

[0080]

[0081] Among them, N is the total number of samples, y i is the observed count data, Count data for model predictions.

[0082] The impurity reduction of a node (weighted impurity reduction) is calculated as follows:

[0083]

[0084] Among them, N t is the number of samples of the current node, N tL is the number of samples of the left node, N tR is the number of samples of the right node, Impurity is the impurity of the current node, Impurity L Impurity of the left child node, R is the impurity of the right child node.

[0085] Positioning accuracy is evaluated by measuring the Euclidean distance between the predicted location and the actual location, thereby verifying the effectiveness of the proposed method. The WiFi indoor positioning algorithm introduced in this embodiment has several advantages. By dynamically adjusting the training data set, the algorithm can quickly adapt to environmental changes, thereby improving positioning accuracy. Depending on the input parameters, the algorithm can select either the K-Nearest Neighbors (KNN) or random forest regression model, thereby improving the adaptability and flexibility of the system. By accurately integrating online and offline RSS data and selecting appropriate training data within a restricted area, this method can achieve high-precision indoor positioning in complex environments.

[0086] Based on the above embodiment, this embodiment further includes, before using the Deep SORT algorithm to track the pedestrian in the video to obtain the second position of the pedestrian:

[0087] Use the external reference matrix to transform the points in the world coordinate system to the camera coordinate system;

[0088] Use the internal reference matrix to project the points in the camera coordinate system onto the imaging plane to obtain pixel coordinates;

[0089] The pixel coordinates acquired by the camera are converted to world coordinates according to the conversion relationship between the world coordinate system and the camera coordinate system, and the conversion relationship between the camera coordinate system and the imaging plane.

[0090] In most indoor surveillance environments, monocular cameras are widely used. However, monocular cameras have difficulty acquiring target depth information, and the target pixel coordinates they acquire cannot directly correspond to the target's actual spatial position. Therefore, this embodiment aims to achieve the conversion from pixel coordinates acquired by a monocular camera to world coordinates (describing the target's actual spatial position). Based on the camera imaging principle, a checkerboard-based camera calibration method is used to achieve the "pixel to world" coordinate conversion.

[0091] The specific process of coordinate transformation includes the following two steps: first, the point in the world coordinate system is transformed into the camera coordinate system using the external reference matrix; second, the point in the camera coordinate system is projected onto the imaging plane using the internal reference matrix to obtain the pixel coordinates.

[0092] Based on the above embodiment, in this embodiment, the pixel coordinates obtained by the camera are converted to world coordinates using the following formula:

[0093]

[0094] The internal reference matrix (IntrinsicMatrix) includes the focal length f, pixel size d x and d y (the number of pixels per unit length), and the optical center coordinates u0 and v0 of the imaging coordinate system (indicating the position of the center point of the imaging plane in the pixel coordinate system), the external reference matrix includes the rotation matrix R and the translation vector T, (x w ,y w ,z w ) is the world coordinate in the world coordinate system, (u, v) is the pixel coordinate on the imaging plane, Z c The depth or distance of the object in the camera coordinate system.

[0095] Equation (6) describes how to use the camera's internal and external references to transform a point (x w ,y w ,z w ) is converted to pixel coordinates (u, v) on the imaging plane.

[0096] The internal reference matrix reflects the properties of the camera itself. Different cameras have different internal reference matrices. The external reference matrix contains the rotation matrix R, which describes the rotation transformation from the world coordinate system to the camera coordinate system, and the translation vector T, which describes the translation transformation from the world coordinate system to the camera coordinate system. The parameters in the equation can be obtained through camera calibration. The architecture of the coordinate transformation model is as follows: Figure 3 Through the above steps, the conversion from pixel coordinates to world coordinates can be successfully achieved, providing theoretical support for indoor positioning based on monocular cameras.

[0097] On the basis of the above embodiments, the method of using an adaptive Kalman filter to fuse the first position and the second position of the pedestrian to obtain the final position of the pedestrian in this embodiment includes:

[0098] At each time step, the Kalman filter performs a prediction step, estimating the state and covariance matrix for the next time step;

[0099] Updating the Kalman gain using the covariance matrix, and updating the state and the covariance matrix as a whole using the updated Kalman gain and the measurement matrices of the Wi-Fi device and the camera;

[0100] determining residuals between the state and a first position and a second position measured by the Wi-Fi device and the camera, respectively, and updating measurement weights of the Wi-Fi device and the camera when no outliers are determined based on the residuals;

[0101] The final position of the pedestrian is determined according to the measurement weights of the Wi-Fi device and the camera.

[0102] The adaptive Kalman filter data fusion module consists of four main steps: prediction, covariance matrix update, adaptive adjustment, outlier detection, and measurement data fusion. By utilizing the adaptive Kalman filter, the system effectively reduces the uncertainty associated with each positioning technique, thereby providing more accurate positioning results.

[0103] This example proposes a modified adaptive Kalman filter (FAKF) algorithm, a data-layer fusion method designed to significantly reduce positioning errors caused by sensor drift. This algorithm uses an observability-based data fusion strategy to improve positioning accuracy. Specifically, it integrates the receiver's wireless signal strength (W-data) with the position coordinates of people in the video frame (V-data), establishing a potential correspondence between wireless signal strength and visual signals for each target.

[0104] The FAKF algorithm consists of four main parts: information reset, time update, measurement update, and information fusion between wireless and visual signals. The algorithm uses state errors to represent the positions of W and V data, thereby promoting the effective fusion of W and V data.

[0105] At each time step, the Kalman filter performs a look-ahead step to estimate the state and covariance for the next time step:

[0106]

[0107] in is the prior state estimate at time t; F t is the state transition matrix; is the prediction covariance matrix; Q t is the process noise covariance matrix, representing the uncertainty of the model; P t-1 is the posterior state covariance matrix at the previous time.

[0108] The new observation data is used to update the state estimate. In this step, the state estimate and covariance matrix are updated using the Kalman gain and the measurement matrices of Wi-Fi and camera:

[0109]

[0110] Among them, K t is the Kalman gain, which is used to weigh the uncertainty of prediction and measurement; H t is the measurement matrix, which transforms the state space into the measurement space; R t is the measurement noise covariance matrix, representing the uncertainty in the observation; z t is the actual measurement value; I is the identity matrix, with the same dimension as P t same.

[0111] In addition, an adaptive process is introduced to uniformly distribute the number of basic AKFs at the beginning. In this process, outlier detection and weight adjustment are achieved by calculating the residual between the predicted state and the actual measurement value:

[0112]

[0113] Then calculate the Euclidean modulus of the residual vector to quantify the size of the residual:

[0114]

[0115] Set an observability threshold observability_factor to detect whether the residual is beyond the normal range, so as to determine whether there are outliers or model mismatch. wifi||>observability_factor or ||e camera When ||>observability_factor, it indicates that an outlier has occurred. In this case, it is necessary to adjust the process noise covariance Q to increase the adaptability of the model and reduce the measurement noise covariance R to enhance the trust of the observation, thereby improving the robustness and stability of the filter.

[0116] Q←a1·Q (16)

[0117] R←a2·R (17)

[0118] Where a1 and a2 are adjustment factors.

[0119] When the residual is within the normal range, the size of the residual will update the sensor data weight, reflecting the contribution of each sensor to the state estimation. If the residual value exceeds a certain threshold, the weight will be reduced, indicating that the confidence in the measurement result is reduced.

[0120]

[0121] The weights are then normalized based on the sum of all weights to ensure that they sum to 1.

[0122]

[0123] The final fused position is obtained based on the measurement weights of the Wi-Fi device and the camera.

[0124] Based on the above embodiment, the weight factor of the adaptive Kalman filter is measured by the following formula:

[0125]

[0126] Among them, O(t0,t f ) represents the time from time t0 to t f The weight factor between is the state transition matrix F t At the power of time step t-t0, Tr represents the trace of the matrix, that is, the sum of the diagonal elements of the matrix, H wifi represents the measurement matrix of the WiFi device, H camera represents the measurement matrix of the camera.

[0127] In order to improve the robustness of perception capability, this embodiment also proposes an improved AKF observability method to measure the weight factor of the filter, as shown in formula (22).

[0128] This adaptive adjustment mechanism allows the filter to dynamically adjust parameters based on real-time data, achieving optimal filtering performance under various environmental conditions. This approach is particularly suitable for situations where sensor data is uncertain or dynamically changing, significantly improving the accuracy and robustness of the positioning system.

[0129] The experiments were conducted in a complex indoor laboratory environment, e.g. Figure 4 (Realistic picture) and Figure 5 (Top view) The laboratory is 17.5 meters long, 8 meters wide, and 3.5 meters high, with numerous built-in lab tables and large cabinets. This complexity poses challenges for pedestrian positioning due to potential WiFi signal interference and visual obstructions. During the experiment, six routers (access points, APs) and a USB camera were strategically placed within the observation area to optimize coverage and minimize blind spots.

[0130] For WiFi signal acquisition, custom Android software developed in Android Studio was used. This software captures the received signal strength indicator (RSSI) of WiFi signals, providing a quantitative measurement of signal strength at specific locations. Through extensive data collection, a comprehensive signal fingerprint database was compiled, with no fewer than 3,000 data entries per acquisition point to ensure robustness against random environmental variations.

[0131] A video camera, strategically placed in the lab environment and connected to a laptop computer that processes the input data, facilitates visual data collection. The area of ​​interest is continuously monitored, capturing video frames and providing pedestrian movement data. Using established extrinsic and intrinsic calibration parameters (including focal length, lens distortion, and scene rotation), the system accurately converts pixel coordinates to world coordinates, thereby improving the accuracy of pedestrian localization.

[0132] This example evaluated the performance of various positioning methods on different experimental paths. Participants walked using their phones, which continuously scanned for Wi-Fi signals and performed Wi-Fi-based positioning. Simultaneously, cameras captured visual data for frame-by-frame visual positioning analysis. The proposed FAKF algorithm integrates Wi-Fi and visual information to achieve fusion indoor positioning, aiming to improve positioning accuracy and reliability. Figure 6 is the trajectory diagram of the three test paths, and Table 1 is the summary table of the corresponding error indicators.

[0133] Figure 6 (a) is the WiFi positioning trajectory diagram of path 1; Figure 6 (b) is the visual positioning trajectory diagram of path 1; Figure 6 (c) in the figure is the fusion positioning trajectory diagram of path 1; Figure 6(d) is the WiFi positioning trajectory diagram of path 2; Figure 6 (e) in the figure is the visual positioning trajectory diagram of path 2; Figure 6 (f) in the figure is the fusion positioning trajectory diagram of path 2; Figure 6 (g) in the figure is the WiFi positioning trajectory of path 3; Figure 6 (h) is the visual positioning trajectory diagram of path 3; Figure 6 (i) in the figure is the fusion positioning trajectory diagram of path 3.

[0134] Table 1 Positioning error results

[0135]

[0136] Experimental results show that the fusion positioning method using the FAKF algorithm exhibits superior positioning performance under different paths and significantly reduces the positioning error compared to single sensor data.

[0137] Experimental results show that the fusion positioning method using the FAKF algorithm shows excellent performance on different paths, and the positioning error is significantly reduced compared with single sensor data. Figure 6 As shown in (a), (b), and (c) of Figure 1, the WiFi positioning trajectory in path 1 is relatively rough, and the visual positioning trajectory exhibits significant deviations. In contrast, the fused positioning trajectory is smoother and closely follows the actual path. Specifically, the root mean square error (RMSE) for WiFi positioning is 1.309 meters, while the RMSE for visual positioning is 1.793 meters. The RMSE for fused positioning is reduced to 0.884 meters, and the mean absolute error (MAE) is only 0.683 meters, demonstrating a significant improvement in positioning accuracy.

[0138] For path 2 ( Figure 6 In (d), (e), and (f), the WiFi positioning trajectory also has significant errors. Although the visual positioning results improved due to the reduced walking range, they were still unsatisfactory. After applying fusion positioning, the trajectory improved significantly. The RMSE for WiFi positioning was 1.464 meters, while the RMSE for visual positioning was 1.049 meters. The RMSE for fusion positioning dropped to 0.896 meters, with a MAE of 0.590 meters. This demonstrates that the fusion method is effective, but there is still room for improvement.

[0139] In path 3 ( Figure 6In Figures (g), (h), and (i), both WiFi and visual positioning exhibit varying degrees of deviation. The data shows an RMSE of 1.439 meters for WiFi positioning and 1.198 meters for visual positioning. In contrast, fusion positioning achieves an RMSE of 1.059 meters and a MAE of 0.834 meters. Although the error of fusion positioning increases, it still significantly improves compared to using a single sensor.

[0140] This embodiment also examines the positioning error rates of different methods on different paths, such as Figure 7 The percentile error data of the three paths are shown in Tables 2 to 4. Figure 7 (a) in the figure is the CDF graph of path 1. Figure 7 (b) in the figure is the error of different percentiles of path 1. Figure 7 (c) in the figure is the CDF graph of path 2. Figure 7 (d) in the figure is the different percentile errors of path 2. Figure 7 (e) in the figure is the CDF graph of path 3. Figure 7 (f) in the figure is the error of different percentiles of path 3.

[0141] Table 2 Percentile error data table of path 1

[0142]

[0143] Table 3 Percentile error data table of path 2

[0144]

[0145]

[0146] Table 4 Percentile error data table of path 3

[0147]

[0148] In path 1, the WiFi positioning error was 1.17 meters at the 50th percentile and increased to 1.81 meters at the 90th percentile. The visual positioning error increased significantly, from 2.42 meters at the 50th percentile to 4.32 meters at the 90th percentile, demonstrating greater instability. In contrast, the fusion positioning technique showed more stable performance, with an error of 1.01 meters at the 50th percentile and a smaller increase to 1.91 meters at the 90th percentile. This demonstrates that fusion positioning provides more accurate results in most situations.

[0149] In path 2, the WiFi positioning error increased significantly, from 0.93 meters at the 50th percentile to 2.78 meters at the 90th percentile. Visual positioning error rose from 1.68 meters at the 50th percentile to 2.83 meters at the 90th percentile, similar to Wi-Fi positioning, but with a higher median error. However, the error for fused positioning remained low, with a 50th percentile error of only 0.47 meters, and its 90th percentile error was lower than that of the individual technologies, indicating that fused positioning remains effective in most cases.

[0150] In path 3, the WiFi positioning error increased from 1.78 meters at the 50th percentile to 2.93 meters at the 90th percentile, showing significant fluctuations. The visual positioning error rose from 1.68 meters at the 50th percentile to 3.01 meters at the 90th percentile, with similar fluctuations to WiFi positioning. The fusion positioning error increased from 1.05 meters at the 50th percentile to 2.46 meters at the 90th percentile. Despite the increase in high-percentile errors, the overall stability was better.

[0151] Data analysis of the three paths shows that the fused localization approach generally outperforms either Wi-Fi or visual localization techniques alone in terms of median and high percentile errors. These results further confirm the effectiveness of fused localization in reducing errors and improving accuracy, particularly in complex indoor environments. These experimental results support the potential application of fused localization in future indoor navigation systems, particularly in scenarios requiring high accuracy and stability.

[0152] The indoor positioning device based on WiFi and visual fusion provided by the present invention is described below. The indoor positioning device based on WiFi and visual fusion described below and the indoor positioning method based on WiFi and visual fusion described above can refer to each other.

[0153] like Figure 8 As shown, the device includes a first positioning module 801, a second positioning module 802 and a fusion module 803, wherein:

[0154] The first positioning module 801 is configured to predict a first position of the pedestrian based on the received signal strength of the pedestrian's target terminal to multiple WiFi devices using a random forest algorithm, wherein the random forest algorithm constrains the positioning area of ​​the target terminal;

[0155] The second positioning module 802 is used to identify the pedestrian in the video using YOLOv7 and track the pedestrian in the video using the Deep SORT algorithm to obtain a second position of the pedestrian;

[0156] The fusion module 803 is used to fuse the first position and the second position of the pedestrian using an adaptive Kalman filter to obtain the final position of the pedestrian.

[0157] This embodiment uses the random forest algorithm in WiFi positioning to process WiFi signal strength data from multiple points, supplemented by regional restrictions, to enhance adaptability to environmental changes and provide more accurate position estimation. Visual positioning integrates the YOLOv7 object detection algorithm and the Deep SORT object tracking algorithm to process image data collected by the camera for target recognition and tracking. The YOLOv7 algorithm performs well in target detection, with high efficiency and accuracy, while the Deep SORT algorithm further improves the stability and continuity of target tracking. Finally, the adaptive Kalman filter technology is used to combine the results of WiFi positioning and visual positioning, effectively reducing the uncertainty associated with each positioning technology, thereby providing more accurate positioning results.

[0158] Figure 9 An example of a physical structure diagram of an electronic device is shown below. Figure 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other via the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute an indoor positioning method based on Wi-Fi and visual fusion, the method comprising: predicting a first position of the pedestrian using a random forest algorithm based on the received signal strength of a pedestrian's target terminal to multiple WiFi devices, the random forest algorithm constraining the positioning area of ​​the target terminal; identifying the pedestrian in the video using YOLOv7, and tracking the pedestrian in the video using the Deep SORT algorithm to obtain a second position of the pedestrian; and fusing the first and second positions of the pedestrian using an adaptive Kalman filter to obtain a final position of the pedestrian.

[0159] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0160] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the indoor positioning method based on Wi-Fi and visual fusion provided by the above methods. The method includes: using a random forest algorithm to predict the first position of the pedestrian based on the received signal strength of the pedestrian's target terminal to multiple WiFi devices, and the random forest algorithm constrains the positioning area of ​​the target terminal; using YOLOv7 to identify the pedestrian in the video, and using the Deep SORT algorithm to track the pedestrian in the video to obtain the second position of the pedestrian; using an adaptive Kalman filter to fuse the first position and second position of the pedestrian to obtain the final position of the pedestrian.

[0161] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the indoor positioning method based on Wi-Fi and visual fusion provided by the above-mentioned methods, the method comprising: predicting the first position of the pedestrian based on the received signal strength of the pedestrian's target terminal to multiple WiFi devices using a random forest algorithm, and constraining the positioning area of ​​the target terminal using the random forest algorithm; identifying the pedestrian in the video using YOLOv7, and tracking the pedestrian in the video using the Deep SORT algorithm to obtain the second position of the pedestrian; and fusing the first position and the second position of the pedestrian using an adaptive Kalman filter to obtain the final position of the pedestrian.

[0162] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0163] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An indoor positioning method based on WiFi and vision fusion, characterized in that: include: Predicting a first location of the pedestrian using a random forest algorithm based on received signal strengths of a target terminal of the pedestrian to multiple WiFi devices, wherein the random forest algorithm constrains a positioning area of ​​the target terminal; Using YOLOv7 to identify the pedestrian in the video, and using the Deep SORT algorithm to track the pedestrian in the video to obtain a second position of the pedestrian; fusing the first position and the second position of the pedestrian using an adaptive Kalman filter to obtain a final position of the pedestrian; The method of fusing the first position and the second position of the pedestrian using an adaptive Kalman filter to obtain a final position of the pedestrian includes: At each time step, the Kalman filter performs a prediction step, estimating the state and covariance matrix for the next time step; Using the covariance matrix to update the Kalman gain, and using the updated Kalman gain and the measurement matrices of the Wi-Fi device and the camera to update the state and the covariance matrix as a whole; determining residuals between the state and a first position and a second position measured by the Wi-Fi device and the camera, respectively, and updating measurement weights of the Wi-Fi device and the camera when no outliers are determined based on the residuals; The final position of the pedestrian is determined according to the measurement weights of the Wi-Fi device and the camera.

2. The indoor positioning method based on WiFi and vision fusion according to claim 1, characterized in that: The predicting the first position of the pedestrian using a random forest algorithm based on the received signal strengths of the pedestrian's target terminal to the multiple WiFi devices includes: Collect the signal strength received by the sample terminal to the WiFi device at multiple locations and establish an offline fingerprint database; Predicting the location of the pedestrian based on the online received signal strength of multiple WiFi devices by the pedestrian's target terminal to obtain the pedestrian's current predicted point; Searching the offline fingerprint database for a position within a preset offset range of the current prediction point and a received signal strength corresponding to the position; Training the random forest algorithm using the position within a preset offset range of the current prediction point and the received signal strength corresponding to the position as a training data set; The online received signal strength is input into the trained random forest algorithm to obtain the first position of the pedestrian.

3. The indoor positioning method based on WiFi and vision fusion according to claim 1, characterized in that: The random forest algorithm uses mean square error, absolute error or Poisson deviation when calculating node impurity.

4. The indoor positioning method based on WiFi and vision fusion according to claim 1, characterized in that: Before using the Deep SORT algorithm to track the pedestrian in the video to obtain the second position of the pedestrian, the method further includes: Use the external reference matrix to transform the points in the world coordinate system to the camera coordinate system; Use the internal reference matrix to project the points in the camera coordinate system onto the imaging plane to obtain pixel coordinates; The pixel coordinates acquired by the camera are converted to world coordinates according to the conversion relationship between the world coordinate system and the camera coordinate system, and the conversion relationship between the camera coordinate system and the imaging plane.

5. The indoor positioning method based on WiFi and vision fusion according to claim 4, characterized in that: The pixel coordinates obtained by the camera are converted to world coordinates using the following formula: The internal reference matrix includes the focal length f, the pixel size d x and d y , and the optical center coordinates u0 and v0 of the imaging coordinate system, the external reference matrix contains the rotation matrix R and the translation vector T, (x w ,y w ,z w ) is the world coordinate in the world coordinate system, (u, v) is the pixel coordinate on the imaging plane, Z c The depth or distance of the object in the camera coordinate system.

6. The indoor positioning method based on WiFi and vision fusion according to claim 1, characterized in that: The weight factor of the adaptive Kalman filter is measured by the following formula: Among them, O(t0,t f ) represents the time from time t0 to t f The weight factor between is the state transition matrix F t At the power of time step t-t0, Tr represents the trace of the matrix, that is, the sum of the diagonal elements of the matrix, H wifi represents the measurement matrix of the WiFi device, H camera represents the measurement matrix of the camera.

7. An indoor positioning device based on WiFi and vision fusion, characterized in that: The indoor positioning method based on WiFi and vision fusion applied to any one of claims 1 to 6 comprises: A first positioning module is configured to predict a first position of the pedestrian based on the received signal strength of a target terminal of the pedestrian to multiple WiFi devices using a random forest algorithm, wherein the random forest algorithm constrains a positioning area of ​​the target terminal; a second positioning module, configured to identify the pedestrian in the video using YOLOv7 and track the pedestrian in the video using a Deep SORT algorithm to obtain a second position of the pedestrian; The fusion module is used to fuse the first position and the second position of the pedestrian using an adaptive Kalman filter to obtain a final position of the pedestrian.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the indoor positioning method based on WiFi and vision fusion as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the indoor positioning method based on WiFi and vision fusion as described in any one of claims 1 to 6 is implemented.

Images