Multimodal indoor positioning method based on heterogeneous feature fusion

Abstract

A multi-modal indoor positioning method based on heterogeneous feature fusion comprises the following steps: in an offline stage, acquiring wireless signal strength RSSI data and a visible light image as a sample set; training the constructed multi-scale deep fusion network based on channel exchange, wherein the multi-scale deep fusion network comprises a preprocessing unit, a data alignment layer, a shared encoder, a channel exchange layer, a multi-modal fusion unit and a multi-layer perceptron layer; and an on-line test stage: predicting an indoor accurate position through the trained multi-scale deep fusion network according to a current RSSI measurement value and a visible light image shot in real time. According to the method, the correlation among multi-modal data is utilized, a mixed shared feature extraction network, a channel-based fusion network structure and a spatial interpolation and channel exchange algorithm are adopted, and heterogeneous and multi-time-scale data are fully fused. The performance of the model is improved, and the calculation complexity is reduced, so that the positioning precision is improved.

Description

Technical Field

[0001] The present invention relates to a technology in the field of wireless positioning, in particular to a multi-modal indoor positioning method based on heterogeneous feature fusion. Background Art

[0002] When processing multimodal data, the existing fusion-based fingerprint indoor positioning technology will have a time scale mismatch problem due to the different sampling rates between data, which will lead to insufficient multimodal feature extraction and fusion; due to the different sampling data structures of multimodal data sources, there will be a problem of heterogeneous data misalignment, which makes it difficult to fuse multimodal data. Traditional fingerprint-based indoor positioning solutions use different feature extraction methods to process heterogeneous data sources, which will lead to the problem of too large a network model. Summary of the invention

[0003] Aiming at the problem that the existing indoor positioning technology has complex calculation features and insufficient feature fusion, which leads to low model positioning accuracy, the present invention proposes a multimodal indoor positioning method based on heterogeneous feature fusion. The method utilizes the correlation between multimodal data, adopts a hybrid shared feature extraction network and a channel-based fusion network structure, and a spatial interpolation and channel exchange algorithm to fully fuse heterogeneous and multi-time scale data. The performance of the model is improved, and the calculation complexity is reduced, thereby improving positioning accuracy.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a multimodal indoor positioning method based on heterogeneous feature fusion. In an offline stage, wireless signal strength RSSI data and visible light images are collected as sample sets, and a constructed multiscale deep fusion network based on channel exchange, which includes a preprocessing unit, a data alignment layer, a shared encoder, a channel exchange layer, a multimodal fusion unit and a multi-layer perception machine layer, is trained. In an online testing stage, the trained multiscale deep fusion network is used to predict the precise indoor position according to the current RSSI measurement value and the visible light image taken in real time. Technical Effects

[0006] The present invention adopts a multimodal heterogeneous data alignment method based on interpolation and a multimodal feature fusion network based on channel exchange; compared with the prior art, the present invention can effectively cope with indoor positioning tasks in complex scenarios, has higher positioning accuracy, and greatly reduces the model size of the network, reduces the computational complexity, and is easier to apply in actual industrial scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 This is a schematic diagram of the deep fusion network of the present invention;

[0008] Figure 2 This is a schematic diagram of the data alignment layer;

[0009] Figure 3 Schematic diagram of shared encoder and channel switching layer;

[0010] Figure 4 Schematic diagram of experimental environment of the embodiment;

[0011] Figure 5 Schematic diagram of CDF of positioning distance error of different algorithms. DETAILED DESCRIPTION

[0012] This embodiment relates to a multimodal indoor positioning method based on heterogeneous feature fusion, including:

[0013] Step 1: Construct a sample set, including:

[0014] 1.1 Construct a training set consisting of wireless RSSI and images, where each sample is collected at a reference point, and 4 visual images are collected in four directions at a fixed height of 1.5 meters; at the same time, the RSSI data is sampled 10 times within a fixed time. The sampling rate of WiFi and Bluetooth is 100Hz. The number of 2.4GHz WiFi and Bluetooth APs is 22 and 14 respectively. The total number of RSSI and image data is 11,000.

[0015] 1.2 Preprocess the image data obtained in step 1.1: crop all images to ensure consistent size, use a Gaussian filter to reduce noise, and subtract each image from the mean of all images in the dataset to remove the common parts of the images and highlight individual differences. All images are then preprocessed to zero mean and the same pixel size with random center points.

[0016] Step 2: Construct Figure 1 and Figure 2 The multi-scale deep fusion network shown in the figure, wherein: the preprocessing unit aligns the multimodal data for preprocessing respectively, the data alignment layer aligns the dimensions of the preprocessed RSSI data and the image to obtain multimodal data with a consistent data dimension; the shared encoder uses an encoder with shared parameters to extract multimodal features and maps the multimodal features to a unified space; the channel exchange layer exchanges features between channels according to the features of each layer in the shared encoder according to the importance of the features, and uses the consistency and complementarity of the models to effectively fuse the multimodal data; the multi-layer perception machine layer classifies according to the multimodal fusion features to finally obtain the positioning result.

[0017] The alignment specifically includes:

[0018] a) To align multimodal data, namely wireless RSSI data and visible light image, the original RSSI data range is changed from [0,100]dBm is adjusted to [0,255], specifically: Where: L is the position to be located, represents the adjusted RSSI data, R(L) represents the original RSSI data, and R min (L) represents the minimum value of the original RSSI data, R max (L) indicates the maximum value of the original RSSI data, I max Indicates the maximum pixel value of the visible light image.

[0019] b) The Wi-Fi and Bluetooth data are merged in a certain proportion to solve the problem that the wireless signal is unavailable at a specific AP access point. Specifically, the two data are directly spliced, that is, Where: L is the location to be located, R(L) is the fused RSSI data, R B (L), R W (L) represents the RSSI data of Bluetooth and WiFi respectively.

[0020] c) Expand the dimension of RSSI data to solve the problem of large dimensional difference between wireless data and image data. Specifically: The constraints are in: The extended RSSI data obtained after interpolation, L0 is the new position coordinate, R(L i ) is the known RSSI data, L i are the known coordinates of several nearby positions, α i is the weight obtained by Kriging interpolation, which is obtained through matrix calculation. d represents the distance between adjacent reference points, and N represents the number of adjacent reference points.

[0021] d) Initialize the location fingerprint database to store the RSSI data and image data of Wi-Fi from NR equally spaced areas, specifically: DB = {(m, (R(L), I(L)), m∈[1,…, N R ]), where: m is the serial number of the reference point location, R(L) is the Ns RSSI data from Nap AP nodes measured in a unit time, I(L) is the image data of the location, L is the location coordinate, N R is the number of reference points.

[0022] The image data are images collected in four directions at equal heights within a fixed time.

[0023] Preferably, since it is difficult to obtain the wireless signal strength of the entire grid area in practical applications, the reference point L obtained by the sensor of the mobile device is m RSSI data H(L m ,T0), that is, the center position of each region The RSSI data of the area is regarded as the RSSI data of the area.

[0024] like Figure 2 As shown, the data alignment layer includes: a range adjustment module, a data fusion module and a data alignment module, wherein: the range adjustment module performs data preprocessing according to the input original multimodal data to make the data ranges of the two modes consistent; the data fusion module performs RSSI data fusion of Bluetooth and WiFi according to the preprocessed RSSI data of the previous level to obtain fused RSSI data; the data alignment module performs RSSI and image data dimension alignment processing according to the RSSI data to obtain aligned RSSI data.

[0025] like Figure 3 As shown, the shared encoder is composed of several levels of cascaded encoders, each encoder includes: a sharing module and a dimensionality reduction module, wherein: the sharing module performs feature extraction and parameter sharing based on multimodal data to obtain multimodal features in a unified space, and the dimensionality reduction module performs feature dimensionality reduction processing based on the multimodal features to obtain reduced dimensionality multimodal features.

[0026] The channel exchange layer includes: a batch normalization (BN) layer and a channel exchange module, wherein: the normalization module performs feature normalization processing according to the characteristics of each level, which is used to eliminate covariate offset and improve the generalization ability of the model to prevent the model from overfitting. The channel exchange module performs channel exchange processing according to the characteristics of each level, fully utilizes the correlation between modes, and fuses features of multiple time scales.

[0027] The channel exchange means that when one mode of any channel has little effect on the final prediction, the characteristics of the channel are replaced by the mean of the characteristics of other modes, specifically: The constraints are Where: μ l，c and σ l，c is the mean and standard deviation; γ l，c and β l，c are the scaling factor and offset used for training respectively; ε is a small constant to avoid division by zero; F' l，c is the feature of mode 1 in channel c; L0 is the number of modes; for this task, L0 is equal to 2; γ l，c It has an impact on the multi-scale output of the channel; when γ l，c →0, feature F' l，cWill lose influence on this mission.

[0028] Step 3: Construct a training set and train the multi-scale deep fusion network obtained in step 2. Specifically, a fully supervised training method is used, the training round dimension is 100, a dynamic learning rate strategy is used, the optimization method uses the Adam optimization algorithm, and the training loss function is the cross entropy loss function. The dropout strategy is used to solve the problem of model overfitting.

[0029] Step 4: In the online testing phase, the real-time RSSI and image are uploaded to a server equipped with the multi-scale deep fusion network based on channel switching obtained after training in step 3, and the corresponding location information is predicted, specifically: in: is the predicted position, f() is the positioning model, They are the pre-processed RSSI data of Bluetooth and WiFi respectively. is the preprocessed image data.

[0030] After specific practical experiments, Figure 4 In the corridor environment shown, the distance between two adjacent points is 0.6m. A total of 220 points are used as reference points for model training, and the other 130 points are used as test points for testing. At each point, 4 visual images are collected at a fixed position with a height of 1.5m in all four directions. RSSI data is sampled 10 times within a fixed time. The number of 2.4GHz Wi-Fi and Bluetooth APs is 22 and 14 respectively. The total amount of RSSI and image data is 11,000. The data training process is carried out on NVIDIA's local server, with a 1080Ti GPU and a Pytorch platform.

[0031] Before the experiment, the image data needs to be preprocessed. Since the images have different sizes, all images are cropped to ensure the same size. The images are cropped to a size of Nr×Nr. Then, data augmentation is used to reduce redundant information and noise. Specifically, a Gaussian filter is used to reduce noise, and then each image is subtracted from the mean of all images in the dataset to remove the common parts of the image and highlight individual differences. Then, all images are preprocessed to have zero mean and pixels with random center points.

[0032] The experimental results are as follows Figure 5As shown, the average positioning error of the deep fusion network of the present invention is 0.22m, which is better than 0.39m of the fusion method HCF based on hybrid cascade, 0.88m of fingerprint fusion LTS, 0.56m of multi-sensor fusion JVWL, and 0.91m based on Transformer fusion Transfuser. In the experimental area of ​​similar size, the proposed invention has a positioning accuracy that is better than other inventions. After analysis, we found that more than 90% of the positioning errors of the present invention are around 1m, while correspondingly, the 1m positioning accuracy of the baseline is only about 83%. The positioning accuracy of the present invention is higher than that of the baseline because the multimodal features are fully fused, while the fusion of the baseline is only at the data level or the fusion is insufficient and introduces noise. The fingerprint fusion LTS method, the fusion method only uses direct splicing. Although this method is simple, it will introduce a lot of noise and reduce the positioning accuracy. In contrast, the present invention adopts a multimodal fusion method of channel exchange, which effectively suppresses noise, can cope with complex scenes, and has higher positioning accuracy.

[0033] As shown in Table 1, Transfuser takes the longest computation time because of the complex structure of Transformer. HCF uses two different encoders to extract multimodal features, which increases the computation time. In contrast, because the method uses a shared network structure and shares parameters between the wireless encoder and the image encoder, the method has high accuracy and low computational complexity, making it a reliable solution for mobile device positioning under resource-limited conditions.

[0034] Table 1 Comparison of total running time of different methods

[0035] Unlike the existing machine learning algorithms that are unsupervised and do not have the ability to automatically extract features, the online test time is linearly related to the fingerprint dimension and the size of the offline fingerprint database. When the fingerprint database is large, the online test speed is very slow. Unlike MPEG online updates, which take a long time and even exhaust the computer's memory. The present invention uses a deep learning method, and the size of the offline database will only affect the training time of offline training. When the network structure, that is, the number of neurons in each layer of the network layer is determined, the online test time is determined. The database is updated in the offline stage and does not take up online testing time. In the online testing stage, you only need to input the real-time RSSI and image information into the trained neural network to obtain the user's real-time location. The calculation complexity is low and the calculation speed is fast, and the user can obtain seamless positioning services.

[0036] In summary, compared with the prior art, the present invention has higher accuracy and better system robustness while significantly reducing the computational complexity. The present invention only requires a number of Wi-Fi and Bluetooth nodes to be deployed, and the model structure of channel switching can cope with a variety of complex scenarios and can be applied to a large number of current residential, office and other scenarios.

[0037] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A multimodal indoor positioning method based on heterogeneous feature fusion, characterized in that: In the offline stage, by collecting wireless signal strength RSSI data and visible light images as sample sets, the constructed multi-scale deep fusion network based on channel exchange, which includes a preprocessing unit, a data alignment layer, a shared encoder, a channel exchange layer, a multimodal fusion unit and a multi-layer perceptron layer, is trained; During the online testing phase, the trained multi-scale deep fusion network predicts the precise indoor location based on the current RSSI measurement value and the visible light image taken in real time.

2. The multimodal indoor positioning method based on heterogeneous feature fusion according to claim 1 is characterized in that: The sample set is constructed in the following way: 1.1 Construct a training set consisting of wireless RSSI and images, where each sample is collected at a reference point, and several visual images are collected at a fixed height of 1.5-2 meters in multiple directions; RSSI data is sampled multiple times within a fixed time; 1.2 Preprocess the image data obtained in step 1.1: crop all images to ensure consistent size, use a Gaussian filter to reduce noise, and subtract each image from the mean of all images in the dataset to remove the common parts of the images and highlight individual differences; then preprocess all images to zero mean and the same pixel size with random center points.

3. The multimodal indoor positioning method based on heterogeneous feature fusion according to claim 1 is characterized in that: The multi-scale deep fusion network based on channel exchange includes: a preprocessing unit, a data alignment layer, a shared encoder, a channel exchange layer, a multimodal fusion unit and a multi-layer perception machine layer, wherein: the preprocessing unit aligns the multimodal data for preprocessing respectively, the data alignment layer aligns the dimensions of the preprocessed RSSI data and the image to obtain multimodal data with a consistent data dimension; the shared encoder uses an encoder with shared parameters to extract multimodal features and maps the multimodal features to a unified space; the channel exchange layer exchanges features between channels according to the features of each layer in the shared encoder and the importance of the features, and uses the consistency and complementarity of the models to effectively fuse the multimodal data; the multi-layer perception machine layer classifies according to the multimodal fusion features to finally obtain the positioning result.

4. The multimodal indoor positioning method based on heterogeneous feature fusion according to claim 1 or 3, characterized in that: The alignment specifically includes: a) To align the multimodal data, namely the wireless RSSI data and the visible light image, the original RSSI data range is adjusted from [0, 100] dBm to [0, 255], specifically: Where: L is the position to be located, represents the adjusted RSSI data, R(L) represents the original RSSI data, and R min (L) represents the minimum value of the original RSSI data, R max (L) indicates the maximum value of the original RSSI data, I max Indicates the maximum pixel value of the visible light image; b) The Wi-Fi and Bluetooth data are merged in a certain proportion to solve the problem that the wireless signal is unavailable at a specific AP access point. Specifically, the two data are directly spliced, that is, Where: L is the location to be located, R(L) is the fused RSSI data, R B (L), R W (L) represents the RSSI data of Bluetooth and WiFi respectively; c) Expand the dimension of RSSI data to solve the problem of large dimensional difference between wireless data and image data. Specifically: The constraints are in: The extended RSSI data obtained after interpolation, L0 is the new position coordinate, R(L i ) is the known RSSI data, L i are the known coordinates of several nearby positions, α i is the weight obtained by Kriging interpolation, which is obtained through matrix calculation, d represents the distance between adjacent reference points, and N represents the number of adjacent reference points; d) Initialize the location fingerprint database to store the RSSI data and image data of Wi-Fi from NR equally spaced areas, specifically: DB = {(m, (R(L), I(L)), m∈[1,..., N R ]), where: m is the serial number of the reference point location, R(L) is the Ns RSSI data from Nap AP nodes measured in a unit time, I(L) is the image data of the location, L is the location coordinate, N R is the number of reference points.

5. The multimodal indoor positioning method based on heterogeneous feature fusion according to claim 4 is characterized in that: The reference point L obtained by the sensor of the mobile device m RSSI data H(L m , T0), that is, the center position of each region The RSSI data of the area is regarded as the RSSI data of the area.

6. The multimodal indoor positioning method based on heterogeneous feature fusion according to claim 1 or 3, characterized in that: The data alignment layer includes: a range adjustment module, a data fusion module and a data alignment module, wherein: the range adjustment module performs data preprocessing according to the input original multimodal data to make the data ranges of the two modes consistent; the data fusion module performs RSSI data fusion of Bluetooth and WiFi according to the preprocessed RSSI data of the previous level to obtain fused RSSI data; the data alignment module performs RSSI and image data dimension alignment processing according to the RSSI data to obtain aligned RSSI data.

7. The multimodal indoor positioning method based on heterogeneous feature fusion according to claim 1 or 3, characterized in that: The shared encoder is composed of several levels of cascaded encoders, each of which includes: a sharing module and a dimensionality reduction module, wherein: the sharing module performs feature extraction and parameter sharing based on multimodal data to obtain multimodal features in a unified space, and the dimensionality reduction module performs feature dimensionality reduction processing based on the multimodal features to obtain reduced dimensionality multimodal features.

8. The multimodal indoor positioning method based on heterogeneous feature fusion according to claim 1 or 3, characterized in that: The channel exchange layer includes: a batch normalization (BN) layer and a channel exchange module, wherein: the normalization module performs feature normalization processing according to the features of each level, which is used to eliminate covariate offset and improve the generalization ability of the model to prevent the model from overfitting. The channel exchange module performs channel exchange processing according to the features of each level, fully utilizes the correlation between modes, and fuses features of multiple time scales.

9. The multimodal indoor positioning method based on heterogeneous feature fusion according to claim 1, 3 or 8, characterized in that: The channel exchange means that when one mode of any channel has little effect on the final prediction, the characteristics of the channel are replaced by the mean of the characteristics of other modes, specifically: The constraint condition is l = {R, I}, Where: μ l，c and σ l，c is the mean and standard deviation; γ l，c and β l，c are the scaling factor and offset used for training respectively; ε is a small constant to avoid division by zero; F′ l，c is the feature of mode l in channel c; L0 is the number of modes; for this task, L0 is equal to 2; γ l，c It has an impact on the multi-scale output of the channel; when γ l，c →0, feature F′ l，c Will lose influence on this mission.

10. The multimodal indoor positioning method based on heterogeneous feature fusion according to claim 1, 2 or 9, characterized in that: The training refers to: adopting a fully supervised training method, a training round dimension of 100, adopting a dynamic learning rate strategy, adopting an Adam optimization algorithm as an optimization method, a cross entropy loss function as a training loss function, and adopting a dropout strategy to solve the problem of model overfitting.

Images