Indoor visible light fusion positioning method based on deep learning

Abstract

The invention discloses an indoor visible light fusion positioning method based on deep learning. The method comprises the following steps: step 1, acquiring AoA information and UWB distance information data of visible light at different positions, preprocessing the data, and converting the data into a two-dimensional matrix format to facilitate network input; 2, constructing a network model for extracting features of the preprocessed data, performing feature fusion, processing the fused features, and outputting positioning continuous coordinate values of the obtained data; step 3, training the network model, using samples including and not including AoA data, and ensuring that the network model can perform effective learning under the condition that only UWB data exists; according to the invention, the penetrability of the UWB signal is utilized to make up for the defect that visible light is easy to block, and high precision and stability can be maintained in different environments.

Description

Technical Field

[0001] The present invention belongs to the field of indoor positioning technology, and specifically relates to an indoor visible light fusion positioning method based on deep learning. Background Art

[0002] Visible light-based positioning technology is a method that uses optical communication technology to determine the precise location of an object or individual indoors. This method relies on the characteristics of light-emitting diode (LED) light sources, because LEDs can be modulated at extremely high frequencies, which enables them to carry and transmit data. The demodulator in the receiving device recovers the information modulated on the optical signal. A database is established in advance to store the characteristics of optical signals at different locations, and then the signal characteristics measured in real time are compared with the characteristics in the database to find the best matching position. However, its disadvantage is that visible light is easily blocked, which will seriously affect the positioning effect.

[0003] Among the existing solutions, the visible light indoor positioning system based on neural network with publication number CN114786119A obtains RSSI information of LED lights through hardware construction, and then sends it to the neural network for training to improve the positioning accuracy. However, it does not solve the problem that LEDs are easily blocked, and the multiple reflection effects of indoor walls, floors, and randomly placed objects lead to poor positioning stability.

[0004] The UWB-based indoor positioning method with the publication number CN118067123A is based on the UWB technical solution. By measuring the distance between the target and each UWB anchor point, the weighted least square positioning algorithm is used for positioning solution. For abnormal observation data, the error of the observation is estimated by biased Kalman filtering, and the weighted least square positioning algorithm is used for positioning solution. It is necessary to deploy multiple UWB anchor points, which greatly increases the cost, and ignores the stability of the UWB technical solution in a complex metal object environment. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide an indoor visible light fusion positioning method based on deep learning, which uses the penetration of UWB signals to make up for the shortcoming that visible light is easily blocked, so as to maintain high accuracy and stability in different environments. At the same time, by optimizing the deployment and use strategy of UWB anchor points, the energy consumption and cost of the overall system can be reduced.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] An indoor visible light fusion positioning method based on deep learning includes the following steps:

[0008] Step 1: Construct a data set, collect AoA data and UWB data of visible light at different positions, perform data preprocessing, and then transform it into a two-dimensional matrix format for network input;

[0009] Among them, AoA information indicates the angle of arrival of visible light, including the yaw angle and the angle of incidence. UWB is based on distance positioning, and the distance information of UWB indicates the distance between the receiving device and the signal source.

[0010] Step 2: Build a network model to extract the features of the preprocessed data, perform feature fusion, process the fused features, and obtain the continuous coordinate values ​​of the data;

[0011] Step 3: Train the network model using samples with and without AoA data to ensure that the network model can learn effectively even with only UWB data.

[0012] The final result is that even when visible light is blocked, good positioning effect is still achieved, achieving high-precision positioning.

[0013] In step 1, the specific steps of pretreatment are:

[0014] a) Data collection: AoA and UWB data are collected from receiving devices. AoA and UWB data are obtained by receiving devices installed in different locations. The receiving devices are used to receive the angle and distance of the signal. AoA and UWB distance information need to correspond one to one.

[0015] b) Data cleaning: remove invalid or erroneous data points;

[0016] c) Data calibration: Calibrate sensor data to eliminate bias;

[0017] d) Feature extraction: extract angle features from AoA and distance features from UWB;

[0018] e) Standardization: scaling the data to the same range.

[0019] In the step 1, the preprocessed data is converted into a two-dimensional matrix format;

[0020] For the two-dimensional matrix format, assume that N sample data are collected. The sample data is a concatenation of AoA data and UWB data, which is a vector. Each sample data contains M features, namely angle and distance features from multiple signals. A two-dimensional matrix with a shape of (N, M) is constructed, where each row represents a sample and each column represents a specific feature, namely, angle features and distance features. The angle features are further divided into those from different signal sources.

[0021] If the data of a sample is represented by a vector x=[x1,x2,…,x M ], where x i Represents the i-th feature of the sample, then the entire data set is represented as a matrix X, whose shape is N×M, that is:

[0022]

[0023] x NM Represents the Mth feature of the Nth sample.

[0024] The step 2 is specifically as follows:

[0025] a) Build a CNN model. For each data type (AoA and UWB), build an independent CNN model;

[0026] b) Feature fusion: Use the Concatenate layer to concatenate the two CNN models in appropriate dimensions, ensuring that before concatenation, the sizes of the remaining dimensions match except for the dimension to be concatenated;

[0027] c) Fully connected layer and output, after the Concatenate layer, add one or more fully connected layers to integrate and further process the fused features, and add an output layer. According to the positioning task, the output layer predicts continuous coordinate values, and the output realizes the regression task.

[0028] In the step a), each model should include a convolution layer, a pooling layer, a normalization layer and an activation function to extract key features from the data; the convolution layer is located at the front end of the network and is responsible for capturing the basic features of the input data. The activation function layer will follow the convolution layer to increase nonlinearity. The pooling layer follows the convolution layer to reduce the dimension of the feature map. The normalization layer is placed after the fully connected layer or the convolution layer, or before the activation function, and is used together with the convolution layer and the fully connected layer to improve the learning dynamics of the entire network.

[0029] Use convolutional layers to capture local features. The size, number, and step length of the convolution kernel can be adjusted according to the specific task.

[0030] Pooling layers are used to reduce spatial dimensions while keeping the most important features;

[0031] The activation function introduces nonlinear elements, allowing the neural network to learn complex mapping relationships;

[0032] Batch Normalization is used to standardize the input or output of the intermediate layer, thereby speeding up training and improving training stability. By reducing internal covariate shift, the normalization layer can make the network easier to learn, especially in deep networks.

[0033] The specific steps of step b) are:

[0034] Use the Concatenate layer to concatenate two CNN models, setting them to be concatenated along the channel dimension (usually the last dimension). The specific operations are as follows:

[0035] i) Import necessary modules: Make sure to import the Keras Concatenate layer;

[0036] ii) Define the outputs of the two models: set output_vlc to be the output of the model that processes visible light data, and output_uwb to be the output of the model that processes UWB data;

[0037] iii) Create a Concatenate layer: Use the Concatenate layer to concatenate along the specified axis (usually the last axis, that is, the channel dimension); concatenated_output = Concatenate(axis = -1)([output_vis, output_uwb]) axis = -1 means concatenation along the last dimension (usually the number of channels). If you want to concatenate along other dimensions, change the value of axis accordingly;

[0038] In the step c), after the concatenate layer is used to fuse in a certain dimension, a fully connected layer is added, which is usually used to further integrate and process the information from different branches or different features.

[0039] The fully connected layer converts the input feature vector into a representation of different dimensions through a weight matrix and a bias term, and usually uses an activation function (such as ReLU or Leaky ReLU) to introduce nonlinearity, thereby enhancing the model's ability to learn complex patterns.

[0040] The number of neurons in the output layer depends on the output dimension of the task. If the task is to predict the coordinates of a point in three-dimensional space, the output layer usually contains three neurons and no activation function is required (or a linear activation function is used), because the output in the regression problem can be any real number.

[0041] The step 3 is specifically as follows:

[0042] Step a): Prepare the dataset and divide it into a training set, a validation set, and a test set; ensure that the training set contains both samples containing AoA data and samples containing only UWB data (occlusion state);

[0043] Step b): Use a data loader or data generator to batch load data to ensure that samples containing AoA data and only UWB data are loaded in each iteration;

[0044] Step c): Make sure that two independent CNN models have been defined, one for VLC data and the other for UWB data, and their outputs have been fused through the Concatenate layer;

[0045] Step d): If it is multi-stage training, you also need to define the final fully connected layer and output layer; use the appropriate loss function, optimizer and evaluation indicators to compile the model.

[0046] Beneficial effects of the present invention:

[0047] The present invention takes into account all possible LED occlusion states (data sets that do not include AoA information) during user movement, and uses UWB technology to supplement it; the integration with visible light positioning greatly reduces the cost and improves the stability of positioning. The present invention uses a large number of data sets to verify the accuracy and stability of the present invention, and uses a deep learning method to fuse the results of UWB and visible light positioning, so that the accuracy of the positioning results is greatly improved, and the positioning accuracy can reach the millimeter level.

[0048] The present invention collects user motion estimation data in different environments to construct a user motion state data set, and performs test and evaluation on user motion state classification. The present invention is used in various indoor complex places and has wide applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of the visible light and UWB fusion positioning algorithm of the present invention.

[0050] Figure 2 This is the neural network model for the fusion of visible light and UWB indoor positioning of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0052] Among the existing technologies, UWB has the highest accuracy, but its cost is relatively high; while visible light positioning has a low cost, but positioning is easily affected by occlusion. Therefore, the present invention integrates visible light technology and UWB technology through a deep learning method, which not only improves positioning accuracy, but also improves stability, reduces costs, and compensates for each other's shortcomings. At the same time, visible light positioning technology uses the existing LED lighting system, and there is no need to deploy additional positioning equipment.

[0053] like Figure 1 As shown, a deep learning-based indoor visible light fusion positioning method includes the following steps:

[0054] Step 1: Construct a data set. First, collect the AoA data of visible light and the distance information data of UWB, perform data preprocessing, and then transform them into a two-dimensional matrix format for network input.

[0055] In step 1, the specific steps of pretreatment are:

[0056] a) Data collection: AoA and UWB data are collected from receiving devices. AoA and UWB data are obtained by receiving devices installed at different locations. The receiving devices are used to receive the angle and distance of the signal.

[0057] b) Data cleaning: remove invalid or erroneous data points;

[0058] c) Data calibration: Calibrate sensor data to eliminate bias;

[0059] d) Feature extraction: extract angle features from AoA and distance features from UWB;

[0060] e) Standardization: Scaling the data to the same range helps improve the performance of model training.

[0061] In the step 1, the preprocessed data is converted into a two-dimensional matrix format;

[0062] For the two-dimensional matrix format, assume that N sample data are collected, each sample data contains M features (such as angles and distances from multiple receiving devices), and construct a two-dimensional matrix with a shape of (N, M), where each row represents a sample and each column represents a specific feature;

[0063] If the data of a sample is represented by a vector x=[x1,x2,…,x M ], where x i represents the i-th feature, then the entire data set is represented as a matrix X, whose shape is N×M, that is:

[0064]

[0065] Step 2: If Figure 2 As shown, a network model is constructed, which includes three parts:

[0066] a) Build a CNN model. For each data type (visible light and UWB), build an independent CNN model;

[0067] In step a), the main difference between these two different types of data lies in the characteristics of the data and how to convert these characteristics into a form suitable for CNN input. Visible light data tends to be naturally suitable for standard image processing processes, while UWB data may require additional preprocessing steps to make it suitable for CNN processing. In addition, due to different data sources, the size, shape, and other parameters of the convolution kernel may need to be adjusted to adapt to the characteristics of the respective data. For example, UWB data may focus more on changes in distance, while visible light data focuses more on details in angles. Therefore, when designing a model, these factors need to be considered to optimize model performance. Each model should contain convolutional layers, pooling layers, normalization layers, and activation functions to extract key features from the data; the convolutional layer is located at the front end of the network and is responsible for capturing the basic features of the input data. The activation function layer will follow the convolutional layer to increase nonlinearity. The pooling layer follows the convolutional layer to reduce the dimension of the feature map. The normalization layer is placed after the fully connected layer or convolutional layer, or before the activation function, and is used together with the convolutional layer and the fully connected layer to improve the learning dynamics of the entire network.

[0068] Use convolutional layers to capture local features. The size, number, and step length of the convolution kernel can be adjusted according to the specific task.

[0069] The pooling layer is used to reduce the spatial dimension while retaining the most important features. Commonly used pooling methods include Max Pooling and Average Pooling.

[0070] The activation function introduces nonlinear elements, allowing the neural network to learn complex mapping relationships; common activation functions include ReLU (Rectified Linear Unit), sigmoid, tanh, etc.

[0071] Batch Normalization is used to standardize the input or output of the intermediate layer, thereby speeding up training and improving training stability. By reducing internal covariate shift, the normalization layer can make the network easier to learn, especially in deep networks.

[0072] b) Feature fusion: Use the Concatenate layer to concatenate the two CNN models in appropriate dimensions, ensuring that before concatenation, the sizes of the remaining dimensions match except for the dimension to be concatenated;

[0073] The specific steps of step b)

[0074] Use the Concatenate layer to concatenate along the channel dimension (usually the last dimension). The specific operations are as follows:

[0075] i) Import necessary modules: Make sure to import the Keras Concatenate layer;

[0076] ii) Define the outputs of the two models: assume that output_vlc is the output of the model that processes visible light data, and output_uwb is the output of the model that processes UWB data;

[0077] iii) Create a Concatenate layer: Use the Concatenate layer to concatenate along the specified axis (usually the last axis, i.e. the channel dimension); concatenated_output = Concatenate(axis = -1)([output_vis, output_uwb])

[0078] In this example, axis = -1 means concatenation along the last dimension (usually the number of channels). If you want to concatenate along other dimensions, you can change the value of axis accordingly.

[0079] In the step c), the fully connected layer and the output, after the Concatenate layer, add one or more fully connected layers to integrate and further process the fused features, and add an output layer. According to the positioning task, the output layer should predict continuous coordinate values, so the output should achieve the regression task.

[0080] Specifically, after the Concatenate layer is fused in a certain dimension, a fully connected layer is added, which is usually used to further integrate and process the information from different branches or different features.

[0081] The fully connected layer converts the input feature vector into a representation of different dimensions through a weight matrix and a bias term, and usually uses an activation function (such as ReLU or Leaky ReLU) to introduce nonlinearity, thereby enhancing the model's ability to learn complex patterns.

[0082] The number of neurons in the output layer depends on the output dimension of the task. If the task is to predict the coordinates of a point in three-dimensional space, the output layer usually contains three neurons and no activation function is required (or a linear activation function is used), because the output in the regression problem can be any real number.

[0083] Step 3: Train the model using samples with and without VLC data to ensure that the model can learn effectively with only UWB data.

[0084] The step 3 is specifically as follows:

[0085] Prepare the dataset and divide it into training set, validation set and test set;

[0086] A general ratio might be 70% training set, 15% validation set, and 15% test set, but this can be adjusted based on specific circumstances.

[0087] Make sure that the training set contains both samples containing AoA data and samples containing only UWB data (occluded state);

[0088] Use a data loader or data generator to batch load data, ensuring that each iteration loads samples containing both AoA data and only UWB data.

[0089] Make sure you have defined two independent CNN models, one for AoA data and another for UWB data, and have fused their outputs through the Concatenate layer;

[0090] If it is multi-stage training, you also need to define the final fully connected layer and output layer;

[0091] Compile the model using appropriate loss function, optimizer, and evaluation metric.

[0092] For example, for classification tasks, you can use the cross entropy loss (categorical_crossentropy) and the Adam optimizer.

[0093] Use a suitable loss function and optimizer, such as Mean Squared Error (MSE) loss and Adam optimizer, to train the model.

[0094] Validation and testing, evaluate the performance of the model on the validation set, adjust the hyperparameters until satisfactory results are obtained, and perform a final evaluation on the test set to check the generalization ability of the model.

[0095] Hyperparameters are parameters that need to be set before training the model. They are not learned through the training process. Hyperparameters include learning rate, batch size, optimizer selection, number of layers, number of neurons in each layer, and regularization coefficient.

[0096] The goal of adjusting hyperparameters is to find the optimal configuration so that the model performs best on the validation set. The methods for adjusting hyperparameters mainly include manual adjustment, grid search, random search, and Bayesian optimization.

[0097] Performance indicator analysis:

[0098] Analyze the model performance based on the returned indicator values. For example, for classification tasks, you can view accuracy, precision, recall, F1-score, etc. For regression tasks, you can view mean squared error (MSE), root mean squared error (RMSE), mean absolute error (MAE), etc.

[0099] If the model performs poorly, further analysis is needed. For example, if the accuracy is very low, the model may be underfitting; if the loss of the validation set is much higher than that of the training set, it may be overfitting.

[0100] The present invention builds a visible light positioning system indoors, collects the AoA information of LEDs at different positions, and constructs a data set based on the distance between the receiving end and the UWB base station.

[0101] In view of the fact that most of the current deep learning methods for solving positioning problems use classification models, the present invention adopts a convolutional neural network and a regression model that uses a concatenate layer for fusion.

[0102] To address the problem of visible light occlusion, we use two data sets, one containing visible light and one excluding visible light, to design a deep learning model that can not only fuse visible light and UWB data, but also make accurate predictions based only on UWB data when visible light data is missing, thereby improving the generalization ability of the model.

[0103] Data preprocessing: processing the raw data accordingly, for example, discarding abnormal data or noise;

[0104] In terms of devices: The present invention uses visible light positioning algorithm and UWB positioning algorithm through the information of light source to achieve preliminary positioning of the user, and adopts deep learning method to fuse the positioning results to improve the stability and accuracy of indoor positioning. In addition to the camera, light source information (AoA) can be obtained by methods other than camera, such as PD array.

[0105] In terms of solutions: Both visible light positioning algorithms and UWB positioning algorithms can use other indoor visible light positioning methods, such as inertial navigation, geomagnetism, and PDR dead reckoning to achieve high-precision and high-robust indoor visible light positioning. For example: replace UWB signals with geomagnetic signals, and use the same method to fuse them with visible light signals to obtain high-precision positioning.

Claims

1. A deep learning-based indoor visible light fusion positioning method, characterized in that: The steps include: Step 1: Construct a data set, collect AoA information of visible light and UWB data at different positions, perform data preprocessing, and then transform it into a two-dimensional matrix format for network input; Step 2: Build a network model to extract the features of the data preprocessed in step 1, perform feature fusion, process the fused features, and obtain the continuous coordinate values ​​of the data; Step 3: Train the network model using samples with and without AoA data to ensure that the network model can learn effectively with only UWB data and still achieve high-precision positioning in the absence of visible light data AoA.

2. According to the deep learning-based indoor visible light fusion positioning method of claim 1, it is characterized in that: In step 1, the specific steps of pretreatment are: a) Data collection: AoA and UWB data are collected from receiving devices. AoA and UWB distance data are obtained by receiving devices installed at different locations. The receiving devices are used to detect the angle and distance of the signal. b) Data cleaning: remove invalid or erroneous data points; c) Data calibration: Calibrate sensor data to eliminate bias; d) Feature extraction: extract angle features from AoA and distance features from UWB; e) Standardization: scaling the data to the same range.

3. According to the indoor visible light fusion positioning method based on deep learning in claim 1, it is characterized in that: In the step 1, the preprocessed data is converted into a two-dimensional matrix format; For the two-dimensional matrix format, assume that N sample data are collected. The sample data is a concatenation of AoA data and UWB data, which is a vector. Each sample data contains M features, namely, angle and distance features from multiple received signals. A two-dimensional matrix with a shape of (N, M) is constructed, where each row represents a sample and each column represents a specific feature. If the data of a sample is represented by a vector x=[x1,x2,…,x M ], where x i represents the i-th feature, then the entire data set is represented as a matrix X, whose shape is N×M, that is: x NM Represents the Mth feature of the Nth sample.

4. The indoor VLC fusion positioning method based on deep learning according to claim 1, characterized in that: The step 2 is specifically as follows: a) Build a CNN model. For each data type, build an independent CNN model; b) Feature fusion: Use the Concatenate layer to concatenate the two CNN models in appropriate dimensions, ensuring that before concatenation, the sizes of the remaining dimensions match except for the dimension to be concatenated; c) Fully connected layer and output, after the Concatenate layer, add one or more fully connected layers to integrate and further process the fused features, and add an output layer. According to the positioning task, the output layer predicts continuous coordinate values, and the output realizes the regression task.

5. The indoor visible light fusion positioning method based on deep learning according to claim 4 is characterized in that: In the step a), each model should include a convolution layer, a pooling layer, a normalization layer and an activation function to extract key features from the data; the convolution layer is located at the front end of the network and is responsible for capturing the basic features of the input data. The activation function layer will follow the convolution layer to increase nonlinearity. The pooling layer follows the convolution layer to reduce the dimension of the feature map. The normalization layer is placed after the fully connected layer or the convolution layer, or before the activation function, and is used together with the convolution layer and the fully connected layer to improve the learning dynamics of the entire network.

6. The indoor visible light fusion positioning method based on deep learning according to claim 5, characterized in that: Use convolutional layers to capture local features. The size, number, and step length of the convolution kernels are adjusted according to the specific task. Pooling layers are used to reduce spatial dimensions while keeping the most important features; The activation function introduces nonlinear elements, allowing the neural network to learn complex mapping relationships; Batch Normalization is used to standardize the input or output of the intermediate layer, thereby speeding up training and improving training stability. By reducing internal covariate shift, the normalization layer can make the network easier to learn, especially in deep networks.

7. The indoor visible light fusion positioning method based on deep learning according to claim 5, characterized in that: The specific steps of step b) are: Use the Concatenate layer to concatenate two CNN models, and set the concatenation to be performed along the channel dimension. The specific operations are as follows: i) Import necessary modules: Make sure to import the Keras Concatenate layer; ii) Define the outputs of the two models: set output_vlc to be the output of the model that processes visible light data, and output_uwb to be the output of the model that processes UWB data; iii) Create a Concatenate layer: Use the Concatenate layer to concatenate along the specified axis; concatenated_output = Concatenate(axis = -1)([output_vis, output_uwb]) axis = -1 means concatenation along the last dimension. If you want to concatenate along other dimensions, change the value of axis accordingly.

8. The indoor visible light fusion positioning method based on deep learning according to claim 7, characterized in that: In the step c), after the concatenate layer is used to fuse in a certain dimension, a fully connected layer is added to further integrate and process the information from different branches or different features; The fully connected layer converts the input feature vector into a representation of different dimensions through a weight matrix and a bias term, usually using an activation function to introduce nonlinearity; The number of neurons in the output layer depends on the output dimension of the task. If the task is to predict the coordinates of a point in three-dimensional space, the output layer usually contains three neurons and no activation function is required.

9. The indoor visible light fusion positioning method based on deep learning according to claim 1, characterized in that: The step 3 is specifically as follows: Step 1): Prepare the dataset and divide it into training set, validation set and test set; ensure that the training set contains both samples containing AoA data and samples containing only UWB data; Step 2): Use a data loader or data generator to batch load data to ensure that samples containing AoA data and only UWB data are loaded in each iteration; Step 3): Make sure you have defined two independent CNN models, one for AoA data and the other for UWB data, and have fused their outputs through the Concatenate layer. Step 4): If it is multi-stage training, you also need to define the final fully connected layer and output layer; use the loss function, optimizer and evaluation indicators to compile the model.

Images