A deep learning-based indoor visible light fusion positioning method

Abstract

The present invention discloses an indoor visible light fusion positioning method based on deep learning, comprising the following steps: Step 1: collecting AoA information of visible light and distance information data of UWB at different positions, performing data preprocessing, and then converting the data into a two-dimensional matrix format for network input; Step 2: constructing a network model to extract features of the preprocessed data, perform feature fusion, process the fused features, and output continuous coordinate values ​​of the positioning data; Step 3: training the network model, using samples containing and excluding AoA data, to ensure that the network model can also be effectively learned when only UWB data is available; the present invention utilizes the penetrability of UWB signals to compensate for the deficiency that visible light is easily blocked, and can maintain high precision and stability in different environments.

Description

Technical Field

[0001] The present invention belongs to the field of indoor positioning technology, and specifically relates to an indoor visible light fusion positioning method based on deep learning. Background Art

[0002] Visible light positioning technology utilizes optical communication technology to precisely determine the location of an object or individual indoors. This method relies on the properties of light-emitting diode (LED) light sources. LEDs can be modulated at extremely high frequencies, enabling them to carry and transmit data. A demodulator in the receiving device recovers the information modulated on the optical signal. A database is pre-established to store the optical signal characteristics at different locations. The signal characteristics measured in real time are then compared with those in the database to find the best match. However, a drawback is that visible light is easily obstructed, which can severely affect positioning effectiveness.

[0003] Among existing solutions, the neural network-based visible light indoor positioning system, published under publication number CN114786119A, uses hardware to acquire RSSI information from LED lights and then feeds it into a neural network for training to improve positioning accuracy. However, this fails to address the issues of LEDs being easily obscured and multiple reflections from indoor walls, floors, and randomly placed objects, which lead to poor positioning stability.

[0004] Publication number CN118067123A describes a UWB-based indoor positioning method. Based on a UWB technology solution, it measures the distance between the target and various UWB anchor points and uses a weighted least squares positioning algorithm for positioning. For abnormal observation data, it estimates the observation error through a biased Kalman filter and uses a weighted least squares positioning algorithm for positioning. This requires the deployment of multiple UWB anchor points, significantly increasing costs and ignoring the stability issues of the UWB technology solution in complex metal environments. Summary of the Invention

[0005] To overcome the shortcomings of the aforementioned existing technologies, the present invention aims to provide a deep learning-based indoor visible light fusion positioning method. This method leverages the penetrating properties of UWB signals to compensate for visible light's vulnerability to obstruction, maintaining high accuracy and stability in diverse environments. Furthermore, by optimizing the deployment and use of UWB anchor points, it reduces overall system energy consumption and costs.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] A deep learning-based indoor visible light fusion positioning method includes the following steps:

[0008] Step 1: Construct a data set, collect visible light AoA data and UWB data at different locations, perform data preprocessing, and then transform it into a two-dimensional matrix format for network input;

[0009] Among them, AoA information represents the angle of arrival of visible light, including the yaw angle and the angle of incidence. UWB is based on distance positioning, and the distance information of UWB represents the distance between the receiving device and the signal source.

[0010] Step 2: Build a network model to extract the features of the preprocessed data, perform feature fusion, process the fused features, and obtain the continuous coordinate values ​​of the data;

[0011] Step 3: Train the network model using samples with and without AoA data to ensure that the network model can effectively learn even with only UWB data.

[0012] The final result is that even when visible light is blocked, good positioning effect is still achieved, achieving high-precision positioning.

[0013] In step 1, the specific steps of pretreatment are:

[0014] a) Data collection: AoA and UWB data are collected from receiving devices installed at different locations. The receiving devices are used to receive the angle and distance of the signal. AoA and UWB distance information need to correspond one-to-one.

[0015] b) Data cleaning: remove invalid or erroneous data points;

[0016] c) Data calibration: Calibrate sensor data to eliminate bias;

[0017] d) Feature extraction: Extract angle features from AoA and distance features from UWB;

[0018] e) Normalization: scaling the data to the same range.

[0019] In the step 1, the pre-processed data is converted into a two-dimensional matrix format;

[0020] For the two-dimensional matrix format, assume that N sample data are collected. The sample data is a concatenation of AoA data and UWB data, and is a vector. Each sample data contains M features, namely angle and distance features from multiple signals. A two-dimensional matrix with a shape of (N, M) is constructed, where each row represents a sample and each column represents a specific feature, namely angle features and distance features. The angle features are further divided into those from different signal sources.

[0021] If the data of a sample is represented by a vector x=[x1,x2,…,x M ], where x i Represents the i-th feature of the sample, then the entire data set is represented as a matrix X, whose shape is N×M, that is:

[0022]

[0023] x NM Represents the Mth feature of the Nth sample.

[0024] The step 2 is specifically as follows:

[0025] a) Build a CNN model. For each data type (AoA and UWB), build an independent CNN model.

[0026] b) Feature fusion: Use the Concatenate layer to concatenate the two CNN models in appropriate dimensions, ensuring that the sizes of all dimensions except the one to be concatenated match before concatenation.

[0027] c) Fully connected layer and output: After the concatenate layer, one or more fully connected layers are added to integrate and further process the fused features, and an output layer is added. According to the positioning task, the output layer predicts continuous coordinate values, and the output realizes the regression task.

[0028] In step a), each model should include a convolution layer, a pooling layer, a normalization layer, and an activation function to extract key features from the data; the convolution layer is located at the front end of the network and is responsible for capturing the basic features of the input data. The activation function layer will follow the convolution layer to increase nonlinearity. The pooling layer follows the convolution layer and is used to reduce the dimension of the feature map. The normalization layer is placed after the fully connected layer or the convolution layer, or before the activation function, and is used together with the convolution layer and the fully connected layer to improve the learning dynamics of the entire network.

[0029] Use convolutional layers to capture local features. The size, number, and stride of the convolution kernel can be adjusted according to the specific task.

[0030] Pooling layers are used to reduce spatial dimensions while retaining the most important features;

[0031] The activation function introduces nonlinear elements, allowing the neural network to learn complex mapping relationships;

[0032] Batch Normalization is used to standardize the input or output of the intermediate layer, thereby speeding up training and improving training stability. By reducing internal covariate shift, the normalization layer can make the network easier to learn, especially in deep networks.

[0033] The specific steps of step b) are:

[0034] Use the Concatenate layer to concatenate two CNN models, setting the concatenation to be performed along the channel dimension (usually the last dimension). The specific operations are as follows:

[0035] i) Import necessary modules: Make sure to import the Keras Concatenate layer;

[0036] ii) Define the outputs of the two models: set output_vlc to be the output of the model that processes visible light data, and output_uwb to be the output of the model that processes UWB data;

[0037] iii) Create a Concatenate layer: Use the Concatenate layer to concatenate along the specified axis (usually the last axis, i.e., the channel dimension); concatenated_output = Concatenate(axis = -1)([output_vis, output_uwb]) axis = -1 means concatenation along the last dimension (usually the number of channels). If you want to concatenate along other dimensions, change the value of axis accordingly;

[0038] In step c), after the concatenate layer fuses at a specific dimension, a fully connected layer is added, which is usually used to further integrate and process the information from different branches or different features.

[0039] The fully connected layer converts the input feature vector into a representation of different dimensions through a weight matrix and a bias term, and usually uses an activation function (such as ReLU or Leaky ReLU) to introduce nonlinearity, thereby enhancing the model's ability to learn complex patterns.

[0040] The number of neurons in the output layer depends on the output dimension of the task. If the task is to predict the coordinates of a point in three-dimensional space, the output layer usually contains three neurons and does not require an activation function (or uses a linear activation function) because the output in regression problems can be any real number.

[0041] The step 3 is specifically as follows:

[0042] Step a): Prepare the dataset and divide it into training, validation, and test sets. Ensure that the training set contains both samples containing AoA data and samples containing only UWB data (occlusion state).

[0043] Step b): Use a data loader or data generator to batch load data, ensuring that samples containing both AoA data and only UWB data are loaded in each iteration.

[0044] Step c): Ensure that two independent CNN models have been defined, one for VLC data and the other for UWB data, and their outputs have been fused through the Concatenate layer;

[0045] Step d): If multi-stage training is used, the final fully connected layer and output layer also need to be defined; use the appropriate loss function, optimizer, and evaluation metrics to compile the model.

[0046] Beneficial effects of the present invention:

[0047] This invention considers all possible LED occlusion states during user movement (datasets that do not include AoA information) and uses UWB technology to supplement this. Its integration with visible light positioning significantly reduces costs and improves positioning stability. This invention uses a large dataset to verify its accuracy and stability, and employs deep learning to fuse the results of UWB and visible light positioning, significantly improving positioning accuracy to the millimeter level.

[0048] The present invention collects user motion estimation data in different environments to construct a user motion state data set, and tests and evaluates the user motion state classification. The present invention is used in various indoor complex places and has wide applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of the visible light and UWB fusion positioning algorithm of the present invention.

[0050] Figure 2 This is the neural network model for the fusion of visible light and UWB indoor positioning of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be described in further detail below with reference to the accompanying drawings.

[0052] Among existing technologies, UWB offers the highest accuracy but comes at a high cost. Visible light positioning, on the other hand, is low-cost but susceptible to occlusion. Therefore, this invention integrates visible light and UWB technologies through deep learning, improving positioning accuracy, enhancing stability, and reducing costs, thus compensating for these shortcomings. Furthermore, visible light positioning utilizes existing LED lighting systems, eliminating the need for additional positioning equipment.

[0053] like Figure 1 As shown, a deep learning-based indoor visible light fusion positioning method includes the following steps:

[0054] Step 1: Build a data set. First, collect visible light AoA data and UWB distance information data, perform data preprocessing, and then transform them into a two-dimensional matrix format for network input.

[0055] In step 1, the specific steps of pretreatment are:

[0056] a) Data collection: AoA and UWB data are collected from receiving devices installed at different locations. The receiving devices are used to receive the angle and distance of the signal.

[0057] b) Data cleaning: remove invalid or erroneous data points;

[0058] c) Data calibration: Calibrate sensor data to eliminate bias;

[0059] d) Feature extraction: extract angle features from AoA and distance features from UWB;

[0060] e) Standardization: Scaling the data to the same range helps improve the performance of model training.

[0061] In the step 1, the pre-processed data is converted into a two-dimensional matrix format;

[0062] For the two-dimensional matrix format, assume that N sample data are collected, each sample data contains M features (such as angles and distances from multiple receiving devices), and construct a two-dimensional matrix of shape (N, M), where each row represents a sample and each column represents a specific feature;

[0063] If the data of a sample is represented by a vector x=[x1,x2,…,x M ], where x i Represents the i-th feature, then the entire data set is represented as a matrix X, whose shape is N×M, that is:

[0064]

[0065] Step 2: If Figure 2 As shown, a network model is constructed, which includes three parts:

[0066] a) Build a CNN model. For each data type (visible light and UWB), build an independent CNN model.

[0067] In step a), the primary difference between the two different types of data lies in the data's characteristics and how these characteristics are converted into a form suitable for CNN input. Visible light data tends to naturally adapt to standard image processing pipelines, while UWB data may require additional preprocessing steps to make it suitable for CNN processing. Furthermore, due to the different data sources, the size, shape, and other parameters of the convolution kernel may need to be adjusted to suit the characteristics of the respective data. For example, UWB data may prioritize changes in distance, while visible light data may focus more on angular details. Therefore, these factors need to be considered when designing the model to optimize performance. Each model should include convolutional layers, pooling layers, normalization layers, and activation functions to extract key features from the data. Convolutional layers are located at the front end of the network, responsible for capturing the essential features of the input data. Activation layers are used after convolutional layers to add nonlinearity. Pooling layers follow convolutional layers to reduce the dimensionality of the feature maps. Normalization layers are placed after fully connected or convolutional layers, or before activation functions, and are used together with convolutional and fully connected layers to improve the learning dynamics of the entire network.

[0068] Use convolutional layers to capture local features. The size, number, and stride of the convolution kernel can be adjusted according to the specific task.

[0069] The pooling layer is used to reduce the spatial dimension while retaining the most important features. Commonly used pooling methods include Max Pooling and Average Pooling.

[0070] The activation function introduces nonlinear elements, allowing the neural network to learn complex mapping relationships; common activation functions include ReLU (Rectified Linear Unit), sigmoid, tanh, etc.

[0071] Batch Normalization is used to standardize the input or output of the intermediate layer, thereby speeding up training and improving training stability. By reducing internal covariate shift, the normalization layer can make the network easier to learn, especially in deep networks.

[0072] b) Feature fusion: Use the Concatenate layer to concatenate the two CNN models in appropriate dimensions, ensuring that the sizes of all dimensions except the one to be concatenated match before concatenation.

[0073] The specific steps of step b)

[0074] Use the Concatenate layer to concatenate along the channel dimension (usually the last dimension). The specific operations are as follows:

[0075] i) Import necessary modules: Make sure to import the Keras Concatenate layer;

[0076] ii) Define the outputs of the two models: assume that output_vlc is the output of the model that processes visible light data, and output_uwb is the output of the model that processes UWB data;

[0077] iii) Create a Concatenate layer: Use the Concatenate layer to concatenate along the specified axis (usually the last axis, i.e. the channel dimension); concatenated_output = Concatenate(axis=-1)([output_vis, output_uwb])

[0078] In this example, axis = -1 means concatenation along the last dimension (usually the number of channels). If you want to concatenate along other dimensions, you can change the value of axis accordingly.

[0079] In step c), after the Concatenate layer, one or more fully connected layers are added to the fully connected layer and output for integrating and further processing the fused features, and an output layer is added. According to the positioning task, the output layer should predict continuous coordinate values, so the output should realize the regression task.

[0080] Specifically, after the concatenate layer fuses in a certain dimension, a fully connected layer is added, which is usually used to further integrate and process information from different branches or different features.

[0081] The fully connected layer converts the input feature vector into a representation of different dimensions through a weight matrix and a bias term, and usually uses an activation function (such as ReLU or Leaky ReLU) to introduce nonlinearity, thereby enhancing the model's ability to learn complex patterns.

[0082] The number of neurons in the output layer depends on the output dimension of the task. If the task is to predict the coordinates of a point in three-dimensional space, the output layer usually contains three neurons and does not require an activation function (or uses a linear activation function) because the output in regression problems can be any real number.

[0083] Step 3: Train the model using samples with and without VLC data to ensure that the model can learn effectively even with only UWB data.

[0084] The step 3 is specifically as follows:

[0085] Prepare the dataset and divide it into training set, validation set and test set;

[0086] A general ratio might be 70% training set, 15% validation set, and 15% test set, but this can be adjusted based on specific circumstances.

[0087] Ensure that the training set contains both samples containing AoA data and samples containing only UWB data (occlusion state);

[0088] Use a data loader or data generator to load data in batches, ensuring that samples containing both AoA data and only UWB data are loaded in each iteration.

[0089] Make sure you have defined two independent CNN models, one for processing AoA data and the other for processing UWB data, and have fused their outputs through the Concatenate layer;

[0090] If multi-stage training is used, the final fully connected layer and output layer also need to be defined;

[0091] Compile the model using the appropriate loss function, optimizer, and evaluation metric.

[0092] For example, for classification tasks, you can use the cross entropy loss (categorical_crossentropy) and the Adam optimizer.

[0093] Use appropriate loss functions and optimizers, such as mean squared error (MSE) loss and Adam optimizer, to train the model.

[0094] Validation and testing: Evaluate the performance of the model on the validation set, adjust the hyperparameters until satisfactory results are obtained, and perform a final evaluation on the test set to check the generalization ability of the model.

[0095] Hyperparameters are parameters that need to be set before training the model. They are not learned during the training process. Hyperparameters include learning rate, batch size, choice of optimizer, number of layers, number of neurons in each layer, and regularization coefficient.

[0096] The goal of adjusting hyperparameters is to find the optimal configuration to achieve the best performance on the validation set. Methods for adjusting hyperparameters include manual adjustment, grid search, random search, and Bayesian optimization.

[0097] Performance indicator analysis:

[0098] Analyze model performance based on the returned metrics. For example, for classification tasks, you can view accuracy, precision, recall, and F1-score. For regression tasks, you can view mean squared error (MSE), root mean squared error (RMSE), and mean absolute error (MAE).

[0099] If the model performs poorly, further analysis is needed. For example, if the accuracy is very low, the model may be underfitting; if the loss on the validation set is much higher than the loss on the training set, it may be overfitting.

[0100] The present invention builds a visible light positioning system indoors to collect the AoA information of LEDs at different positions and the distance between the receiving end and the UWB base station to construct a data set.

[0101] In view of the fact that most of the current deep learning methods used to solve positioning problems use classification models, the present invention adopts a convolutional neural network and a regression model that uses a concatenate layer for fusion.

[0102] To address the problem of visible light occlusion, we use two datasets, one containing visible light and the other excluding visible light, to design a deep learning model that can not only fuse visible light and UWB data, but also make accurate predictions based solely on UWB data when visible light data is missing, thereby improving the model's generalization ability.

[0103] Data preprocessing: performing corresponding processing on the original data, for example, discarding abnormal data or noise;

[0104] In terms of devices: The present invention uses visible light positioning algorithm and UWB positioning algorithm through the information of light source to achieve preliminary positioning of the user, and adopts deep learning method to fuse the positioning results to improve the stability and accuracy of indoor positioning. In addition to the camera, light source information (AoA) can be obtained by using methods other than the camera, such as the PD array.

[0105] In terms of solutions, both visible light and UWB positioning algorithms can utilize other indoor visible light positioning methods, such as inertial navigation, geomagnetism, and PDR dead reckoning, to achieve high-precision and robust indoor visible light positioning. For example, geomagnetic signals can be used to replace UWB signals and then fused with visible light signals using the same method to achieve high-precision positioning.

Claims

1. A deep learning-based indoor visible light fusion positioning method, characterized in that: The following steps are included: Step 1: Construct a data set to collect visible light AoA information and UWB data at different locations, perform data preprocessing, and then transform it into a two-dimensional matrix format for network input. Step 2: Build a network model to extract the features of the data preprocessed in step 1, perform feature fusion, process the fused features, and obtain the continuous coordinate values ​​of the data; Step 3: Train the network model using samples with and without AoA data to ensure that the network model can effectively learn with only UWB data and still achieve high-precision positioning in the absence of visible light AoA data. The step 2 is specifically as follows: a) Build a CNN model. For each data type, build an independent CNN model; b) Feature fusion: Use the Concatenate layer to concatenate the two CNN models. Ensure that the sizes of all dimensions except the one to be concatenated match before concatenation. c) Fully connected layer and output: After the concatenate layer, one or more fully connected layers are added to integrate and further process the fused features. An output layer is also added. Based on the positioning task, the output layer predicts continuous coordinate values ​​and outputs them to achieve the regression task. The step 3 is specifically as follows: Step 1): Prepare the dataset and divide it into training, validation, and test sets. Ensure that the training set contains both samples containing AoA data and samples containing only UWB data. Step 2): Use a data loader or data generator to batch load data, ensuring that samples containing both AoA data and only UWB data are loaded in each iteration. Step 3): Ensure that two independent CNN models have been defined, one for processing AoA data and the other for processing UWB data, and their outputs have been fused through the Concatenate layer; Step 4): If multi-stage training is used, the final fully connected layer and output layer must also be defined; the model is compiled using the loss function, optimizer, and evaluation metrics.

2. The indoor visible light fusion positioning method based on deep learning according to claim 1 is characterized in that: In step 1, the specific steps of pretreatment are: a) Data collection: AoA and UWB data are collected from receiving devices. AoA and UWB distance data are acquired by receiving devices installed in different locations. The receiving devices are used to detect the angle and distance of the signal. b) Data cleaning: remove invalid or erroneous data points; c) Data calibration: Calibrate sensor data to eliminate bias; d) Feature extraction: Extract angle features from AoA and distance features from UWB; e) Normalization: scaling the data to the same range.

3. The indoor visible light fusion positioning method based on deep learning according to claim 1 is characterized in that: In the step 1, the pre-processed data is converted into a two-dimensional matrix format; For the two-dimensional matrix format, assume that N sample data are collected. The sample data is a concatenation of AoA data and UWB data, which is a vector. Each sample data contains M features, namely the angle and distance features from multiple received signals. A two-dimensional matrix with a shape of (N, M) is constructed, where each row represents a sample and each column represents a specific feature. If the data of a sample is represented by a vector x=[x1,x2,…,x M ], where x i Represents the i-th feature, then the entire data set is represented as a matrix X, whose shape is N×M, that is: x NM Represents the Mth feature of the Nth sample.

4. The indoor visible light fusion positioning method based on deep learning according to claim 1 is characterized in that: In step a), each model should include a convolution layer, a pooling layer, a normalization layer, and an activation function to extract key features from the data; the convolution layer is located at the front end of the network and is responsible for capturing the basic features of the input data. The activation function layer will follow the convolution layer to increase nonlinearity. The pooling layer follows the convolution layer and is used to reduce the dimension of the feature map. The normalization layer is placed after the fully connected layer or the convolution layer, or before the activation function, and is used together with the convolution layer and the fully connected layer to improve the learning dynamics of the entire network.

5. The indoor visible light fusion positioning method based on deep learning according to claim 4 is characterized in that: Use convolutional layers to capture local features, and the size, number, and stride of the convolution kernels are adjusted according to the specific task; Pooling layers are used to reduce spatial dimensions while retaining the most important features; The activation function introduces nonlinear elements, allowing the neural network to learn complex mapping relationships; Batch Normalization is used to standardize the input or output of the intermediate layer, thereby speeding up training and improving training stability. By reducing internal covariate shift, the normalization layer can make the network easier to learn, especially in deep networks.

6. The indoor visible light fusion positioning method based on deep learning according to claim 4 is characterized in that: The specific steps of step b) are: Use the Concatenate layer to concatenate two CNN models, setting the concatenation to be performed along the channel dimension. The specific operations are as follows: i) Import necessary modules: Make sure to import the Keras Concatenate layer; ii) Define the outputs of the two models: set output_vlc to be the output of the model that processes visible light data, and output_uwb to be the output of the model that processes UWB data; iii) Create a Concatenate layer: Use the Concatenate layer to concatenate along the specified axis; concatenated_output = Concatenate(axis = -1)([output_vis, output_uwb]) axis = -1 means concatenation along the last dimension. If you want to concatenate along other dimensions, change the value of axis accordingly.

7. The indoor visible light fusion positioning method based on deep learning according to claim 6, characterized in that: In step c), after the concatenate layer fuses the information in a specific dimension, a fully connected layer is added to further integrate and process the information from different branches or different features; The fully connected layer converts the input feature vector into a representation of different dimensions through a weight matrix and a bias term, usually using an activation function to introduce nonlinearity; The number of neurons in the output layer depends on the output dimension of the task. If the task is to predict the coordinates of a point in three-dimensional space, the output layer usually contains three neurons and no activation function is required.

Images