Indoor and outdoor detection method and system based on pulse neural network
By using densely connected pulsed neural network to process smartphone sensor data, extract and encode features, combined with LIF model and batch normalization processing, the computational complexity and adaptability problems of indoor and outdoor detection in the prior art are solved, and efficient indoor and outdoor detection and generalization capabilities.
Patent Information
- Application Number
- CN202310025877.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-01-09
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-01-09
AI Technical Summary
The existing technology has detection methods based on predefined values in indoor and outdoor detection, which are not suitable for generalization to other scenarios, and the calculation complexity of deep learning methods is high, making it difficult to deploy on platforms with limited resources, and cannot efficiently process mobile phone sensor data. Meet accuracy and computing efficiency requirements.
The pulsed neural network (SNN) with dense connections is used to combine smartphone sensor measurement for indoor and outdoor detection. The feature is extracted, encoding and feature embedding is achieved by combining batch normalization processing and voting mechanisms.
It achieves higher computing efficiency and detection and recognition accuracy, can perform well in different scenarios and devices, has good generalization capabilities and low computing consumption.
Smart Images

Figure CN115774850B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of indoor and outdoor positioning technology, in particular to the field of indoor and outdoor detection methods based on neural networks, and relates to an indoor and outdoor detection method and system based on a pulse neural network. Background Art
[0002] Modern smart devices (e.g., smartphones, smart watches) integrate a large number of sensors, such as accelerometers, gyroscopes, light sensors, barometers, GNSS (Global Navigation Satellite System), and magnetometers, and are becoming an important platform for providing ubiquitous sensing and communication. Indoor / outdoor (IO) detection is the basic foundation for many applications and services, such as pedestrian navigation, power management, human tracking, and activity recognition. For example, IO information allows smart applications to automatically adjust screen brightness, or turn on / off GNSS services to reduce the power consumption of the device.
[0003] So far, many efforts have been made to deal with the IO detection problem. Early works utilized multiple sensing resources, such as light sensors, magnetic sensors, GNSS signals, and cell tower signals, to identify the user's environment with fixed predefined value detection rules. However, the fixed values are usually obtained through experimental measurements, and these values will change as the environment switches. Therefore, rule-based detection methods face obstacles when generalizing to other scenarios. To address the above problems, some learning-based solutions have been proposed. Some works use traditional machine learning (ML) methods to infer the user's environment. In this case, various artificial features are first extracted from smartphone sensor measurements to train ML models offline. Then, these trained models can be used for IO detection in real-time scenarios. In recent years, some deep learning-based methods have been proposed to further improve IO detection accuracy or reduce the workload of manual engineering of features. However, deep learning-based IO detection methods often suffer from huge complexity, which makes them unsuitable for deployment on resource-limited platforms.
[0004] Therefore, it is necessary to focus on how to efficiently process mobile phone sensor data to meet the accuracy requirements and computational efficiency requirements of IO detection scenarios. Summary of the invention
[0005] In view of this, the purpose of the present invention is to provide an indoor and outdoor detection method and system based on a spiking neural network, which uses a spiking neural network (SNN) with dense connections and smartphone sensor measurements for IO detection. Compared with traditional deep learning models, the method and system are more biologically credible and can be used for more efficient neuromorphic hardware; compared with previous learning-based methods, SNN is inherently suitable for utilizing temporal information and has high computational efficiency.
[0006] In order to achieve the above object, the present invention provides the following technical solutions:
[0007] An indoor and outdoor detection method based on a spiking neural network, which uses a spiking neural network (SNN) with dense connections combined with smartphone sensor measurements for indoor / outdoor (IO) detection, specifically comprising the following steps: S1: extracting main features from smartphone sensor measurements; S2: encoding the main features extracted from the sensor measurements; S3: using a spiking neural network (SNN) to further learn feature embedding; S4: using a voting mechanism to obtain a user's IO status to achieve indoor and outdoor detection of the user.
[0008] Further, in step S1, the extracting main features from the smartphone sensor measurements includes extracting a total of 43 features from 5 sensors: 9 features are extracted from satellite measurements, WiFi measurements, and cellular signal tower measurements, and 8 features are extracted from light sensors and magnetometer measurements, respectively; these features include the number of visible satellites or signal towers (for GNSS, WiFi, and cellular networks), mean, variance, standard deviation, maximum, minimum, median, range, and quartiles.
[0009] Further, in step S2, the main features are encoded. Common peak encoding methods include frequency encoding, time-to-priority peak encoding, and delay encoding schemes. Although these schemes have achieved good results in experiments, they will introduce an additional time step, resulting in a longer reasoning time for IO detection. In the present invention, a simpler encoding scheme is adopted, that is, the first convolutional layer of the model is used as the encoding layer, and the time step is the step size on the time series of the input signal.
[0010] Furthermore, in step S3, SNN is used to further learn feature embedding to achieve accurate IO detection; compared with traditional deep neural networks, SNN has a special activation mechanism, that is, each neuron accumulates the input of the previous layer into its membrane potential and triggers a pulse when its membrane potential exceeds a threshold;
[0011] A 1D convolutional layer is used to process the input feature vector, and then the LIF model is used as the dynamic model of SNN to perform nonlinear mapping on the features. The dynamics of the LIF model is generally described as:
[0012]
[0013] Where U(t) represents the membrane potential of the neuron at time t, and τ is a time constant, which represents the time it takes for the membrane potential to drop to its initial value. -1 time, X(t) represents the input to the neuron, U restRefers to the resting potential of a neuron; when a neuron is not stimulated or its membrane potential exceeds the firing threshold U th (0.5 in the present invention), the membrane potential of the neuron will be reset to U rest The value of ; rewrite the dynamics function into discrete form:
[0014]
[0015] S(t)=H(U(t)-U th ) (3)
[0016] Where S(t) represents the output pulse at time t, and H(*) represents the Heaviside step function; LIF neurons can use time information in the integration and pulse emission stages. However, it resets the membrane potential after each discharge process, which can only represent limited information, that is, each discharge will cause some information to be lost. Therefore, dense connections between Deep layers are used to better capture useful features. Let x0, x1, ..., x l-1 As the input features of the previous layer of the lth layer, the output of the lth densely connected layer is written as:
[0017] x l =f[x0, x1, ..., x l-1 ] (4)
[0018] Since SNN has a special spatiotemporal information processing mechanism, the gradient propagation tends to disappear, so it is difficult to train a deep SNN model. In order to solve this problem, a batch normalization processing mode of temporal features is adopted to implement a deep SNN model. The normalization process is described as follows:
[0019]
[0020]
[0021] in, represents the x of the i-th channel l (t) feature map, U th represents the peak emission threshold, and They represent the expectation and variance of the i-th channel feature map of the l-th layer through all time steps respectively; From formula (5), it can be seen that the output is normalized to 0 and the threshold U th between; i and β i are two learnable parameters; in the last stage, each category indicating the user environment is associated with an output spike train through a voting operation; the final output C in a given time window is written as:
[0022]
[0023] Where M is a constant voting matrix for decoding peaks, and O(t) is the output feature of the last layer at time t. However, since the impulse function is not differentiable, the popular error back propagation method cannot be used directly to train the network. In order to enable the model to be trained directly using the error back propagation algorithm, the present invention uses a rectangular function to approximate the derivative of the peak function, i.e.
[0024]
[0025] Wherein a is a width parameter, which can be set to 0.5 in the present invention.
[0026] Further, in step S4, the output of the system is associated with the target IO state through a voting mechanism.
[0027] The present invention also provides an indoor and outdoor detection system based on a pulse neural network for performing indoor and outdoor environment detection.
[0028] The beneficial effects of the present invention are:
[0029] This method can not only classify the user environment according to the current mobile sensor's indication, but also consider the measurement values of the previous time step for classification. Compared with the existing methods, the present invention has higher computational efficiency and detection recognition accuracy. Experimental results show that this method has good performance and can be well generalized to different scenarios and different mobile devices.
[0030] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:
[0032] Figure 1 The IO detection accuracy of the present invention and the benchmark method when generalized to new scenarios;
[0033] Figure 2 The IO detection accuracy of the present invention and the benchmark method when generalized to new devices;
[0034] Figure 3Comparison of the accuracy of DeepSIO using different sensor combinations, where G, C, L, M, and W represent GNSS, cellular network, optical sensor, optical sensor, magnetometer, and WiFi, respectively;
[0035] Figure 4 The DeepSIO architecture takes five sensors as input measurements and outputs the user's IO status. DETAILED DESCRIPTION
[0036] The technical solution of the present invention is described in detail below with reference to the accompanying drawings.
[0037] The present invention aims at the problems existing in the IO detection methods of the prior art: the IO detection methods based on predefined value detection rules rely on experimental data with the environment, which will face obstacles when being extended to other scenarios; the learning-based methods have high computational complexity and are difficult to deploy on resource-limited platforms. A DeepSIO IO detection method is proposed, which uses a densely connected pulse neural network (SNN) and smartphone sensor measurements for IO detection. The IO status of the user is detected by the measurement values of the mobile phone sensors. Compared with the existing methods, the present invention has higher computational efficiency and detection recognition accuracy. In this embodiment, a data set containing three recognition scenarios (campus, shopping mall, hospital) and 10 samples is used to evaluate the proposed method. The following comparisons are made during the evaluation process:
[0038] The recognition accuracy of the proposed method is compared with that of advanced IO detection methods, including MagIO, IODetector, NeuralIO, a popular rule-based method and a popular traditional machine learning-based method. The experimental results are shown in Table 1. It can be seen that DeepSIO has the highest detection accuracy in all test scenarios. Specifically, the IO detection accuracy of DeepSIO in campus, shopping mall and hospital scenarios reached 99.78%, 99.71% and 99.81% respectively.
[0039] Table 1. Comparison of IO detection accuracy (%) of different algorithms on datasets collected in different scenarios
[0040] method campus shopping center Hospital MagIO 80.07 72.69 82.78 IODetector 77.87 73.46 85.99 Rule-based approach 83.06 70.81 67.87 Machine learning based methods 99.23 91.81 90.62 NeuralIO 99.18 98.60 99.34 The present invention 99.78 99.71 99.81
[0041] The generalization ability of the proposed method in the present invention and the widely used machine learning method in new scenarios is compared. To this end, the present invention re-divides the collected data into a training set and a test set. Specifically, the data collected in shopping malls is used as the training set, and the data collected in campuses and hospitals is used as the test set. Figure 1It is shown that DeepSIO outperforms other methods and is able to detect IO status in new scenarios with an accuracy of 96%, which is much higher than other methods.
[0042] The generalization ability of the proposed method and the widely used machine learning method under new devices is compared. The collected data is re-divided into training set and test set, that is, the data collected by some mobile devices is used as the training set, and the data collected by other mobile devices is used as the test set. Figure 2 As shown, Figure 2 It is described that the method proposed in the present invention performs much better than other benchmark methods and can achieve an accuracy rate of more than 96.5%.
[0043] The experimental results show that the method proposed in the present invention has a high IO detection accuracy rate and also shows excellent performance in generalization ability. At the same time, the present invention adopts SNN as the backbone network and uses its characteristics of processing binary data (0 and 1) to greatly reduce the computational efficiency, so that the method proposed in the present invention can be applied to more lightweight devices.
[0044] The specific framework of the present invention is as follows Figure 4 As shown, the method proposed in the present invention takes measurements from various sensors as input, including GNSS, light sensors, WiFi, magnetometers, and cellular networks (such as GSM), and finally outputs the user's IO status. Specifically, the network uses LIF neurons, including a pulse coding module, two Deep blocks, a transition layer, two fully connected (FC) layers, and finally a voting layer for classification. Each Deep block consists of two Deep layers, each of which consists of two one-dimensional convolutional layers and two event-based batch normalization (EBN) layers. The present invention uses LIF as the dynamic model of the pulse neural network because it is very simple and has good performance. In each Deep block, dense connections between Deep layers are used to better capture features from the data.
[0045] Let x typeRepresents the measurement value of the sensor, where type represents the type of sensor. The main features of the sensor measurement value are extracted according to a one-second time window. These features include the number of visible satellites or signal towers (for GNSS, WiFi and cellular networks, light sensors and magnetometers are filled with the number 0), mean, variance, standard deviation, maximum, minimum, median, range and quartiles. After extracting these main features, these features are encoded into a pulse sequence and input to the SNN. Common peak coding methods include frequency coding, time-to-priority peak coding and delay coding schemes. Although these methods can approximate the intensity value well, it introduces a fixed-length time window, which leads to excessive time consumption in the inference stage. In the present invention, a simpler coding scheme is adopted, and the first convolutional layer of the model is used as the coding layer, which enables the model to consume less time during IO detection.
[0046] After pulse encoding, SNN is used to further learn feature embedding to achieve accurate IO detection. Compared with traditional deep neural networks, SNN has a special activation mechanism, that is, each neuron accumulates the input of the previous layer into its membrane potential and triggers a pulse when its membrane potential exceeds a threshold. Here, a 1-dimensional convolutional layer is used to process the input feature vector. Then the LIF model is used as the dynamic model of SNN to perform nonlinear mapping on the features. The dynamics of the LIF model is generally described as:
[0047]
[0048] Where U(t) represents the membrane potential of the neuron at time t, and τ is a time constant, which represents the time it takes for the membrane potential to drop to its initial value. -1 time, X(t) represents the input to the neuron. rest Refers to the resting potential of a neuron. When a neuron is not stimulated or its membrane potential exceeds the firing threshold U th (0.5 in this study), the membrane potential of the neuron will be reset to U rest To better understand the update of membrane potential, the kinetic function is rewritten in discrete form:
[0049]
[0050] S(t)=H(U(t)-U th ) (3)
[0051] Where S(t) represents the output pulse at time t, and H(*) represents the Heaviside step function. LIF neurons can use time information in the integration and pulse emission stages. However, it resets the membrane potential after each discharge process, which can only represent limited information, that is, each discharge will cause some information to be lost. Therefore, the present invention uses dense connections between Deep layers to better capture useful features. Let x0, x1, ..., x l-1 As the input features of the previous layer of the lth layer, the output of the lth densely connected layer is written as:
[0052] x l =f[x0, x1, ..., x l-1 ] (4)
[0053] Since SNN has a special spatiotemporal information processing mechanism, the gradient propagation tends to disappear, so it is difficult to train a deep SNN model. In order to solve this problem, the present invention adopts a batch normalization processing mode for temporal features to realize a deep SNN model. The normalization process is described as follows:
[0054]
[0055]
[0056] in, represents the x of the i-th channel l (t) feature map, U th represents the peak emission threshold, and They represent the expectation and variance of the i-th channel feature map of the l-th layer through all time steps. From formula (5), it can be seen that the output is normalized to 0 and the threshold U th Between. i and β i are two learnable parameters. In the last stage, the model associates each category indicating the user environment with an output spike train through a voting operation. The final output C in a given time window is written as:
[0057]
[0058] Where M is the constant voting matrix used to decode the peak value, and O(t) is the output feature of the last layer at time t. However, since the impulse function is not differentiable, the popular error back propagation method cannot be used directly to train the network. In order to allow the model to be trained directly using the error back propagation algorithm, a rectangular function is used to approximate the derivative of the peak function, that is,
[0059]
[0060] Where a is a width parameter, which is set to 0.5 in this embodiment.
[0061] In this embodiment, the proposed method is implemented using PyTorch. The method uses the Adam optimizer for training. The number of training iterations is set to 100, the batch size is set to 512, and the initial learning rate is set to 0.001, which drops to one-tenth of the previous value after 40 training periods. The method was evaluated in experiments conducted in three typical scenarios: campus, shopping mall, and hospital. The total number of data samples collected exceeded 100,000. 9 participants participated in the experiment and used 12 different mobile phones. After collecting the data, 50% of the data was randomly selected as the test set, and the rest was used as the training set. The following comparison was made and the following comparison conclusions were drawn:
[0062] The recognition accuracy of the proposed method is compared with that of advanced IO detection methods, including MagIO, IODetector, NeuralIO, a popular rule-based method and a popular traditional machine learning-based method. The experimental results are shown in Table 1. It can be seen that DeepSIO has the highest detection accuracy in all test scenarios. Specifically, the IO detection accuracy of DeepSIO in campus, shopping mall and hospital scenarios reached 99.78%, 99.71% and 99.81% respectively.
[0063] The generalization ability of the proposed method in the new scenario is compared with that of the widely used machine learning method. To this end, the collected data is re-divided into a training set and a test set. Specifically, the data collected from shopping malls is used as the training set, and the data collected from campuses and hospitals is used as the test set. Figure 1 It is shown that DeepSIO outperforms other methods and is able to detect IO status in new scenarios with an accuracy of 96%, which is much higher than other methods.
[0064] The generalization ability of the proposed method and the widely used machine learning method under new devices is compared. The collected data is re-divided into training set and test set, that is, the data collected by some mobile devices is used as the training set, and the data collected by other mobile devices is used as the test set. Figure 2 It is described that the method proposed in the present invention performs much better than other benchmark methods and can achieve an accuracy rate of more than 96.5%.
[0065] The effects of different sensor combinations on the performance of the proposed method are analyzed. Since the number of different combinations can reach 31, due to space limitations, only the results of using a few typical combinations are demonstrated. Figure 3 , Figure 3It shows that the best IO detection accuracy can be obtained using data from all 5 sensors, with an accuracy of about 99.5%. The method proposed in the present invention can also achieve relatively high accuracy (about 88%) using only the ubiquitous magnetic field, which is much higher than MagIO (less than 80%) working on the same data, indicating the superiority of this method. Cellular signals and light intensity can help improve the performance of DeepSIO very well. The combination of GNSS and magnetometer performs better than these combinations without GNSS. WiFi is also useful in helping IO detection, although the improvement is large. In general, regardless of the sensor combination, the method proposed in the present invention can achieve relatively satisfactory accuracy.
[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.
Claims
1. An indoor and outdoor detection method based on a pulse neural network, characterized in that: The method uses a densely connected Spiking Neural Network (SNN) combined with smartphone sensor measurements for indoor / outdoor (IO) detection, which includes the following steps: S1: Extracting key features from smartphone sensor measurements; S2: Encode the main features extracted from the sensor measurement; S3: Use spiking neural network (SNN) to further learn feature embedding; S4: A voting mechanism is used to obtain the user's IO status and realize indoor and outdoor detection of the user; In step S1, extracting main features from smartphone sensor measurements includes extracting a total of 43 features from 5 sensors: 9 features are extracted from satellite measurements, WiFi measurements, and cellular signal tower measurements, and 8 features are extracted from light sensor and magnetometer measurements respectively; these features include the number, mean, variance, standard deviation, maximum, minimum, median, range, and quartile of visible satellites or signal towers; In step S2, the main features are encoded using a simpler encoding scheme, that is, the first convolutional layer of the model is used as the encoding layer, and the time step is the step size of the input signal time series; In step S3, SNN is used to further learn feature embedding to achieve accurate IO detection; compared with traditional deep neural networks, SNN has a special activation mechanism, that is, each neuron accumulates the input of the previous layer into its membrane potential and triggers a pulse when its membrane potential exceeds a threshold; A 1D convolutional layer is used to process the input feature vector, and then the LIF model is used as the dynamic model of SNN to perform nonlinear mapping on the features. The dynamics of the LIF model is generally described as: Where U(t) represents the membrane potential of the neuron at time t, and τ is a time constant, which represents the time it takes for the membrane potential to drop to its initial value. -1 time, X(t) represents the input to the neuron, U rest Refers to the resting potential of a neuron; when a neuron is not stimulated or its membrane potential exceeds the firing threshold U th When the neuron's membrane potential is reset to U rest The value of ; rewrite the dynamics function into discrete form: S(t)=H(U(t)-U th ) (3) In the formula, S(t) represents the output pulse at time t, H(*) represents the Heaviside step function; dense connections between Deep layers are used to better capture useful features. Let x0,x1,...,x l-1 As the input features of the previous layer of the lth layer, the output of the lth densely connected layer is written as: x l =f[x0,x1,...,x l-1 ] (4) A batch normalization processing mode is used to implement the deep SNN model. The normalization process is described as follows: in, represents the x of the i-th channel l (t) feature map, U th represents the peak emission threshold, and They represent the expectation and variance of the i-th channel feature map of the l-th layer through all time steps respectively; From formula (5), it can be seen that the output is normalized to 0 and the threshold U th between; i and β i are two learnable parameters; in the last stage, each category indicating the user environment is associated with an output spike train through a voting operation; the final output C in a given time window is written as: Where M is the constant voting matrix used to decode the peak value, and O(t) is the output feature of the last layer at time r.
2. The indoor and outdoor detection method based on pulse neural network according to claim 1 is characterized in that: In step S4, the output of the system is associated with the target IO state through a voting mechanism.
3. An indoor and outdoor detection system based on a pulse neural network, characterized in that: The system adopts the method described in any one of claims 1 to 2 to perform indoor and outdoor environment detection.
