Fine-grained indoor and outdoor scene recognition method, device and system and storage medium

Abstract

The invention discloses a fine-grained indoor and outdoor scene recognition method, device and system and a storage medium, multi-modal data in different scenes are synchronously captured at high frequency by using GNSS, IMU, geomagnetism, air pressure, light intensity and other multi-source sensors built in a smart phone, and data consistency is processed by adopting time interpolation synchronization and filtering noise reduction, so that the recognition accuracy of the indoor and outdoor scenes is improved. According to the method, time-frequency domain features are extracted through short-time Fourier transform (STFT), time domain local features are extracted by means of CNN, frequency domain global dependence is captured by ViT, and association between structured features and a scene is mined through TabTransform, so that the precision and generalization ability of fine-grained scene recognition are improved.

Description

Technical Field

[0001] This invention belongs to the field of pattern recognition technology, specifically relating to a fine-grained indoor and outdoor scene recognition method, device, system, and storage medium. Background Technology

[0002] Fine-grained scene recognition, as a core technology for perceiving environmental attributes and supporting intelligent decision-making, demonstrates irreplaceable application value in fields such as smart mobility, navigation and positioning, smart homes, and emergency rescue. Traditional scene recognition methods typically rely on a single type of perceptual signal (such as visual images, GNSS positioning information, or single motion sensor data), analyzing environmental features (such as light intensity, location coordinates, and motion state) within the signal to achieve scene discrimination. Early research focused on simple scene classification (such as a binary division between indoor and outdoor), for example, determining whether one is outdoors solely based on GNSS signal strength, or distinguishing between light and dark environments using light sensor data, and then inputting these features into traditional machine learning models such as support vector machines and random forests to complete the recognition. However, traditional scene recognition methods face many challenges in practical applications: First, most methods rely on a single sensor, resulting in limited data dimensions and susceptibility to environmental interference. For example, GNSS signals suffer severe loss indoors or in obstructed environments, and optical sensors struggle to distinguish between different scenes with similar lighting (such as an enclosed elevator and an underground parking garage), leading to incomplete scene feature characterization and poor robustness. Second, traditional models lack the ability to extract spatiotemporal features. Scene information includes not only instantaneous states (such as air pressure at a given moment) but also dynamic change patterns (such as continuous air pressure changes when going up or down stairs). Traditional methods are often limited to temporal statistical features (such as mean and variance), neglecting temporal characteristics (such as periodic fluctuations) and spatiotemporal correlations, making it difficult to distinguish fine-grained scenes with similar features (such as going up or down stairs). Furthermore, existing methods are poorly adaptable to complex environments. Real-world scenarios often involve complex spatial structures (such as multi-story buildings) and dynamic interference (such as vehicle vibrations). Due to their limited generalization ability, traditional models experience a significant drop in recognition accuracy in non-line-of-sight and multi-obstructed scenarios.

[0003] In recent years, with the widespread adoption of smartphones, solutions utilizing their built-in multi-sensor arrays (such as GNSS, IMU, geomagnetism, and barometers) for scene recognition have gained attention due to their versatility. These solutions enhance feature richness by fusing multi-source data and strengthen feature extraction capabilities by combining deep learning models, thus providing new possibilities for fine-grained scene recognition.

[0004] However, fine-grained scene recognition based on multiple sensors of smartphones still faces key challenges: (1) The sampling rates of multiple sensors vary greatly and the data formats are heterogeneous, which can easily lead to time synchronization problems, resulting in high noise and poor consistency of fused data; (2) Existing feature extraction strategies mostly focus on a single time domain or frequency domain, failing to fully explore spatiotemporal correlation features and making it difficult to capture the dynamic change patterns of the scene; (3) Single deep learning models (such as CNN and LSTM) have inherent limitations—CNN is good at local feature extraction but lacks global dependency modeling, and Transformer can capture global correlation but is sensitive to local details, making it difficult to adapt to the heterogeneous characteristics of multimodal data; (4) In complex scenes (such as semi-obscured outdoor and underground parking garages), there are sensor signal distortions (such as GNSS obstruction and magnetic field interference), and existing models are not robust enough to such interference, resulting in large fluctuations in recognition accuracy. Summary of the Invention

[0005] To address the problems existing in the prior art, the present invention provides a fine-grained indoor and outdoor scene recognition method, device, system, and storage medium.

[0006] To achieve the above objectives, the present invention provides the following solution: A fine-grained indoor and outdoor scene recognition method includes: S1: Collects data from multiple sensors built into the smartphone, including GNSS, IMU, geomagnetism, air pressure, and light intensity. S2: Preprocess the multimodal sensor data, including time interpolation synchronization, filtering and noise reduction, and time-frequency domain feature extraction, to obtain enhanced spatiotemporal feature data; S3: Input the enhanced spatiotemporal feature data into the CNN-ViT-TabTransformer fusion network to achieve real-time recognition of fine-grained indoor and outdoor scenes.

[0007] As a preferred approach, in the CNN-ViT-TabTransformer fusion network, CNN extracts local temporal features, ViT captures global frequency domain dependencies, and the two features are concatenated. Then, TabTransformer mines the deep correlation between air pressure, illumination structured data and scene categories, and finally, the MLP classification head achieves accurate recognition of fine-grained scenes.

[0008] Preferably, the loss function of the CNN-ViT-TabTransformer fusion network is cross-entropy:

[0009] in, N For the sample size, n For the number of categories, For the true distribution, This represents the network output distribution.

[0010] The present invention also provides a fine-grained indoor and outdoor scene recognition device, comprising: The first processing module is used to collect data from multiple sources of sensors built into the smartphone, including GNSS, IMU, geomagnetism, air pressure, and light intensity. The second processing module is used to preprocess the multimodal sensor data, including time interpolation synchronization, filtering and noise reduction, and time-frequency domain feature extraction, to obtain enhanced spatiotemporal feature data. The third processing module is used to input the enhanced spatiotemporal feature data into the CNN-ViT-TabTransformer fusion network to achieve real-time recognition of fine-grained indoor and outdoor scenes.

[0011] As a preferred approach, in the CNN-ViT-TabTransformer fusion network, CNN extracts local temporal features, ViT captures global frequency domain dependencies, and the two features are concatenated. Then, TabTransformer mines the deep correlation between air pressure, illumination structured data and scene categories, and finally, the MLP classification head achieves accurate recognition of fine-grained scenes.

[0012] Preferably, the loss function of the CNN-ViT-TabTransformer fusion network is cross-entropy:

[0013] in, N For the sample size, n For the number of categories, For the true distribution, This represents the network output distribution.

[0014] The present invention also provides a fine-grained indoor and outdoor scene recognition system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a fine-grained indoor and outdoor scene recognition method when executed by the processor.

[0015] The present invention also provides a storage medium storing a computer program, which executes a fine-grained indoor and outdoor scene recognition method when running.

[0016] The technical solution of this invention first constructs a collaborative acquisition mechanism based on multiple sensors in a smartphone. This mechanism can simultaneously acquire and locally store high-frequency data from multiple sensors built into the smartphone, such as GNSS, IMU (accelerometer, gyroscope), geomagnetism, air pressure, and light intensity, in different indoor and outdoor environments, vehicles, and buildings. This provides rich and high-quality basic data for scene recognition. Subsequently, this paper proposes a feature extraction strategy that integrates time interpolation synchronization and time-frequency domain analysis: linear interpolation achieves millisecond-level time alignment of multi-sensor data; exponential moving average filtering eliminates high-frequency noise; and short-time Fourier transform (STFT) is used to extract frequency domain features from accelerometer and gyroscope data, thereby comprehensively capturing the dynamic changes and spatial correlation characteristics of the scene. Based on this, a CNN-ViT-TabTransformer fusion network architecture is designed: CNN extracts local temporal features, ViT captures global frequency domain dependencies, and the features from both are concatenated and input into the TabTransformer to mine the deep correlation between structured data and scene categories. Finally, scene recognition is achieved through an MLP classification head. Using this method, we not only achieved a recognition accuracy of over 99% in tests encompassing 10 fine-grained scenes, but also maintained excellent generalization ability in complex environments, further expanding the application potential of multimodal sensor data in fine-grained scene recognition. It has the following technical effects: 1. A collaborative acquisition mechanism based on multiple sensors of smartphones was constructed to achieve high-frequency synchronous acquisition and integration of multi-source data such as accelerometers, gyroscopes, magnetometers, barometers, optical sensors and GNSS, providing rich and high-quality basic data support for fine-grained scene recognition.

[0017] 2. A multimodal spatiotemporal feature extraction strategy integrating time interpolation synchronization and time-frequency domain analysis is proposed. Compared with traditional single temporal feature extraction methods, it can more comprehensively capture the dynamic change patterns and spatial correlation characteristics of the scene, and significantly improve the effectiveness of feature representation.

[0018] 3. A CNN-ViT-TabTransformer fusion network architecture was designed. By synergistically leveraging the local feature extraction capability of convolutional neural networks, the global dependency modeling advantage of visual Transformers, and the strengths of TabTransformers in processing structured data, the problem of insufficient generalization ability of single models in complex scene recognition was effectively solved, enabling fine-grained scene recognition accuracy to exceed 99%. Attached Figure Description

[0019] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the fine-grained indoor and outdoor scene recognition method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the architecture of the CNN-ViT-TabTransformer fusion network provided in an embodiment of the present invention; Figure 3 A comparison of the recognition accuracy of the CNN-ViT-TabTransformer fusion network provided in this embodiment of the invention with other classic models; Figure 4 The confusion matrix of the CNN-ViT-TabTransformer fusion network provided in this embodiment of the invention when performing different scene recognition. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Example 1 like Figure 1As shown, this invention provides a fine-grained indoor and outdoor scene recognition method. Based on multimodal sensors (accelerometer, gyroscope, magnetometer, barometer, light sensor, screen brightness sensor, orientation sensor, and GNSS) built into a smartphone, high-frequency data is collected in indoor and outdoor scenes, vehicles, and buildings. A time synchronization algorithm is implemented to achieve millisecond-level alignment of multi-sensor data. A filtering algorithm (such as exponential moving average) is used to eliminate high-frequency noise. Short-time Fourier transform (STFT) is combined to extract frequency domain features from accelerometer and gyroscope data, constructing a spatiotemporal multidimensional feature set. The processed data is input into a designed CNN-ViT-TabTransformer fusion network: CNN extracts local temporal features, ViT captures global frequency domain dependencies, and the two features are concatenated. Then, TabTransformer mines the deep correlation between structured data such as air pressure and illumination and scene categories. Finally, an MLP classification head is used to achieve accurate fine-grained scene recognition. The method includes the following steps: S1: Using a multi-sensor data acquisition system based on the Android / iOS platform, it simultaneously collects and stores high-frequency data from multiple sources of sensors, such as GNSS, IMU (accelerometer, gyroscope), geomagnetism, air pressure, and light intensity, built into smartphones in indoor and outdoor, transportation and building scenarios. S2: Preprocess the acquired multimodal sensor data, including time interpolation synchronization, filtering and noise reduction, and time-frequency domain feature extraction, to obtain enhanced spatiotemporal feature data; S3: The preprocessed multimodal feature data is transmitted to the cloud via a mobile communication network and input into the CNN-ViT-TabTransformer fusion network to achieve real-time recognition of fine-grained indoor and outdoor scenes.

[0024] As one embodiment of the present invention, step S1 includes: S1-1: The multi-sensor data acquisition system supports high-frequency synchronous acquisition with a sampling rate of 10-100Hz, covering the following sensor types and parameters: Accelerometer (3-axis, unit m / s²), gyroscope (3-axis, unit rad / s), magnetometer (3-axis, unit μT), barometer (unit hPa), light sensor (unit lux), screen brightness sensor (unit %), orientation sensor (unit °), GNSS sensor (including 6 parameters such as visible satellite number, horizontal velocity, and SNR statistics); S1-2: The fine-grained scenes include 10 scenarios: outdoor semi-obscured, outdoor open, outdoor obscured, driving, cycling, indoor hall, elevator, going down stairs, going up stairs, and underground parking garage.

[0025] As one embodiment of the present invention, step S2 includes: S2-1: Select a specific sensor as the baseline to construct a time axis, and use linear interpolation to complete the asynchronous data from other sensors. For the interpolation time... The interpolation function is: ; in, , They are respectively , Sensor data at any given moment is used to achieve millisecond-level time synchronization through weighted calculations.

[0026] S2-2: High-frequency noise is filtered using the Exponential Moving Average (EMA) algorithm. Let the first... One sensor in The observed value at time is The forgetting factor is The filtered data for: ; Dynamic tracking of recent data is achieved through index weight allocation, preserving the continuity of time series.

[0027] S2-3: Short-Time Fourier Transform (STFT) is used to extract frequency domain features from the preprocessed time-domain data (accelerometer, gyroscope). The transform formula is as follows: ; in The Hanning window function has the following expression: ; The power spectral density of the output spectrum is used as a frequency domain feature.

[0028] As one embodiment of the present invention, in step S3... Figure 2 This is a schematic diagram of the architecture of the CNN-ViT-TabTransformer fusion network provided in this embodiment of the invention. The CNN-ViT-TabTransformer fusion network consists of three core functional modules: a CNN submodule for extracting local temporal features, a ViT submodule for capturing global frequency domain dependencies, and a TabTransformer submodule for multimodal feature fusion and scene classification. The overall structure aims to utilize the complementarity of temporal and frequency domain features simultaneously to improve the accuracy and stability of fine-grained scene recognition.

[0029] (1) The CNN submodule takes temporal augmentation data as input and is composed of TimeConv1, BatchNorm1, MaxPool1, TimeConv2, BatchNorm2, MaxPool2 and Dropout layers connected sequentially. This module extracts local structural information and short-range dynamic change features from the time series signal through continuous convolution, normalization and pooling operations, thereby forming a stable temporal local feature representation.

[0030] (2) The ViT submodule takes frequency domain augmented data as the processing object and consists of a PatchEmbed layer and four cascaded Transformer Encoders. The PatchEmbed layer is responsible for dividing the temporal spectrum into equal-scale patches and mapping them to a high-dimensional embedding space; each subsequent Transformer Encoder consists of LayerNorm, Multi-HeadAttention with self-attention mechanism and feedforward network FFN, which can effectively model the long-distance dependencies between frequency domain features and thus obtain the global semantic representation of the frequency domain.

[0031] (3) The TabTransformer submodule is used to complete multimodal feature fusion and final classification. It consists of six cascaded Transformer Encoders, FinalLayerNorm layers, and MLP mapping layers. Based on a multi-layer self-attention structure, this module performs deep interactive modeling of features from the time domain and frequency domain, which significantly improves the expressive power of multimodal information fusion and makes the fused high-dimensional features more suitable for downstream fine-grained scene recognition tasks.

[0032] The data processing flow of this fusion network is as follows: The enhanced original spatiotemporal feature data is divided into a temporal branch and a frequency branch. The temporal branch data undergoes a two-level convolution-normalization-pooling structure and Dropout operation to obtain local temporal features; the frequency branch data is embedded in blocks using PatchEmbed and then encoded by a four-layer Transformer Encoder to extract global frequency features. The two types of features are then concatenated along the feature dimension and input into the TabTransformer submodule for deep fusion using a six-layer Transformer Encoder. After MLP mapping and FinalLayerNorm normalization, the final output is the recognition results corresponding to ten fine-grained scenes.

[0033] Specifically, it includes the following sub-steps: S3-1: The CNN sub-network in the CNN-ViT-TabTransformer network structure is used to extract local temporal features. It contains two convolutional layers, each followed by a ReLU activation function, a BatchNorm normalization layer, and a max-pooling layer. The output of the convolutional layer is: ; S3-2: The ViT sub-network in the CNN-ViT-TabTransformer network structure is used to extract global features in the frequency domain. It segments the time-frequency spectrum (dimension B×C×T×F) obtained by STFT into patches and maps them to vectors. After adding learnable classification tokens and positional encodings, it is input into a 4-layer Transformer encoder, each layer containing a multi-head self-attention mechanism. ; The formula for calculating attention is as follows: ; in For query vector, For key vectors, The value vector is obtained by multiplying the input vector with the model's three weight vectors. for Vector dimension This is the output weight matrix.

[0034] S3-3: The TabTransformer subnetwork in the CNN-ViT-TabTransformer network structure is used to integrate multimodal features. It concatenates the output features of CNN and ViT, then inputs them into a 6-layer Transformer encoder. After LayerNorm normalization and processing by a feedforward network (FFN), it finally outputs the scene category through an MLP classification head. The loss function used is cross-entropy. ; in N For the sample size, n For the number of categories, For the true distribution, This represents the network output distribution.

[0035] Figure 3 and Figure 4These figures show a comparison of various indicators obtained from scene recognition prediction using the CVT model provided in this embodiment of the invention, and other common models, as well as the confusion matrix of the CVT model prediction. As can be seen from the figures, the fine-grained scene recognition method based on the spatiotemporal features of smartphone multimodal sensors and the CVT model proposed in this invention can achieve more stable and superior prediction results compared to other common models.

[0036] Example 2 The present invention also provides a fine-grained indoor and outdoor scene recognition device, comprising: The first processing module is used to collect data from multiple sources of sensors built into the smartphone, including GNSS, IMU, geomagnetism, air pressure, and light intensity. The second processing module is used to preprocess the multimodal sensor data, including time interpolation synchronization, filtering and noise reduction, and time-frequency domain feature extraction, to obtain enhanced spatiotemporal feature data. The third processing module is used to input the enhanced spatiotemporal feature data into the CNN-ViT-TabTransformer fusion network to achieve real-time recognition of fine-grained indoor and outdoor scenes.

[0037] As one embodiment of the present invention, in the CNN-ViT-TabTransformer fusion network, CNN extracts local temporal features, ViT captures global frequency domain dependencies, the two features are concatenated, and then TabTransformer mines the deep association between air pressure, illumination structured data and scene categories, and finally the MLP classification head achieves accurate recognition of fine-grained scenes.

[0038] As one embodiment of the present invention, the loss function of the CNN-ViT-TabTransformer fusion network adopts cross-entropy:

[0039] in, N For the sample size, n For the number of categories, For the true distribution, This represents the network output distribution.

[0040] Example 3 The present invention also provides a fine-grained indoor and outdoor scene recognition system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a fine-grained indoor and outdoor scene recognition method when executed by the processor.

[0041] Example 4 The present invention also provides a storage medium storing a computer program, which executes a fine-grained indoor and outdoor scene recognition method when running.

[0042] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A fine-grained indoor and outdoor scene recognition method, characterized in that, include: S1: Collects data from multiple sensors built into the smartphone, including GNSS, IMU, geomagnetism, air pressure, and light intensity. S2: Preprocess the multimodal sensor data, including time interpolation synchronization, filtering and noise reduction, and time-frequency domain feature extraction, to obtain enhanced spatiotemporal feature data; S3: Input the enhanced spatiotemporal feature data into the CNN-ViT-TabTransformer fusion network to achieve real-time recognition of fine-grained indoor and outdoor scenes.

2. The fine-grained indoor and outdoor scene recognition method as described in claim 1, characterized in that, In the CNN-ViT-TabTransformer fusion network, CNN extracts local temporal features, ViT captures global frequency domain dependencies, and the two features are concatenated. Then, TabTransformer mines the deep correlation between air pressure, illumination structured data and scene categories, and finally, the MLP classification head achieves accurate recognition of fine-grained scenes.

3. The fine-grained indoor and outdoor scene recognition method as described in claim 2, characterized in that, The loss function of the CNN-ViT-TabTransformer fusion network uses cross-entropy: ； in, N For the sample size, n For the number of categories, For the true distribution, This represents the network output distribution.

4. A fine-grained indoor and outdoor scene recognition device, characterized in that, include: The first processing module is used to collect data from multiple sources of sensors built into the smartphone, including GNSS, IMU, geomagnetism, air pressure, and light intensity. The second processing module is used to preprocess the multimodal sensor data, including time interpolation synchronization, filtering and noise reduction, and time-frequency domain feature extraction, to obtain enhanced spatiotemporal feature data. The third processing module is used to input the enhanced spatiotemporal feature data into the CNN-ViT-TabTransformer fusion network to achieve real-time recognition of fine-grained indoor and outdoor scenes.

5. The fine-grained indoor and outdoor scene recognition device as described in claim 4, characterized in that, In the CNN-ViT-TabTransformer fusion network, CNN extracts local temporal features, ViT captures global frequency domain dependencies, and the two features are concatenated. Then, TabTransformer mines the deep correlation between air pressure, illumination structured data and scene categories, and finally, the MLP classification head achieves accurate recognition of fine-grained scenes.

6. The fine-grained indoor and outdoor scene recognition device as described in claim 5, characterized in that, The loss function of the CNN-ViT-TabTransformer fusion network uses cross-entropy: ； in, N For the sample size, n For the number of categories, For the true distribution, This represents the network output distribution.

7. A fine-grained indoor and outdoor scene recognition system, characterized in that, include: A memory and a processor, wherein the memory stores a computer program executed by the processor, the computer program performing the fine-grained indoor and outdoor scene recognition method as described in any one of claims 1-3 when executed by the processor.

8. A storage medium, characterized in that, The storage medium stores a computer program, which executes the fine-grained indoor and outdoor scene recognition method as described in any one of claims 1-3 when it runs.

Images