A method for monitoring parking space status based on multi-sensor fusion

Abstract

This invention discloses a parking space status monitoring method based on multi-sensor fusion, comprising: Step 1: collecting multi-modal sensor data; Step 2: preprocessing the multi-modal sensor data to generate a standardized multi-modal dataset; Step 3: constructing an improved SwinFusion model, performing multi-modal feature extraction on the standardized multi-modal dataset, and generating an asynchronous attention feature tensor sequence; Step 4: performing multi-scale feature decoding and spatiotemporal mapping to generate a feature mapping map and an occupancy probability map; Step 5: performing status discrimination based on the feature mapping map and the occupancy probability map and outputting parking space status labels; Step 6: generating a real-time parking space status visualization map; Step 7: collecting edge terminal recognition results and sensor feedback data, and performing incremental updates on the improved SwinFusion model. This invention improves the multi-sensor data fusion capability and the accuracy and robustness of parking space status recognition in complex environments.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation and multimodal perception technology, and in particular to a method for monitoring the status of parking spaces based on multi-sensor fusion. Background Technology

[0002] With the rapid development of smart cities and intelligent transportation infrastructure, parking space monitoring and management have gradually become a key aspect of urban traffic governance. Traditional parking status recognition methods mostly rely on single-type sensors, such as geomagnetic detectors or video surveillance systems, which only provide localized sensing capabilities and suffer from low accuracy in complex environments. Existing methods generally suffer from the following problems:

[0003] Single-modal sensors exhibit poor stability in low-light, high-obstruction, or strong-interference scenarios. For example, cameras are greatly affected by weather and lighting changes, and geomagnetic sensors are easily triggered by metal interference, leading to frequent misjudgments of parking space status. Different types of sensors have inherent differences in data format, sampling frequency, and synchronization mechanisms, making it difficult to model multi-source heterogeneous data in a unified manner. This can easily lead to cross-modal alignment mismatches, which in turn affect the fusion effect. Most existing methods rely on static fusion strategies and lack dynamic modeling and adaptive mechanisms. They are not robust enough to deal with diverse vehicle types, complex occupancy behaviors, and frequent sensor anomalies in parking scenarios, making it difficult to achieve stable, real-time, and refined parking status recognition and management.

[0004] Therefore, how to provide a parking space status monitoring method based on multi-sensor fusion is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a parking space status monitoring method based on multi-sensor fusion. This invention fully utilizes intelligent traffic sensing technology, deep learning networks, and edge intelligent computing to construct an improved SwinFusion model, achieving joint modeling and asynchronous alignment of multi-source heterogeneous data such as geomagnetic, infrared, ultrasonic, and image data. The improved SwinFusion model achieves high-precision identification and dynamic visualization of parking space status in complex parking environments through modal window partitioning, modal channel reorganization, visual attention filtering, and asynchronous attention alignment. Furthermore, when deployed at the edge, a threshold and lightweight network collaborative discrimination method is used to improve the real-time performance and energy efficiency of parking space status identification, effectively reducing computational load. It possesses advantages such as high identification accuracy, strong anti-interference capability, flexible deployment, and good real-time performance.

[0006] A parking space status monitoring method based on multi-sensor fusion according to an embodiment of the present invention includes the following steps: Step 1: Collect multimodal sensor data of the target parking area; Step 2: Preprocess the multimodal sensor data, including time synchronization, noise filtering, normalization and tensor quantization, to generate a standardized multimodal dataset; Step 3: Construct an improved SwinFusion model, which includes a modal window partitioning module, a modal channel reorganization module, a visual attention filtering module, and an asynchronous attention alignment module. Perform multimodal feature extraction on the standardized multimodal dataset to generate an asynchronous attention feature tensor sequence. Step 4: Perform multi-scale feature decoding and spatiotemporal mapping on the asynchronous attention feature tensor sequence to generate feature mapping map and occupancy probability map of the parking space area; Step 5: Based on the feature map and occupancy probability map of the parking space area, extract the fused feature vector of the target parking space and perform state discrimination to output the parking space state label; Step 6: Match the parking space status labels with the parking area coordinate information to generate a real-time parking space status visualization map on the edge terminal; Step 7: Collect edge terminal identification results and sensor feedback data, and perform incremental updates on the improved SwinFusion model.

[0007] Optionally, the multimodal sensor data includes image modal data, geomagnetic modal data, infrared modal data, and ultrasonic modal data.

[0008] Optionally, step two specifically includes: Multimodal sensor data is time-synchronized according to sampling time to generate a multimodal time-series sample set; Noise filtering is performed on the multimodal time series sample set to generate a multimodal denoised sample set. Specifically, the image modal data is denoised using the median filtering method, the geomagnetic modal data and infrared modal data are denoised using the moving mean filtering method, and the ultrasonic modal data is denoised using the threshold shearing method to eliminate abrupt interference signals. The minimum-maximum normalization method is used to normalize each modal data separately to form a multimodal normalized sample set; The multimodal normalized sample set is mapped into tensor form. The image modal data is constructed into an image modal tensor sequence according to the number of time steps, image height, image width, and number of image channels. The geomagnetic modal data, infrared modal data, and ultrasonic modal data are constructed into a time step tensor sequence according to the number of time steps and modal feature dimensions. The modal feature dimension refers to the sum of the channel dimensions of the geomagnetic modal data, infrared modal data, and ultrasonic modal data. The time step tensor sequence includes a geomagnetic modal tensor, an infrared modal tensor, and an ultrasonic modal tensor at each time step. The image modal tensor sequence and the time step tensor sequence are organized into a standardized multimodal dataset.

[0009] Optionally, the modal window partitioning module processes the image modal tensor sequence in the standardized multimodal dataset frame by frame to obtain a sliding window partitioning matrix, and performs a non-uniform window partitioning operation on the image modal tensor sequence to generate a set of local feature blocks, specifically: Each frame of the image modality tensor sequence is weighted and compressed along the channel dimension to obtain a single-channel grayscale image; The horizontal and vertical gradient maps of a single-channel grayscale image are calculated using the Sobel edge detection operator. The gradient magnitude map is obtained by summing the squares of the horizontal and vertical gradient maps pixel by pixel and taking the square root. The gradient magnitude map is then normalized pixel by pixel to generate a saliency map. The saliency map is divided into several saliency sub-blocks according to a preset window size, and block average pooling is performed on each saliency sub-block to obtain the average saliency score of each saliency sub-block. The saliency score matrix is ​​then formed according to the original spatial arrangement order. The saliency score matrices of each time frame are used to construct a sliding window partitioning matrix according to the time step. Based on the sliding window partitioning matrix, a non-uniform window partitioning operation is performed on the image modal tensor sequence to generate the image local feature tensor sequence.

[0010] Optionally, the modal channel reconstruction module performs cross-modal alignment and channel reconstruction on the image local feature tensor sequence and the time step tensor sequence to generate a modal fusion feature tensor sequence, specifically: The image local feature tensor sequence and the time step tensor sequence are paired according to the time step to form a modality-aligned feature pair sequence. Each time step modality-aligned feature pair includes the image local feature tensor, the geomagnetic mode tensor, the infrared mode tensor, and the ultrasonic mode tensor. At each time step, the local feature tensor of the image is transformed by a one-dimensional convolution operation to generate an image modality mapping tensor; At each time step, the geomagnetic mode tensor, infrared mode tensor, and ultrasonic mode tensor are linearly transformed to generate the geomagnetic mode mapping tensor, infrared mode mapping tensor, and ultrasonic mode mapping tensor, respectively. The image mode mapping tensor, geomagnetic mode mapping tensor, infrared mode mapping tensor, and ultrasonic mode mapping tensor have the same channel dimension; At each time step, the geomagnetic mode mapping tensor, the infrared mode mapping tensor, and the ultrasonic mode mapping tensor are concatenated to generate a non-image mode combination tensor. At each time step, the image modality mapping tensor is flattened in the spatial dimension to generate a two-dimensional image modality tensor; The two-dimensional image modal tensor and the non-image modal combination tensor are concatenated in the sample dimension, and then linear mixing and weighted recombination between channels are performed through a one-dimensional convolution and ReLU activation function to obtain the modal recombination tensor. The modality reconstruction tensor is restored to the same spatial dimension as the local feature tensor of the image using the Reshape function, forming a modality fusion feature tensor, which is then organized into a sequence of modality fusion feature tensors according to time steps.

[0011] Optionally, the visual attention filtering module uses the image modality mapping tensor as visual guidance and performs feature weighting on the modality fusion feature tensor, specifically: At each time step, the image modality mapping tensor is subjected to average pooling and max pooling along the spatial dimension to generate two one-dimensional compressed vectors. The two one-dimensional compressed vectors are then passed through one-dimensional convolution to output two activation vectors. The two activation vectors are then summed and passed through the Sigmoid activation function to generate channel attention weight vectors. The image modality mapping tensor is subjected to average pooling and max pooling along the channel dimension to obtain the spatial average map and the spatial max map respectively. The spatial average map and the spatial max map are concatenated along the channel dimension and then generated by two-dimensional convolution to generate a spatial attention activation map. The spatial attention activation map is then activated by Sigmoid to generate a spatial attention weight matrix. The channel attention weight vector is extended into a three-dimensional weight tensor that is consistent with the modality fusion feature tensor in the channel dimension and has a single scalar in the spatial dimension, thus obtaining the channel attention weight tensor. The spatial attention matrix is ​​extended into a three-dimensional weight tensor that has the same spatial dimension as the modality fusion feature tensor and a single scalar channel dimension, thus obtaining the spatial attention weight tensor. The modality fusion feature tensor is weighted channel-wise and pixel-wise by channel attention weight tensor and spatial attention weight tensor respectively to obtain the visual attention enhancement tensor, which is then organized into a visual attention enhancement tensor sequence according to time steps.

[0012] Optionally, the asynchronous attention alignment module performs asynchronous attention operations on the visual attention enhancement tensor sequence, specifically: At each time step, the visual attention enhancement tensor is subjected to GEM pooling along the spatial dimension to generate modal description vectors, and the modal description vectors are organized into a sequence of modal description vectors at each time step; The modal description vector sequence is mapped to query sequence, key sequence and value sequence through three sets of shared one-dimensional convolutions, and the attention weight between the current time step and the historical time step is calculated based on dot product attention to form an asynchronous attention weight matrix; A weighted summation of the value sequence is performed based on the asynchronous attention weight matrix to generate an asynchronous attention feature sequence; The asynchronous attention feature sequence is input into a fully connected mapping layer, and a linear mapping is performed at each time step to generate tensorized asynchronous attention features. The tensorized asynchronous attention features are then reshaped using a reshape function to be consistent with the dimension of the visual attention enhancement tensor, and organized into an asynchronous attention feature tensor sequence according to the time steps.

[0013] Optionally, step four specifically includes: At each time step, the asynchronous attention feature tensor is used to extract multi-scale features through two-dimensional convolution with kernel sizes of 3×3, 5×5 and 7×7 to obtain small-scale feature tensors, medium-scale feature tensors and large-scale feature tensors, and then concatenated in the channel dimension to obtain scale fusion feature tensors. The scale-fused feature tensor is compressed and linearly corrected by one-dimensional convolution and ReLU activation function to obtain a compressed feature tensor. The compressed feature tensor is upsampled to restore the feature map of the parking space region with the same spatial size as the original image and a channel number of 1: Perform a 1×1 convolution and sigmoid activation on the feature map to generate a preliminary occupancy probability map; The occupancy probability map of the parking space area is generated by normalizing each pixel value in the initial occupancy probability map using the Poisson normalization function.

[0014] Optionally, step five specifically includes: At each time step, the two-dimensional spatial region corresponding to the target parking space is extracted based on the feature map, and average pooling is performed to obtain the structural feature vector; Extract the pixel sub-blocks corresponding to the target parking space from the occupancy probability map, and calculate the average occupancy probability value of the pixel sub-blocks; The structural feature vector and the occupancy probability value are concatenated to generate a fused feature vector. Set state threshold With state threshold If the average occupancy probability value is less than or equal to the state threshold If the average occupancy probability value is greater than or equal to the state threshold, then a parking space status label L0 is generated: vacant state; Then, a parking space status label L1 is generated: occupancy status; if the average occupancy probability value is greater than the status threshold... And less than the state threshold Then, a parking space status label L2 is generated: Abnormal status; If the parking space status label is L2: abnormal state, then the corresponding fused feature vector is subjected to a three-class classification operation through an MLP structure to obtain a three-dimensional probability vector, and the index of the maximum probability value of the three-dimensional probability vector is used as the parking space status label; each of the three-dimensional probability vectors corresponds to the probability value of the idle state, the probability value of the occupied state, and the probability value of the abnormal state, respectively.

[0015] Optionally, step six specifically includes: The parking space status label of the target parking space is matched with the spatial coordinate information to construct a parking space labeling information unit. The labeling information unit includes the parking space number, coordinate bounding box and parking space status label. Based on the parking space labeling information unit, all parking space status information is rendered into a parking area image to generate a parking space status visualization map. In the parking space status visualization map, the idle state is rendered in green, the occupied state is rendered in red, and the abnormal state is rendered in yellow. The parking space status visualization map is stored at the edge terminal, and the parking space status label, status probability value and coordinate boundary information at each time step are cached in the edge database in a structured data format.

[0016] The beneficial effects of this invention are: This invention discloses a parking space status monitoring method based on multi-sensor fusion, aiming to solve problems such as insufficient fusion of multi-source heterogeneous data and susceptibility to interference in status recognition. First, by deploying an image acquisition camera, a geomagnetic sensor, an infrared detector, and an ultrasonic ranging module, image modal data, geomagnetic modal data, infrared modal data, and ultrasonic modal data of the target parking area are collected, forming a multi-modal data source covering multiple dimensions such as viewing angle, magnetic signal, temperature change, and distance perception.

[0017] Secondly, an improved SwinFusion model is employed for deep fusion of multimodal data. This improved SwinFusion model designs a modality-guided sliding window partitioning mechanism in the feature modeling stage. Based on the perceptual distribution and spatial correlation of different modalities, it adaptively generates perceptual window boundaries, achieving feature alignment across heterogeneous modalities. Through modal channel recombination, multimodal feature channels are rearranged and combined according to semantic consistency, improving the alignment accuracy and semantic aggregation capability across modalities. In the feature fusion stage, an image vision-driven attention filtering module is introduced. Based on the image modality mapping tensor, a channel attention weight tensor is generated and applied to the modality fusion feature tensor, thereby strengthening the saliency of the corresponding image modality's channel response, suppressing potentially invalid or interfered channel feature responses, and improving the discriminativeness and robustness of the fused representation. To address the issue of inconsistent sampling frequencies across modalities, asynchronous cross-modal attention alignment is adopted, combined with temporal embedding and deformable attention operations, to achieve keyframe semantic alignment and temporal consistency modeling.

[0018] In the state discrimination stage, a discrimination method based on threshold rules and lightweight neural networks is used to quickly identify the "occupied," "vacant," and "abnormal" states of parking spaces, and supports state visualization output. This method can still achieve highly robust and accurate parking space state recognition in complex occlusion, low-light, or electromagnetic interference environments, and has excellent edge deployment adaptability and engineering practical value.

[0019] In summary, this invention achieves deep fusion and robust identification of multi-sensor data, improving the accuracy and real-time performance of parking space status monitoring in complex environments. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0021] Figure 1 This is a schematic diagram of a parking space status monitoring method based on multi-sensor fusion proposed in this invention; Figure 2 This is a flowchart of the improved SwinFusion model structure in a parking space status monitoring method based on multi-sensor fusion proposed in this invention. Figure 3 This is a flowchart of the modal channel reconfiguration module in a parking space status monitoring method based on multi-sensor fusion proposed in this invention; Figure 4 This is a flowchart of the state discrimination and visualization process in a parking space state monitoring method based on multi-sensor fusion proposed in this invention. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0023] refer to Figures 1-4 A parking space status monitoring method based on multi-sensor fusion includes the following steps: Step 1: Collect multimodal sensor data of the target parking area; Step 2: Preprocess the multimodal sensor data, including time synchronization, noise filtering, normalization and tensor quantization, to generate a standardized multimodal dataset; Step 3: Construct an improved SwinFusion model, which includes a modal window partitioning module, a modal channel reorganization module, a visual attention filtering module, and an asynchronous attention alignment module. Perform multimodal feature extraction on the standardized multimodal dataset to generate an asynchronous attention feature tensor sequence. Step 4: Perform multi-scale feature decoding and spatiotemporal mapping on the asynchronous attention feature tensor sequence to generate feature mapping map and occupancy probability map of the parking space area; Step 5: Based on the feature map and occupancy probability map of the parking space area, extract the fused feature vector of the target parking space and perform state discrimination to output the parking space state label; Step 6: Match the parking space status labels with the parking area coordinate information to generate a real-time parking space status visualization map on the edge terminal; Step 7: Collect edge terminal identification results and sensor feedback data, and perform incremental updates on the improved SwinFusion model.

[0024] In this embodiment, the multimodal sensor data includes image modal data, geomagnetic modal data, infrared modal data, and ultrasonic modal data.

[0025] In this invention, the image modal data is a top-down view of the parking space captured by a camera installed above the parking space; the geomagnetic modal data is the magnetic field change collected by a ground-embedded triaxial magnetic flux sensor; the infrared modal data is the ground temperature radiation intensity collected by an infrared pyroelectric sensor installed above or to the side of the parking space, used to sense residual engine heat and the movement of people or objects; the ultrasonic modal data is the vertical distance information collected above the parking space by a periodically transmitting and receiving sound wave signals by an ultrasonic ranging module fixed at the front or top of the parking space, used to determine whether there is a parked vehicle or an object obstructing the view.

[0026] In this embodiment, step two specifically includes: Multimodal sensor data is time-synchronized according to sampling time to generate a multimodal time-series sample set; Noise filtering is performed on the multimodal time series sample set to generate a multimodal denoised sample set. Specifically, the image modal data is denoised using the median filtering method, the geomagnetic modal data and infrared modal data are denoised using the moving mean filtering method, and the ultrasonic modal data is denoised using the threshold shearing method to eliminate abrupt interference signals. The minimum-maximum normalization method is used to normalize each modal data separately to form a multimodal normalized sample set; The multimodal normalized sample set is mapped into tensor form. The image modal data is constructed into an image modal tensor sequence according to the number of time steps, image height, image width, and number of image channels. The geomagnetic modal data, infrared modal data, and ultrasonic modal data are constructed into a time step tensor sequence according to the number of time steps and modal feature dimensions. The modal feature dimension refers to the sum of the channel dimensions of the geomagnetic modal data, infrared modal data, and ultrasonic modal data. The time step tensor sequence includes a geomagnetic modal tensor, an infrared modal tensor, and an ultrasonic modal tensor at each time step. The image modal tensor sequence and the time step tensor sequence are organized into a standardized multimodal dataset.

[0027] In this embodiment, the modal window partitioning module processes the image modal tensor sequence in the standardized multimodal dataset frame by frame to obtain a sliding window partitioning matrix, and performs a non-uniform window partitioning operation on the image modal tensor sequence to generate a set of local feature blocks, specifically: Each frame of the image modality tensor sequence is weighted and compressed along the channel dimension to obtain a single-channel grayscale image; The horizontal and vertical gradient maps of a single-channel grayscale image are calculated using the Sobel edge detection operator. The gradient magnitude map is obtained by summing the squares of the horizontal and vertical gradient maps pixel by pixel and taking the square root. The gradient magnitude map is then normalized pixel by pixel to generate a saliency map. The saliency map is divided into several saliency sub-blocks according to a preset window size, and block average pooling is performed on each saliency sub-block to obtain the average saliency score of each saliency sub-block. The saliency score matrix is ​​then formed according to the original spatial arrangement order. The saliency score matrices of each time frame are used to construct a sliding window partitioning matrix according to the time step. Based on the sliding window partitioning matrix, a non-uniform window partitioning operation is performed on the image modal tensor sequence to generate the image local feature tensor sequence.

[0028] This invention introduces an adaptive sliding window mechanism based on saliency maps, which dynamically adjusts the window partitioning method according to the texture complexity and edge distribution of image modalities at different time frames. This enables fine-grained feature extraction of key regions and compressed representation of non-key regions. This design effectively avoids the boundary feature loss and spatial redundancy problems caused by traditional fixed-window methods, making local feature blocks more representative and structurally sensitive, thereby improving the spatial alignment accuracy and overall recognition robustness in the multimodal fusion stage.

[0029] In this embodiment, the modal channel reconstruction module performs cross-modal alignment and channel reconstruction on the image local feature tensor sequence and the time step tensor sequence to generate a modal fusion feature tensor sequence, specifically: The image local feature tensor sequence and the time step tensor sequence are paired according to the time step to form a modality-aligned feature pair sequence. Each time step modality-aligned feature pair includes the image local feature tensor, the geomagnetic mode tensor, the infrared mode tensor, and the ultrasonic mode tensor. At each time step, the local feature tensor of the image is transformed by a one-dimensional convolution operation to generate an image modality mapping tensor; At each time step, the geomagnetic mode tensor, infrared mode tensor, and ultrasonic mode tensor are linearly transformed to generate the geomagnetic mode mapping tensor, infrared mode mapping tensor, and ultrasonic mode mapping tensor, respectively. The image mode mapping tensor, geomagnetic mode mapping tensor, infrared mode mapping tensor, and ultrasonic mode mapping tensor have the same channel dimension; At each time step, the geomagnetic mode mapping tensor, the infrared mode mapping tensor, and the ultrasonic mode mapping tensor are concatenated to generate a non-image mode combination tensor. At each time step, the image modality mapping tensor is flattened in the spatial dimension to generate a two-dimensional image modality tensor; The two-dimensional image modal tensor and the non-image modal combination tensor are concatenated in the sample dimension, and then linear mixing and weighted recombination between channels are performed through a one-dimensional convolution and ReLU activation function to obtain the modal recombination tensor. The modality reconstruction tensor is restored to the same spatial dimension as the local feature tensor of the image using the Reshape function, forming a modality fusion feature tensor, which is then organized into a sequence of modality fusion feature tensors according to time steps.

[0030] In this invention, by unifying and linearly recombining image modalities and non-image modalities in both channel and spatial dimensions, semantic alignment and collaborative expression between cross-modal features are effectively achieved. During the fusion process, inter-channel linear mixing and weighting are introduced, enabling deep fusion of spatial detail features of the image modality with environmental perception information from geomagnetic, infrared, and ultrasonic modalities at the tensor level. This improves the semantic integrity and discriminability of the fused features, enhancing the accuracy and robustness of the state discrimination process.

[0031] In this embodiment, the visual attention filtering module uses the image modality mapping tensor as visual guidance and performs feature weighting on the modality fusion feature tensor, specifically: At each time step, the image modality mapping tensor is subjected to average pooling and max pooling along the spatial dimension to generate two one-dimensional compressed vectors. The two one-dimensional compressed vectors are then passed through one-dimensional convolution to output two activation vectors. The two activation vectors are then summed and passed through the Sigmoid activation function to generate channel attention weight vectors. The image modality mapping tensor is subjected to average pooling and max pooling along the channel dimension to obtain the spatial average map and the spatial max map respectively. The spatial average map and the spatial max map are concatenated along the channel dimension and then generated by two-dimensional convolution to generate a spatial attention activation map. The spatial attention activation map is then activated by Sigmoid to generate a spatial attention weight matrix. The channel attention weight vector is extended into a three-dimensional weight tensor that is consistent with the modality fusion feature tensor in the channel dimension and has a single scalar in the spatial dimension, thus obtaining the channel attention weight tensor. The spatial attention matrix is ​​extended into a three-dimensional weight tensor that has the same spatial dimension as the modality fusion feature tensor and a single scalar channel dimension, thus obtaining the spatial attention weight tensor. The modality fusion feature tensor is weighted channel-wise and pixel-wise by channel attention weight tensor and spatial attention weight tensor respectively to obtain the visual attention enhancement tensor, which is then organized into a visual attention enhancement tensor sequence according to time steps.

[0032] This invention uses image modality mapping tensors as guidance and combines channel attention and spatial attention for joint modeling, enabling accurate identification and enhancement of salient regions or channel responses in multimodal fusion features. By introducing attention weights for image modalities in both spatial and channel dimensions, invalid features and interfering information are suppressed, effectively strengthening the expression of salient features highly correlated with the target parking state. This module significantly improves the representational and discriminative power of fused features, enhancing the model's recognition stability and robustness in complex scenarios such as low light, occlusion, and geomagnetic interference.

[0033] In this embodiment, the asynchronous attention alignment module performs asynchronous attention operations on the visual attention enhancement tensor sequence, specifically as follows: At each time step, the visual attention enhancement tensor is subjected to GEM pooling along the spatial dimension to generate modality description vectors, and these modality description vectors are organized into a sequence of modality description vectors over the time steps. ; in, The first tensor in the visual attention enhancement tensor The pixel value at each spatial location. This represents the total number of pixels in the spatial dimension. These are learnable pooling parameters; The modal description vector sequence is mapped to query sequence, key sequence and value sequence through three sets of shared one-dimensional convolutions, and the attention weight between the current time step and the historical time step is calculated based on dot product attention to form an asynchronous attention weight matrix; A weighted summation of the value sequence is performed based on the asynchronous attention weight matrix to generate an asynchronous attention feature sequence; The asynchronous attention feature sequence is input into a fully connected mapping layer, and a linear mapping is performed at each time step to generate tensorized asynchronous attention features. The tensorized asynchronous attention features are then reshaped using a reshape function to be consistent with the dimension of the visual attention enhancement tensor, and organized into an asynchronous attention feature tensor sequence according to the time steps.

[0034] In this invention, the asynchronous attention alignment module uses GEM pooling to extract modal description vectors at key moments, and achieves dynamic correlation modeling and important frame enhancement between features at different time steps through temporal attention matching of query, key, and value sequences. While ensuring feature continuity, it enhances semantic consistency across time steps, thereby significantly improving the model's response sensitivity to dynamic parking state changes and its ability to fuse cross-temporal features, ensuring that it can still output high-precision state recognition results stably under asynchronous perception conditions.

[0035] In this embodiment, step four specifically includes: At each time step, the asynchronous attention feature tensor is used to extract multi-scale features through two-dimensional convolution with kernel sizes of 3×3, 5×5 and 7×7 to obtain small-scale feature tensors, medium-scale feature tensors and large-scale feature tensors, and then concatenated in the channel dimension to obtain scale fusion feature tensors. The scale-fused feature tensor is compressed and linearly corrected by one-dimensional convolution and ReLU activation function to obtain a compressed feature tensor. The compressed feature tensor is upsampled to restore the feature map of the parking space region with the same spatial size as the original image and a channel number of 1: Perform a 1×1 convolution and sigmoid activation on the feature map to generate a preliminary occupancy probability map; The occupancy probability map of the parking space area is generated by normalizing each pixel value in the initial occupancy probability map using the Poisson normalization function.

[0036] In this embodiment, step five specifically includes: At each time step, the two-dimensional spatial region corresponding to the target parking space is extracted based on the feature map, and average pooling is performed to obtain the structural feature vector; Extract the pixel sub-blocks corresponding to the target parking space from the occupancy probability map, and calculate the average occupancy probability value of the pixel sub-blocks; The structural feature vector and the occupancy probability value are concatenated to generate a fused feature vector. Set state threshold With state threshold If the average occupancy probability value is less than or equal to the state threshold If the average occupancy probability value is greater than or equal to the state threshold, then a parking space status label L0 is generated: vacant state; Then, a parking space status label L1 is generated: occupancy status; if the average occupancy probability value is greater than the status threshold... And less than the state threshold Then, a parking space status label L2 is generated: Abnormal status; If the parking space status label is L2: abnormal state, then the corresponding fused feature vector is subjected to a three-class classification operation through an MLP structure to obtain a three-dimensional probability vector, and the index of the maximum probability value of the three-dimensional probability vector is used as the parking space status label; each of the three-dimensional probability vectors corresponds to the probability value of the idle state, the probability value of the occupied state, and the probability value of the abnormal state, respectively.

[0037] In this invention, feature maps are used to extract spatial structure features of parking spaces, while probability information of local areas of parking spaces is obtained from occupancy probability maps. These are then concatenated to generate a fused feature vector, enabling the model to consider both spatial structure and probability intensity features when determining the parking status. A dual threshold is designed to classify the status into three categories: idle, occupied, and abnormal. For abnormal states, an MLP structure is further introduced for three-class probabilistic inference, ensuring that the model can still output stable and accurate status labels under conditions of boundary ambiguity and perceptual interference. This effectively improves the robustness and error correction capability of parking status recognition in complex environments.

[0038] In this embodiment, step six specifically includes: The parking space status label of the target parking space is matched with the spatial coordinate information to construct a parking space labeling information unit. The labeling information unit includes the parking space number, coordinate bounding box and parking space status label. Based on the parking space labeling information unit, all parking space status information is rendered into a parking area image to generate a parking space status visualization map. In the parking space status visualization map, the idle state is rendered in green, the occupied state is rendered in red, and the abnormal state is rendered in yellow. The parking space status visualization map is stored at the edge terminal, and the parking space status label, status probability value and coordinate boundary information at each time step are cached in the edge database in a structured data format.

[0039] Example 1: To verify the feasibility of this invention in practice, it was applied to an intelligent renovation project of an underground parking garage in a city's intelligent parking system. This renovation project included multiple enclosed parking units with poor lighting conditions, obstructions such as pipe supports and pillars in some areas, and geomagnetic disturbances and signal reflections, representing a typical challenging scenario for parking space status recognition in a complex environment.

[0040] During implementation, a multimodal sensing device consisting of an image acquisition camera, a geomagnetic sensor, an infrared sensing module, and an ultrasonic ranging sensor was deployed in each parking space. All modal data were uniformly accessed and preprocessed at edge computing nodes, including time synchronization, noise filtering, normalization, and tensor quantization, forming a standardized modal dataset. The multimodal data was then input into an improved SwinFusion model, which sequentially performed modal guidance window partitioning, modal channel reorganization, visual attention filtering, and asynchronous attention alignment to achieve deep fusion and temporal consistency modeling of multimodal features, outputting an asynchronous attention feature tensor sequence. After multi-scale feature decoding and spatiotemporal mapping, the asynchronous attention feature tensor sequence generated a feature mapping map and occupancy probability map of the parking space area. Based on a threshold and lightweight neural network collaborative discrimination strategy, it output "idle," "occupied," or "abnormal" status labels. Simultaneously, a parking space status visualization map was generated in real time at the edge terminal, realizing intelligent recognition and dynamic display of parking space status.

[0041] In practical application, data from 100 parking spaces in the renovation project were collected for 15 consecutive days, with data samples collected every 5 seconds, resulting in approximately 259,200 valid samples. The method of this invention was compared with three mainstream parking status recognition schemes: Scheme A: traditional single image recognition method, Scheme B: image + geomagnetic dual-modal fusion method, and Scheme C: multimodal fusion method with Transformer structure. The experimental results are shown in Table 1.

[0042] Table 1. Performance Comparison of the Invention Method and Comparative Schemes in Parking Space Status Recognition

[0043] As shown in Table 1, the method of this invention significantly outperforms the comparative schemes in multiple performance indicators. Specifically, the method of this invention significantly outperforms comparative schemes A, B, and C in several key performance indicators. In terms of average recognition accuracy, the method of this invention reaches 96.8%, an improvement of 4.3 percentage points compared to the best comparative scheme C. Regarding the abnormal state recognition rate, the method of this invention reaches 85.1%, far exceeding the 67.3% of comparative scheme C, an improvement of 17.8 percentage points, indicating that the improved SwinFusion model has stronger abnormal scene understanding and interference recognition capabilities. In terms of occupancy false positive rate and idle false positive rate, the method of this invention is 1.9% and 2.4% respectively, significantly lower than the 3.7% and 4.9% of comparative scheme C, indicating that the method of this invention is more accurate and reliable in state discrimination under occlusion, low light, and multi-vehicle intersection scenarios. In terms of model inference latency, the present invention achieves 46.2ms, which is slightly higher than schemes A and B, but significantly lower than scheme C by 28.4ms. This indicates that the present invention maintains recognition accuracy while achieving fast response and has good adaptability to edge deployment. Regarding the false alarm rate, the present invention achieves 2.1%, significantly better than the three comparative schemes, demonstrating that the present invention can effectively avoid false alarms in dynamic scenarios.

[0044] The method of this invention combines high-precision recognition, low false positives, rapid response, and anomaly detection capabilities, making it suitable for real-time parking space status monitoring needs in complex urban parking scenarios.

[0045] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A parking space status monitoring method based on multi-sensor fusion, characterized in that, Includes the following steps: Step 1: Collect multimodal sensor data of the target parking area; Step 2: Preprocess the multimodal sensor data, including time synchronization, noise filtering, normalization and tensor quantization, to generate a standardized multimodal dataset; Step 3: Construct an improved SwinFusion model, which includes a modal window partitioning module, a modal channel reorganization module, a visual attention filtering module, and an asynchronous attention alignment module. Perform multimodal feature extraction on the standardized multimodal dataset to generate an asynchronous attention feature tensor sequence. The modal window partitioning module processes the image modal tensor sequence in the standardized multimodal dataset frame by frame to obtain a sliding window partitioning matrix, and performs a non-uniform window partitioning operation on the image modal tensor sequence to generate a set of local feature blocks, specifically: Each frame of the image modality tensor sequence is weighted and compressed along the channel dimension to obtain a single-channel grayscale image; The horizontal and vertical gradient maps of a single-channel grayscale image are calculated using the Sobel edge detection operator. The gradient magnitude map is obtained by summing the squares of the horizontal and vertical gradient maps pixel by pixel and taking the square root. The gradient magnitude map is then normalized pixel by pixel to generate a saliency map. The saliency map is divided into several saliency sub-blocks according to a preset window size, and block average pooling is performed on each saliency sub-block to obtain the average saliency score of each saliency sub-block. The saliency score matrix is ​​then formed according to the original spatial arrangement order. The saliency score matrices of each time frame are used to construct a sliding window partitioning matrix according to the time steps. Based on the sliding window partitioning matrix, a non-uniform window partitioning operation is performed on the image modal tensor sequence to generate the image local feature tensor sequence. The modal channel reconstruction module performs cross-modal alignment and channel reconstruction between the image local feature tensor sequence and the time step tensor sequence to generate a modal fusion feature tensor sequence, specifically: The image local feature tensor sequence and the time step tensor sequence are paired according to the time step to form a modality-aligned feature pair sequence. Each time step modality-aligned feature pair includes the image local feature tensor, the geomagnetic mode tensor, the infrared mode tensor, and the ultrasonic mode tensor. At each time step, the local feature tensor of the image is transformed by a one-dimensional convolution operation to generate an image modality mapping tensor; At each time step, the geomagnetic mode tensor, infrared mode tensor, and ultrasonic mode tensor are linearly transformed to generate the geomagnetic mode mapping tensor, infrared mode mapping tensor, and ultrasonic mode mapping tensor, respectively. The image mode mapping tensor, geomagnetic mode mapping tensor, infrared mode mapping tensor, and ultrasonic mode mapping tensor have the same channel dimension; At each time step, the geomagnetic mode mapping tensor, the infrared mode mapping tensor, and the ultrasonic mode mapping tensor are concatenated to generate a non-image mode combination tensor. At each time step, the image modality mapping tensor is flattened in the spatial dimension to generate a two-dimensional image modality tensor; The two-dimensional image modal tensor and the non-image modal combination tensor are concatenated in the sample dimension, and then linear mixing and weighted recombination between channels are performed through a one-dimensional convolution and ReLU activation function to obtain the modal recombination tensor. The modality reconstruction tensor is restored to the same spatial dimension as the local feature tensor of the image using the Reshape function, forming a modality fusion feature tensor, which is then organized into a sequence of modality fusion feature tensors according to time steps; The visual attention filtering module uses the image modality mapping tensor as visual guidance and performs feature weighting on the modality fusion feature tensor, specifically: At each time step, the image modality mapping tensor is subjected to average pooling and max pooling along the spatial dimension to generate two one-dimensional compressed vectors. The two one-dimensional compressed vectors are then passed through one-dimensional convolution to output two activation vectors. The two activation vectors are then summed and passed through the Sigmoid activation function to generate channel attention weight vectors. The image modality mapping tensor is subjected to average pooling and max pooling along the channel dimension to obtain the spatial average map and the spatial max map respectively. The spatial average map and the spatial max map are concatenated along the channel dimension and then generated by two-dimensional convolution to generate a spatial attention activation map. The spatial attention activation map is then activated by Sigmoid to generate a spatial attention weight matrix. The channel attention weight vector is extended into a three-dimensional weight tensor that is consistent with the modality fusion feature tensor in the channel dimension and has a single scalar in the spatial dimension, thus obtaining the channel attention weight tensor. The spatial attention matrix is ​​extended into a three-dimensional weight tensor that has the same spatial dimension as the modality fusion feature tensor and a single scalar channel dimension, thus obtaining the spatial attention weight tensor. The modality fusion feature tensor is weighted channel-wise and pixel-wise by channel attention weight tensor and spatial attention weight tensor respectively to obtain the visual attention enhancement tensor, and then organized into a visual attention enhancement tensor sequence according to time steps. The asynchronous attention alignment module performs asynchronous attention operations on the visual attention enhancement tensor sequence, specifically as follows: At each time step, the visual attention enhancement tensor is subjected to GEM pooling along the spatial dimension to generate modal description vectors, and the modal description vectors are organized into a sequence of modal description vectors at each time step; The modal description vector sequence is mapped to query sequence, key sequence and value sequence through three sets of shared one-dimensional convolutions, and the attention weight between the current time step and the historical time step is calculated based on dot product attention to form an asynchronous attention weight matrix; A weighted summation of the value sequence is performed based on the asynchronous attention weight matrix to generate an asynchronous attention feature sequence; The asynchronous attention feature sequence is input into a fully connected mapping layer, and a linear mapping is performed at each time step to generate tensorized asynchronous attention features. The tensorized asynchronous attention features are reshaped using a reshape function to be consistent with the dimension of the visual attention enhancement tensor, and then organized into an asynchronous attention feature tensor sequence according to the time steps. Step 4: Perform multi-scale feature decoding and spatiotemporal mapping on the asynchronous attention feature tensor sequence to generate feature mapping map and occupancy probability map of the parking space area; Step 5: Based on the feature map and occupancy probability map of the parking space area, extract the fused feature vector of the target parking space and perform state discrimination to output the parking space state label; Step 6: Match the parking space status labels with the parking area coordinate information to generate a real-time parking space status visualization map on the edge terminal; Step 7: Collect edge terminal identification results and sensor feedback data, and perform incremental updates on the improved SwinFusion model.

2. The parking space status monitoring method based on multi-sensor fusion according to claim 1, characterized in that, The multimodal sensor data includes image modal data, geomagnetic modal data, infrared modal data, and ultrasonic modal data.

3. The parking space status monitoring method based on multi-sensor fusion according to claim 1, characterized in that, Step two specifically includes: Multimodal sensor data is time-synchronized according to sampling time to generate a multimodal time-series sample set; Noise filtering is performed on the multimodal time series sample set to generate a multimodal denoised sample set. Specifically, the image modal data is denoised using the median filtering method, the geomagnetic modal data and infrared modal data are denoised using the moving mean filtering method, and the ultrasonic modal data is denoised using the threshold shearing method to eliminate abrupt interference signals. The minimum-maximum normalization method is used to normalize each modal data separately to form a multimodal normalized sample set; The multimodal normalized sample set is mapped into tensor form. The image modal data is constructed into an image modal tensor sequence according to the number of time steps, image height, image width, and number of image channels. The geomagnetic modal data, infrared modal data, and ultrasonic modal data are constructed into a time step tensor sequence according to the number of time steps and modal feature dimensions. The modal feature dimension refers to the sum of the channel dimensions of the geomagnetic modal data, infrared modal data, and ultrasonic modal data. The time step tensor sequence includes a geomagnetic modal tensor, an infrared modal tensor, and an ultrasonic modal tensor at each time step. The image modal tensor sequence and the time step tensor sequence are organized into a standardized multimodal dataset.

4. The parking space status monitoring method based on multi-sensor fusion according to claim 1, characterized in that, Step four specifically includes: At each time step, the asynchronous attention feature tensor is used to extract multi-scale features through two-dimensional convolution with kernel sizes of 3×3, 5×5 and 7×7 to obtain small-scale feature tensors, medium-scale feature tensors and large-scale feature tensors, and then concatenated in the channel dimension to obtain scale fusion feature tensors. The scale-fused feature tensor is compressed and linearly corrected by one-dimensional convolution and ReLU activation function to obtain a compressed feature tensor. The compressed feature tensor is upsampled to restore the feature map of the parking space region with the same spatial size as the original image and a channel number of 1: Perform a 1×1 convolution and sigmoid activation on the feature map to generate a preliminary occupancy probability map; The occupancy probability map of the parking space area is generated by normalizing each pixel value in the initial occupancy probability map using the Poisson normalization function.

5. The parking space status monitoring method based on multi-sensor fusion according to claim 1, characterized in that, Step five specifically includes: At each time step, the two-dimensional spatial region corresponding to the target parking space is extracted based on the feature map, and average pooling is performed to obtain the structural feature vector; Extract the pixel sub-blocks corresponding to the target parking space from the occupancy probability map, and calculate the average occupancy probability value of the pixel sub-blocks; The structural feature vector and the occupancy probability value are concatenated to generate a fused feature vector. Set state threshold With state threshold If the average occupancy probability value is less than or equal to the state threshold If the average occupancy probability value is greater than or equal to the state threshold, then a parking space status label L0 is generated: vacant state; Then, a parking space status label L1 is generated: occupancy status; if the average occupancy probability value is greater than the status threshold... And less than the state threshold Then, a parking space status label L2 is generated: Abnormal status; If the parking space status label is L2: abnormal state, then the corresponding fused feature vector is subjected to a three-class classification operation through an MLP structure to obtain a three-dimensional probability vector, and the index of the maximum probability value of the three-dimensional probability vector is used as the parking space status label; each of the three-dimensional probability vectors corresponds to the probability value of the idle state, the probability value of the occupied state, and the probability value of the abnormal state, respectively.

6. The parking space status monitoring method based on multi-sensor fusion according to claim 1, characterized in that, Step six specifically includes: The parking space status label of the target parking space is matched with the spatial coordinate information to construct a parking space labeling information unit. The labeling information unit includes the parking space number, coordinate bounding box and parking space status label. Based on the parking space labeling information unit, all parking space status information is rendered into a parking area image to generate a parking space status visualization map. In the parking space status visualization map, the idle state is rendered in green, the occupied state is rendered in red, and the abnormal state is rendered in yellow. The parking space status visualization map is stored at the edge terminal, and the parking space status label, status probability value and coordinate boundary information at each time step are cached in the edge database in a structured data format.

Images