Parking space state monitoring method based on multi-sensor fusion

Abstract

The invention discloses a parking space state monitoring method based on multi-sensor fusion. The parking space state monitoring method comprises the steps of 1, collecting multi-modal sensor data; 2, preprocessing the multi-modal sensor data to generate a standardized multi-modal data set; step 3, constructing an improved SwinFusion model, performing multi-modal feature extraction on the standardized multi-modal data set, and generating an asynchronous attention feature tensor sequence; 4, multi-scale feature decoding and space-time mapping are executed, and a feature mapping graph and an occupation probability graph are generated; 5, based on the feature mapping graph and the occupancy probability graph, executing state judgment and outputting a parking space state label; step 6, generating a real-time parking space state visual graph; and step 7, collecting an edge terminal identification result and sensor feedback data, and executing incremental updating on the improved SwinFusion model. According to the invention, the multi-sensing data fusion capability and the parking space state identification accuracy and robustness in a complex environment are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent transportation and multi-modal perception technology, and in particular to a parking space state monitoring method based on multi-sensor fusion. BACKGROUND

[0002] With the rapid development of smart city and intelligent transportation infrastructure, parking space monitoring and management has gradually become a key link in urban traffic governance. Traditional parking state recognition methods mostly rely on a single type of sensor, such as a geomagnetic detector or a video monitoring system, which can only provide local perception capability and has low recognition accuracy in complex environments. Existing methods generally have the following problems:

[0003] Single modal sensors have poor stability in low light, high occlusion or strong interference scenarios, such as cameras being greatly affected by weather and light changes, and geomagnetic sensors being easily triggered by metal interference, resulting in frequent misjudgment of parking space state; different types of sensors have natural differences in data format, sampling frequency, synchronization mechanism, etc., making it difficult to model multi-source heterogeneous data uniformly, which may lead to cross-modal alignment mismatch and affect the fusion effect; most existing methods rely on static fusion strategies and lack dynamic modeling and adaptive mechanisms, which have insufficient robustness in the face of various types of vehicles, complex occupancy behaviors and frequent sensor anomalies in parking scenarios, making it difficult to achieve stable, real-time and refined parking state recognition and management.

[0004] Therefore, how to provide a parking space state monitoring method based on multi-sensor fusion is a problem that needs to be solved by those skilled in the art. SUMMARY

[0005] One object of the present application is to provide a parking space state monitoring method based on multi-sensor fusion, which makes full use of intelligent transportation perception technology, deep learning network and edge intelligent computing, and constructs an improved SwinFusion model to realize joint modeling and asynchronous alignment of multi-source heterogeneous data such as geomagnetic, infrared, ultrasonic and image data. The improved SwinFusion model realizes high-precision recognition and dynamic visualization of parking space state in complex parking environments by designing modal window division, modal channel reorganization, visual attention filtering and asynchronous attention alignment, and uses threshold and lightweight network collaborative discrimination method to improve the real-time performance and energy efficiency of parking space state recognition when deployed on the edge, effectively reducing the computational load, and having the advantages of high recognition accuracy, strong anti-interference ability, flexible deployment and good real-time performance.

[0006] According to the parking space state monitoring method based on multi-sensor fusion of the present application, the method comprises the following steps: Step 1: Collect multi-modal sensor data of the target parking area; Step two: pre-process the multi-modal sensor data, including time synchronization, noise filtering, normalization and tensorization, to generate a standardized multi-modal data set; Step three: build an improved SwinFusion model, which includes a modal window division module, a modal channel reorganization module, a visual attention filtering module and an asynchronous attention alignment module, to perform multi-modal feature extraction on the standardized multi-modal data set and generate an asynchronous attention feature tensor sequence; Step four: perform multi-scale feature decoding and spatio-temporal mapping on the asynchronous attention feature tensor sequence to generate a feature mapping graph and an occupancy probability graph of the parking area; Step five: based on the feature mapping graph and the occupancy probability graph of the parking area, extract the fusion feature vector of the target parking space and perform state discrimination to output the parking space state label; Step six: match the parking space state label with the parking area coordinate information to generate a real-time parking space state visualization graph on the edge terminal; Step seven: collect the edge terminal recognition result and sensor feedback data to perform incremental update on the improved SwinFusion model.

[0007] Optionally, the multi-modal sensor data includes image modal data, geomagnetic modal data, infrared modal data and ultrasonic modal data.

[0008] Optionally, the step two specifically includes: Time synchronization processing of multi-modal sensor data according to sampling time to generate a multi-modal time series sample set; Perform noise filtering on the multi-modal time series sample set to generate a multi-modal denoising sample set, specifically: use median filtering method to denoise image modal data, use sliding mean filtering method to denoise geomagnetic modal data and infrared modal data, and use threshold clipping method to eliminate sudden interference signals of ultrasonic modal data; Normalize each modal data respectively by the min-max normalization method to form a multi-modal normalized sample set; Map the multi-modal normalized sample set to tensor form, the image modal data is constructed into an image modal tensor sequence according to the number of time steps, image height, image width and image channel number, the geomagnetic modal data, infrared modal data and ultrasonic modal data are constructed into a time step tensor sequence according to the number of time steps and modal feature dimensions; the modal feature dimension refers to the sum of the channel dimensions of the geomagnetic modal data, infrared modal data and ultrasonic modal data; the time step tensor sequence includes geomagnetic modal tensor, infrared modal tensor and ultrasonic modal tensor at each time step; The image modality tensor sequence and the time step tensor sequence are organized into a standardized multi-modality dataset.

[0009] Optionally, the modality window division module performs frame-by-frame processing on the image modality tensor sequence in the standardized multi-modality dataset, obtains a sliding window division matrix, and performs a non-uniform window division operation on the image modality tensor sequence to generate a local feature block set, specifically as follows. Each frame of image in the image modality tensor sequence is compressed in the channel dimension by weighting to obtain a single-channel grayscale image; The horizontal direction gradient map and the vertical direction gradient map of the single-channel grayscale image are calculated respectively by using a Sobel edge detection operator; The horizontal direction gradient map and the vertical direction gradient map are pixel-by-pixel squared summed and then square rooted to obtain a gradient amplitude map, and the gradient amplitude map is pixel-by-pixel normalized to generate a saliency map; The saliency map is divided into a plurality of saliency sub-blocks according to a preset window size, and each saliency sub-block is block average-pooled to obtain an average saliency score of each saliency sub-block, and the saliency scores are arranged in the original spatial order to form a saliency score matrix; The saliency score matrices of the time frames are constructed into a sliding window division matrix according to the time steps, and a non-uniform window division operation is performed on the image modality tensor sequence based on the sliding window division matrix to generate an image local feature tensor sequence.

[0010] Optionally, the modality channel reorganization module performs cross-modality alignment and channel reorganization on the image local feature tensor sequence and the time step tensor sequence to generate a modality fusion feature tensor sequence, specifically as follows. The image local feature tensor sequence and the time step tensor sequence are paired according to the time steps to form a modality alignment feature pair sequence, wherein the modality alignment feature pair at each time step includes an image local feature tensor, a geomagnetic modality tensor, an infrared modality tensor, and an ultrasonic modality tensor; At each time step, the image local feature tensor is channel-transformed by a one-dimensional convolution operation to generate an image modality mapping tensor; At each time step, the geomagnetic modality tensor, the infrared modality tensor, and the ultrasonic modality tensor are respectively subjected to linear transformation to generate a geomagnetic modality mapping tensor, an infrared modality mapping tensor, and an ultrasonic modality mapping tensor; The channel dimensions of the image modality mapping tensor, the geomagnetic modality mapping tensor, the infrared modality mapping tensor, and the ultrasonic modality mapping tensor are consistent; At each time step, the geomagnetic modality mapping tensor, the infrared modality mapping tensor, and the ultrasonic modality mapping tensor are spliced to generate a non-image modality combined tensor; At each time step, the image modality mapping tensor is flattened in the spatial dimension to generate a two-dimensional image modality tensor; The two-dimensional image modality tensor is spliced with the non-image modality combined tensor in the sample dimension, and is linearly mixed and weighted in the channel by a one-dimensional convolution and a ReLU activation function to obtain a modality reorganization tensor; The modality reorganization tensor is restored to the same spatial dimension as the image local feature tensor by a Reshape function to form a modality fusion feature tensor, and is organized into a modality fusion feature tensor sequence according to the time step.

[0011] Optionally, the visual attention filtering module takes the image modality mapping tensor as a visual guide and performs feature weighting on the modality fusion feature tensor, specifically: At each time step, the image modality mapping tensor is averaged and maximized along the spatial dimension to generate two one-dimensional compression vectors, the two one-dimensional compression vectors are respectively output by a one-dimensional convolution to generate two activation vectors, and the two activation vectors are added and then passed through a Sigmoid activation function to generate a channel attention weight vector; The image modality mapping tensor is averaged and maximized along the channel dimension to obtain a spatial average map and a spatial maximum map, respectively. The spatial average map and the spatial maximum map are spliced in the channel dimension and then passed through a two-dimensional convolution to generate a spatial attention activation map, and the spatial attention activation map is passed through a Sigmoid activation to generate a spatial attention weight matrix; The channel attention weight vector is expanded into a three-dimensional weight tensor consistent with the modality fusion feature tensor in the channel dimension and a single scalar in the spatial dimension to obtain a channel attention weight tensor; The spatial attention matrix is expanded into a three-dimensional weight tensor consistent with the modality fusion feature tensor in the spatial dimension and a single scalar in the channel dimension to obtain a spatial attention weight tensor; The modality fusion feature tensor is weighted channel by channel and pixel by pixel by the channel attention weight tensor and the spatial attention weight tensor, respectively, to obtain a visual attention enhanced tensor, and the visual attention enhanced tensor is organized into a visual attention enhanced tensor sequence according to the time step.

[0012] Optionally, the asynchronous attention alignment module performs asynchronous attention operation on the visual attention enhanced tensor sequence, specifically: At each time step, the visual attention enhanced tensor is GEM-pooled along the spatial dimension to generate a modality description vector, and the modality description vector is organized into a modality description vector sequence according to the time step. The modal description vector sequence is mapped into a query sequence, a key sequence and a value sequence through three groups of structure-shared one-dimensional convolution respectively, and attention weights between the current time step and the historical time steps are calculated based on dot product attention to form an asynchronous attention weight matrix; A weighted sum is performed on the value sequence based on the asynchronous attention weight matrix to generate an asynchronous attention feature sequence; The asynchronous attention feature sequence is input into a fully connected mapping layer to perform linear mapping at each time step to generate a tensorized asynchronous attention feature, and the tensorized asynchronous attention feature is reconstructed to be consistent with the dimension of the visual attention enhancement tensor through a reshape function and organized into an asynchronous attention feature tensor sequence according to the time steps.

[0013] Optionally, the step four specifically includes: At each time step, multi-scale features are extracted from the asynchronous attention feature tensor through two-dimensional convolution with a convolution kernel size of 3x3, 5x5 and 7x7 respectively to obtain a small-scale feature tensor, a medium-scale feature tensor and a large-scale feature tensor, and the scale fusion feature tensor is obtained by concatenating in the channel dimension; The scale fusion feature tensor is compressed in the channel and linearly corrected through one-dimensional convolution and ReLU activation function to obtain a compressed feature tensor; The compressed feature tensor is restored to a feature mapping map of the parking space region with a spatial size consistent with the original image and a channel number of 1 through up-sampling: 1x1 convolution and Sigmoid activation are performed on the feature mapping map to generate a preliminary occupancy probability map; Each pixel value in the preliminary occupancy probability map is normalized through a Poisson normalization function to generate an occupancy probability map of the parking space region.

[0014] Optionally, the step five specifically includes: At each time step, a two-dimensional spatial region corresponding to the target parking space is extracted based on the feature mapping map, and average pooling is performed to obtain a structural feature vector; A pixel sub-block corresponding to the target parking space is extracted from the occupancy probability map, and an average occupancy probability value of the pixel sub-block is calculated; The structural feature vector and the occupancy probability value are concatenated to generate a fusion feature vector; A state threshold is set and the state threshold , if the average occupancy probability value is less than or equal to the state threshold , a parking space state label L0: free state is generated; if the average occupancy probability value is greater than or equal to the state threshold , a parking space state label L1: occupied state is generated; if the average occupancy probability value is greater than the state threshold and less than the state threshold If the parking space state label is L2: abnormal state, then a parking space state label L2: abnormal state is generated. If the parking space state label is L2: abnormal state, then a corresponding fusion feature vector is subjected to a three-classification operation through an MLP structure to obtain a three-dimensional probability vector, and the maximum probability value index of the three-dimensional probability vector is taken as the parking space state label; each of the three-dimensional probability vector corresponds to an idle state probability value, an occupied state probability value and an abnormal state probability value, respectively.

[0015] Optionally, the step six specifically comprises: The parking space state label of the target parking space is matched with the spatial coordinate information to construct a parking space labeling information unit, the labeling information unit comprising a parking space number, a coordinate bounding box and a parking space state label. Based on the parking space labeling information unit, all parking space state information is rendered into a parking area image to generate a parking space state visualization map, wherein the idle state is rendered in green, the occupied state is rendered in red, and the abnormal state is rendered in yellow. The parking space state visualization map is stored in the edge terminal, and the parking space state label, the state probability value and the coordinate boundary information of each time step are cached to the edge database in a structured data format.

[0016] The present application has the following advantages: The present application discloses a parking space state monitoring method based on multi-sensor fusion, aiming to solve the problems of insufficient multi-source heterogeneous data fusion and state recognition susceptible to interference. First, by deploying image acquisition cameras, geomagnetic sensors, infrared detectors and ultrasonic ranging modules, image modal data, geomagnetic modal data, infrared modal data and ultrasonic modal data of the target parking area are collected to form multi-modal data sources covering multiple dimensions such as visual angle, magnetic signal, temperature change and distance perception.

[0017] Secondly, an improved SwinFusion model is used for deep fusion of multi-modal data. The improved SwinFusion model designs a modality-guided sliding window division mechanism in the feature modeling stage, adaptively generates a perception window boundary according to the perception distribution and spatial correlation of different modalities, and realizes feature alignment of heterogeneous modalities. Through modality channel reorganization, the multi-modal feature channels are rearranged and combined according to semantic consistency, so as to improve the alignment accuracy and semantic aggregation ability between cross-modalities. In the feature fusion stage, an image visual dominant attention filtering module is introduced, a channel attention weight tensor is generated based on the image modality mapping tensor, and the channel attention weight tensor is applied to the modality fusion feature tensor, so as to strengthen the significant channel response corresponding to the image modality, suppress the potential invalid or disturbed channel feature response, and improve the discriminability and robustness of the fusion representation. In order to cope with the problem of inconsistent sampling frequency of each modality, asynchronous cross-modality attention alignment is used in combination with timing embedding and deformable attention operation to realize key frame semantic alignment and timing consistency modeling.

[0018] In the state discrimination stage, a discrimination method based on threshold rules and a lightweight neural network is used to quickly complete the identification of the "occupied", "idle" and "abnormal" states of the parking space, and support state visualization output. This method can still realize high-robustness and high-precision parking space state recognition in complex occlusion, low illumination or electromagnetic interference environment, and has excellent edge deployment adaptability and engineering practical value.

[0019] In summary, the present application realizes deep fusion and robust recognition of multi-sensor data, and improves the accuracy and real-time performance of parking space state monitoring in complex environments. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the specification, together with the embodiments of the application, to explain the application, and do not constitute a limitation on the application. In the drawings:

[0021] Figure 1 is a schematic diagram of a parking space state monitoring method based on multi-sensor fusion proposed by the present application; Figure 2 is a structure flow chart of an improved SwinFusion model in a parking space state monitoring method based on multi-sensor fusion proposed by the present application; Figure 3 is a modality channel reorganization module flow chart in a parking space state monitoring method based on multi-sensor fusion proposed by the present application; Figure 4 is a state discrimination and visualization flow chart in a parking space state monitoring method based on multi-sensor fusion proposed by the present application. DETAILED DESCRIPTION

[0022] The application will now be described in further detail with reference to the drawings. These drawings are simplified schematic diagrams which only show the basic structure of the application in a diagrammatic manner and thus only show the components which are relevant to the application.

[0023] Reference Figures 1-4 A parking space state monitoring method based on multi-sensor fusion includes the following steps: Step one: collect multi-modal sensor data of the target parking area; Step two: preprocess the multi-modal sensor data, which includes time synchronization, noise filtering, normalization and tensorization processing, to generate a standardized multi-modal data set; Step three: build an improved SwinFusion model, which includes a modal window division module, a modal channel reorganization module, a visual attention filtering module and an asynchronous attention alignment module, to perform multi-modal feature extraction on the standardized multi-modal data set and generate an asynchronous attention feature tensor sequence; Step four: perform multi-scale feature decoding and spatiotemporal mapping on the asynchronous attention feature tensor sequence to generate a feature mapping graph and an occupancy probability graph of the parking space area; Step five: based on the feature mapping graph and the occupancy probability graph of the parking space area, extract the fusion feature vector of the target parking space and perform state discrimination to output the parking space state label; Step six: match the parking space state label with the parking area coordinate information to generate a real-time parking space state visualization graph on the edge terminal; Step seven: collect the edge terminal recognition result and sensor feedback data to perform incremental update on the improved SwinFusion model.

[0024] In this embodiment, the multi-modal sensor data includes image modal data, geomagnetic modal data, infrared modal data and ultrasonic modal data.

[0025] In this application, the image modal data is a parking space overhead image collected by a camera installed above the parking space; the geomagnetic modal data is a magnetic field change collected by a buried three-axis magnetic flux sensor; the infrared modal data is a ground temperature radiation intensity collected by an infrared pyroelectric sensor installed above or beside the parking space, which is used to sense engine heat residues and human or object movement; the ultrasonic modal data is a vertical distance information collected above the parking space by an ultrasonic ranging module fixed at the front or top end of the parking space, which is used to determine whether there is a vehicle parked or an obstructing object.

[0026] In this embodiment, step two specifically includes: The multi-modal sensor data is time-synchronized according to sampling time to generate a multi-modal time sequence sample set; Noise filtering is performed on the multi-modal time sequence sample set to generate a multi-modal denoising sample set, specifically: the image modal data is denoised by a median filtering method, the geomagnetic modal data and the infrared modal data are denoised by a sliding mean filtering method, and the ultrasonic modal data is processed by a threshold clipping method to eliminate sudden interference signals; Each modal data is normalized by a min-max normalization method to form a multi-modal normalized sample set; The multi-modal normalized sample set is mapped into a tensor form, the image modal data is structured into an image modal tensor sequence according to the number of time steps, image height, image width and image channel number, and the geomagnetic modal data, infrared modal data and ultrasonic modal data are structured into a time step tensor sequence according to the number of time steps and modal feature dimensions; the modal feature dimension refers to the sum of the channel dimensions of the geomagnetic modal data, infrared modal data and ultrasonic modal data; the time step tensor sequence includes a geomagnetic modal tensor, an infrared modal tensor and an ultrasonic modal tensor at each time step; The image modal tensor sequence and the time step tensor sequence are organized into a standardized multi-modal data set.

[0027] In the embodiment, the modal window division module processes the image modal tensor sequence in the standardized multi-modal data set frame by frame to obtain a sliding window division matrix, and performs a non-uniform window division operation on the image modal tensor sequence to generate a local feature block set, specifically: Each frame of image in the image modal tensor sequence is compressed in the channel dimension to obtain a single-channel grayscale image; The horizontal direction gradient map and the vertical direction gradient map of the single-channel grayscale image are calculated by a Sobel edge detection operator respectively; The horizontal direction gradient map and the vertical direction gradient map are summed pixel by pixel and then take the square root to obtain a gradient amplitude map, and the gradient amplitude map is normalized pixel by pixel to generate a saliency map; The saliency map is divided into a plurality of saliency sub-blocks according to a preset window size, and each saliency sub-block is block average pooled to obtain an average saliency score of each saliency sub-block, and the saliency score matrix is formed according to the original spatial arrangement order; The saliency score matrix of each time frame is constructed into a sliding window division matrix according to the time step, and a non-uniform window division operation is performed on the image modal tensor sequence based on the sliding window division matrix to generate an image local feature tensor sequence.

[0028] In the present application, by introducing an adaptive sliding window mechanism based on a saliency map, the window division method can be dynamically adjusted according to the texture complexity and edge distribution of the image modalities under different time frames, realizing fine-grained feature extraction of key areas and compressed expression of non-key areas. This design effectively avoids the boundary feature loss and spatial redundancy problems caused by traditional fixed window methods, making the local feature block more representative and structure-sensitive, thereby improving the spatial alignment accuracy and overall recognition robustness in the multi-modal fusion stage.

[0029] In the present embodiment, the modal channel reorganization module cross-modally aligns and reorganizes the image local feature tensor sequence and the time step tensor sequence to generate a modal fusion feature tensor sequence, specifically: The image local feature tensor sequence and the time step tensor sequence are paired according to the time step to form a modal alignment feature pair sequence, wherein the modal alignment feature pair at each time step includes an image local feature tensor, a geomagnetic modal tensor, an infrared modal tensor, and an ultrasonic modal tensor. At each time step, the image local feature tensor is transformed into a channel through one-dimensional convolution operation to generate an image modal mapping tensor. At each time step, the geomagnetic modal tensor, the infrared modal tensor, and the ultrasonic modal tensor are respectively subjected to linear transformation to generate a geomagnetic modal mapping tensor, an infrared modal mapping tensor, and an ultrasonic modal mapping tensor. The channel dimensions of the image modal mapping tensor, the geomagnetic modal mapping tensor, the infrared modal mapping tensor, and the ultrasonic modal mapping tensor are consistent. At each time step, the geomagnetic modal mapping tensor, the infrared modal mapping tensor, and the ultrasonic modal mapping tensor are spliced to generate a non-image modal combination tensor. At each time step, the image modal mapping tensor is flattened in the spatial dimension to generate a two-dimensional image modal tensor. The two-dimensional image modal tensor and the non-image modal combination tensor are spliced in the sample dimension, and are subjected to linear mixing and weighted reorganization between channels through one layer of one-dimensional convolution and ReLU activation function to obtain a modal reorganization tensor. The modal reorganization tensor is restored to the same spatial dimension as the image local feature tensor through the Reshape function to form a modal fusion feature tensor, and is organized into a modal fusion feature tensor sequence according to the time step.

[0030] In the present application, by unified mapping and linear reorganization of image modalities and non-image modalities in channel dimension and spatial dimension, semantic alignment and collaborative expression between cross-modal features are effectively realized. In the fusion process, inter-channel linear mixing and weighting are introduced, so that the spatial detail features of the image modality and the environmental perception information of the geomagnetic, infrared and ultrasonic modalities are deeply fused at the tensor level, improving the semantic integrity and discriminability of the fused features, and enhancing the discrimination accuracy and robustness of the state discrimination process.

[0031] In the present embodiment, the visual attention filtering module maps the image modality tensor as a visual guide and performs feature weighting on the modality fusion feature tensor, specifically: At each time step, the image modality mapping tensor is averaged and maximized along the spatial dimension to generate two one-dimensional compressed vectors, the two one-dimensional compressed vectors are output through one-dimensional convolution to generate two activation vectors, and the two activation vectors are added and then passed through a Sigmoid activation function to generate a channel attention weight vector; The image modality mapping tensor is averaged and maximized along the channel dimension to obtain a spatial average graph and a spatial maximum graph, respectively. The spatial average graph and the spatial maximum graph are concatenated in the channel dimension and then passed through a two-dimensional convolution to generate a spatial attention activation graph, and the spatial attention activation graph is passed through a Sigmoid activation to generate a spatial attention weight matrix; The channel attention weight vector is expanded into a three-dimensional weight tensor consistent with the modality fusion feature tensor in the channel dimension and a single scalar in the spatial dimension, obtaining a channel attention weight tensor; The spatial attention matrix is expanded into a three-dimensional weight tensor consistent with the modality fusion feature tensor in the spatial dimension and a single scalar in the channel dimension, obtaining a spatial attention weight tensor; The modality fusion feature tensor is weighted channel by channel and pixel by pixel by the channel attention weight tensor and the spatial attention weight tensor, respectively, to obtain a visual attention enhanced tensor, and the visual attention enhanced tensor sequence is organized according to the time step.

[0032] In the present application, the image modality mapping tensor is used as a guide, and the channel attention and spatial attention are jointly modeled, which can accurately identify and enhance the significant regions or channel responses in the multi-modal fusion features. By introducing the attention weights of the image modality in the spatial and channel dimensions, the suppression of invalid features and interference information is realized, and the significant feature expression highly related to the target parking state is effectively enhanced. This module significantly improves the representation ability and discriminability of the fusion features, and enhances the recognition stability and robustness of the model in complex scenes such as low light, occlusion and geomagnetic interference.

[0033] In this embodiment, the asynchronous attention alignment module performs asynchronous attention operation on the visual attention enhanced tensor sequence, specifically: At each time step, the visual attention enhanced tensor is subjected to GEM pooling along the spatial dimension to generate a modal description vector, and the modal description vector is organized into a modal description vector sequence at the time step: ; Wherein, represents the pixel value of the th spatial position in the visual attention enhanced tensor, represents the total number of pixels in the spatial dimension, is a learnable pooling parameter; The modal description vector sequence is mapped into a query sequence, a key sequence and a value sequence through three groups of one-dimensional convolution with structure sharing respectively, and the attention weights between the current time step and the historical time step are calculated based on the dot product attention to form an asynchronous attention weight matrix; Based on the asynchronous attention weight matrix, weighted summation is performed on the value sequence to generate an asynchronous attention feature sequence; The asynchronous attention feature sequence is input into a fully connected mapping layer to perform linear mapping at each time step to generate a tensorized asynchronous attention feature, and the tensorized asynchronous attention feature is reconstructed to be consistent with the dimension of the visual attention enhanced tensor through the reshape function, and is organized into an asynchronous attention feature tensor sequence according to the time step.

[0034] In the present application, the asynchronous attention alignment module extracts the modal description vector at the key moment by using GEM pooling, and realizes dynamic association modeling and important frame reinforcement between features at different time steps through the time sequence attention matching of the query, key and value sequences, thereby enhancing the semantic consistency across time steps while ensuring the continuity of the features, so as to significantly improve the response sensitivity of the model to the dynamic parking state change and the cross-time sequence feature fusion capability, and ensure that the high-precision state recognition result can still be stably output under asynchronous perception conditions.

[0035] In this embodiment, the step four specifically comprises: At each time step, the asynchronous attention feature tensor is extracted through two-dimensional convolution with convolution kernel size of 3x3, 5x5 and 7x7 to obtain a small-scale feature tensor, a medium-scale feature tensor and a large-scale feature tensor, and the scale fusion feature tensor is obtained by splicing in the channel dimension; The scale fusion feature tensor is subjected to channel compression and linear correction through one-dimensional convolution and ReLU activation function to obtain a compressed feature tensor; The compressed feature tensor is restored to a feature mapping map of the parking space area with the same spatial size as the original image and a channel number of 1 through up-sampling: performing 1x1 convolution and Sigmoid activation on the feature map to generate a preliminary occupancy probability map; normalizing each pixel value in the preliminary occupancy probability map by a Poisson normalization function to generate an occupancy probability map of the parking space region.

[0036] In this embodiment, the step five specifically includes: At each time step, a two-dimensional spatial region corresponding to the target parking space is extracted based on the feature map, and average pooling is performed to obtain a structural feature vector; extracting a pixel sub-block corresponding to the target parking space from the occupancy probability map and calculating an average occupancy probability value of the pixel sub-block; concatenating the structural feature vector and the occupancy probability value to generate a fusion feature vector; setting a state threshold and the state threshold If the average occupancy probability value is less than or equal to the state threshold , a parking space state label L0 is generated: free state; if the average occupancy probability value is greater than or equal to the state threshold , a parking space state label L1 is generated: occupied state; if the average occupancy probability value is greater than the state threshold and less than the state threshold , a parking space state label L2 is generated: abnormal state. If the parking space state label is L2: abnormal state, the corresponding fusion feature vector is subjected to a three-classification operation through an MLP structure to obtain a three-dimensional probability vector, and the maximum probability value index of the three-dimensional probability vector is taken as the parking space state label; each of the three-dimensional probability vector corresponds to an idle state probability value, an occupied state probability value, and an abnormal state probability value.

[0037] In this embodiment, the spatial structure features of the parking space are extracted using the feature map, and the probability information of the local region of the parking space is obtained from the occupancy probability map. The fusion feature vector is generated by concatenation, so that the model considers both the spatial structure and the probability intensity features when judging the state. The double threshold is designed to divide the state into three categories: free, occupied, and abnormal. The MLP structure is further introduced for three-classification probability reasoning for the abnormal state, ensuring that the model can still output stable and accurate state labels under the conditions of fuzzy boundaries and perception interference, thereby effectively improving the robustness and error correction ability of parking state recognition in complex environments.

[0038] In this embodiment, the step six specifically includes: matching the parking space state label of the target parking space with the spatial coordinate information to construct a parking space labeling information unit, the labeling information unit including a parking space number, a coordinate bounding box, and a parking space state label; Based on the parking space labeling information unit, all parking space state information is rendered into a parking area image to generate a parking space state visualization diagram, in which the idle state is rendered in green, the occupied state is rendered in red, and the abnormal state is rendered in yellow. The parking space state visualization diagram is stored in the edge terminal, and the parking space state label, state probability value and coordinate boundary information of each time step are cached in a structured data format to the edge database.

[0039] Embodiment 1 In order to verify the feasibility of the application in implementation, the application is applied to the intelligent reconstruction project of an underground garage of a certain city intelligent parking system. The reconstruction project includes multiple closed parking units, the light condition is poor, there are obstructions in some areas, such as pipeline supports, columns, etc., and is accompanied by geomagnetic disturbance and signal reflection phenomenon, which belongs to the difficult scene of parking space state recognition in typical complex environment.

[0040] In the implementation process, a set of multi-modal sensing equipment composed of an image acquisition camera, a geomagnetic sensor, an infrared sensing module and an ultrasonic ranging sensor is deployed at each parking space. All modal data are uniformly accessed to the edge computing node for preprocessing, including time synchronization, noise filtering, normalization and tensorization processing, to form a standardized modal data set. The multi-modal data are input into the improved SwinFusion model, and the modal guided window division, modal channel reorganization, visual attention filtering and asynchronous attention alignment are sequentially performed to realize the deep fusion of multi-modal features and the time sequence consistency modeling, and output the asynchronous attention feature tensor sequence. After the asynchronous attention feature tensor sequence is processed by multi-scale feature decoding and space-time mapping, the feature mapping graph and the occupancy probability graph of the parking space area are generated, and based on the threshold and the lightweight neural network cooperative judgment strategy, the "idle", "occupied" or "abnormal" state label is output, and at the same time, the parking space state visualization diagram is generated in real time at the edge terminal, realizing the intelligent recognition and dynamic display of the parking space state.

[0041] In the actual application process, 100 parking spaces in the reconstruction project are continuously collected for 15 days, and data samples are collected every 5 seconds, forming about 259200 valid samples, and the application method is compared with three kinds of mainstream parking state recognition schemes, which are scheme A: traditional single image recognition method, scheme B: image + geomagnetic dual modal fusion method, scheme C: multi-modal fusion method with Transformer structure, and the experimental results are shown in Table 1.

[0042] Table 1 Performance comparison table of the application method and the comparison schemes in parking space state recognition

[0043] From the data in Table 1, it can be seen that the method of the present application is significantly better than the comparison schemes in multiple performance indicators. The method of the present application is significantly better than comparison schemes A, B and C in multiple key performance indicators. In terms of average recognition accuracy, the method of the present application reaches 96.8%, which is 4.3 percentage points higher than the optimal comparison scheme C; in terms of abnormal state recognition rate, the method of the present application reaches 85.1%, which is much higher than 67.3% of comparison scheme C, and is improved by 17.8 percentage points, indicating that the improved SwinFusion model has stronger abnormal scene understanding and interference recognition capability. In terms of occupancy misjudgment rate and idle misjudgment rate, the method of the present application is 1.9% and 2.4% respectively, which is significantly lower than 3.7% and 4.9% of comparison scheme C, indicating that the method of the present application is more accurate and reliable in state discrimination under the scenes of occlusion, low light and multiple vehicle interlacing. In the model reasoning delay index, the present application is controlled at 46.2ms, which is slightly higher than schemes A and B, but is significantly reduced by 28.4ms compared with scheme C, indicating that the method of the present application can maintain recognition accuracy and realize fast response, and has good edge deployment adaptability. In terms of abnormal false alarm rate, the method of the present application is controlled at 2.1%, which is significantly better than the three comparison schemes, indicating that the method of the present application can effectively avoid false alarms in dynamic scenes.

[0044] The method of the present application considers high-precision recognition, low misjudgment, fast response and abnormal perception capability, and is suitable for real-time parking space state monitoring requirements in complex urban parking scenes.

[0045] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can make equivalent replacement or change according to the technical solution and inventive concept of the present application within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A parking space state monitoring method based on multi-sensor fusion, characterized in that, The method comprises the following steps: Step 1: Collecting multi-modal sensor data of the target parking area; Step 2: Preprocessing the multi-modal sensor data, which includes time synchronization, noise filtering, normalization and tensorization processing, to generate a standardized multi-modal data set; Step 3: Constructing an improved SwinFusion model, which includes a modal window division module, a modal channel reorganization module, a visual attention filtering module and an asynchronous attention alignment module, to perform multi-modal feature extraction on the standardized multi-modal data set and generate an asynchronous attention feature tensor sequence; Step 4: Performing multi-scale feature decoding and space-time mapping on the asynchronous attention feature tensor sequence to generate a feature mapping graph and an occupancy probability graph of the parking space area; Step 5: Based on the feature mapping graph and the occupancy probability graph of the parking space area, extracting the fusion feature vector of the target parking space and performing state discrimination to output the parking space state label; Step 6: Matching the parking space state label with the parking area coordinate information to generate a real-time parking space state visualization graph on the edge terminal; Step 7: Collecting the edge terminal recognition result and sensor feedback data to perform incremental update on the improved SwinFusion model. 2.The parking space state monitoring method based on multi-sensor fusion according to claim 1, characterized in that, The multi-modal sensor data includes image modal data, geomagnetic modal data, infrared modal data and ultrasonic modal data. 3.The parking space state monitoring method based on multi-sensor fusion according to claim 1, characterized in that, The step 2 specifically comprises: Time synchronization processing of the multi-modal sensor data according to the sampling time to generate a multi-modal time series sample set; Noise filtering of the multi-modal time series sample set to generate a multi-modal denoising sample set, specifically: using a median filtering method to denoise the image modal data, using a sliding mean filtering method to denoise the geomagnetic modal data and infrared modal data, and using a threshold clipping method to eliminate sudden interference signals of the ultrasonic modal data; Normalizing each modal data respectively through the min-max normalization method to form a multi-modal normalized sample set; Mapping the multi-modal normalized sample set into a tensor form, the image modal data is constructed into an image modal tensor sequence according to the time step, image height, image width and image channel number, the geomagnetic modal data, infrared modal data and ultrasonic modal data are constructed into a time step tensor sequence according to the time step and modal feature dimension; the modal feature dimension refers to the sum of the channel dimensions of the geomagnetic modal data, infrared modal data and ultrasonic modal data; the time step tensor sequence includes a geomagnetic modal tensor, an infrared modal tensor and an ultrasonic modal tensor at each time step; Organizing the image modal tensor sequence and the time step tensor sequence into a standardized multi-modal data set.

4. The method according to claim 1, wherein, The modal window division module performs frame-by-frame processing on the image modal tensor sequence in the standardized multi-modal data set to obtain a sliding window division matrix, and performs a non-uniform window division operation on the image modal tensor sequence to generate a local feature block set, specifically: Performing weighted compression on each frame of image in the channel dimension in the image modal tensor sequence to obtain a single-channel grayscale image; The horizontal direction gradient map and the vertical direction gradient map of the single-channel grayscale image are calculated respectively by a Sobel edge detection operator; The horizontal direction gradient map and the vertical direction gradient map are pixel-by-pixel squared summed and then square rooted to obtain a gradient amplitude map, and the gradient amplitude map is pixel-by-pixel normalized to generate a saliency map; The saliency map is divided into a plurality of saliency sub-blocks according to a preset window size, and block average pooling is performed on each saliency sub-block to obtain an average saliency score of each saliency sub-block, and the saliency score matrix is arranged according to the original spatial arrangement order. The saliency score matrix of each time frame is divided into a sliding window division matrix according to the time step, and a non-uniform window division operation is performed on the image modal tensor sequence based on the sliding window division matrix to generate an image local feature tensor sequence.

5. The method of claim 1, wherein the method further comprises: The modal channel reorganization module performs cross-modal alignment and channel reorganization on the image local feature tensor sequence and the time step tensor sequence to generate a modal fusion feature tensor sequence, specifically: The image local feature tensor sequence and the time step tensor sequence are paired according to the time step to form a modal alignment feature pair sequence, wherein the modal alignment feature pair at each time step includes an image local feature tensor, a geomagnetic modal tensor, an infrared modal tensor, and an ultrasonic modal tensor; At each time step, the image local feature tensor is transformed into a channel through a one-dimensional convolution operation to generate an image modal mapping tensor; At each time step, the geomagnetic modal tensor, the infrared modal tensor, and the ultrasonic modal tensor are respectively subjected to linear transformation to generate a geomagnetic modal mapping tensor, an infrared modal mapping tensor, and an ultrasonic modal mapping tensor; The channel dimensions of the image modal mapping tensor, the geomagnetic modal mapping tensor, the infrared modal mapping tensor, and the ultrasonic modal mapping tensor are consistent; At each time step, the geomagnetic modal mapping tensor, the infrared modal mapping tensor, and the ultrasonic modal mapping tensor are spliced to generate a non-image modal combination tensor; At each time step, the image modal mapping tensor is flattened in the spatial dimension to generate a two-dimensional image modal tensor; The two-dimensional image modal tensor and the non-image modal combination tensor are spliced in the sample dimension, and are subjected to channel linear mixing and weighted reorganization through one layer of one-dimensional convolution and ReLU activation function to obtain a modal reorganization tensor; The modal reorganization tensor is restored to the same spatial dimension as the image local feature tensor through a Reshape function to form a modal fusion feature tensor, and the modal fusion feature tensor sequence is organized according to the time step.

6. The method of claim 1, wherein the method further comprises: The visual attention filtering module takes the image modal mapping tensor as a visual guide and performs feature weighting on the modal fusion feature tensor, specifically: At each time step, the image modal mapping tensor is subjected to average pooling and maximum pooling along the spatial dimension to generate two one-dimensional compression vectors, the two one-dimensional compression vectors are respectively output through one-dimensional convolution to generate two activation vectors, and the two activation vectors are added and then input into a Sigmoid activation function to generate a channel attention weight vector. The image modality mapping tensor is subjected to average pooling and maximum pooling along the channel dimension to obtain a spatial average graph and a spatial maximum graph, respectively, the spatial average graph and the spatial maximum graph are spliced in the channel dimension, and a spatial attention activation graph is generated through two-dimensional convolution, and the spatial attention activation graph is subjected to Sigmoid activation to generate a spatial attention weight matrix; The channel attention weight vector is expanded into a three-dimensional weight tensor consistent with the channel dimension of the modality fusion feature tensor and a single scalar in the spatial dimension, to obtain a channel attention weight tensor; The spatial attention matrix is expanded into a three-dimensional weight tensor consistent with the spatial dimension of the modality fusion feature tensor and a single scalar in the channel dimension, to obtain a spatial attention weight tensor; The modality fusion feature tensor is subjected to channel-by-channel weighting and pixel-by-pixel weighting through the channel attention weight tensor and the spatial attention weight tensor, respectively, to obtain a visual attention enhanced tensor, and the visual attention enhanced tensor is organized into a visual attention enhanced tensor sequence according to the time steps.

7. The method according to claim 1, wherein, The asynchronous attention alignment module performs asynchronous attention operation on the visual attention enhanced tensor sequence, specifically: At each time step, the visual attention enhanced tensor is subjected to GEM pooling along the spatial dimension to generate a modality description vector, and the modality description vector is organized into a modality description vector sequence at the time steps; The modality description vector sequence is mapped into a query sequence, a key sequence and a value sequence through three groups of one-dimensional convolution sharing the same structure, respectively, and the attention weights between the current time step and the historical time steps are calculated based on the dot product attention to form an asynchronous attention weight matrix; The value sequence is subjected to weighted summation based on the asynchronous attention weight matrix to generate an asynchronous attention feature sequence; The asynchronous attention feature sequence is input into a fully connected mapping layer to perform linear mapping at each time step to generate a tensorized asynchronous attention feature, and the tensorized asynchronous attention feature is reconstructed to be consistent with the dimension of the visual attention enhanced tensor through a reshape function and is organized into an asynchronous attention feature tensor sequence according to the time steps. 8.The parking space state monitoring method based on multi-sensor fusion according to claim 1, characterized in that, The step four specifically includes: At each time step, the asynchronous attention feature tensor is subjected to two-dimensional convolution with convolution kernel sizes of 3x3, 5x5 and 7x7 to extract multi-scale features, to obtain a small-scale feature tensor, a medium-scale feature tensor and a large-scale feature tensor, and the scale fusion feature tensor is obtained by splicing in the channel dimension; The scale fusion feature tensor is subjected to one-dimensional convolution and ReLU activation function for channel compression and linear correction to obtain a compressed feature tensor; The compressed feature tensor is restored to a feature mapping graph of the parking space region with the same spatial size as the original image and a channel number of 1 through up-sampling: 1x1 convolution and Sigmoid activation are performed on the feature mapping graph to generate a preliminary occupancy probability graph; The Poisson normalization function is used to normalize each pixel value in the preliminary occupancy probability graph to generate an occupancy probability graph of the parking space region.

9. The method according to claim 1, wherein, The step five specifically includes: At each time step, a two-dimensional spatial region corresponding to the target parking space is extracted based on the feature mapping graph, and average pooling is performed to obtain a structural feature vector; extracting a pixel sub-block corresponding to the target parking space from the occupancy probability map, and calculating an average occupancy probability value of the pixel sub-block; concatenating the structural feature vector and the occupancy probability value to generate a fusion feature vector; Setting a state threshold With the state threshold , if the average occupancy probability value is less than or equal to the state threshold , a parking space state label L0: free state is generated; if the average occupancy probability value is greater than or equal to the state threshold , a parking space state label L1: occupied state is generated; if the average occupancy probability value is greater than the state threshold and less than the state threshold , a parking space state label L2: abnormal state is generated; if the parking space state label is L2: abnormal state, performing a three-classification operation on the corresponding fusion feature vector through an MLP structure to obtain a three-dimensional probability vector, and taking the maximum probability value index of the three-dimensional probability vector as the parking space state label; each of the three-dimensional probability vector corresponds to an idle state probability value, an occupancy state probability value, and an abnormal state probability value, respectively.

10. The method of claim 1, wherein, The step six specifically includes: matching the parking space state label of the target parking space with the spatial coordinate information to construct a parking space labeling information unit, the labeling information unit including a parking space number, a coordinate bounding box, and a parking space state label; based on the parking space labeling information unit, rendering all parking space state information into a parking area image to generate a parking space state visualization map, in which the idle state is rendered in green, the occupancy state is rendered in red, and the abnormal state is rendered in yellow; storing the parking space state visualization map in the edge terminal, and caching the parking space state label, the state probability value, and the coordinate boundary information of each time step to the edge database in a structured data format.

Images