A door and window security alarm method and system based on multimodal sensing

Abstract

This invention relates to the field of security protection technology and discloses a door and window security alarm method and system based on multimodal perception. The method includes: acquiring multimodal perception data through vibration sensors and acoustic sensors and performing short-time Fourier transform; inputting the time-spectrum matrix into an environmental interference source identification convolutional network to output environmental interference type labels and energy contribution estimates for each frequency band; subtracting environmental interference energy components from each frequency band to generate a residual time-spectrum matrix; detecting subthreshold micro-events in the residual domain and extracting temporal features; calculating the cross-modal synchronization correlation score in the residual domain; inputting the comprehensive feature vector into a slow intrusion mode classification model, and generating an alarm notification when the cumulative alarm threshold is exceeded.

Description

Technical Field

[0001] This invention relates to the field of security protection technology, specifically to a door and window security alarm method and system based on multimodal perception. Background Technology

[0002] Door and window security systems typically deploy vibration and acoustic sensors on the door and window frames to continuously collect environmental signals and compare the collected values ​​with adaptive detection thresholds to identify abnormal events. To detect low-intensity intrusion behaviors such as slow prying, existing solutions register subthreshold micro-events by using the inter-frame signal energy change rate and analyze the temporal regularity and cross-modal synchronization correlation of the micro-event sequence within an extended observation time window.

[0003] However, when the environment around doors and windows experiences continuous moderate-intensity disturbances, wind pressure pulsations and raindrop impacts simultaneously generate frequent and time-structured energy fluctuations in both the vibration and acoustic channels. Because environmental disturbances affect both channels simultaneously, a large number of environmental source events are registered as subthreshold micro-events, exhibiting cross-modal co-occurrence characteristics. The temporal regularity of actual intrusion micro-events is masked by the dense distribution of environmental micro-events, and the cross-modal synchronization correlation score is inflated to an indiscriminate level by the environmental co-occurrence effect. This results in slow intrusion pattern classification being unable to extract effective intrusion accumulation features from the contaminated micro-event sequence, leading to systematic underreporting. Summary of the Invention

[0004] This invention provides a door and window security alarm method and system based on multimodal perception, which solves the technical problem in related technologies that it is difficult to effectively identify slow prying intrusion behavior in the context of environmental interference, resulting in a high false alarm rate.

[0005] This invention discloses a door and window security alarm method based on multimodal perception, comprising: Short-time Fourier transforms are performed on the vibration signals and acoustic signals collected by the vibration sensor and acoustic sensor, respectively, to obtain the vibration time-frequency matrix and the acoustic frequency characteristic matrix. The vibration time spectrum matrix and acoustic spectrum feature matrix are respectively input into the environmental interference source identification convolutional network to output the environmental interference type label and energy contribution estimate for each frequency band. Based on the energy contribution estimate, the environmental interference energy component is subtracted from the vibration time-spectrum matrix and acoustic residual spectrum feature matrix band by band to generate the vibration residual time-spectrum matrix and acoustic residual spectrum feature matrix. Perform inverse short-time Fourier transform on the vibration residual time spectrum matrix and the acoustic residual spectrum feature matrix to obtain the residual signal, and detect subthreshold micro-events in the residual domain based on the residual energy change rate; Temporal features are extracted from the subthreshold micro-event sequence in the residual domain within the extended observation time window, and the cross-modal synchronization correlation score in the residual domain is calculated. The temporal features and the residual domain cross-modal synchronization correlation score are concatenated and input into the slow intrusion mode classification model. When the output slow prying intrusion probability value exceeds the cumulative alarm threshold, an alarm notification is generated.

[0006] Furthermore, the output layer of the environmental interference source identification convolutional network contains two parallel branches: a classification branch outputs environmental interference type labels for each frequency band through a Softmax activation function, and a regression branch outputs corresponding energy contribution estimates through a Sigmoid activation function; the environmental interference type labels include wind pressure periodic vibration, raindrop impact response, and structural conducted vibration; the energy contribution estimates characterize the proportion of signal energy occupied by each type of environmental interference source in the corresponding frequency band, with a value range of [value range missing]. ,in This indicates that this type of interference source does not contribute energy. This indicates that all the energy in this frequency band is contributed by this type of interference source.

[0007] Furthermore, it also includes: obtaining environmental parameter data for the current time period from a meteorological data interface, the environmental parameter data including wind speed, wind direction, and rainfall intensity; performing Z-score normalization on wind speed and rainfall intensity, and performing one-hot encoding on wind direction; concatenating the normalized environmental parameter data with the hidden state of the last fully connected layer of the environmental interference source identification convolutional network and then participating in the output layer operation; when the environmental parameter data indicates high wind speed or moderate to high rainfall intensity, the confidence weight of the energy contribution estimate of the environmental interference source identification convolutional network for the wind pressure periodic vibration or raindrop impact response category is increased accordingly.

[0008] Further, the operation of subtracting the environmental interference energy component band by band is as follows: for each band energy value in each time frame of the vibration time-frequency spectrum matrix and the acoustic spectrum feature matrix, subtract the product of the identified energy contribution estimate of that band and the energy value of that band; when the subtraction result is negative, set it to zero; apply a conservative coefficient to the energy contribution estimate, the conservative coefficient being less than... ,in The baseline ratio without any attenuation is such that the actual stripped energy is lower than the energy indicated by the energy contribution estimate.

[0009] Furthermore, the detection of subthreshold micro-events in the residual domain based on the residual energy change rate includes: calculating the residual energy change rate between adjacent sampling frames of the residual signal, assuming the th... The energy of the frame residual signal is , No. The energy of the frame residual signal is Then the first Rate of change of residual energy of frames for: ; When the Rate of change of residual energy of frames If the micro-event sensitivity threshold is exceeded and the residual signal amplitude of the frame does not exceed the adaptive detection threshold, the frame is marked as a residual domain subthreshold micro-event, and its occurrence timestamp and residual energy change rate value are recorded.

[0010] Furthermore, the temporal features include the frequency of micro-event occurrence, the average time interval between adjacent micro-events, the standard deviation of the time interval, the slope of the increasing trend of the residual energy change rate, and the peak value of the autocorrelation coefficient; wherein, the slope of the increasing trend of the residual energy change rate is the slope value obtained after performing linear regression fitting on the residual energy change rate values ​​of each residual domain subthreshold micro-event within the extended observation time window in chronological order; the peak value of the autocorrelation coefficient is the maximum value of the autocorrelation coefficient at all non-zero delays after calculating the Pearson autocorrelation coefficient for the time interval sequence of adjacent micro-events.

[0011] Furthermore, Z-score standardization is performed on the frequency of micro-event occurrence, the average time interval between adjacent micro-events, and the standard deviation of the time interval. Mean normalization based on the range is performed on the slope of the increasing trend of the residual energy change rate and the peak value of the autocorrelation coefficient. After scaling each time series feature to a similar numerical range, the time series features are concatenated into a time series feature vector.

[0012] Furthermore, the calculation method for the residual domain cross-modal synchronization correlation score is as follows: within the extended observation time window, for each residual domain subthreshold micro-event timestamp of the vibration mode, it is determined whether there is a corresponding residual domain subthreshold micro-event in the acoustic mode within the synchronization tolerance time range before and after the timestamp. The count of corresponding residual domain subthreshold micro-events is divided by the total number of residual domain subthreshold micro-events of the vibration mode to obtain the co-occurrence ratio from vibration to acoustic direction; the co-occurrence ratio from acoustic to vibration direction is calculated in the same way; the residual domain cross-modal synchronization correlation score is the average of the co-occurrence ratio from vibration to acoustic direction and the co-occurrence ratio from acoustic to vibration direction.

[0013] Furthermore, it also includes: when the probability value of slow prying intrusion exceeds the proportion threshold of the cumulative alarm threshold but does not reach the cumulative alarm threshold, the current extended observation time window is marked as a warning state and the extended observation time window is slid forward; if the probability values ​​of slow prying intrusion in multiple consecutive sliding windows remain above the warning range, an alarm notification is generated by combining the probability values ​​of multiple windows.

[0014] This invention discloses a door and window security alarm system based on multimodal perception, used to execute the aforementioned door and window security alarm method based on multimodal perception, comprising: The signal acquisition and transformation module is used to acquire multimodal sensing data through vibration sensors and acoustic sensors, and to perform short-time Fourier transform on vibration signals and acoustic signals to obtain vibration time-frequency matrix and acoustic frequency characteristic matrix; The environmental interference source identification module is used to input the vibration time spectrum matrix and the acoustic spectrum feature matrix into the environmental interference source identification convolutional network, and output the environmental interference type label and energy contribution estimate for each frequency band. The spectrum stripping module is used to subtract the environmental interference energy component from the vibration time spectrum matrix and the acoustic spectrum feature matrix band by band based on the energy contribution estimate, so as to generate the vibration residual time spectrum matrix and the acoustic residual spectrum feature matrix. The residual domain micro-event detection module is used to perform inverse short-time Fourier transform on the vibration residual time spectrum matrix and the acoustic residual spectrum feature matrix to obtain the residual signal, and to detect and mark the residual domain subthreshold micro-events based on the residual energy change rate. The temporal feature extraction module is used to extract temporal features from the subthreshold micro-event sequence in the residual domain within an extended observation time window; The cross-modal synchronization analysis module is used to perform time alignment analysis on the residual domain subthreshold micro-event timestamp sequences of vibration modes and acoustic modes, and to calculate the cross-modal synchronization correlation score in the residual domain. The classification and alarm module is used to concatenate the temporal features with the cross-modal synchronization correlation score of the residual domain and input them into the slow intrusion mode classification model. When the probability value of slow intrusion exceeds the cumulative alarm threshold, an alarm notification is generated.

[0015] The beneficial effects of this invention are as follows: This invention introduces environmental interference spectrum stripping processing before subthreshold micro-event detection in the residual domain. An environmental interference source identification convolutional network identifies the energy contributions of environmental interference sources such as wind pressure periodic vibration and raindrop impact response in each frequency band, and subtracts the corresponding components from the vibration time-spectrum matrix and acoustic spectral feature matrix band by band. Since the energy contributions of environmental interference sources are stripped before the signal enters the subthreshold micro-event detection process, the vibration and acoustic residual signals no longer contain frequent energy fluctuations caused by wind pressure pulsations and raindrop impacts. False micro-events generated by environmental interference are no longer registered in batches, and the event density of the subthreshold micro-event sequence in the residual domain is reduced. Simultaneously, because the synchronous energy contributions of environmental interference sources in the vibration and acoustic channels are synchronously stripped, the cross-modal synchronization correlation score in the residual domain is no longer inflated by environmental co-occurrence effects. The slow intrusion pattern classification model can effectively extract the temporal regularity and true cross-modal correlation features of intrusion operations from the subthreshold micro-event sequence in the residual domain, thereby maintaining the detection capability for slow-moving intrusion behavior under continuous severe weather conditions and improving the systematic underreporting problem of existing schemes under such environmental conditions. Attached Figure Description

[0016] Figure 1 This is a flowchart of a door and window security alarm method based on multimodal perception provided in an embodiment of the present invention; Figure 2 This is a schematic diagram comparing the estimated values ​​of environmental interference energy contribution of vibration and acoustic channel at different frequency bands provided in the embodiments of the present invention; Figure 3 This is a schematic diagram showing the energy comparison before and after stripping in each time frame of the low-frequency band of the vibration channel according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the time-series distribution of the subthreshold micro-event residual energy change rate in the vibration channel residual domain provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the distribution of adjacent time intervals between vibration and acoustic channel micro-events provided in an embodiment of the present invention; Figure 6 This is a schematic diagram comparing the timing characteristics of vibration and acoustic channels provided in an embodiment of the present invention; Figure 7 This is a schematic diagram illustrating the sum of cross-modal temporal alignment co-occurrence ratio and intrusion probability provided in an embodiment of the present invention; Figure 8 This is a schematic diagram comparing the effects of the conservative coefficient for environmental interference stripping in different frequency bands provided in the embodiments of the present invention; Figure 9 This is a schematic diagram of the scatter distribution of environmental interference before and after stripping in each frame of the low-frequency band of the vibration channel provided in this embodiment of the invention. Detailed Implementation

[0017] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0018] At least one embodiment of the present invention discloses a door and window security alarm method based on multimodal perception, see [link to relevant documentation]. Figure 1 This includes the following steps: Step 1, Multimodal signal acquisition and time-frequency transformation: Multimodal sensing data is continuously acquired using vibration and acoustic sensors. Short-time Fourier transforms are performed on the vibration and acoustic signals respectively to obtain the vibration time-frequency matrix and acoustic frequency characteristic matrix. Simultaneously, environmental parameter data for the current time period is obtained from a meteorological data interface.

[0019] It should be noted that the vibration spectrum matrix is ​​arranged with time frames as rows and frequency components as columns, and the matrix elements are the signal energy amplitudes of the corresponding time frames and frequency bands. The acoustic spectrum feature matrix adopts the same time-frequency characterization structure. The window length and frame shift parameters of the short-time Fourier transform are determined according to the typical frequency range of the vibration response of the door and window structure, so that the short-time Fourier transform results cover the low-frequency components generated by the prying operation.

[0020] Step 2, Identification of the spectrum of environmental interference sources: The vibration frequency spectrum matrix and acoustic frequency spectrum feature matrix are input into a pre-trained environmental interference source identification convolutional network, which outputs environmental interference type labels and their energy contribution estimates for each frequency band. Environmental interference types include wind pressure periodic vibration, raindrop impact response, and structurally conducted vibration.

[0021] It should be noted that the environmental interference source identification convolutional network is a convolutional neural network. The input layer of this network receives a single-modal time-spectrum matrix, and the output layer contains two parallel branches: a classification branch outputs environmental interference type labels for each frequency band using a Softmax activation function, and a regression branch outputs corresponding energy contribution estimates using a Sigmoid activation function. The energy contribution estimates characterize the proportion of signal energy occupied by each type of environmental interference source in the corresponding frequency band, and the range of the energy contribution estimates is [insert range here]. Type label classification uses the cross-entropy loss function, and energy contribution estimation regression uses the mean squared error loss function. The two losses are weighted and summed before training using the Adam optimization algorithm. Training samples are obtained by labeling door and window sensor data collected under known non-intrusion conditions and different weather environments.

[0022] In this embodiment, to improve the accuracy of energy contribution estimation for environmental disturbance sources, environmental parameter data is used as auxiliary input features and concatenated with the hidden state of the final fully connected layer of the environmental disturbance source identification convolutional network before participating in the output layer computation. The environmental parameter data includes current wind speed, wind direction, and rainfall intensity. Wind speed and rainfall intensity are continuous numerical values, while wind direction is categorical data. Before concatenation, Z-score normalization is performed on wind speed and rainfall intensity, and one-hot encoding is performed on wind direction, ensuring that all environmental parameter data are uniformly represented numerically before concatenation with the hidden state. When the environmental parameter data indicates high wind speed or moderate to high rainfall intensity, the confidence weight of the energy contribution estimate for the wind pressure periodic vibration or raindrop impact response category in the environmental disturbance source identification convolutional network is correspondingly increased.

[0023] Step 3, Environmental interference energy spectrum stripping: Based on the estimated environmental interference energy contribution values ​​for each frequency band, the corresponding environmental interference energy components are subtracted from the vibration time-frequency spectrum matrix and the acoustic residual spectrum feature matrix band by band to generate the vibration residual time-frequency spectrum matrix and the acoustic residual spectrum feature matrix.

[0024] It should be noted that the frequency band subtraction operation is as follows: for each frequency band energy value in each time frame of the vibration time-frequency spectrum matrix and the acoustic spectral feature matrix, subtract the product of the estimated energy contribution of the identified environmental interference source in that frequency band and the energy value of that frequency band; when the subtraction result is negative, it is set to zero. The energy contribution estimate is a dimensionless proportion, and after multiplying it by the corresponding frequency band energy value, its dimensions are consistent with the subtracted quantity, so the subtraction operation can be performed directly. The vibration residual time-frequency spectrum matrix and the acoustic residual spectral feature matrix retain the residual signal energy distribution after excluding the contribution of the identified environmental interference sources.

[0025] In this embodiment, to avoid the accidental deletion of weak intrusion signal components due to excessive stripping, a conservative coefficient is applied to the estimated value of environmental interference energy contribution. The conservative coefficient is less than [value missing]. This ensures that the actual stripped energy is slightly lower than the estimated contribution of environmental interference energy, thus preserving edge band energy that may overlap with intrusion signals.

[0026] Step 4, Subthreshold micro-event detection in the residual domain: Perform inverse short-time Fourier transforms on the vibration residual time-spectrum matrix and the acoustic residual spectral feature matrix respectively to obtain the vibration residual signal and the acoustic residual signal. Calculate the residual energy change rate between adjacent sampling frames of the vibration residual signal and the acoustic residual signal.

[0027] When the residual energy change rate of a certain frame exceeds the preset micro-event sensitivity threshold, and the residual signal amplitude of the frame does not exceed the corresponding adaptive detection threshold, the frame is marked as a residual domain subthreshold micro-event, and the occurrence timestamp and residual energy change rate value of the residual domain subthreshold micro-event are recorded.

[0028] It should be noted that, let the first... The energy of the frame residual signal is The residual energy change rate is defined as: ;

[0029] in For the first Frame residual signal energy, For the first The rate of change of residual energy of the frame Index of the current frame. Molecules With denominator Both have the same dimensions, being units of signal energy; after division... This is a dimensionless ratio. The micro-event sensitivity threshold is used to filter out low-amplitude random fluctuations that still exist in the residual signal. The adaptive detection threshold is a conventional anomaly detection threshold independent of micro-event detection, used to distinguish between subthreshold micro-events in the residual domain and directly exceeding-threshold anomaly events.

[0030] Step 5, Extraction of temporal features of micro-events in the residual domain: Within a preset extended observation time window, temporal features are extracted from the residual domain subthreshold micro-event sequences detected for each modality. These temporal features include the frequency of micro-event occurrences, the average time interval between adjacent micro-events, the standard deviation of the time interval, and the slope of the increasing trend in the residual energy change rate.

[0031] It should be noted that the slope of the increasing trend of the residual energy change rate is the slope value obtained by performing a linear regression fitting on the residual energy change rate values ​​of subthreshold micro-events in each residual domain within the extended observation time window in chronological order. The positive slope characterizes the trend of increasing intensity of micro-events over time, and the characteristic of the increasing trend of the residual energy change rate corresponds to the physical process of gradually increasing force response due to the gradual loosening of the frame during a slow prying process.

[0032] In this embodiment of the application, to distinguish between the periodic force application pattern of the levering operation and random residual fluctuations, the time series characteristics also include the peak value of the autocorrelation coefficient calculated from the time interval sequence of adjacent micro-events. Assume there are a total of [number missing] events within the extended observation time window. Subthreshold micro-events in the residual domain To expand the total number of subthreshold micro-events in the residual domain within the observation time window, the first... The and the first The time interval between each residual domain subthreshold micro-event is , The sequence number of the micro-event and Then the time interval sequence of adjacent micro-events is The Pearson autocorrelation coefficient method is used to calculate the autocorrelation coefficient of the time interval sequence of adjacent micro-events. The peak value of the autocorrelation coefficient is the maximum value of the autocorrelation coefficient at all non-zero delays. When the peak value of the autocorrelation coefficient exceeds a preset periodicity threshold, it indicates that the subthreshold micro-event sequence in the residual domain has quasi-periodic distribution characteristics.

[0033] Z-score standardization is performed on the frequency of micro-events, the average time interval between adjacent micro-events, and the standard deviation of the time interval. Mean normalization based on the range is performed on the slope of the increasing trend of the residual energy change rate and the peak value of the autocorrelation coefficient. After scaling each time series feature to a similar numerical range, the time series features are concatenated into a time series feature vector to avoid the impact of dimensional differences on subsequent classification operations.

[0034] Step 6, Calculation of cross-modal synchronization correlation in the residual domain: Cross-modal time alignment analysis was performed on the residual domain subthreshold micro-event timestamp sequences of vibration and acoustic modes within the extended observation time window. The co-occurrence ratio of residual domain subthreshold micro-events occurring simultaneously in both modes within the preset synchronization tolerance time range was calculated, and the cross-modal synchronization correlation score of the residual domain was obtained.

[0035] It should be noted that the co-occurrence ratio is calculated as follows: Within the extended observation time window, for each residual domain subthreshold micro-event timestamp of the vibration mode, it is determined whether a corresponding residual domain subthreshold micro-event exists in the acoustic mode within the synchronization tolerance time range before and after that timestamp; the count of corresponding residual domain subthreshold micro-events is divided by the total number of residual domain subthreshold micro-events of the vibration mode to obtain the co-occurrence ratio from vibration to acoustic direction; the co-occurrence ratio from acoustic to vibration direction is calculated in the same way, that is, for each residual domain subthreshold micro-event timestamp of the acoustic mode, it is determined whether a corresponding residual domain subthreshold micro-event exists in the vibration mode within the synchronization tolerance time range before and after that timestamp, and the count of corresponding residual domain subthreshold micro-events is divided by the total number of residual domain subthreshold micro-events of the acoustic mode; the residual domain cross-modal synchronization correlation score is the mean of the co-occurrence ratio from vibration to acoustic direction and the co-occurrence ratio from acoustic to vibration direction. The co-occurrence ratios in both directions are dimensionless ratios, and the units of the mean calculation are consistent.

[0036] Step 7, Slow Intrusion Pattern Classification and Alarm Generation: The temporal feature vectors of each modality are concatenated with the cross-modal synchronization correlation scores in the residual domain to form a comprehensive feature vector, which is then input into a pre-trained slow intrusion pattern classification model, and the output is a slow intrusion probability value.

[0037] When the probability of a slow, spurious intrusion exceeds a preset cumulative alarm threshold, an alarm notification is generated and pushed to the user's terminal. The alarm notification includes an anomaly category label, an intrusion probability quantification value, and event time information.

[0038] It should be noted that the slow intrusion pattern classification model is a fully connected neural network classifier. The input layer of the slow intrusion pattern classification model receives the comprehensive feature vector, and the output layer outputs the probability value of slow intrusion through the Softmax activation function. The comprehensive feature vector is composed of vibration mode temporal feature vectors, acoustic mode temporal feature vectors, and residual domain cross-modal synchronization correlation scores. The vibration mode temporal feature vectors and acoustic mode temporal feature vectors each contain five temporal features: frequency of micro-event occurrences in the corresponding mode, average time interval between adjacent micro-events, standard deviation of time interval, slope of the increasing trend of residual energy change rate, and peak value of autocorrelation coefficient. The residual domain cross-modal synchronization correlation score is a scalar feature, and the comprehensive feature vector has eleven dimensions. The training samples include intrusion micro-event sequence features and non-intrusion micro-event sequence features annotated after residual domain processing under different environmental interference conditions. The cross-entropy loss function is used, and the Adam optimization algorithm is employed for training.

[0039] In this embodiment of the application, in order to reduce the risk of false alarms in a single window determination, the following steps are also included: when the probability value of slow prying intrusion exceeds the proportion threshold of the cumulative alarm threshold but does not reach the cumulative alarm threshold, the current extended observation time window is marked as a warning state, and the extended observation time window is slid forward; if the probability values ​​of slow prying intrusion in multiple consecutive sliding windows are all maintained above the warning range, an alarm notification is generated by combining the probability values ​​of multiple windows.

[0040] On the other hand, the present invention also proposes a door and window security alarm system based on multimodal perception, comprising: The signal acquisition and transformation module is used to acquire multimodal sensing data through vibration sensors and acoustic sensors, perform short-time Fourier transform on vibration signals and acoustic signals to obtain vibration time-frequency matrix and acoustic frequency characteristic matrix, and obtain environmental parameter data from meteorological data interface.

[0041] The environmental interference source identification module is used to input the vibration time spectrum matrix and the acoustic spectrum feature matrix into the environmental interference source identification convolutional network, and output the environmental interference type label and its energy contribution estimate for each frequency band.

[0042] The spectrum stripping module is used to subtract the corresponding environmental interference energy components from the vibration time spectrum matrix and acoustic spectrum feature matrix band by band based on the estimated environmental interference energy contribution values ​​of each frequency band, thereby generating the vibration residual time spectrum matrix and acoustic residual spectrum feature matrix.

[0043] The residual domain micro-event detection module is used to perform inverse short-time Fourier transform on the vibration residual time spectrum matrix and the acoustic residual spectrum feature matrix to obtain the vibration residual signal and the acoustic residual signal, and to detect and mark subthreshold micro-events in the residual domain based on the residual energy change rate.

[0044] The temporal feature extraction module is used to extract temporal features from the subthreshold micro-event sequence in the residual domain within an extended observation time window, including the frequency of micro-event occurrence, the average time interval between adjacent micro-events, the standard deviation of the time interval, and the slope of the increasing trend of the residual energy change rate.

[0045] The cross-modal synchronization analysis module is used to perform time alignment analysis on the residual domain subthreshold micro-event timestamp sequences of vibration modes and acoustic modes, and to calculate the cross-modal synchronization correlation score in the residual domain.

[0046] The classification and alarm module is used to concatenate the temporal feature vector and the cross-modal synchronization correlation score of the residual domain and input it into the slow intrusion mode classification model. It outputs the slow intrusion probability value and generates an alarm notification when the slow intrusion probability value exceeds the cumulative alarm threshold.

[0047] By way of example, the system proposed in this invention may also include various features and combinations thereof in the method embodiments, which will not be elaborated here.

[0048] This invention introduces environmental interference spectrum stripping processing before residual domain subthreshold micro-event detection. An environmental interference source identification convolutional network identifies the energy contributions of environmental interference sources such as wind pressure periodic vibration and raindrop impact response in each frequency band, and subtracts the corresponding components from the vibration time-spectrum matrix and acoustic spectral feature matrix band by band. Because the energy contributions of environmental interference sources are stripped before the signal enters the residual domain subthreshold micro-event detection process, the vibration and acoustic residual signals no longer contain frequent energy fluctuations caused by wind pressure pulsations and raindrop impacts. Pseudo-micro-events generated by environmental interference are no longer registered in batches, and the event density of the residual domain subthreshold micro-event sequence is significantly reduced. Simultaneously, because the synchronous energy contributions of environmental interference sources in the vibration and acoustic channels are synchronously stripped, the residual domain cross-modal synchronization correlation score is no longer inflated by environmental co-occurrence effects. The slow intrusion pattern classification model can effectively extract the temporal regularity and true cross-modal correlation features of intrusion operations from the residual domain subthreshold micro-event sequence, thus maintaining the ability to detect slow-moving intrusion behavior even in continuously severe weather conditions.

[0049] See Figures 2-9 The following application examples are proposed: The intelligent door and window security system of a residential community operated continuously during a period of continuous rainfall in the autumn of a certain year. The monitored object was a north-facing exterior window, with vibration and acoustic sensors deployed on the window frame. During this period, the wind speed remained at a moderate to high level, and the rainfall intensity was moderate rain. Wind pressure pulsations and raindrop impacts simultaneously generated frequent energy fluctuations on both sensors. An edge computing processor performed signal processing and model inference in real time. Under these adverse weather conditions, the system needed to identify any slow prying intrusion attempts.

[0050] Step 1 execution process: Vibration and acoustic sensors continuously acquire signals at a fixed sampling rate. Short-time Fourier transforms are performed on the acquired vibration and acoustic signals respectively, with the window length determined based on the low-frequency response range of the prying operation to ensure low-frequency components are covered. The transform results form a vibration time-frequency matrix and an acoustic frequency-frequency feature matrix, with rows corresponding to time frames and columns corresponding to frequency components. Simultaneously, current environmental parameter data is read from a meteorological data interface to obtain real-time values ​​for wind speed, wind direction, and rainfall intensity.

[0051] Table 1. Overview of environmental parameter data and time-frequency matrix for Step 1

[0052]

[0053] Step 2 execution process: The vibration frequency spectrum matrix and acoustic frequency spectrum feature matrix are fed into the environmental interference source identification convolutional network. With a wind speed of 8.3 m / s and moderate rainfall, the wind speed and rainfall intensity provided by the meteorological interface are Z-score normalized, and the wind direction "northwest" is converted into a numerical vector through one-heat encoding. These three types of parameters are concatenated to the hidden state of the final fully connected layer before participating in the output layer computation. Under high wind speed and moderate rain conditions, the confidence weights of the convolutional network for wind pressure periodic vibration and raindrop impact response categories are correspondingly increased. The output results show that the low-frequency band (2 Hz to 20 Hz) is mainly identified as wind pressure periodic vibration, the mid-frequency band (20 Hz to 50 Hz) is mainly identified as raindrop impact response, and the high-frequency band (50 Hz to 80 Hz) is mainly identified as structurally conducted vibration.

[0054] Table 2. Environmental interference identification results for each frequency band of the vibration channel.

[0055]

[0056] Table 3. Environmental interference identification results for each frequency band of the acoustic channel

[0057]

[0058] Step 3 execution process:

[0059] Based on the energy contribution estimates for each frequency band output in step 2, environmental interference energy stripping is performed band by band on the vibration time-spectrum matrix and acoustic spectral feature matrix. To avoid over-stripping, a conservative coefficient of 0.85 is applied to all energy contribution estimates, resulting in an actual stripping ratio lower than the estimated value. Taking the low-frequency band of the vibration channel as an example, the actual stripping ratio after multiplying the wind pressure cycle vibration energy contribution estimate of 0.78 by the conservative coefficient is 0.66. The energy value of each time frame in this frequency band is subtracted from its own product with 0.66, and the result is set to zero when it is negative. After stripping, the vibration residual time-spectrum matrix and acoustic residual spectral feature matrix are generated, both of which retain the residual signal energy distribution after excluding environmental interference contributions.

[0060] Table 4. Energy comparison before and after low-frequency stripping of the vibration channel (partial time frames)

[0061]

[0062] Step 4 execution process: Inverse short-time Fourier transforms are performed on the vibration residual time-frequency matrix and the acoustic residual spectral characteristic matrix, respectively, to restore the time-domain sequences of the vibration residual signal and the acoustic residual signal. The rate of change of residual energy is calculated for each sampling frame. ,in For the first Frame residual signal energy, For the first Frame residual signal energy, It is a dimensionless ratio. When a certain frame... If the micro-event sensitivity threshold (set to 0.18) is exceeded and the residual amplitude of the frame does not exceed the adaptive detection threshold, the frame is marked as a residual domain subthreshold micro-event and its timestamp is recorded. Since environmental interference energy has been stripped away in step 3, the frequent energy fluctuations caused by wind pressure pulsations and raindrop impacts in the residual signal are significantly reduced, and the event density of registered subthreshold micro-events in the residual domain is significantly lower than that without stripping.

[0063] Table 5. Vibration Channel Residual Domain Subthreshold Micro-Event Registration Record

[0064]

[0065] Step 5 execution process: Within a preset extended observation time window (120 seconds), temporal features were extracted from the residual domain subthreshold micro-event sequences of vibration and acoustic modes. Four residual domain subthreshold micro-events were detected in the vibration channel, and four were also detected in the acoustic channel. Taking the vibration channel as an example: the micro-events occurred four times, with adjacent time intervals of 17 seconds, 18 seconds, and 19 seconds, respectively, an average time interval of 18 seconds, and a standard deviation of 0.82 seconds; for the four micro-events... The values ​​(0.23, 0.27, 0.31, 0.36) were subjected to linear regression in chronological order. The positive slope of the fitted values ​​indicates a gradually increasing response to each applied force, corresponding to the physical process of gradual loosening of the framework. This was applied to the time interval sequence. The Pearson autocorrelation coefficient was calculated, and a high peak value was observed at a 1-second delay, indicating that the time interval sequence exhibits a quasi-periodic distribution. Z-score standardization was applied to the frequency of micro-events, average time interval, and standard deviation of time interval. The slope of the increasing trend and the peak value of the autocorrelation coefficient were normalized to mean based on the range and then concatenated to form a five-dimensional time series feature vector.

[0066] Table 6 Summary of temporal characteristics of vibration and acoustic channels

[0067]

[0068] Step 6 execution process: Cross-modal time alignment analysis was performed on the residual domain subthreshold micro-event timestamp sequences of vibration and acoustic modes within the extended observation time window, with a synchronization tolerance time range set to 1.5 seconds. Of the four micro-events in the vibration channel, three co-occurred within 1.5 seconds before and after the corresponding timestamp in the acoustic channel, with a vibration-to-acoustic co-occurrence ratio of 0.75. Similarly, of the four micro-events in the acoustic channel, three co-occurred within 1.5 seconds before and after the corresponding timestamp in the vibration channel, with an acoustic-to-vibration co-occurrence ratio of 0.75. The residual domain cross-modal synchronization correlation score, calculated as the average of the co-occurrence ratios in both directions, was 0.75. It should be noted that since the synchronization energy contribution of environmental interference was synchronously stripped in step 3, this correlation score reflects the true intrusion correlation after excluding environmental co-occurrence effects, rather than a false co-occurrence boosted by environmental synchronization.

[0069] Step 7 execution process: The five-dimensional temporal feature vectors of the vibration mode, the five-dimensional temporal feature vectors of the acoustic mode, and the cross-modal synchronization correlation score (0.75) in the residual domain are concatenated into an eleven-dimensional comprehensive feature vector, which is then input into the slow intrusion pattern classification model. The output layer of the slow intrusion pattern classification model, after being activated by the Softmax function, outputs a slow prying intrusion probability value of 0.88. This exceeds the preset cumulative alarm threshold of 0.75, prompting the system to generate an alarm notification and push it to the user's terminal. The alarm notification includes the anomaly category label "slow prying intrusion," the intrusion probability quantification value of 0.88, and event time information.

[0070] Table 7. Comprehensive Feature Vectors and Classification Output Results

[0071]

[0072] Throughout the processing, the data starts from the original vibration and acoustic signals. Step 1 involves time-frequency transformation to form a matrix representation. Step 2 introduces meteorological parameters to assist in identifying the types and energy contributions of environmental interference in each frequency band. Step 3 transforms the identification results into a frequency band stripping operation, ensuring that the subthreshold micro-events registered in the residual domain in Step 4 no longer include pseudo-events caused by environmental interference. Step 5 extracts time-series features with quasi-periodicity and increasing trends from the sparsified micro-event sequence. Step 6 calculates the cross-modal synchronization correlation degree on the dual channels, which have also undergone interference stripping, ensuring that this score is unaffected by environmental co-occurrence effects. Step 7 aggregates the above features into an eleven-dimensional comprehensive feature vector, based on which the classification model outputs a high-confidence probability of slow, lurking intrusion, triggering an alarm. The output data from each step is passed sequentially, with the stripping effect of the preceding step directly determining the quality of the subsequent feature, forming a complete causal data flow chain.

[0073] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A door and window security alarm method based on multimodal perception, characterized in that, include: Short-time Fourier transforms are performed on the vibration signals and acoustic signals collected by the vibration sensor and acoustic sensor, respectively, to obtain the vibration time-frequency matrix and the acoustic frequency characteristic matrix. The vibration time spectrum matrix and acoustic spectrum feature matrix are respectively input into the environmental interference source identification convolutional network to output the environmental interference type label and energy contribution estimate for each frequency band. Based on the energy contribution estimate, the environmental interference energy component is subtracted from the vibration time-spectrum matrix and acoustic residual spectrum feature matrix band by band to generate the vibration residual time-spectrum matrix and acoustic residual spectrum feature matrix. Perform inverse short-time Fourier transform on the vibration residual time spectrum matrix and the acoustic residual spectrum feature matrix to obtain the residual signal, and detect subthreshold micro-events in the residual domain based on the residual energy change rate; Temporal features are extracted from the subthreshold micro-event sequence in the residual domain within the extended observation time window, and the cross-modal synchronization correlation score in the residual domain is calculated. The temporal features and the residual domain cross-modal synchronization correlation score are concatenated and input into the slow intrusion mode classification model. When the output slow prying intrusion probability value exceeds the cumulative alarm threshold, an alarm notification is generated.

2. The door and window security alarm method based on multimodal perception according to claim 1, characterized in that, The output layer of the environmental interference source identification convolutional network contains two parallel branches: a classification branch outputs environmental interference type labels for each frequency band using a Softmax activation function, and a regression branch outputs corresponding energy contribution estimates using a Sigmoid activation function. The environmental interference type labels include wind pressure periodic vibration, raindrop impact response, and structural conducted vibration. The energy contribution estimates characterize the proportion of signal energy occupied by each type of environmental interference source in the corresponding frequency band, with a value range of [range missing]. ,in This indicates that this type of interference source does not contribute energy. This indicates that all the energy in this frequency band is contributed by this type of interference source.

3. The door and window security alarm method based on multimodal perception according to claim 2, characterized in that, Also includes: The environmental parameter data for the current time period is obtained from the meteorological data interface, including wind speed, wind direction, and rainfall intensity. Z-score normalization is performed on wind speed and rainfall intensity, and one-hot encoding is performed on wind direction. The normalized environmental parameter data is concatenated with the hidden state of the last fully connected layer of the environmental interference source identification convolutional network and then used in the output layer operation. When the environmental parameter data indicates high wind speed or moderate to high rainfall intensity, the confidence weight of the energy contribution estimate of the environmental interference source identification convolutional network for the wind pressure periodic vibration or raindrop impact response category is increased accordingly.

4. The door and window security alarm method based on multimodal perception according to claim 1, characterized in that, The operation of subtracting the environmental interference energy component band by band is as follows: for each frequency band energy value in each time frame of the vibration time-frequency spectrum matrix and the acoustic spectrum feature matrix, subtract the product of the identified energy contribution estimate of that frequency band and the energy value of that frequency band; when the subtraction result is negative, set it to zero; apply a conservatism coefficient to the energy contribution estimate, the conservatism coefficient being less than... ,in The baseline ratio without any attenuation is such that the actual stripped energy is lower than the energy indicated by the energy contribution estimate.

5. The door and window security alarm method based on multimodal perception according to claim 1, characterized in that, The method for detecting subthreshold micro-events in the residual domain based on the residual energy change rate includes: calculating the residual energy change rate between adjacent sampling frames of the residual signal, assuming the ... The energy of the frame residual signal is , No. The energy of the frame residual signal is Then the first Rate of change of residual energy of frames for: ； When the Rate of change of residual energy of frames If the micro-event sensitivity threshold is exceeded and the residual signal amplitude of the frame does not exceed the adaptive detection threshold, the frame is marked as a residual domain subthreshold micro-event, and its occurrence timestamp and residual energy change rate value are recorded.

6. The door and window security alarm method based on multimodal perception according to claim 1, characterized in that, The temporal features include the frequency of micro-event occurrence, the average time interval between adjacent micro-events, the standard deviation of the time interval, the slope of the increasing trend of the residual energy change rate, and the peak value of the autocorrelation coefficient; wherein, the slope of the increasing trend of the residual energy change rate is the slope value obtained by performing linear regression fitting on the residual energy change rate values ​​of each residual domain subthreshold micro-event within the extended observation time window in chronological order; the peak value of the autocorrelation coefficient is the maximum value of the autocorrelation coefficient at all non-zero delays after calculating the Pearson autocorrelation coefficient for the time interval sequence of adjacent micro-events.

7. The door and window security alarm method based on multimodal perception according to claim 6, characterized in that, Z-score standardization was performed on the frequency of micro-events, the average time interval between adjacent micro-events, and the standard deviation of the time interval. Mean normalization based on the range was performed on the slope of the increasing trend of the residual energy change rate and the peak value of the autocorrelation coefficient. After scaling each time series feature to a similar numerical range, the time series features were concatenated into a time series feature vector.

8. The door and window security alarm method based on multimodal perception according to claim 1, characterized in that, The calculation method for the residual domain cross-modal synchronization correlation score is as follows: within the extended observation time window, for each residual domain subthreshold micro-event timestamp of the vibration mode, it is determined whether there is a corresponding residual domain subthreshold micro-event in the acoustic mode within the synchronization tolerance time range before and after the timestamp. The count of the corresponding residual domain subthreshold micro-event is divided by the total number of residual domain subthreshold micro-events of the vibration mode to obtain the co-occurrence ratio from vibration to acoustic direction. The co-occurrence ratio from acoustic to vibration direction is calculated in the same manner; the residual domain cross-modal synchronization correlation score is the average of the co-occurrence ratio from vibration to acoustic direction and the co-occurrence ratio from acoustic to vibration direction.

9. The door and window security alarm method based on multimodal perception according to claim 1, characterized in that, Also includes: When the probability value of the slow prying intrusion exceeds the proportion threshold of the cumulative alarm threshold but does not reach the cumulative alarm threshold, the current extended observation time window is marked as an early warning state and the extended observation time window is slid backward. If the probability values ​​of slow intrusion through multiple sliding windows remain above the warning range, an alarm notification will be generated based on the combined probability values ​​of the multiple windows.

10. A door and window security alarm system based on multimodal perception, used to execute the door and window security alarm method based on multimodal perception as described in any one of claims 1 to 9, characterized in that, include: The signal acquisition and transformation module is used to acquire multimodal sensing data through vibration sensors and acoustic sensors, and to perform short-time Fourier transform on vibration signals and acoustic signals to obtain vibration time-frequency matrix and acoustic frequency characteristic matrix; The environmental interference source identification module is used to input the vibration time spectrum matrix and the acoustic spectrum feature matrix into the environmental interference source identification convolutional network, and output the environmental interference type label and energy contribution estimate for each frequency band. The spectrum stripping module is used to subtract the environmental interference energy component from the vibration time spectrum matrix and the acoustic spectrum feature matrix band by band based on the energy contribution estimate, so as to generate the vibration residual time spectrum matrix and the acoustic residual spectrum feature matrix. The residual domain micro-event detection module is used to perform inverse short-time Fourier transform on the vibration residual time spectrum matrix and the acoustic residual spectrum feature matrix to obtain the residual signal, and to detect and mark the residual domain subthreshold micro-events based on the residual energy change rate. The temporal feature extraction module is used to extract temporal features from the subthreshold micro-event sequence in the residual domain within an extended observation time window; The cross-modal synchronization analysis module is used to perform time alignment analysis on the residual domain subthreshold micro-event timestamp sequences of vibration modes and acoustic modes, and to calculate the cross-modal synchronization correlation score in the residual domain. The classification and alarm module is used to concatenate the temporal features with the cross-modal synchronization correlation score of the residual domain and input them into the slow intrusion mode classification model. When the probability value of slow intrusion exceeds the cumulative alarm threshold, an alarm notification is generated.

Images