Security alarm method, device and system based on multi-sensor fusion

Abstract

The application is suitable for the technical field of security data processing, and provides a security early warning method, device and system based on multi-sensor fusion. The method comprises the following steps: when a vibration event is detected in the signal collected by a DAS unit, classifying the vibration event to obtain a vibration classification result, the vibration event comprising a vibration position and a vibration time; acquiring an acoustic signal collected by a target microphone within a preset time after the vibration time, and an image sequence of the vibration position region collected by a target camera within a preset time after the vibration time; classifying the acoustic signal to obtain an acoustic classification result, and classifying the image sequence to obtain an image classification result; determining a threat value of the vibration event according to the vibration classification result, the acoustic classification result and the image classification result, and performing security early warning based on the threat value. The application can improve the detection accuracy and reliability of perimeter security.

Description

Technical Field

[0001] This application relates to the field of security data processing technology, specifically to a security early warning method, device, and system based on multi-sensor fusion. Background Technology

[0002] Perimeter security is a core technological means to protect the safety of important facilities, construction areas, and critical infrastructure. Integrating physical protection with intelligent sensing technology, perimeter security uses detection, interception, and alarm systems to provide early warning and proactive deterrence against intrusion, serving as the first line of defense for these areas. Traditional perimeter security technologies mostly rely on single-sensor technology, resulting in low detection accuracy and poor reliability. Summary of the Invention

[0003] In view of this, embodiments of this application provide a security early warning method, device and system based on multi-sensor fusion, which can improve the detection accuracy and reliability of perimeter security.

[0004] To achieve the above objectives, this application adopts the following technical solution:

[0005] In a first aspect, this application provides a security early warning method based on multi-sensor fusion, comprising: when a vibration event is detected in a signal collected by a DAS unit, classifying the vibration event to obtain a vibration classification result, wherein the vibration event includes a vibration location and a vibration time; wherein a DAS unit, a microphone array, and a camera array are provided at the boundary of the protected area; acquiring an acoustic signal collected by a target microphone within a preset time after the vibration time, and an image sequence of the vibration location area collected by a target camera within a preset time after the vibration time; wherein the target microphone is at least one microphone in the microphone array closest to the vibration location, and the target camera is at least one camera in the camera array closest to the vibration location; classifying the acoustic signal to obtain an acoustic classification result, and classifying the image sequence to obtain an image classification result; determining the threat value of the vibration event based on the vibration classification result, the acoustic classification result, and the image classification result, and providing a security early warning based on the threat value.

[0006] Based on the first aspect, in some embodiments, the process of detecting vibration events from signals acquired by the DAS unit includes: denoising the original signals acquired by the DAS unit using a bandpass filter with a passband of 1Hz to 500Hz; and processing the denoised signals... ,according to Calculate signal The short-term energy, according to Calculate signal The short-time zero-crossing rate; where W is the number of continuous signal sampling points included in the sliding time window; when the short-time energy Exceeding the energy threshold E th And short-term zero crossing rate When within the preset range, the determination signal is... A vibration event occurs at the corresponding location; where the energy threshold E th Determined based on the mean and standard deviation of the background noise energy of the original signal.

[0007] Based on the first aspect, in some embodiments, classifying the vibration event to obtain a vibration classification result includes: extracting multi-domain features from the signal corresponding to the vibration event, wherein the multi-domain features include time-domain features, frequency-domain features, and time-frequency features; inputting the multi-domain features into a lightweight CNN classifier to classify the vibration event, thereby obtaining a vibration classification result of the vibration event and a first confidence level of the vibration classification result.

[0008] Based on the first aspect, in some embodiments, acquiring the acoustic signal collected by the target microphone within a preset time after the vibration moment, and the image sequence of the vibration location area collected by the target camera within a preset time after the vibration moment, includes: when a vibration event is detected in the signal collected by the DAS unit, waking up the target microphone and the target camera according to the vibration location; acquiring the acoustic signal collected by the target microphone within a preset time after the vibration moment, and the image sequence of the vibration location area collected by the target camera within a preset time after the vibration moment;

[0009] The process of classifying the acoustic signal to obtain an acoustic classification result and classifying the image sequence to obtain an image classification result includes: extracting the Mel frequency cepstral coefficients of the acoustic signal to form a voiceprint feature vector, inputting the voiceprint feature vector into a voiceprint recognition model for classification and recognition, and obtaining an acoustic classification result and a second confidence level of the acoustic classification result; acquiring an image sequence of the vibration location area collected by the target camera within a preset time after the vibration moment, preprocessing the image sequence and inputting it into an infrared target detection model, and obtaining an image classification result and a third confidence level of the image classification result.

[0010] Based on the first aspect, in some embodiments, determining the threat value of the vibration event according to the vibration classification result, the acoustic classification result, and the image classification result includes:

[0011] According to S=w1×P DAS +w2×P audio +w3×P IR +w4×I(F DA )+w5×I(F DI )+w6×I(FAI Calculate the threat value, P, of the vibration event. DAS P represents the confidence level of the vibration classification results. audio P represents the confidence level of the acoustic classification results. IR F represents the confidence level of the image classification result. DA For vibrational acoustic marking, F DI For marking vibration images, F AI For acoustic image labeling, vibration acoustic labeling represents the consistency verification results between vibration classification results and acoustic classification results; for vibration image labeling, it represents the consistency verification results between vibration classification results and image classification results; for acoustic image labeling, it represents the consistency verification results between acoustic classification results and image classification results. The indicator function is w1 to w6, which are the weights.

[0012] Based on the first aspect, in some embodiments, if the vibration classification result is an intrusion-type result, the acoustic classification result is consistent with the vibration classification result, and P DAS With P audio If the product of F and F is greater than the bimodal verification threshold, then F DA The value is 1, otherwise F DA The value is 0; if the vibration classification result is an intrusion result, a human target exists in the vibration location region in the image classification result, and P IR If it is greater than the image verification threshold, then F DI The value is 1, otherwise F DI The value is 0; if the acoustic classification result is an intrusion result, and the image classification result shows a heat source target in the vibration location region, then F AI The value is 1, otherwise F AI The value is 0.

[0013] Based on the first aspect, in some embodiments, the method includes: according to S'=α1×w1×P DAS +α2×w2×P audio +α3×w3×P IR +α4×w4×I(F DA )+α5×w5×I(F DI )+α6×w6×I(F AI Adaptive compensation is performed on the threat value of the vibration event. i can be 1 to 6. Let be the function of the influence of environmental factors on the i-th component;

[0014] Where, δ1(env) = -η wind ×max(0,(V-V0) / (V th -V0), where V is the current wind speed, V0 is the lower limit of the wind speed at which compensation is initiated, and V thη is the wind speed threshold. wind For the preset coefficient, 0.5≤η wind ≤0.8; δ2(env)= -η rain ×max(0,(R-R0) / (R th -R0), where R is the current rainfall intensity, R0 is the lower limit of rainfall intensity for triggering compensation, and R th η is the rainfall intensity threshold. rain For the preset coefficient, 0.6≤η rain ≤0.8; δ3(env)= -η temp ×(1-|T env -T human | / T range ), T env For ambient temperature, T human For human body temperature, T range For reference temperature range, i.e., the baseline value of the temperature difference between typical ambient temperature and human body temperature, η is generally taken as 12 in a comfortable environment. temp A preset coefficient for temperature sensitivity is provided, 0 ≤ η temp ≤1.

[0015] Based on the first aspect, in some embodiments, the security early warning based on the threat value includes: if the threat value is greater than the high-level alarm threshold, an audible and visual alarm is triggered, and alarm information is pushed to the management personnel terminal. The alarm information includes the coordinates of the intrusion location, a thermal imaging screenshot of the scene, and the voiceprint classification result. At the same time, the video recording system is linked to save complete audio and video evidence of the scene. If the threat value is greater than the intermediate alarm threshold but less than or equal to the high-level alarm threshold, an early warning prompt is issued to the management personnel terminal, and the management personnel perform manual verification by remotely retrieving real-time thermal imaging footage and real-time voiceprint audio. The high-level alarm threshold is greater than the intermediate alarm threshold. If the threat value is less than or equal to the intermediate alarm threshold, it is determined to be an environmental disturbance event, and the event log is recorded but no alarm is triggered.

[0016] Secondly, embodiments of this application provide a security early warning device based on multi-sensor fusion, comprising:

[0017] The vibration classification module is used to classify vibration events when they are detected in the signals collected by the DAS unit to obtain vibration classification results. The vibration events include vibration location and vibration time. DAS unit, microphone array and camera array are set at the boundary of the protected area.

[0018] The acquisition module is used to acquire acoustic signals collected by the target microphone within a preset time after the vibration moment, and image sequences of the vibration location area collected by the target camera within a preset time after the vibration moment; wherein, the target microphone is at least one microphone in the microphone array that is closest to the vibration location, and the target camera is at least one camera in the camera array that is closest to the vibration location;

[0019] An acoustic image classification module is used to classify the acoustic signals to obtain acoustic classification results and to classify the image sequences to obtain image classification results;

[0020] The early warning module is used to determine the threat value of the vibration event based on the vibration classification result, the acoustic classification result, and the image classification result, and to provide security early warning based on the threat value.

[0021] Thirdly, embodiments of this application provide a security early warning method system based on multi-sensor fusion, including a DAS unit, a microphone array, a camera array, and a terminal device. The DAS unit includes a sensing optical cable laid along the perimeter of the protection area and a DAS demodulation host. The microphone array and the camera array are deployed at a preset interval along the perimeter of the protection area. The terminal device includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the security early warning method based on multi-sensor fusion as described in any of the first aspects above.

[0022] The beneficial effects of the embodiments of this application compared with the prior art include:

[0023] This application first utilizes DAS (Direct Adaptive Sensor) to achieve long-distance, full-perimeter vibration event detection and precise positioning. Secondly, the DAS positioning results activate a silicon photonic acoustic microphone and a thermal imaging camera at the event location for directional secondary detection. Finally, a three-level progressive fusion decision-making process—dual-modal initial screening, weighted comprehensive scoring, and adaptive environmental compensation—achieves mutual verification and complementarity of the three sensor detection results. This application overcomes the inherent deficiencies of single-sensor technologies in terms of false alarm rate, false alarm rate, and environmental adaptability, achieving high-precision, low-false-alarm, all-weather intelligent protection for scenarios such as anti-theft and anti-illegal construction intrusion in construction areas. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating the security early warning method based on multi-sensor fusion provided in an embodiment of this application;

[0026] Figure 2 This is a flowchart illustrating a specific application of the security early warning method based on multi-sensor fusion provided in this application embodiment;

[0027] Figure 3 This is a flowchart of DAS signal preprocessing and time-frequency feature extraction provided in the embodiments of this application;

[0028] Figure 4 This is a flowchart of the voiceprint recognition processing provided in the embodiments of this application;

[0029] Figure 5 This is a flowchart of the thermal imaging target detection processing provided in the embodiments of this application;

[0030] Figure 6 This is a structural block diagram of a security early warning device based on multi-sensor fusion provided in an embodiment of this application;

[0031] Figure 7 This is a structural block diagram of the terminal device provided in the embodiments of this application. Detailed Implementation

[0032] The present application will be described more clearly below with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the function of the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0034] Infrared beam detectors experience a significantly higher false alarm rate in adverse weather conditions such as rain, snow, fog, and haze, making them unreliable for all-weather operation. Visible light video surveillance systems suffer a sharp decline in performance at night in complete darkness or under strong backlight conditions, leading to numerous missed alarms. Vibration cable and leaky cable systems are highly susceptible to electromagnetic interference, resulting in limited positioning accuracy and requiring continuous power supply, making them ill-suited for harsh environments. While traditional electronic fences provide physical obstruction, they suffer from drawbacks such as being readily observable and easily circumvented, and are sensitive to environmental factors like strong winds.

[0035] Distributed Acoustic Sensing (DAS) is a rapidly developing new sensing technology in recent years. Based on the principle of coherent Rayleigh scattering, DAS utilizes the sensitivity of optical fibers to sound and vibration to achieve distributed measurement. When external vibration acts on the sensing fiber, due to the photoelastic effect, the refractive index and length of the fiber undergo minute changes, causing a phase change in the transmitted signal within the fiber. By detecting the intensity change (differential signal) of the Rayleigh scattered light signal before and after the vibration, vibration events can be detected and multiple events can be simultaneously and accurately located. DAS systems offer ultra-long measurement distances of up to 100 kilometers, positioning accuracy of 1 meter, and a typical fast response time of 1 second. Furthermore, the optical fiber itself serves as the sensor, integrating transmission and sensing functions. It is completely electrically insulated, resistant to electromagnetic interference, and suitable for various complex environments.

[0036] Despite the significant advantages of DAS technology, such as long-distance operation, distributed operation, and all-weather capability, relying solely on DAS technology in practical applications still presents the following problems: It is difficult to accurately distinguish intrusion types (e.g., pedestrians, vehicles, construction machinery, animal activity, wind and rain disturbances) based solely on vibration signals; the signal-to-noise ratio of vibration signals may decrease in complex environmental noise backgrounds, leading to a decline in identification accuracy; and the lack of visual confirmation means that it cannot provide on-site image evidence. Similarly, while a single thermal imaging camera can operate in completely dark conditions and detect intruders through temperature differences, it lacks vibration sensing and acoustic recognition capabilities; while a single silicon photonics acoustic microphone can capture ambient sound, its localization capabilities are limited and it lacks image evidence.

[0037] Based on this, this application provides a security early warning method, device, and system based on multi-sensor fusion. The system includes a distributed fiber optic acoustic / vibration sensing (DAS) unit, a point-type silicon photonic acoustic microphone array, a thermal imaging camera array, and terminal equipment. First, the DAS is used to achieve long-distance, full-perimeter vibration event detection and precise positioning. Second, the DAS positioning result is used to activate the silicon photonic acoustic microphone and thermal imaging camera at the event location for directional secondary detection. Finally, a three-level progressive fusion decision-making process—dual-modal initial screening, weighted comprehensive scoring, and adaptive environmental compensation—is used to achieve mutual verification and complementarity of the detection results from the three sensors. This application overcomes the inherent defects of single-sensor technology in terms of false alarm rate, false alarm rate, and environmental adaptability, achieving high-precision, low-false-alarm, and all-weather intelligent protection for scenarios such as anti-theft in construction areas and prevention of illegal construction intrusion.

[0038] Specifically, a security early warning system based on multi-sensor fusion may include a DAS unit, a microphone array (e.g., a dot-type silicon photoacoustic microphone array), a camera array (e.g., a thermal imaging camera array), and terminal equipment. The DAS unit includes a sensing optical cable laid along the protection perimeter and a DAS demodulation host. The microphone array and camera array are deployed at preset intervals along the protection perimeter.

[0039] The sensing optical cable can be attached to a perimeter fence or laid underground to collect vibration signals along the entire fiber in real time. The DAS demodulation host injects narrow-linewidth laser pulses into the optical fiber and analyzes the phase and intensity changes of the Rayleigh backscattered signal. Through a phase demodulation algorithm, the original optical signal is converted into vibration waveform data, enabling multi-point real-time detection, location, and recording of vibration events.

[0040] Silicon photonic acoustic microphones can be deployed at 50m intervals at key perimeter locations (such as fence corners, near entrances / exits, and boundaries of important equipment storage areas) for high-sensitivity acoustic signal acquisition. Utilizing silicon photonic integration technology, these microphones convert the diaphragm vibrations caused by sound waves into optical signals, which are then converted into electrical signals, achieving high-sensitivity acoustic signal acquisition with a sensitivity exceeding 100mV / Pa and a frequency response range covering 20Hz to 20kHz.

[0041] Thermal imaging camera arrays can be deployed along the perimeter at intervals of 200m to 300m, mounted on poles 3m to 5m high. The field of view of each camera covers the area between its adjacent cameras, achieving seamless coverage of the entire perimeter. Based on the principle of infrared thermal radiation, thermal imaging cameras detect the temperature difference between the target and the environment to achieve all-weather imaging and detection of intrusion targets. They can operate normally under adverse weather conditions such as complete darkness, strong light, backlight, and fog, with a detection range of up to several hundred meters.

[0042] The DAS unit, silicon photonic acoustic microphone array, and thermal imaging camera array together form a three-in-one sensing network. The DAS unit provides continuous vibration sensing and precise positioning capabilities across the entire perimeter, the silicon photonic acoustic microphone provides acoustic feature sensing capabilities at key nodes, and the thermal imaging camera provides full-coverage infrared visual sensing capabilities.

[0043] The terminal device processes the signals collected by the DAS unit, microphone array, and camera array to implement the aforementioned security early warning method based on multi-sensor fusion. For details, please refer to the detailed description of the security early warning method based on multi-sensor fusion.

[0044] Figure 1 This is a flowchart illustrating the security early warning method based on multi-sensor fusion provided in an embodiment of this application. Figure 2For a flowchart illustrating the specific application of the security early warning method based on multi-sensor fusion provided in this application embodiment, please refer to... Figure 1 and Figure 2 The above-mentioned security early warning method based on multi-sensor fusion is described in detail below:

[0045] Step 101: When a vibration event is detected in the signal acquired by the DAS unit, the vibration event is classified to obtain a vibration classification result. The vibration event includes the vibration location and vibration time.

[0046] In some embodiments, the process of detecting vibration events from signals acquired by the DAS unit includes: denoising the original signal acquired by the DAS unit using a bandpass filter with a passband of 1Hz to 500Hz; and processing the denoised signal... ,according to Calculate signal The short-term energy, according to Calculate signal The short-time zero-crossing rate; where W is the number of continuous signal sampling points included in the sliding time window; when the short-time energy Exceeding the energy threshold E th And short-term zero crossing rate When within the preset range, the determination signal is... A vibration event occurs at the corresponding location; where the energy threshold E th Determined based on the mean and standard deviation of the background noise energy of the original signal.

[0047] For details, see Figure 3 A bandpass filter with a bandwidth of 1Hz to 500Hz can be used to denoise the original phase differential signal. This 1Hz to 500Hz range covers the vibration frequencies of major intrusion behaviors such as walking, climbing, and mechanical construction, while effectively filtering out low-frequency ground noise and high-frequency circuit noise. Then, an adaptive noise canceller is applied, using environmental noise during event-free periods as a reference, to further suppress periodic interference such as power frequency harmonics. Vibration event segments are extracted using a dual-threshold mechanism of Short-Time Energy (STE) and Zero-Crossing Rate (ZCR). Exceeding the energy threshold E th and Within the preset range [Z] min Z max When the vibration event occurs within a certain range, it is determined that a vibration event has occurred at that location.

[0048] Energy threshold E th An adaptive mechanism is used to calculate the average energy E of the background noise in real time within a 10-minute background learning window. bg and standard deviation δbg E th = E bg + k·δ bg (k is typically set to 4–6). This mechanism ensures that the system maintains optimal detection sensitivity in both noisy daytime and quiet nighttime environments.

[0049] In this embodiment of the application, the arrival time difference of the Rayleigh backscattered light signal is calculated, and the refractive index of the optical fiber is combined to solve the location x of the event, and finally the precise spatiotemporal coordinates (x,t) of the event are output. This is a well-known technique and will not be described in detail here.

[0050] In some embodiments, classifying vibration events to obtain vibration classification results may include: extracting multi-domain features from the signal corresponding to the vibration event, wherein the multi-domain features include time-domain features, frequency-domain features, and time-frequency features; inputting the multi-domain features into a lightweight CNN classifier to classify the vibration event, thereby obtaining the vibration classification result of the vibration event and a first confidence level of the vibration classification result.

[0051] For details, see Figure 3 When the DAS unit detects a vibration event, it extracts multi-domain features from the vibration signal: time-domain features (including short-time energy, peak amplitude, and waveform duration), frequency-domain features (including spectral centroid, spectral bandwidth, and spectral peak frequency), and time-frequency features (extracted through wavelet transform or short-time Fourier transform). These multi-domain features are normalized and concatenated into a feature vector, which is then input into a pre-trained lightweight CNN (Convolutional Neural Network) classifier to perform preliminary classification of the vibration event. The classification categories include: pedestrian walking, pedestrian climbing, vehicle driving, mechanical construction, wind disturbance, rain disturbance, small animal activity, and non-threatening interference. The classification result C output by this classifier is... DAS and confidence level P DAS .

[0052] The lightweight CNN classifier employs a 5-layer structure: Input layer (accepting 1xN vibration feature vectors) → Convolutional layer C1 (16 3x3 convolutional kernels, ReLU) → Pooling layer P1 (2x2 max pooling) → Convolutional layer C2 (32 3x3 convolutional kernels, ReLU) → Pooling layer P2 (2x2 max pooling) → Flattening layer → Fully connected layer FC1 (64 units, Dropout=0.5) → Output layer (Softmax, 8 nodes corresponding to 8 classification categories). This CNN model is a lightweight improvement on existing mature architectures, focusing on reducing the number of parameters while preserving the ability to extract time-frequency features.

[0053] Training dataset construction: Test optical fibers were deployed in three types of scenarios: typical construction areas, wilderness, and urban fringe. Vibration data of eight types of events were collected, including people walking / climbing, vehicles driving, excavators / electric drills operating, wind / rain disturbance, and small animal (cat / dog) activities. No less than 2,000 samples were collected for each type of event. Gaussian white noise with different signal-to-noise ratios (5dB~20dB), random pruning, and time stretching were added to the original samples to expand the sample size to more than 5,000 samples per type. Professional security personnel accurately labeled the event type of each sample.

[0054] Training process: Cross-entropy loss function and Adam optimizer (initial learning rate 0.001, decay coefficient 0.95 / epoch) were used; 80% of the samples were used as the training set, 10% as the validation set, and 10% as the test set; the training process lasted for 20 epochs, and was terminated early when the validation set loss no longer decreased for 3 consecutive epochs to prevent overfitting. The final model achieved an average classification accuracy of >92% on the test set.

[0055] Step 102: Acquire the acoustic signal collected by the target microphone within a preset time after the vibration moment, and the image sequence of the vibration location area collected by the target camera within a preset time after the vibration moment.

[0056] In some embodiments, the above-mentioned acquisition of the acoustic signal collected by the target microphone within a preset time after the vibration moment and the image sequence of the vibration location area collected by the target camera within a preset time after the vibration moment includes: when a vibration event is detected in the signal collected by the DAS unit, waking up the target microphone and the target camera according to the vibration location; and acquiring the acoustic signal collected by the target microphone within a preset time after the vibration moment and the image sequence of the vibration location area collected by the target camera within a preset time after the vibration moment.

[0057] Specifically, the location coordinates (x, t) of the vibration event are used for space-time window matching, waking up the K nearest silicon photonic acoustic microphones (K is usually 1 to 3) and triggering the M nearest thermal imaging cameras (M is usually 1 to 2) to perform directional detection on the target area. The introduction of the wake-up mechanism can significantly reduce the unnecessary computational load on the acoustic microphones and thermal imaging cameras, thereby optimizing system energy efficiency.

[0058] When the vibration event localization result wakes up multiple silicon photonic acoustic microphones, a "weighted feature fusion" strategy is adopted: 1. Single microphone feature extraction: Each woken microphone i independently collects acoustic signals and extracts features such as MFCC, short-time energy, and spectral centroid to form a feature vector F. audio_i 2. Quality Assessment: Calculate the signal-to-noise ratio (SNR) of the signal acquired by each microphone. i weight λ i= SNR i / ΣSNR j The higher the signal-to-noise ratio, the greater the contribution; 3. Feature-level fusion: The fused global speaker feature vector is: F audio_fused =Σλ i ·F audio_i The vector is then input into the voiceprint recognition model; 4. Decision output: The model outputs the final voiceprint classification result C. audio and confidence level P audio .

[0059] For cases where multiple thermal imaging cameras are simultaneously activated, the processing logic is as follows: 1. Independent detection: Each activated camera j independently acquires infrared images and outputs the target detection result (Class) through the YOLO model. j BBox j , P IR_j 2. Spatial coordinate alignment: Using camera calibration parameters and a known scene map, the bounding boxes of the targets detected by each camera are aligned. Box_j 3. Target Association and Fusion: If different cameras detect targets in the same world coordinate region (Intersection over Union (IoU) > 0.5), then it is considered that the same intruder has been detected; The maximum value is used for fusion of the overall confidence score: P IR = max(P IR_j The target location is calculated as the average of all detection results; if only a single camera detects the target, that result is used. If none of the cameras detect the target, then P... IR =0; 4. Output: Output the fused target category, location, and confidence level P. IR .

[0060] Step 103: Classify the acoustic signal to obtain the acoustic classification result, and classify the image sequence to obtain the image classification result.

[0061] In some embodiments, step 103 may include: extracting the Mel frequency cepstral coefficients of the acoustic signal to form a voiceprint feature vector, inputting the voiceprint feature vector into a voiceprint recognition model for classification and recognition, obtaining an acoustic classification result and a second confidence level of the acoustic classification result; acquiring an image sequence of the vibration location area collected by the target camera within a preset time after the vibration moment, preprocessing the image sequence and inputting it into an infrared target detection model, obtaining an image classification result and a third confidence level of the image classification result.

[0062] For details, see Figure 4After being awakened by DAS, the silicon photonics acoustic microphone begins to collect ambient audio within the Tacp time period (Tacp is usually 2 to 5 seconds). The acoustic feature extraction module processes the collected audio signal, including: (1) pre-emphasis processing to enhance high-frequency components; frame-by-frame windowing (frame length 25ms, frame shift 10ms, using Hamming window); endpoint detection to remove silence; (2) extraction of Mel frequency cepstral coefficients (MFCC, usually 13-dimensional); (3) calculation of auxiliary features such as short-time energy, spectral centroid, and spectral flux of the audio signal. The MFCC and auxiliary features form the acoustic feature vector F. audio .

[0063] The voiceprint feature vector F audio The input is fed into a pre-trained voiceprint recognition model for classification and recognition, and the output voiceprint classification result C is generated. audio and confidence level P audio The voiceprint recognition model adopts an improved SENet (Squeeze-and-Excitation Network) architecture, which adaptively enhances the weight of key frequency band features through a channel attention mechanism. This architecture can effectively distinguish between human footsteps, construction machinery sounds (such as the voiceprint features of excavators, electric drills, and impact drills), vehicle engine sounds, metal knocking / cutting sounds, and environmental background noise (such as wind and rain sounds).

[0064] The voiceprint recognition model can be comprised of an input layer, convolutional layers, a Squeeze module, an Excitation module, an improved multi-scale attention layer, a time-frequency attention layer, and a classification layer, all connected in sequence. The input layer receives a 128×T Log-Mel spectrogram, converting the one-dimensional audio signal into a two-dimensional time-frequency representation. The convolutional layer includes two 3×3 convolutional blocks, with 32 and 64 channels respectively, used for initial extraction of low-level acoustic features. The Squeeze module performs global average pooling, generating 1×1×C feature descriptors, compressing global spatial information into a single channel descriptor. The Excitation module includes two fully connected layers (dimensionality reduction ratio r=16) and a Sigmoid activation layer, used to learn the weights of each feature channel and model inter-channel dependencies. The improved multi-scale attention layer uses convolutional kernels of three different scales (3×3, 5×5, and 7×7) in parallel for feature extraction, fusing the results before inputting them into the SENet module. This enhances the model's ability to capture voiceprint patterns across different time spans, making it more robust to footsteps (short-term) and engine sounds (long-term). Time-frequency attention applies attention mechanisms in both the time and frequency dimensions to compute the time attention graph T. att and frequency attention map F attThis is used to enhance the representation of key time segments (e.g., the instant footsteps occur) and key frequency bands (e.g., characteristic frequencies of construction machinery). The classification layer includes global average pooling, Dropout(0.5), a fully connected layer, and Softmax, used to output the probability distribution of 8 types of voiceprint events.

[0065] The training process of the voiceprint recognition model includes: 1. Collecting 8 types of acoustic event samples (footsteps, excavator, electric drill, vehicle engine, metal knocking, wind, rain, and quiet background) in a construction area environment, with ≥3000 samples per type, and dividing the training / validation / test sets in an 8:1:1 ratio; 2. Preprocessing: All audio is downsampled to 16kHz, framed (25ms, frame shift 10ms, Hamming window), and a 128-dimensional Log-Mel spectrum is extracted as the model input; 3. Training configuration: The loss function is Focal Loss (focusing on hard-to-classify samples to solve the class imbalance problem), the transformer is AdamW (initial learning rate 0.001, weight decay 1e-4), the batch size is 64, the training period is 50 epochs, and the learning rate decays by 0.5 every 10 epochs; 4. Evaluation metrics: The weighted F1-score on the test set reaches 0.94, and the classification accuracy for the highly confusing "footsteps" and "small animal activities" is improved by about 8%.

[0066] See Figure 5 When triggered by a vibration event, the thermal imaging camera automatically adjusts its pan-tilt unit (PTZ) to align with the target area (PTZ response time ≤ 0.5 seconds), continuously acquiring a sequence of infrared images within the TIR time period. The infrared images are preprocessed (including non-uniformity correction, adaptive histogram equalization, and normalized scaling) and then input into a pre-trained infrared target detection model. This model, based on the YOLO (You Only Look Once) architecture, is specifically optimized for training on the thermal radiation characteristics of the human body in infrared thermal images, outputting detection results including target category, bounding box location, and confidence level P. IR .

[0067] The training process of the infrared target detection model is as follows:

[0068] 1. Dataset Construction: Under different lighting and weather conditions, such as daytime, nighttime, dusk, rainy days, and foggy days, video footage of the perimeter of the construction area was collected using deployed thermal imaging cameras, with a total duration of over 500 hours. Target categories included people, vehicles (excavators, trucks, cars), and animals (cats, dogs, birds). The original infrared images were randomly horizontally flipped, rotated (±15°), scaled (0.8~1.2 times), and their brightness / contrast was adjusted (simulating changes in ambient temperature). Gaussian noise was added, and the sample size was expanded from 20,000 images to 100,000 images. The LabelImg tool was used to manually annotate the bounding box and category label of each target.

[0069] 2. Training process: The loss function is CIoU Loss (Complete Intersection over Union Loss) for bounding box regression; the optimizer is SGD with momentum (momentum = 0.937), with an initial learning rate of 0.01 and cosine annealing scheduling; the batch size is 32, the training period is 300 epochs, and transfer learning is performed using weights pre-trained on the FLIR infrared dataset to accelerate convergence and improve performance.

[0070] 3. Model Optimization and Quantization: After training, the model is quantized with INT8 to enable real-time inference at a speed of ≥25 FPS on embedded devices (such as NVIDIA Jetson Xavier); a larger YOLOv5l model is used as the teacher network to guide the training of the YOLOv5s student network, further improving the detection rate of small targets (human body at 50 meters away).

[0071] 4. Deployment parameters: The final model achieved an mAP@0.91 on the test set, and its detection performance remained stable under conditions of complete darkness, strong backlight, and haze.

[0072] Step 104: Determine the threat value of the vibration event based on the vibration classification results, acoustic classification results, and image classification results, and conduct security early warning based on the threat value.

[0073] In some embodiments, determining the threat value of a vibration event based on vibration classification results, acoustic classification results, and image classification results may include: based on S=w1×P DAS +w2×P audio +w3×P IR +w4×I(F DA )+w5×I(F DI )+w6×I(F AI Calculate the threat value, P, of the vibration event. DAS P represents the confidence level of the vibration classification results. audio P represents the confidence level of the acoustic classification results. IR F represents the confidence level of the image classification result. DA For vibrational acoustic marking, F DI For marking vibration images, F AI For acoustic image labeling, vibration acoustic labeling represents the consistency verification results between vibration classification results and acoustic classification results; for vibration image labeling, it represents the consistency verification results between vibration classification results and image classification results; for acoustic image labeling, it represents the consistency verification results between acoustic classification results and image classification results. The indicator function is w1 to w6, which are the weights.

[0074] For example, if the vibration classification result is an intrusion result, the acoustic classification result is consistent with the vibration classification result, and P DAS With P audio If the product of F and F is greater than the bimodal verification threshold, then F DA The value is 1, otherwise F DA The value is 0; if the vibration classification result is an intrusion result, a human target exists in the vibration location region in the image classification result, and P IR If it is greater than the image verification threshold, then F DI The value is 1, otherwise F DI The value is 0; if the acoustic classification result is an intrusion result, and the image classification result shows a heat source target in the vibration location region, then F AI The value is 1, otherwise F AI The value is 0.

[0075] Specifically, after obtaining the vibration classification results, acoustic classification results, image classification results, and the confidence scores of each classification result, a three-level progressive fusion decision is made, including bimodal initial screening, weighted comprehensive scoring, and adaptive environmental compensation.

[0076] I. Dual-modal primary screening

[0077] Preliminary pairing verification was performed on any pairwise combination of the three sensors:

[0078] (1) DAS + microphone verification: If the vibration classification result C DAS The category is intrusion (including human walking / climbing / mechanical construction / vehicle movement), and the acoustic recognition result is C. audio The results are consistent with the vibration classification results (e.g., the vibration classification result is "walking" and the acoustic recognition result is "footsteps"), and the product of their confidence scores, P, is also consistent. DAS ×P audio >T va (T) va If the preset dual-modal verification threshold is used (typically 0.5), then the first-level "DAS-voiceprint" verification is marked as passed (F). DA =1.

[0079] (2) DAS + thermal imaging verification: If an intrusion event is detected by DAS signal, and a human target is present in the vibration location area in the image classification result (confidence level P) IR >T IR_Mmin T IR_Mmin (Typically 0.6), then "DAS-thermal imaging" verification is achieved by marking F. DI =1.

[0080] (3) Microphone + Thermal Imaging Verification: If the acoustic recognition result is an intrusive voiceprint (e.g., footsteps, construction machinery sounds), and the image classification result shows a heat source target (human heat source or vehicle heat source) in the vibration location area, then the "voiceprint-thermal imaging" verification is passed and marked F. AI =1.

[0081] It should be noted that the above-mentioned dual-modal verification marker F DA F DI and F AI It is not necessary for all three markers to be 1 simultaneously. In real-world scenarios, due to sensor characteristics, environmental interference, or incomplete intrusion behavior, the three markers may take any combination of 1 or 0 values. A subsequent weighted comprehensive score is used to quantify the intrusion threat level under different marker combinations: the score is highest when all three markers are true, decreases accordingly when some are true, and is lowest when none are true. This design ensures that the system can still make reasonable decisions even if any sensor fails or the sensor response is incomplete in a specific scenario. Dual-modal verification marker F DA F DI and F AI See Table 1 for the possible combinations.

[0082] Table 1. Dual-modal verification marker F DA F DI and F AI Combination

[0083]

[0084] II. Weighted Comprehensive Score

[0085] Construct the overall threat value S:

[0086] S=w1×P DAS +w2×P audio +w3×P IR +w4×I(F DA )+w5×I(F DI )+w6×I(F AI )

[0087] Where w1 to w6 are the weights of each dimension. This is an indicator function.

[0088] The weight coefficients w1 to w6 are determined as follows: During the initial stage of system deployment, an on-site calibration test is conducted for N days (N is typically 7-14 days). Sample data of real intrusion events and false alarm events are collected. Multi-objective optimization methods (such as grid search or Bayesian optimization) are used to search for the optimal weight combination, maximizing the system's detection rate while minimizing the false alarm rate. The initial default weights are: w1=0.20, w2=0.20, w3=0.20, w4=0.15, w5=0.15, w6=0.10.

[0089] III. Adaptive Environmental Compensation

[0090] Considering the impact of environmental factors (such as current wind speed, current rainfall intensity, ambient temperature, and lighting conditions) on sensor performance, adaptive compensation is performed on the threat value S based on environmental factors.

[0091] Specifically, if the current wind speed exceeds the wind speed threshold, the weight w1 is reduced; if the current rainfall intensity exceeds the rainfall intensity threshold, the weight w2 is increased; if the current illuminance is less than the illuminance threshold, the contribution weight w3 to the thermal imaging confidence is increased; if the ambient temperature is within the preset temperature range, the weight w3 is reduced, and the preset temperature range is determined based on human body temperature.

[0092] Specifically, (1) if the current wind speed exceeds V th (e.g., 8 m / s), then appropriately reduce the weight w1, because strong winds will cause the fence to vibrate, increasing the probability of DAS false alarms; (2) if the current rainfall intensity exceeds R th (e.g., 10mm / h), then appropriately increase the microphone detection threshold to reduce the noise impact caused by rain; (3) if the current environment is completely dark (ambient illumination < 0.1 lux), then appropriately decrease the visible light detection weight (if using a thermal imaging dual-light camera), and increase the contribution weight w3 of the thermal imaging confidence; (4) if the ambient temperature is between 32℃ and 38℃ (close to human body temperature), then appropriately decrease the contribution weight w3 of the thermal imaging confidence, because the temperature contrast between the human body and the background is reduced at this time.

[0093] The compensated threat value is: S'=α1×w1×P DAS +α2×w2×P audio +α3×w3×P IR +α4×w4×I(F DA )+α5×w5×I(F DI )+α6×w6×I(F AI ),in, Here, i represents the environmental adaptability coefficient for each component, ranging from 1 to 6. Let α1 be the function of the influence of environmental factors on the i-th component. For example, when the wind speed is too high, α1 decreases (reducing the contribution of DAS confidence); when the ambient temperature is close to the human body temperature, α3 decreases (reducing the contribution of thermal imaging confidence).

[0094] For example, δ1(env) = -η wind ×max(0,(V-V0) / (V th -V0), where V is the current wind speed, V0 is the lower limit of the wind speed at which compensation is initiated, and V th For wind speed threshold, 0.5 ≤ η wind ≤0.8 determines the maximum discount strength of the DAS confidence when the threshold wind speed is reached. DAS depends on fence vibration, and strong winds will cause a large number of non-intrusive vibrations, so their weight needs to be reduced. When the wind speed increases, δ1 becomes negative, making α1 < 1, thus reducing the contribution of DAS.

[0095] δ2(env)= -η rain ×max(0,(R-R0) / (R th -R0), where R is the current rainfall intensity, R0 is the lower limit of rainfall intensity for triggering compensation, and R th For rainfall intensity threshold, 0.6 ≤ η rain ≤0.8, η rain This represents the maximum attenuation of the voiceprint signal by rainfall. The sound of raindrops hitting the sound can drown out key intrusion voiceprint features, so it is necessary to increase the detection threshold or reduce its contribution to the score. When the rainfall intensity increases, δ2 becomes negative, making α2 < 1, thus reducing the voiceprint contribution.

[0096] δ3(env)= -η temp ×(1-|T env -T human | / T range ), T env For ambient temperature, T human For human body temperature, T range For the reference temperature difference range (i.e., the baseline value of the temperature difference between typical ambient temperature and human body temperature, ranging from 10 to 13, generally taken as 12 in comfortable environments), η temp A preset coefficient, η, is set for temperature sensitivity. temp The value range is 0 to 1, for example, η temp Take 0.5. η temp A value approaching 0 indicates that the effect of temperature on thermal imaging is almost negligible, η temp A value approaching 1 indicates that the thermal imaging contribution can be completely reduced to zero at extreme high temperatures (i.e., α3 is 0), η temp A value of 0.5 acknowledges the effects of high temperatures while retaining at least half of the contribution from thermal imaging. When the ambient temperature rises from 25°C to 37°C, |T env -T human| It decreases from 12 to 0. If T range =12, then α3=1- 0.5 * (0 / 12) =1 (the discount is greatest at 37℃). As the temperature difference between the target and the background decreases, the thermal imaging detection capability declines, and its confidence contribution needs to be reduced. When the ambient temperature is close to the human body temperature, δ3 is negative, so that α3< 1 (the smaller the temperature difference, the more negative it becomes).

[0097] δ4(env) = 0, δ5(env) = 0, δ6(env) = 0. The verification marker is a logical judgment result (0 or 1), representing the consensus formed between sensors. Environmental factors mainly affect the confidence of the sensors themselves, and should not easily negate the established cross-validation relationship. That is, the verification marker is a logical quantity and is not directly affected by the environment.

[0098] In some embodiments, the aforementioned security early warning based on threat values ​​may include: if the threat value is greater than a high-level alarm threshold, triggering an audible and visual alarm and pushing alarm information to the management personnel terminal. The alarm information includes the coordinates of the intrusion location, a thermal imaging screenshot of the scene, and the voiceprint classification result. Simultaneously, the video recording system is activated to save complete audio and video evidence of the scene. If the threat value is greater than a medium-level alarm threshold but less than or equal to the high-level alarm threshold, issuing an early warning to the management personnel terminal, and the management personnel can remotely retrieve real-time thermal imaging footage and real-time voiceprint audio for manual verification. The high-level alarm threshold is greater than the medium-level alarm threshold. If the threat value is less than or equal to the medium-level alarm threshold, it is determined to be an environmental disturbance event, and the event log is recorded without triggering an alarm. The high-level alarm threshold can be 0.75, and the medium-level alarm threshold can be 0.5.

[0099] Specifically, (1) if S' > T H (T) H If the threshold is set to high alert level, it is determined to be a highly definite human intrusion event, immediately triggering an audible and visual alarm and pushing alarm information (including intrusion location coordinates, on-site thermal imaging screenshots, and voiceprint classification results) to the management personnel terminal. At the same time, the video recording system is linked to save complete on-site audio and video evidence.

[0100] (2) If T M <S'≤T H (T) M If the threshold for medium-level alarm is reached, it is determined to be a suspected intrusion event, and the system issues an early warning. Administrators can remotely retrieve real-time thermal imaging footage and real-time audio from the voiceprint for manual verification.

[0101] (3) If S'≤T M If the event is identified as an environmental disturbance (such as wind, rain, or small animals), the system will record the event log but will not trigger an alarm. It will only generate an environmental event statistical report periodically.

[0102] This application supports continuous optimization: when the system issues an alarm and it is confirmed by management (whether it is a genuine intrusion or a false alarm), the confirmation result will be automatically added to the incremental learning sample library as a new labeled sample. During low-load periods (e.g., 3:00 AM to 5:00 AM), the system automatically uses the accumulated incremental samples to fine-tune the classification model online (using a small learning rate to avoid catastrophic forgetting), enabling the system to continuously improve its recognition ability for specific scenarios as the running time increases. This mechanism solves the problem of decreased recognition performance in traditional security systems after scene migration.

[0103] Event logging and playback mechanism: The system fully records all detected events, including: event timestamp, DAS location coordinates, DAS vibration signal waveform (5 seconds before and after the event), audio clips from the voiceprint microphone, thermal imaging image sequence, detection classification results and confidence levels of each sensor, fusion decision process (including comprehensive score 'X' and its various dimensions), final alarm level, and manual review results (if any). This mechanism not only supports event tracing and responsibility determination but also provides a complete data foundation for subsequent algorithm optimization and forensic analysis.

[0104] DAS utilizes its ultra-long-range, fully distributed, continuous sensing capability (up to 100 kilometers) as the spatial positioning backbone of the entire system. When a vibration event occurs at any location along the fiber optic cable, DAS can provide precise location information within one second and activate nearby silicon photoacoustic microphones and thermal imaging cameras for secondary orientation confirmation. DAS solves the problem of "where it happened".

[0105] Silicon photonics acoustic microphones utilize their ultra-high sensitivity to collect ambient sound within the DAS (Directional Analysis) positioning area. Through acoustic signature analysis, the acoustic characteristics of intrusion behavior can be identified, such as the rhythmic characteristics of footsteps, the engine noise characteristics of construction machinery, and the high-frequency characteristics of metal cutting. Acoustic microphones solve the "what" problem.

[0106] After receiving a location trigger from the DAS (Directional Analysis and Detection System), the thermal imaging camera automatically aims at the target area for imaging detection, using deep learning target detection algorithms to determine whether there are heat source targets such as people or vehicles. Thermal imaging solves the problem of "whether there is actually a person / vehicle".

[0107] The following example, using a typical construction area security scenario, illustrates the complete workflow of this application:

[0108] Scenario: The total perimeter of the fence in a construction area is about 2 kilometers. The system is deployed with one DAS demodulation host connected to a 2-kilometer sensing optical cable, 10 point-type silicon photonic acoustic microphones (spaced 200m apart), and 8 thermal imaging cameras (spaced 250m apart) to achieve full coverage.

[0109] Step 1: When an intruder climbs over the fence at night, the fence vibration is triggered first. The DAS unit detects an abnormal vibration signal at a certain location along the fence (such as location coordinate P10), records the location and waveform of the vibration event, and completes detection and localization within 1 second.

[0110] Step 2: The DAS vibration classifier initially identifies the event as "person climbing fence" (confidence level R). 𝐷𝐴𝑆 =0.85). The system automatically wakes up the two silicon photonic acoustic microphones (M5 and M6) and one thermal imaging camera (C4) closest to the event location P10.

[0111] Step 3: The activated silicon photonics microphones M5 and M6 collect multiple sound signals, including fence vibration, fabric rubbing, and footsteps. The voiceprint recognition model classifies these as "intrusion" voiceprints (confidence level P). audio =0.90), and perform Doppler frequency shift analysis on footsteps to help determine the target's direction and speed of movement.

[0112] Step 4: Simultaneously, the thermal imaging camera C4 is aimed at the target area and detects a moving target with human thermal radiation characteristics in the infrared image (confidence level P). IR =0.88).

[0113] Step 5: Fusion Decision Module Calculation: All three sensors detected the intrusion event, and the dual-modal verification flag F was set. DA =1、F DI =1、F AI =1 All positions are set. The comprehensive threat score is calculated as follows: S = 0.20 × 0.85 + 0.20 × 0.90 + 0.20 × 0.88 + 0.15 × 1 + 0.15 × 1 + 0.10 × 1 = 0.17 + 0.18 + 0.176 + 0.15 + 0.15 + 0.10 = 0.926. Current environmental parameters are (no wind at night, temperature approximately 15℃), and the environmental adaptability coefficients α for each element are... i Both are approximately equal to 1.0, S'=S=0.926.

[0114] Step 6: S' = 0.926 > T H =0.75, the system determines this to be a highly definite human intrusion event, immediately triggers an audible and visual alarm, and automatically pushes a notification containing the intrusion location P. 10 The alarm, which includes thermal imaging screenshots and voiceprint information, is sent to the duty officer's terminal, and the video recording system is linked to save the entire event's audio and video evidence.

[0115] The aforementioned security early warning method based on multi-sensor fusion first utilizes DAS to achieve long-distance, full-perimeter vibration event detection and precise location. Second, the DAS location results are used to activate microphones and cameras at the vibration event location for directional secondary detection. Finally, a three-level progressive fusion decision-making process—dual-modal initial screening, weighted comprehensive scoring, and adaptive environmental compensation—achieves mutual verification and complementarity of the three sensor detection results. This application overcomes the inherent shortcomings of single-sensor technology in terms of false alarm rate, false alarm rate, and environmental adaptability, achieving high-precision, low-false-alarm, all-weather intelligent protection for scenarios such as anti-theft and anti-illegal construction intrusion in construction areas.

[0116] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0117] Corresponding to the security early warning method based on multi-sensor fusion described in the above embodiments, Figure 6 The diagram shows a structural block diagram of a security early warning device based on multi-sensor fusion provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0118] See Figure 6 This application provides a security early warning device based on multi-sensor fusion, which may include a vibration classification module 201, an acquisition module 202, an acoustic image classification module 203, and an early warning module 204.

[0119] The vibration classification module 201 is used to classify vibration events when they are detected in the signals collected by the DAS unit to obtain vibration classification results. The vibration events include vibration location and vibration time. The boundary of the protected area is equipped with a DAS unit, a microphone array and a camera array.

[0120] The acquisition module 202 is used to acquire the acoustic signal collected by the target microphone within a preset time after the vibration moment, and the image sequence of the vibration location area collected by the target camera within a preset time after the vibration moment; wherein, the target microphone is at least one microphone in the microphone array that is closest to the vibration location, and the target camera is at least one camera in the camera array that is closest to the vibration location.

[0121] The acoustic image classification module 203 is used to classify acoustic signals to obtain acoustic classification results and to classify image sequences to obtain image classification results.

[0122] The early warning module 204 is used to determine the threat value of the vibration event based on the vibration classification results, acoustic classification results and image classification results, and to provide security early warning based on the threat value.

[0123] Optionally, the process of detecting vibration events from signals acquired by the DAS unit includes: denoising the original signals acquired by the DAS unit using a bandpass filter with a bandwidth of 1Hz to 500Hz; and processing the denoised signals... ,according to Calculate signal The short-term energy, according to Calculate signal The short-time zero-crossing rate; where W is the number of continuous signal sampling points included in the sliding time window; when the short-time energy Exceeding the energy threshold E th And short-term zero crossing rate When within the preset range, the determination signal is... A vibration event occurs at the corresponding location; where the energy threshold E th Determined based on the mean and standard deviation of the background noise energy of the original signal.

[0124] Optionally, the vibration event is classified to obtain a vibration classification result, including: extracting multi-domain features from the signal corresponding to the vibration event, the multi-domain features including time-domain features, frequency-domain features and time-frequency features; inputting the multi-domain features into a lightweight CNN classifier to classify the vibration event, and obtaining the vibration classification result of the vibration event and the first confidence level of the vibration classification result.

[0125] Optionally, the acquisition module 202 is specifically used to: when a vibration event is detected in the signal acquired by the DAS unit, wake up the target microphone and the target camera according to the vibration position; acquire the acoustic signal acquired by the target microphone within a preset time after the vibration moment, and the image sequence of the vibration position area acquired by the target camera within a preset time after the vibration moment;

[0126] The acoustic image classification module 203 is specifically used for: extracting the Mel frequency cepstral coefficients of the acoustic signal to form a voiceprint feature vector, inputting the voiceprint feature vector into the voiceprint recognition model for classification and recognition, and obtaining the acoustic classification result and the second confidence level of the acoustic classification result; acquiring the image sequence of the vibration location area collected by the target camera within a preset time after the vibration moment, preprocessing the image sequence and inputting it into the infrared target detection model to obtain the image classification result and the third confidence level of the image classification result.

[0127] Optionally, the early warning module 204 is specifically used for: based on S=w1×P DAS +w2×P audio +w3×P IR +w4×I(F DA )+w5×I(F DI )+w6×I(F AI Calculate the threat value, P, of the vibration event.DAS P represents the confidence level of the vibration classification results. audio P represents the confidence level of the acoustic classification results. IR F represents the confidence level of the image classification result. DA For vibrational acoustic marking, F DI For marking vibration images, F AI For acoustic image labeling, vibration acoustic labeling represents the consistency verification results between vibration classification results and acoustic classification results; for vibration image labeling, it represents the consistency verification results between vibration classification results and image classification results; for acoustic image labeling, it represents the consistency verification results between acoustic classification results and image classification results. The indicator function is w1 to w6, which are the weights.

[0128] Optionally, if the vibration classification result is an intrusion result, the acoustic classification result is consistent with the vibration classification result, and P DAS With P audio If the product of F and F is greater than the bimodal verification threshold, then F DA The value is 1, otherwise F DA The value is 0; if the vibration classification result is an intrusion result, a human target exists in the vibration location region in the image classification result, and P IR If it is greater than the image verification threshold, then F DI The value is 1, otherwise F DI The value is 0; if the acoustic classification result is an intrusion result, and the image classification result shows a heat source target in the vibration location region, then F AI The value is 1, otherwise F AI The value is 0.

[0129] Optionally, the aforementioned security early warning device based on multi-sensor fusion includes a compensation module, and the supplementary module is used to: according to S'=α1×w1×P DAS +α2×w2×P audio +α3×w3×P IR +α4×w4×I(F DA )+α5×w5×I(F DI )+α6×w6×I(F AI Adaptive compensation is applied to the threat value of vibration events. i can be 1 to 6. Let be the function of the influence of environmental factors on the i-th component;

[0130] Where, δ1(env) = -η wind ×max(0,(V-V0) / (V th -V0), where V is the current wind speed, V0 is the lower limit of the wind speed at which compensation is initiated, and V th η is the wind speed threshold. wind For the preset coefficient, 0.5≤ηwind ≤0.8; δ2(env)= -η rain ×max(0,(R-R0) / (R th -R0), where R is the current rainfall intensity, R0 is the lower limit of rainfall intensity for triggering compensation, and R th η is the rainfall intensity threshold. rain For the preset coefficient, 0.6≤η rain ≤0.8; δ3(env)= -η temp ×(1-|T env -T human | / T range ), T env For ambient temperature, T human For human body temperature, T range For reference temperature range, i.e., the baseline value of the temperature difference between typical ambient temperature and human body temperature, η is generally taken as 12 in a comfortable environment. temp A preset coefficient for temperature sensitivity is provided, 0 ≤ η temp ≤1.

[0131] Optionally, the above-mentioned security early warning based on threat values ​​includes: if the threat value is greater than the high-level alarm threshold, an audible and visual alarm is triggered, and alarm information is pushed to the management personnel terminal. The alarm information includes the coordinates of the intrusion location, a thermal imaging screenshot of the scene, and the voiceprint classification result. At the same time, the video recording system is linked to save complete audio and video evidence of the scene. If the threat value is greater than the intermediate alarm threshold but less than or equal to the high-level alarm threshold, an early warning prompt is issued to the management personnel terminal. The management personnel can remotely retrieve real-time thermal imaging footage and real-time voiceprint audio for manual verification. The high-level alarm threshold is greater than the intermediate alarm threshold. If the threat value is less than or equal to the intermediate alarm threshold, it is determined to be an environmental disturbance event, and the event log is recorded but no alarm is triggered.

[0132] Figure 7 This is a schematic diagram of a terminal device provided in an embodiment of the present invention. Figure 7 As shown, the terminal device 300 in this embodiment includes a processor 310 and a memory 320. The memory 320 stores a computer program that can run on the processor 310, such as a security early warning program based on multi-sensor fusion. When the processor 310 executes the computer program, it implements the steps in the above-described security early warning method embodiment based on multi-sensor fusion, for example... Figure 1 As shown in 101 to 104. Alternatively, when the processor 310 executes the computer program, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 6 The functions of the vibration classification module 201 to the early warning module 204 are shown.

[0133] The terminal device 300 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. The terminal device may include, but is not limited to, a processor 310 and a memory 320. Those skilled in the art will understand that... Figure 7 This is merely an example of terminal device 300 and does not constitute a limitation on terminal device 300. It may include more or fewer components than shown, or combine certain components, or different components. For example, the terminal device may also include input / output devices, network access devices, buses, etc.

[0134] The processor 310 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0135] The memory 320 can be an internal storage unit of the terminal device 300, such as a hard disk or memory of the terminal device 300. The memory 320 can also be an external storage device of the terminal device 300, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device 300. Furthermore, the memory 320 can include both internal and external storage units of the terminal device 300. The memory 320 is used to store the computer program and other programs and data required by the terminal device. The memory 320 can also be used to temporarily store data that has been output or will be output.

[0136] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A security early warning method based on multi-sensor fusion, characterized in that, include: When a vibration event is detected in the signal acquired by the DAS unit, the vibration event is classified to obtain a vibration classification result. The vibration event includes the vibration location and vibration time. A DAS unit, a microphone array and a camera array are set at the boundary of the protected area. Acquire acoustic signals collected by the target microphone within a preset time after the vibration moment, and image sequences of the vibration location area collected by the target camera within a preset time after the vibration moment; wherein, the target microphone is at least one microphone in the microphone array that is closest to the vibration location, and the target camera is at least one camera in the camera array that is closest to the vibration location; The acoustic signals are classified to obtain acoustic classification results, and the image sequences are classified to obtain image classification results; The threat value of the vibration event is determined based on the vibration classification results, the acoustic classification results, and the image classification results, and a security warning is issued based on the threat value.

2. The security early warning method based on multi-sensor fusion according to claim 1, characterized in that, The process of detecting vibration events from signals acquired by the DAS unit includes: A bandpass filter with a passband of 1Hz to 500Hz is used to denoise the raw signal acquired by the DAS unit; The signal after noise reduction ,according to Calculate signal The short-term energy, according to Calculate signal The short-time zero-crossing rate; where W is the number of continuous signal sampling points contained in the sliding time window; When short-time energy Exceeding the energy threshold E th And short-term zero crossing rate When within the preset range, the determination signal is... A vibration event occurs at the corresponding location; where the energy threshold E th Determined based on the mean and standard deviation of the background noise energy of the original signal.

3. The security early warning method based on multi-sensor fusion according to claim 1, characterized in that, The classification of the vibration events to obtain vibration classification results includes: Multi-domain features are extracted from the signal corresponding to the vibration event, and the multi-domain features include time-domain features, frequency-domain features, and time-frequency features. The multi-domain features are input into a lightweight CNN classifier to classify the vibration event, thereby obtaining the vibration classification result of the vibration event and the first confidence level of the vibration classification result.

4. The security early warning method based on multi-sensor fusion according to claim 3, characterized in that, The step of acquiring the acoustic signal collected by the target microphone within a preset time after the vibration moment, and the image sequence of the vibration location area collected by the target camera within a preset time after the vibration moment, includes: when a vibration event is detected in the signal collected by the DAS unit, waking up the target microphone and the target camera according to the vibration location; acquiring the acoustic signal collected by the target microphone within a preset time after the vibration moment, and the image sequence of the vibration location area collected by the target camera within a preset time after the vibration moment. The process of classifying the acoustic signal to obtain an acoustic classification result and classifying the image sequence to obtain an image classification result includes: extracting the Mel frequency cepstral coefficients of the acoustic signal to form a voiceprint feature vector, inputting the voiceprint feature vector into a voiceprint recognition model for classification and recognition, and obtaining an acoustic classification result and a second confidence level of the acoustic classification result; acquiring an image sequence of the vibration location area collected by the target camera within a preset time after the vibration moment, preprocessing the image sequence and inputting it into an infrared target detection model, and obtaining an image classification result and a third confidence level of the image classification result.

5. The security early warning method based on multi-sensor fusion according to any one of claims 1 to 4, characterized in that, Determining the threat value of the vibration event based on the vibration classification result, the acoustic classification result, and the image classification result includes: According to S=w1×P DAS +w2×P audio +w3×P IR +w4×I(F DA )+w5×I(F DI )+w6×I(F AI Calculate the threat value, P, of the vibration event. DAS P represents the confidence level of the vibration classification results. audio P represents the confidence level of the acoustic classification results. IR F represents the confidence level of the image classification result. DA For vibrational acoustic marking, F DI For marking vibration images, F AI For acoustic image labeling, vibration acoustic labeling represents the consistency verification results between vibration classification results and acoustic classification results; for vibration image labeling, it represents the consistency verification results between vibration classification results and image classification results; for acoustic image labeling, it represents the consistency verification results between acoustic classification results and image classification results. The indicator function is w1 to w6, which are the weights.

6. The security early warning method based on multi-sensor fusion according to claim 5, characterized in that, If the vibration classification result is an intrusion result, the acoustic classification result is consistent with the vibration classification result, and P DAS With P audio If the product of F and F is greater than the bimodal verification threshold, then F DA The value is 1, otherwise F DA The value is 0; If the vibration classification result is an intrusion-related result, then the image classification result shows a human target in the vibration location region, and P IR If it is greater than the image verification threshold, then F DI The value is 1, otherwise F DI The value is 0; If the acoustic classification result is an intrusion-related result, and the image classification result shows a heat source target in the vibration location region, then F AI The value is 1, otherwise F AI The value is 0.

7. The security early warning method based on multi-sensor fusion according to claim 5, characterized in that, The method includes: based on S'=α1×w1×P DAS +α2×w2×P audio +α3×w3×P IR +α4×w4×I(F DA )+α5×w5×I(F DI )+α6×w6×I(F AI Adaptive compensation is performed on the threat value of the vibration event. i can be 1 to 6. Let be the function of the influence of environmental factors on the i-th component; Where, δ1(env) = -η wind ×max(0,(V-V0) / (V th -V0), where V is the current wind speed, V0 is the lower limit of the wind speed at which compensation is initiated, and V th η is the wind speed threshold. wind For the preset coefficient, 0.5≤η wind ≤0.8; δ2(env)= -η rain ×max(0,(R-R0) / (R th -R0), where R is the current rainfall intensity, R0 is the lower limit of rainfall intensity for triggering compensation, and R th η is the rainfall intensity threshold. rain For the preset coefficient, 0.6≤η rain ≤0.8; δ3(env)= -η temp ×(1-|T env -T human | / T range ), T env For ambient temperature, T human For human body temperature, T range For reference temperature difference range, η temp A preset coefficient for temperature sensitivity is provided, 0 ≤ η temp ≤1.

8. The security early warning method based on multi-sensor fusion according to claim 1, characterized in that, The security early warning based on the threat value includes: If the threat value is greater than the advanced alarm threshold, an audible and visual alarm is triggered, and alarm information is pushed to the management personnel terminal. The alarm information includes the coordinates of the intrusion location, a thermal imaging screenshot of the scene, and the voiceprint classification result. At the same time, the video recording system is linked to save complete audio and video evidence of the scene. If the threat value is greater than the intermediate alarm threshold but less than or equal to the advanced alarm threshold, an early warning is issued to the management personnel terminal, and the management personnel can manually verify the data by remotely retrieving real-time thermal imaging footage and real-time audio from the voiceprint; wherein, the advanced alarm threshold is greater than the intermediate alarm threshold. If the threat value is less than or equal to the intermediate alarm threshold, it is determined to be an environmental disturbance event, the event log is recorded but no alarm is triggered.

9. A security early warning device based on multi-sensor fusion, characterized in that, include: The vibration classification module is used to classify vibration events when they are detected in the signals collected by the DAS unit to obtain vibration classification results. The vibration events include vibration location and vibration time. DAS unit, microphone array and camera array are set at the boundary of the protected area. The acquisition module is used to acquire acoustic signals collected by the target microphone within a preset time after the vibration moment, and image sequences of the vibration location area collected by the target camera within a preset time after the vibration moment; wherein, the target microphone is at least one microphone in the microphone array that is closest to the vibration location, and the target camera is at least one camera in the camera array that is closest to the vibration location; An acoustic image classification module is used to classify the acoustic signals to obtain acoustic classification results and to classify the image sequences to obtain image classification results; The early warning module is used to determine the threat value of the vibration event based on the vibration classification result, the acoustic classification result, and the image classification result, and to provide security early warning based on the threat value.

10. A security early warning method system based on multi-sensor fusion, characterized in that, The device includes a DAS unit, a microphone array, a camera array, and a terminal device. The DAS unit includes a sensing optical cable laid along the perimeter of the protection area and a DAS demodulation host. The microphone array and the camera array are deployed at preset intervals along the perimeter of the protection area. The terminal device includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the security early warning method based on multi-sensor fusion as described in any one of claims 1 to 8.

Images