Smart park safety protection system and method based on intelligent video analysis and multi-mode integration

Abstract

The invention discloses a smart park safety protection system and method based on smart video analysis and multi-mode integration, and relates to the technical field of smart park management. The system comprises a camera, an environment microphone and an access control terminal which are deployed in a park, an edge computing node is used for extracting structured video, audio and access control features, a central server realizes multi-modal space-time alignment through a self-calibration module, a multi-modal fusion module constructs a joint representation vector based on a low-rank tensor algorithm, and the joint representation vector is used for realizing multi-modal space-time alignment. And the security decision module identifies an abnormal event and triggers a response strategy in combination with a classification and anomaly detection model. According to the invention, multi-mode cooperative identification, high-robustness behavior analysis and intelligent response linkage are realized, and the real-time performance, accuracy and intelligent level of a park security system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of smart park management, and in particular to a smart park security protection system and method based on intelligent video analysis and multimodal integration. Background Art

[0002] With the rapid development of smart cities and the Internet of Things (IoT) technology, campus security systems are gradually evolving towards intelligence, data-based systems, and networking. Traditional campus security systems primarily rely on manual monitoring, single-source video surveillance, and access control card swiping. These systems struggle to achieve real-time perception and collaborative processing of multi-source information, early identification and high-confidence judgment of abnormal behavior, and robust anomaly protection and response linkage in complex scenarios.

[0003] Some existing systems have introduced deep learning recognition algorithms for video surveillance, which can detect and classify people or vehicles in some scenes. However, they still have the disadvantages of single modality, lack of multimodal collaboration mechanisms such as audio and access control, and inability to build complete event semantics. They also lack the ability to self-calibrate video offset problems caused by camera deployment deviations, blind spot occlusions, and lighting changes. There is also a lack of effective privacy protection mechanisms for identity data, making it difficult to perform large-scale behavior recognition under the premise of compliance.

[0004] Therefore, there is an urgent need to build a distributed, intelligent, reconfigurable multimodal security system that integrates front-end perception, mid-end processing, and back-end fusion to support efficient identification and emergency response of complex behavior patterns in smart parks. Summary of the Invention

[0005] To address the above issues, the present invention proposes a smart campus security protection system and method based on intelligent video analysis and multimodal integration, which integrates multimodal data such as video, audio, and access control to achieve spatiotemporal alignment and intelligent fusion of heterogeneous security data, and uses edge computing and new optical acceleration technology to reduce the latency and energy consumption of large-scale video analysis.

[0006] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0007] The smart campus security protection system based on intelligent video analysis and multimodal integration includes front-end sensing devices deployed throughout the campus and back-end processing units located at the network edge and center. The front-end sensing devices include several cameras, ambient microphones, and access control terminals, which are used to collect video, audio, and access information respectively; the back-end processing units include edge computing nodes and central servers; the edge computing nodes include:

[0008] The video analysis module receives video streams and performs target detection, tracking, and behavior recognition on image frames. It uses a multi-layer convolutional neural network and Transformer self-attention mechanism to extract the location, category, trajectory, and facial features of people or vehicles, and outputs structured video features.

[0009] The audio analysis module is used to receive audio signals for spectrum analysis and abnormal event identification, and extract audio features including abnormal event type, occurrence time and corresponding spectrum pattern;

[0010] The access control recognition module is used to identify the identity of the person based on the access information, record the access time and the corresponding access control terminal number, and determine whether it is an abnormal access based on the behavior recognition results. If so, it outputs the access control features including the person's identity, access time, access control terminal number and behavior label;

[0011] The central server includes:

[0012] A self-calibration module is used to calculate the temporal and spatial transformation parameters between cameras based on the trajectory of the same target in different video frames, and to perform spatiotemporal alignment of the video features, audio features, and access control features;

[0013] The multimodal fusion module is used to perform product weighted operations on the spatiotemporally aligned modal features based on a low-rank tensor fusion algorithm to construct a multimodal joint representation vector.

[0014] The security decision module is used to receive the joint representation vector, identify the behavior category through a neural network model with a residual structure, and generate an alarm signal and trigger a security response strategy if an abnormal event occurs.

[0015] Preferably, the video analysis module includes an optical neural network acceleration unit for accelerating the convolution operation and attention operation in the image frame, specifically including:

[0016] An input modulation module, which is used to convert the pixel tensor of the image frame into a multi-channel light intensity signal through an electro-optical modulator based on a modulation network composed of a Mach-Zehnder interferometer array, thereby realizing parallel encoding of high-dimensional input features in the optical domain;

[0017] A reconfigurable optical computing array includes multiple optical convolution kernel channels and optical self-attention channels. The optical convolution kernel channels are constructed based on the tensor decomposition method, using tensor column or CP decomposition to decompose the original high-dimensional weight tensor into low-rank sub-kernels, which are loaded into the optical path through a spatial light modulator and a phase controller network to complete the point multiplication operation of the low-rank convolution kernel. The optical self-attention channel adopts a multi-wavelength interference path structure and realizes parallel operation of different attention heads through wavelength division multiplexing. Each attention calculation channel is composed of an optical waveguide matrix multiplier and an adjustable microcavity array, which is used to perform matrix point multiplication and weighted sum operations in the query-key-value ternary relationship.

[0018] The interference and detection module is equipped with two free spectral range paths and a balanced detector to differentially obtain the optical calculation results of positive and negative weight contributions, and realize convolution and attention operation outputs that support negative weights;

[0019] The photoelectric conversion and sampling module is used to convert the light intensity signal output by the interference into a digital signal through a photoelectric detector and send it to the subsequent neural network classifier for target classification and behavior judgment;

[0020] An optical storage and multiplexing module is used to cache intermediate feature light signals output by the optical convolution kernel channel and the optical self-attention channel within the reconfigurable optical computing array, and transmit the intermediate feature light signals to downstream or parallel channels through an optical waveguide feedback structure to achieve cross-layer sharing and reuse of feature information;

[0021] The control and programming interface module is connected to the input modulation module, spatial light modulator, phase controller and wavelength division multiplexing path, and is used to load the neural network weight configuration file according to different tasks, dynamically control the optical channel structure, tensor core arrangement and the number of attention heads, and realize structural switching and hardware reconstruction for different video processing tasks.

[0022] Preferably, the audio analysis module includes:

[0023] A frequency domain conversion unit is used to perform a fast Fourier transform (FFT) on the collected original audio signal to obtain a complex spectrum vector X(f);

[0024] A dynamic filter weight generation unit is used to input the complex spectrum vector X(f) into a spectrum attention network containing trainable parameters, and dynamically generate frequency domain filter weights W(f) based on the amplitude characteristics and phase distribution of the spectrum in the local and global frequency ranges, wherein the frequency domain filter weights W(f) are a set of real-valued weight vectors corresponding to the frequency f, which are used to control the spectrum energy distribution of different frequency bands in the frequency dimension; for frequency points corresponding to the noise frequency band, the values ​​of the corresponding frequency components in the frequency domain filter weights W(f) are set to be close to zero to suppress the amplitude response of the frequency points in the filtering results; for frequency points carrying audio abnormal event characteristics, the values ​​of the corresponding frequency components in the frequency domain filter weights W(f) are set to be close to 1 to retain the energy of the frequency points, thereby achieving selective enhancement of the target audio event spectrum and suppression of the background noise spectrum;

[0025] A spectrum filtering unit is configured to perform a frequency-by-frequency element-wise multiplication operation on the complex spectrum vector X(f) and the frequency domain filtering weight W(f) to obtain a filtered spectrum vector Y(f), specifically:

[0026] Y(f)=W(f)⊙X(f);

[0027] Where "⊙" represents the multiplication of the corresponding frequency elements;

[0028] The inverse transform unit is used to perform inverse fast Fourier transform (IFFT) on the filtered spectrum vector Y(f) to reconstruct the filtered time domain audio signal.

[0029] Abnormal sound event recognition unit, used to filter the time domain audio signal Input the pre-trained audio event classification network to identify the abnormal event type e, the occurrence time τ, and the spectral pattern φ of the abnormal event, and use {e, τ, φ} as the structured output of the audio analysis module.

[0030] Preferably, the self-calibration module includes:

[0031] The trajectory extraction and filtering unit is used to extract the time-series trajectory of the same target in multiple camera images based on the video features output by the video analysis module, and perform credibility assessment on the time-series trajectory, eliminating unreliable trajectory points caused by occlusion, light interference or countermeasure disturbance to obtain the filtered target trajectory;

[0032] The camera parameter estimation unit is used to calculate the relative spatial pose transformation parameters between different cameras based on the filtered target trajectory through multi-view reprojection consistency constraints, including the rotation matrix R and displacement vector T, and to infer the time synchronization deviation Δt between the cameras by combining trajectory timestamp matching and trajectory frame difference minimization strategy;

[0033] The low-visibility compensation unit is used to compensate for calibration interruptions caused by missing target tracks by performing pose interpolation based on historical trajectory predictions, scene geometric edge features, or inertial measurement unit (IMU) data when any camera has a blind spot or insufficient lighting for target recognition for a short period of time.

[0034] The audio auxiliary calibration unit is used to calculate the three-dimensional spatial coordinates of the abnormal sound source using the time difference localization (TDOA) algorithm based on the microphone array when the audio analysis module detects an abnormal sound source with spatial positioning characteristics, and introduce the three-dimensional spatial coordinates as an auxiliary constraint into the camera pose solution process;

[0035] A calibration mapping construction unit is used to construct a unified spatiotemporal transformation relationship between cameras based on the rotation matrix R and displacement vector T, the time synchronization deviation Δt, and the three-dimensional spatial positioning coordinates of the abnormal sound source, thereby forming a global spatiotemporal mapping function for multimodal feature normalization processing;

[0036] A multimodal feature alignment unit is used to perform unified conversion of the time and space dimensions on the video features, audio features and access control features according to the global spatiotemporal mapping function.

[0037] Preferably, the scoring function used by the trajectory extraction and filtering unit to evaluate the credibility of the time series trajectory is expressed as:

[0038] s(p,t)=α·IoU(p,t)+β·conf(p,t)+γ·Δθ(p,t);

[0039] Where s(p,t) is the score value; p represents the target candidate box detected in the t-th frame; IoU(p,t) is the overlap rate with the prediction box of the previous frame; conf(p,t) is the detection confidence; Δθ(p,t) represents the degree of trajectory direction mutation; α, β, γ are weighting coefficients;

[0040] When the score value s(p,t) is lower than the set score threshold, the trajectory point is marked as untrustworthy and removed.

[0041] Preferably, the multimodal fusion module includes:

[0042] Modality mapping unit, used to align spatiotemporal video features Audio characteristics Access control features Mapped to the unified fusion subspace, we get: z v =W v ·f v ;z a =W a ·f a ;z c =Wc ·f c ;

[0043] in, Indicates that the video feature is a column vector consisting of mv real numbers, Indicates that the audio feature is a column vector consisting of ma real numbers, Indicates that the access control feature is a column vector consisting of mc real numbers; z v 、z a 、z c Map video features, audio features and access control features to a unified fusion subspace The intermediate representation vector after ; Respectively represent the linear mapping matrices of the corresponding modes; r is the fusion rank;

[0044] A fusion rank control unit is used to dynamically adjust the value of the fusion rank r based on the complexity index of the current input feature. The complexity index includes the number of targets, modal information entropy, and historical anomaly frequency, so as to control the representation capacity of the fusion structure and achieve a dynamic trade-off between fusion accuracy and computational load;

[0045] Product fusion unit, used to calculate the intermediate representation vector z of each modality v 、z a 、z c , and perform sum operations on the fusion dimensions to construct a joint tensor and the final joint representation vector The calculation method is:

[0046] H[i]=z v [i]×z a [i]×z c [i], where i∈{1,…,r};

[0047]

[0048] Where H[i] represents the trimodal product result on the i-th fusion dimension;

[0049] The modality completion unit is used to estimate the alternative representation vector of the missing modality based on the acquired modality features and the modality co-occurrence distribution in historical similar scenarios when any modality input is detected to be missing or abnormal, and replace the original input to ensure the continuity of the fusion calculation.

[0050] Preferably, the security decision module includes:

[0051] The abnormality discrimination and behavior recognition unit is used to receive the joint representation vector h, identify the behavior category through the residual structure neural network model, and evaluate the abnormality of the behavior category. The formula is:

[0052] Where μ is the center vector of the normal behavior sample, which is obtained in advance through normal data statistics; D(h) is the square of the Euclidean distance between the joint representation vector h and the center vector μ, which indicates the degree of deviation between the current sample and the normal sample; P anomaly is the probability that the current joint representation vector is judged as abnormal; σ(·) is the Sigmoid function; ρ is the abnormal score amplification factor; δ is the abnormal discrimination threshold;

[0053] Modal robust reconstruction and defense mechanism, used when the probability P is detected anomaly When the probability of abnormality is greater than the threshold, the modal defense mechanism is automatically enabled based on the modal robustness factor ε j Readjust the contribution of each modal fusion and call the modal replacement function Enable redundant modes for structural compensation;

[0054] The response strategy optimization unit is used to call the reinforcement learning model to calculate the response score Q(s,a) based on the current state s and the response action set A, and select the optimal action to generate the response strategy:

[0055]

[0056] Where, π * (s) represents the optimal response action that the system should take in state s; a is a candidate response action;

[0057] It means selecting the action that maximizes the Q value in the response action set A, that is, obtaining the optimal response strategy.

[0058] Preferably, the central server further includes a privacy protection and modal defense module, specifically including:

[0059] A feature desensitization processing unit is used to perform differential privacy perturbation or homomorphic encryption on video features or access control features containing personal identity information before uploading modal feature data to the edge computing node to prevent the original identity features from being restored;

[0060] A dynamic gating control unit is used to dynamically generate a gating coefficient based on the confidence level of each modal feature and the scenario risk before the joint representation vector is generated, and to preemptively suppress modal features with redundancy higher than a threshold or confidence lower than a threshold;

[0061] The adversarial interference detection and trust reconstruction unit is used to judge the change in modal credibility based on the contribution factor of multimodal input to output after the joint representation vector is generated, identify potential modal attack risks in combination with the inter-modal consistency indicator, and adjust the fusion contribution of the modality before triggering the response of the security decision module.

[0062] Preferably, the system also includes a federated collaborative learning unit, which is used to establish an encrypted communication link with multiple security protection systems deployed in different parks, periodically exchange anonymized model parameters or statistical summary information, and collaboratively optimize the neural network model parameters in the security decision-making module without transmitting the original modal data.

[0063] A smart park security protection method based on intelligent video analysis and multimodal integration is applicable to the smart park security protection system based on intelligent video analysis and multimodal integration as described above, and the method includes:

[0064] Front-end sensing devices deployed throughout the park collect video, audio, and traffic information;

[0065] At the edge computing node, feature extraction is performed on the video, audio, and traffic information, and structured video features, audio features, and access control features are output;

[0066] Performing differential privacy perturbation or homomorphic encryption on the data containing identity information in the video features or access control features to prevent unauthorized data leakage;

[0067] In the central server, based on multi-view target trajectory reconstruction, audio positioning and time synchronization technology, a unified spatiotemporal mapping relationship is established between each front-end sensing device, and video features, audio features and access control features are uniformly aligned in time and space.

[0068] The aligned features of each modality are mapped to a unified fusion subspace, and the element-by-element product and sum operation of the intermediate vectors of the three modalities are calculated using the product fusion method to construct the fused joint representation vector;

[0069] The joint representation vector is input into the residual neural network to identify the current behavior category and calculate the probability that the current joint representation vector is judged to be abnormal based on the distance from the center vector of the normal sample. When the probability exceeds the set abnormal probability threshold, the modal robust reconstruction and defense mechanism is triggered;

[0070] Based on the current state and the type of abnormal event, the reinforcement learning model is called to select the optimal response action, generate an alarm signal and trigger the security response strategy.

[0071] The beneficial effects of the present invention are: through the all-round collection of three types of information, namely video, audio and passage, the system's perception integrity and discrimination ability for complex behavioral scenarios are enhanced. Structured video, audio (such as and access control features) are extracted separately on the edge computing node to reduce the communication load and improve the real-time processing capability; the self-calibration module automatically calculates the time synchronization difference and spatial posture mapping between devices based on the projection relationship of the target trajectory in multiple cameras, solving the problems of occlusion, offset, and asynchrony in the deployment of multi-source cameras in the park, ensuring the consistency and comparability of multimodal data before fusion. The multimodal fusion module projects the three modal features into a unified subspace through product weighting and performs low-rank fusion to form a joint representation vector with compactness, discrimination and modal complementarity; based on the residual neural network model, it identifies whether there are abnormal events in the park, generates an alarm signal and activates the corresponding security response strategy at the same time, and realizes closed-loop event processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0073] in:

[0074] Figure 1 It is a system structure block diagram of the present invention;

[0075] Figure 2 This is a structural block diagram of an optical neural network acceleration unit in an embodiment of the present invention;

[0076] Figure 3 is a structural block diagram of an audio analysis module in an embodiment of the present invention;

[0077] Figure 4 4 is a flow chart of a method in an embodiment of the present invention. DETAILED DESCRIPTION

[0078] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of the present invention.

[0079] like Figure 1As shown, it is an embodiment of the present invention, which provides a smart campus security protection system based on intelligent video analysis and multimodal integration, and generally adopts a three-layer architecture of "front-end perception-edge processing-center decision-making".

[0080] The front-end perception layer includes front-end perception devices deployed throughout the campus, including high-definition cameras, ambient microphones, access control terminals (card swipe / face recognition terminals), infrared / smoke sensors, and more. Each device is connected to the edge computing node via wired / wireless networks. Cameras capture video streams, ambient microphones capture audio streams, access control terminals record entry and exit information and corresponding identity information, and other sensors provide signals indicating environmental anomalies (such as fire alarms).

[0081] Edge intelligent processing layer: It is composed of multiple edge computing nodes distributed in the campus LAN, which carry out preliminary data analysis tasks nearby to reduce the pressure and response delay of the center. Each node is integrated with an optical neural network acceleration unit to offload heavy deep learning reasoning (such as video analysis) to the optical neural network accelerator for execution, achieving efficient and low-energy computing. The edge computing node runs various modal analyses (video, audio, access control, etc.) and local data cache to perform real-time processing and temporary storage of front-end data. The preliminary analysis results (such as detected targets, identified identities, and sensor readings) are timestamped and sent to the center.

[0082] Central Decision-Making Layer: This integrated security platform, deployed within the monitoring and command center, boasts enhanced computing power and a comprehensive data view. The central server receives event streams and feature data uploaded by edge computing nodes, performing multimodal data fusion and advanced analytical decision-making. It is responsible for correlating information from various sources to form a holistic understanding of the campus's overall situation, issuing alarms, activating security devices (such as door locks and broadcasts), and notifying management personnel.

[0083] The entire architecture adopts a distributed collaborative working mode: front-end perception is ubiquitously deployed to collect campus dynamics in all directions; edge computing nodes intelligently filter and accelerate processing to protect privacy on-site; central servers integrate and synchronize, make comprehensive decisions, and achieve closed-loop security protection from terminal to center.

[0084] Specifically, the edge computing node includes a video analysis module, an audio analysis module and an access control recognition module.

[0085] The video analysis module receives camera video streams for intelligent analysis. Its functions include target detection and tracking (person and vehicle identification and trajectory), abnormal behavior recognition (fighting, climbing fences, etc.), and face recognition and comparison. It uses a multi-layer convolutional neural network and Transformer self-attention mechanism to extract features and output structured video features, such as "person Y was detected climbing a fence at time T and location X."

[0086] The audio analysis module performs spectral analysis and identifies abnormal events (such as glass breaking, screaming, and alarm sounds) on the audio signals collected by the ambient microphone. It uses a convolutional neural network + MFCC feature method to extract acoustic features, and introduces FFT filtering preprocessing to suppress irrelevant noise bands in the audio spectrum to improve recognition accuracy. It generates audio alarms in real time and outputs audio features including the abnormal event type, occurrence time, and corresponding spectral pattern, such as "glass breaking sound detected at time T, area Z."

[0087] The access control recognition module is responsible for processing the communication information of access control card swiping, fingerprint, face and other identity authentication devices. On the one hand, it records data such as personnel identity, pass time, access control terminal number, etc. On the other hand, it calls the face recognition results of the video analysis module for comparison and verification, and combines the behavior recognition results to determine whether it is abnormal passage. If there is an abnormal passage such as unauthorized personnel following in, the abnormal event can be marked immediately (such as "a person who did not swipe the card follows the cardholder to enter"), and the access control features such as personnel identity, pass time, access control terminal number, behavior label, etc. are output. The module also converts the access control log into semantic embedding, and reduces the interference of redundant ID signals through the gating mechanism when necessary for the central fusion reference.

[0088] like Figure 2 As shown, in one preferred embodiment, the video analysis module is configured with an optical neural network acceleration unit for accelerating the execution of convolution operations in image frames and fast reasoning of the self-attention mechanism in the edge computing node, specifically including:

[0089] The input modulation module receives image frame input and converts it into a light intensity signal that can be used for optical computing. Specifically, this module uses an electro-optical modulator (EOM) array combined with a Mach-Zehnder interferometer (MZI) network to perform high-dimensional encoding of the pixel tensor of each image frame. Each pixel value is modulated into a light signal of different intensity or phase through the modulator array, enabling parallel loading of input features in the optical domain.

[0090] A reconfigurable optical computing array, which contains multiple parallel optical convolution kernel channels and optical self-attention channels, corresponding to the convolutional layer and attention layer in the deep neural network, respectively;

[0091] The optical convolution kernel channel uses tensor decomposition methods, such as tensor train (TT) or CP decomposition, to decompose the original high-dimensional weight tensor into several low-rank sub-kernels. Each sub-kernel is loaded into the optical path in a specific way through a spatial light modulator (SLM) and a phase controller, and performs a dot multiplication operation with the input optical signal to achieve convolution calculation; the optical self-attention channel is used to accelerate the multi-head self-attention mechanism in the Transformer architecture. The channel adopts a multi-wavelength interference path structure and realizes parallel operation of different attention heads through wavelength division multiplexing (WDM). In each attention calculation, the query, key, and value tensors are input through an optical waveguide array, and the matrix dot multiplication and weighted sum operation are completed using an adjustable microcavity array to obtain the attention aggregation result;

[0092] The Interference and Detection module, which supports the expression of positive and negative weights in the network, is equipped with two Free Spectral Range (FSR) paths and balanced detectors for interferometric and differential detection of positive and negative light intensity signals. This structure ensures effective separation of positive and negative contributions to the tensor product results, expanding the expressive power and model adaptability of optical computing.

[0093] Photoelectric conversion and sampling module: After the optical calculation results are output by the interference and detection module, they are converted into electrical signals by high-responsivity photoelectric detectors and digitized by analog-to-digital conversion circuits. The resulting feature vectors are input into the subsequent neural network classifier in the edge computing device to identify structured information such as the category, location, trajectory, and behavior label of the target in the image frame;

[0094] To improve resource utilization within the optical computation path, this embodiment employs multiple optical cache structures and optical waveguide feedback paths to temporarily store intermediate feature results within the optical domain and enable cross-layer or cross-channel information reuse. This module can direct some intermediate layer feature signals to subsequent convolution kernels or attention head paths for computation, thereby reducing repetitive computation and improving energy efficiency.

[0095] The control and programming interface module provides operation configuration and task switching capabilities for the optical acceleration unit. It is connected to the input modulation module, spatial light modulator, phase controller and wavelength division multiplexing path. It is used to load the neural network weight configuration file according to different tasks, dynamically set the tensor decomposition parameters, adjust the number of attention heads, and configure the convolution kernel size and channel structure, thereby realizing the structural reconstruction and operation scheduling of the optical computing path.

[0096] The optical neural network acceleration unit achieves high-throughput, low-latency, and low-energy processing capabilities for convolution and attention calculations in image frames through parallel optical computing, tensor low-rank decomposition loading, optical domain feature storage, and programmable path reconstruction. Compared with traditional GPU inference solutions, it has higher energy efficiency and edge deployment feasibility.

[0097] Through the above structural design, the optical neural network acceleration unit of this embodiment can complete the feature extraction and forward reasoning of image frames at the edge node with a delay of milliseconds, significantly reducing the power consumption and heat dissipation burden required by traditional GPUs. In typical scenarios, the inference energy efficiency ratio can reach more than 50 times that of traditional electronic solutions, which is particularly suitable for multi-point edge analysis scenarios that require the deployment of large-scale high-definition video streams in campus security. In addition, the acceleration unit has a modular design feature that can flexibly expand optical computing arrays of different sizes, and can quickly reconstruct weights and network structures through the control module to adapt to the diverse needs of different areas and different functional nodes in the campus for video recognition accuracy and real-time performance.

[0098] like Figure 3 As shown, in one preferred embodiment, ambient microphones deployed at park entrances, corridors, parking lots, and other areas collect analog audio signals s(t), which are converted into discrete audio sequences after A / D conversion. The sampling rate is set to 16kHz, the frame length is set to 512 sampling points (approximately 32 milliseconds), the frame shift is 256 points, and a Hamming window is used for windowing to form frame-level audio data. The audio analysis module is based on a dynamic spectrum filtering structure. By analyzing the collected audio signals in real time, it realizes the recognition and structured extraction of abnormal events (such as glass breakage, screaming, alarms, etc.), which specifically includes:

[0099] The frequency domain conversion unit performs fast Fourier transform FFT on each frame of audio signal s(t) to obtain a complex spectrum vector X(f). represents the complex number field, that is, the set of all complex numbers, N = 257 (due to the use of real number symmetry, the positive frequency part is retained), and X(f) contains the amplitude and phase information of each frequency point.

[0100] The dynamic filter weight generation unit inputs the complex spectrum vector X(f) into a spectrum attention network containing trainable parameters. The network consists of two one-dimensional convolutional layers (kernel width 5, stride 1), one normalization layer, and a Sigmoid activation function. Based on the amplitude characteristics and phase distribution of the spectrum in the local and global frequency ranges, the frequency domain filter weight W(f) is dynamically generated. The frequency domain filter weight W(f) is a set of real-valued weight vectors corresponding to the frequency f, which is used to control the spectrum energy distribution of different frequency bands in the frequency dimension. For the frequency points corresponding to the noise band, the value of the corresponding frequency component in the frequency domain filter weight W(f) is set to zero to suppress the amplitude response of the frequency point in the filtering result. For the frequency points carrying the characteristics of audio abnormal events, the value of the corresponding frequency component in the frequency domain filter weight W(f) is set to 1 to retain the energy of the frequency point, thereby achieving selective enhancement of the target audio event spectrum and suppression of the background noise spectrum.

[0101] During the training phase, the spectral attention network aims to accurately identify abnormal audio events and automatically learns the correlation between frequency points and event labels through backpropagation: during network training, the network tends to generate weights close to 1 for abnormal event frequencies (such as glass breakage concentrated in 2-4kHz), and generates weights close to 0 for non-discriminative frequency bands such as low-frequency motor noise (such as 50-300Hz), thereby achieving selective retention and suppression of spectral energy.

[0102] The spectrum filtering unit performs a frequency-by-frequency element-wise multiplication operation on the complex spectrum vector X(f) and the frequency domain filtering weight W(f) to obtain the filtered spectrum vector Y(f), specifically:

[0103] Y(f)=W(f)⊙X(f);

[0104] Here, “⊙” represents the multiplication of the corresponding frequency elements; the frequency bands close to 0 in W(f) have no contribution to Y(f), and the energy of the frequency bands close to 1 is retained.

[0105] The inverse transform unit performs inverse fast Fourier transform IFFT on the filtered spectrum vector Y(f), retains the real part, and reconstructs the filtered time domain audio signal. This signal is more semantically significant than the original s(t) and the background noise is effectively suppressed.

[0106] The abnormal sound event recognition unit converts the filtered time domain audio signal Input the pre-trained audio event classification network (a deep neural network consisting of 3 convolutional layers and 1 GRU layer can be used), output the abnormal event type e, confidence branch, occurrence time τ, etc. corresponding to the current frame, and output the spectrum pattern φ of the frame (such as the filtered feature map) to form the structured audio feature {e, τ, φ} for subsequent alignment and fusion with video / access control modal features.

[0107] The spectral attention network and audio event classification network are trained end-to-end, using recognition accuracy as the optimization objective for a joint loss function. After training, they are deployed on the campus' edge computing nodes (such as the Jetson Xavier NX). Field testing has shown that recognition accuracy in complex environments has increased by 15%, the false alarm rate has decreased to 8.1%, and recognition latency has been kept within 150ms, meeting the campus's real-time response requirements for abnormal events.

[0108] This embodiment shows that the audio analysis structure based on dynamic spectrum filtering has better frequency band separation and background noise suppression capabilities than the traditional MFCC+CNN method, and has efficient edge inference performance, which can effectively improve the perception and response capabilities of smart parks to sudden acoustic events.

[0109] The central server includes a self-calibration module, a multimodal fusion module and a security decision module.

[0110] The self-calibration module achieves dynamic spatial calibration and temporal synchronization between multiple cameras based on the trajectory of the same target in different video frames, ensuring that the video, audio, and access control feature data from multiple front-end sensing devices can be fused and analyzed in a unified time axis and spatial coordinate system, avoiding inconsistencies in multimodal data caused by perspective deviation, device drift, or time differences, which may affect subsequent abnormal behavior recognition and security incident detection.

[0111] In one embodiment, the self-calibration module includes:

[0112] The trajectory extraction and filtering unit is used to extract the position data of the same target (such as a person or vehicle) in consecutive frames of different camera images based on the video features output by the video analysis module. It uses a cross-camera identity association algorithm (such as ReID + time series matching) to integrate the trajectories of the same target under different viewing angles, and performs credibility assessment on the integrated time series trajectory. It removes unreliable trajectory points caused by occlusion, light interference, or adversarial disturbances to obtain the filtered target trajectory.

[0113] The camera parameter estimation unit is used to calculate the relative spatial pose transformation parameters between different cameras, including the rotation matrix R and the displacement vector T, based on the filtered target trajectory and through multi-view reprojection consistency constraints. In addition, to synchronize multiple cameras due to time offsets caused by different frame rates, network latency, and trigger mechanisms, the time synchronization deviation Δt between the cameras is further calculated by matching trajectory timestamps, minimizing the inter-frame error of reprojection frames, and aligning acceleration trajectories, achieving frame-level time domain alignment.

[0114] The low visibility compensation unit is used to interpolate or extrapolate the camera pose in the missing segment based on historical trajectory predictions, scene geometric edge features (obtained through SLAM or NeRF scene set models), or relative motion information provided by the inertial measurement unit (IMU) data when any camera has a blind spot (such as temporary structures or vehicles blocking the view) or insufficient lighting (such as no fill light at night) and cannot identify the target. This maintains the continuity and stability of the spatiotemporal alignment chain and compensates for the calibration discontinuity caused by the missing target trajectory.

[0115] The audio-assisted calibration unit addresses situations in which abnormal events may be invisible but audio-significant (such as nighttime screams, crashing sounds, and explosions). When the audio analysis module detects an abnormal sound source with spatial positioning characteristics (such as a short pulse), it uses the Time Difference of Arrival (TDOA) algorithm based on the microphone array to calculate the three-dimensional spatial positioning coordinates of the abnormal sound source. These three-dimensional spatial positioning coordinates are introduced as auxiliary constraints into the camera pose solution process, improving the stability and accuracy of the camera parameter solution. In particular, it can maintain the system's global perception of the scene structure when visual targets are missing.

[0116] A calibration mapping construction unit is used to construct a unified spatiotemporal transformation relationship between cameras based on the rotation matrix R and displacement vector T, the time synchronization deviation Δt, and the three-dimensional spatial positioning coordinates of the abnormal sound source, forming a global spatiotemporal mapping function for multimodal feature normalization processing, whose input is the timestamp and local coordinates (x, t) of the original modal feature, and the output is the position in the global reference coordinate system and a unified timestamp;

[0117] The multimodal feature alignment unit is used to perform unified transformation of the time and space dimensions of video features, audio features, and access control features according to the global spatiotemporal mapping function, and output alignment features with global three-dimensional coordinates and standardized timestamps. The alignment results can be directly input into the multimodal fusion module for subsequent anomaly detection and decision analysis, ensuring the alignment accuracy and scene consistency between data.

[0118] In one embodiment of the present invention, to enhance the robustness of the self-calibration module in complex campus scenarios, the trajectory extraction and filtering unit incorporates a trajectory credibility scoring mechanism. This mechanism is used to screen and cleanse candidate trajectory points detected in multi-camera images under conditions such as target occlusion, sudden changes in illumination, or interference, preventing unreliable data from being input into the pose estimation process.

[0119] Specifically, the scoring function for credibility evaluation of the time series trajectory by the trajectory extraction and filtering unit is expressed as:

[0120] s(p,t)=α·IoU(p,t)+β·conf(p,t)+γ·Δθ(p,t);

[0121] Where t represents the time index of the current frame (in frame numbers), which is used to identify the temporal position of p in the video; p represents the target candidate box detected in the t-th frame; s(p,t) is the score value; IoU(p,t) is the overlap rate with the prediction box of the previous frame, that is, the intersection-over-union ratio between the candidate box p and its corresponding predicted trajectory box in the t-1th frame, which is used to measure the consistency of the trajectory position; conf(p,t) is the detection confidence of the candidate box p in the t-th frame, which is output by the target detection neural network and generally ranges from 0 to 1; Δθ(p,t) represents the degree of trajectory direction mutation, that is, the rate of change of the motion direction of the candidate point p between the t-th frame and the previous frame, which is used to identify trajectory mutations, such as obtained by calculating the angle or velocity change rate of the motion vector between the two frames; α, β, and γ are weighting coefficients, which can be set according to actual scene experience or automatically optimized through the training process;

[0122] When the score value s(p,t) is lower than the set score threshold, the trajectory point is marked as unreliable and removed, and does not participate in the subsequent camera pose estimation process.

[0123] Through this scoring mechanism, false trajectory data caused by false detection, occlusion, jitter or physical attacks (such as covering key areas of the image) can be effectively identified and eliminated, thereby improving the stability and accuracy of the self-calibration module in complex environments.

[0124] The multimodal fusion module is used to perform product weighted operations on the spatiotemporally aligned modal features based on the low-rank tensor fusion algorithm to construct a multimodal joint representation vector.

[0125] In one optional embodiment, the multimodal fusion module includes:

[0126] Modality mapping unit, used to align spatiotemporal video features Audio characteristics Access control features Mapped to the unified fusion subspace, we get: z v =W v ·f v ;z a =W a ·f a ;z c =W c ·f c ;

[0127] in, Indicates that the video feature is a column vector consisting of mv real numbers, Indicates that the audio feature is a column vector consisting of ma real numbers, Indicates that the access control feature is a column vector consisting of mc real numbers; z v 、z a 、z c Map video features, audio features and access control features to a unified fusion subspace The intermediate representation vector after ; They represent the linear mapping matrices of the corresponding modes respectively; r is the fusion rank.

[0128] A fusion rank control unit is used to dynamically adjust the value of the fusion rank r based on the complexity index of the current input feature. The complexity index includes the number of targets, modal information entropy, and historical anomaly frequency, so as to control the representation capacity of the fusion structure and achieve a dynamic trade-off between fusion accuracy and computational load;

[0129] In the scenario of this embodiment, mv=128, ma=64, and mc=32. If the number of detected targets in the current period is large (>4 people) and the mutual information between the modalities is significantly improved (>0.85), r=64 is automatically adjusted. On the contrary, in the high-confidence scenario with no traffic at night, r is reduced to 32 to reduce the computational overhead.

[0130] Product fusion unit, used to calculate the intermediate representation vector z of each modality v 、z a 、z c , and perform sum operations on the fusion dimensions to construct a joint tensor and the final joint representation vector The calculation method is:

[0131] H[i]=z v [i]×z a [i]×z c [i], where i∈{1,…,r};

[0132]

[0133] Where H[i] represents the trimodal product result on the i-th fusion dimension.

[0134] The modality completion unit is used to estimate the alternative representation vector of the missing modality based on the acquired modality features and the modality co-occurrence distribution in similar historical scenarios when any modality input is detected to be missing or abnormal, and replace the original input to ensure the continuity of the fusion calculation. For example, in an actual operation, the access control device is disconnected from the network and cannot upload the card swiping data, resulting in the missing access control feature fc. In this case, the modality completion unit constructs the input z based on the historical audio and video co-occurrence pattern. v 、z a The fusion code is input into the adversarial generation network G to generate It is used to replace the missing access control feature input and continue to complete the joint representation vector calculation to ensure the consistency of the fusion results.

[0135] Through the above implementation, the multimodal fusion module of this embodiment realizes the efficient fusion, dynamic rank adjustment and modal robustness compensation of heterogeneous data sources in complex scenarios, and can link the downstream response system to realize intelligent security identification and real-time response of smart parks.

[0136] The security decision module is used to receive the joint representation vector and identify the behavior category through the neural network model of the residual structure. If an abnormal event occurs, an alarm signal is generated and a security response strategy is triggered.

[0137] In one specific embodiment, the security decision module includes:

[0138] The abnormality discrimination and behavior recognition unit is used to receive the joint representation vector h, which represents the scene state after the fusion of video, audio, and access control modes at the current time t and spatial coordinates (x, y). In this embodiment, the joint representation vector h is classified and inferred by the multi-layer perceptron (MLP) model with a residual structure, and the current behavior category label is output, such as "normal passage", "illegal crossing", "nighttime intrusion", etc. At the same time, the center vector μ of the preset normal behavior sample is used to calculate the Euclidean distance square D(h) between the joint representation vector h and the center vector μ as the deviation:

[0139] Then the deviation is converted into abnormal probability, the formula is: P anomaly =σ(ρ·(D(h)-δ));

[0140] Where, P anomaly is the probability that the current joint representation vector is judged as abnormal. The closer the value is to 1, the more likely the current behavior is abnormal. σ(·) is the Sigmoid function. ρ is the abnormal score amplification factor. δ is the adjustable abnormality discrimination threshold.

[0141] Modal robust reconstruction and defense mechanism, used when the probability P is detected anomaly When the probability of abnormality is greater than the set threshold, the system automatically activates the modal defense mechanism. The modal robustness factor ε is pre-set for each mode j. j , for example, in nighttime conditions, the ε of the video modality j Can be set to a lower value, audio mode and access control mode ε j The value is relatively high; the modal robustness factor ε j Used to regulate the modal fusion contribution, reduce the fusion weight of unreliable modalities, and call the modal replacement function (When modality j is identified as abnormal, its redundant candidate modality j' is called to replace the original modality input.) Redundant modalities are enabled for structural compensation. For example, a video modality affected by strong light interference is switched to a millimeter-wave radar modality, or a noisy audio channel is switched to a vibration sensing channel, to maintain the stability and reliability of the system's fusion output.

[0142] The response strategy optimization unit is used to calculate the response score Q(s, a) based on the current state s (including information such as anomaly type, confidence, time location, etc.) and the preset response action set A (such as turning on the alarm, remotely locking the door, notifying security, etc.), call the reinforcement learning model, and select the optimal action to generate a response strategy:

[0143]

[0144] Where, π * (s) represents the optimal response action that the system should take in state s; a is a candidate response action;

[0145] It means selecting the action that maximizes the Q value in the response action set A, that is, obtaining the optimal response strategy.

[0146] In this embodiment, if forced entry by no one using a card swipe is detected at night (an abnormal event), the optimal response action selected by the system in state s is: "lock the access control + push security notification + turn on the alarm light", and the response log is recorded for subsequent model optimization and tracking audit.

[0147] Through the above implementation, the security decision module of the present invention not only has the ability to identify real-time anomalies of fusion vectors, but also has the intelligent security protection capability of dynamically selecting multimodal strategies and self-learning response solutions according to the scenario.

[0148] In one optional embodiment, the central server further includes a privacy protection and modal defense module, specifically including:

[0149] A feature desensitization processing unit is used to perform differential privacy perturbation or homomorphic encryption on video features or access control features containing personal identity information before uploading modal feature data to the edge computing node to prevent the original identity features from being restored;

[0150] A dynamic gating control unit is used to dynamically generate a gating coefficient based on the confidence level of each modal feature and the scenario risk before the joint representation vector is generated, and to preemptively suppress modal features with redundancy higher than a threshold or confidence lower than a threshold;

[0151] The adversarial interference detection and trust reconstruction unit is used to judge the change in modal credibility based on the contribution factor of multimodal input to output after the joint representation vector is generated, identify potential modal attack risks in combination with the inter-modal consistency indicator, and adjust the fusion contribution of the modality before triggering the response of the security decision module.

[0152] Preferably, the system of the present invention also includes a federated collaborative learning unit, which is used to establish an encrypted communication link with multiple security protection systems deployed in different parks, periodically exchange anonymized model parameters or statistical summary information, and collaboratively optimize the neural network model parameters in the security decision-making module without transmitting the original modal data.

[0153] like Figure 4 FIG. 1 is another embodiment of the present invention, which provides a smart park security protection method based on intelligent video analysis and multimodal integration, applicable to the smart park security protection system based on intelligent video analysis and multimodal integration as described above, and includes the following contents:

[0154] Front-end sensing devices deployed throughout the park collect video, audio, and traffic information;

[0155] At the edge computing node, feature extraction is performed on video, audio, and access information, and structured video features, audio features, and access control features are output;

[0156] Perform differential privacy perturbation or homomorphic encryption on data containing identity information in video features or access control features to prevent unauthorized data leakage;

[0157] In the central server, based on multi-view target trajectory reconstruction, audio positioning and time synchronization technology, a unified spatiotemporal mapping relationship is established between each front-end sensing device, and video features, audio features and access control features are uniformly aligned in time and space.

[0158] The aligned features of each modality are mapped to a unified fusion subspace, and the element-by-element product and sum operation of the intermediate vectors of the three modalities are calculated using the product fusion method to construct the fused joint representation vector;

[0159] The joint representation vector is input into the residual neural network to identify the current behavior category and calculate the probability that the current joint representation vector is judged to be abnormal based on the distance from the center vector of the normal sample. When the probability exceeds the set abnormal probability threshold, the modal robust reconstruction and defense mechanism is triggered;

[0160] Based on the current state and the type of abnormal event, the reinforcement learning model is called to select the optimal response action, generate an alarm signal and trigger a security response.

[0161] In summary, the present invention achieves a comprehensive improvement in the timeliness, accuracy, energy efficiency and safety of the smart campus security system through a series of innovative designs such as multimodal deep fusion, optical accelerated computing, adaptive calibration and security protection, and has significant practical application value.

[0162] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included within the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A smart campus security system based on intelligent video analysis and multimodal integration, including front-end sensing devices deployed throughout the campus and back-end processing units located at the network edge and center, including: The front-end sensing device includes several cameras, environmental microphones and access control terminals, which are respectively used to collect video, audio and traffic information; the back-end processing unit includes edge computing nodes and a central server; it is characterized in that the edge computing node includes: The video analysis module receives video streams and performs target detection, tracking, and behavior recognition on image frames. It uses a multi-layer convolutional neural network and Transformer self-attention mechanism to extract the location, category, trajectory, and facial features of people or vehicles, and outputs structured video features. The audio analysis module is used to receive audio signals for spectrum analysis and abnormal event identification, and extract audio features including abnormal event type, occurrence time and corresponding spectrum pattern; The access control recognition module is used to identify the identity of the person based on the access information, record the access time and the corresponding access control terminal number, and determine whether it is an abnormal access based on the behavior recognition results. If so, it outputs the access control features including the person's identity, access time, access control terminal number and behavior label; The central server includes: A self-calibration module is used to calculate the temporal and spatial transformation parameters between cameras based on the trajectory of the same target in different video frames, and to perform spatiotemporal alignment of the video features, audio features, and access control features; The multimodal fusion module is used to perform product weighted operations on the spatiotemporally aligned modal features based on a low-rank tensor fusion algorithm to construct a multimodal joint representation vector. The security decision module is used to receive the joint representation vector, identify the behavior category through a neural network model with a residual structure, and generate an alarm signal and trigger a security response strategy if an abnormal event occurs.

2. The smart park security protection system based on intelligent video analysis and multimodal integration according to claim 1 is characterized in that: The video analysis module includes an optical neural network acceleration unit for accelerating the convolution operation and attention operation in the image frame, specifically including: An input modulation module, which is used to convert the pixel tensor of the image frame into a multi-channel light intensity signal through an electro-optical modulator based on a modulation network composed of a Mach-Zehnder interferometer array, thereby realizing parallel encoding of high-dimensional input features in the optical domain; A reconfigurable optical computing array includes multiple optical convolution kernel channels and optical self-attention channels. The optical convolution kernel channels are constructed based on the tensor decomposition method, using tensor column or CP decomposition to decompose the original high-dimensional weight tensor into low-rank sub-kernels, which are loaded into the optical path through a spatial light modulator and a phase controller network to complete the point multiplication operation of the low-rank convolution kernel. The optical self-attention channel adopts a multi-wavelength interference path structure and realizes parallel operation of different attention heads through wavelength division multiplexing. Each attention calculation channel is composed of an optical waveguide matrix multiplier and an adjustable microcavity array, which is used to perform matrix point multiplication and weighted sum operations in the query-key-value ternary relationship. The interference and detection module is equipped with two free spectral range paths and a balanced detector to differentially obtain the optical calculation results of positive and negative weight contributions, and realize convolution and attention operation outputs that support negative weights; The photoelectric conversion and sampling module is used to convert the light intensity signal output by the interference into a digital signal through a photoelectric detector and send it to the subsequent neural network classifier for target classification and behavior judgment; An optical storage and multiplexing module is used to cache intermediate feature light signals output by the optical convolution kernel channel and the optical self-attention channel within the reconfigurable optical computing array, and transmit the intermediate feature light signals to downstream or parallel channels through an optical waveguide feedback structure to achieve cross-layer sharing and reuse of feature information; The control and programming interface module is connected to the input modulation module, spatial light modulator, phase controller and wavelength division multiplexing path, and is used to load the neural network weight configuration file according to different tasks, dynamically control the optical channel structure, tensor core arrangement and the number of attention heads, and realize structural switching and hardware reconstruction for different video processing tasks.

3. The smart park security protection system based on intelligent video analysis and multimodal integration according to claim 1 is characterized in that: The audio analysis module includes: A frequency domain conversion unit is used to perform a fast Fourier transform (FFT) on the collected original audio signal to obtain a complex spectrum vector X(f); A dynamic filter weight generation unit is used to input the complex spectrum vector X(f) into a spectrum attention network containing trainable parameters, and dynamically generate frequency domain filter weights W(f) based on the amplitude characteristics and phase distribution of the spectrum in the local and global frequency ranges, wherein the frequency domain filter weights W(f) are a set of real-valued weight vectors corresponding to the frequency f, which are used to control the spectrum energy distribution of different frequency bands in the frequency dimension; for frequency points corresponding to the noise frequency band, the values ​​of the corresponding frequency components in the frequency domain filter weights W(f) are set to be close to zero to suppress the amplitude response of the frequency points in the filtering results; for frequency points carrying audio abnormal event characteristics, the values ​​of the corresponding frequency components in the frequency domain filter weights W(f) are set to be close to 1 to retain the energy of the frequency points, thereby achieving selective enhancement of the target audio event spectrum and suppression of the background noise spectrum; A spectrum filtering unit is configured to perform a frequency-by-frequency element-wise multiplication operation on the complex spectrum vector X(f) and the frequency domain filtering weight W(f) to obtain a filtered spectrum vector Y(f), specifically: Y(f)=W(f)⊙X(f); Where "⊙" represents the multiplication of the corresponding frequency elements; The inverse transform unit is used to perform inverse fast Fourier transform (IFFT) on the filtered spectrum vector Y(f) to reconstruct the filtered time domain audio signal. Abnormal sound event recognition unit, used to filter the time domain audio signal Input the pre-trained audio event classification network to identify the abnormal event type e, the occurrence time τ, and the spectral pattern φ of the abnormal event, and use {e, τ, φ} as the structured output of the audio analysis module.

4. The smart park security protection system based on intelligent video analysis and multimodal integration according to claim 1 is characterized in that: The self-calibration module comprises: The trajectory extraction and filtering unit is used to extract the time-series trajectory of the same target in multiple camera images based on the video features output by the video analysis module, and perform credibility assessment on the time-series trajectory, eliminating unreliable trajectory points caused by occlusion, light interference or countermeasure disturbance to obtain the filtered target trajectory; The camera parameter estimation unit is used to calculate the relative spatial pose transformation parameters between different cameras based on the filtered target trajectory through multi-view reprojection consistency constraints, including the rotation matrix R and displacement vector T, and to infer the time synchronization deviation Δt between the cameras by combining trajectory timestamp matching and trajectory frame difference minimization strategy; The low-visibility compensation unit is used to compensate for calibration interruptions caused by missing target tracks by performing pose interpolation based on historical trajectory predictions, scene geometric edge features, or inertial measurement unit (IMU) data when any camera has a blind spot or insufficient lighting for target recognition for a short period of time. The audio auxiliary calibration unit is used to calculate the three-dimensional spatial coordinates of the abnormal sound source using the time difference localization (TDOA) algorithm based on the microphone array when the audio analysis module detects an abnormal sound source with spatial positioning characteristics, and introduce the three-dimensional spatial coordinates as an auxiliary constraint into the camera pose solution process; A calibration mapping construction unit is used to construct a unified spatiotemporal transformation relationship between cameras based on the rotation matrix R and displacement vector T, the time synchronization deviation Δt, and the three-dimensional spatial positioning coordinates of the abnormal sound source, thereby forming a global spatiotemporal mapping function for multimodal feature normalization processing; A multimodal feature alignment unit is used to perform unified conversion of the time and space dimensions on the video features, audio features and access control features according to the global spatiotemporal mapping function.

5. The smart park security protection system based on intelligent video analysis and multimodal integration according to claim 4 is characterized in that: The scoring function for evaluating the credibility of the time series trajectory by the trajectory extraction and filtering unit is expressed as: s(p,t)=α·IoU(p,t)+β·conf(p,t)+γ·Δθ(p,t); Where s(p,t) is the score value; p represents the target candidate box detected in the t-th frame; IoU(p,t) is the overlap rate with the prediction box of the previous frame; conf(p,t) is the detection confidence; Δθ(p,t) represents the degree of trajectory direction mutation; α, β, γ are weighting coefficients; When the score value s(p,t) is lower than the set score threshold, the trajectory point is marked as untrustworthy and removed.

6. The smart park security protection system based on intelligent video analysis and multimodal integration according to claim 1 is characterized in that: The multimodal fusion module includes: Modality mapping unit, used to align spatiotemporal video features Audio characteristics Access control features Mapped to the unified fusion subspace, we get: z v =W v ·f v ;z a =W a ·f a ;z c =W c ·f c ; in, Indicates that the video feature is a column vector consisting of mv real numbers, Indicates that the audio feature is a column vector consisting of ma real numbers, Indicates that the access control feature is a column vector consisting of mc real numbers; z v 、z a 、z c Map video features, audio features and access control features to a unified fusion subspace The intermediate representation vector after ; Respectively represent the linear mapping matrices of the corresponding modes; r is the fusion rank; A fusion rank control unit is used to dynamically adjust the value of the fusion rank r based on the complexity index of the current input feature. The complexity index includes the number of targets, modal information entropy, and historical anomaly frequency, so as to control the representation capacity of the fusion structure and achieve a dynamic trade-off between fusion accuracy and computational load; Product fusion unit, used to calculate the intermediate representation vector z of each modality v 、z a 、z c , and perform sum operations on the fusion dimensions to construct a joint tensor and the final joint representation vector The calculation method is: H[i] = z v [i] × z a [i] × z c [i], where i ∈ {1,..., r}; Where H[i] represents the trimodal product result on the i-th fusion dimension; The modality completion unit is used to estimate the alternative representation vector of the missing modality based on the acquired modality features and the modality co-occurrence distribution in historical similar scenarios when any modality input is detected to be missing or abnormal, and replace the original input to ensure the continuity of the fusion calculation.

7. The smart park security protection system based on intelligent video analysis and multimodal integration according to claim 1 is characterized in that: The security decision module includes: The abnormality discrimination and behavior recognition unit is used to receive the joint representation vector h, identify the behavior category through the residual structure neural network model, and evaluate the abnormality of the behavior category. The formula is: P anomaly =σ(ρ·(D(h)-δ)); where μ is the center vector of the normal behavior sample, which is obtained in advance through normal data statistics; D(h) is the square of the Euclidean distance between the joint representation vector h and the center vector μ, indicating the degree of deviation between the current sample and the normal sample; P anomaly is the probability that the current joint representation vector is judged as abnormal; σ(·) is the Sigmoid function; ρ is the abnormal score amplification factor; δ is the abnormal discrimination threshold; Modal robust reconstruction and defense mechanism, used when the probability P is detected anomaly When the probability of abnormality is greater than the threshold, the modal defense mechanism is automatically enabled based on the modal robustness factor ε j Readjust the contribution of each modal fusion and call the modal replacement function Enable redundant modes for structural compensation; The response strategy optimization unit is used to call the reinforcement learning model to calculate the response score Q(s,a) based on the current state s and the response action set A, and select the optimal action to generate the response strategy: Where, π * (s) represents the optimal response action that the system should take in state s; a is a candidate response action; It means selecting the action that maximizes the Q value in the response action set A, that is, obtaining the optimal response strategy.

8. The smart park security protection system based on intelligent video analysis and multimodal integration according to claim 1 is characterized in that: The central server also includes a privacy protection and modal defense module, specifically including: A feature desensitization processing unit is used to perform differential privacy perturbation or homomorphic encryption on video features or access control features containing personal identity information before uploading modal feature data to the edge computing node to prevent the original identity features from being restored; A dynamic gating control unit is used to dynamically generate a gating coefficient based on the confidence level of each modal feature and the scenario risk before the joint representation vector is generated, and to preemptively suppress modal features with redundancy higher than a threshold or confidence lower than a threshold; The adversarial interference detection and trust reconstruction unit is used to judge the change in modal credibility based on the contribution factor of multimodal input to output after the joint representation vector is generated, identify potential modal attack risks in combination with the inter-modal consistency indicator, and adjust the fusion contribution of the modality before triggering the response of the security decision module.

9. The smart park security protection system based on intelligent video analysis and multimodal integration according to claim 1 is characterized in that: The system also includes a federated collaborative learning unit for establishing encrypted communication links with multiple security protection systems deployed in different parks, periodically exchanging anonymized model parameters or statistical summary information, and collaboratively optimizing the neural network model parameters in the security decision-making module without transmitting the original modal data.

10. A smart park security protection method based on intelligent video analysis and multimodal integration, applicable to a smart park security protection system based on intelligent video analysis and multimodal integration as described in any one of claims 1 to 9, characterized in that: The method comprises: Front-end sensing devices deployed throughout the park collect video, audio, and traffic information; At the edge computing node, feature extraction is performed on the video, audio, and traffic information, and structured video features, audio features, and access control features are output; Performing differential privacy perturbation or homomorphic encryption on the data containing identity information in the video features or access control features to prevent unauthorized data leakage; In the central server, based on multi-view target trajectory reconstruction, audio localization, and time synchronization technology, a unified spatiotemporal mapping relationship is established between each front-end sensing device. Video features, audio features, and access control features are uniformly aligned in time and space. The aligned modal features are mapped to a unified fusion subspace, and the product fusion method is used to calculate the element-by-element product and sum of the intermediate vectors of the three modalities to construct the fused joint representation vector. The joint representation vector is input into the residual neural network to identify the current behavior category and calculate the probability that the current joint representation vector is judged to be abnormal based on the distance from the center vector of the normal sample. When the probability exceeds the set abnormal probability threshold, the modal robust reconstruction and defense mechanism is triggered; Based on the current state and the type of abnormal event, the reinforcement learning model is called to select the optimal response action, generate an alarm signal and trigger the security response strategy.

Images