A Smart Park Security Protection System and Method Based on Intelligent Video Analytics and Multimodal Integration

Abstract

This invention discloses a smart park security system and method based on intelligent video analytics and multimodal integration, belonging to the field of smart park management technology. The system includes cameras, environmental microphones, and access control terminals deployed throughout the park. Edge computing nodes are used to extract structured video, audio, and access control features. A central server achieves multimodal spatiotemporal alignment through a self-calibration module. A multimodal fusion module constructs a joint representation vector based on a low-rank tensor algorithm. A security decision module combines classification and anomaly detection models to identify abnormal events and trigger response strategies. This invention achieves multimodal collaborative recognition, highly robust behavior analysis, and intelligent response linkage, improving the real-time performance, accuracy, and intelligence level of the park security system.

Description

Technical Field

[0001] This invention relates to the field of smart park management technology, specifically to a smart park security protection system and method based on intelligent video analysis and multimodal integration. Background Technology

[0002] With the rapid development of smart city and IoT technologies, park security systems are gradually evolving towards intelligence, datafication, and networking. Traditional park security systems mainly rely on manual duty, single video surveillance, and access control card swiping, which are insufficient to achieve real-time perception and collaborative processing of multi-source information, early identification and high-confidence judgment of abnormal behavior, and highly robust anomaly protection and response linkage in complex scenarios.

[0003] Some existing technologies have introduced deep learning recognition algorithms for video surveillance, which can detect and classify people or vehicles in some scenarios. However, they still suffer from limitations such as single modality, lack of multimodal collaboration mechanisms such as audio and access control, inability to construct complete event semantics, and lack of self-calibration capabilities for video offset problems caused by camera deployment deviations, blind spot occlusion, and changes in lighting. Furthermore, they lack effective privacy protection mechanisms for identity data, making it difficult to perform large-scale behavior recognition under compliant conditions.

[0004] Therefore, there is an urgent need to build a distributed, intelligent, and reconfigurable multimodal security system that integrates front-end perception, mid-end processing, and back-end fusion to support efficient identification and emergency response to complex behavioral patterns in smart parks. Summary of the Invention

[0005] To address the above issues, this invention proposes a smart park security system and method based on intelligent video analytics and multimodal integration. It integrates multimodal data such as video, audio, and access control to achieve spatiotemporal alignment and intelligent fusion of heterogeneous security data. Furthermore, it utilizes edge computing and novel optical acceleration technologies to reduce latency and energy consumption in large-scale video analytics.

[0006] The present invention achieves the above objectives through the following technical solutions:

[0007] A smart park security system based on intelligent video analytics and multimodal integration includes front-end sensing devices deployed throughout the park and back-end processing units located at the network edge and center. The front-end sensing devices include several cameras, environmental microphones, and access control terminals, used to collect video, audio, and access information, respectively. The back-end processing unit includes edge computing nodes and a central server. The edge computing nodes include:

[0008] The video analysis module is used to receive video streams, perform target detection, tracking and behavior recognition on image frames, and use multi-layer convolutional neural networks and Transformer self-attention mechanism to extract the location, category, trajectory and facial features of people or vehicles, and output structured video features.

[0009] The audio analysis module is used to receive audio signals for spectrum analysis and anomaly identification, and to extract audio features including the type of anomaly, the time of occurrence, and the corresponding spectrum pattern.

[0010] The access control identification module is used to identify personnel identity based on access information, record access time and corresponding access control terminal number, and determine whether it is abnormal access by combining the behavior identification results. If so, it outputs access control features including personnel identity, access time, access control terminal number and behavior tag.

[0011] The central server includes:

[0012] The self-calibration module is used to calculate the temporal and spatial transformation parameters between cameras based on the trajectory of the same target in different video frames, and to perform spatiotemporal alignment of the video features, audio features and access control features.

[0013] The multimodal fusion module is used to perform multiplication and weighting operations on the spatiotemporally aligned modal features based on the low-rank tensor fusion algorithm to construct a joint representation vector of the multimodal features;

[0014] The security decision module is used to receive the joint representation vector, identify behavior categories through a neural network model with residual structure, and generate alarm signals and trigger security response strategies if abnormal events occur.

[0015] Preferably, the video analysis module includes an optical neural network acceleration unit for accelerating the execution of convolution operations and attention operations in image frames, specifically including:

[0016] The input modulation module is used to convert the pixel tensor of the image frame into a multi-channel light intensity signal through an electro-optic modulator based on a modulation network composed of a Mach-Zehnder interferometer array, so as to realize the parallel encoding of high-dimensional input features in the optical domain.

[0017] A reconfigurable optical computing array includes multiple optical convolution kernel channels and optical self-attention channels. The optical convolution kernel channels are constructed based on tensor decomposition, using tensor columns or CP decomposition to decompose the original high-dimensional weight tensor into low-rank sub-kernels. These sub-kernels are then loaded into the optical path through a spatial light modulator and phase modulator network to perform dot product operations on the low-rank convolution kernels. The optical self-attention channels employ a multi-wavelength interference path structure, enabling parallel computation of different attention heads through wavelength division multiplexing. Each attention calculation consists of an optical waveguide matrix multiplier and a tunable microcavity array, used to perform matrix dot product and weighted sum operations in the Query-Key-Value ternary relation.

[0018] The interferometry and detection module is equipped with two free spectral range paths and a balanced detector, which are used to differentially obtain the optical calculation results of positive and negative weight contributions, and realize convolution and attention operation output that support negative weights;

[0019] The photoelectric conversion and sampling module is used to convert the light intensity signal of the interference output into a digital signal via a photodetector, and then send it to the subsequent neural network classifier for target classification and behavior judgment.

[0020] An optical storage and multiplexing module is used to cache intermediate feature optical signals output by optical convolution kernel channels and optical self-attention channels within the reconfigurable optical computing array, and transmit the intermediate feature optical signals to downstream or parallel channels through an optical waveguide feedback structure, thereby realizing cross-layer sharing and reuse of feature information.

[0021] The control and programming interface module is connected to the input modulation module, spatial light modulator, phase controller and wavelength division multiplexing path. It is used to load neural network weight configuration files according to different tasks, dynamically control the optical channel structure, tensor kernel arrangement and attention head number, and realize the structure switching and hardware reconstruction for different video processing tasks.

[0022] Preferably, the audio analysis module includes:

[0023] The frequency domain conversion unit is used to perform a Fast Fourier Transform (FFT) on the acquired raw audio signal to obtain a complex spectral vector X(f);

[0024] A dynamic filtering weight generation unit is used to input the complex spectrum vector X(f) into a spectrum attention network containing trainable parameters. Based on the amplitude characteristics and phase distribution of the spectrum in the local and global frequency ranges, it dynamically generates frequency domain filtering weights W(f), where the frequency domain filtering weights W(f) are a set of real-valued weight vectors corresponding to frequency f, used to control the spectral energy distribution of different frequency bands in the frequency dimension. For frequency points corresponding to noise frequency bands, the value of the corresponding frequency component in the frequency domain filtering weights W(f) is set to approach zero to suppress the amplitude response of that frequency point in the filtering result. For frequency points carrying audio abnormal event characteristics, the value of the corresponding frequency component in the frequency domain filtering weights W(f) is set to approach 1 to retain the energy of that frequency point, thereby achieving selective enhancement of the spectrum of the target audio event and suppression of the background noise spectrum.

[0025] The spectrum filtering unit is used to perform a frequency-wise element-wise multiplication operation on the complex spectrum vector X(f) and the frequency domain filtering weight W(f) to obtain the filtered spectrum vector Y(f), specifically:

[0026] Y(f) = W(f)⊙X(f);

[0027] Where “⊙” represents the multiplication of the corresponding frequency elements;

[0028] The inverse transform unit performs an inverse fast Fourier transform (IFFT) on the filtered spectral vector Y(f) to reconstruct the filtered time-domain audio signal.

[0029] An abnormal sound event recognition unit is used to identify the filtered time-domain audio signal. Input the pre-trained audio event classification network to identify the abnormal event type e, occurrence time τ, and spectral pattern φ of the abnormal event, and use {e,τ,φ} as the structured output of the audio analysis module.

[0030] Preferably, the self-calibration module includes:

[0031] The trajectory extraction and filtering unit is used to extract the temporal trajectory of the same target in multiple camera images based on the video features output by the video analysis module, and to evaluate the credibility of the temporal trajectory, remove unreliable trajectory points caused by occlusion, light interference or anti-disturbance, and obtain the filtered target trajectory.

[0032] The camera parameter estimation unit is used to calculate the relative spatial pose transformation parameters between different cameras based on the filtered target trajectory through multi-view reprojection consistency constraints, including the rotation matrix R and the displacement vector T, and to calculate the time synchronization deviation Δt between cameras by combining trajectory timestamp matching and trajectory frame difference minimization strategies.

[0033] The low visibility compensation unit is used to compensate for the calibration interruption caused by the lack of target trajectory by performing pose interpolation based on historical trajectory prediction, scene geometric edge features, or inertial measurement unit (IMU) data when any camera has a blind spot or insufficient lighting to identify the target for a short period of time.

[0034] The audio auxiliary calibration unit is used to calculate the three-dimensional spatial positioning coordinates of the abnormal sound source based on the microphone array and the Time Difference of Origin (TDOA) algorithm when the audio analysis module detects an abnormal sound source with spatial positioning characteristics. The three-dimensional spatial positioning coordinates are then used as auxiliary constraints in the camera pose solving process.

[0035] The calibration mapping construction unit is used to construct a unified spatiotemporal transformation relationship between cameras based on the rotation matrix R, displacement vector T, time synchronization deviation Δt, and the three-dimensional spatial positioning coordinates of the abnormal sound source, and form a global spatiotemporal mapping function for multimodal feature normalization processing.

[0036] The multimodal feature alignment unit is used to perform a unified transformation of the time and space dimensions on the video features, audio features and access control features respectively according to the global spatiotemporal mapping function.

[0037] Preferably, the scoring function used by the trajectory extraction and filtering unit to evaluate the credibility of the time-series trajectory is expressed as:

[0038] s(p,t)=α·IoU(p,t)+β·conf(p,t)+γ·Δθ(p,t);

[0039] In the formula, s(p,t) is the score value; p represents the target candidate box detected in frame t; IoU(p,t) is the overlap rate with the predicted box in the previous frame; conf(p,t) is the detection confidence; Δθ(p,t) represents the degree of abrupt change in trajectory direction; α, β, and γ are weighting coefficients.

[0040] When the score value s(p,t) is lower than the set score threshold, the trajectory point is marked as unreliable and removed.

[0041] Preferably, the multimodal fusion module includes:

[0042] Modality mapping unit, used to align spatiotemporally video features Audio features Access control features Mapping to the unified fusion subspace, we get: z v =W v ·f v ;z a =W a ·f a ;z c =Wc ·f c ;

[0043] in, The video feature is represented as a column vector consisting of mv real numbers. The audio features are represented as a column vector consisting of ma real numbers. The access control feature is represented as a column vector consisting of mc real numbers; z v z a z c Video features, audio features, and access control features are mapped to a unified fusion subspace, respectively. The intermediate representation vector is shown below; Let represent the linear mapping matrices for the corresponding modes; r is the fusion rank;

[0044] The fusion rank control unit is used to dynamically adjust the value of the fusion rank r according to the complexity index of the current input features. The complexity index includes the number of targets, modal information entropy and historical anomaly frequency, which is used to control the representation capacity of the fusion structure and achieve a dynamic trade-off between fusion accuracy and computational load.

[0045] The product fusion unit is used to calculate the intermediate representation vector z of each modality. v z a z c Element-wise multiplication and summation over the fusion dimension are performed to construct a joint tensor. and the final joint representation vector The calculation method is as follows:

[0046] H[i]=z v [i]×z a [i]×z c [i], where i∈{1,…,r};

[0047]

[0048] In the formula, H[i] represents the result of the three-modal product on the i-th fusion dimension;

[0049] The modal completion unit is used to estimate the alternative representation vector of the missing modality based on the acquired modal features and the modal co-occurrence distribution in similar historical scenarios when any modal input is detected to be missing or abnormal. The original input is then replaced to ensure the continuity of the fusion computation.

[0050] Preferably, the security decision module includes:

[0051] The anomaly detection and behavior recognition unit receives the joint representation vector h, identifies behavior categories through a residual neural network model, and evaluates the degree of anomaly of the behavior category, using the following formula:

[0052] In the formula, μ is the center vector of the normal behavior sample, which is obtained in advance through statistical analysis of normal data; D(h) is the squared Euclidean distance between the joint representation vector h and the center vector μ, representing the degree of deviation between the current sample and the normal sample; P anomaly σ(·) represents the probability that the current joint representation vector is judged as an anomaly; ρ is the anomaly scoring amplification factor; and δ is the anomaly discrimination threshold.

[0053] Modal robust reconstruction and defense mechanisms are used to detect probabilities P. anomaly When the probability exceeds the anomaly threshold, a modal defense mechanism is automatically activated based on the modal robustness factor ε. j The contributions of each modality fusion are readjusted, and the modality replacement function is invoked. Activate redundant modes for structural compensation;

[0054] The response policy optimization unit is used to calculate the response score Q(s,a) by calling the reinforcement learning model based on the current state s and the set of response actions A, and to select the optimal action to generate the response policy.

[0055]

[0056] In the formula, π * (s) represents the optimal response action that the system should take in state s; a is a candidate response action;

[0057] This means selecting the action that maximizes the Q value from the set of response actions A, thus obtaining the optimal response strategy.

[0058] Preferably, the central server further includes a privacy protection and modality defense module, specifically comprising:

[0059] The feature desensitization processing unit is used to perform differential privacy perturbation or homomorphic encryption on video features or access control features containing personal identity information before uploading modal feature data to the edge computing node, so as to prevent the original identity features from being restored.

[0060] The dynamic gating control unit is used to dynamically generate gating coefficients based on the confidence level and scenario risk of each modality feature before the joint representation vector is generated, and to perform preemptive suppression on modality features with redundancy higher than the threshold or confidence lower than the threshold.

[0061] The anti-interference detection and trust reconstruction unit is used to determine the modality credibility change based on the contribution influence factor of multimodal input to output after the joint representation vector is generated, identify potential modality attack risks by combining the intermodal consistency index, and adjust the fusion contribution of the modality before the security decision module responds.

[0062] Preferably, the system further includes a federated collaborative learning unit, which is used to establish encrypted communication links with multiple security protection systems deployed in different parks, periodically exchange anonymized model parameters or statistical summary information, and collaboratively optimize the neural network model parameters in the security decision module without transmitting the original modal data.

[0063] The smart park security protection method based on intelligent video analysis and multimodal integration is applicable to the smart park security protection system based on intelligent video analysis and multimodal integration as described above. The method includes:

[0064] Video, audio, and access information are collected by front-end sensing devices deployed throughout the park;

[0065] At the edge computing node, features are extracted from the video, audio, and access information respectively, and structured video features, audio features, and access control features are output.

[0066] Differential privacy perturbation or homomorphic encryption is applied to the video features or access control features containing identity information to prevent unauthorized data leakage;

[0067] In the central server, based on multi-view target trajectory reconstruction, audio localization and time synchronization technologies, a unified spatiotemporal mapping relationship is constructed between various front-end sensing devices to perform unified temporal and spatial alignment of video features, audio features and access control features;

[0068] The aligned modal features are mapped to a unified fusion subspace. The element-wise product and summation of the intermediate vectors of the three modalities are calculated using a product fusion method to construct the fused joint representation vector.

[0069] The joint representation vector is input into the residual neural network to identify the current behavior category, and the probability that the current joint representation vector is judged as abnormal is calculated based on the distance to the center vector of normal samples. When the probability exceeds the set abnormal probability threshold, the modal robust reconstruction and defense mechanism is triggered.

[0070] Based on the current state and the type of abnormal event, the reinforcement learning model is invoked to select the optimal response action, generate an alarm signal, and trigger the security response strategy.

[0071] The beneficial effects of this invention are as follows: By comprehensively collecting video, audio, and access control information, the system's perception integrity and discrimination capabilities for complex behavioral scenarios are enhanced. Structured video and audio data (such as access control features) are extracted at edge computing nodes, reducing communication load and improving real-time processing capabilities. The self-calibration module automatically calculates the time synchronization difference and spatial pose mapping between devices based on the projection relationship of the target trajectory in multiple cameras, solving problems such as occlusion, offset, and asynchrony in the deployment of multi-source cameras in the park, ensuring the consistency and comparability of multimodal data before fusion. The multimodal fusion module projects the three modal features onto a unified subspace and performs low-rank fusion through a product-weighted method, forming a joint representation vector with compactness, discriminability, and modal complementarity. Based on the residual neural network model, the system identifies whether abnormal events exist in the park, generates alarm signals, and activates corresponding security response strategies, achieving closed-loop event processing. Attached Figure Description

[0072] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] in:

[0074] Figure 1 This is a system structure block diagram of the present invention;

[0075] Figure 2 This is a structural block diagram of the optical neural network acceleration unit in an embodiment of the present invention;

[0076] Figure 3 This is a structural block diagram of the audio analysis module in an embodiment of the present invention;

[0077] Figure 4 This is a flowchart of the method in an embodiment of the present invention. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0079] like Figure 1As shown, this is an embodiment of the present invention, which provides a smart park security protection system based on intelligent video analysis and multimodal integration, and adopts a three-layer architecture of "front-end perception - edge processing - central decision-making".

[0080] Front-end perception layer: This includes front-end perception devices deployed throughout the park, such as several high-definition cameras, environmental microphones, access control terminals (card swipe / facial recognition terminals), infrared / smoke sensors, etc. Each device connects to the edge computing node via wired / wireless networks. Cameras acquire video streams, environmental microphones acquire audio streams, access control terminals record personnel entry and exit information including card swipes and corresponding identification, and other sensors provide signals for environmental anomalies (such as fire alarms).

[0081] Edge intelligent processing layer: Composed of multiple edge computing nodes distributed within the campus LAN, this layer handles preliminary data analysis tasks locally, reducing the pressure on the central processing unit and response latency. Each node integrates an optical neural network acceleration unit, offloading heavy deep learning inference (such as video parsing) to the optical neural network accelerator for efficient and low-power computing. Edge computing nodes run various modal analyses (video, audio, access control, etc.) and local data caching, performing real-time processing and temporary storage of front-end data. Preliminary analysis results (such as detected targets, identified identities, and sensor readings) are timestamped and sent to the central processing unit.

[0082] Central Decision-Making Layer: Deployed in the monitoring and command center, this comprehensive security platform boasts enhanced computing power and a complete data view. The central server receives event streams and feature data uploaded from various edge computing nodes, performing multimodal data fusion and advanced analysis for decision-making. It is responsible for correlating information from different sources to form a holistic understanding of the overall situation within the park, and issuing alarms, triggering security equipment (such as door locks and broadcasts), or notifying management personnel.

[0083] The entire architecture adopts a distributed collaborative working approach: the front-end is ubiquitously deployed to collect dynamic information about the park in all aspects; edge computing nodes intelligently filter and accelerate processing to protect privacy on-site; and the central server integrates and synchronizes data, makes comprehensive decisions, and achieves closed-loop security protection from the terminal to the center.

[0084] Specifically, the edge computing node includes a video analytics module, an audio analytics module, and an access control recognition module.

[0085] The video analysis module receives video streams from cameras and performs intelligent analysis. Its functions include: target detection and tracking (personnel and vehicle recognition and trajectory), abnormal behavior recognition (fighting, climbing fences, etc.), and face recognition comparison. It uses multi-layer convolutional neural networks and Transformer self-attention mechanism to extract features and output structured video features, such as "Time T, Location X, Person Y was detected climbing fence".

[0086] The audio analysis module performs spectrum analysis and abnormal event identification (such as glass breaking sound, screams, alarm sounds, etc.) on the audio signals collected by the environmental microphone. It uses a convolutional neural network + MFCC feature extraction method to extract acoustic features, and introduces FFT filtering preprocessing to suppress irrelevant noise frequency bands in the audio spectrum to improve the recognition accuracy. It generates audio alarms in real time and outputs audio features including abnormal event type, occurrence time and corresponding spectrum mode, such as "glass breaking sound detected in region Z at time T".

[0087] The access control identification module is responsible for processing communication information from access control card swipes, fingerprints, facial recognition and other identity authentication devices. On the one hand, it records data such as personnel identification, passage time and access control terminal number. On the other hand, it calls the facial recognition results from the video analysis module for comparison and verification, and combines the behavioral recognition results to determine whether there is abnormal passage. If there is abnormal passage such as unauthorized personnel following into the door, the abnormal event can be marked in real time (such as "person without card swipe follows cardholder into the door"), and the access control features such as personnel identification, passage time, access control terminal number and behavioral tags can be output. The module also converts access control logs into semantic embeddings, and reduces redundant ID signal interference through the gate control mechanism when necessary, for central fusion reference.

[0088] like Figure 2 As shown, in one preferred embodiment, the video analysis module is configured with an optical neural network acceleration unit for accelerating the execution of convolution operations and fast inference of self-attention mechanisms in image frames at edge computing nodes, specifically including:

[0089] The input modulation module is used to receive image frame input and convert it into light intensity signals that can be used for optical computing processing. Specifically, the module uses an electro-optic modulator (EOM) array combined with a Mach-Zehnder interferometer (MZI) network to encode the pixel tensor of each image frame in a high dimension. Each pixel value is modulated into light signals of different intensities or phases through the modulator array, so as to realize the parallel loading of input features in the optical domain.

[0090] A reconfigurable optical computing array contains multiple parallel-configured optical convolutional kernel channels and optical self-attention channels, corresponding to convolutional layers and attention layers in a deep neural network, respectively.

[0091] The optical convolution kernel channel employs tensor decomposition methods, such as Tensor Train (TT) or CP decomposition, to decompose the original high-dimensional weight tensor into several low-rank sub-kernels. Each sub-kernel is loaded into the optical path in a specific manner through a Spatial Light Modulator (SLM) and a phase modulator, and performs a dot product operation with the input optical signal to achieve convolution calculation. The optical self-attention channel is used to accelerate the multi-head self-attention mechanism in the Transformer architecture. This channel adopts a multi-wavelength interference path structure and achieves parallel operation of different attention heads through Wavelength Division Multiplexing (WDM). In each attention calculation, the Query, Key, and Value tensors are input through an optical waveguide array, and matrix multiplication and weighted sum operations are performed using a tunable microcavity array to obtain the attention aggregation result.

[0092] The interferometry and detection module, configured with two Free Spectral Range (FSR) paths and a balanced detector to support the representation of positive and negative weights in the network, performs interferometry and differential detection on the positive and negative light intensity signals. This structure ensures effective separation of positive and negative contributions in the tensor product results, expanding the expressive power of optical computation and the model's adaptability.

[0093] The photoelectric conversion and sampling module outputs the optical calculation results through the interferometry and detection module. The results are then converted into electrical signals by a high-response photodetector and digitized by an analog-to-digital converter. The resulting feature vector is then input into the subsequent neural network classifier in the edge computing device to identify the structured information of the target in the image frame, such as its category, location, trajectory, and behavioral labels.

[0094] To improve resource utilization in the optical computing path, this embodiment incorporates multiple optical buffer structures and optical waveguide feedback paths. These structures temporarily store intermediate feature results in the optical domain and enable information reuse across layers or channels. This module can guide some intermediate layer feature signals to subsequent convolutional kernels or attention head paths for computation, thereby reducing redundant computation and improving energy efficiency.

[0095] The control and programming interface module provides the optical acceleration unit with the ability to configure operation and switch tasks. It connects with the input modulation module, spatial light modulator, phase controller and wavelength division multiplexing path. It is used to load neural network weight configuration files according to different tasks, dynamically set tensor decomposition parameters, adjust the number of attention heads, configure convolution kernel size and channel structure, thereby realizing the structural reconstruction and operation scheduling of optical computing path.

[0096] The optical neural network acceleration unit achieves high throughput, low latency, and low power consumption processing capabilities for convolution and attention calculations in image frames through parallel optical computing, tensor low-rank decomposition loading, optical domain feature storage, and programmable path reconstruction. Compared with traditional GPU inference solutions, it has higher energy efficiency and edge deployment feasibility.

[0097] Through the above structural design, the optical neural network acceleration unit in this embodiment can complete feature extraction and forward inference of image frames at edge nodes with millisecond-level latency, significantly reducing the power consumption and heat dissipation burden required by traditional GPUs. In typical scenarios, the inference energy efficiency ratio can reach more than 50 times that of traditional electronic solutions, making it particularly suitable for multi-point edge analysis scenarios in campus security that require the deployment of large-scale high-definition video streams. In addition, this acceleration unit has modular design characteristics, which can flexibly expand optical computing arrays of different sizes, and the weights and network structure can be quickly reconstructed through the control module to adapt to the diverse needs of different areas and functional nodes in the campus for video recognition accuracy and real-time performance.

[0098] like Figure 3 As shown, in one preferred embodiment, environmental microphones deployed in areas such as park entrances, corridors, and parking lots collect analog audio signals s(t), which are then converted by an A / D converter to obtain discrete audio sequences. The sampling rate is set to 16kHz, the frame length to 512 sampling points (approximately 32 milliseconds), the frame shift to 256 points, and a Hamming window is used for windowing processing to form frame-level audio data. The audio analysis module, based on a dynamic spectrum filtering structure, analyzes the collected audio signals in real time to identify and structurally extract abnormal events (such as broken glass, screams, alarms, etc.), specifically including:

[0099] The frequency domain conversion unit performs a Fast Fourier Transform (FFT) on each frame of audio signal s(t) to obtain a complex spectrum vector X(f). The complex number field is the set of all complex numbers, N = 257 (the positive frequency part is retained due to the use of real number symmetry), and X(f) contains the amplitude and phase information of each frequency point.

[0100] The dynamic filter weight generation unit inputs the complex spectral vector X(f) into a spectral attention network containing trainable parameters. This network consists of two one-dimensional convolutional layers (kernel width 5, stride 1), one normalization layer, and a sigmoid activation function. Based on the amplitude characteristics and phase distribution of the spectrum in the local and global frequency ranges, it dynamically generates frequency domain filter weights W(f), where the frequency domain filter weights W(f) are a set of real-valued weight vectors corresponding to frequency f, used to control the spectral energy distribution of different frequency bands in the frequency dimension. For frequency points corresponding to noise frequency bands, the value of the corresponding frequency component in the frequency domain filter weights W(f) is set to approach zero to suppress the amplitude response of that frequency point in the filtering result. For frequency points carrying audio abnormal event characteristics, the value of the corresponding frequency component in the frequency domain filter weights W(f) is set to approach 1 to retain the energy of that frequency point, thereby achieving selective enhancement of the spectrum of the target audio event and suppression of the background noise spectrum.

[0101] This spectrum attention network aims to improve the accuracy of identifying anomalous audio events during the training phase. It automatically learns the correlation between frequency points and event labels through backpropagation. During network training, the network tends to generate weights close to 1 for anomalous event frequencies (such as glass breakage concentrated in 2–4kHz), while generating weights close to 0 for non-discriminatory frequency bands such as low-frequency motor noise (such as 50–300Hz), thus achieving selective retention and suppression of spectrum energy.

[0102] The spectrum filtering unit performs a frequency-wise element-wise multiplication operation on the complex spectrum vector X(f) and the frequency domain filtering weights W(f) to obtain the filtered spectrum vector Y(f), specifically:

[0103] Y(f) = W(f)⊙X(f);

[0104] Where “⊙” represents the multiplication of the corresponding frequency elements; the frequency bands close to 0 in W(f) do not contribute to Y(f), while the energy of the frequency bands close to 1 is retained.

[0105] The inverse transform unit performs an inverse fast Fourier transform (IFFT) on the filtered spectral vector Y(f), retains the real part, and reconstructs the filtered time-domain audio signal. The signal is more semantically significant than the original s(t) and the background noise is effectively suppressed.

[0106] The abnormal sound event recognition unit will filter the time-domain audio signal. Input a pre-trained audio event classification network (which can be a deep neural network consisting of 3 convolutional layers and 1 GRU layer) and output the abnormal event type e, confidence branch, occurrence time τ, etc. corresponding to the current frame. At the same time, output the spectral pattern φ of the frame (such as the filtered feature map) to form structured audio features {e,τ,φ}, which are used for subsequent alignment and fusion with video / access control modal features.

[0107] The aforementioned spectrum attention network and audio event classification network are trained end-to-end during the training phase, with recognition accuracy as the joint loss function optimization objective. After training, they are deployed on edge computing nodes (such as Jetson XavierNX) within the park. Actual testing shows that in complex environments, recognition accuracy is improved by 15%, the false alarm rate is reduced to 8.1%, and the recognition latency is controlled within 150ms, meeting the park's real-time response requirements for abnormal events.

[0108] This embodiment shows that, compared with the traditional MFCC+CNN method, the audio analysis structure based on dynamic spectrum filtering has better frequency band separation and background noise suppression capabilities, and has efficient edge inference performance, which can effectively improve the ability of smart parks to perceive and respond to sudden acoustic events.

[0109] The central server includes a self-calibration module, a multimodal fusion module, and a security decision module.

[0110] The self-calibration module achieves dynamic spatial calibration and time synchronization between multiple cameras based on the trajectory of the same target in different video frames. This ensures that video, audio, and access control feature data from multiple front-end sensing devices can be fused and analyzed under a unified time axis and spatial coordinate system. This avoids the impact on subsequent abnormal behavior recognition and security event detection caused by inconsistencies in multimodal data due to perspective deviation, device drift, or time differences.

[0111] In one embodiment, the self-calibration module includes:

[0112] The trajectory extraction and filtering unit is used to extract the position data of the same target (such as people or vehicles) in continuous frames in different camera images based on the video features output by the video analysis module. It adopts a cross-camera identity association algorithm (such as ReID + temporal matching) to integrate the trajectory of the same target under different viewpoints, and evaluates the credibility of the integrated temporal trajectory. It removes untrusted trajectory points caused by occlusion, light interference or adversarial perturbation to obtain the filtered target trajectory.

[0113] The camera parameter estimation unit is used to calculate the relative spatial pose transformation parameters between different cameras based on the filtered target trajectory and through multi-view reprojection consistency constraints, including the rotation matrix R and the displacement vector T. In addition, in order to synchronize the time offset between multiple cameras caused by different frame rates, network latency, and different triggering mechanisms, the unit further calculates the time synchronization deviation Δt between cameras by matching trajectory timestamps, minimizing reprojection inter-frame errors, and aligning acceleration trajectories, thereby achieving frame-level precision temporal alignment.

[0114] The low visibility compensation unit is used to interpolate or extrapolate the camera pose within the currently missing segment when any camera has a blind spot (such as temporary structures or vehicle obstruction) or insufficient lighting (such as no supplementary lighting at night) and cannot identify the target for a short period of time. This is based on historical trajectory prediction, scene geometric edge features (obtained by building a scene set model through SLAM or NeRF), or relative motion information provided by the inertial measurement unit (IMU) data. This maintains the continuity and stability of the spatiotemporal alignment chain and compensates for the calibration interruption caused by the missing target trajectory.

[0115] The audio-assisted calibration unit is designed for situations in abnormal events where the sound is not visually visible but is significant (such as screams at night, breaking sounds, explosions, etc.). When the audio analysis module detects an abnormal sound source with spatial positioning characteristics (such as a short pulse), it calculates the three-dimensional spatial positioning coordinates of the abnormal sound source based on the microphone array using the Time Difference of Arrival (TDOA) algorithm. The three-dimensional spatial positioning coordinates are then introduced as an auxiliary constraint into the camera pose solution process to improve the stability and accuracy of the camera parameter solution. In particular, it can maintain the system's global perception of the scene structure even when visual targets are missing.

[0116] The calibration mapping construction unit is used to construct a unified spatiotemporal transformation relationship between cameras based on the rotation matrix R, displacement vector T, time synchronization deviation Δt, and the three-dimensional spatial positioning coordinates of the abnormal sound source, forming a global spatiotemporal mapping function for multimodal feature normalization processing. Its input is the timestamp of the original modal feature and the local coordinates (x,t), and the output is the position and unified timestamp in the global reference coordinate system.

[0117] The multimodal feature alignment unit performs a unified transformation of the time and space dimensions of video features, audio features, and access control features according to the global spatiotemporal mapping function, and outputs aligned features with global three-dimensional coordinates and standardized timestamps. The alignment result can be directly input into the multimodal fusion module for subsequent anomaly detection and decision analysis to ensure the alignment accuracy and scene consistency between data.

[0118] In one embodiment of the present invention, to improve the robustness of the self-calibration module in complex campus scenarios, a trajectory extraction and filtering unit introduces a trajectory credibility scoring mechanism. This mechanism is used to filter and clean candidate trajectory points detected in multi-camera images under conditions such as target occlusion, sudden changes in illumination, or adversarial interference, thereby avoiding inputting unreliable data into the pose estimation process.

[0119] Specifically, the scoring function for the reliability assessment of time-series trajectories by the trajectory extraction and filtering unit is expressed as follows:

[0120] s(p,t)=α·IoU(p,t)+β·conf(p,t)+γ·Δθ(p,t);

[0121] In the formula, t represents the time index of the current frame (unit: frame number), used to identify the temporal position of p in the video; p represents the target candidate box detected in frame t; s(p,t) is the score value; IoU(p,t) is the overlap rate with the predicted box of the previous frame, that is, the intersection-over-union ratio between the candidate box p and its corresponding predicted trajectory box in frame t-1, used to measure the consistency of trajectory position; conf(p,t) is the detection confidence of the candidate box p in frame t, output by the target detection neural network, and the range is generally 0 to 1; Δθ(p,t) represents the degree of trajectory direction change, that is, the rate of change of the motion direction of the candidate point p between frame t and the previous frame, used to identify trajectory changes, such as by calculating the angle between the motion vectors or the rate of change of velocity between the two frames; α, β, γ are weighting coefficients, which can be set according to the experience of the actual scene, or automatically optimized through the training process;

[0122] When the score s(p,t) is lower than the set score threshold, the trajectory point is marked as unreliable and removed, and does not participate in the subsequent camera pose estimation process.

[0123] This scoring mechanism can effectively identify and eliminate false trajectory data caused by false detections, occlusion, jitter, or physical attacks (such as occluding key areas of an image), thereby improving the stability and accuracy of the self-calibration module in complex environments.

[0124] The multimodal fusion module is used to perform multiplication and weighting operations on the spatiotemporally aligned modal features based on the low-rank tensor fusion algorithm to construct a joint representation vector of the multimodal features;

[0125] In one optional embodiment, the multimodal fusion module includes:

[0126] Modality mapping unit, used to align spatiotemporally video features Audio features Access control features Mapping to the unified fusion subspace, we get: z v =W v ·f v ;z a =W a ·f a ;z c =W c ·f c ;

[0127] in, The video feature is represented as a column vector consisting of mv real numbers. The audio features are represented as a column vector consisting of ma real numbers. The access control feature is represented as a column vector consisting of mc real numbers; z v z a z c Video features, audio features, and access control features are mapped to a unified fusion subspace, respectively. The intermediate representation vector is shown below; Let r represent the linear mapping matrices for the corresponding modes; r is the fusion rank.

[0128] The fusion rank control unit is used to dynamically adjust the value of the fusion rank r according to the complexity index of the current input features. The complexity index includes the number of targets, modal information entropy and historical anomaly frequency, which is used to control the representation capacity of the fusion structure and achieve a dynamic trade-off between fusion accuracy and computational load.

[0129] In this scenario, mv = 128, ma = 64, and mc = 32. If the number of targets detected in the current time period is large (>4 people) and the mutual information between modalities is significantly improved (>0.85), then r = 64 is automatically adjusted. Conversely, in a high-confidence scenario with no traffic at night, r is reduced to 32 to reduce computational overhead.

[0130] The product fusion unit is used to calculate the intermediate representation vector z of each modality. v z a z c Element-wise multiplication and summation over the fusion dimension are performed to construct a joint tensor. and the final joint representation vector The calculation method is as follows:

[0131] H[i]=z v [i]×z a [i]×z c [i], where i∈{1,…,r};

[0132]

[0133] In the formula, H[i] represents the result of the three-modal product on the i-th fusion dimension.

[0134] The modal completion unit is used to estimate the alternative representation vector of the missing modality based on the acquired modal features and the modal co-occurrence distribution of similar historical scenarios when any modal input is detected to be missing or abnormal. This alternative representation vector replaces the original input to ensure the continuity of fusion computation. For example, in a certain actual operation, if the access control device loses network access and cannot upload card swipe data, resulting in the missing access control feature fc, the modal completion unit will construct the input z based on the historical audio and video co-occurrence patterns. v z a The fusion encoding is input into the generative adversarial network G to generate This is used to replace the missing access control feature input and continue to complete the joint representation vector calculation, ensuring the consistency of the fusion result.

[0135] Through the above implementation methods, the multimodal fusion module of this embodiment achieves efficient fusion of heterogeneous data sources in complex scenarios, dynamic rank adjustment and modal robustness compensation, and can link with downstream response systems to realize intelligent security identification and real-time response in smart parks.

[0136] The security decision module receives the joint representation vector, identifies the behavior category through a neural network model with residual structure, and generates an alarm signal and triggers a security response strategy if an abnormal event occurs.

[0137] In one specific embodiment, the security decision module includes:

[0138] The anomaly detection and behavior recognition unit receives a joint representation vector h, which represents the scene state at the current time t and spatial coordinates (x, y) after modal fusion of video, audio, and access control. In this embodiment, a multilayer perceptron (MLP) model with a residual structure is used to classify and infer the joint representation vector h, outputting the current behavior category label, such as "normal passage," "illegal crossing," and "nighttime intrusion." Simultaneously, using the preset center vector μ of normal behavior samples, the squared Euclidean distance D(h) between the joint representation vector h and the center vector μ is calculated as the deviation.

[0139] The deviation is then converted into anomaly probability, as shown in the formula: P anomaly =σ(ρ·(D(h)-δ));

[0140] In the formula, P anomaly σ(·) represents the probability that the current joint representation vector is judged as abnormal. The closer the value is to 1, the more likely the current behavior is abnormal. σ(·) is the Sigmoid function. ρ is the anomaly scoring amplification factor. δ is an adjustable anomaly discrimination threshold.

[0141] Modal robust reconstruction and defense mechanisms are used to detect probabilities P. anomaly When the anomaly probability exceeds the set anomaly probability threshold, the system automatically activates the modal defense mechanism. A modal robustness factor ε is pre-set for each modality j. j For example, under nighttime conditions, the ε of the video modality j It can be set to a lower value, for both audio and access control modes ε. j The value is relatively high; the modal robustness factor ε j Used to adjust the modal fusion contribution, reduce the fusion weight of unreliable modes, and call the modal replacement function. (When mode j is identified as abnormal, its redundant candidate mode j' is called to replace the original mode input) Enable redundant modes for structural compensation, such as switching the video mode affected by strong light interference to the millimeter-wave radar mode, or switching the noisy audio channel to the vibration sensing channel, to maintain the stability and reliability of the system fusion output.

[0142] The response strategy optimization unit is used to calculate the response score Q(s,a) by calling the reinforcement learning model based on the current state s (including information such as anomaly type, confidence level, time and location) and the preset set of response actions A (such as activating the alarm, remotely locking the door, notifying security, etc.), and to select the optimal action to generate the response strategy.

[0143]

[0144] In the formula, π * (s) represents the optimal response action that the system should take in state s; a is a candidate response action;

[0145] This means selecting the action that maximizes the Q value from the set of response actions A, thus obtaining the optimal response strategy.

[0146] In this embodiment, if an unauthorized entry at night is detected (an abnormal event), the optimal response action selected by the system in state s is: "lock access control + push security notification + turn on alarm lights", and the response log is recorded for subsequent model optimization and tracking audit.

[0147] Through the above implementation, the security decision module of the present invention not only has the ability to identify anomalies in the fused vector in real time, but also has the intelligent security protection capability to dynamically select multimodal strategies and self-learning response schemes according to the scenario.

[0148] In one alternative embodiment, the central server further includes a privacy protection and modality defense module, specifically comprising:

[0149] The feature desensitization processing unit is used to perform differential privacy perturbation or homomorphic encryption on video features or access control features containing personal identity information before uploading modal feature data to the edge computing node, so as to prevent the original identity features from being restored.

[0150] The dynamic gating control unit is used to dynamically generate gating coefficients based on the confidence level and scenario risk of each modality feature before the joint representation vector is generated, and to perform preemptive suppression on modality features with redundancy higher than the threshold or confidence lower than the threshold.

[0151] The anti-interference detection and trust reconstruction unit is used to determine the modality credibility change based on the contribution influence factor of multimodal input to output after the joint representation vector is generated, identify potential modality attack risks by combining the intermodal consistency index, and adjust the fusion contribution of the modality before the security decision module responds.

[0152] Preferably, the system of the present invention further includes a federated collaborative learning unit, which is used to establish encrypted communication links with multiple security protection systems deployed in different parks, periodically exchange anonymized model parameters or statistical summary information, and collaboratively optimize the neural network model parameters in the security decision module without transmitting the original modal data.

[0153] like Figure 4 As shown, another embodiment of the present invention provides a smart park security protection method based on intelligent video analysis and multimodal integration, applicable to the smart park security protection system based on intelligent video analysis and multimodal integration as described above, including the following:

[0154] Video, audio, and access information are collected by front-end sensing devices deployed throughout the park;

[0155] At the edge computing node, features are extracted from video, audio, and access information respectively, and structured video features, audio features, and access control features are output.

[0156] Differential privacy perturbation or homomorphic encryption is applied to data containing identity information in video or access control features to prevent unauthorized data leakage;

[0157] In the central server, based on multi-view target trajectory reconstruction, audio localization and time synchronization technologies, a unified spatiotemporal mapping relationship is constructed between various front-end sensing devices to perform unified temporal and spatial alignment of video features, audio features and access control features;

[0158] The aligned modal features are mapped to a unified fusion subspace. The element-wise product and summation of the intermediate vectors of the three modalities are calculated using a product fusion method to construct the fused joint representation vector.

[0159] The joint representation vector is input into the residual neural network to identify the current behavior category, and the probability that the current joint representation vector is judged as abnormal is calculated based on the distance to the center vector of normal samples. When the probability exceeds the set abnormal probability threshold, the modal robust reconstruction and defense mechanism is triggered.

[0160] Based on the current state and the type of abnormal event, the reinforcement learning model is invoked to select the optimal response action, generate an alarm signal, and trigger a security response.

[0161] In summary, this invention, through a series of innovative designs such as multimodal deep fusion, optical accelerated computing, adaptive calibration, and security protection, has achieved a comprehensive improvement in the timeliness, accuracy, energy efficiency, and security of smart park security systems, and has significant practical application value.

[0162] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A smart park security system based on intelligent video analytics and multimodal integration, comprising front-end sensing devices deployed throughout the park and back-end processing units located at the network edge and center, wherein: The front-end sensing device includes several cameras, environmental microphones, and access control terminals, respectively used to collect video, audio, and access information; the back-end processing unit includes edge computing nodes and a central server; characterized in that the edge computing node includes: The video analysis module is used to receive video streams, perform target detection, tracking and behavior recognition on image frames, and use multi-layer convolutional neural networks and Transformer self-attention mechanism to extract the location, category, trajectory and facial features of people or vehicles, and output structured video features. The audio analysis module is used to receive audio signals for spectrum analysis and anomaly identification, and to extract audio features including the type of anomaly, the time of occurrence, and the corresponding spectrum pattern. The access control identification module is used to identify personnel identity based on access information, record access time and corresponding access control terminal number, and determine whether it is abnormal access by combining the behavior identification results. If so, it outputs access control features including personnel identity, access time, access control terminal number and behavior tag. The video analysis module includes an optical neural network acceleration unit for accelerating the execution of convolution operations and attention operations in image frames, specifically including: The input modulation module is used to convert the pixel tensor of the image frame into a multi-channel light intensity signal through an electro-optic modulator based on a modulation network composed of a Mach-Zehnder interferometer array, so as to realize the parallel encoding of high-dimensional input features in the optical domain. A reconfigurable optical computing array includes multiple optical convolution kernel channels and optical self-attention channels. The optical convolution kernel channels are constructed based on tensor decomposition, using tensor columns or CP decomposition to decompose the original high-dimensional weight tensor into low-rank sub-kernels. These sub-kernels are then loaded into the optical path through a spatial light modulator and phase modulator network to perform dot product operations on the low-rank convolution kernels. The optical self-attention channels employ a multi-wavelength interference path structure, enabling parallel computation of different attention heads through wavelength division multiplexing. Each attention calculation consists of an optical waveguide matrix multiplier and a tunable microcavity array, used to perform matrix dot product and weighted sum operations in the Query-Key-Value ternary relation. The interferometry and detection module is equipped with two free spectral range paths and a balanced detector, which are used to differentially obtain the optical calculation results of positive and negative weight contributions, and realize convolution and attention operation output that support negative weights; The photoelectric conversion and sampling module is used to convert the light intensity signal of the interference output into a digital signal via a photodetector, and then send it to the subsequent neural network classifier for target classification and behavior judgment. An optical storage and multiplexing module is used to cache intermediate feature optical signals output by optical convolution kernel channels and optical self-attention channels within the reconfigurable optical computing array, and transmit the intermediate feature optical signals to downstream or parallel channels through an optical waveguide feedback structure, thereby realizing cross-layer sharing and reuse of feature information. The control and programming interface module is connected to the input modulation module, spatial light modulator, phase controller and wavelength division multiplexing path. It is used to load neural network weight configuration files according to different tasks, dynamically control the optical channel structure, tensor kernel arrangement and attention head number, and realize the structure switching and hardware reconstruction for different video processing tasks. The central server includes: The self-calibration module is used to calculate the temporal and spatial transformation parameters between cameras based on the trajectory of the same target in different video frames, and to perform spatiotemporal alignment of the video features, audio features and access control features. The multimodal fusion module is used to perform multiplication and weighting operations on the spatiotemporally aligned modal features based on the low-rank tensor fusion algorithm to construct a joint representation vector of the multimodal features; The security decision module is used to receive the joint representation vector, identify behavior categories through a neural network model with residual structure, and generate alarm signals and trigger security response strategies if abnormal events are found.

2. The smart park security system based on intelligent video analysis and multimodal integration according to claim 1, characterized in that, The audio analysis module includes: The frequency domain transformation unit is used to perform a Fast Fourier Transform (FFT) on the acquired raw audio signal to obtain a complex spectrum vector. ; The dynamic filter weight generation unit is used to generate the complex spectrum vector. The input consists of a spectral attention network with trainable parameters. Based on the amplitude features and phase distribution of the spectrum in local and global frequency ranges, frequency domain filter weights are dynamically generated. The frequency domain filter weights For frequency A corresponding set of real-valued weight vectors is used to control the spectral energy distribution of different frequency bands in the frequency dimension; for the frequency points corresponding to the noise frequency band, frequency domain filtering weights are set. The values ​​of the corresponding frequency components are made close to zero to suppress the amplitude response of that frequency point in the filtering result; for frequency points carrying characteristics of audio anomalies, frequency domain filtering weights are set. The value of the corresponding frequency component is close to 1 to preserve the energy of that frequency point, thereby achieving selective enhancement of the spectrum of the target audio event and suppression of the background noise spectrum; A spectrum filtering unit is used to filter the complex spectrum vector. With the frequency domain filter weights Performing frequency-wise element-wise multiplication yields the filtered spectrum vector. Specifically: ； Where "⊙" represents the multiplication of the corresponding frequency elements; The inverse transform unit is used to process the filtered spectrum vector. Performing the inverse fast Fourier transform (IFFT) reconstructs the filtered time-domain audio signal. ; An abnormal sound event recognition unit is used to identify the filtered time-domain audio signal. Input a pre-trained audio event classification network to identify abnormal event types. Time of occurrence and the spectral pattern of the anomaly. and will As a structured output of the audio analysis module.

3. The smart park security system based on intelligent video analysis and multimodal integration according to claim 1, characterized in that, The self-calibration module includes: The trajectory extraction and filtering unit is used to extract the temporal trajectory of the same target in multiple camera images based on the video features output by the video analysis module, and to evaluate the credibility of the temporal trajectory, remove unreliable trajectory points caused by occlusion, light interference or anti-disturbance, and obtain the filtered target trajectory. The camera parameter estimation unit is used to calculate the relative spatial pose transformation parameters between different cameras based on the filtered target trajectory through multi-view reprojection consistency constraints, including the rotation matrix R and the displacement vector T, and to calculate the time synchronization deviation Δt between cameras by combining trajectory timestamp matching and trajectory frame difference minimization strategies. The low visibility compensation unit is used to compensate for the calibration interruption caused by the lack of target trajectory by performing pose interpolation based on historical trajectory prediction, scene geometric edge features, or inertial measurement unit (IMU) data when any camera has a blind spot or insufficient lighting to identify the target for a short period of time. The audio auxiliary calibration unit is used to calculate the three-dimensional spatial positioning coordinates of the abnormal sound source based on the microphone array and the Time Difference of Origin (TDOA) algorithm when the audio analysis module detects an abnormal sound source with spatial positioning characteristics. The three-dimensional spatial positioning coordinates are then used as auxiliary constraints in the camera pose solving process. The calibration mapping construction unit is used to construct a unified spatiotemporal transformation relationship between cameras based on the rotation matrix R, displacement vector T, time synchronization deviation Δt, and the three-dimensional spatial positioning coordinates of the abnormal sound source, and form a global spatiotemporal mapping function for multimodal feature normalization processing. The multimodal feature alignment unit is used to perform a unified transformation of the time and space dimensions on the video features, audio features and access control features respectively according to the global spatiotemporal mapping function.

4. The smart park security protection system based on intelligent video analysis and multimodal integration according to claim 3, characterized in that, The scoring function for the reliability assessment of the time-series trajectory by the trajectory extraction and filtering unit is expressed as follows: ； In the formula, This is the score; Indicates the first Target candidate boxes detected in the frame; The overlap rate with the predicted bounding box of the previous frame; To test the confidence level; Indicates the degree of abrupt change in trajectory direction; , , These are weighting coefficients; When the rating value When the score falls below the set threshold, the trajectory point is marked as unreliable and removed.

5. The smart park security system based on intelligent video analysis and multimodal integration according to claim 1, characterized in that, The multimodal fusion module includes: Modality mapping unit, used to align spatiotemporally video features Audio features Access control features Mapping to the unified fusion subspace, we get: ; ; ; in, This indicates that a video feature is a combination of... A column vector consisting of real numbers, An audio feature is a combination of... A column vector consisting of real numbers, The access control feature is a combination of... A column vector consisting of real numbers; , , Video features, audio features, and access control features are mapped to a unified fusion subspace, respectively. The intermediate representation vector is shown below; , , These represent the linear mapping matrices for the corresponding modes; For the fusion order; The fusion rank control unit is used to dynamically adjust the fusion rank based on the complexity index of the current input features. The value of the complexity index includes the number of targets, modal information entropy and historical anomaly frequency, which are used to control the representation capacity of the fusion structure and achieve a dynamic trade-off between fusion accuracy and computational load. The product fusion unit is used to calculate the intermediate representation vectors of each modality. , , Element-wise multiplication and summation over the fusion dimension are performed to construct a joint tensor. and the final joint representation vector The calculation method is as follows: ； In the formula, Indicates the first The result of the trimodal product in the fusion dimension; The modal completion unit is used to estimate the alternative representation vector of the missing modality based on the acquired modal features and the modal co-occurrence distribution in similar historical scenarios when any modal input is detected to be missing or abnormal. The original input is then replaced to ensure the continuity of the fusion computation.

6. The smart park security system based on intelligent video analysis and multimodal integration according to claim 1, characterized in that, The security decision module includes: The anomaly detection and behavior recognition unit is used to receive the joint representation vector. The residual structure neural network model is used to identify behavior categories and assess the degree of anomalousness of those categories. The formula is as follows: ; In the formula, The center vector of the normal behavior sample is obtained in advance through statistical analysis of normal data. Joint representation vector With the center vector The squared Euclidean distance represents the degree of deviation between the current sample and the normal sample; This represents the probability that the current joint representation vector is judged as an anomaly. For the Sigmoid function; This is an amplification factor for abnormal scores; This is the threshold for anomaly detection; Modal robust reconstruction and defense mechanisms are used to detect probabilistic... When the probability exceeds the anomaly threshold, a modal defense mechanism is automatically activated based on a modal robustness factor. The contributions of each modality fusion are readjusted, and the modality replacement function is invoked. Activate redundant modes for structural compensation; The response strategy optimization unit is used to optimize the response strategy based on the current state. With response action set Call the reinforcement learning model to calculate the response score And select the optimal action to generate a response strategy: ； In the formula, Indicates the system in state The optimal response action to be taken; Candidate response actions; Indicates the set of response actions The optimal response strategy is obtained by selecting the action that maximizes the Q value.

7. The smart park security system based on intelligent video analysis and multimodal integration according to claim 1, characterized in that, The central server also includes a privacy protection and modality defense module, specifically including: The feature desensitization processing unit is used to perform differential privacy perturbation or homomorphic encryption on video features or access control features containing personal identity information before uploading modal feature data to the edge computing node, so as to prevent the original identity features from being restored. The dynamic gating control unit is used to dynamically generate gating coefficients based on the confidence level and scenario risk of each modality feature before the joint representation vector is generated, and to perform preemptive suppression on modality features with redundancy higher than the threshold or confidence lower than the threshold. The anti-interference detection and trust reconstruction unit is used to determine the modality credibility change based on the contribution influence factor of multimodal input to output after the joint representation vector is generated, identify potential modality attack risks by combining the intermodal consistency index, and adjust the fusion contribution of the modality before the security decision module responds.

8. The smart park security system based on intelligent video analysis and multimodal integration according to claim 1, characterized in that, The system also includes a federated collaborative learning unit, which is used to establish encrypted communication links with multiple security protection systems deployed in different parks, periodically exchange anonymized model parameters or statistical summary information, and collaboratively optimize the neural network model parameters in the security decision module without transmitting the original modal data.

9. A smart park security protection method based on intelligent video analysis and multimodal integration, applicable to the smart park security protection system based on intelligent video analysis and multimodal integration as described in any one of claims 1-8, characterized in that, The method includes: Video, audio, and access information are collected by front-end sensing devices deployed throughout the park; At the edge computing node, features are extracted from the video, audio, and access information respectively, and structured video features, audio features, and access control features are output. Differential privacy perturbation or homomorphic encryption is applied to the video features or access control features containing identity information to prevent unauthorized data leakage; In the central server, based on multi-view target trajectory reconstruction, audio localization and time synchronization technologies, a unified spatiotemporal mapping relationship is constructed between various front-end sensing devices to perform unified temporal and spatial alignment of video features, audio features and access control features; The aligned modal features are mapped to a unified fusion subspace. The element-wise product and summation of the intermediate vectors of the three modalities are calculated using a product fusion method to construct the fused joint representation vector. The joint representation vector is input into the residual neural network to identify the current behavior category, and the probability that the current joint representation vector is judged as abnormal is calculated based on the distance to the center vector of normal samples. When the probability exceeds the set abnormal probability threshold, the modal robust reconstruction and defense mechanism is triggered. Based on the current state and the type of abnormal event, the reinforcement learning model is invoked to select the optimal response action, generate an alarm signal, and trigger the security response strategy.

Images