Intelligent security and protection monitoring method, system and equipment and storage medium

Abstract

The invention discloses an intelligent security and protection monitoring method, system and device and a storage medium, and the method comprises the steps: carrying out the cross-modal fusion analysis of a roughly extracted first visual feature and first radar feature of a target, and generating a dynamic fuzzy intensity coefficient based on an abnormal behavior result after the fusion analysis, so as to dynamically adjust the fuzzy intensity of a target face region; according to the change of the target dynamic fuzzy intensity coefficient, judging whether a high-risk early warning signal is output or not; if the high-risk early warning signal is output, cross-modal deep fusion analysis is carried out on the extracted fine second visual feature and second radar feature of the target, and a target abnormal behavior judgment result is output based on the features after deep fusion; and tracking the target based on the judgment result, associating a visual detection target with a radar tracking target when the target is visually shielded, predicting the future motion trajectory of the shielded target, and recovering visual tracking when the shielding is removed. Accurate detection of abnormal behaviors is realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an intelligent security monitoring method, system, device and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence and the Internet of Things (IoT) technologies, intelligent security systems have been widely applied in retail, accommodation, and public safety sectors. Traditional security systems primarily rely on cameras deployed in key areas, using computer vision technology to analyze video streams to detect abnormal behavior such as theft, trespassing, and falls. However, systems based on a single visual modality have many limitations in complex real-world scenarios, such as susceptibility to changes in lighting and visual obstruction, and continuous high-definition recording raises increasingly serious privacy risks. To improve system robustness, multimodal sensing technology has been introduced into the security field.

[0003] The shortcomings of existing technologies are as follows: 1. They mostly employ traditional data fusion strategies such as weighted fusion and Kalman filtering, failing to achieve deep cross-modal semantic association and feature interaction. When the target is partially or completely occluded (e.g., a customer is obscured by a shelf, or a guest is obscured by a door), visual information is lost, and the system loses target tracking or causes anomaly detection to be interrupted due to the lack of an effective cross-modal association mechanism. 2. In pursuit of detection accuracy, high-definition video stream analysis is usually maintained throughout the process, indiscriminately recording and processing sensitive personal information, including faces and clothing.

[0004] Therefore, how to achieve accurate, continuous, and efficient intelligent security monitoring of abnormal behavior in complex and obscured scenarios without infringing on the privacy of ordinary individuals is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] The main objective of this invention is to provide an intelligent security monitoring method, system, device, and storage medium that achieves accurate detection and all-time tracking of abnormal behavior while ensuring privacy compliance. Furthermore, through intelligent resource scheduling and system self-optimization mechanisms, it constructs an intelligent security strategy that combines high efficiency, adaptability, and evolvability.

[0006] In a first aspect, this application provides an intelligent security monitoring method, wherein the method includes the following steps: The first visual features and first radar features of the target are roughly extracted, cross-modal fusion analysis is performed, and dynamic blur intensity coefficients are generated based on the abnormal behavior results after fusion analysis to dynamically adjust the blur intensity of the target face region. Based on the change in the dynamic fuzzy intensity coefficient of the target, determine whether to output a high-risk warning signal; If a high-risk warning signal is output, the refined second visual features and second radar features of the target will be extracted and subjected to cross-modal deep fusion analysis. Based on the deep fusion features, the abnormal behavior judgment result of the target will be output. Based on the determination result, the target is tracked, and when the target is visually occluded, the visually detected target is associated with the radar-tracked target, and the future trajectory of the occluded target is predicted. Visual tracking is resumed when the occlusion is removed.

[0007] In conjunction with the first aspect mentioned above, as an optional implementation method, video streams from high-definition cameras and point cloud data from millimeter-wave radar can be acquired simultaneously through timestamp alignment. The video stream is used to detect and locate faces using a lightweight visual Transformer model, and the AlphaPose model is used to obtain the first visual features of the target. The first visual features include: the coordinates of human key points and the distance between key points. Extract a rough first radar feature of the target from the radar data, the first radar feature including: the target motion spectrum; By calculating cosine similarity, visual pose features and radar micro-motion features are cross-modal aligned and weighted fused to generate a unified multimodal feature tensor, which is the behavior risk index of the current frame; The multimodal feature tensor is input into a privacy decision network for behavioral risk analysis, and a time series score representing the cumulative confidence of abnormal behavior is output. The time series score is passed through the Sigmoid activation function output from the privacy decision network to generate dynamic fuzzy intensity coefficients; The radius of the Gaussian blur kernel is adjusted based on the dynamic blur intensity coefficient to sharpen the target face region.

[0008] In conjunction with the first aspect mentioned above, as an optional implementation, multimodal data within a time window centered on the trigger moment is loaded from a circular data buffer pool. The multimodal data includes: human body key point coordinates, radar point cloud and human motion spectrum, and sharpened video frames. The sharpened video frames are input into the SlowFast network to extract detailed target second visual features, and output a visual feature vector representing the spatiotemporal behavior pattern. The second visual features include: spatial semantic features and subtle motion change features. The millimeter-wave radar point cloud and micro-motion spectrum are input to a timing encoder to extract fine target second radar features and output radar timing feature vectors. The second radar features include: sudden increase in the vibration frequency of human joint points and duration features. A heterogeneous graph network is constructed, with visual key points and radar tracking points as nodes and spatial distance and motion consistency as edge weights. The visual feature vector and radar temporal feature vector are fused through a message passing mechanism to output cross-modal fusion features. The cross-modal fusion features are input into the classifier to obtain the initial behavior category probabilities; The initial probability distribution is weighted by multimodal evidence strength using a pre-trained evidence importance weight matrix; Based on the visual feature vector of the spatiotemporal behavior pattern output by the SlowFast network, a temporal confidence correction factor is calculated to correct the continuity of action timing. By combining the time-series confidence correction factor and the evidence strength weighting result, the final confidence level of the target abnormal behavior is calculated and output using the Sigmoid function, thus completing the determination of the abnormal behavior.

[0009] In conjunction with the first aspect mentioned above, as an optional implementation method, if the target behavior is determined to be abnormal based on the judgment result, the target is tracked, and at the moment the target is occluded, the position and velocity of the target before visual loss are matched with the nearest neighbor of the point cloud cluster detected by the radar using spatiotemporal consistency, so as to establish the identity association between the visual target and the radar-tracked target. After establishing the identity association, the radar trajectory is initialized to output the initial state vector of the target in the radar coordinate system. The state vector includes: position, velocity, acceleration and direction. Based on the point cloud data provided by the radar, the initial state vector is updated in real time to obtain the updated state vector; Based on the state vector and the real-time radial distance, azimuth angle and Doppler velocity measured by the radar, the future trajectory of the occluded target is predicted using an extended Kalman filter model. When the distance between the target and the edge of the occlusion is less than a set threshold and the direction of movement is pointing towards the visual monitoring area, the occlusion detection is lifted, and the appearance features of the appearing target are compared with those of the target recorded before the occlusion using a visual re-identification model. If the similarity of the two features exceeds the threshold, they are identified as the same target and visual tracking is resumed.

[0010] In conjunction with the first aspect mentioned above, as an optional implementation, if the dynamic fuzzy intensity coefficient remains below a set threshold for more than a preset duration, a high-risk warning signal is generated and output.

[0011] In conjunction with the first aspect mentioned above, as an optional implementation method, the acquired state vector is input into the PPO agent that has been trained offline. The state vector includes: the number of active targets, the probability of abnormal events, the proportion of occluded targets, the processor utilization rate, and the ambient light intensity. The intelligent agent outputs a resource scheduling action vector, which includes: a visual processing resolution scaling factor, a pose estimation model complexity level, and a radar sampling frequency. The allocation of computing resources is dynamically adjusted based on the resource scheduling action vector.

[0012] In conjunction with the first aspect mentioned above, as an optional implementation method, a Bayesian network is used to fuse the target's behavioral confidence, behavioral anomaly during occlusion, surrounding environment, and historical behavioral patterns to comprehensively assess the probability of target behavioral risk. The overall risk probability is compared with multiple preset thresholds, and different levels of alarm response actions are triggered based on the comparison results.

[0013] Secondly, this application provides an intelligent security monitoring system, which includes: The dynamic privacy control module is used to extract the first visual features and first radar features of the target, perform cross-modal fusion analysis, and generate a dynamic blur intensity coefficient based on the abnormal behavior results after fusion analysis, so as to dynamically adjust the blur intensity of the target face region. The judgment module is used to determine whether to output a high-risk warning signal based on the change in the dynamic fuzziness intensity coefficient of the target. The cross-modal anomaly detection module is used to extract the refined second visual features and second radar features of the target for cross-modal deep fusion analysis if a high-risk warning signal is output, and output the target abnormal behavior judgment result based on the deep fusion features. The occlusion processing module is used to track the target based on the determination result, and when the target is visually occluded, it associates the visually detected target with the radar-tracked target, predicts the future trajectory of the occluded target, and resumes visual tracking when the occlusion is removed.

[0014] Thirdly, this application also provides an electronic device, the electronic device comprising: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the method described in any one of the first aspects.

[0015] Fourthly, this application also provides a computer-readable storage medium storing computer program instructions that, when executed by a computer, cause the computer to perform the method described in any of the first aspects.

[0016] This application provides an intelligent security monitoring method, system, device, and storage medium. The method includes the following steps: extracting coarse first visual features and first radar features of the target, performing cross-modal fusion analysis, and generating a dynamic blur intensity coefficient based on the abnormal behavior results after fusion analysis to dynamically adjust the blur intensity of the target's face region; determining whether to output a high-risk warning signal based on the change in the target's dynamic blur intensity coefficient; if a high-risk warning signal is output, extracting refined second visual features and second radar features of the target, performing cross-modal deep fusion analysis, and outputting a target abnormal behavior judgment result based on the deep fusion features; tracking the target based on the judgment result, and when visual occlusion occurs, associating the visually detected target with the radar-tracked target, predicting the future trajectory of the occluded target, and resuming visual tracking when the occlusion is removed. This application achieves accurate detection and all-time tracking of abnormal behavior while ensuring privacy compliance, and constructs an intelligent security strategy with high efficiency, adaptability, and evolvability through intelligent resource scheduling and system self-optimization mechanisms.

[0017] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit the invention. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0019] Figure 1 This is a flowchart of an intelligent security monitoring method provided in the embodiments of this application; Figure 2 This is a schematic diagram of an intelligent security monitoring system provided in the embodiments of this application; Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of this application; Figure 4 This is a schematic diagram of a computer-readable program medium provided in an embodiment of this application. Detailed Implementation

[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0021] The embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0022] Reference Figure 1 , Figure 1 The diagram shown is a flowchart of an intelligent security monitoring method provided by the present invention. Figure 1 As shown, the method includes the following steps: Step S101: Extract the rough first visual features and first radar features of the target, perform cross-modal fusion analysis, and generate a dynamic blur intensity coefficient based on the abnormal behavior results after fusion analysis to dynamically adjust the blur intensity of the target face region.

[0023] Specifically, by aligning timestamps, video streams from high-definition cameras and point cloud data from millimeter-wave radar are collected simultaneously. The video stream is used to detect and locate faces using a lightweight visual Transformer model, and the AlphaPose model is used to obtain the first visual features of the target. The first visual features include: the coordinates of human key points and the distance between key points. Extract a rough first radar feature of the target from the radar data, the first radar feature including: the target motion spectrum; By calculating cosine similarity, visual pose features and radar micro-motion features are cross-modal aligned and weighted fused to generate a unified multimodal feature tensor, which is the behavior risk index of the current frame; The multimodal feature tensor is input into a privacy decision network for behavioral risk analysis, and a time series score representing the cumulative confidence of abnormal behavior is output. The time series score is passed through the Sigmoid activation function output from the privacy decision network to generate dynamic fuzzy intensity coefficients; The radius of the Gaussian blur kernel is adjusted based on the dynamic blur intensity coefficient to sharpen the target face region.

[0024] To fully demonstrate the collaborative work of each module, let's take a supermarket security scenario as an example. This scenario involves a customer (Customer A) engaging in unusual behavior (theft) while shopping in the supermarket, triggering a full-process system response. Scenario details are as follows: Location: Large retail supermarket with densely packed shelves and visual obstructions (such as blind spots between shelves).

[0025] Character: Customer A is shopping normally, but then tries to hide a box of expensive chocolates in his coat pocket (theft).

[0026] System Deployment: A multimodal sensor network is deployed within the supermarket, including high-definition cameras (covering the main aisles and shelf areas) and millimeter-wave radar equipment (installed on the ceiling to detect movement through the shelves).

[0027] Module Interaction: Scene triggers the S201 Dynamic Privacy Control Module (handles face blurring), the S203 Cross-Modal Anomaly Detection Module (identifies theft behavior), and the S204 Occlusion Robust Handling Module (handles target tracking under shelf obstruction). All examples are based on this unified scenario to ensure seamless input and output.

[0028] First, the specific implementation method for handling face blurring is as follows: S1011, Synchronous Acquisition of Video Stream and Radar Data High-definition cameras continuously capture video streams (1080p resolution, 30fps) of the supermarket's public areas, while millimeter-wave radar collects point cloud data at a 60Hz sampling rate, including target distance, azimuth, and micro-motion features (such as hand vibration frequency). It should be noted that this step synchronizes the acquisition of the high-definition camera's video stream and the millimeter-wave radar's point cloud data using timestamp alignment.

[0029] For example, in a supermarket scenario, a camera focuses on customer A's shopping path, while radar synchronously tracks their movement. To ensure data synchronization, each video frame and radar point cloud data packet is timestamped to millisecond precision and aligned via Network Time Protocol (NTP). This step outputs a time-synchronized multimodal data stream, where the video stream contains customer A's facial area, and the radar data contains their motion vector (velocity 0.8 m / s, direction towards the snack shelf).

[0030] S1012. Real-time face detection and localization based on the ViT-S model A lightweight visual Transformer model (ViT-S) is used as the input to a synchronous video stream. This model, after distillation and compression, requires only 4.2 GFLOPS of computing power to achieve frame-by-frame face detection. ViT-S analyzes image patches through a self-attention mechanism, outputting the face bounding box coordinates (x_min, y_min, x_max, y_max) and a segmentation mask. It should be noted that this step extracts and locates the face, facilitating subsequent extraction of primary visual features.

[0031] For example, in the supermarket example, when customer A is facing the camera, ViT-S accurately detects their face, generates a bounding box [120, 80, 180, 140] (pixel coordinates), and labels it with a confidence score of 0.98. Simultaneously, the model processes multiple face scenes in parallel (up to 32 targets / frame) to avoid missed detections. This step outputs high-precision face region information, providing input for subsequent blurring.

[0032] S1013, Micro-motion Feature Extraction from Millimeter-Wave Radar In parallel, the millimeter-wave radar data stream is processed by a point cloud clustering algorithm to extract the micro-motion features of customer A. The radar signal processing unit calculates the Doppler frequency shift, generates the target motion spectrum, and focuses on analyzing the vibration frequency (range 0-10Hz) of the hand area. It should be noted that this step is a rough extraction of the first radar features. S101 only needs to quickly capture data to determine who is suspicious; the main analysis of abnormal behavior is performed in S102.

[0033] In a supermarket scenario, customer A's hand swings naturally while shopping normally, and the radar detects a low-frequency vibration (1.5Hz). However, when their hand approaches their pocket to conceal items, the vibration frequency suddenly increases to 5.8Hz. This step outputs a temporal micro-motion feature vector, including physical quantities such as frequency peaks and acceleration changes.

[0034] S1014, Visual Feature Analysis The video stream is simultaneously input into the improved AlphaPose model to extract the coordinates of key skeletal points (such as hand and hip positions) of customer A. The model outputs the two-dimensional coordinates of 17 key points based on a convolutional neural network and calculates the distance between joints (such as hand-pocket distance). It should be noted that this step is also a rough extraction of first visual features.

[0035] In the example, when customer A's hand approaches the coat pocket, AlphaPose detects that the hand-hip distance decreases from the normal value of more than 50 pixels to 15 pixels and marks the pose as abnormal with a confidence level of 0.75. This step outputs a visual feature vector, which, together with the radar micro-motion features, constitutes a chain of evidence for abnormal behavior.

[0036] S1015, Multimodal Feature Alignment and Fusion The module aligns the radar features of S1013 with the visual features of S1014. This module calculates cross-modal consistency (e.g., spatiotemporal matching between radar hand vibration frequency and visual hand movement trajectory) using cosine similarity, generating a unified multimodal feature tensor.

[0037] In a supermarket scenario, when customer A's hand gesture is partially obscured by a shelf, the radar provides vibration data (5.8Hz), while the visual system's confidence level decreases due to the obstruction. The alignment module weighted and fused the two data points to ensure feature reliability. The output is a fused multimodal feature tensor with 256 dimensions, representing the behavioral risk index of the current frame.

[0038] S1016 Privacy Decision Network (PDN) Temporal Modeling A privacy-preserving decision network (PDN) is constructed by fusing feature tensors as inputs. This network outputs temporal scores representing the cumulative confidence of anomalous behavior based on gated recurrent units (GRUs). The GRU network learns cross-frame temporal dependencies, such as analyzing the persistence of hand vibration frequencies (theft often lasts for multiple frames).

[0039] In the example, when customer A's hand vibration frequency exceeds 5Hz for 5 consecutive frames (approximately 166ms) and the distance between their hand and pocket is less than 20 pixels, the GRU hidden state is updated, and a high-risk probability is output. This step outputs a temporal anomaly score, ranging from 0 to 1, representing the cumulative confidence level of the anomalous behavior.

[0040] Dynamic calculation of S1017 fuzzy intensity coefficient α The temporal score is processed by the Sigmoid activation function output from the privacy decision network to generate a dynamic blur intensity coefficient α (range 0-1, 0 representing sharpness, 1 representing full blur). The formula for calculating α is α = 1 - σ(w) h_t + b), where h_t is the final hidden state of GRU, and w and b are the training parameters.

[0041] During normal shopping in the supermarket, α remains at 0.95 (high-intensity blur); when signs of theft are detected (such as a rapid hand concealment movement), α decays linearly. In the example, α drops from 0.95 to 0.1 within 300ms after the theft is triggered. This step outputs the real-time α value, updated every frame.

[0042] S1018 performs face blurring based on α. The α value drives the Gaussian blur kernel to process the face region. The blur kernel radius r is linearly related to α (r = α). (15 pixels). When α=0.95, face blurring with a radius of 14.25 pixels is applied to the video stream; when α drops to 0.1, the blur is almost completely removed.

[0043] In the supermarket example, customer A's face remained blurred during normal shopping; after an anomaly was triggered, a clear facial image was recorded for source tracing. This step outputs the processed video stream, in which the facial privacy status is dynamically adjusted.

[0044] Step S102: Determine whether to output a high-risk warning signal based on the change in the target's dynamic fuzziness intensity coefficient.

[0045] Specifically, if the dynamic fuzzy intensity coefficient remains below a set threshold for more than a preset duration, a high-risk warning signal is generated and output.

[0046] If the blur intensity coefficient α remains below the threshold (e.g., α < 0.3) for a certain duration (e.g., 5 frames, approximately 166ms), the S201 module determines that the current scene poses a potentially high risk. At this point, the module does not directly issue an alarm, but instead generates a high-risk warning signal, which, along with the clarified video frame, timestamp, and triggering cause (e.g., persistent abnormal high-frequency hand vibration), is output to the S202 cross-modal anomaly detection module. The core function of this signal is to activate the high-precision verification process of the S202 module, notifying S202 that the current target behavior has a high probability of being abnormal and requesting immediate initiation of a multimodal evidence chain for final determination. Simultaneously, this decision log is recorded for system tracing and model optimization.

[0047] Step S103: If a high-risk warning signal is output, the refined second visual features and second radar features of the target will be extracted and subjected to cross-modal deep fusion analysis. Based on the deep fusion features, the abnormal behavior judgment result of the target will be output.

[0048] Specifically, multimodal data within a time window centered on the trigger moment is loaded from a circular data buffer pool. The multimodal data includes: coordinates of key human body points, radar point clouds and human motion spectrum, and sharpened video frames. The sharpened video frames are input into the SlowFast network to extract detailed target second visual features, and output a visual feature vector representing the spatiotemporal behavior pattern. The second visual features include: spatial semantic features and subtle motion change features. The millimeter-wave radar point cloud and micro-motion spectrum are input to a timing encoder to extract fine target second radar features and output radar timing feature vectors. The second radar features include: sudden increase in the vibration frequency of human joint points and duration features. A heterogeneous graph network is constructed, with visual key points and radar tracking points as nodes and spatial distance and motion consistency as edge weights. The visual feature vector and radar temporal feature vector are fused through a message passing mechanism to output cross-modal fusion features. The cross-modal fusion features are input into the classifier to obtain the initial behavior category probabilities; The initial probability distribution is weighted by multimodal evidence strength using a pre-trained evidence importance weight matrix; Based on the visual feature vector of the spatiotemporal behavior pattern output by the SlowFast network, a temporal confidence correction factor is calculated to correct the continuity of action timing. By combining the time-series confidence correction factor and the evidence strength weighting result, the final confidence level of the target abnormal behavior is calculated and output using the Sigmoid function, thus completing the determination of the abnormal behavior.

[0049] For ease of understanding, please provide an example: As the core decision-making unit of the system, the S102 cross-modal anomaly detection module is responsible for the accurate verification and classification of high-risk events marked in step S101. Based on a unified supermarket scenario (customer A suspected of stealing chocolate), the S102 module receives the clarified video stream, preliminary behavioral features, and trigger signals output by the S101 module. Through deep multimodal fusion and spatiotemporal analysis, it ultimately confirms the type and confidence level of the abnormal behavior. The S102 design follows the principles of functional specialization and data reuse: to avoid redundant calculations, the S102 module directly reuses the unified pose estimation results (AlphaPose keypoints) already generated by the S101 module; simultaneously, to improve the accuracy of behavior recognition, a SlowFast network is introduced to model temporal behavior, and a graph neural network is used to achieve semantic-level fusion of visual and radar data.

[0050] S1031, High-risk event triggering and data buffer pool loading When the privacy decision network (PDN) of S101 outputs a consistently low ambiguity coefficient (α < 0.3 for more than 5 frames), S102 is activated. The system first loads time window data centered on the trigger moment (e.g., 2 seconds before and after, totaling 60 frames @ 30fps) from the circular data buffer. This data includes: the unified human keypoint sequence processed by S101 (coordinates of 17 keypoints output by AlphaPose), synchronized millimeter-wave radar point cloud and micro-motion spectrum, and sharpened video frames.

[0051] In a supermarket setting, this step ensures that the complete context of customer A's journey from normal shopping to abnormal hand movements is captured, providing a temporal basis for in-depth analysis.

[0052] S1032. Spatiotemporal behavioral feature extraction based on SlowFast network Loaded video frame segments are input into the SlowFast network, which employs a dual-path architecture: the Slow path analyzes spatial semantic features (such as the interaction posture of the hand with the product) at a low frame rate (e.g., 15fps), while the Fast path captures subtle motion changes (such as the concealment action of a rapid wrist flip) at a high frame rate (e.g., 30fps). The network outputs a high-dimensional feature vector that represents the spatiotemporal pattern of the behavior.

[0053] In the example, the SlowFast network successfully identified the continuous motion of customer A's hand as it picked up an item from the shelf and quickly hid it in his coat pocket, distinguishing it from the normal action pattern of putting the item into a shopping basket.

[0054] S1033, Timing Coding of Radar Micromotion Characteristics In parallel, the radar data stream passes through a temporal encoder, which converts the point cloud clustering results and micro-motion spectra (such as the hand vibration frequency of 5.8 Hz) into temporal feature vectors. The encoder focuses on capturing dynamic changes in features, such as sudden increases in vibration frequency and their duration, which are highly correlated with visual theft actions. This step outputs radar temporal features that complement the SlowFast visual features.

[0055] S1034, Construction and Feature Alignment of Cross-Modal Graph Neural Networks A heterogeneous graph neural network (GNN) is constructed, with visual keypoints and radar tracking points as nodes and spatial distance and motion consistency as edge weights. The fine visual features of S1022 and the fine radar features of S1023 are fused through a message passing mechanism to output cross-modal fused features.

[0056] It should be noted that the nodes in the graph include visual keypoints (such as the right wrist and left wrist) and radar tracking points. Edge weights are dynamically calculated based on spatial distance (such as the distance between the hand keypoint and the radar point cloud) and motion consistency (such as the angle between velocity vectors). GNN achieves feature alignment through a message passing mechanism: for example, it associates the semantic meaning of "hand near pocket" in vision with the physical signal of "high-frequency hand vibration" in radar to the same graph node. This step solves the feature fragmentation problem in occluded scenes.

[0057] S1035. Classification of Abnormal Behavior and Confidence Calculation The fused cross-modal features are input into a multi-classifier, which outputs the probability distributions of various abnormal behaviors (theft, falls, smoking, etc.). The classifier is trained based on cross-entropy loss and outputs Top-K candidate behaviors and their confidence scores. In the example, the system calculates a confidence score of 0.92 for "theft" and 0.05 for "normal," ultimately classifying it as theft.

[0058] In step S1035, the classification of abnormal behavior and the calculation of confidence scores are key decision-making steps. The core of this step lies in transforming the multimodal features extracted in previous steps into quantifiable behavioral judgment results. Based on a supermarket scenario (customer A stealing chocolate), this step achieves accurate classification through a multi-level probability fusion model, where the calculation of confidence scores follows strict mathematical logic and the principle of evidence weighting.

[0059] First, the cross-modal fused feature vector from step S1034 is fed into a three-layer fully connected neural network classifier. The output layer of this classifier uses the Softmax activation function to generate an initial class probability distribution P_initial, covering five categories: theft, falling, smoking, trespassing, and normal behavior. In the supermarket scenario example, assuming the classifier's original output for the "theft" category is 2.8, after Softmax transformation, P_initial(theft) is approximately 0.65. This initial probability primarily reflects the pattern matching degree based on the fused features, but it does not fully consider temporal continuity and the strength of evidence.

[0060] Subsequently, the system introduces a temporal confidence correction factor λ_temporal, which is calculated by analyzing the continuity of spatiotemporal features extracted by the SlowFast network in S1032. Specifically, the system detects the duration and completeness of abnormal actions: when customer A's "hand concealment" action lasts for more than 10 frames (approximately 330ms) and contains a complete "reaching out-touching the item-concealing" action chain, λ_temporal is set to 1.2; if the action is interrupted or incomplete, the value is proportionally reduced to below 0.8. Simultaneously, the consistency of radar micro-motion features also affects the correction factor; when the hand vibration frequency (5.8Hz) remains stable during the action, λ_temporal can be further increased to 1.3. In the example, customer A's continuous theft actions resulted in λ_temporal reaching 1.25.

[0061] More importantly, the system employs an evidence strength weighting mechanism, using a learnable importance weight matrix W_evidence to differentiate the weights of multimodal evidence. This weight matrix automatically learns the reliability of different modalities of evidence during training: for example, the clarity weight of the "hand-product" interaction in visual evidence is 0.6, while the weight of micro-motion signals in a specific frequency band (5-8Hz) in radar evidence is 0.4. In a supermarket scenario, when the camera clearly captures the interaction between the hand and the chocolate, and the radar detects a stable concealed vibration pattern, both types of evidence exhibit high intensity, maximizing the weighted product. The final confidence level P_final is calculated as: P_final = σ(λ_temporal × W_evidence × P_initial + b), where σ is the Sigmoid function and b is the bias term. After this series of calculations, the final confidence level of customer A's theft behavior increased from the initial 0.65 to 0.92, significantly exceeding the preset alarm threshold of 0.85.

[0062] The innovation of this confidence calculation mechanism lies in its multi-level fusion strategy: it not only considers static feature matching, but also introduces dynamic time-series correction and evidence reliability weighting to effectively distinguish between real anomalies and transient false alarms (such as a customer organizing clothes). At the same time, all intermediate calculation results are recorded in the system log, providing data support for the visual traceability of step S1036, ensuring the interpretability of classification decisions and audit trail capabilities.

[0063] S1036, Visualization and Tracing of Multimodal Evidence Chains To enhance the interpretability of the results, the module generates a gradient-weighted class activation map (Grad-CAM) that highlights the regions in the image that contribute most to the classification decision (such as the area where a hand contacts a pocket). Simultaneously, the system records the time-alignment curves between radar micro-motion spectral peaks and visual actions, forming a traceable chain of evidence. This output provides security personnel with intuitive criteria for judgment.

[0064] S1037. Detection Result Transmission and System Linkage Finally, the S201 module packages and outputs the anomaly type (theft), confidence level (0.92), timestamp, and visual evidence to the S203 occlusion processing module (for continuous tracking).

[0065] In the example, this result triggers the S203 module to continuously track customer A, regardless of whether they enter the blind spot.

[0066] Step S104: Based on the determination result, track the target, and when the target is visually occluded, associate the visually detected target with the radar-tracked target, predict the future trajectory of the occluded target, and resume visual tracking when the occlusion is removed.

[0067] Specifically, if the target behavior is determined to be abnormal based on the judgment result, the target is tracked, and at the moment the target is occluded, the position and velocity of the target before visual loss are matched with the nearest neighbor of the point cloud cluster detected by the radar using spatiotemporal consistency, so as to establish the identity association between the visual target and the radar-tracked target. After establishing the identity association, the radar trajectory is initialized to output the initial state vector of the target in the radar coordinate system. The state vector includes: position, velocity, acceleration and direction. Based on the point cloud data provided by the radar, the initial state vector is updated in real time to obtain the updated state vector; Based on the state vector and the real-time radial distance, azimuth angle and Doppler velocity measured by the radar, the future trajectory of the occluded target is predicted using an extended Kalman filter model. When the distance between the target and the edge of the occlusion is less than a set threshold and the direction of movement is pointing towards the visual monitoring area, the occlusion detection is lifted, and the appearance features of the appearing target are compared with those of the target recorded before the occlusion using a visual re-identification model. If the similarity of the two features exceeds the threshold, they are identified as the same target and visual tracking is resumed.

[0068] For ease of understanding, please provide an example: The S203 occlusion handling module is responsible for maintaining continuous tracking and behavior monitoring capabilities when the target enters a visual blind spot or is partially occluded, ensuring full-time coverage in complex environments. Based on a supermarket scenario (customer A steals chocolate and moves towards the fresh produce section, being obstructed by shelves along the way), this module solves the target loss problem of traditional single-vision systems under occlusion through deep fusion of vision and radar and predictive modeling. The module design emphasizes smooth state transitions and continuous behavior analysis, achieving robust tracking through multimodal motion pattern analysis.

[0069] S1041, Visual Occlusion Detection and Modality Switching Trigger The module continuously receives unified pose estimation results (AlphaPose keypoints) from the S201 module and abnormal behavior markers from the S202 module. When the system detects a continuous decrease in the confidence of the target keypoints (e.g., the confidence of the wrist keypoints drops from 0.95 to below 0.3 within 5 frames), or the overlap area between the target bounding box and scene occluders (e.g., shelves, stacks) exceeds 60%, a visual occlusion state is triggered.

[0070] In a supermarket scenario, when customer A's shopping cart passes the shelf, the confidence levels of the shoulder and elbow keypoints output by AlphaPose significantly decrease. At this point, the system generates an occlusion event marker and initiates a radar-driven tracking mode, while simultaneously recording the last known position, velocity vector, and appearance features (based on the output of the ViT-S feature extractor of the S201 module) before visual loss.

[0071] S1042, Radar point cloud target association and motion trajectory initialization At the instant visual occlusion occurs, the system immediately retrieves radar point cloud clusters associated with the visual target from the millimeter-wave radar data stream. The association is based on the principle of spatiotemporal consistency: the position before visual loss is matched with the 3D coordinates detected by the radar using nearest neighbor matching, requiring a spatial distance of less than 1 meter and a velocity vector angle of less than 20 degrees.

[0072] In the example, before customer A was obscured by the stack, his visual position (x=5.2m, y=3.1m) closely matched that of a point cloud cluster (x=5.3m, y=3.0m, v=0.8m / s) tracked by the radar. The system successfully established the association and initialized the radar trajectory, so as to provide a reference for subsequent prediction by outputting the target identifier and initial motion state vector in the radar coordinate system after initialization.

[0073] S1043, Short-time motion prediction based on extended Kalman filter For sparse point cloud data provided by radar, the initial motion state vector is updated in real time. The state vector includes position, velocity, acceleration, and direction. Radial range, azimuth angle, and Doppler velocity are observed in real time by radar.

[0074] Based on the real-time updated motion state vector and the real-time observed radial distance, azimuth angle and Doppler velocity, the short-term motion trajectory of the target during occlusion is predicted by the extended Kalman filter (EKF) model.

[0075] In a supermarket scenario, customer A, after being obscured by a display shelf, continues to move towards the fresh produce section at a speed of approximately 0.8 m / s. The EKF predicts the trajectory for the next 10 frames (approximately 333 ms) based on the customer's historical motion patterns and updates the state estimate for each frame. This prediction effectively compensates for the discreteness caused by the radar data sampling interval (16.7 ms), outputting smooth, continuous trajectory coordinates.

[0076] S1044, Multimodal Behavioral Feature Extraction and Abnormal Pattern Continuity Analysis During periods of visual obstruction, the system does not pause behavior analysis but instead switches to radar-driven anomaly pattern detection. This phase focuses on analyzing two types of features: macroscopic motion trajectory anomalies and microscopic motion pattern anomalies. At the macroscopic level, the spatial relationship between the radar trajectory and the electronic fence determines whether a restricted area has been entered. At the microscopic level, the system analyzes patterns such as the periodicity and sudden pauses in the target's overall movement (e.g., brief observation behavior commonly seen after theft).

[0077] In the example, customer A's trajectory paused for 2 seconds (radar speed dropped to 0.1 m / s) after being obscured by the shelf, followed by a sudden acceleration and turn. This "pause-turn" pattern was recorded as a suspicious behavioral characteristic. It is worth noting that this analysis is based on overall kinematic characteristics, making it more widely applicable.

[0078] S1045, Occlusion Removal Detection and Cross-Modal Re-identification When radar data indicates that a target is approaching the edge of an obstruction, the system preloads visual re-identification resources. The conditions for determining whether the obstruction is removed include: the predicted distance between the target and the obstruction is less than 0.5 meters, and the direction of movement is pointing towards the visible area.

[0079] In the example, when customer A is about to appear from the other side of the shelf, the system initiates a re-identification process: first, the camera prefocuses by predicting the location using radar, and then a visual re-identification model is used to compare the appearing target with the appearance features (clothing color, body shape, etc.) recorded before the occlusion. When the feature similarity exceeds 0.9, it is confirmed as the same target, and visual tracking takes over again.

[0080] S1046, Seamless Trajectory Stitching and Fusion of Behavioral Evidence Chain After successful re-identification, the system spatiotemporally aligns and stitches the radar trajectory during the occlusion period with the visual trajectory. By minimizing positional errors, the radar coordinate system is aligned to the visual coordinate system, ensuring trajectory continuity. Simultaneously, abnormal behavioral features recorded during the occlusion period (such as the pause-sharp turn pattern detected by S1044) are fused with visual behavioral evidence to improve the final anomaly confidence.

[0081] In the example, once customer A appears off the shelf, the system immediately resumes tracking him and uses the unusual "pause-turn" behavior during the obstruction as supporting evidence of suspected theft.

[0082] S1047, Occlusion Learning and Model Adaptive Update After each occlusion processing, the system records metrics such as occlusion duration, re-identification accuracy, and trajectory prediction error, which are used to optimize EKF parameters and the re-identification model offline. For example, when a certain type of occlusion (such as a tall shelf) causes a large re-identification error, the system automatically increases the radar sampling frequency in that area or adjusts the weight allocation of visual features to achieve continuous performance improvement.

[0083] In one embodiment, the system further includes a resource scheduling step during monitoring, which includes: The acquired state vector is input into the PPO agent trained offline. The state vector includes: number of active targets, probability of abnormal events, proportion of occluded targets, processor utilization and ambient light intensity. The agent outputs a resource scheduling action vector, which includes: a visual processing resolution scaling factor, a pose estimation model complexity level, and a radar sampling frequency; and dynamically adjusts the allocation of computing resources based on the resource scheduling action vector.

[0084] Specifically, the multimodal resource scheduling module, as the system's intelligent resource management center, undertakes the core responsibility of dynamically optimizing computing resource allocation and balancing system performance and energy efficiency. Based on a supermarket scenario (the entire process from customer A's normal shopping to theft), this module uses a reinforcement learning-driven adaptive strategy to monitor the load status of each module and the scenario's risk level in real time, achieving precise on-demand allocation of computing resources. The module design follows the principle of maximizing efficiency, significantly reducing the system's normal operating energy consumption while ensuring real-time anomaly detection, thus solving the resource waste problem caused by the continuous operation of high-load models.

[0085] 1. Multi-dimensional system status monitoring and feature vector construction Obtain the dynamic state vector. The state vector includes: the number of active targets processed by module S201, the probability of abnormal events in module S202 (based on the risk score output by the PDN network), the proportion of occluded targets in module S203, the overall CPU / GPU utilization of the system, the remaining battery power, and the ambient light intensity. In the initial stage of the supermarket scenario, when customer A is shopping normally, the system state vector is as follows: number of targets 1, abnormal probability 0.05, no occluded targets, GPU utilization 35%, and sufficient light intensity of 500 lux. These data are updated every 200 milliseconds, forming a feature tensor describing the real-time load of the system.

[0086] 2. Resource scheduling decision generation based on Proximity Policy Optimization (PPO) The state feature vector is input to the PPO agent, which is trained offline. This agent employs an Actor-Critic architecture and outputs resource scheduling decisions in a continuous action space through a policy network. The action space contains three core adjustable parameters: visual processing resolution (scaling factor from 0.2 to 1.0), pose estimation model complexity, and millimeter-wave radar sampling frequency (10Hz to 60Hz). In the example, when customer A is in a normal shopping state, the PPO network outputs an action vector [resolution 0.5, model complexity "lightweight", radar frequency 20Hz], corresponding to the energy-saving mode in a low-risk scenario.

[0087] 3. Multi-module collaborative load balancing The module analyzes the computational dependencies of each subtask in real time and optimizes the task scheduling sequence. When the system needs to handle high-priority tasks (such as the anomaly verification in S202) and regular tasks simultaneously, the resource scheduler adopts a preemptive allocation strategy: task S220 receives the highest priority, and the regular detection task in S201 is temporarily downgraded (e.g., skipping non-critical frame processing). During the theft by customer A, module S202 receives 90% of the GPU resources, while modules S201 and S203 share the remaining resources to ensure the real-time performance of anomaly verification. It can be understood that steps 1-3 determine which stage has the highest resource demand based on requirements, then allocate resources accordingly. After allocation, resources are specifically assigned to the corresponding stages. For example, if ABC needs support, resource D is allocated to ABC; specifically, A receives d1, B receives d2, and C receives d3, allocated on demand.

[0088] 4. Real-time trade-off optimization between energy efficiency and performance The reward function of the PPO agent is designed as a multi-objective optimization: R = w1 × detection accuracy + w2 × (1 - energy efficiency ratio) + w3 × reciprocal of response latency. Through online learning, the system gradually masters the optimal trade-off points in different scenarios. For example, after closing at night, the system maintains a low-power mode; while during peak business hours, it maintains a higher resource level to cope with emergencies.

[0089] 5. Continuous learning and updating of scheduling strategies The module records the system performance after each scheduling decision (such as anomaly detection accuracy and energy consumption changes), and continuously optimizes by fine-tuning PPO network parameters online. When the supermarket layout changes or the flow of people changes, the system can automatically adapt to the new resource demand pattern, forming a self-optimizing scheduling ecosystem. It should be explained that 4-5 is based on self-learning from each allocation pattern, so that when the same allocation method is encountered again, scheduling can be quickly carried out.

[0090] In one embodiment, after resuming visual tracking upon removal of occlusion, the process includes: By using Bayesian networks to fuse the target’s behavioral confidence, behavioral anomaly during occlusion, surrounding environment and historical behavioral patterns, the probability of target behavioral risk is comprehensively assessed. The overall risk probability is compared with multiple preset thresholds, and different levels of alarm response actions are triggered based on the comparison results.

[0091] Specifically, a dynamic Bayesian network is used to probabilistically assess the behavioral risk of customer A. Network nodes include the outputs of each module (e.g., theft confidence, abnormality of behavior during obstruction), environmental factors (e.g., crowd density, time period), and historical behavioral patterns. When new evidence emerges, the posterior probability is calculated through Bayesian updates. In the example, the initial prior probability is 0.7 for normal shopping. After S202 provides evidence of theft (likelihood ratio 8.5), the posterior probability becomes 0.88 for theft; subsequently, the evidence of "accelerated departure after obstruction" provided by S203 (likelihood ratio 2.3) further increases the probability to 0.91. This probability assessment provides a quantitative basis for subsequent decision-making.

[0092] Based on the risk assessment results, the system triggers differentiated responses according to preset multi-level thresholds. When the overall risk probability exceeds 0.85, a Level 1 alarm is triggered (recorded in the background log); when it exceeds 0.9, a Level 2 alarm is triggered (pushed to security personnel via their app); when it exceeds 0.96 and the target is near the exit, a Level 3 alarm is triggered (on-site audible and visual warning). In the example, customer A's overall risk probability eventually reached 0.91, triggering a Level 2 alarm and sending an alert message containing a trajectory map and a summary of key evidence to the security supervisor.

[0093] In one embodiment, a multi-camera collaborative tracking mechanism is activated when the target moves within the monitored area. The system uses visual re-identification technology to correlate images of customer A captured by different cameras and reconstructs their complete path throughout the store using radar trajectory data. In the example, during customer A's movement from the snack area through the fresh produce area to the exit, the system successfully correlated footage from three cameras. This step outputs a global trajectory heatmap and behavioral density analysis of customer A, revealing anomalies in their behavioral patterns.

[0094] In one embodiment, a human decision-making interface is provided, allowing security personnel to confirm or reject the automatic judgment results. When security personnel confirm the theft through on-site verification, they can send a "confirmation" signal through the system; if it is a false alarm, a "false alarm" and the reason (e.g., "employee restocking") are sent. In the example, after receiving an alarm, security personnel intercept customer A on-site and find unpaid chocolates in his coat pocket, then confirm the theft in the system. This feedback signal is recorded for model optimization.

[0095] All confirmed results (whether correctly detected or false positives) enter the incremental learning pipeline. The system periodically fine-tunes key models (such as the anomaly classifier in S202 and the PDN network in S101) using new confirmed samples, paying particular attention to boundary cases (such as difficult samples with confidence levels between 0.8 and 0.9). In the example, the theft case of customer A (a confirmed positive example) is used to strengthen the detection weights of similar hand-movement patterns.

[0096] Continuously monitor overall system performance metrics, including module processing latency, detection accuracy, and resource utilization, and generate a visual dashboard. When performance deviations are detected (such as the false alarm rate of S102 exceeding the threshold), automatic flagging requires manual review. Simultaneously, generate detailed PDF reports for each complete event (such as the theft incident involving Customer A), including a timeline, evidence screenshots, and explanations of decision-making logic, meeting auditing and judicial requirements.

[0097] In summary, this application performs real-time analysis of surveillance video streams and performs privacy blurring on faces under normal conditions. When potential abnormal behavior is detected, a high-risk warning signal is generated and output. Based on the high-risk warning signal, a high-precision analysis model is activated to perform cross-modal fusion analysis on the triggering target to verify and determine the type and confidence level of the abnormal behavior. Based on the anomaly determination result, the target is continuously tracked, and when the target is visually obstructed, the system switches to radar-dominated mode to maintain tracking continuity.

[0098] Reference Figure 2 , Figure 2 The diagram shown is a schematic of an intelligent security monitoring system provided by the present invention. Figure 2 As shown, the system includes: Dynamic privacy control module 201: It is used to extract the first visual features and first radar features of the target, perform cross-modal fusion analysis, and generate dynamic blur intensity coefficients based on the abnormal behavior results after fusion analysis, so as to dynamically adjust the blur intensity of the target face region. Based on the change in the dynamic fuzziness intensity coefficient of the target, determine whether to output a high-risk warning signal.

[0099] Cross-modal anomaly detection module 202: If a high-risk warning signal is output, it will extract the refined second visual features and second radar features of the target, perform cross-modal deep fusion analysis, and output the target abnormal behavior judgment result based on the deep fusion features.

[0100] Occlusion processing module 203: It is used to track the target based on the determination result, and when the target is visually occluded, it associates the visually detected target with the radar-tracked target, predicts the future trajectory of the occluded target, and resumes visual tracking when the occlusion is removed.

[0101] Furthermore, in one possible implementation, the dynamic privacy control module is also used to synchronously acquire video streams from high-definition cameras and point cloud data from millimeter-wave radar by means of timestamp alignment; The video stream is used to detect and locate faces using a lightweight visual Transformer model, and the AlphaPose model is used to obtain the first visual features of the target. The first visual features include: the coordinates of human key points and the distance between key points. Extract a rough first radar feature of the target from the radar data, the first radar feature including: the target motion spectrum; By calculating cosine similarity, visual pose features and radar micro-motion features are cross-modal aligned and weighted fused to generate a unified multimodal feature tensor, which is the behavior risk index of the current frame; The multimodal feature tensor is input into a privacy decision network for behavioral risk analysis, and a time series score representing the cumulative confidence of abnormal behavior is output. The time series score is passed through the Sigmoid activation function output from the privacy decision network to generate dynamic fuzzy intensity coefficients; The radius of the Gaussian blur kernel is adjusted based on the dynamic blur intensity coefficient to sharpen the target face region.

[0102] Furthermore, in one possible implementation, the cross-modal anomaly detection module is also used to load multimodal data within a time window centered on the trigger moment from a circular data buffer pool. The multimodal data includes: coordinates of human key points, radar point clouds and human motion spectrum, and sharpened video frames. The sharpened video frames are input into the SlowFast network to extract detailed target second visual features, and output a visual feature vector representing the spatiotemporal behavior pattern. The second visual features include: spatial semantic features and subtle motion change features. The millimeter-wave radar point cloud and micro-motion spectrum are input to a timing encoder to extract fine target second radar features and output radar timing feature vectors. The second radar features include: sudden increase in the vibration frequency of human joint points and duration features. A heterogeneous graph network is constructed, with visual key points and radar tracking points as nodes and spatial distance and motion consistency as edge weights. The visual feature vector and radar temporal feature vector are fused through a message passing mechanism to output cross-modal fusion features. The cross-modal fusion features are input into the classifier to obtain the initial behavior category probabilities; The initial probability distribution is weighted by multimodal evidence strength using a pre-trained evidence importance weight matrix; Based on the visual feature vector of the spatiotemporal behavior pattern output by the SlowFast network, a temporal confidence correction factor is calculated to correct the continuity of action timing. By combining the time-series confidence correction factor and the evidence strength weighting result, the final confidence level of the target abnormal behavior is calculated and output using the Sigmoid function, thus completing the determination of the abnormal behavior.

[0103] Furthermore, in one possible implementation, the occlusion processing module is also used to track the target if the target behavior is determined to be abnormal based on the judgment result, and at the moment the target is occluded, to use spatiotemporal consistency to perform nearest neighbor matching between the position and velocity of the target before visual loss and the point cloud cluster detected by the radar, so as to establish the identity association between the visual target and the radar-tracked target. After establishing the identity association, the radar trajectory is initialized to output the initial state vector of the target in the radar coordinate system. The state vector includes: position, velocity, acceleration and direction. Based on the point cloud data provided by the radar, the initial state vector is updated in real time to obtain the updated state vector; Based on the state vector and the real-time radial distance, azimuth angle and Doppler velocity measured by the radar, the future trajectory of the occluded target is predicted using an extended Kalman filter model. When the distance between the target and the edge of the occlusion is less than a set threshold and the direction of movement is pointing towards the visual monitoring area, the occlusion detection is lifted, and the appearance features of the appearing target are compared with those of the target recorded before the occlusion using a visual re-identification model. If the similarity of the two features exceeds the threshold, they are identified as the same target and visual tracking is resumed.

[0104] Furthermore, in one possible implementation, the dynamic privacy control module is also used to generate and output a high-risk warning signal if the dynamic blur intensity coefficient remains below a set threshold for a preset duration.

[0105] Furthermore, in one possible implementation, a resource allocation module is used to input the acquired state vector into an offline-trained PPO agent, the state vector including: number of active targets, probability of abnormal events, proportion of occluded targets, processor utilization, and ambient light intensity. The intelligent agent outputs a resource scheduling action vector, which includes: a visual processing resolution scaling factor, a pose estimation model complexity level, and a radar sampling frequency. The allocation of computing resources is dynamically adjusted based on the resource scheduling action vector.

[0106] Furthermore, in one possible implementation, the occlusion processing module is used to comprehensively assess the target's behavioral risk probability by fusing the target's behavioral confidence, behavioral anomaly during occlusion, surrounding environment, and historical behavioral patterns using a Bayesian network. The overall risk probability is compared with multiple preset thresholds, and different levels of alarm response actions are triggered based on the comparison results.

[0107] The following reference Figure 3 To describe an electronic device 300 according to this embodiment of the present invention. Figure 3 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0108] like Figure 3 As shown, the electronic device 300 is presented in the form of a general-purpose computing device. The components of the electronic device 300 may include, but are not limited to: at least one processing unit 310, at least one storage unit 320, and a bus 330 connecting different system components (including storage unit 320 and processing unit 310).

[0109] The storage unit stores program code that can be executed by the processing unit 310, causing the processing unit 310 to perform the steps described in the "Embodiment Methods" section of this specification according to various exemplary embodiments of the present invention.

[0110] Storage unit 320 may include readable media in the form of volatile storage units, such as random access memory (RAM) 321 and / or cache memory 322, and may further include read-only memory (ROM) 323.

[0111] Storage unit 320 may also include a program / utility 324 having a set (at least one) of program modules 325, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0112] Bus 330 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0113] Electronic device 300 can also communicate with one or more external devices (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 300, and / or any device that enables electronic device 300 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 350. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 360. As shown, network adapter 360 communicates with other modules of electronic device 300 via bus 330. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0114] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0115] According to the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the present invention can also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the present invention described in the "Exemplary Methods" section above.

[0116] refer to Figure 4 As shown, a program product 400 for implementing the above-described method according to an embodiment of the present invention is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0117] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0118] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0119] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0120] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0121] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0122] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

[0123] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

Claims

1. An intelligent security monitoring method, characterized in that, include: The first visual features and first radar features of the target are roughly extracted, cross-modal fusion analysis is performed, and dynamic blur intensity coefficients are generated based on the abnormal behavior results after fusion analysis to dynamically adjust the blur intensity of the target face region. Based on the change in the dynamic fuzzy intensity coefficient of the target, determine whether to output a high-risk warning signal; If a high-risk warning signal is output, the refined second visual features and second radar features of the target will be extracted and subjected to cross-modal deep fusion analysis. Based on the deep fusion features, the abnormal behavior judgment result of the target will be output. Based on the determination result, the target is tracked, and when the target is visually occluded, the visually detected target is associated with the radar-tracked target, and the future trajectory of the occluded target is predicted. Visual tracking is resumed when the occlusion is removed.

2. The method according to claim 1, characterized in that, The process involves extracting coarse target first visual features and first radar features, performing cross-modal fusion analysis, and generating dynamic blur intensity coefficients based on the results of the fusion analysis, including: By aligning with timestamps, video streams from high-definition cameras and point cloud data from millimeter-wave radar are collected synchronously. The video stream is used to detect and locate faces using a lightweight visual Transformer model, and the AlphaPose model is used to obtain the first visual features of the target. The first visual features include: coordinates of human key points and distances between key points. Extract a rough first radar feature of the target from the radar data, the first radar feature including: the target motion spectrum; By calculating cosine similarity, visual pose features and radar micro-motion features are cross-modal aligned and weighted fused to generate a unified multimodal feature tensor, which is the behavior risk index of the current frame; The multimodal feature tensor is input into a privacy decision network for behavioral risk analysis, and a time series score representing the cumulative confidence of abnormal behavior is output. The time series score is passed through the Sigmoid activation function output from the privacy decision network to generate dynamic fuzzy intensity coefficients; The radius of the Gaussian blur kernel is adjusted based on the dynamic blur intensity coefficient to sharpen the target face region.

3. The method according to claim 1, characterized in that, The process of extracting refined second visual features and second radar features of the target, performing cross-modal deep fusion analysis, and outputting a result determining the abnormal behavior of the target based on the deep fusion features includes: Multimodal data within a time window centered on the trigger moment is loaded from a circular data buffer pool. The multimodal data includes: coordinates of human key points, radar point cloud and human motion spectrum, and sharpened video frames. The sharpened video frames are input into the SlowFast network to extract the fine target second visual features, and output a visual feature vector representing the spatiotemporal behavior pattern. The second visual features include: spatial semantic features and subtle motion change features. The millimeter-wave radar point cloud and micro-motion spectrum are input to a timing encoder to extract fine target second radar features and output radar timing feature vectors. The second radar features include: sudden increase in vibration frequency of human joint points and duration. A heterogeneous graph network is constructed, with visual key points and radar tracking points as nodes and spatial distance and motion consistency as edge weights. The visual feature vector and radar temporal feature vector are fused through a message passing mechanism to output cross-modal fusion features. The cross-modal fusion features are input into the classifier to obtain the initial behavior category probabilities; The initial probability distribution is weighted by multimodal evidence strength using a pre-trained evidence importance weight matrix; Based on the visual feature vector of the spatiotemporal behavior pattern output by the SlowFast network, a temporal confidence correction factor is calculated to correct the continuity of action timing. By combining the time-series confidence correction factor and the evidence strength weighting result, the final confidence level of the target abnormal behavior is calculated and output using the Sigmoid function, thus completing the determination of the abnormal behavior.

4. The method according to claim 1, characterized in that, The step of tracking the target based on the determination result, associating the visually detected target with the radar-tracked target when visual occlusion occurs, predicting the future trajectory of the occluded target, and resuming visual tracking when the occlusion is removed includes: If the target behavior is determined to be abnormal based on the judgment result, the target is tracked. At the moment the target is occluded, the position and velocity of the target before visual loss are matched with the nearest neighbor of the point cloud cluster detected by the radar, so as to establish the identity association between the visual target and the radar-tracked target. After establishing the identity association, the radar trajectory is initialized to output the initial state vector of the target in the radar coordinate system. The state vector includes: position, velocity, acceleration and direction. Based on the point cloud data provided by the radar, the initial state vector is updated in real time to obtain the updated state vector; Based on the state vector and the real-time radial distance, azimuth angle and Doppler velocity measured by the radar, the future trajectory of the occluded target is predicted using an extended Kalman filter model. When the distance between the target and the edge of the occlusion is less than a set threshold and the direction of movement is pointing towards the visual monitoring area, the occlusion detection is lifted, and the appearance features of the appearing target are compared with those of the target recorded before the occlusion using a visual re-identification model. If the similarity of the two features exceeds the threshold, they are identified as the same target and visual tracking is resumed.

5. The method according to claim 1, characterized in that, The step of determining whether to output a high-risk warning signal based on changes in the target's dynamic fuzziness intensity coefficient includes: If the dynamic fuzzy intensity coefficient remains below a set threshold for more than a preset duration, a high-risk warning signal is generated and output.

6. The method according to claim 1, characterized in that, It also includes resource scheduling steps during monitoring, which include: The acquired state vector is input into the PPO agent trained offline. The state vector includes: number of active targets, probability of abnormal events, proportion of occluded targets, processor utilization and ambient light intensity. The intelligent agent outputs a resource scheduling action vector, which includes: a visual processing resolution scaling factor, a pose estimation model complexity level, and a radar sampling frequency. The allocation of computing resources is dynamically adjusted based on the resource scheduling action vector.

7. The method according to claim 1, characterized in that, After resuming visual tracking upon removal of occlusion, the process includes: By using Bayesian networks to fuse the target’s behavioral confidence, behavioral anomaly during occlusion, surrounding environment and historical behavioral patterns, the probability of target behavioral risk is comprehensively assessed. The overall risk probability is compared with multiple preset thresholds, and different levels of alarm response actions are triggered based on the comparison results.

8. An intelligent security monitoring system, characterized in that, include: The dynamic privacy control module is used to extract the first visual features and first radar features of the target, perform cross-modal fusion analysis, and generate a dynamic blur intensity coefficient based on the abnormal behavior results after fusion analysis, so as to dynamically adjust the blur intensity of the target face region. Based on the change in the dynamic fuzzy intensity coefficient of the target, determine whether to output a high-risk warning signal; The cross-modal anomaly detection module is used to extract the refined second visual features and second radar features of the target for cross-modal deep fusion analysis if a high-risk warning signal is output, and output the target abnormal behavior judgment result based on the deep fusion features. The occlusion processing module is used to track the target based on the determination result, and when the target is visually occluded, it associates the visually detected target with the radar-tracked target, predicts the future trajectory of the occluded target, and resumes visual tracking when the occlusion is removed.

9. An electronic device, characterized in that, The electronic device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores computer program instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 7.

Images