Semantic analysis system and method for public security monitoring

Abstract

The invention discloses a semantic analysis system and method for public security monitoring. The system comprises a multi-source data acquisition and fusion module; the multi-modal semantic feature extraction module comprises a video textualization analysis unit and a social semantic feature construction unit; the semantic comprehension and correlation analysis engine is connected with the feature extraction module; the real-time early warning and feedback module is connected with the engine; the model updating module is connected with the real-time early warning and feedback module, the feature extraction module and the semantic understanding and correlation analysis engine and is used for incremental learning; according to the method, the unstructured monitoring data is converted into the structured natural language description sequence in real time, and the structured natural language description sequence and the audio transcription text are fused in the same semantic space, so that deep semantic analysis of the multi-source monitoring information is realized; the problems that in the prior art, multi-modal information is simply spliced only on the data or feature level, and deep semantic association is lacked are solved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing and sentiment analysis technology, and in particular to a semantic analysis system and method for public safety monitoring. Background Technology

[0002] Current mainstream intelligent monitoring systems mainly use computer vision technology to analyze videos in real time, enabling functions such as face recognition, vehicle detection, or detection of abnormal behaviors (such as running or gathering). Meanwhile, audio monitoring can convert speech into text through speech recognition. The popularization of Internet of Things (IoT) technology allows data from sensors such as temperature, smoke, and access control to be connected to the monitoring network. Currently, there are studies using rule-based or traditional machine learning methods for simple event detection and alarms.

[0003] First, existing "multimodal fusion" solutions mostly remain at the level of simple data layer or feature layer concatenation and joint decision-making, failing to delve into semantic understanding for in-depth analysis. For example, the system may independently detect "someone raising their hand" (visual) and "shouting 'help'" (audio), but cannot semantically connect and understand them as a complete "call for help" event. Furthermore, it struggles to combine environmental sensor data (such as lights suddenly going out) to infer context and urgency. Second, existing methods heavily rely on predefined rules or shallow feature matching to identify complex social behaviors (such as fraud, arguments, and stalking), lacking deep semantic modeling capabilities for behavioral intent, emotions, and social relationships between multiple targets. This results in insufficient predictability of potential risks and high false positive and false negative rates. Third, the systems are typically statically configured, making it difficult for their analysis models and rule bases to effectively learn and dynamically evolve online based on new cases or operator feedback. Knowledge cannot be continuously accumulated, leading to poor adaptability to new security threats.

[0004] Therefore, in response to the problems mentioned above, this invention proposes a semantic analysis system and method for public safety monitoring. Summary of the Invention

[0005] To overcome the shortcomings of existing public safety monitoring systems in terms of insufficient multimodal semantic fusion and lack of deep social behavioral semantic understanding, this invention proposes a semantic analysis system and method for public safety monitoring. This system unifies and integrates multi-source monitoring data into a natural language semantic space, constructs a public safety knowledge graph for deep association and reasoning, and introduces an update-based mechanism to achieve accurate early warning from low-level perception data to high-level security semantics.

[0006] The technical solution of this invention is: a semantic analysis system for public safety monitoring, comprising: The data acquisition and fusion module is used to acquire monitoring data in real time from multiple monitoring sources deployed in public areas. The monitoring data includes at least video, audio and sensor event sequences collected from sensors. The module also parses the video to extract visual feature sequences, performs speech recognition and voiceprint feature extraction on the audio, and generates corresponding text transcription information and voiceprint identifiers. The semantic feature extraction module, connected to the acquisition and fusion module, includes: The video text analysis unit is used to transform visual feature sequences into descriptive natural language text sequences using a visual semantic description model. The visual semantic description model is a pre-trained model based on the Transformer architecture, which is trained with a large number of video-description text pairs and can understand and generate behavioral descriptions that conform to the context of public safety monitoring. The model adopts an encoder-decoder architecture and has been customized and improved for the spatiotemporal characteristics of video.

[0007] (1) The video encoder adopts a spatiotemporally separated Transformer encoder. First, a pre-trained 2DCNN backbone network is used to extract dense grid features of each frame image. Then, these frame-level features are input into a stacked multi-layer Transformer encoder for temporal modeling. The encoder also uses a spatiotemporal self-attention mechanism: in the spatial dimension, it focuses on the relationship between different regions within the same frame (such as the positional relationship between people and objects); in the temporal dimension, it focuses on the motion and state changes of the same target or related targets across frames (such as a person's walking trajectory or the delivery process of a package). The final output of the encoder is a feature sequence rich in spatiotemporal context.

[0008] The text decoder employs an autoregressive Transformer decoder. The decoder uses the feature sequence output by the encoder as contextual memory and dynamically focuses on the most relevant video spatiotemporal segments when generating each descriptive word through a cross-attention mechanism. The decoder itself also models the semantic dependencies of the generated text through masked self-attention. To enhance the security domain relevance of the generated descriptions, this invention integrates a domain-aware module into the output layer of the decoder. During each generation step, this module references a pre-built high-frequency vocabulary of the public safety domain (containing verbs such as "loitering," "tailing," "passing," "gathering," and "climbing," and nouns such as "suspicious package" and "warning line") to bias the probability distribution of the vocabulary, thereby guiding the model to generate more domain-specific terms.

[0009] (2) The model is trained using large-scale and publicly available general video-description pairing datasets, such as MSR-VTT (containing approximately 10,000 video clips and 200,000 descriptions) and ActivityNet Captions (containing 100,000 temporal descriptions of 20,000 untrimmed videos). The total number of training samples in this stage exceeds 500,000 video-text pairs, with the aim of enabling the model to learn the general alignment relationship between visual concepts and language descriptions, basic grammar, and common ways of expressing objects and actions.

[0010] This invention also constructs a proprietary video description dataset for security monitoring scenarios based on this model. This dataset is constructed in the following manner: Anonymized surveillance video clips from transportation hubs, city streets, and campuses, totaling over 5,000 hours, were collected. Detailed descriptive text was then written for each video clip according to strict standards. The annotation standards require the inclusion of: the subject's physical characteristics, specific actions (avoiding ambiguity), interacting objects, and location information. For example, instead of using "suspicious behavior," the description should be "A man repeatedly tried to use different cards at an ATM and frequently observed his surroundings." Video clips from historical real-world security incidents (after anonymization) were then highlighted to create high-quality positive samples. Finally, this dataset contains approximately 200,000 finely annotated video clip-description pairs, covering over 100 categories of high-frequency behaviors and events related to public safety.

[0011] (3) Its pre-training objective mainly adopts a combination of sequence-to-sequence masked language modeling and video-text contrastive learning. At the encoder end, random masking of video frame or region features is required for model reconstruction; at the decoder end, standard next-word prediction loss is used to train text generation ability. At the same time, the video-text contrastive learning loss function prompts the model to learn a shared embedding space, making the matched video and text representations close to each other.

[0012] For domain-specific data, end-to-end fine-tuning was performed using a relatively small learning rate, with standard cross-entropy loss as the primary loss function. To mitigate the problem of description templates, minimum-risk training was introduced, encouraging the model to generate sentences that are semantically similar to multiple reference descriptions (the same video may be annotated by multiple people) but with diverse expressions. Mixed-precision training and gradient accumulation techniques were used during training to optimize computational resources.

[0013] The audio filtering and parsing unit has a built-in legal broadcast semantic library and comparison unit and an audio fingerprint fast comparison unit. It is used to identify and filter environmental broadcast interference and parse the semantics of the target sound source. At the same time, it quickly matches the fingerprint of the real-time environmental broadcast signal with the pre-stored common broadcast audio fingerprint library to achieve real-time filtering of high-frequency and fixed content broadcasts. The social semantic feature construction unit is used to extract intent, sentiment, and instruction information based on text transcription information and descriptive natural language text sequences, and construct social semantic feature vectors. The semantic understanding and correlation analysis engine, connected to the semantic feature extraction module, is used to correlate with a public safety knowledge graph that stores high-risk behavior patterns, abnormal event templates, and interference source information. The semantic understanding and correlation analysis engine is used to correlate, match, and reason with social semantic feature vectors, voiceprint identifiers, and sensor data with the knowledge graph to generate semantic alarm information. It also has multimodal data consistency verification logic to reduce false alarms caused by broadcast or noise interference. The real-time early warning and feedback module is connected to the semantic understanding and correlation analysis engine. It is used to push semantic alarm information to the monitoring terminal and trigger the early warning plan, and collect the processing results and operator feedback. The model update module, connected to the real-time early warning and feedback module, the semantic feature extraction module, and the semantic understanding and association analysis engine, is used to learn and update the models and knowledge graphs within the system using the returned labeled data.

[0014] Preferably, the social semantic feature construction unit includes a dependency parser, a public safety domain sentiment dictionary, and an intent lexicon, and its output is a quintuple sequence, which is a (subject, action, object, intent, sentiment) quintuple sequence.

[0015] Preferably, the public safety knowledge graph in the semantic understanding and association analysis engine is stored and queried using a graph database, and integrates a graph neural network reasoning module to calculate the correlation between current event features and graph patterns and perform logical reasoning.

[0016] This invention proposes a semantic analysis method for public safety monitoring, comprising the following steps: (1) Real-time acquisition of monitoring data from multiple monitoring sources deployed in public areas, including at least video, audio and sensor-collected data; (2) Perform real-time analysis on the video, extract visual feature sequences that contain at least human posture, movement trajectory and facial orientation, and perform speech recognition and voiceprint feature extraction on the audio stream to generate corresponding text transcription information and voiceprint identifiers; The text content of the environmental broadcast signal is parsed and compared with a preset legal broadcast semantic library for the scene. If the broadcast signal content matches the legal broadcast semantics, it is marked and filtered out from the semantic analysis of the current monitoring event. Then, speech recognition and sentiment analysis are performed on the target sound source signal obtained after sound source separation to obtain the target audio semantic information. Based on the visual feature sequence and the semantic information of the target audio after filtering out broadcast interference, intention and emotion are extracted to construct an interference-resistant semantic feature vector. The anti-interference semantic feature vector is input into the public safety knowledge graph for matching. This knowledge graph has pre-set high-risk event patterns corresponding to the scene and containing information on typical interference sources. This step performs multimodal consistency verification, specifically: when the audio semantics indicate high risk but the video semantics do not provide support, the system will combine the environmental broadcast tag information and the knowledge graph to verify whether the audio is a broadcast echo, background noise or other interference, and reduce the confidence of the event or determine it as a false alarm accordingly. (3) Using a visual semantic description model, the visual feature sequence is transformed into a descriptive natural language text sequence, which includes at least a dynamic description of the behavior of the target individual, the interaction relationship between multiple targets and the scene state; preferably, the visual semantic description model adopts an encoder-decoder architecture based on an attention mechanism, in which the encoder encodes the visual feature sequence and the decoder generates a structured natural language description containing the behavioral subject, action predicate and scene object frame by frame based on the encoded context vector; (4) Based on the text transcription information and descriptive natural language text sequence, use natural language processing technology to extract the intent keywords, sentiment words and imperative sentences to construct the target social semantic feature vector; preferably, this step uses dependency parsing to process the text transcription information and descriptive text sequence, identify the core predicates and agent-patient relationships, and perform sentiment analysis and intent classification on the identified core predicates and related words based on the pre-constructed public safety domain sentiment dictionary and intent word library, forming a (subject, action, object, intent, sentiment) quintuple as the basic element of the social semantic feature vector, wherein the intent word library contains classifications that are highly related to public safety, such as "threat", "help", "transaction", "guidance"; (5) Input the social semantic feature vector, voiceprint identifier and sensor event sequence into the public safety knowledge graph for association matching and reasoning. The public safety knowledge graph pre-stores semantic features of high-risk behavior patterns, abnormal event templates and historical cases. The nodes of the public safety knowledge graph include: personnel entities, behavior entities, item entities, location entities and time entities; the edges represent the relationship between entities or event logic links. This step calculates the correlation between the current multimodal features and the nodes and edges in the knowledge graph through graph neural network, identifies the current event that matches the predefined high-risk semantic pattern, and generates semantic alarm information containing event type, risk level, involved target and confidence level. (6) Push semantic alarm information to the monitoring terminal and trigger different early warning plans according to the risk level. At the same time, use the handling results of this event and the operator's feedback as labeled data and send them back to the model for updating.

[0017] Preferably, the association matching and reasoning specifically involves: performing similarity matching between the five-tuple of the current event and the event subgraph in the knowledge graph, and using the reasoning rule chain of the graph to predict potential subsequent high-risk events.

[0018] Preferably, the model update module receives the returned labeled data and periodically learns and updates the multimodal semantic understanding model, visual semantic description model, and public safety knowledge graph. The update of the public safety knowledge graph includes creating or strengthening nodes and edges in the graph based on the five-tuple of new events and their processing results, or adjusting the risk weights of event templates.

[0019] The beneficial effects of this invention are: 1. This invention solves the problem of false alarms caused by interference sources such as environmental broadcasts or media sound leakage in noisy public monitoring scenarios by introducing mechanisms such as "sound source separation processing", "legitimate broadcast semantic library filtering" and "multimodal consistency verification logic". Experimental data fully demonstrates that in a special test containing 300 high-interference negative samples, this invention increases the false alarm suppression rate in interference scenarios to 96.7% and reduces the overall false alarm rate to 3.2%, achieving an order-of-magnitude leap compared to traditional solutions, thereby ensuring the alarm reliability of the system in real complex environments.

[0020] 2. The alarm information generated by this invention transforms the focus from whether an alarm is present to why it is being triggered, demonstrating strong interpretability and significantly improving the decision-making efficiency and trustworthiness of monitoring personnel. The system not only outputs risk conclusions but also clearly explains the semantic reasoning process behind those conclusions within the alarm, particularly by proactively indicating eliminated interference factors (such as "audio content matches the current broadcast semantics and has been filtered" or "no supporting actions were detected in the video; suspected media audio"). In the evaluation of alarm information quality, this invention received an exceptionally high score of 4.5, far exceeding traditional solutions. This allows monitoring personnel to quickly understand the logic behind the alarm and take targeted measures, precisely focusing limited security resources on the real threats. Attached Figure Description

[0021] Figure 1 The diagram shown is a schematic representation of the system framework of this invention. Figure 2 The diagram shown is a schematic representation of the implementation process of the railway station system of the present invention. Figure 3 The diagram shown illustrates the implementation process of the campus system according to the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Please see Figure 1 This invention provides an embodiment of a semantic analysis system for public safety monitoring: In this embodiment, the data acquisition and fusion module is responsible for acquiring data in real time from various acquisition devices (cameras, microphone arrays, infrared sensors, door magnetic switches, etc.) throughout the monitored area. This module decodes the video and performs preliminary computer vision analysis, extracting visual feature sequences represented by human skeletal key point sequences, target tracking trajectories, and micro-expression features; it performs noise reduction, track splitting, and automatic speech recognition on the audio, generating text transcription information with timestamps, and simultaneously extracting voiceprint features to generate voiceprint identifiers; it processes the data reported by various sensors to form structured sensor event sequences (e.g., "{Time: T, Location: East Gate of Area A, Event Type: Abnormal Access Control Opening, Confidence: 0.95}"). All data is uniformly stamped with a spatiotemporal stamp.

[0024] In this embodiment, the semantic feature extraction module is described in detail: (1) The video text analysis unit of this module integrates a large-scale pre-trained visual semantic description model. This model adopts a "video encoder-text decoder" architecture, in which the encoder is responsible for spatiotemporal modeling of the input visual feature sequence and capturing dynamic information, while the decoder generates fluent and accurate natural language description sentences based on the context vector output by the encoder. The model of this invention is specifically optimized for public safety scenarios. Its training corpus contains a large number of video and text pairs labeled with specific behaviors such as "tailing", "argument", "leaving objects", and "climbing", which makes the accuracy of the generated descriptions far exceed that of general image description models in terms of behavioral verbs and scene elements.

[0025] For example, for a video clip, the model might output: "At 18:30:05, a man wearing a black coat (ID: P001) loitered in front of ticket window number 3 and had brief physical contact with a woman wearing a red top (ID: P002), after which the woman handed the man a small black package."

[0026] (2) The audio filtering and parsing unit of this module has a built-in legal broadcast semantic library and comparison unit and an audio fingerprint fast comparison unit, which is used to identify and filter environmental broadcast interference and parse the semantics of the target sound source. At the same time, it quickly matches the fingerprint of the real-time environmental broadcast signal with the pre-stored common broadcast audio fingerprint library to achieve real-time filtering of high-frequency and fixed content broadcasts.

[0027] (3) The social semantic feature construction unit of this module receives descriptive text from the video textification analysis unit and transcribed text from audio recognition. First, it uses a dependency parser and named entity recognition technology to perform deep syntactic parsing on the merged text stream, identifying the core action (predicate), performer (subject), receiver (object), and time, location, and other components in the sentence. Then, the public safety domain sentiment analyzer and intent recognition classifier integrated within the unit begin to work. The sentiment analyzer is based on a dictionary that integrates general sentiment vocabulary and domain-specific vocabulary (such as "panic", "arrogance", "pleading"), and judges the sentiment polarity (positive, negative, neutral) and intensity of the sentence in combination with the context. The intent recognition classifier is a multi-label classification model that can identify dozens of public safety-related intent labels such as "seek help", "threat", "transaction", "inquiry", "gathering", and "escape". Finally, the unit integrates all the above analysis results to generate a structured semantic quintuple for each detected target or interaction event.

[0028] For example: (Subject: P001, Action: Delivery, Object: Black Package, Intent: Transaction, Emotion: Neutral) and (Subject: P002, Action: Receiving, Object: Black Package, Intent: Transaction, Emotion: Tension).

[0029] In this embodiment, the semantic understanding and association analysis engine will be described in detail: The engine's public safety knowledge graph is stored in the form of a graph database. Node types include entities such as people, behaviors, items, locations, times, and organizations. Edges represent various relationships between entities (such as "located in", "implemented", "owned", "accomplices", "occurred in") and higher-level event logic links (such as "argument" may "lead to" "fight", "long-term loitering" may "predict" "theft"). At the same time, multimodal data consistency verification logic is set up to reduce false alarms caused by broadcast or noise interference.

[0030] The workflow is as follows: For the semantic quintuple sequence pushed in real time by the social semantic feature construction unit, the engine first links it with entities in the knowledge graph. For example, it associates "P001" with existing records of suspicious persons in the graph, and associates "black package" with the concept of "suspicious item". Then, the graph neural network reasoning module is activated. This module matches and calculates the current event subgraph (temporarily composed of a series of related quintuples) with the predefined "high-risk event pattern subgraph" in the knowledge graph. For example, a "premeditated pickpocketing" pattern may contain specific combinations of multiple behavioral nodes such as "looking around", "deliberately crowding", and "focusing attention on other people's property". Graph neural networks calculate the semantic similarity between the current subgraph and each pattern subgraph through a message passing mechanism. When the matching degree exceeds a preset threshold, the engine will not only immediately generate a semantic alert (e.g., "Alert: Suspected illegal transaction of goods detected at ticket window 3, involving personnel P001 (previously recorded) and P002, risk level: medium, confidence level: 87%), but also provide further warnings based on the inference rule chain in the graph (e.g., based on the pattern of "quickly separating after the transaction", suggest "tracking the subsequent travel direction of both parties").

[0031] When the system detects a suspected high-risk audio semantic event, the multimodal data consistency verification logic is immediately activated, performing a detailed backtracking and comparison of the video semantic stream within the same spatiotemporal range. It first checks action consistency, determining whether the target person's body movements in the video match the audio semantics; for example, when an audio alarm says "help," are there running, waving, or other distress-related actions in the video? Next, it performs lip-sync timing analysis, comparing the timing of dangerous words with the mouth movements of the person in the frame to determine if it was a direct human voice. Simultaneously, it combines sound source separation and localization technology to verify whether the direction of the suspected sound source matches the spatial location of any potential target in the video. Any serious inconsistency will trigger a primary anomaly marker.

[0032] Then, it performs active correlation verification with the tagged environmental interference sources. This logic queries the output of the audio filtering unit in real time to check whether the system has identified the playing environmental broadcast signal within the time window of the target audio event. If so, it calculates the deep semantic similarity between the text content of the current high-risk audio and the "legitimate broadcast semantic library" and analyzes their temporal overlap. (For example, if a station broadcast is playing "Please keep your belongings safe and prevent theft," and the system happens to identify the dangerous word "theft," the logic will determine that this audio is very likely a fragment of the broadcast content, thus significantly reducing its risk weight). This step can isolate known systemic interference from the events to be judged.

[0033] Finally, based on the contextual semantic coherence and historical pattern reasoning of the knowledge graph, building upon the first two steps, this logic projects the current cross-modal evidence sequence into the public safety knowledge graph for high-level behavioral chain rationality judgment. The graph stores the typical development logic of events (for example, a real "robbery" usually includes a sequence of "approach-threat-robbery-escape." If there is only isolated "threat" audio without video or other sensor evidence to support its antecedents and consequences, the logic will determine that it does not conform to the real high-risk event pattern). At the same time, the logic matches the multimodal features of the current event with the historical false alarm case patterns stored in the knowledge graph. If the features are highly similar, it is directly classified as a reproduction of a known interference pattern. After these three progressive verifications, the logic outputs a comprehensive confidence adjustment factor, ultimately deciding whether to generate an alert, downgrade the handling, or silently filter, thus forming an intelligent firewall against false alarms.

[0034] In this embodiment, the real-time early warning and feedback module is described in detail: This module is responsible for sending semantic alerts generated by the engine to operators in the monitoring center. Low-risk events generate logs on the console; medium-risk events trigger screen pop-ups and sound alerts; and high-risk events directly link the broadcast system and physical devices such as warning lights.

[0035] This module records the operator's actions for each alarm (such as confirming, false alarm, ignoring, and dispatching an alarm), and feeds these processing results, along with the original alarm data and the data fragment that triggered the alarm, back to the model update module.

[0036] In this embodiment, the model update module will be described in detail: The model update module processes data from the real-time early warning and feedback module every 24 hours or based on the amount of accumulated data.

[0037] For visual semantic description models and intent / sentiment classifiers, incremental learning or online learning strategies are adopted. Data corresponding to correctly confirmed alarms are used as positive samples, and false alarm data are used as difficult negative samples to fine-tune the existing models, making them increasingly better at distinguishing edge cases.

[0038] For the public safety knowledge graph, the system will automatically analyze the feedback data. For newly emerging and valid event patterns, it will attempt to create new nodes and relationships in the graph. For existing patterns, it will adjust their risk weights and trigger thresholds based on the frequency with which they are manually confirmed after triggering alarms.

[0039] For example, if a specific pattern of crowd gathering is repeatedly confirmed to be square dancing activities (low risk) at night, the system will gradually reduce the risk weight of that pattern to reduce false alarms; conversely, if a new type of fraud is identified, its corresponding semantic pattern will be quickly strengthened and shared throughout the entire monitoring network.

[0040] Please see Figure 2 The present invention provides an embodiment 1: In this embodiment, more than 200 high-definition cameras, 50 microphones, and hundreds of smoke detectors and infrared beam sensors are deployed in areas such as the entrances and exits of the train station, the waiting hall, the ticket office, and the platform.

[0041] Implementation process: (1) The system accesses all collected data in real time. Video analysis detected that a passenger (ID: T401) lingered next to the security scanner for a long time and glanced at other people's luggage multiple times. At the same time, the audio system captured a vague whisper and automatically transcribed it into "...that bag...".

[0042] (2) "T401 lingered by the security conveyor belt, frequently scanning the luggage of passing passengers." Combining the audio text, the social semantic construction unit generates a quintuple: (T401, lingering and scanning, passenger luggage, intention: spying / waiting for an opportunity, emotion: tension).

[0043] (3) The engine matches the quintuple with the knowledge graph. The graph contains a subgraph of the pattern "premeditated baggage theft", which includes nodes such as "loitering in dense baggage areas", "focusing on baggage other than one's own" and "no normal communication with others". The current event matches the pattern with a degree of 82%. At the same time, the engine's related query found that there was no ticket purchase record for T401.

[0044] (4) At this time, the system generates a high-level semantic alert: "Alert: Suspicious person T401 has been found on the east side of the security checkpoint. The behavior characteristics are consistent with the premeditated baggage theft pattern. It is recommended that on-site security personnel intervene for questioning. Risk level: high, confidence level: 85%". The alert is pushed to the mobile terminals of patrolling security personnel and the large screen of the command center.

[0045] (5) After the security personnel arrived, T401 left quickly. The operator marked the incident as "effective warning, target escaped". This positive feedback was used to strengthen the connection between the "premeditated theft" pattern and related features such as "unlicensed stay" in the knowledge graph.

[0046] Please see Figure 3 The present invention provides an embodiment 2: In this embodiment, an infrared intrusion alarm system, a public address system, and cameras facing both inside and outside the perimeter of the school wall are installed.

[0047] Implementation process: (1) At night, the infrared sensor triggers an “intrusion” alarm on a section of the fence, and the camera quickly adjusts its view to capture two individuals (ID: S01, S02) loitering outside the fence and making limb movements that attempt to climb (visual features: human posture estimated to be climbing posture).

[0048] (2) The broadcast system turned on the area to pick up audio and captured a dialogue segment: "...from here...hurry up...". The video text description is: "Two unidentified people are trying to climb the wall from the outside, and their movements are covert." (3) Social semantic units integrate information to generate a quintuple: (S01 and S02, attempting to climb, campus wall, intention: illegal intrusion, emotion: urgent).

[0049] (4) The semantic understanding engine matched the "illegal intrusion" pattern. At the same time, the engine queried the knowledge graph and found that the current time was late at night on a teaching day, which was not a normal entry and exit time. Moreover, the wall was close to the key laboratory area on campus. The graph association rule assessment risk was "high".

[0050] (5) The system immediately responds by sending a red alert and real-time video to the campus security center. At the same time, it automatically controls the lighting system in the area to turn on the strong light to deter intrusion. It also issues a voice warning through the intercom system: "Warning, you have entered the monitored area. Please leave immediately!" Finally, it pushes the screenshot and characteristics of the target personnel to the mobile terminals of all patrol posts.

[0051] (6) The two people fled quickly after being exposed to strong light and voice warning. At this time, the operator confirmed that the alarm was effective. This successful "deterrence-drive-away" case was used by the model update module to optimize the response strategy threshold in similar scenarios. For example, in the late night, the confidence requirement for the recognition of "climbing intention" can be appropriately reduced, thereby achieving a faster early warning response.

[0052] This invention provides a comparative example 1: This experiment compares the present invention with two comparative examples: Comparative Example 1 is a purely vision-based behavior recognition system that uses a deep convolutional network to recognize predefined behaviors. Comparative Example 2 is a multimodal system that directly concatenates visual feature vectors and audio feature vectors and inputs them into a classification network for event recognition.

[0053] This experiment constructed a test dataset covering scenarios such as transportation hubs, squares, and campuses, containing 1000 labeled security event segments (such as premeditated theft, fighting, illegal intrusion, and group gatherings) and 2000 normal behavior segments. Each segment includes synchronized video, audio, and simulated sensor events. This test dataset was used to verify the various metrics of the present invention.

[0054] Table 1 Test Results

[0055] As shown in the table above, the present invention has the highest accuracy rate. This is because the present invention, through semantic understanding, can identify complex scenarios that are difficult to distinguish in comparison examples 1 and 2. For example, for "intense debate" and "physical conflict," the former two may both be falsely reported as "fighting" due to the large range of motion, while the present invention can distinguish them by analyzing the semantics of dialogue content (intent) and interaction posture, thereby simultaneously improving accuracy and significantly reducing false positives and false negatives.

[0056] This invention provides a comparative example 2: In this comparative study, data was collected from train station waiting halls and campus perimeter environments, including 200 real high-risk event segments (such as pickpocketing disputes and illegal trespassing) and 300 carefully designed "high-interference negative sample" segments. The negative samples specifically simulate scenarios prone to false alarms, such as: (1) Broadcast interference: When the background is playing easily confusing announcements such as "Passengers in emergency vehicles, please go to the service desk" or "Keep your belongings safe to prevent theft", people are doing normal activities.

[0057] (2) The media leaked a short video played on a passenger's mobile phone, which contained dialogues involving "fighting" and "fire".

[0058] (3) The environment is noisy, such as the "lie down" command in the school broadcast exercise and the screams of children playing games.

[0059] Three sets of comparative examples were designed based on the above scenario, as follows: Comparative Example 1, representing the feature-layer fusion baseline system, embodies the current mainstream multimodal monitoring solutions. It possesses video behavior recognition and audio speech recognition capabilities, directly concatenating visual and audio feature vectors at the feature layer before inputting them into a classification network for event determination. However, it lacks sound source separation, broadcast semantic filtering, and knowledge graph-based multimodal consistency verification logic. Its alarms rely entirely on the output confidence level of the classification network.

[0060] Comparative Example 2 is an enhanced system based on rule filtering. This system adds simple rule-based filtering and has a built-in keyword blacklist (containing common broadcast words such as "train", "broadcast", and "notification"). When these keywords appear in the audio recognition text, the audio analysis results are directly discarded or their weight is reduced.

[0061] Comparative Example 3 uses an advanced semantic matching system. This system incorporates preliminary semantic analysis, converting video content into simple behavioral description text (such as "someone is running") and performing keyword matching with the audio text. An alarm is only triggered when both contain high-risk words such as "fighting" or "help."

[0062] Table 2 Test Results

[0063] As shown in the table above, Comparative Example 1 performed the worst, with an extremely high false positive rate (28.3%), mainly originating from interference scenarios. This indicates that simple feature concatenation is completely unreliable in noisy environments, and audio noise can severely mislead the classification model. Comparative Example 2 filtered out some obvious broadcast words through a keyword blacklist, resulting in a lower false positive rate. However, its false positive suppression rate was only 62%, indicating serious limitations. Firstly, it cannot handle variant expressions and semantically related interference (such as "emergency vehicle"); secondly, it may mistakenly reject real alarms (such as a real emergency call that happens to contain the word "notification"). Its alarm information quality is low and cannot provide a basis for decision-making. Comparative Example 3 showed a significant improvement, but its interference suppression rate of 78.5% is still low. Because it lacks sound source separation, when the target human voice is mixed with the broadcast sound, its audio semantics are already contaminated, leading to an unreliable basis for consistency judgment. Furthermore, the flat keyword library cannot understand the essential difference between "theft mentioned in the broadcast" and "theft actually occurred" in context, making it prone to misjudgment when faced with "media leakage" interference.

[0064] This invention achieves the best results across all metrics, especially a false alarm suppression rate of 96.7% in interference scenarios and an alarm quality score of 4.8, which fully demonstrates the synergistic effectiveness of the new module.

[0065] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A semantic analysis system for public safety monitoring, characterized in that, Including: The data acquisition and fusion module is used to acquire monitoring data in real time from multiple monitoring sources deployed in public areas. The monitoring data includes at least video, audio and sensor data collected from sensors. The module analyzes the video to extract visual feature sequences, performs speech recognition and voiceprint feature extraction on the audio, and generates corresponding text transcription information and voiceprint identifiers. The semantic feature extraction module, connected to the acquisition and fusion module, includes: The video text analysis unit is used to transform visual feature sequences into descriptive natural language text sequences using a visual semantic description model. The audio filtering and parsing unit has a built-in legal broadcast semantic library and comparison unit and an audio fingerprint fast comparison unit. It is used to identify and filter environmental broadcast interference and parse the semantics of the target sound source. At the same time, it quickly matches the fingerprint of the real-time environmental broadcast signal with the pre-stored common broadcast audio fingerprint library to achieve real-time filtering of high-frequency and fixed content broadcasts. The social semantic feature construction unit is used to extract intent, sentiment, and instruction information based on text transcription information and descriptive natural language text sequences, and construct social semantic feature vectors. The semantic understanding and correlation analysis engine, connected to the semantic feature extraction module, is used to correlate with a public safety knowledge graph that stores high-risk behavior patterns, abnormal event templates, and interference source information. The semantic understanding and correlation analysis engine is used to correlate, match, and reason with social semantic feature vectors, voiceprint identifiers, and sensor data with the knowledge graph to generate semantic alarm information. It also has multimodal data consistency verification logic to reduce false alarms caused by broadcast or noise interference. The real-time early warning and feedback module is connected to the semantic understanding and correlation analysis engine. It is used to push semantic alarm information to the monitoring terminal and trigger the early warning plan, and collect the processing results and operator feedback. The model update module, connected to the real-time early warning and feedback module, the semantic feature extraction module, and the semantic understanding and association analysis engine, is used to learn and update the models and knowledge graphs within the system using the returned labeled data.

2. The semantic analysis system for public safety monitoring according to claim 1, characterized in that: The visual semantic description model in the video textual analysis unit is a pre-trained model based on the Transformer architecture. This pre-trained model is trained on a large number of videos and descriptive texts, and is able to understand and generate behavioral descriptions that conform to the context of public safety monitoring.

3. The semantic analysis system for public safety monitoring according to claim 1, characterized in that: The social semantic feature construction unit includes a dependency parser, a public safety domain sentiment dictionary, and an intent lexicon. Its output is a quintuple sequence, the structure of which is "subject, action, object, intent, sentiment".

4. The semantic analysis system for public safety monitoring according to claim 1, characterized in that: The public safety knowledge graph in the semantic understanding and association analysis engine is stored and queried using a graph database, and integrates a graph neural network reasoning module to calculate the correlation between current event features and graph patterns and perform logical reasoning.

5. A semantic analysis method for public safety monitoring, employing the semantic analysis system for public safety monitoring as described in any one of claims 1-4, characterized in that, It includes the following steps: S1, real-time acquisition of monitoring data from multiple monitoring sources deployed in public areas, the monitoring data including at least video, audio and sensor data; S2 performs real-time analysis of the video, extracts visual feature sequences containing at least human posture, movement trajectory, and facial orientation, and performs speech recognition and voiceprint feature extraction on the audio stream to generate corresponding text transcription information and voiceprint identifiers. S201, parse the text content of the environmental broadcast signal and compare it with the preset legal broadcast semantic library in this scenario. If the broadcast signal content matches the legal broadcast semantics, mark it and filter it out from the semantic analysis of the current monitoring event. Then, perform speech recognition and sentiment analysis on the target sound source signal obtained after sound source separation to obtain the target audio semantic information. S202, based on the visual feature sequence and the semantic information of the target audio after filtering out broadcast interference, extract the intent and emotion, and construct an interference-resistant semantic feature vector; S203, the anti-interference semantic feature vector is input into the public safety knowledge graph for matching. The knowledge graph has pre-set high-risk event patterns corresponding to the scene and containing information on typical interference sources. This step performs multimodal consistency verification, specifically: when the audio semantics indicate high risk but the video semantics do not provide support, the system will combine the environmental broadcast marker information with the knowledge graph to verify whether the audio is a broadcast echo, background noise or other interference, and reduce the confidence of the event or determine it as a false alarm accordingly. S4, based on a pre-trained visual semantic description model, fuses and analyzes visual feature sequences, text transcription information, and sensor data. This step specifically includes: S401, using a visual semantic description model, transforms a visual feature sequence into a descriptive natural language text sequence, which includes at least a dynamic description of the target individual's behavior, the interaction relationship between multiple targets, and the scene state; S402, based on text transcription information and descriptive natural language text sequences, uses natural language processing technology to extract intent keywords, sentiment words and instructional statements to construct target social semantic feature vectors; S403, input the social semantic feature vector, voiceprint identifier and sensor data into the public safety knowledge graph for association matching and reasoning. The public safety knowledge graph pre-stores semantic features of high-risk behavior patterns, abnormal event templates and historical cases. This step uses graph neural network to calculate the correlation between the current multimodal features and the nodes and edges in the knowledge graph, identify the current event that matches the predefined high-risk semantic pattern, and generate semantic alarm information containing event type, risk level, involved target and confidence level. S5 pushes semantic alarm information to the monitoring terminal and triggers different early warning plans according to the risk level. At the same time, the handling results of this event and the operator's feedback are used as labeled data and sent back to the model for updating.

6. The semantic analysis method for public safety monitoring according to claim 5, characterized in that: In step S301, the visual semantic description model adopts an encoder-decoder architecture based on an attention mechanism. The encoder encodes the visual feature sequence, and the decoder generates a natural language description containing the subject of the behavior, the action predicate, and the scene object frame by frame based on the encoded context vector.

7. The semantic analysis method for public safety monitoring according to claim 5, characterized in that, Step S302 specifically includes: using dependency parsing to process text transcription information and descriptive natural language text sequences, identifying core predicates and agent-patient relationships, and based on a public safety domain sentiment dictionary and intent lexicon, performing sentiment analysis and intent classification on the identified core predicates and related words to form a five-tuple sequence as the basic element of the social semantic feature vector.

8. The semantic analysis method for public safety monitoring according to claim 5, characterized in that: In step S303, the nodes of the public safety knowledge graph include personnel entities, behavior entities, item entities, location entities, and time entities, where edges represent the relationships or event logic links between entities.

9. The semantic analysis method for public safety monitoring according to claim 5, characterized in that: The association matching and reasoning specifically involves: performing similarity matching between the five-tuple of the current event and the event subgraph in the knowledge graph, and using the reasoning rule chain of the graph to predict potential subsequent high-risk events.

10. A semantic analysis method for public safety monitoring according to claim 5, characterized in that: The model update module receives the returned labeled data and periodically learns and updates the multimodal semantic understanding model, visual semantic description model, and public safety knowledge graph. The update of the public safety knowledge graph includes creating or strengthening nodes and edges in the graph based on the five-tuple of new events and their processing results, or adjusting the risk weights of event templates.

Images