Video early warning method and system based on visual large model

Abstract

The invention discloses a video early warning method and system based on a visual large model. The method comprises the following steps: a data acquisition step: acquiring structured visual data generated by a video monitoring system; a visual analysis step: performing deep analysis on the panoramic image by using a pre-trained visual large model; a feature modeling step: constructing a multi-dimensional feature vector based on a deep analysis result; a semantic deployment and control step: establishing an abnormal behavior deployment and control mechanism based on natural language description, analyzing the abnormal behavior deployment and control mechanism into feature vectors which can be understood by a machine, and forming a deployment and control strategy; and a real-time comparison and early warning step: performing real-time comparison on the multi-dimensional feature vector and a deployment and control strategy to generate early warning information conforming to an early warning condition. According to the method, deep understanding of personnel behaviors and appendages in the video image is realized, the accuracy and real-time performance of abnormal event identification are improved, and an intelligent early warning analysis means is provided for public security actual combat.

Description

Technical Field

[0001] This invention relates to the field of public safety technology, specifically to a video early warning method and system based on a large visual model. Background Technology

[0002] With the acceleration of urbanization and the increasing demand for public safety, video surveillance systems have become an important component of the social security and prevention system. While traditional video structuring technology can detect and identify targets such as people and vehicles in videos, it suffers from the following technical bottlenecks: Limitations of feature extraction: Existing video structuring technologies mainly target specific targets such as faces, bodies, and vehicles for detection and attribute extraction, but lack the ability to deeply understand human behavior patterns, features of objects, and scene relationships.

[0003] Insufficient identification of abnormal behavior: For abnormal behaviors involving public safety, such as fighting, illegal gatherings, carrying dangerous items and other complex scenarios, existing technologies are unable to effectively identify and issue warnings.

[0004] High computational resource consumption: Existing real-time video stream analysis solutions based on small models require a large amount of computational resources, which is not cost-effective and makes it difficult to deploy them comprehensively in large-scale monitoring networks.

[0005] Delayed early warning response: Traditional methods for identifying and issuing early warnings of abnormal events often have a time delay, which cannot meet the real-time requirements of public security operations.

[0006] The model has poor generalization ability: special models need to be trained and deployed for different scenarios and tasks, resulting in poor system scalability and difficulty in adapting to the complex and ever-changing needs of public security operations. Summary of the Invention

[0007] To address the aforementioned shortcomings of existing technologies, this invention provides a video early warning method and system based on a large visual model. This method enables in-depth understanding of human behavior and objects in video images, improving the accuracy and real-time performance of abnormal event identification, and providing intelligent early warning and analysis tools for public security operations.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following solution: A video early warning method based on a large visual model includes the following steps: Data acquisition steps: Acquire structured visual data generated by the video surveillance system, including panoramic images of the target personnel, as well as associated device information and time information; Visual analysis steps: The panoramic image is subjected to deep analysis using a pre-trained large visual model to generate analysis results that include global scene understanding, multi-object detection and association, fine-grained attributes and behavior patterns; Feature modeling steps: Based on the deep analysis results, construct a multi-dimensional feature vector, which includes target features, behavioral features, scene features, and correlation features between targets; Semantic control steps: Establish an abnormal behavior control mechanism based on natural language description, parse the abnormal behavior control mechanism into machine-understandable feature vectors, and form a control strategy; Real-time comparison and early warning steps: The multi-dimensional feature vector is compared with the deployment strategy in real time to generate early warning information that meets the early warning conditions.

[0009] Preferably, in the visual analysis step, the global scene understanding information is obtained by: extracting the global semantic features of the panoramic image, and based on the global semantic features, identifying the scene type, environmental features and spatial layout corresponding to the panoramic image; Multi-target detection and association information is obtained by simultaneously detecting and recognizing multiple targets, including faces, bodies, vehicles, and objects, and establishing spatial and semantic associations between different targets. Fine-grained attribute information is obtained in the following way: the clothing features, carried items, and behavioral posture attributes of the target person are extracted using a multi-task inference head structure based on a large visual model; Behavioral patterns are obtained by feeding local features of the target personnel into a static attribute classification model and a temporal action recognition model to identify and classify the personnel's behavioral patterns.

[0010] Preferably, in the feature modeling step, the target features are obtained in the following way: based on the face features and body features in the deep analysis results, the corresponding original visual feature vectors are extracted; the original visual feature vectors are standardized to obtain target feature vectors of face features and body features with uniform modulus length. Behavioral features are obtained by sampling image sequences containing the target person from consecutive video frames using a sliding window approach; encoding the image sequences using a temporal action recognition model to output temporal feature vectors representing behavior type, action sequence, and interaction mode; Scene features are obtained by extracting the global semantic feature vector of the panoramic image, wherein the global semantic feature vector is the feature vector corresponding to the [CLS] label output by the visual encoder of the large visual model.

[0011] Preferably, in the feature modeling step, the associated features are obtained through the following steps: Spatial relationship modeling: Calculate the spatial geometric metrics between different targets, including the intersection-union ratio of the target bounding boxes, the distance between the center points, and the relative azimuth angle; Semantic relation modeling: Using graph neural networks, modeling the category associations and behavioral interactions between targets; Temporal relationship modeling: Calculate the trajectory similarity between different target motion trajectories to measure the stability of spatiotemporal co-occurrence of different targets; Feature fusion: The spatial geometric measure, the output of the graph neural network, and the trajectory similarity are concatenated to generate a comprehensive relational feature vector.

[0012] Preferably, the feature modeling step further includes a feature alignment step, which includes mapping the target features, behavioral features, and scene features to a common semantic space of the same preset dimension through a shared projection layer, so as to achieve scale alignment and semantic unification of cross-modal features.

[0013] Preferably, the semantic control steps include: Semantic parsing: Using a pre-trained language model, the deployment conditions described in natural language are parsed into machine-understandable feature vectors; Deployment strategy definition: Supports defining multi-level deployment logic rules that include time, space, number of targets, behavior type, and duration; Dynamic deployment update: Based on manual analysis and feedback or preset spatiotemporal context rules, dynamically adjust the matching threshold or activation status of deployment conditions.

[0014] Preferably, the real-time comparison and early warning steps include: Multi-dimensional similarity calculation: The similarity between real-time features and deployment strategies is calculated on multiple feature dimensions, and then weighted and fused to obtain a comprehensive matching score. The multiple feature dimensions include target attributes, behavior patterns, scene context, and association relationship dimensions. Adaptive threshold adjustment: Based on statistical feedback from historical warnings, dynamically adjust the threshold for warning determination; Warning information generation: For targets whose matching scores exceed the threshold, warning information is generated. The warning information includes target identity attribute tags, behavior tags, scene type, spatiotemporal coordinates, device information, time information, and association with other targets.

[0015] Preferably, the adaptive threshold adjustment method is as follows: based on the true positive rate and false positive rate of historical early warnings within the sliding time window, the threshold is dynamically adjusted using an exponential smoothing algorithm.

[0016] Preferably, the video early warning method further includes an early warning assessment and handling step, which includes: aggregating and displaying the generated early warning information, performing multi-source information correlation analysis, and providing visualization auxiliary tools to support manual assessment and handling process management.

[0017] A system for implementing the above-mentioned video early warning method based on a large visual model includes: Data acquisition module: used to acquire structured visual data generated by the video surveillance system, including panoramic images of the target personnel, as well as associated device information and time information; Visual analysis module: used to perform deep analysis of the panoramic image using a pre-trained large visual model, generating analysis results that include global scene understanding, multi-object detection and association, fine-grained attributes and behavior patterns; Feature modeling module: used to construct multi-dimensional feature vectors based on the deep analysis results, wherein the multi-dimensional feature vectors include target features, behavioral features, scene features and correlation features between targets; Semantic control module: used to establish an abnormal behavior control mechanism based on natural language description, parse the abnormal behavior control mechanism into machine-understandable feature vectors, and form a control strategy; Real-time comparison and early warning module: used to compare the multi-dimensional feature vector with the deployment strategy in real time and generate early warning information that meets the early warning conditions.

[0018] Compared with the prior art, the present invention has the following advantages: 1. This invention, through the deep semantic understanding capabilities of a large visual model, surpasses the limitations of traditional methods based on simple appearance and trajectory, enabling accurate identification and semantic description of complex abnormal behaviors such as "fighting," "illegal gatherings," and "carrying dangerous items." For example, it can understand the complex behavioral pattern of "two or more people, close proximity, continuous physical contact accompanied by drastic posture changes," greatly improving the accuracy of abnormal behavior identification and early warning.

[0019] 2. This invention performs secondary intelligent analysis based on existing structured capture data, avoiding the huge computational overhead caused by directly processing the original massive video stream, significantly reducing the consumption of computing resources and improving the cost-effectiveness of the system.

[0020] 3. This invention significantly improves inference speed while ensuring accuracy by performing lightweight processing such as pruning and quantization on large visual models, making real-time early warning of complex models possible and buying critical time for public security response.

[0021] 4. Based on the generalization ability and semantic control mechanism of the visual large model, this invention allows users to flexibly define complex abnormal behavior rules using natural language. Combined with the dynamic update strategy, this method can quickly adapt to different scenarios (such as subway stations, squares, and night markets) and task requirements, and has good flexibility and scalability.

[0022] 5. This invention can fully utilize the technical advantages of large visual models to achieve a deep understanding of human behavior and objects in video images, improve the accuracy and real-time performance of abnormal event identification, and provide intelligent early warning and analysis methods for public security operations.

[0023] 6. This invention, through an integrated platform for early warning analysis, realizes full-process management from early warning generation to handling feedback, greatly improving the efficiency and collaborative capabilities of public security operations. Attached Figure Description

[0024] Appendix Figure 1 This is a flowchart of the video early warning method based on a large visual model in Embodiment 1 of the present invention; Appendix Figure 2 This is a system block diagram of the video early warning system based on a large visual model in Embodiment 1 of the present invention; Appendix Figure 3 This is a flowchart of the video early warning method based on a large visual model in Embodiment 2 of the present invention; Appendix Figure 4 This is a system block diagram of the video early warning system based on a large visual model in Embodiment 2 of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] Furthermore, in describing representative embodiments, the specification may have presented methods and / or processes as a specific sequence of steps. However, the method or process should not be limited to the specific order of steps described herein, to the extent that it does not depend on such a specific order. As will be understood by those skilled in the art, other sequences of steps are also possible. Therefore, the specific order of steps set forth in the specification should not be construed as a limitation on the embodiments. Moreover, the method and / or process should not be limited to the steps performed in the written order; those skilled in the art will readily understand that these orders can be varied and still remain within the spirit and scope of the embodiments of this application.

[0028] Example 1: This specific embodiment provides a video early warning method based on a large visual model, as shown in the attached figure. Figure 1 As shown, it includes the following steps: Step S1) Data acquisition step: Acquire structured visual data generated by the video surveillance system. The structured visual data includes panoramic images of the target personnel, as well as associated device information and time information.

[0029] In practice, if there is no video surveillance system's view library as a data source, structured visual data can be obtained by periodically extracting frames from real-time video to generate large images.

[0030] Step S2) Visual analysis step: Use a pre-trained large visual model to perform deep analysis on the panoramic image to generate analysis results that include global scene understanding, multi-object detection and association, fine-grained attributes and behavior patterns.

[0031] The foundational model for this solution's large-scale visual model is a Transformer-based model, such as ViT or Swin Transformer. Then, the model is optimized for the characteristics of public security video images, employing domain-adaptive fine-tuning: on a public security surveillance image dataset, masked image modeling (MIM) and contrastive learning are jointly used for optimization, focusing on enhancing the feature reconstruction capabilities for small targets (less than 32×32 pixels) and occluded human bodies, thereby improving the detection capabilities for small and occluded targets.

[0032] Simultaneously, inference speed for large-scale visual models is accelerated. Specifically, channel pruning (30% pruning rate) combined with INT8 quantization is used to improve model inference speed and meet real-time requirements. Pruning is based on the scaling factor sorting of the BN layer, preserving important channels; quantization uses a calibration set for MinMax calibration. Performance is guaranteed through distillation loss constraints: the original model output is used as a soft-label supervised compressed model to ensure that mAP decreases by <1.5%.

[0033] Specifically, scene understanding information is obtained through the following methods: extracting global semantic features from the panoramic image, and identifying the scene type, environmental features, and spatial layout corresponding to the panoramic image based on the global semantic features. Specifically, scene type, environmental features, and spatial layout are identified through MLP classification using image-level [CLS] token output.

[0034] Multi-target detection and association information is obtained by simultaneously detecting and recognizing multiple targets, including faces, bodies, vehicles, and objects, and establishing spatial and semantic associations between different targets.

[0035] Fine-grained attribute information is obtained by using a multi-task inference head structure based on the visual large model InternVL to extract detailed attributes of the local features of the target person, such as clothing features, carried items, and behavioral posture attributes.

[0036] Behavioral patterns are obtained by feeding local features of the target individuals into a static attribute classification model and a temporal action recognition model to identify and classify their behavioral patterns. Specifically, RoI-aligned local features are fed into a dedicated static attribute classification model and a temporal action recognition TimeSformer module to identify and classify behavioral patterns such as walking, running, gathering, and fighting.

[0037] Step S3) Feature modeling step: Based on the deep analysis results, construct a multi-dimensional feature vector, which includes target features, behavioral features, scene features and correlation features between targets.

[0038] The target features are obtained as follows: based on the facial and human features in the deep analysis results, their corresponding original visual feature vectors are extracted; the original visual feature vectors are then standardized to obtain target feature vectors of facial and human features with uniform modulus. Specifically, the original visual feature vectors are extracted by the Transformer backbone of the large visual model and then normalized by L2 to obtain target feature vectors of facial features, human features, and object features with uniform modulus.

[0039] Behavioral features are obtained as follows: image sequences containing the target person are sampled from consecutive video frames using a sliding window approach; the image sequences are encoded using a temporal action recognition model to output temporal feature vectors representing behavior type, action sequence, and interaction mode. Specifically, the temporal action recognition model is a VideoMAE model pre-trained based on a masked autoencoder, which samples consecutive panoramic frames through a sliding window and inputs them into the VideoMAE model to encode temporal feature vectors representing behavior type, action sequence, and interaction mode.

[0040] Scene features are obtained by extracting the global semantic feature vector of the panoramic image, wherein the global semantic feature vector is the feature vector corresponding to the [CLS] label output by the visual encoder of the large visual model.

[0041] Association features are obtained through the following steps: Spatial relationship modeling: Calculate the spatial geometric metrics between different targets, including the intersection-over-union ratio (IoU) of the target bounding boxes, the distance between the center points, and the relative azimuth angle; Semantic relation modeling: Using graph neural networks (GNNs) to model the category associations and behavioral interactions between targets; Temporal relationship modeling: Calculate the trajectory similarity (DTW) between different target motion trajectories to measure the stability of spatiotemporal co-occurrence of different targets; Feature fusion: The spatial geometric metrics, the output of the graph neural network, and the trajectory similarity are concatenated to generate a comprehensive relational feature vector. This generated relational feature vector is input to the real-time comparison and early warning module to assist in early warning judgment.

[0042] Specifically, the feature modeling step further includes a feature alignment step, which involves mapping the target features, behavioral features, and scene features to a common semantic space of the same preset dimension through a shared projection layer, so as to achieve scale alignment and semantic unification of cross-modal features. The preset dimension is 512 dimensions.

[0043] Step S4) Semantic control step: Establish an abnormal behavior control mechanism based on natural language description, parse the abnormal behavior control mechanism into machine-understandable feature vectors, and form a control strategy.

[0044] Specifically, the semantic control steps include semantic parsing, control strategy definition, and dynamic control update.

[0045] The semantic parsing step involves using a pre-trained language model to parse the deployment conditions described in natural language into machine-understandable feature vectors, i.e., semantic vectors.

[0046] The deployment strategy definition steps support defining multi-level deployment logic rules that include time, space, number of targets, behavior type, and duration. Specifically, this step supports setting differentiated deployment strategies for different levels and types of abnormal events. For example, the deployment strategy for "suspected fighting" is defined as: two or more people having continuous physical contact within 3 meters for more than 2 seconds, accompanied by rapid displacement and drastic changes in posture (such as swinging arms or falling to the ground); "illegal gathering" is defined as: a sudden increase in the density of people in the same area within 5 minutes without normal movement paths. Deployment strategies are expressed using a combination of natural language templates and logical rules.

[0047] Dynamic deployment update steps: Based on manual assessment feedback or preset spatiotemporal context rules, dynamically adjust the matching threshold or activation status of deployment conditions. Specifically, this step can dynamically adjust deployment conditions and parameters according to actual combat needs and early warning effects. Dynamic deployment update adopts a dual mechanism: First, online fine-tuning is triggered based on manual assessment feedback (confirmation / false alarm) to adjust semantic vector thresholds; second, based on spatiotemporal context rules (such as automatically increasing "loitering" sensitivity at night), specific deployment items are activated / suppressed in real time through a preset strategy engine.

[0048] This involves constructing a triplet that includes entities (personnel, vehicles, locations), relationships (traveling together, entering and exiting, possessing), and events (fighting, theft, gathering), such as "Person A - carrying - knife" and "Subway station B - high incidence - pickpocketing," etc., to provide domain knowledge support.

[0049] Meanwhile, it provides commonly used templates for monitoring abnormal behaviors, including: "long-term stay (>10 minutes) + frequent looking around", "multiple people surrounding a single person + physical conflict", "carrying a bucket-shaped object into the square", and "climbing over the isolation fence", to simplify the monitoring operation.

[0050] Step S5) Real-time comparison and early warning step: The multi-dimensional feature vector is compared with the deployment strategy in real time to generate early warning information that meets the early warning conditions.

[0051] Specifically, the real-time comparison and early warning process includes multi-dimensional similarity calculation, adaptive threshold adjustment, and early warning information generation.

[0052] The multi-dimensional similarity calculation involves calculating the similarity between real-time features and the deployment strategy across multiple feature dimensions, then performing weighted fusion to obtain a comprehensive matching score. These multiple feature dimensions include target attributes, behavioral patterns, scene context, and relational dimensions. Specifically, similarity calculation can employ various methods such as cosine similarity and Euclidean distance for feature matching. This step performs similarity calculation and comprehensive evaluation across multiple feature dimensions. Simultaneously, after independently calculating the cosine similarity for each dimension, the results are weighted and fused through a learnable weighted gating network, and then output as a comprehensive matching score via a sigmoid function. This score is used to compare with an adaptive threshold to determine whether an alert should be issued.

[0053] Adaptive Threshold Adjustment: Based on statistical feedback from historical warnings, the threshold for warning determination is dynamically adjusted. Specifically, based on the true positive rate (TPR) and false positive rate (FPR) of historical warnings within a sliding time window, the threshold is dynamically adjusted using an exponential smoothing algorithm. When the FPR increases, the threshold rises; when the TPR decreases, the threshold falls, ensuring that the system maintains Pareto optimality between accuracy and recall.

[0054] Warning Information Generation: Warning information is generated for targets whose matching scores exceed the threshold. This warning information includes target identity attribute tags, behavioral tags, scene type, spatiotemporal coordinates, device information, time information, and relationships with other targets. Specifically, for targets meeting the warning conditions, a warning message is generated containing multi-dimensional feature information from structured fields such as target identity attribute tags (e.g., clothing, carried items), behavioral tags (e.g., running, fighting), scene type (e.g., subway station, night market), spatiotemporal coordinates, and interaction relationships with other targets (e.g., distance, co-occurrence duration).

[0055] To improve matching accuracy, multimodal fusion technology is also used in this step. By using cross-modal attention fusion, text features are used as queries and visual region features are used as keys / values. After calculating attention weights, visual information is aggregated in a weighted manner and then connected with the residuals of the original text features to form a semantically enhanced joint representation, thereby improving matching accuracy.

[0056] In addition, incremental learning is performed in this step. Based on the feedback results, the manual judgment results (effective / ineffective) of each warning are recorded to construct a feedback sample set; the incremental learning process is triggered monthly to fine-tune the feature space with Triplet Loss, bringing effective samples closer and pushing away ineffective samples, thereby continuously optimizing the comparison accuracy.

[0057] In addition, this specific embodiment also provides a system for implementing the above-mentioned video early warning method based on a large visual model, as shown in the attached figure. Figure 2 As shown, it includes a data acquisition module, a visual analysis module, a feature modeling module, a semantic control module, and a real-time comparison and early warning module.

[0058] The data acquisition module is used to acquire structured visual data generated by the video surveillance system. The structured visual data includes panoramic images of the target personnel, as well as associated equipment information, time information, and preliminary structured results. The visual analysis module is used to perform deep analysis on the panoramic image using a pre-trained large visual model, generating analysis results that include global scene understanding, multi-object detection and association, fine-grained attributes and behavior patterns; The feature modeling module is used to construct a multi-dimensional feature vector based on the deep analysis results. The multi-dimensional feature vector includes target features, behavioral features, scene features, and correlation features between targets. The semantic control module is used to establish an abnormal behavior control mechanism based on natural language description, parse the abnormal behavior control mechanism into machine-understandable feature vectors, and form a control strategy. The real-time comparison and early warning module is used to compare the multi-dimensional feature vector with the deployment strategy in real time and generate early warning information that meets the early warning conditions.

[0059] Example 2: The video early warning method based on a large visual model in this specific embodiment is shown in the appendix. Figure 3 As shown, it includes the following steps: Data acquisition steps: Acquire structured visual data generated by the video surveillance system, including panoramic images of the target personnel, as well as associated device information and time information; Visual analysis steps: The panoramic image is subjected to deep analysis using a pre-trained large visual model to generate analysis results that include global scene understanding, multi-object detection and association, fine-grained attributes and behavior patterns; Feature modeling steps: Based on the deep analysis results, construct a multi-dimensional feature vector, which includes target features, behavioral features, scene features, and correlation features between targets; Semantic control steps: Establish an abnormal behavior control mechanism based on natural language description, parse the abnormal behavior control mechanism into machine-understandable feature vectors, and form a control strategy; Real-time comparison and early warning steps: The multi-dimensional feature vector is compared with the deployment strategy in real time to generate early warning information that meets the early warning conditions; Early warning assessment and response steps: Aggregate and display the generated early warning information, perform multi-source information correlation analysis, and provide visualization tools to support manual assessment and response process management.

[0060] Specifically, this step involves unified analysis and handling at the early warning center. The center provides an intuitive visual interface for displaying and operating early warning information, while utilizing AI technology to offer intelligent assistance suggestions for manual analysis and supporting collaborative analysis and handling by multiple police units and departments. This includes: Aggregated display of early warning information: Aggregated display of early warning information for the same event or related events.

[0061] Multi-source information correlation analysis: The early warning event is matched with the face database, checkpoint vehicle records, and police information database in terms of spatiotemporal intersection and graph path to identify the suspect's trajectory, accomplice association, or prior criminal behavior and conduct correlation analysis with other data sources.

[0062] Assistance for manual analysis: Provides features including spatiotemporal heatmaps, multi-target trajectory playback, behavioral time-series graphs, and correlation diagrams. Supports one-click retrieval of related video clips, attribute comparison panels, and similar case recommendations, among other visualization tools and analytical methods, to assist police officers in conducting manual analysis.

[0063] Handling process management: Supports the hierarchical handling, flow tracking and result feedback of early warning information.

[0064] In addition, this specific embodiment also provides a system for implementing the above-mentioned video early warning method based on a large visual model, as shown in the attached figure. Figure 4 As shown, it includes a data acquisition module, a visual analysis module, a feature modeling module, a semantic control module, a real-time comparison and early warning module, and an early warning analysis and handling module.

[0065] The data acquisition module is used to acquire structured visual data generated by the video surveillance system. The structured visual data includes panoramic images of the target personnel, as well as associated equipment information, time information, and preliminary structured results. The visual analysis module is used to perform deep analysis on the panoramic image using a pre-trained large visual model, generating analysis results that include global scene understanding, multi-object detection and association, fine-grained attributes and behavior patterns; The feature modeling module is used to construct a multi-dimensional feature vector based on the deep analysis results. The multi-dimensional feature vector includes target features, behavioral features, scene features, and correlation features between targets. The semantic control module is used to establish an abnormal behavior control mechanism based on natural language description, parse the abnormal behavior control mechanism into machine-understandable feature vectors, and form a control strategy. The real-time comparison and early warning module is used to compare the multi-dimensional feature vector with the deployment strategy in real time and generate early warning information that meets the early warning conditions.

[0066] The early warning analysis and response module is used to aggregate and display the generated early warning information, perform multi-source information correlation analysis, and provide visualization tools to support manual analysis and response process management.

[0067] Compared with existing technologies, this invention, through the deep semantic understanding capabilities of a large visual model, transcends the limitations of traditional methods based on simple appearance and trajectory, enabling accurate identification and semantic description of complex abnormal behaviors such as "fighting," "illegal gatherings," and "carrying dangerous items." For example, it can understand the complex behavioral pattern of "two or more people, close-range, continuous physical contact accompanied by drastic posture changes," greatly improving the accuracy of abnormal behavior identification and early warning.

[0068] This invention performs secondary intelligent analysis based on existing structured capture data, avoiding the huge computational overhead caused by directly processing the original massive video stream, significantly reducing computing resource consumption, and improving the system's cost-effectiveness.

[0069] This invention significantly improves inference speed while maintaining accuracy by performing lightweight processing such as pruning and quantization on large visual models, making real-time early warning of complex models possible and buying critical time for public security response.

[0070] Based on the generalization ability and semantic control mechanism of large visual models, this invention allows users to flexibly define complex abnormal behavior rules using natural language. Combined with a dynamic update strategy, this method can quickly adapt to different scenarios (such as subway stations, squares, and night markets) and task requirements, and has good flexibility and scalability.

[0071] This invention can fully utilize the technical advantages of large visual models to achieve a deep understanding of human behavior and objects in video images, improve the accuracy and real-time performance of abnormal event identification, and provide intelligent early warning and analysis methods for public security operations.

[0072] This invention, through an integrated platform for early warning analysis, enables full-process management from early warning generation to response and feedback, greatly improving the efficiency and collaborative capabilities of public security operations.

[0073] In addition, the present invention also provides a computer storage medium storing executable instructions, which, when executed by a processor, enable the processor to execute the aforementioned video warning method based on a large visual model.

[0074] In addition, the present invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied as a computer storage medium; in another optional embodiment, the computer program product is specifically embodied as a software product, such as a Software Development Kit (SDK), etc.

[0075] Furthermore, in this embodiment, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0076] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0077] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0078] This invention is described with reference to schematic and / or block diagrams illustrating the implementation of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the schematic and / or block diagrams can be implemented by computer program instructions, and combinations of blocks in the schematic and / or block diagrams can be implemented. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the schematic and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 Devices that specify the functions in one or more boxes.

[0079] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction device, which is implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0080] These computer program instructions can also be loaded onto a computer or other programmable data processing equipment, causing a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A video early warning method based on a large visual model, characterized in that, Includes the following steps: Data acquisition steps: Acquire structured visual data generated by the video surveillance system, including panoramic images of the target personnel, as well as associated device information and time information; Visual analysis steps: The panoramic image is subjected to deep analysis using a pre-trained large visual model to generate analysis results that include global scene understanding, multi-object detection and association, fine-grained attributes and behavior patterns; Feature modeling steps: Based on the deep analysis results, construct a multi-dimensional feature vector, which includes target features, behavioral features, scene features, and correlation features between targets; Semantic control steps: Establish an abnormal behavior control mechanism based on natural language description, parse the abnormal behavior control mechanism into machine-understandable feature vectors, and form a control strategy; Real-time comparison and early warning steps: The multi-dimensional feature vector is compared with the deployment strategy in real time to generate early warning information that meets the early warning conditions.

2. The video early warning method based on a large visual model according to claim 1, characterized in that, In the visual analysis step, global scene understanding information is obtained in the following way: extracting global semantic features of the panoramic image, and based on the global semantic features, identifying the scene type, environmental features and spatial layout corresponding to the panoramic image; Multi-target detection and association information is obtained by simultaneously detecting and recognizing multiple targets, including faces, bodies, vehicles, and objects, and establishing spatial and semantic associations between different targets. Fine-grained attribute information is obtained in the following way: the clothing features, carried items, and behavioral posture attributes of the target person are extracted using a multi-task inference head structure based on a large visual model; Behavioral patterns are obtained by feeding local features of the target personnel into a static attribute classification model and a temporal action recognition model to identify and classify the personnel's behavioral patterns.

3. The video early warning method based on a large visual model according to claim 1, characterized in that, In the feature modeling step, the target features are obtained in the following way: based on the face features and body features in the deep parsing results, the corresponding original visual feature vectors are extracted; the original visual feature vectors are standardized to obtain target feature vectors of face features and body features with uniform modulus. Behavioral features are obtained by sampling image sequences containing the target person from consecutive video frames using a sliding window approach; encoding the image sequences using a temporal action recognition model to output temporal feature vectors representing behavior type, action sequence, and interaction mode; Scene features are obtained by extracting the global semantic feature vector of the panoramic image, wherein the global semantic feature vector is the feature vector corresponding to the [CLS] label output by the visual encoder of the large visual model.

4. The video early warning method based on a large visual model according to claim 3, characterized in that, In the feature modeling step, the associated features are obtained through the following steps: Spatial relationship modeling: Calculate the spatial geometric metrics between different targets, including the intersection-union ratio of the target bounding boxes, the distance between the center points, and the relative azimuth angle; Semantic relation modeling: Using graph neural networks, modeling the category associations and behavioral interactions between targets; Temporal relationship modeling: Calculate the trajectory similarity between different target motion trajectories to measure the stability of spatiotemporal co-occurrence of different targets; Feature fusion: The spatial geometric measure, the output of the graph neural network, and the trajectory similarity are concatenated to generate a comprehensive relational feature vector.

5. The video early warning method based on a large visual model according to claim 4, characterized in that, The feature modeling step also includes a feature alignment step, which includes mapping the target features, behavioral features, and scene features to a common semantic space of the same preset dimension through a shared projection layer, so as to achieve scale alignment and semantic unification of cross-modal features.

6. The video early warning method based on a large visual model according to claim 1, characterized in that, Semantic control steps include: Semantic parsing: Using a pre-trained language model, the deployment conditions described in natural language are parsed into machine-understandable feature vectors; Deployment strategy definition: Supports defining multi-level deployment logic rules that include time, space, number of targets, behavior type, and duration; Dynamic deployment update: Based on manual analysis and feedback or preset spatiotemporal context rules, dynamically adjust the matching threshold or activation status of deployment conditions.

7. The video early warning method based on a large visual model according to claim 1, characterized in that, Real-time comparison and early warning steps include: Multi-dimensional similarity calculation: The similarity between real-time features and deployment strategies is calculated on multiple feature dimensions, and then weighted and fused to obtain a comprehensive matching score. The multiple feature dimensions include target attributes, behavior patterns, scene context, and association relationship dimensions. Adaptive threshold adjustment: Based on statistical feedback from historical warnings, dynamically adjust the threshold for warning determination; Warning information generation: For targets whose matching scores exceed the threshold, warning information is generated. The warning information includes target identity attribute tags, behavior tags, scene type, spatiotemporal coordinates, device information, time information, and association with other targets.

8. The video early warning method based on a large visual model according to claim 7, characterized in that, The adaptive threshold adjustment method is as follows: based on the true positive rate and false positive rate of historical early warnings within the sliding time window, the threshold is dynamically adjusted using an exponential smoothing algorithm.

9. The video early warning method based on a large visual model according to claim 1, characterized in that, The video early warning method also includes early warning assessment and handling steps, which include: aggregating and displaying the generated early warning information, performing multi-source information correlation analysis, and providing visualization auxiliary tools to support manual assessment and handling process management.

10. A system for implementing the video early warning method based on a large visual model as described in claim 1, characterized in that, include: Data acquisition module: used to acquire structured visual data generated by the video surveillance system, including panoramic images of the target personnel, as well as associated device information and time information; Visual analysis module: used to perform deep analysis of the panoramic image using a pre-trained large visual model, generating analysis results that include global scene understanding, multi-object detection and association, fine-grained attributes and behavior patterns; Feature modeling module: used to construct multi-dimensional feature vectors based on the deep analysis results, wherein the multi-dimensional feature vectors include target features, behavioral features, scene features and correlation features between targets; Semantic control module: used to establish an abnormal behavior control mechanism based on natural language description, parse the abnormal behavior control mechanism into machine-understandable feature vectors, and form a control strategy; Real-time comparison and early warning module: used to compare the multi-dimensional feature vector with the deployment strategy in real time and generate early warning information that meets the early warning conditions.

Images