An intelligent violation behavior detection system and method based on a visual large model

Abstract

The application discloses a kind of based on visual large model's rule violation intelligent detection system and method, the system includes multi-source video access module, video frame pre-processing and scheduling module, real-time preview and marking module, offline detection module, visual large model inference engine, detection result management module, user authentication and authority management module, operation data statistics module and system upgrade management module.The application is encoded to single frame image by visual large model, rule violation classification and target positioning, and high-precision rule violation identification is realized;Support four kinds of video sources, such as USB, HDMI, RTSP and RTMP, unified access;Support real-time detection and offline detection dual mode.The application effectively solves the technical problems that the feature expression ability of detection model is limited, the multi-source video access ability is insufficient, the system function is incomplete and the authority management is weak in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of intelligent video surveillance and computer vision technology, specifically relating to a single-frame image violation detection system and method based on a large visual model, which can be applied to the automatic identification and early warning of violations in scenarios such as safety production supervision, traffic management, and public security. Background Technology

[0002] With the widespread deployment of video surveillance equipment, violation detection based on video analytics has become an important tool in fields such as workplace safety, traffic management, and public security. Currently, traditional violation detection solutions mainly fall into the following categories: 1. Detection scheme based on fixed rules Early violation detection systems analyzed video footage by manually setting detection rules (such as area intrusion detection and tripwire detection). This approach relied heavily on pre-defined rules, making it difficult to handle complex and ever-changing real-world scenarios, resulting in high false positive and false negative rates.

[0003] 2. Detection scheme based on a dedicated model Recent technical solutions employ pre-trained deep learning models (such as YOLO series object detection models, Faster R-CNN, convolutional neural networks, etc.) to extract features and identify violations in video frames. For example, some existing technologies extract key points of objects and human bodies in the scene using object detection models, and then determine whether a violation exists based on preset rules. While such solutions offer improvements over rule-based methods, the models have limited feature representation capabilities, insufficient ability to handle blur, occlusion, lighting changes, and generalization to different scenes, and require separate model training for each violation type, resulting in high maintenance costs.

[0004] 3. Single-path video access solution Existing video surveillance systems typically only support a single video access method (such as only supporting RTSP streams or only supporting USB cameras), which limits the applicability and deployment flexibility of the system.

[0005] 4. Separation of testing and business management Existing technical solutions mostly focus on the detection algorithm itself, lacking comprehensive supporting functions such as user permission management, automatic generation of detection reports, and remote system upgrades. This results in high maintenance costs and weak data security management in actual system deployment.

[0006] In summary, the existing technology has the following shortcomings: limited feature representation capabilities of the detection model, insufficient multi-source video access capabilities, incomplete system functions, lack of flexible user permission management mechanisms, and lack of automated report generation capabilities. Summary of the Invention

[0007] The technical problem to be solved by this invention is to provide an intelligent detection system and method for violations based on a large visual model, which aims to solve the technical problems of limited feature expression capability of the detection model, insufficient multi-source video access capability, incomplete system functions, lack of flexible user permission management mechanism and automated report generation capability in the prior art.

[0008] This invention is implemented as follows: Firstly, a violation detection system based on a large visual model includes: The multi-source video access module is used to connect to video sources with different protocols and interfaces, including a protocol adaptation submodule and a video stream decoding submodule; The video frame preprocessing and scheduling module is used for video decoding, frame extraction, and image quality enhancement, including a frame extraction scheduling submodule and an image preprocessing submodule. The real-time preview and annotation module is used for real-time rendering of video footage and visual annotation of violations, including a video rendering submodule and a violation annotation submodule. The offline detection module is used to detect uploaded offline video files, and includes a file upload submodule and an offline task scheduling submodule. The visual large model inference engine is used for intelligent identification of violations in single-frame input images. It includes an image encoding submodule, a violation classification submodule, a target localization submodule, a confidence evaluation submodule, and a model management and update submodule. The test result management module is used for storing, querying, and generating test data, and includes a result storage submodule, a query retrieval submodule, and a report generation submodule. The user authentication and permission management module manages users, roles, and permissions based on the RBAC model. The operational data statistics module is used for the statistical analysis and visualization of system operation data. The system upgrade management module is used for remote online system upgrades.

[0009] Furthermore, the visual big model inference engine implements single-frame image violation detection based on the visual big model, which is a backbone network model with visual feature representation capability and zero-sample transfer capability obtained by pre-training with massive image data.

[0010] Furthermore, the offline detection module supports users uploading local video files. The upload process uses segmented breakpoint resume technology, and the supported file formats include MP4, AVI, MOV, and MKV.

[0011] Furthermore, the user authentication and permission management module adopts the RBAC model, which includes a user management submodule, a role management submodule, and a permission allocation submodule. It abstracts system functionalities into permission resources and supports granting permissions to roles and granting roles to users.

[0012] Furthermore, the system upgrade management module supports remote online updates of the visual large model weight file, including a version detection submodule and an upgrade execution submodule. The upgrade process supports breakpoint resume and upgrade rollback.

[0013] Secondly, the present invention provides an intelligent detection method for violations based on a large visual model, applied to the system described in the first aspect, characterized by comprising the following steps: Step S1: User login and permission verification. The user authentication and permission management module verifies the user's identity. After successful verification, a JWT token is generated and returned. Step S2: The multi-source video access module performs protocol adaptation and video stream decoding on the video source, and the real-time preview and annotation module begins to render the video frame; Step S3: The video frame preprocessing and scheduling module extracts frames from each video stream according to a preset frame rate, and sends the extracted images to the visual large model inference engine after preprocessing. Step S4: The visual large model inference engine performs image encoding, violation classification, and target localization on the input image, and outputs structured detection results; Step S5: If a violation is detected, the real-time preview and annotation module marks the violation area on the screen and displays the violation type and confidence level, while the detection result is stored in the database; Step S6: Users can retrieve historical test records by criteria through the test result management module. The system automatically generates a test report, which users can then export as Word or PDF format.

[0014] Furthermore, in step S3, each video stream is frame-sampling at a preset frame rate, and a multi-threaded concurrent scheduling strategy is adopted for multiple video streams.

[0015] Furthermore, the reasoning process of the visual large model inference engine in step S4 includes: encoding the input image into a high-dimensional visual feature vector through an image encoder; classifying violations based on the visual feature vector and outputting the violation type; locating the violation target based on the feature map or attention map of the visual large model and outputting the bounding box coordinates; and outputting the confidence score of the detection result.

[0016] Furthermore, in step S4, the violation classification of the visual large model inference engine supports rapid adaptation using a small number of example images to fit the specific violation definition of the scenario.

[0017] Furthermore, in step S5, when the confidence level is lower than a preset threshold, the detection result is marked as pending manual review.

[0018] The advantages of this invention are: 1. Detection accuracy and generalization ability are significantly improved. This invention uses a large visual model as the core image encoder. This model obtains powerful visual feature representation capabilities through pre-training on massive image data. Compared with existing solutions based on dedicated models such as YOLO and CNN, the detection accuracy is significantly improved under complex conditions such as drastic changes in lighting, target occlusion, and migration between different scenes. It also has strong zero-shot / few-shot generalization ability and does not require retraining the model for each new scene.

[0019] 2. Fine-grained violation detection capability The high-dimensional feature representation of the visual big model can capture fine-grained semantic information in the image (such as the color of the safety helmet, details of the protective clothing, and the posture of the tool being used), supporting the identification of more refined violation rules.

[0020] 3. Unified access to multi-source video This invention supports four mainstream video access methods: USB, HDMI, RTSP, and RTMP. It can simultaneously access multiple video streams for parallel detection, and is applicable to various existing monitoring infrastructures. It does not require the replacement of hardware equipment and has low deployment costs.

[0021] 4. Real-time and offline dual-mode detection This invention supports both real-time video stream detection and offline video file detection, meeting both real-time monitoring needs and post-event traceability analysis, making its application scenarios more comprehensive.

[0022] 5. Standardized management of test results This invention provides complete functions for storing, retrieving, and automatically generating test results, and supports exporting standard test reports in Word and PDF formats, solving the problems of extensive test result management and reliance on manual report creation in existing solutions.

[0023] 6. Online model update capability The visual large model weight file of the present invention can be remotely updated online through the system upgrade module, which can continuously improve the detection capability without on-site deployment.

[0024] 7. Fine-grained access control This invention uses the RBAC model to implement three-level management of users, roles, and permissions, supporting flexible permission allocation and adjustment, and meeting the compliance requirements of enterprises for data security and operational auditing. Attached Figure Description

[0025] The present invention will now be further described with reference to the accompanying drawings and embodiments.

[0026] Figure 1 This is a schematic diagram of the overall architecture of the system of the present invention; Figure 2 This is a business process diagram of the method of the present invention. Detailed Implementation

[0027] like Figure 1 As shown, the intelligent violation detection system based on a large visual model of the present invention consists of the following core modules: a multi-source video access module, a video frame preprocessing and scheduling module, a real-time preview and annotation module, an offline detection module, a large visual model inference engine, a detection result management module, a user authentication and permission management module, an operational data statistics module, and a system upgrade management module.

[0028] I. Multi-source video access module This module includes a protocol adaptation submodule and a video stream decoding submodule.

[0029] The protocol adapter submodule encapsulates USB Video Class (UVC) drivers, HDMI video capture card drivers, RTSP clients, and RTMP clients, enabling unified access to video sources with different protocols.

[0030] The video stream decoding submodule uses the FFmpeg decoding library to decode video streams of different formats, convert them into YUV format frame data, and then convert them into RGB format for subsequent processing.

[0031] The output of this module is connected to the video frame preprocessing and scheduling module, as well as the real-time preview and annotation module.

[0032] II. Video Frame Preprocessing and Scheduling Module This module includes a frame-skipping scheduling submodule and an image preprocessing submodule.

[0033] The frame extraction scheduling submodule extracts frames from each video stream at a preset frame rate (e.g., 1-2 frames per second), and puts the extracted image frames into the detection queue. It adopts a multi-threaded concurrent scheduling strategy for multiple video streams to avoid single-channel blocking from affecting the detection of other channels.

[0034] The image preprocessing submodule performs preprocessing operations such as size normalization (uniformly scaling to the input size required by the large visual model, such as 448×448 or 640×640), contrast adaptive enhancement, and noise reduction on the images obtained by frame extraction, thereby improving the quality of the input image.

[0035] The input of this module is connected to the multi-source video access module, and the output is connected to the visual large model inference engine.

[0036] III. Real-time Preview and Annotation Module This module includes a screen rendering submodule and a violation labeling submodule.

[0037] The screen rendering submodule is based on GPU-accelerated rendering and supports simultaneous display of multiple video feeds, with a maximum of 16 split-screen displays per screen.

[0038] The violation labeling submodule receives the detection results (violation type, bounding box coordinates of the violation target, and confidence score) returned by the visual big model inference engine, and visually labels the violation behavior on the original video frame using rectangular boxes, semi-transparent overlays, and label text, with different colors used to distinguish different violation types.

[0039] The input end of this module is connected to the multi-source video access module and the visual large model inference engine, and the output end is connected to the display device.

[0040] IV. Offline Detection Module This module includes a file upload submodule and an offline task scheduling submodule.

[0041] The file upload submodule supports users uploading local video files (supported formats include but are not limited to MP4, AVI, MOV, and MKV), and the upload process uses segmented breakpoint resume technology.

[0042] The offline task scheduling submodule demultiplexes the uploaded video file and extracts frames in the frame sequence. Each frame is then sent to the visual big model inference engine for detection. After the detection is completed, the results are summarized, stored in the database, and the user is notified.

[0043] This module is bidirectionally connected to the visual large model inference engine and the detection result management module.

[0044] V. Visual Large Model Inference Engine This engine is the core innovation of this invention, achieving single-frame image violation detection based on a Visual Foundation Model (VFM). A Visual Foundation Model refers to a backbone network model (such as a large-scale visual model based on the Transformer architecture) pre-trained with massive amounts of image data, possessing powerful visual feature representation capabilities and zero-sample transfer capabilities. It comprises the following sub-modules: Image encoding submodule: The input preprocessed image is encoded into a high-dimensional visual feature vector by the image encoder of the large visual model. This encoder has hundreds of millions to billions of parameters and has obtained powerful visual semantic understanding capabilities through pre-training on massive amounts of data.

[0045] Violation Classification Submodule: Based on a large visual model classification head or through linear probing, this module maps the visual feature vectors output by the image encoder to output whether violations exist in the image and the specific category of the violation. This classification head allows users to quickly adapt it using a small number of sample images to fit the specific violation definition for the given scenario.

[0046] Target localization submodule: Based on the feature map of the large visual model or with the help of a lightweight localization head (such as an attention map based on ViT), the module locates the position of the illegal target in the image and outputs the bounding box coordinates (x_min, y_min, x_max, y_max) of the illegal target.

[0047] Confidence assessment submodule: Outputs the confidence score of the detection results (between 0 and 1). When the confidence score is lower than the preset threshold, the result is marked as pending manual review.

[0048] Model Management and Update Submodule: Manages the version and weight files of large visual models, and supports online updates and iterations of models through the upgrade management module.

[0049] The engine takes a single-frame RGB image as input and outputs structured detection results (JSON format), which include information such as violation type, bounding box coordinates, and confidence score. It communicates with various modules through an API interface.

[0050] VI. Test Result Management Module This module includes a results storage submodule, a query retrieval submodule, and a report generation submodule.

[0051] The results storage submodule stores the detection results (including screenshots of violations, video clips of violations, violation type, confidence level, timestamp, camera number, etc.) into the database.

[0052] The query and retrieval submodule supports multi-condition combined retrieval based on dimensions such as time range, violation type, camera number, and processing status.

[0053] The report generation submodule automatically populates the detection results into a detection report based on a preset report template. It supports exporting to Word (.docx) and PDF formats. The report content includes a summary of detection statistics, a list of violations, and screenshot attachments of violations.

[0054] VII. User Authentication and Access Control Module This module adopts the RBAC (Role-Based Access Control) model and includes the following sub-modules: The user management submodule supports the creation, disabling, deletion, and information modification of users. User authentication uses the JWT token mechanism and supports encrypted password storage.

[0055] The role management submodule supports the custom creation and deletion of system roles, with pre-defined roles such as administrator, auditor, and general operator.

[0056] The permission allocation submodule abstracts system functions (such as "viewing camera footage", "exporting detection reports", "managing users") into permission resources, supporting the granting of permissions to roles and the granting of roles to users, thereby achieving fine-grained access control.

[0057] This module is coupled with other functional modules of the system through a permission interceptor, which performs permission verification when a user accesses any function.

[0058] VIII. Operational Data Statistics Module This module includes a data acquisition submodule and a visualization submodule.

[0059] The data acquisition submodule collects data in real time, including the number of detection tasks, statistics of detection results (distribution of violation types, trend of violation quantity), user operation logs, system operating status, and other data.

[0060] The visualization submodule displays the above statistical data on the system dashboard page using charts (line charts, bar charts, pie charts, etc.), and supports data filtering and aggregation display by day, week, month, and custom time interval.

[0061] IX. System Upgrade Management Module This module includes a version detection submodule and an upgrade execution submodule.

[0062] The version detection submodule periodically or manually sends a version query request to the upgrade server to obtain the latest version information and compare it with the current version.

[0063] Upon detecting a new version, the upgrade execution submodule downloads the upgrade package using either incremental or full upgrade methods. After verifying the integrity of the upgrade package, it executes the upgrade operation. The upgrade process supports resuming interrupted downloads and rollback. The upgrade content includes system functional modules and the visual large model weight file.

[0064] like Figure 2 As shown, the operation process of this invention mainly includes the following steps: Step S1: User Login and Permission Verification Users initiate a login request by entering their account and password through the client (web or desktop). The user authentication and permission management module verifies the user's identity. After successful verification, a JWT token is generated and returned. Subsequent user requests will carry this token for permission verification. Step S2: Video source access and real-time preview Users select the video access method (USB, HDMI, RTSP or RTMP), configure the corresponding parameters (such as RTSP URL, port, etc.), the multi-source video access module completes protocol adaptation and video stream decoding, and the real-time preview and annotation module begins to render the video image; Step S3: Frame extraction and large visual model detection (real-time mode) The video frame preprocessing and scheduling module extracts frames from each video stream at a preset frame rate (e.g., 1 frame per second). The extracted images are preprocessed and then fed into the visual large-scale model inference engine. The visual large-scale model inference engine encodes the input images, classifies violations, and locates targets, outputting detection results. If a violation is detected, the real-time preview and annotation module outlines the violation area on the screen and displays the violation type and confidence level, while simultaneously storing the detection results in the database. Step S4: Offline Detection Mode Users upload video files through the offline detection module. The system adds the files to the task queue, demultiplexes the video, extracts each frame, and sends each frame to the visual big model inference engine for detection. After completion, the system stores the summarized detection results in the database and notifies the user. The user can view the results in the detection result management module. Step S5: Report Generation and Export Users can access the test results management module, search for historical test records by criteria, select the records to be included in the report, choose a report template, and the system will automatically generate the test report. Users can choose to export it as a Word or PDF file. Step S6: View Operational Data Users with the appropriate permissions can access the operational data statistics module to view the system's operational overview, violation trend analysis, and statistical charts of various dimensions. The system also supports filtering by time interval and data export. Step S7: System Upgrade The administrator enters the system upgrade management module, clicks "Check for updates," and the system queries the upgrade server to obtain version information. If a new version is available, the update content (including feature updates and visual large model weight updates) will be displayed. After the administrator confirms, the upgrade will be executed.

[0065] The above embodiments and figures are not intended to limit the product form and style of the present invention. Any appropriate changes or modifications made by those skilled in the art should be considered as not departing from the patent scope of the present invention.

Claims

1. A violation detection system based on a large visual model, characterized in that, include: The multi-source video access module is used to connect to video sources with different protocols and interfaces, including a protocol adaptation submodule and a video stream decoding submodule; The video frame preprocessing and scheduling module is used for video decoding, frame extraction, and image quality enhancement, including a frame extraction scheduling submodule and an image preprocessing submodule. The real-time preview and annotation module is used for real-time rendering of video footage and visual annotation of violations, including a video rendering submodule and a violation annotation submodule. The offline detection module is used to detect uploaded offline video files, and includes a file upload submodule and an offline task scheduling submodule. The visual large model inference engine is used for intelligent identification of violations in single-frame input images. It includes an image encoding submodule, a violation classification submodule, a target localization submodule, a confidence evaluation submodule, and a model management and update submodule. The test result management module is used for storing, querying, and generating test data, and includes a result storage submodule, a query retrieval submodule, and a report generation submodule. The user authentication and permission management module manages users, roles, and permissions based on the RBAC model. The operational data statistics module is used for the statistical analysis and visualization of system operation data. The system upgrade management module is used for remote online system upgrades.

2. The intelligent detection system for violations based on a large visual model according to claim 1, characterized in that, The visual big model inference engine is based on a visual big model to detect violations in a single frame image. The visual big model is a backbone network model that is pre-trained with massive image data and has visual feature representation capabilities and zero-sample transfer capabilities.

3. The intelligent detection system for violations based on a large visual model according to claim 1, characterized in that, The offline detection module supports users uploading local video files. The upload process uses segmented breakpoint resume technology, and the supported file formats include MP4, AVI, MOV and MKV.

4. The intelligent detection system for violations based on a large visual model according to claim 1, characterized in that, The user authentication and permission management module adopts the RBAC model and includes a user management submodule, a role management submodule, and a permission allocation submodule. It abstracts system functions into permission resources and supports granting permissions to roles and roles to users.

5. The intelligent detection system for violations based on a large visual model according to claim 1, characterized in that, The system upgrade management module supports remote online updates of visual large model weight files, including a version detection submodule and an upgrade execution submodule. The upgrade process supports breakpoint resume and upgrade rollback.

6. A method for intelligent detection of violations based on a large visual model, applied to the system described in any one of claims 1-5, characterized in that, Includes the following steps: Step S1: User login and permission verification. The user authentication and permission management module verifies the user's identity. After successful verification, a JWT token is generated and returned. Step S2: The multi-source video access module performs protocol adaptation and video stream decoding on the video source, and the real-time preview and annotation module begins to render the video frame; Step S3: The video frame preprocessing and scheduling module extracts frames from each video stream according to a preset frame rate, and sends the extracted images to the visual large model inference engine after preprocessing. Step S4: The visual large model inference engine performs image encoding, violation classification, and target localization on the input image, and outputs structured detection results; Step S5: If a violation is detected, the real-time preview and annotation module marks the violation area on the screen and displays the violation type and confidence level, while the detection result is stored in the database; Step S6: Users can retrieve historical test records by criteria through the test result management module. The system automatically generates a test report, which users can then export as Word or PDF format.

7. The intelligent detection method for violations based on a large visual model according to claim 6, characterized in that, In step S3, each video stream is frame-squeezed at a preset frame rate, and a multi-threaded concurrent scheduling strategy is adopted for multiple video streams.

8. The intelligent detection method for violations based on a large visual model according to claim 6, characterized in that, The reasoning process of the visual large model inference engine in step S4 includes: encoding the input image into a high-dimensional visual feature vector through an image encoder; classifying violations based on the visual feature vector and outputting the violation type; locating the violation target based on the feature map or attention map of the visual large model and outputting the bounding box coordinates; and outputting the confidence score of the detection result.

9. The intelligent detection method for violations based on a large visual model according to claim 6, characterized in that, In step S4, the violation classification of the visual large model inference engine supports rapid adaptation using a small number of example images to fit the specific violation definition of the scenario.

10. The intelligent detection method for violations based on a large visual model according to claim 6, characterized in that, In step S5, when the confidence level is lower than the preset threshold, the detection result is marked as pending manual review.

Images