A programmable multi-model collaborative video intelligent analysis system and method

Abstract

The application discloses a programmable multi-model cooperative video intelligent analysis system and method, relates to the technical field of computer vision, and comprises the following units: a video access and frame reading unit, a skill graph programming unit, an alarm merging unit, an asynchronous re-trial unit, and a result pushing and storage unit.The video access and frame reading unit receives a video stream and outputs a video frame; the skill graph programming unit receives a video frame, an electronic fence, and a task parameter corresponding to an analysis task, and outputs an alarm candidate; the alarm merging unit generates a merged alarm according to a plurality of alarm candidates, generates alarm information, and performs secondary semantic verification on a false alarm scene based on the alarm information and a screenshot cache address, and outputs an alarm result after rechecking; and the result pushing and storage unit pushes the alarm result to a monitoring terminal and writes the alarm result into a database and object storage.In a multi-path industrial video real-time analysis scene, a complete video intelligent analysis scheme is provided, which is flexible to configure, efficient in multi-model cooperation, low in alarm noise, traceable, supports continuous self-optimization of a model, and simultaneously considers real-time concurrency performance, computing power cost, and business rapid iteration requirements in engineering landing.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a programmable multi-model collaborative video intelligent analysis system and method. Background Technology

[0002] In various industrial and public places, managers typically install fixed cameras in key areas such as production areas, storage corridors, power distribution rooms, control rooms, and loading and unloading areas. The video streams are usually connected to a video access management platform via the GB28181 national standard protocol, ONVIF, or RTSP, and are viewed by monitoring center personnel in rotation or reviewed afterward. This model is a well-known method of video acquisition and manual supervision in this field.

[0003] In recent years, industrial video intelligent analysis has gradually shifted from "manual video viewing" to "algorithm-assisted identification." A common existing engineering approach involves a streaming media server sending frames to an inference server. The inference server deploys lightweight object detection models such as YOLO, performing object detection on each frame or by sampling frames at a fixed frequency. Once the confidence level exceeds a threshold, an alarm module immediately generates a message and pushes it to the monitoring terminal. The entire system's logical connection is linear—camera, streaming media, inference, alarm, and terminal are sequentially connected, without an independent workflow orchestration layer or a second model involved in the analysis.

[0004] Taking "not wearing a safety helmet detection" as an example, the actual operation of the existing solution is illustrated: the camera captures images at 25fps, and the inference service samples frames at 2fps and feeds them into the YOLO model; after the model outputs the no_helmet category detection box, the system triggers alarms frame by frame. Within 5 seconds that a worker on the work surface is not wearing a helmet, 10 highly repetitive alarms may be generated. The on-duty personnel need to check each one individually, which can easily lead to alarm fatigue. Increasing the threshold to reduce the alarm frequency will increase the risk of missed alarms. For complex behaviors that are difficult to describe with a single detection box, such as sleeping on duty, using mobile phones, or incorrect power supply sequence, the existing solution often requires developing a separate post-processing script or retraining a dedicated model. The monitoring logic is scattered across different code files, and the deployment of new scenarios usually requires modifying the source code, stopping the service, and redeploying, with a cycle of weeks.

[0005] On the other hand, multimodal visual language models are strong in image semantic understanding, capable of distinguishing subtle differences such as "not wearing a hat" versus "headscarf occlusion" and "shadow misjudgment." However, a single inference cycle typically takes seconds. If the semantic model is called for each frame of 20 video streams, existing GPU clusters struggle to meet the engineering requirements of "single-frame inference time not exceeding 200 milliseconds and concurrency not less than 20 streams." Therefore, the industry has yet to develop a complete solution that, under real-time multi-stream analysis conditions, can orchestrate various monitoring strategies, allow detection and semantic models to perform their respective functions, suppress alarm noise, and perform closed-loop optimization.

[0006] The aforementioned existing technologies have three prominent shortcomings: First, the analysis process is rigid, making it difficult to flexibly combine steps such as "detection-filtering-semantic understanding-process compliance" according to different scenarios; second, the detection model and semantic model lack a layered collaboration mechanism, making it difficult to balance real-time performance and accuracy; third, there are many duplicate alarms, the false alarm rate is high in complex scenarios, and false alarm samples are difficult to be automatically fed back for subsequent model optimization. As a result, video data remains at the "storage cost" level for a long time and is difficult to be transformed into a sustainable and evolving analytical asset.

[0007] Therefore, in view of the shortcomings of the existing technology, how to provide a programmable multi-model collaborative video intelligent analysis system and method is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0008] In view of this, the present invention provides a programmable multi-model collaborative video intelligent analysis system and method, which provides a complete video intelligent analysis solution in the scenario of real-time analysis of multiple industrial videos. It is flexibly configurable, has efficient multi-model collaboration, low-noise and traceable alarms, and supports continuous self-optimization of models. At the same time, it takes into account the real-time concurrent performance, computing cost, and rapid business iteration requirements for engineering implementation.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: a programmable multi-model collaborative video intelligent analysis system, comprising: a task scheduling unit, a video access and frame reading unit, a skill graph arrangement unit, a model inference unit, an alarm merging unit, an asynchronous re-judgment unit, and a result push and storage unit; The task scheduling unit is connected to the video access and frame reading unit. The skill graph arrangement unit is connected to the model inference unit and the alarm merging unit respectively. The model inference unit works in a multi-model collaborative mode. The task scheduling unit is used to start or stop the analysis task according to the analysis task configuration, and output the runtime segment, frame sampling frequency, skill map, electronic fence and task parameter configuration bound to the analysis task. The video access and frame reading unit is used to acquire video streams and output video frames according to the analysis task; The skill graph orchestration unit is used to receive video frames, electronic fences and task parameters corresponding to the analysis task, and output alarm candidates according to the skill graph definition; The alarm merging unit is used to generate a merged alarm based on multiple alarm candidates, simultaneously capture screenshots and pre- and post-buffered videos as complete evidence, and generate alarm information; when necessary, the alarm information is sent to the asynchronous review unit. The asynchronous review unit is used to perform secondary semantic verification on false alarm scenarios based on the alarm information and screenshot cache address, and output the reviewed alarm result. The result push and storage unit is used to push the alarm results after semantic verification to the monitoring terminal and write them to the database and object storage.

[0010] Preferably, the model inference unit has a built-in detection model and a multimodal visual language model; The detection model is used to locate targets in video frames; The multimodal visual language model is used to perform semantic verification based on the detection model.

[0011] Preferably, the skill graph arrangement unit includes a directed acyclic DAG skill graph, which includes a start node, a detection model node, a rule processing node, a judgment node, and an end node. The start node is used to receive video frames, electronic fences, and task parameters; The detection model node is used to call the detection model of the model inference unit and output a unified detection result; The rule processing node is used to filter the unified detection results to obtain valid target images; The judgment node is used to perform condition judgment, time sequence judgment, and compliance judgment of the operation process on the valid target image; The end node is used to summarize the output results of the sub-nodes and generate a single-frame level alarm candidate when the violation judgment conditions are met.

[0012] Preferably, it also includes: semantic model nodes; the semantic model nodes and the detection model nodes can be directly connected in the same skill graph, and multi-model collaboration can be completed in the same frame processing process to form a synchronous collaborative link mode; Alternatively, after an alarm is generated and merged, the asynchronous review unit can independently call the multimodal visual language model to perform secondary semantic review on the evidence screenshots according to the task review configuration, thus forming an asynchronous collaborative link mode. The synchronous collaborative link mode and the asynchronous collaborative link mode can be used individually or in combination in the same task.

[0013] Preferably, in the synchronous collaborative link mode, the semantic model node is connected in series between the rule processing node and the judgment node; the semantic model node receives the effective target image filtered by the rule processing node, calls the multimodal visual language model built into the model inference unit, outputs a structured semantic conclusion based on the business-defined prompt words, and transmits the structured semantic conclusion to the judgment node.

[0014] Preferably, in the asynchronous collaborative link mode, the alarm candidates are based on the alarm candidates output by the skill graph orchestration unit; the alarm merging unit merges and reduces noise on the alarm candidates, and then decides whether to enter the asynchronous re-judgment unit for re-judgment according to the task re-judgment configuration; For alarm messages entering the asynchronous review unit, the asynchronous review unit calls the multimodal visual language model in the model inference unit according to the task review configuration, performs global semantic review on the entire evidence screenshot, and outputs the review judgment result.

[0015] Preferably, it further includes: an evidence generation unit, which is connected to an alarm merging unit, wherein the evidence generation unit configures a video circular buffer for each analysis task, and the video circular buffer continuously caches the original video frames within a preset duration; Before the alarm merging unit generates a merged alarm and prepares to push it, the evidence generation unit reads the alarm's first trigger time and last end time recorded in the merged object record, extracts the corresponding continuous buffer frames before and after the alarm event from the video circular buffer, encodes them to generate evidence video, and uploads and stores it.

[0016] Preferably, a programmable multi-model collaborative video intelligent analysis method includes: Start or stop the analysis task according to the analysis task configuration, and output the runtime segment, frame sampling frequency, skill map, electronic fence and task parameter configuration bound to the analysis task. The video stream is acquired and video frames are output based on the analysis task; Receive video frames, electronic fences, and task parameters corresponding to the analysis task, and output alarm candidates according to the skill graph definition; A merged alarm is generated based on multiple alarm candidates, and screenshots and pre- and post-buffered videos are captured simultaneously as complete evidence to generate alarm information. Based on the alarm information and screenshot cache address, a second semantic verification is performed on the false alarm scenario, and the verified alarm result is output. The alarm results, after semantic verification, are pushed to the monitoring terminal and written to the database and object storage.

[0017] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a programmable multi-model collaborative video intelligent analysis system and method. Compared with the "single-model frame-by-frame alarm" scheme in the background art, the present invention shows significant engineering advantages in typical industrial field deployments: Skill graph arrangement allows for the flexible activation of various custom scenario skills by region, time period, and level. New strategy releases no longer require restarting the entire platform, and the strategy adjustment cycle has been shortened from weeks to hours.

[0018] When the detection model maintains 2fps analysis, the inference time per frame is approximately 15–200 milliseconds, and it can run stably with multiple concurrent connections. The semantic model is only invoked when there are people within the electronic fence area or when a re-judgment is required after merging. Under normal working conditions, the number of semantic model calls is about one-tenth of the total number of calls in the entire frame.

[0019] After false positives are archived and entered into the annotation platform, the detection model can be retrained and published to the inference service, forming a closed loop of deployment-running-feedback-retraining-redeployment.

[0020] Persistent message queues, dead-letter queues, and interface / execution processes are deployed separately, enabling the system to achieve an engineering availability level of 99.99% for effective uptime. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the overall system architecture provided for an embodiment of the present invention.

[0023] Figure 2 A schematic diagram of the data element closed loop and intelligent supervision link provided for embodiments of the present invention.

[0024] Figure 3 This is a schematic diagram of the platform capabilities and application scenarios provided in the embodiments of the present invention.

[0025] Figure 4 This is a schematic diagram of the asynchronous collaboration and alarm merging process provided in an embodiment of the present invention.

[0026] Figure 5 This is a schematic diagram of the deployment topology and system boundary provided for an embodiment of the present invention.

[0027] Figure 6The flowchart for skill graph arrangement and multi-model collaborative analysis provided in the embodiments of the present invention.

[0028] Figure 7 This is a schematic diagram of system deployment and module connection provided for an embodiment of the present invention.

[0029] Figure 8 This is a flowchart illustrating the synchronous and collaborative process for monitoring sleep deprivation in an embodiment of the present invention.

[0030] Figure 9 The flowchart for asynchronous collaboration and re-judgment of safety helmet detection is provided for embodiments of the present invention.

[0031] Figure 10 This is a schematic diagram illustrating the compliance judgment of the standard operating procedure for power outages and restorations provided in an embodiment of the present invention.

[0032] Figure 11 This is a schematic diagram of low-frequency equipment inspection and frame reading mode switching provided in an embodiment of the present invention.

[0033] Figure 12 This is a deployment diagram of a non-ferrous metal mine scenario provided in an embodiment of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] The technical problems to be solved by the embodiments of the present invention are: how to enable lightweight target detection models and multimodal visual language models to form a hierarchical collaboration through configurable analysis process orchestration under the condition of real-time intelligent analysis of multi-channel industrial videos; how to call semantic models on demand to complete complex scene confirmation while ensuring low-latency operation of the detection link; how to merge and reduce noise of high-frequency repetitive alarms and retain complete evidence; and how to feed the analysis operation data back to the model training stage to form a closed loop of sustainable optimization.

[0036] To this end, embodiments of the present invention provide a programmable multi-model collaborative video intelligent analysis method and system, which transforms the regulatory analysis strategy from "hard-coded into code" to "configured in the skill graph", transforms multi-model collaboration from "either / or" to "real-time detection and on-demand semantic analysis", and transforms alarm output from "one alarm after another" to "merged and pushed with timing and counting information".

[0037] This invention discloses a programmable multi-model collaborative video intelligent analysis system, comprising: a task scheduling unit, a video access and frame reading unit, a skill graph arrangement unit, a model inference unit, an alarm merging unit, an asynchronous re-judgment unit, and a result push and storage unit; The task scheduling unit is connected to the video access and frame reading unit. The skill graph arrangement unit is connected to the model inference unit and the alarm merging unit respectively. The model inference unit works in a multi-model collaborative mode. The task scheduling unit is used to start or stop the analysis task according to the analysis task configuration, and output the runtime segment, frame sampling frequency, skill map, electronic fence and task parameter configuration bound to the analysis task. The video access and frame reading unit is used to acquire video streams and output video frames according to the analysis task; The skill graph orchestration unit is used to receive video frames, electronic fences and task parameters corresponding to the analysis task, and output alarm candidates according to the skill graph definition; The alarm merging unit is used to generate a merged alarm based on multiple alarm candidates, simultaneously capture screenshots and pre- and post-buffered videos as complete evidence, and generate alarm information; when necessary, the alarm information is sent to the asynchronous review unit. The asynchronous review unit is used to perform secondary semantic verification on false alarm scenarios based on the alarm information and screenshot cache address, and output the reviewed alarm result. The result push and storage unit is used to push the alarm results after semantic verification to the monitoring terminal and write them to the database and object storage.

[0038] Specifically, in this embodiment of the invention, the electronic fence and task parameters are obtained through analysis of task configuration. Maintenance personnel draw an electronic fence for the camera in the front-end operation plan, including polygonal regions of interest or tripwires, and configure skill parameters, frame extraction frequency, warning level, and video pre- and post-buffering durations, etc.; after the operation plan is issued, the above configuration is written into the corresponding analysis task; when the task is executed, the video access and frame reading unit outputs the video frame of the task, and the skill graph orchestration unit synchronously receives the electronic fence and task parameters corresponding to the task, and performs skill graph analysis and outputs alarm candidates accordingly.

[0039] In a specific embodiment of the present invention, the system proposed in this embodiment can be implemented in actual engineering as a video intelligent analysis engine and its supporting components. To avoid limiting the present invention to a specific software name, the following description will use "video intelligent analysis engine" or "analysis engine". The system as a whole consists of a video access and frame reading unit, a task scheduling unit, a skill graph orchestration unit, a model inference unit, an alarm merging unit, an asynchronous review unit, and a result push and storage unit. The logical connection relationship between each unit is as follows: the camera or the existing video access management platform outputs video streams to the video access and frame reading unit; the frame reading unit outputs video frames according to the runtime segment and frame extraction frequency given by the task scheduling unit; the skill graph orchestration unit receives video frames, electronic fences, and task parameters, and calls the model inference unit and rule processing nodes according to the skill graph definition; after the skill graph end node outputs alarm candidates, it enters the alarm merging unit; the alarm merging unit generates merged alarms, screenshots, and video evidence, and sends the alarms to the asynchronous review unit when necessary; the asynchronous review unit calls the semantic model to confirm the results, and finally the result push and storage unit pushes them to the monitoring terminal and writes them to the database and object storage. The above connection relationship corresponds to Figure 1 , Figure 4 and Figure 6 The structure shown is... Figure 1 This is a schematic diagram of the overall system architecture. It illustrates the layered relationships between the front-end presentation layer, interface service layer, intelligent analysis layer, inference and data resource layer, and infrastructure layer. It is designed to ensure that those skilled in the art can implement the system by module boundaries.

[0040] Specifically, the technical means of the video access and frame reading unit is adaptive frame reading. The input end of this unit can be connected to a camera, a video access management platform, or other video sources that support protocols such as RTSP, GB28181, and ONVIF. The output end is connected to the frame processing queue of the execution process. The system maintains a shared stream reader for each camera and records the reference count. When multiple analysis tasks share the same camera, video stream connections are not repeatedly established; instead, the latest frame is distributed from the shared stream reader. If the task frame extraction interval is lower than a preset connection overhead threshold, such as a personnel monitoring task with 2 frames per second, a continuous connection reading mode is adopted. The acquisition thread continuously updates the latest video frame, and the detection thread reads the latest frame according to the task frequency. If the task frame extraction interval is higher than the threshold, such as a device inspection task every 30 seconds, an on-demand screenshot mode is adopted, where a single frame image is obtained through the video access management platform before each analysis. This embodiment of the invention enables the same system to simultaneously adapt to high-frequency real-time monitoring and low-frequency inspection scenarios, avoiding the resource waste caused by all tasks maintaining long connections or all tasks establishing connections on demand in the prior art.

[0041] Specifically, the task scheduling and execution unit adopts a structure that separates the interface service process and the execution service process. The interface service process is responsible for camera management, task configuration, skill graph editing, early warning query, and real-time message push; the execution service process is responsible for video streaming, skill graph execution, model invocation, alarm merging, and re-judgment queuing. The execution service process is further divided into acquisition threads, detection threads, and optional push streaming threads, which pass data through a queue: the acquisition thread only guarantees to acquire the latest frame, the detection thread is only responsible for performing AI analysis and producing structured results, and the push streaming thread only draws detection boxes and outputs framed video when preview is needed. This structure differs from the existing single-process serial solution, and its function is to ensure that the external interface, video acquisition, and model inference do not block each other, and to maintain the responsiveness of the configuration interface and monitoring screen even under multi-channel concurrency.

[0042] Specifically, the model inference unit incorporates a detection model and a multimodal visual language model; The detection model is used to locate targets in video frames; The multimodal visual language model is used to perform semantic verification based on the detection model.

[0043] Specifically, the multimodal visual language model described in this embodiment of the invention is used as a semantic model node within the skill graph to participate in synchronous collaborative analysis, and is used by the asynchronous review unit to perform secondary semantic verification on the evidence screenshots after alarm merging. As a preferred implementation, it can also be combined with business rules or a knowledge base to generate handling suggestions. In one embodiment, when the asynchronous review confirms that an alarm is a genuine alarm, the system matches a safety production procedure clause template according to the alarm type to generate text containing risk descriptions and handling suggestions; or the text model retrieves knowledge from the knowledge base to generate handling suggestions for reference by on-duty personnel.

[0044] The model inference unit employs a multi-model collaborative approach, rather than relying on a single model for independent alerts. The first type of model is the detection model, such as the YOLO series, responsible for low-latency, high-recall target localization of video frames. The second type is the multimodal visual language model, such as the Qwen-VL model deployed on the Llama-Factory service, responsible for confirming behavioral semantics, occlusion, and suspected false alarms that the detection model struggles to determine. Detection model nodes and semantic model nodes can be linked within the same skill graph, or after an alert is generated and merged, the asynchronous review unit can independently call the multimodal visual language model for secondary semantic review based on the task review configuration, thus forming both synchronous and asynchronous collaborative modes.

[0045] Specifically, the skill graph orchestration unit includes a directed acyclic DAG skill graph, which includes a start node, a detection model node, a rule processing node, a judgment node, and an end node. The start node is used to receive video frames, electronic fences, and task parameters; The detection model node is used to call the detection model of the model inference unit and output a unified detection result including target category, confidence level, location box, and tracking number; The rule processing nodes include nodes for region filtering, size filtering, counting, intersection, distance, tripwire, displacement, target matting, and video slicing; these are used to filter the unified detection results to obtain valid target images and remove invalid targets that are outside the fence, have abnormal sizes, or do not meet the quantity conditions. The judgment node is used to perform condition judgment, time sequence judgment, and compliance judgment of the operation process on the valid target image; The termination node is used to summarize the output results of the sub-nodes. When the violation judgment conditions are met, a single-frame level alarm candidate is generated and the alarm candidate is sent to the alarm merging unit.

[0046] Existing technologies typically hardcode the "detection-filtering-judgment-alarm" process into the program code. However, the skill graph orchestration unit in this invention abstracts each analytical capability as a node and the data transfer between nodes as directed edges, forming a storable, verifiable, publishable, and rollback-capable directed acyclic skill graph. The skill graph includes at least a start node, a detection model node, a rule processing node, a judgment node, and an end node. The start node outputs the current frame image, the electronic fence area, and task-defined parameters; the detection model node calls a lightweight target detection model, outputting unified detection results including target category, confidence level, bounding box, and tracking number; the rule processing node includes nodes for region filtering, size filtering, counting, intersection, distance, tripwire, displacement, target matting, and video slicing; the judgment node includes conditional judgment, temporal judgment, and workflow compliance judgment; and the end node summarizes upstream results and generates alarm candidates. The skill graph is stored as JSON, and after publication, it is written to a skill class table and registered with a skill factory via an adapter, thus being mounted as a regular AI skill on the camera task, achieving runtime hot loading. Figure 6 As shown, Figure 6 This is a flowchart for skill graph orchestration and multi-model collaborative analysis. It directly reflects the invention point of "arrangeability". The diagram includes structures such as video frame input, skill graph DAG, start node, detection model node, rule processing node, semantic model node, condition judgment node, end node, skill release, alarm merging, video evidence collection, asynchronous review, push archiving, and sample feedback, clearly showing the connection relationships between the detection model, semantic model, rule nodes, and closed-loop modules.

[0047] Specifically, during skill graph execution, the orchestration engine first performs structural verification on nodes and edges, and then verifies port types to prevent incorrect connections of different types of data, such as image data, detection result data, electronic fence area data, and judgment result data. After verification, the nodes are topologically sorted and called sequentially. For nodes requiring cross-frame judgment, such as condition judgment nodes, timing judgment nodes, and work process compliance nodes, the orchestration engine saves the running status in isolation by task and node, ensuring that timing rules such as "continuously satisfy for 2 seconds," "allow short buffers," and "execute in A→B→C order" can be valid across frames. The purpose of this mechanism is to transform complex regulatory strategies from hard-coded logic into configurable processes, and to ensure that the configured processes can be reproduced by ordinary technicians according to the node inputs, outputs, and connection relationships. Figure 5 As shown, Figure 5 This diagram illustrates the deployment topology and system boundaries. It shows the deployment boundaries between cameras, the video access management layer, the intelligent analysis platform, the application monitoring layer, and the unified authentication and access control, as well as the security boundaries such as external interface services and internal network isolation of execution services.

[0048] Specifically, it also includes: semantic model nodes; the semantic model nodes and the detection model nodes can be directly connected in the same skill graph, and complete multi-model collaboration in the same frame processing process to form a synchronous collaborative link mode; Alternatively, after an alarm is generated and merged, the asynchronous review unit can independently call the multimodal visual language model to perform secondary semantic review on the evidence screenshots according to the task review configuration, thus forming an asynchronous collaborative link mode. The synchronous collaborative link mode and the asynchronous collaborative link mode can be used individually or in combination in the same task; The synchronous collaborative link mode and the asynchronous collaborative link mode can be used individually or in combination in the same task.

[0049] In the asynchronous collaborative link mode, the asynchronous review unit and the detection model node in the skill graph are not connected in series, nor do they depend on the connection of the semantic model node in the skill graph.

[0050] The semantic model node is used to call the multimodal visual language model and output structured semantic results according to the prompt words.

[0051] Specifically, in the synchronous collaborative link mode, the semantic model node is connected in series between the rule processing node and the judgment node; the semantic model node receives the valid target image filtered by the rule processing node, calls the multimodal visual language model built into the model inference unit, outputs a structured semantic conclusion containing whether it violates the rules, the reason for the violation, and the semantic confidence based on the business-defined prompt words, and transmits the structured semantic conclusion to the judgment node.

[0052] Specifically, the synchronous and collaborative method is as follows: the detection model node first performs target detection on the entire frame of the image, outputting candidate targets such as personnel, equipment, vehicles, or devices; rule nodes such as region filtering, counting, and size filtering filter the candidate targets, excluding targets outside the fence, those with abnormal sizes, or those whose quantity does not meet the conditions; only targets or image regions that meet preset conditions are passed to the semantic model node, which outputs structured results such as whether it violates the rules, the reason for the violation, and the confidence level based on prompt words; the judgment node decides whether to trigger the termination node based on the semantic results, duration, and buffer time. For example, in the sleep deprivation analysis, the detection model is only responsible for locating personnel, the region filtering node only retains personnel within the duty area, and the semantic model only judges whether personnel in that area are sleeping on duty or using their mobile phones. The purpose of this technique is to limit the use of the semantic model to necessary scenarios and avoid calling the semantic model for the entire image in every frame.

[0053] Specifically, in the asynchronous collaborative link mode, alarm candidates are output by the skill graph orchestration unit; the alarm merging unit merges and reduces noise from the alarm candidates, and then decides whether to enter the asynchronous review unit for review based on the task review configuration; For alarm messages entering the asynchronous review unit, the asynchronous review unit, according to the task review configuration, calls the multimodal visual language model in the model inference unit to perform a global semantic review of the entire evidence screenshot and outputs the review judgment result. The asynchronous review is decoupled from the main link of real-time skill graph analysis and does not block frame-by-frame analysis.

[0054] Specifically, when the analysis task has enabled review and has been bound to a review skill or multimodal review model, the alarm information and screenshot cache address are written to the message queue and reviewed by the asynchronous review unit; when the analysis task has not enabled review or has not been configured with a review skill, it does not enter the asynchronous review unit and is directly pushed or archived according to the merged alarm results.

[0055] Specifically, the asynchronous collaborative method is as follows: After generating alarm candidates at the end node of the skill graph, the alarm merging unit first performs merging and noise reduction, and then decides whether to enter asynchronous semantic review based on the task configuration. The alarm merging unit extracts stable fields such as task type, task number, camera number, skill number, alarm type, and alarm level, concatenates them to generate an MD5 warning unique key, and does not include dynamic text such as "3 people detected" or "5 people detected" in the key to ensure that consecutive frames of the same event can be merged. The merging object records the first time, last time, instance list, and cumulative number of times; if a new alarm is within the merging window and does not exceed the maximum duration, it is appended to the existing merging object; if it exceeds the window or duration limit, the old object is sent and a new object is created. The sending delay is calculated according to the alarm level. Level 1 emergency alarms are sent immediately or near real-time, and ordinary alarms are sent with a delay based on the base delay and level factor. When the cumulative number of times reaches the fast sending threshold, it is sent immediately. The purpose of this technique is to transform multi-frame repeated alarms into an event with a time span and cumulative number of times, which reduces both manual interference and the number of subsequent semantic review calls. like Figure 4 As shown, Figure 4 This is a schematic diagram of the asynchronous collaboration and alarm merging process. It illustrates that after the detection model generates candidate alarms, the closed loop is completed through processes such as MD5 deduplication, hierarchical delay merging, screenshot and video evidence collection, review queue, multimodal model review, knowledge-assisted suggestions, and reporting, archiving, and statistics.

[0056] Specifically, it also includes: an evidence generation unit, which is connected to an alarm merging unit, and the evidence generation unit independently configures a video circular buffer for each analysis task, and the video circular buffer continuously caches the original video frames within a preset duration; Before the alarm merging unit generates a merged alarm and prepares to push it, the evidence generation unit reads the alarm's first trigger time and last end time recorded in the merged object record, extracts the corresponding continuous buffer frames before and after the alarm event from the video circular buffer, encodes them to generate MP4 format evidence video, and uploads and stores it.

[0057] Specifically, the evidence generation unit is connected to the alarm merging unit to generate traceable evidence before merging alarms is sent. The system maintains a video circular buffer for the task, continuously storing frame data from the most recent few seconds. When a merged alarm is ready to be sent, the system extracts the buffered video before and after the alarm from the circular buffer according to the first and last times of the merging object, encodes it as MP4, and uploads it to the object storage; within the merging group... Figure 1 The data is then archived, with the main image prioritized for instances that are located near the center of the merge group and actually exist. This technique avoids insufficient context due to saving only a single screenshot, allowing alerts to be used for post-event review, data archiving, and model training.

[0058] Specifically, the asynchronous review unit is connected to the alarm merging unit, object storage, and semantic model inference service. After the merged alarm is sent, if the task is bound to a review skill, the system packages the alarm information, screenshot cache key, object storage address, detection box, and review prompt words into a review message and writes it to a persistent message queue. After retrieving the review message, the consumer thread first reads the screenshot from the cache service. If the cache is invalid, it reads the original image or labeled image from the object storage, and then calls the multimodal visual language model to output the review conclusion. When the review fails, an exponential backoff retrieval is used, and after multiple failures, the image enters the dead-letter queue. The purpose of this embodiment is to decouple the time-consuming semantic confirmation from the real-time detection link, ensuring the real-time performance of the detection link while improving the accuracy of alarms in complex scenarios.

[0059] Specifically, the result push and closed-loop optimization unit is used to transform the analysis results into usable business data. After merging and re-judgment, alarms are pushed to the monitoring terminal via SSE, while structured alarms, re-judgment records, screenshot addresses, video addresses, and handling status are written to the database. After on-duty personnel confirm, archive, or mark false alarms, false alarm samples can be fed back to the annotation and training platform to form new training samples; the trained detection model is redeployed to the inference service, and the skill graph can be updated or rolled back through version management. The purpose of this invention is to make the video analysis system not just a one-time alarm tool, but to form a continuous optimization closed loop of "analysis—handling—feedback—retraining—redeployment".

[0060] In one specific embodiment of the present invention, a programmable multi-model collaborative video intelligent analysis method includes: Start or stop the analysis task according to the analysis task configuration, and output the runtime segment, frame sampling frequency, skill map, electronic fence and task parameter configuration bound to the analysis task. The video stream is acquired and video frames are output based on the analysis task; Receive video frames, electronic fences, and task parameters corresponding to the analysis task, and output alarm candidates according to the skill graph definition; A merged alarm is generated based on multiple alarm candidates, and screenshots and pre- and post-buffered videos are captured simultaneously as complete evidence to generate alarm information. Based on the alarm information and screenshot cache address, a second semantic verification is performed on the false alarm scenario, and the verified alarm result is output. The alarm results, after semantic verification, are pushed to the monitoring terminal and written to the database and object storage.

[0061] Specifically, in this embodiment of the invention, the video stream is accessed and a frame reading mode is selected according to the frame extraction interval; a skill graph containing detection model nodes, semantic model nodes, and rule nodes is created or loaded; the skill graph is verified, topologically sorted, and executed according to node connections; alarm candidates are generated based on the output of the end node; alarm merging and level-delayed sending are performed based on the unique warning key; screenshots and pre- and post-buffered video evidence are generated; merged alarms requiring semantic confirmation are written into the review queue; the semantic model is called for asynchronous review; the analysis results are pushed and stored; and false alarms and handling results are fed back to the training stage. Figure 2 As shown, Figure 2 This diagram illustrates the data element closed-loop and intelligent supervision link. It explains the data closed-loop relationship between video perception data, business data, knowledge rule data, false alarm feedback, labeled samples, and model iteration.

[0062] like Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the platform's capabilities and application scenarios. It serves to demonstrate that the invention is not limited to a single industry and can be extended to various video analytics scenarios such as personnel protection, area access, equipment operation, special operations, fire protection, and vehicle logistics.

[0063] Specifically, the embodiments of the present invention also propose the following alternative technical solutions: In addition to MD5, the unique warning key can use SHA-256, SM3, or other hash algorithms; the review queue can use any message middleware with persistence, retry, and dead-letter processing capabilities; the detection model can be replaced with YOLO, RT-DETR, Faster R-CNN, Transformer detection models, or industry-specific detection models; the semantic model can be replaced with a local multimodal model, a cloud-based visual language model, or a rule-enhanced semantic classifier; the object storage can be replaced with a local file system, a distributed file system, or cloud object storage; skill graph nodes can be extended into new image processing nodes, time-series nodes, business rule nodes, or external system call nodes through a registration mechanism; synchronous collaboration and asynchronous collaboration can be used individually or in combination within the same task; the threshold for adaptive frame reading can be changed from a fixed threshold to dynamically adjusted based on network quality and historical connection establishment time.

[0064] The English abbreviations involved in the embodiments of this invention are explained as follows: RTSP (Real-time Streaming Protocol), GB28181 (Public Security Video Surveillance Network Standard), YOLO (Single Detection Target Algorithm), DAG (Directed Acyclic Graph), VLM (Multimodal Visual Language Model), SSE (Server Push Event), and MD5 (Message Digest Algorithm Version 5).

[0065] Specifically, the embodiments of the present invention relate to a method and system for real-time intelligent analysis of fixed camera video streams, which can be widely applied to fixed camera scenarios such as factory workshops, logistics warehouses, energy and chemical industries, construction sites, mines, industrial parks, and transportation hubs where video analysis of safety production, order supervision, or equipment inspection is required.

[0066] like Figure 7 The diagram illustrates the system deployment and module connection of an embodiment of the present invention. In a typical industrial site deployment, two GPU servers equipped with Hygon 7490 processors and Kunlun Core accelerator cards run the Kylin V10 operating system. The video access management platform connects to cameras at various monitoring points via protocols such as GB28181. The detection execution process of the video intelligent analysis engine retrieves streams from the video access management platform via standard video stream protocols. The detection model inference service deploys models for safety helmet recognition, personnel recognition, etc.; the semantic model inference service deploys multimodal visual language models for semantic nodes within the skill graph and alarm review. The relational database stores camera logs, AI tasks, skill graph versions, alarm and review records; the caching service stores screenshots used for review; the object storage organizes warning images and video clips by task and date; the message queue carries review tasks and dead-letter queues. The front-end monitoring page subscribes to the warning stream through a real-time push interface. The task scheduler starts and stops tasks according to a preset time expression. When a task resumes, the merge manager sends merged alarms that were not sent under that task to avoid missed reports.

[0067] In a specific embodiment of the present invention, synchronous collaboration involves detecting different targets / elements separately within the same skill map, with semantic models supplementing different dimensions of criteria. This embodiment uses compliance checks of personal protective equipment (PPE) upon entry as an example. After the analysis task is bound to the corresponding skill map, multiple detection model nodes are set up in parallel or in a branching manner within the same skill map to detect different protective elements such as safety helmets, reflective vests, and work clothes. After area filtering, a structured result is obtained indicating whether each element is missing. Simultaneously, semantic model nodes can be chained together within the skill map: based on the target boxes of personnel and protective equipment provided by the detection model, a multimodal visual language model specifically judges semantic items that are not easily given directly by the detection boxes themselves, such as "whether the wearing position is correct" or "whether there is improper wearing such as crooked wearing / not fastened," and outputs structured semantic conclusions. The converging judgment node combines the "missing item results" and "improper wearing semantic results" to generate alarm candidates, and the ending node outputs an alarm. This process is completed within the same frame processing: each detection node detects different objects, and the semantic model supplements different dimensions of criteria, jointly supporting the same regulatory conclusion, which constitutes synchronous collaboration. Similarly, in scenarios where vehicles are in motion but pedestrians are not, vehicle detection and pedestrian detection can be performed separately within the same skill map, combined with distance or intersection judgments; if necessary, the semantic model can be used to supplement the judgment on the intersection risk level, which also reflects synchronous collaboration.

[0068] In one specific embodiment of the present invention, asynchronous collaboration involves the detection model issuing an alarm first, followed by semantic verification using a multimodal model. This embodiment uses the monitoring of workers not wearing safety helmets in a production area as an example. The analysis task is bound to a skill graph of "start node → no safety helmet detection node → work area filtering node → end node". The detection model node identifies targets without safety helmets in real time and outputs alarm candidates; the alarm merging unit merges and reduces noise from high-frequency repetitive alarms, generating evidence screenshots and pre- and post-buffered videos. If the analysis task has enabled review and bound to a review skill, the asynchronous review unit writes the alarm information and screenshot cache address to a message queue, calls the multimodal visual language model to perform a global semantic review of the entire evidence screenshot, distinguishing between "no safety helmet" and "false alarm due to wearing a dark headscarf," etc.; if the review confirms a genuine alarm, it is pushed; if the review determines it is a false alarm, it is marked and the error is fed back for annotation. In this process, the second model does not compete with the detection node for the same detection target within the skill graph, but is called independently after the alarm is generated, which is asynchronous collaboration.

[0069] like Figure 8 The diagram shows a synchronous collaborative flowchart for monitoring sleep deprivation provided in this embodiment of the invention. The dispatch room camera is bound to the "Sleep Deprivation Recognition in Duty Room" skill graph in the task configuration. The runtime is from 8:00 AM to 6:00 PM daily, with a frame sampling frequency of 2 frames per second. The skill graph structure is: Start Node → Personnel Detection Node → Duty Room Area Filtering Node → Sleep Deprivation / Mobile Phone Use Semantic Judgment Node → Duration Condition Judgment Node → End Node. After the task starts, the adaptive frame reader operates in continuous connection mode. If a suspected person is detected by a frame detection node within 15ms; after area filtering, only one person remains within the fenced area; the semantic node returns "Suspected sleep deprivation behavior exists" and the reason; the condition judgment node triggers an alarm after the condition is met for 2 seconds. Within a 5-second window, the cumulative number of similar frames increases without repeated pushes. After the timer expires, the preceding and following 5-second video clips in the attached object storage are merged. This embodiment illustrates that the semantic model only runs when "someone is in the duty room," and the duration mechanism filters out accidental triggers due to instantaneous posture.

[0070] like Figure 9 The diagram shows the asynchronous collaboration and review process for helmet detection provided in this embodiment of the invention. A camera in a production area uses a skill graph consisting of "start node → helmet-less detection node → work area filtering node → end node," and is bound to the "helmet semantic confirmation" review skill. If a suspected helmet-less situation is detected within 10 frames, this invention generates the same unique warning key based on the same task, camera, alarm type, and alarm level, merging them into one group. After merging and sending, the review queue receives the alarm screenshot and the prompt "distinguish between helmet-less and headscarf / shadow." The semantic model returns "not a violation, because a dark headscarf does not necessarily mean helmet-less," and the alarm is marked as a false alarm. The screenshot is added to the annotation set. The semantic call is only performed once, ensuring the detection chain is not slowed down.

[0071] like Figure 10 The diagram shown illustrates the compliance judgment of the standard operating procedure for power outages and restorations provided in this embodiment of the invention. The substation camera skill diagram includes three detection model nodes, corresponding to voltage verification, grounding, and switch operation actions, which are then integrated into the work process compliance judgment node. When a worker operates the switch without voltage verification, the work process compliance judgment node determines that the preceding steps are missing when the switch operation action occurs, and the end node outputs a level one alarm. Because the level one alarm is an emergency alarm, it is sent to the monitoring terminal via a real-time push interface approximately 0.5 seconds later, and simultaneously enters the review and object storage evidence retrieval process. This demonstrates the orchestration layer's support for process-based supervision.

[0072] like Figure 11 The diagram illustrates the switching between low-frequency equipment inspection and frame reading modes provided in this embodiment of the invention. A critical equipment analysis task analyzes one frame approximately every 30 seconds. The adaptive frame reader determines that this interval exceeds a preset connection overhead threshold and switches to on-demand screenshot mode. Each analysis acquires one image through the video access management platform, decodes it, and sends it to the skill map. After completion, the video stream connection is not maintained. A 10-channel low-frequency task multiplexing connection pool allows the same platform to handle both high-frequency personnel supervision and low-frequency equipment inspection in parallel.

[0073] like Figure 12 As shown, the embodiments of the present invention can be applied to non-ferrous metal mining scenarios. Under a general architecture, the embodiments of the present invention have been demonstrated and deployed in high-risk industries such as non-ferrous metal mines, connecting cameras in areas such as underground transportation, mining faces, substations, and ore dressing workshops, and implementing more than 30 analysis skills, including safety helmet monitoring, sleeping on duty monitoring, conveyor belt anomaly monitoring, and power outage compliance monitoring. In specific implementation, after camera access is completed via protocols such as GB28181, the task scheduling unit starts each analysis task according to the operation plan and binds skill maps, electronic fences, frame extraction configurations, and re-judgment configurations; the video access and frame reading unit extracts streams and frames according to tasks; the skill map orchestration unit combines detection model nodes and rule nodes or semantic model nodes to output alarm candidates; the alarm merging unit performs noise reduction, and the evidence generation unit generates screenshots and pre- and post-buffered evidence videos; asynchronous re-judgment can be selected according to task configuration; finally, the result push and storage unit pushes the results to the monitoring terminal and archives them.

[0074] In summary, the embodiments of this invention liberate the analysis strategy from the code through skill graphs, connect the detection model and semantic model within the same engineering system through synchronous and asynchronous collaboration, achieve both real-time detection and accurate confirmation through early warning merging and asynchronous review, and transform alarm data into reusable analytical assets through evidence storage and false alarm feedback. All of the above embodiments have been verified as feasible under the video intelligent analysis engine architecture. The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0075] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A programmable multi-model collaborative video intelligent analysis system, characterized in that, include: The system includes a task scheduling unit, a video access and frame reading unit, a skill graph arrangement unit, a model inference unit, an alarm merging unit, an asynchronous re-judgment unit, and a result push and storage unit. The task scheduling unit is connected to the video access and frame reading unit. The skill graph arrangement unit is connected to the model inference unit and the alarm merging unit respectively. The model inference unit works in a multi-model collaborative mode. The task scheduling unit is used to start or stop the analysis task according to the analysis task configuration, and output the runtime segment, frame sampling frequency, skill map, electronic fence and task parameter configuration bound to the analysis task. The video access and frame reading unit is used to acquire video streams and output video frames according to the analysis task; The skill graph orchestration unit is used to receive video frames, electronic fences and task parameters corresponding to the analysis task, and output alarm candidates according to the skill graph definition; The alarm merging unit is used to generate a merged alarm based on multiple alarm candidates, and simultaneously capture screenshots and pre- and post-buffered videos as complete evidence to generate alarm information. The asynchronous review unit is used to perform secondary semantic verification on false alarm scenarios based on the alarm information and screenshot cache address, and output the reviewed alarm result. The result push and storage unit is used to push the alarm results after semantic verification to the monitoring terminal and write them to the database and object storage.

2. The programmable multi-model collaborative video intelligent analysis system according to claim 1, characterized in that, The model inference unit incorporates a detection model and a multimodal visual language model; The detection model is used to locate targets in video frames; The multimodal visual language model is used to perform semantic verification based on the detection model.

3. The programmable multi-model collaborative video intelligent analysis system according to claim 2, characterized in that, The skill graph arrangement unit includes a directed acyclic skill graph, which includes a start node, a detection model node, a rule processing node, a judgment node, and an end node. The start node is used to receive video frames, electronic fences, and task parameters; The detection model node is used to call the detection model of the model inference unit and output a unified detection result; The rule processing node is used to filter the unified detection results to obtain valid target images; The judgment node is used to perform condition judgment, time sequence judgment, and compliance judgment of the operation process on the valid target image; The end node is used to summarize the output results of the sub-nodes and generate a single-frame level alarm candidate when the violation judgment conditions are met.

4. The programmable multi-model collaborative video intelligent analysis system according to claim 3, characterized in that, Also includes: Semantic model nodes; the semantic model nodes and the detection model nodes can be directly connected in the same skill graph, and multi-model collaboration can be completed in the same frame processing to form a synchronous collaborative link mode. Alternatively, after an alarm is generated and merged, the asynchronous review unit can independently call the multimodal visual language model to perform secondary semantic review on the evidence screenshots according to the task review configuration, thus forming an asynchronous collaborative link mode. The synchronous collaborative link mode and the asynchronous collaborative link mode can be used individually or in combination in the same task.

5. A programmable multi-model collaborative video intelligent analysis system according to claim 4, characterized in that, In the synchronous collaborative link mode, the semantic model node is connected in series between the rule processing node and the judgment node; the semantic model node receives the effective target image filtered by the rule processing node, calls the multimodal visual language model built into the model inference unit, outputs a structured semantic conclusion based on the business-defined prompt words, and transmits the structured semantic conclusion to the judgment node.

6. The programmable multi-model collaborative video intelligent analysis system according to claim 4, characterized in that, In the asynchronous collaborative link mode, alarm candidates are output by the skill graph orchestration unit; the alarm merging unit merges and reduces noise from the alarm candidates, and then decides whether to enter the asynchronous review unit for review based on the task review configuration; For alarm messages entering the asynchronous review unit, the asynchronous review unit calls the multimodal visual language model in the model inference unit according to the task review configuration, performs global semantic review on the entire evidence screenshot, and outputs the review judgment result.

7. The programmable multi-model collaborative video intelligent analysis system according to claim 1, characterized in that, Also includes: An evidence generation unit is connected to an alarm merging unit. The evidence generation unit configures a video circular buffer for each analysis task. The video circular buffer continuously caches original video frames within a preset duration. Before the alarm merging unit generates a merged alarm and prepares to push it, the evidence generation unit reads the alarm's first trigger time and last end time recorded by the merging object, extracts the corresponding continuous buffered frames before and after the alarm event from the video circular buffer, encodes them to generate evidence-gathering video, and uploads it to the storage analysis task.

8. A programmable multi-model collaborative video intelligent analysis method, applied to a programmable multi-model collaborative video intelligent analysis system according to any one of claims 1-7, characterized in that, include: Start or stop the analysis task according to the analysis task configuration, and output the runtime segment, frame sampling frequency, skill map, electronic fence and task parameter configuration bound to the analysis task. Based on the analysis task, acquire the video stream and output video frames; Receive video frames, electronic fences, and task parameters corresponding to the analysis task, and output alarm candidates according to the skill graph definition; A merged alarm is generated based on multiple alarm candidates, and screenshots and pre- and post-buffered videos are captured simultaneously as complete evidence to generate alarm information. Based on the alarm information and screenshot cache address, a second semantic verification is performed on the false alarm scenario, and the verified alarm result is output. The alarm results, after semantic verification, are pushed to the monitoring terminal and written to the database and object storage.

Images