Mine ai video monitoring detection and identification method based on combination of small model and large model

Abstract

This invention belongs to the fields of computer vision, artificial intelligence, and mine video surveillance technology. Specifically, it relates to a mine AI video surveillance detection and recognition method based on a combination of small and large models. First, the mine operation video stream is preprocessed by frame extraction to obtain a discrete image sequence. A lightweight YOLOv11 model is used to perform coarse target detection and filter irrelevant background. Then, a Qwen3VL multimodal large model is used to perform fine recognition of the target area and generate a structured scene description. A Deepseekr1 large model, in conjunction with the mine safety standard RAG knowledge base, completes semantic matching and anomaly judgment. Subsequently, the Qwen3VL large model is used again to fuse multiple types of data to generate standardized anomaly conclusions. Finally, Deepseekr1 generates disposal instructions, realizing closed-loop management of the entire process of early warning, rectification, and recording. This invention fully leverages the advantages of different models, rationally allocates computing power, dynamically iterates judgment rules, and ensures traceability of recognition results, effectively improving the intelligence and standardization of mine safety supervision.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, artificial intelligence, and mine video surveillance technology, specifically relating to a mine AI video surveillance detection and recognition method based on the combination of small and large models. Background Technology

[0002] Mining environments are complex, often characterized by poor lighting, long target distances, small targets, and numerous obstructions. Traditional video surveillance relies on manual monitoring, which is not only costly in terms of manpower but also prone to missed detections and delays, making it difficult to achieve efficient 24 / 7 monitoring. With the development of deep learning technology, YOLO series object detection models and multimodal large models are gradually being applied to the field of industrial visual inspection, and various AI video recognition solutions have emerged in the industry. However, existing technologies still have significant shortcomings.

[0003] Patent CN118227984A discloses a general industrial AI analysis framework of "small model for coarse screening + large model for fine judgment." However, this solution is not customized and optimized for mining scenarios and lacks multimodal data preprocessing mechanisms such as video frame extraction and target information extraction. It cannot adapt to the complex video data in mines and has poor scenario applicability. Patent CN120635764A relies solely on a single visual model to complete the detection. Currently, commonly used models in mines such as YOLOv4 and YOLOv5 also have similar problems: lightweight models meet real-time performance standards, but their recognition accuracy is insufficient under low light or target occlusion conditions; high-precision models consume a lot of computing power and cannot meet the real-time detection requirements of video streams. Furthermore, a single model can only complete basic target detection and is difficult to achieve behavior recognition and compliance judgment.

[0004] In addition, there are two other common solutions in the existing technology: one is to use a large multimodal model to complete the detection independently, which has high recognition accuracy, but large inference latency, high hardware cost, and is difficult to deploy on the edge of the mine; the other is to use a combination of "small model + fixed rule engine", where the rules rely on manual preset, cannot be dynamically updated with safety regulations, are prone to misjudgment in complex scenarios, and cannot form a complete safety supervision closed loop.

[0005] In summary, existing technical solutions cannot simultaneously address the needs for real-time performance, detection accuracy, scenario adaptability, and closed-loop management, thus failing to meet the requirements of intelligent safety supervision in mines. Therefore, there is an urgent need to propose a multi-model collaborative intelligent detection and recognition method for mine AI video surveillance, incorporating a safety knowledge base. Summary of the Invention

[0006] To address the technical problems of existing AI video recognition solutions, this invention provides a mine AI video surveillance detection and recognition method based on the combination of small and large models.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A mine AI video surveillance detection and recognition method based on the combination of small and large models includes the following steps: S1. Video data acquisition and preprocessing: Real-time acquisition of operation video streams through high-definition monitoring equipment at the mine site, frame extraction processing of continuous video streams, and conversion of dynamic video streams into discrete image sequences as standard input data for model detection; S2. Layered target coarse detection: The YOLOv11 lightweight model is called to perform global target detection on the discrete image sequence, identify visual targets in the image, output target category and target bounding box coordinate information, filter out target areas related to mining operations, and filter out irrelevant background elements; S3, Multimodal Fine Recognition: Call the Qwen3VL multimodal large model to perform multi-dimensional analysis on the operation-related target area obtained in S2, extract personnel action posture, operation tool type, protective equipment wearing status information, and generate structured scene description text; S4. Safety rule matching and anomaly determination: The Deepseekr1 thinking model is invoked, and the pre-built mine safety standard RAG knowledge base is linked. The structured scenario description text is semantically matched with the safety standard clauses in the knowledge base to determine whether there is a safety anomaly in the current operation and to mark the legal basis for the violation. S5. Multi-source result fusion: The Qwen3VL multimodal large model is called again to integrate the target coordinates of S2, the scene description of S3, and the anomaly judgment results of S4 to generate standardized anomaly conclusions with traceability. S6. Abnormal Closed-Loop Handling: The Deepseekr1 thinking model generates on-site handling instructions based on standardized abnormal conclusions, triggers the linkage early warning mechanism, retains execution records, and outputs rectification and penalty suggestions to complete the closed-loop process of mine safety supervision.

[0008] In S1, the standard frame rate of the mining operation video stream is 25 frames per second, and a single frame image is the smallest input unit for a single model inference.

[0009] In S2, the targets detected by the YOLOv11 lightweight model include workers, tools, and personal protective equipment; the output target bounding box uses quadruple coordinates [x1, y1, x2, y2], and removes non-work background areas such as idle equipment and walls.

[0010] In S3, the Qwen3VL multimodal large model recognition content includes personnel operation actions, tool type, integrity of protective equipment wearing, and on-site operation environment status, and organizes the recognition content into standardized structured text.

[0011] The mine safety regulations RAG knowledge base linked to S4 stores the "Safety Regulations for Metal and Non-metal Mines", the "Production Safety Law", relevant national mandatory standards for mining operations, and internal safety management regulations of mining enterprises.

[0012] In S4, after Deepseekr1 considers the large model and matches it with the knowledge base, it synchronously outputs the type of security anomaly, the level of violation, and the corresponding legal provisions for accountability.

[0013] The standardized anomaly conclusion generated by S5 includes at least three items: target location coordinates, description of unsafe behavior, and violation of rules and regulations.

[0014] Specifically, S6 includes: triggering on-site audible and visual warnings and online platform warnings, automatically generating violation notices, issuing rectification requirements, and completing rectification review within the prescribed time limit.

[0015] In step S2, the YOLOv11 lightweight model is used to perform front-end target coarse screening, reducing the amount of computation in invalid image areas and lowering the computational load of the large model in subsequent steps S3 and S4.

[0016] In S4, the method of manually pre-setting fixed logical rules for judgment is abandoned. Instead, the RAG knowledge base is used to support incremental input and dynamic updates of security specifications, thereby achieving automatic rule iteration and intelligent semantic matching.

[0017] Compared with the prior art, the beneficial effects of this invention are: 1. This invention adopts a layered collaborative working mode of YOLOv11 lightweight model and Qwen3VL and Deepseekr1 large model. The small model completes the fast coarse detection of video stream, filters invalid background, reduces the calculation range, and ensures the real-time response capability of video surveillance. Then, the large model is relied on to complete the fine recognition of complex scenes and fine targets. This effectively solves the problems of insufficient accuracy of single lightweight model and excessive computing power of pure large model, and is suitable for the deployment conditions of edge equipment in mining sites.

[0018] 2. This invention combines the Deepseekr1 inference model with the RAG knowledge base of mine safety regulations for compliance assessment, abandoning the traditional model of manually pre-setting fixed rules, and enabling dynamic updates and incremental maintenance of safety regulations. Simultaneously, it can accurately associate on-site operation scenarios with corresponding laws, regulations, and clauses, resulting in rigorous and clear logic for judging violations, significantly reducing the probability of misjudgments in complex operation scenarios.

[0019] 3. This invention forms a complete business loop from video acquisition, target detection, and anomaly judgment to automatic early warning, issuance of handling instructions, and retention of execution records, changing the shortcomings of traditional monitoring that "only detects, but does not handle, and is difficult to trace." The entire process operates automatically, reducing manual intervention, and can achieve real-time early warning and standardized management of mine operation safety risks, thereby improving the level of intelligence in mine safety supervision. Attached Figure Description

[0020] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0021] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0022] Figure 1 This is a schematic diagram of the workflow of the present invention; Figure 2 This is a schematic diagram of the target detection and anomaly recognition of the present invention; Figure 3 This is a schematic diagram illustrating the anomaly identification process of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. These descriptions are only for further illustrating the features and advantages of the present invention, and not for limiting the claims of the present invention. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0025] This invention is a mine AI video surveillance detection and recognition method based on a combination of small and large models. It is mainly applied to safety supervision scenarios such as painting, coating, hoisting, and handheld power tool operations in mines. It relies on high-definition monitoring equipment and edge computing devices deployed on-site in the mine to complete the entire process of automated detection. Figure 1 As shown, this invention consists of a YOLOv11 small model, a Qwen3VL multimodal large model, and a Deepseekr1 thinking large model working in a hierarchical and collaborative manner, executed sequentially according to six steps from step 1 to step 6. The specific implementation process is as follows: Step 1: Video Data Acquisition and Preprocessing High-definition monitoring equipment is deployed in various work areas of the mine. The equipment collects on-site video streams 24 / 7. In this embodiment, the standard frame rate of the video stream is set to 25 frames per second. Frame extraction processing is performed on the collected continuous dynamic video stream, decomposing the video into independent discrete image sequences. Each frame serves as the smallest input unit for a single model inference. After standardizing the image format, this discrete image sequence is used as the standard input data for subsequent model detection.

[0026] Step 2: Coarse detection of layered targets The discrete image sequence output from step 1 is input into the deployed YOLOv11 lightweight model, which then performs global object detection on the images. For example... Figure 2 As shown, this step is one of the core reasoning stages of the system. The model mainly detects three types of targets: workers, various work tools, and personal protective equipment such as goggles. After the model detection is completed, it outputs the category information of each type of target and the target bounding box coordinates in the form of a four-tuple [x1,y1,x2,y2].

[0027] Based on target category and coordinate information, the system automatically filters out target areas relevant to mining operations while removing irrelevant background elements such as idle equipment and walls. This step, through front-end target coarse screening, removes a large number of invalid image areas, effectively reducing the computational load on the large model in subsequent steps 3 and 4, and lowering the overall computing power burden.

[0028] Step 3: Multimodal Fine Recognition The target areas related to the operation obtained from step 2 are extracted and input into the Qwen3VL multimodal large model for multi-dimensional analysis. The model focuses on identifying four main categories of information: real-time personnel actions, types of on-site tools, completeness of protective equipment, and the overall on-site working environment. After identification, the above multi-dimensional information is organized and a structured scene description text with a unified format is output, providing standardized semantic data for subsequent safety rule matching.

[0029] Step 4: Security rule matching and anomaly detection This step involves building a mining safety regulations (RAG) knowledge base in advance. The knowledge base stores the "Safety Regulations for Metal and Non-metal Mines", the "Production Safety Law", relevant national mandatory standards for mining operations, and internal safety management regulations of mining enterprises.

[0030] The Deepseekr1 framework is invoked to perform intelligent semantic matching between the structured scenario description text generated in step 3 and the regulatory provisions in the RAG knowledge base. This determines whether the current work behavior exhibits any safety anomalies and marks the corresponding legal basis for any violations. After matching is complete, the specific type of safety anomaly, the level of violation, and the corresponding legal provisions for accountability are output synchronously.

[0031] This solution abandons the traditional judgment mode of manually writing fixed logical rules. Relying on the RAG knowledge base, it supports incremental input and dynamic updates of security specifications, and can realize automatic rule iteration and continuous intelligent semantic matching.

[0032] Step 5: Multi-source result fusion The Qwen3VL multimodal large model is invoked again to integrate the data from the entire process: summarizing the target bounding box coordinates output in step 2, the structured scene description generated in step 3, and the anomaly judgment results output in step 4. Finally, a standardized anomaly conclusion with complete source tracing capabilities is generated. This conclusion consistently includes three core elements: target location coordinates, description of unsafe behavior, and the corresponding regulatory clauses for the violation. For example... Figure 3 As shown, this step can intuitively output the identification results of various typical unsafe behaviors in mines, such as not wearing safety goggles during painting operations, violations during hoisting operations, and lack of protective measures when operating power tools.

[0033] Step 6: Abnormal Closed-Loop Handling Based on the standardized anomaly conclusions output in step 5, the Deepseekr1 large-scale thinking model automatically generates on-site handling instructions. The system simultaneously executes multiple handling actions: triggering on-site audible and visual warnings and online warnings on the backend management platform; automatically generating violation notices and issuing rectification requirements to on-site and management personnel; and strictly adhering to management regulations to organize and complete rectification review work within the specified time limit.

[0034] The system automatically records all execution records, including early warnings, responses, and reviews, and outputs corresponding rectification and penalty suggestions based on safety regulations. This completes a closed loop for the entire mine safety supervision process, from video collection, target detection, and anomaly determination to early warning response, rectification, and review.

[0035] The above is a preferred embodiment of the present invention. The steps of the entire method are closely coordinated, relying on the three types of models YOLOv11, Qwen3VL, and Deepseekr1 to work in a layered and collaborative manner. This not only meets the real-time detection requirements of mine video streams, but also achieves high-precision recognition and intelligent compliance judgment in complex scenarios. The overall technical solution does not exceed the protection scope described in the claims of the present invention.

[0036] The above description only illustrates the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention, and all such changes should be included within the protection scope of the present invention.

Claims

1. A mine AI video surveillance detection and recognition method based on the combination of small and large models, characterized by: Includes the following steps: S1. Video data acquisition and preprocessing: Real-time acquisition of operation video streams through high-definition monitoring equipment at the mine site, frame extraction processing of continuous video streams, and conversion of dynamic video streams into discrete image sequences as standard input data for model detection; S2. Layered target coarse detection: The YOLOv11 lightweight model is called to perform global target detection on the discrete image sequence, identify visual targets in the image, output target category and target bounding box coordinate information, filter out target areas related to mining operations, and filter out irrelevant background elements; S3, Multimodal Fine Recognition: Call the Qwen3VL multimodal large model to perform multi-dimensional analysis on the operation-related target area obtained in S2, extract personnel action posture, operation tool type, protective equipment wearing status information, and generate structured scene description text; S4. Safety rule matching and anomaly determination: The Deepseekr1 thinking model is invoked, and the pre-built mine safety standard RAG knowledge base is linked. The structured scenario description text is semantically matched with the safety standard clauses in the knowledge base to determine whether there is a safety anomaly in the current operation and to mark the legal basis for the violation. S5. Multi-source result fusion: The Qwen3VL multimodal large model is called again to integrate the target coordinates of S2, the scene description of S3, and the anomaly judgment results of S4 to generate standardized anomaly conclusions with traceability. S6. Abnormal Closed-Loop Handling: The Deepseekr1 thinking model generates on-site handling instructions based on standardized abnormal conclusions, triggers the linkage early warning mechanism, retains execution records, and outputs rectification and penalty suggestions to complete the closed-loop process of mine safety supervision.

2. The mine AI video surveillance detection and recognition method based on the combination of small and large models as described in claim 1, characterized in that: In S1, the standard frame rate of the mining operation video stream is 25 frames per second, and a single frame image is the smallest input unit for a single model inference.

3. The mine AI video surveillance detection and recognition method based on the combination of small and large models as described in claim 1, characterized in that: In S2, the targets detected by the YOLOv11 lightweight model include workers, tools, and personal protective equipment; the output target bounding box uses quadruple coordinates [x1, y1, x2, y2], and removes non-work background areas such as idle equipment and walls.

4. The mine AI video surveillance detection and recognition method based on the combination of small and large models according to claim 1, characterized in that: In S3, the Qwen3VL multimodal large model recognition content includes personnel operation actions, tool type, integrity of protective equipment wearing, and on-site operation environment status, and organizes the recognition content into standardized structured text.

5. The mine AI video surveillance detection and recognition method based on the combination of small and large models according to claim 1, characterized in that: The mine safety regulations RAG knowledge base linked to S4 stores the "Safety Regulations for Metal and Non-metal Mines", the "Production Safety Law", relevant national mandatory standards for mining operations, and internal safety management regulations of mining enterprises.

6. The mine AI video surveillance detection and recognition method based on the combination of small and large models according to claim 1, characterized in that: In S4, after Deepseekr1 considers the large model and matches it with the knowledge base, it synchronously outputs the type of security anomaly, the level of violation, and the corresponding legal provisions for accountability.

7. The mine AI video surveillance detection and recognition method based on the combination of small and large models according to claim 1, characterized in that: The standardized anomaly conclusion generated by S5 includes at least three items: target location coordinates, description of unsafe behavior, and violation of rules and regulations.

8. The mine AI video surveillance detection and recognition method based on the combination of small and large models according to claim 1, characterized in that: Specifically, S6 includes: triggering on-site audible and visual warnings and online platform warnings, automatically generating violation notices, issuing rectification requirements, and completing rectification review within the prescribed time limit.

9. The mine AI video surveillance detection and recognition method based on the combination of small and large models according to claim 1, characterized in that: In step S2, the YOLOv11 lightweight model is used to perform front-end target coarse screening, reducing the amount of computation in invalid image areas and lowering the computational load of the large model in subsequent steps S3 and S4.

10. The mine AI video surveillance detection and recognition method based on the combination of small and large models according to claim 1, characterized in that: In S4, the method of manually pre-setting fixed logical rules for judgment is abandoned. Instead, the RAG knowledge base is used to support the incremental input and dynamic updating of security specifications, thereby realizing automatic rule iteration and intelligent semantic matching.

Images