AI vision safety production intelligent supervision architecture based on end-cloud cooperation

Abstract

The application provides an AI vision safety production intelligent supervision architecture based on end-cloud cooperation, which comprises: an end side for real-time video data acquisition and preprocessing, completing real-time violation preliminary screening; an edge side for receiving feature data transmitted from the end side, completing semantic feature extraction and sparse coding, realizing data compression and privacy protection transmission; a cloud side for receiving feature data transmitted from the edge side, outputting an interpretable supervision report and rectification suggestion. The application realizes real-time and efficient supervision of safety production; while significantly reducing the end side computing power and storage occupation, a lightweight perception mechanism with real-time detection efficiency is constructed; the end-edge-cloud data transmission volume is greatly reduced, and the transmission efficiency is improved; the limitation of traditional visual detection which can only identify single violation behavior is made up, and joint reasoning of visual features, text semantics and regulation clauses is realized.

Description

Technical Field

[0001] This invention relates to the field of safety production technology, and in particular to an AI-based visual intelligent supervision architecture for safety production based on edge-cloud collaboration. Background Technology

[0002] Safety in production is the bottom line and core guarantee for the high-quality development of the manufacturing industry, directly related to enterprise production efficiency and the sustainable development of the industry. However, as production scenarios become increasingly complex and the complexity of operating environments continues to rise, the traditional supervision model of manual inspection and fixed-point single video monitoring is no longer suitable for the real-time and intelligent supervision needs in complex production scenarios. Therefore, it is urgent to leverage next-generation information technologies such as artificial intelligence, computer vision, and edge computing to achieve intelligent supervision. AI visual inspection technology, with its advantages of non-contact, automation, and all-weather operation, has become the core of intelligent safety production supervision.

[0003] Edge-cloud collaborative architecture is an innovative technology that deeply integrates edge computing and cloud services. It breaks the limitations of single-edge or centralized cloud computing by deploying computing tasks in layers across the edge and cloud. The edge performs processing such as perception inference, feature extraction, and encoding transmission, while complex computations such as deep network prediction and large model inference are deployed in the cloud, forming a full-link collaborative visual supervision system of "edge / device preprocessing - cloud intelligent analysis." This architecture optimizes computing power configuration and improves data transmission and processing efficiency, providing an effective technical approach to solving problems such as limited edge computing power and high cloud transmission pressure in AI visual supervision.

[0004] Currently, domestic researchers have conducted extensive research and practice on edge-cloud collaborative architecture and AI-based visual safety production supervision, achieving a series of research results in scenarios such as violation recognition and equipment anomaly detection. For example, some studies have designed a novel lightweight and high-precision object detection model based on YOLOv8, using a lightweight backbone and multi-scale enhancements, specifically designed for robust safety wear detection, adaptable to complex safety supervision scenarios. Others have integrated edge computing with industry-specific basic classes, achieving a more real-time safety monitoring system than centralized processing. Still other studies have designed lightweight object detection methods to meet the needs of various edge devices for accurate, fast, and low-latency models. However, these studies only optimize the processing at the edge or device side, without performing hierarchical collaborative inference.

[0005] In applied research on hierarchical collaborative reasoning, some studies have proposed a smart construction site real-time monitoring system based on edge computing, which preprocesses 80% of the raw data at the edge through edge-cloud hierarchical processing, reducing the pressure on the cloud. Others have proposed CombiNet, a lightweight edge-cloud collaborative branch network for industrial monitoring, customizing intelligent edge devices to achieve low-latency real-time target detection in smart grid substation scenarios, significantly reducing end-to-end latency. Other research focuses on edge feature extraction and transmission. Existing research either emphasizes lightweight edge models or only focuses on edge data transmission speed, with little or no comprehensive consideration of edge simplification, feature extraction and encoding transmission, and deep interpretable compliance analysis in the cloud.

[0006] In summary, existing visual surveillance systems based on edge-cloud collaborative architectures still face many technical challenges that urgently need to be addressed. First, edge devices are generally constrained by computing power and storage resources. Deploying general-purpose visual inspection models directly on the edge can easily lead to high inference latency and high false negative rates, making it difficult to meet the real-time monitoring requirements of safe production. Second, many traditional architectures use raw images or general features for transmission, resulting in massive amounts of invalid data being uploaded to the cloud and causing a serious waste of bandwidth resources. Finally, existing solutions mostly focus on the visual recognition of single violations, lacking the ability to analyze the logical relationships between multiple violations, and are not deeply integrated with the safe production regulatory system. This makes it impossible to achieve full-process analysis from violation identification to compliance interpretation, resulting in detection results that are difficult to directly support regulatory decisions and final rectification implementation. Summary of the Invention

[0007] To address the aforementioned shortcomings in existing technologies, this invention provides an AI-based visual safety production intelligent supervision architecture based on edge-cloud collaboration, which solves technical problems such as limited computing power on the edge side, high cloud transmission pressure, and lack of compliance analysis in existing AI visual supervision.

[0008] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows: An AI-based visual intelligent supervision architecture for safe production based on edge-cloud collaboration, comprising edge side, cloud side, and cloud side; End-side: Used for real-time video data acquisition and preprocessing, completing real-time initial screening of violations; Edge side: Used to receive feature data transmitted from the receiving end side, complete semantic feature extraction and sparse coding, and realize data compression and privacy-preserving transmission; Cloud side: Used to receive feature data transmitted from the edge side, complete compliance analysis through multimodal large models, knowledge graphs and thought chain reasoning, achieve model privacy iteration through hierarchical federated learning, and finally output interpretable regulatory reports and rectification suggestions.

[0009] Furthermore, the edge-side includes a YOLO-NAS lightweight object detection model and a violation keyframe screening module; The YOLO-NAS lightweight object detection model achieves compression optimization through a multi-level compression strategy. The violation keyframe filtering module calculates the motion vectors and inter-frame pixel change rate between adjacent video frames based on optical flow method, and presets a scene change threshold. When the inter-frame pixel change rate is greater than the preset scene change threshold, the current frame is determined to be a valid violation keyframe and transmitted to the edge side for further processing.

[0010] Furthermore, multi-level compression strategies include: Structured channel pruning: A channel importance scoring mechanism based on the sum of absolute weights is used to calculate the contribution of convolutional channels, according to the formula. Calculate the first k Importance score of each channel In the formula, Representing the k Each channel corresponds to a convolutional kernel weight; all channels are sorted by importance score, and redundant channels with importance scores below a preset threshold are removed; INT8 Quantization: During the training phase of the YOLO-NAS lightweight object detection model, INT8 quantization is introduced to achieve numerical precision compression. The basic transformation from floating-point feature mapping to integer during the conversion to INT8 quantization is as follows: in, For FP32 format floating-point eigenvalues, As a quantitative scaling factor, This refers to the quantized INT8 integer characteristics; Operator fusion: The convolution operation, batch normalization layer, and activation function are combined into a single composite operator.

[0011] Furthermore, the edge side uses the MobileViT lightweight visual Transformer model to extract and transmit value features from the feature data initially screened from the endpoint. The process of feature extraction and transmission using the MobileViT lightweight visual Transformer model is as follows: First, local feature extraction of the image is performed using convolution: Then, the extracted feature maps are divided into blocks and fed into a multi-head self-attention layer to complete global context modeling: Finally, the features calculated through self-attention are folded and reconstructed to restore the feature map to its original size and output: .

[0012] Furthermore, the 256×256 feature map output from the edge is split into 16×16 patch image blocks, and the feature encoding process is completed by the MobileViT Block module, finally generating a 32-dimensional high information density feature vector.

[0013] Furthermore, the extracted 32-dimensional high-information-density feature vector is subjected to feature sparsification and customized encoding for transmission, as follows: First, L1 regularization is applied to the feature vectors output by the MobileViT lightweight visual Transformer model, causing more than 70% of the feature weights to approach 0, thereby achieving feature sparsity and compressing the feature dimension to 30% of the original feature. Subsequently, taking into account the data distribution characteristics of sparse features, a customized encoding strategy of Huffman coding-differential coding hybrid compression is adopted; specifically, Huffman coding is used to compress non-zero feature values, and differential coding is used to compress the differences between adjacent features.

[0014] Furthermore, the cloud side is based on the LLaVA multimodal large model and uses the LoRA low-rank adapter for lightweight fine-tuning; The calculation process of the LLaVA multimodal large model is as follows: First, image features are extracted using a visual encoder: in, Input image; It is a CLIP-ViT visual encoder used to extract global and local visual features of images; The visual features output by the visual encoder; Then, the visual features are mapped to the LLaMA multimodal large model space through a cross-modal projection layer: in, The visual features output by the visual encoder; This is a trainable cross-modal projection layer used to map the visual feature dimension to the word embedding dimension of the LLaMA multimodal large model space, thereby achieving visual-language space alignment. The visual fusion features are aligned after projection layers; Finally, LLaMA performs instruction reasoning and compliance interpretation in a multimodal large model space: in, The visual fusion features are aligned after projection layers; These are prompt words; Provide compliance judgments and textual descriptions for the model output; This is a lightweight LLaMA large language model backbone network.

[0015] Furthermore, before the lightweight fine-tuning, the backbone network of the LLaVA multimodal large model is frozen: the CLIP-ViT visual encoder and all backbone parameters of the LLaVA multimodal large model are frozen, and LoRA low-rank adapters are inserted only next to the query matrix Q, key matrix K, and value matrix V projection matrix of the attention layer.

[0016] Furthermore, the lightweight fine-tuning adopts a segmented fine-tuning strategy, including feature alignment stage fine-tuning and instruction following stage fine-tuning; Feature alignment stage fine-tuning: Training is performed only on the visual projection layer of the LLaVA multimodal large model and the inserted LoRA adapter, so that the 32-dimensional high information density visual features transmitted by the MobileViT lightweight visual Transformer model to the cloud side are aligned with the word embedding space of the LLaVA multimodal large model. Instruction follow-up phase fine-tuning: By jointly fine-tuning the LoRA low-rank adapter and the top-level decoder of the LLaMA multimodal large model, the ability of the LLaMA multimodal large model to understand various safety production compliance analysis instructions and generate compliance judgment results is enhanced.

[0017] Furthermore, on the cloud side, a knowledge graph of safety production regulations is constructed, and a mind chain prompt is set up to guide the engineering process for logical reasoning of the LLaMA multimodal large model. The process of logical reasoning through thought chains for the LLaMA multimodal large model is as follows: First, the construction of a legal knowledge graph; Then, the graph Prompt Template is stored and retrieved; Finally, the mind chain prompting project.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention presents an AI-based visual intelligent supervision method for safe production based on an edge-cloud collaborative architecture, aiming to achieve real-time and efficient supervision of safe production. It employs simplified reasoning on the edge side, high-value feature extraction and customized encoding transmission mechanisms on the edge side, and a compliance analysis framework built in the cloud using a multimodal large model integrating a knowledge graph. Based on the YOLO-NAS lightweight object detection model, structural pruning and quantization are performed, significantly reducing edge computing power and storage consumption while constructing a lightweight perception mechanism with real-time detection efficiency. A customized sparse feature encoding and efficient transmission mechanism for safe production scenarios is proposed, combining lightweight MobileViT to achieve high-value semantic feature extraction. L1 regularization sparse constraints and customized Huffman coding are used to complete multi-level feature compression, significantly reducing the amount of data transmitted between the edge and cloud and improving transmission efficiency. Finally, an interpretable compliance analysis framework integrating a multimodal large model and a knowledge graph of safe production regulations is constructed, overcoming the limitation of traditional visual detection that can only identify single violations, and enabling joint reasoning of visual features, textual semantics, and regulatory clauses. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the AI-based visual intelligent supervision architecture for safe production based on edge-cloud collaboration of the present invention. Figure 2 A structural diagram showing the comparison of the YOLO-NAS structure before and after optimization at the end side; Figure 3 This is a diagram of the cloud-based LLaMA multimodal large model compliance analysis architecture. Figure 4 For safety regulations knowledge graph; Figure 5 This is a screenshot showing the effect of scene violation detection. Detailed Implementation

[0020] The technical solution of the present invention will be further described in a non-limiting manner below with reference to the accompanying drawings and specific embodiments.

[0021] like Figure 1 As shown, this embodiment provides an AI-based visual safety production intelligent supervision architecture based on edge-cloud collaboration, including the edge side, the cloud side, and the terminal side.

[0022] End-side: Used for real-time video data acquisition and preprocessing, completing real-time initial screening of violations; Edge side: Used to receive feature data transmitted from the receiving end side, complete high-value semantic feature extraction and sparse coding, and achieve efficient data compression and privacy-preserving transmission; Cloud side: Used to receive feature data transmitted from the edge side, complete compliance analysis through multimodal large models, knowledge graphs and thought chain reasoning, achieve model privacy iteration through hierarchical federated learning, and finally output interpretable regulatory reports and rectification suggestions.

[0023] The architecture is based on a core link of minimalist edge perception, efficient edge preprocessing, and intelligent compliance interpretation in the cloud. It employs technologies such as lightweight object detection, sparse feature encoding, and multimodal large model fine-tuning, and achieves model privacy iteration through hierarchical federated learning.

[0024] Furthermore, the edge component includes a lightweight YOLO-NAS object detection model and a violation keyframe filtering module. As the perception entry point in the entire architecture, the core objective of the edge component is to achieve real-time preliminary violation detection with limited computing resources, and to achieve privacy protection and data lightweighting from the data source, laying the foundation for subsequent collaborative processing between the edge and cloud sides. In practical design applications, the edge component is an industrial camera.

[0025] The YOLO-NAS lightweight object detection model achieves extreme performance optimization through a multi-level compression strategy, such as... Figure 2 As shown.

[0026] This invention utilizes the YOLO-NAS lightweight real-time object detection model, developed by Deci AI based on Neural Architecture Search (NAS) technology, for edge-side visual perception, primarily applied to resource-constrained edge computing scenarios. This model, through its quantized perception structure and efficient branching structure, is naturally adapted to resource-constrained edge computing environments and can be applied to scenarios involving the identification of violations in safe production. To further adapt to the low computing power and low power consumption hardware constraints of edge terminals, this invention performs multi-level compression on the YOLO-NAS lightweight object detection model (structured channel pruning, INT8 quantization, and operator fusion) to achieve ultimate optimization of the model's inference performance.

[0027] Structured channel pruning: A channel importance scoring mechanism based on the sum of absolute weights is used to calculate the contribution of convolutional channels, according to the formula. Calculate the first k Importance score of each channel In the formula, Representing the k The convolutional kernel weights corresponding to each channel; The importance scores of all channels are sorted, and redundant channels with importance scores below a preset threshold are removed. The number of parameters of the YOLO-NAS lightweight object detection model is compressed to 30% of the original version, while retaining the core feature extraction capability, and structured pruning optimization is completed.

[0028] INT8 Quantization: During the training phase of the YOLO-NAS lightweight object detection model, INT8 quantization is introduced to achieve numerical precision compression.

[0029] During the conversion to INT8 quantization, the basic transformation for converting floating-point feature maps to integers is as follows: in, For FP32 format floating-point eigenvalues, As a quantitative scaling factor, This is the quantized INT8 integer feature.

[0030] By using INT8 quantization, the floating-point 32-bit weights and activation values ​​of the YOLO-NAS lightweight object detection model are quantized into 8-bit integers. Under the premise of controlling the accuracy loss to within 1%, the model size is compressed to within 20MB, and the single-frame inference latency is stably controlled to within 8ms, meeting the real-time detection requirements of the scene.

[0031] Operator fusion: By fusing convolution operations, batch normalization layers (BatchNorm), and activation functions (ReLU / ReLU6) into a single composite operator, the number of memory read / write operations and data read / write overhead during the inference process of the YOLO-NAS lightweight object detection model is reduced, further reducing the inference time of edge devices.

[0032] The YOLO-NAS lightweight target detection model, after the above multi-level optimization, significantly reduces inference latency while maintaining high target detection accuracy. It can quickly identify personnel, equipment, and environmental targets on edge devices, making it suitable for applications such as real-time target detection, security monitoring, and safety production violation identification. (See also...) Figure 2 The structure of the YOLO-NAS lightweight object detection model before and after optimization is shown.

[0033] The violation keyframe filtering module adopts a violation keyframe filtering mechanism to avoid the leakage of privacy data in the original video stream, while reducing the pressure of transmitting feature data to the cloud.

[0034] Before the violation keyframe screening module is deployed on the edge side, it calculates the motion vectors and inter-frame pixel change rate between adjacent video frames based on optical flow. A scene change threshold (such as pixel change rate > 15%) is preset. When the inter-frame pixel change rate is greater than the preset scene change threshold, the current frame is determined to be a valid violation keyframe. Only keyframes with significant changes in image content are transmitted to the edge side for subsequent processing, while static frames without changes are directly discarded. Privacy-pre-sensitization is achieved at the data source level, reducing the amount of invalid data transmission and reducing the data processing burden on the cloud side.

[0035] The edge computing platform utilizes the MobileViT lightweight visual Transformer model to extract high-value features and efficiently transmit them from the feature data initially screened at the device side. In practical applications, the edge computing platform is configured with TPU or GPU servers.

[0036] The process of feature extraction and transmission in the MobileViT lightweight visual Transformer model is as follows: First, local feature extraction of the image is performed using convolution: This is the original feature map output from the endpoint; This is the local spatial feature map extracted by convolution; Then, the extracted feature maps are divided into blocks and fed into a multi-head self-attention layer to complete global context modeling: This is the local spatial feature map extracted by convolution; Features calculated after self-attention; Finally, the features calculated through self-attention are folded and reconstructed to restore the feature map to its original size and output: Features calculated after self-attention; To restore the feature map to its original size; Specifically, the MobileViT lightweight visual Transformer model performs semantic enhancement on the feature data initially screened from the edge. Adopting the idea of ​​"Transformer as Convolution", it combines local convolution with global self-attention, achieving global context modeling while keeping the number of parameters < 6M, thereby improving the discriminative power of features for violations. During feature mapping and encoding, the 256×256 feature map output from the edge is split into 16×16 patch image blocks, which are then processed by the MobileViT Block module to generate a 32-dimensional high information density feature vector, providing a compact semantic representation for subsequent compliance analysis.

[0037] To improve transmission efficiency, this invention also performs feature sparsification and customized encoding on the extracted 32-dimensional high-information-density feature vector for transmission, as follows: First, an L1 regularization constraint (λ=0.01) is applied to the feature vector output by the MobileViT lightweight visual Transformer model, so that more than 70% of the feature weights approach 0, thereby achieving feature sparsity and compressing the feature dimension to 30% of the original feature.

[0038] Subsequently, considering the data distribution characteristics of sparse features, a customized encoding strategy combining Huffman coding and differential coding was employed. Specifically, Huffman coding was used to compress non-zero feature values, while differential coding was used for the differences between adjacent features. Through this customized encoding, the overall feature compression ratio reached 1 / 60 of the original feature compression ratio. While ensuring the integrity and validity of violation information, the amount of data transmitted is significantly reduced.

[0039] The MobileViT lightweight visual Transformer model, with fewer than 6M parameters, achieves a Top-1 accuracy of 78.4% on the ImageNet-1k dataset. It efficiently extracts global and local features from images, making it suitable for high-value semantic feature extraction and compression tasks in complex environments within safe production scenarios. This provides an efficient method for edge-side feature extraction and compressed transmission. Despite having significantly fewer parameters than traditional ViT models, the MobileViT model effectively captures global semantic information in complex scenes, making it ideal for feature encoding and dimensionality reduction transmission of safe production images at the edge.

[0040] On the cloud side, the model is based on the LLaVA multimodal large model and uses LoRA (Low-Rank Adaptation) technology for lightweight fine-tuning. This significantly reduces training costs while avoiding catastrophic forgetting, enabling the model to be quickly adapted to safe production scenarios.

[0041] The calculation process of the LLaVA multimodal large model is as follows: First, image features are extracted using a visual encoder: in, Input image; It is a CLIP-ViT visual encoder used to extract global and local visual features of images; The visual features output by the visual encoder; Then, the visual features are mapped to the LLaMA multimodal large model space through a cross-modal projection layer: in, The visual features output by the visual encoder; This is a trainable cross-modal projection layer used to map the visual feature dimension to the word embedding dimension of the LLaMA multimodal large model space, thereby achieving visual-language space alignment. The visual fusion features are aligned after projection layers; Finally, LLaMA performs instruction reasoning and compliance interpretation in a multimodal large model space: in, The visual fusion features are aligned after projection layers; These are prompt words; Provide compliance judgments and textual descriptions for the model output; This is a lightweight LLaMA large language model backbone network.

[0042] Before lightweight fine-tuning, the backbone network of the LLaVA multimodal large model is frozen: the CLIP-ViT visual encoder and all backbone parameters of the LLaVA multimodal large model are frozen, and low-rank adapters (rank r=8~16) are inserted only next to the query matrix Q, key matrix K, and value matrix V projection matrix of the attention layer. The number of trainable parameters accounts for only 0.1~0.5% of the original model, effectively reducing training overhead.

[0043] Lightweight fine-tuning employs a segmented fine-tuning strategy, including feature alignment stage fine-tuning and instruction following stage fine-tuning.

[0044] Feature alignment stage fine-tuning: Training is performed only on the visual projection layer of the LLaVA multimodal large model and the inserted LoRA adapter. This training method aligns the 32-dimensional high information density visual features transmitted from the MobileViT lightweight visual Transformer model to the cloud with the word embedding space of the LLaVA multimodal large model, ensuring that the visual features can be accurately recognized and interpreted by the language model, laying the foundation for subsequent compliance analysis.

[0045] Instruction follow-up phase fine-tuning: By jointly fine-tuning the LoRA adapter and the top-level decoder of the LLaMA multimodal large model, the ability of the LLaMA multimodal large model to understand various safety production compliance analysis instructions and generate compliance judgment results is enhanced.

[0046] To ensure the real-time performance of compliance analysis on the cloud side, this invention optimizes the inference process of the LLaMA multimodal large model: during inference, the LoRA adapter weights are merged into the original model weights without introducing additional inference latency, and the time taken for compliance analysis of a single image is controlled within 200ms.

[0047] Furthermore, to conduct compliance analysis, this invention constructs a knowledge graph of safety production regulations on the cloud side and sets up mind chain prompts and engineering guidance to complete logical reasoning of the LLaMA multimodal large model, such as... Figure 3 As shown.

[0048] LLaVA (Large Language-and-Vision Assistant) is a large vision-language multimodal model.

[0049] like Figure 4 As shown, the process of logical reasoning through the thought chain for the LLaMA multimodal large model is as follows: First, a legal knowledge graph is constructed. Entities and relationships are extracted from legal documents such as the "Production Safety Law" and the "Safety Regulations for Industrial and Commercial Enterprises," as well as typical cases. Entities such as violations, legal clauses, rectification measures, and responsible parties are extracted, and relationships such as "violation," "correspondence," and "requirement" are defined. Next, the knowledge graph is stored and queried using the graph prompt template. The Neo4j graph database is used to store the knowledge graph, supporting fast legal retrieval and related path queries based on violations.

[0050] Finally, the thought chain prompting process is implemented. A structured Prompt template forces the model to output the reasoning process step-by-step. The Prompt template is shown in Table 1.

[0051] Table 1: Furthermore, this invention employs a hierarchical federated learning framework to achieve privacy-preserving iterative updates of the model. The edge layer consists of independent edge nodes for each factory area. Each node performs local model training based on local data, calculating and uploading only the model's gradient parameters (rather than the original data). The cloud-side server receives the gradient parameters from each edge node and generates a global model through a secure aggregation algorithm. A federated averaging algorithm (FedAvg) is used to perform a weighted average of the gradients of each edge node, with the weight allocation proportional to the amount of local training data at each node, ensuring the fairness and effectiveness of the global model update. The cloud distributes the aggregated and updated global model to each edge node, where the edge nodes replace their local old models with the global model, forming an optimized process of "local training - gradient upload - cloud aggregation - model distribution".

[0052] Experiment and Results Analysis: To verify the feasibility and superiority of this invention, comparative experiments were designed across multiple dimensions. From three core dimensions—end-side resource consumption, data transmission efficiency, and compliance judgment performance—it was compared with three typical end-cloud collaborative regulatory architectures: traditional end-cloud direct data transmission, simple edge forwarding, and centralized cloud inference. Furthermore, it was compared with a smart construction site real-time monitoring system based on end-edge-cloud layering proposed in existing technologies.

[0053] The experimental dataset uses a self-built safety production supervision dataset, supplemented by publicly available industrial safety datasets to ensure data diversity and multi-scenario coverage. The self-built dataset collects on-site videos from manufacturing production lines, extracting 120,000 valid frames and labeling typical safety violations such as not wearing safety helmets / goggles, working at heights, hot work, and unauthorized operations in confined spaces. The publicly available datasets are supplemented with the SAFE-Industrial and UA-DETRAC industrial safety sub-datasets, adding 40,000 samples for special scenarios such as complex lighting, occlusion, and multiple violations overlapping.

[0054] Subsequently, all samples were divided into training, validation, and test sets in an 8:1:1 ratio, and all were subjected to data augmentation processing such as standardization (resolution uniformly set to 640×640), random flipping, and brightness adjustment to improve the model's generalization ability.

[0055] The experimental comparison objects of this invention are: 1. Traditional edge-to-cloud direct data transmission architecture, where the edge only collects video streams and the raw images are directly uploaded to the cloud for detection; 2. Simple edge forwarding architecture, where the edge completes video frame acquisition and keyframe filtering, and the edge does not perform feature processing, directly forwarding the keyframes to the cloud for analysis; 3. Cloud-centralized inference architecture, where the edge and edge only perform data transmission, and all detection, feature extraction, and analysis tasks are centrally completed in the cloud; 4. CombiNet branch network, an industrial monitoring solution based on lightweight edge detection, edge feature forwarding, and cloud-centralized inference.

[0056] End-side performance comparison analysis: Table 2 compares the edge performance metrics of each architecture, showing that the architecture of this invention significantly outperforms the experimental comparison objects in terms of edge resource consumption and real-time inference. The architecture of this invention is based on YOLO-NAS structural pruning and INT8 quantization, compressing the model parameter size to 18.6MB, far lower than the general YOLOv8 model (253.8MB) of the cloud-based centralized inference architecture in the comparison group; the single-frame inference latency is only 7.8ms, meeting the real-time detection requirements of industrial scenarios. Simultaneously, the edge CPU and GPU utilization rates of the architecture of this invention are 28.3% and 35.6%, respectively, avoiding stuttering and disconnection issues caused by resource exhaustion on the edge device, and adapting to low edge computing power.

[0057] Table 2 Comparison of end-side performance indicators: Comparative analysis of data transmission performance: By employing a multi-level data compression strategy—using edge-side keyframe filtering for transmission to the edge, followed by L1 regularization and sparsification, and then customized Huffman coding—the average data transmission volume per frame in this invention's architecture is only 1.2KB. Compared to the 1286.5KB of the traditional edge-to-cloud direct data transmission architecture, this represents a 99.9% reduction in output volume; compared to the 156.8KB of the simple edge forwarding architecture, this represents a 99.2% reduction in output volume. The transmission efficiency is improved to 8.3MB / s, with a bandwidth utilization rate of only 0.98%, far lower than the bandwidth utilization level of traditional architectures. This effectively alleviates the transmission pressure under limited bandwidth in industrial plants and significantly improves transmission efficiency. Table 3 shows the comparison results of data transmission efficiency for each architecture.

[0058] Table 3 Comparison of data transmission efficiency: Cloud-based compliance analysis performance comparison: To improve the accuracy of compliance analysis and judgment, this invention constructs a multimodal compliance analysis framework that integrates a knowledge graph of safety production regulations with a mind chain prompting engineering approach.

[0059] First, a legal knowledge graph is constructed. Knowledge extraction, specifically entity and relation extraction, is performed from legal documents such as the *Production Safety Law* and the *Safety Regulations for Industrial and Commercial Enterprises*, as well as typical cases. Entities such as violations, legal clauses, rectification measures, and responsible parties are extracted (over 500 in total). Relationships such as "violation," "correspondence," and "requirement" are defined (over 1,000 in total). Based on these entities and relationships, a knowledge graph is constructed, such as... Figure 4 As shown.

[0060] Table 4 compares cloud-based compliance analysis. The compliance determination accuracy of this invention's architecture reaches 94.2%, an 18% improvement compared to traditional architectures with compliance analysis capabilities. Simultaneously, the high-value semantic feature extraction from MobileViT at the edge enhances feature discriminative power, providing a more accurate feature foundation for cloud-based compliance identification. Therefore, the violation identification accuracy of this invention's architecture is also slightly higher than that of traditional architectures. Furthermore, the multi-layer preprocessing based on the edge and endpoint significantly reduces the computational load in the cloud, allowing the cloud to focus on core compliance reasoning tasks. The comparison shows that the compliance analysis latency of this invention's architecture is significantly reduced, by more than 30% compared to traditional architectures, meeting the needs of real-time analysis.

[0061] Table 4 Comparison of Cloud Compliance Analysis: To intuitively verify the violation identification effect of the invention architecture in real production scenarios, two typical scenarios, namely not wearing a safety helmet and violating regulations, were selected for visualization. Figure 5The images show the detection results in different scenarios. The left column shows the original scene images, and the right column shows the model detection output results. Red boxes indicate illegal targets, and green boxes indicate compliant targets.

[0062] As can be seen from the renderings, the architecture of this invention can still accurately identify the wearing status of safety helmets and illegal operation behaviors in actual production environments such as complex lighting, occlusion, and multiple target superposition. It has accurate positioning, low false detection rate, and strong scenario robustness, and can meet the needs of industrial site safety production supervision.

[0063] This invention proposes an optimized edge-cloud collaborative intelligent safety production supervision platform architecture. This architecture achieves rational allocation of computing resources and efficient data transmission through simplified edge-side reasoning, high-value semantic feature extraction and customized sparse coding at the edge, and compliance analysis design that integrates a multimodal large-scale model with a regulatory knowledge graph in the cloud. This constructs a real-time and interpretable safety production supervision system. Experimental and real-world scenario tests demonstrate that this architecture effectively reduces edge-side resource consumption and data transmission volume, significantly improving transmission efficiency and compliance judgment accuracy.

[0064] This invention can be further optimized. First, it improves cross-scenario generalization by expanding multi-industry safety production datasets and combining domain-adaptive learning to enhance cross-scenario generalization capabilities. Second, it employs a dynamic scheduling strategy for edge computing power, combining reinforcement learning techniques to achieve dual dynamic adaptive adjustment of key parameters and computing resources, adapting to dynamic changes in equipment within production scenarios and further improving computing power utilization efficiency.

[0065] This invention proposes an AI-based visual intelligent supervision method for safe production based on an edge-cloud collaborative architecture to achieve real-time and efficient supervision of safe production. It utilizes simplified reasoning on the edge side, high-value feature extraction and customized encoding transmission mechanisms on the edge side, and a compliance analysis framework built in the cloud, incorporating a multimodal large model with a knowledge graph. Based on the YOLO-NAS lightweight object detection model, structural pruning and quantization are performed, significantly reducing edge computing power and storage consumption while constructing a lightweight perception mechanism with real-time detection efficiency. A customized sparse feature encoding and efficient transmission mechanism for safe production scenarios is proposed, combining lightweight MobileViT to extract high-value semantic features. L1 regularization sparse constraints and customized Huffman coding are used to achieve multi-level feature compression, significantly reducing the amount of data transmitted between the edge and cloud and improving transmission efficiency. An interpretable compliance analysis framework integrating a multimodal large model and a knowledge graph of safe production regulations is constructed, overcoming the limitation of traditional visual detection that can only identify single violations, and enabling joint reasoning of visual features, textual semantics, and regulatory clauses.

[0066] The above description is an explanation of the present invention and not a limitation thereof. The scope of the present invention is defined by the claims. Within the scope of protection of the present invention, any form of modification may be made.

Claims

1. An AI-based visual intelligent supervision architecture for safe production based on edge-cloud collaboration, characterized in that: Including the endpoint, edge, and cloud sides; End-side: Used for real-time video data acquisition and preprocessing, completing real-time initial screening of violations; Edge side: Used to receive feature data transmitted from the receiving end side, complete semantic feature extraction and sparse coding, and realize data compression and privacy-preserving transmission; Cloud side: Used to receive feature data transmitted from the edge side, complete compliance analysis through multimodal large models, knowledge graphs and thought chain reasoning, achieve model privacy iteration through hierarchical federated learning, and finally output interpretable regulatory reports and rectification suggestions.

2. The AI-based visual intelligent supervision architecture for safe production based on edge-cloud collaboration as described in claim 1, characterized in that, The edge-side includes a lightweight YOLO-NAS object detection model and a violation keyframe filtering module; The YOLO-NAS lightweight object detection model achieves compression optimization through a multi-level compression strategy. The violation keyframe filtering module calculates the motion vectors and inter-frame pixel change rate between adjacent video frames based on optical flow method, and presets a scene change threshold. When the inter-frame pixel change rate is greater than the preset scene change threshold, the current frame is determined to be a valid violation keyframe and transmitted to the edge side for further processing.

3. The AI-based visual intelligent supervision architecture for safe production based on edge-cloud collaboration as described in claim 2, characterized in that, Multi-level compression strategies include: Structured channel pruning: A channel importance scoring mechanism based on the summation of absolute weights is used to calculate the contribution of convolutional channels, according to the formula. Calculate the first k Importance score of each channel In the formula, Representing the k Each channel corresponds to a convolutional kernel weight; all channels are sorted by importance score, and redundant channels with importance scores below a preset threshold are removed; INT8 Quantization: During the training phase of the YOLO-NAS lightweight object detection model, INT8 quantization is introduced to achieve numerical precision compression. The basic transformation from floating-point feature mapping to integer during the conversion to INT8 quantization is as follows: in, For FP32 format floating-point eigenvalues, As a quantitative scaling factor, This refers to the quantized INT8 integer characteristics; Operator fusion: The convolution operation, batch normalization layer, and activation function are combined into a single composite operator.

4. The AI-based visual intelligent supervision architecture for safe production based on edge-cloud collaboration as described in claim 1, characterized in that, On the edge side, the MobileViT lightweight visual Transformer model is used to extract and transmit value features from the feature data initially screened on the end side. The feature extraction and transmission process of the MobileViT lightweight visual Transformer model is as follows: First, local feature extraction of the image is performed using convolution: Then, the extracted feature maps are divided into blocks and fed into a multi-head self-attention layer to complete global context modeling: Finally, the features calculated through self-attention are folded and reconstructed to restore the feature map to its original size and output: 。 5. The AI-based visual intelligent supervision architecture for safe production based on edge-cloud collaboration as described in claim 4, characterized in that, The 256×256 feature map output from the edge is split into 16×16 patch image blocks, and the feature encoding process is completed by the MobileViT Block module, finally generating a 32-dimensional high information density feature vector.

6. The AI-based visual intelligent supervision architecture for safe production based on edge-cloud collaboration as described in claim 4, characterized in that, The extracted 32-dimensional high-information-density feature vector is subjected to feature sparsification and customized encoding for transmission, as follows: First, L1 regularization is applied to the feature vectors output by the MobileViT lightweight visual Transformer model, causing more than 70% of the feature weights to approach 0, thereby achieving feature sparsity and compressing the feature dimension to 30% of the original feature. Subsequently, taking into account the data distribution characteristics of sparse features, a customized encoding strategy of Huffman coding-differential coding hybrid compression is adopted; specifically, Huffman coding is used to compress non-zero feature values, and differential coding is used to compress the differences between adjacent features.

7. The AI-based visual intelligent supervision architecture for safe production based on edge-cloud collaboration as described in claim 1, characterized in that, The cloud-based model is based on the LLaVA multimodal large model and uses the LoRA low-rank adapter for lightweight fine-tuning. The calculation process of the LLaVA multimodal large model is as follows: First, image features are extracted using a visual encoder: in, Input image; It is a CLIP-ViT visual encoder used to extract global and local visual features of images; The visual features output by the visual encoder; Then, the visual features are mapped to the LLaMA multimodal large model space through a cross-modal projection layer: in, The visual features output by the visual encoder; This is a trainable cross-modal projection layer used to map the visual feature dimension to the word embedding dimension of the LLaMA multimodal large model space, thereby achieving visual-language space alignment. The visual fusion features are aligned after projection layers; Finally, LLaMA performs instruction reasoning and compliance interpretation in a multimodal large model space: in, The visual fusion features are aligned after projection layers; These are prompt words; Provide compliance judgments and textual descriptions for the model output; This is a lightweight LLaMA large language model backbone network.

8. The AI-based visual intelligent supervision architecture for safe production based on edge-cloud collaboration as described in claim 7, characterized in that, Before the lightweight fine-tuning, the backbone network of the LLaVA multimodal large model is frozen: the CLIP-ViT visual encoder and all backbone parameters of the LLaVA multimodal large model are frozen, and LoRA low-rank adapters are inserted only next to the query matrix Q, key matrix K, and value matrix V projection matrix of the attention layer.

9. The AI-based visual intelligent supervision architecture for safe production based on edge-cloud collaboration as described in claim 8, characterized in that, Lightweight fine-tuning employs a segmented fine-tuning strategy. This includes fine-tuning during the feature alignment stage and fine-tuning during the instruction following stage; Feature alignment stage fine-tuning: Training is performed only on the visual projection layer of the LLaVA multimodal large model and the inserted LoRA adapter, so that the 32-dimensional high information density visual features transmitted by the MobileViT lightweight visual Transformer model to the cloud side are aligned with the word embedding space of the LLaVA multimodal large model. Instruction follow-up phase fine-tuning: By jointly fine-tuning the LoRA low-rank adapter and the top-level decoder of the LLaMA multimodal large model, the ability of the LLaMA multimodal large model to understand various safety production compliance analysis instructions and generate compliance judgment results is enhanced.

10. The AI-based visual intelligent supervision architecture for safe production based on edge-cloud collaboration as described in claim 1, characterized in that, On the cloud side, a knowledge graph of safety production regulations is constructed, and a mind chain prompt is set up to guide the engineering process for logical reasoning of the LLaMA multimodal large model. The process of logical reasoning through thought chains for the LLaMA multimodal large model is as follows: First, the construction of a legal knowledge graph; Then, the graph Prompt Template is stored and retrieved; Finally, the mind chain prompting project.

Images