Factory safety inspection system and method based on multi-source video AI analysis

Abstract

This invention discloses a factory safety inspection system and method based on multi-source video AI analysis, relating to the fields of artificial intelligence and computer vision. It involves spatiotemporal alignment of various video streams; initial screening using a lightweight model with adaptive frame rate based on optical flow to intercept suspected abnormal events; extraction of the three-dimensional coordinates of suspected events; dynamic scheduling of the optimal surrounding mobile video sources for cross-view verification using a collaborative cost function; uploading multi-view video slices to the cloud for secondary inference and final judgment; and back-optimizing the edge through knowledge distillation; calculating a static risk index by combining the event coordinates with the distance to surrounding high-risk sources, triggering corresponding audible and visual alarms and equipment linkage commands. This invention effectively eliminates visual blind spots in fixed monitoring, reduces false alarm and missed alarm rates, optimizes edge-cloud computing power allocation, and achieves intelligent closed-loop safety management of factories with three-dimensional spatial quantitative assessment capabilities.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and computer vision technology, and specifically relates to a factory safety inspection system and method based on multi-source video AI analysis. Background Technology

[0002] With the deepening of intelligent manufacturing in industry, safe production in factories has become the lifeline of enterprise development. Especially in heavy industries such as glass manufacturing and metallurgy, the factory area contains high-temperature furnaces, high-pressure gas pipelines, and densely packed heavy automated material handling equipment. The complex working environment requires the establishment of a 24 / 7, comprehensive safety inspection mechanism to promptly detect potential hazards such as personnel violations, equipment leaks, and fires, thereby ensuring the safety of life and property.

[0003] Current factory safety inspections primarily rely on scheduled manual patrols and traditional AI video surveillance based on fixed cameras. In practical applications, traditional AI video surveillance typically employs a simple "end-to-cloud" architecture, where fixed-view network cameras are installed throughout the factory area, transmitting the entire video stream in real-time to a central server room or cloud server. Conventional target detection algorithms are then used to identify single-frame images of events such as personnel not wearing safety helmets, smoke, or flames, triggering an alarm pop-up upon detection. However, this approach faces significant challenges in the complex environment of a glass factory: First, the raw glass sheets have strong reflective properties, and the factory area is densely packed with steel structural columns, making the single-viewpoint of fixed cameras susceptible to optical refraction interference or physical obstruction. Second, traditional algorithms can only output the pixel coordinates of bounding boxes in two-dimensional images, lacking three-dimensional depth information from the real world. The system cannot perceive the specific spatial location of the alarm event, nor can it determine its distance from nearby high-risk equipment such as natural gas valves. Finally, uploading hundreds or thousands of high-definition video streams to the cloud without differentiation not only consumes massive amounts of network bandwidth but also results in a significant waste of cloud computing resources due to a large number of static, absolutely secure, and invalid images.

[0004] This reveals the following problems with existing technologies: 1. Limited by a fixed single perspective and optical interference, the visual blind zone is large, which easily leads to false alarms and missed alarms; 2. The allocation of computing power between the edge and cloud is unreasonable, and the full cloud migration without a screening mechanism results in serious waste of bandwidth and computing power; 3. The identification results of each monitoring device are only in a two-dimensional plane, lacking three-dimensional spatial perception capabilities, and cannot achieve quantitative risk assessment and closed-loop linkage of devices. Summary of the Invention

[0005] (a) Technical problems to be solved In response to the problems in related technologies, this invention provides a factory safety inspection system and method based on multi-source video AI analysis to overcome the aforementioned technical problems existing in the existing related technologies.

[0006] (II) Technical Solution To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution: S1. Add a global timestamp to the multi-source video streams within the factory, and map the spatial perspective of the multi-source video streams to a unified three-dimensional physical coordinate system of the factory through the factory's 3D BIM model to obtain spatiotemporally aligned multi-source video data. S2. Use a lightweight object detection model to perform real-time preliminary screening of spatiotemporally aligned multi-source video streams to obtain the initial category and confidence score of security events; Set safety thresholds and alarm thresholds; if the confidence score is between the safety threshold and the alarm threshold, it is judged as a suspected abnormal event. S3. Extract the two-dimensional pixel coordinates of the suspected abnormal event in the first video source, and convert them into three-dimensional spatial coordinates in the three-dimensional physical coordinate system of the factory using a perspective transformation matrix; Based on the occlusion relationships in the three-dimensional spatial coordinates and the three-dimensional BIM model, the collaborative scheduling cost function of the surrounding candidate video sources is calculated; the second video source with the smallest scheduling cost function performs cross-view verification shooting on the three-dimensional spatial coordinates. S4. Spatially align and stitch together the suspected abnormal video slice from the first video source and the cross-view verification video slice from the second video source, and input them into the visual big model for secondary reasoning, and output the final judgment result of the security event. S5. Based on the final judgment result, calculate the static risk index by combining the three-dimensional spatial coordinates of the safety event with the distance to the high-risk area of ​​the factory, and trigger the corresponding audible and visual alarm and equipment linkage command.

[0007] Preferably, step S1 includes the following steps: S11. Connect to multi-source video acquisition equipment within the factory; the multi-source video acquisition equipment includes fixed PTZ cameras, cameras built into mobile inspection AGVs, and cameras mounted on drones; The clocks of all multi-source video acquisition devices are synchronized using the Network Time Protocol (NTP), and a global timestamp is assigned to each frame of the image. S12. Obtain the factory's 3D BIM model and extract the geometric topology data and spatial coordinate data from the factory's 3D BIM model; S13. Calibrate the camera intrinsic parameter matrix and initial extrinsic parameter matrix of each multi-source video acquisition device; based on the camera intrinsic parameter matrix, extrinsic parameter matrix and the factory 3D BIM model, establish the mapping relationship matrix between the two-dimensional image pixel coordinate system and the three-dimensional physical coordinate system to obtain spatiotemporally aligned multi-source video data. Preferably, step S2 includes the following steps: S21. The spatiotemporally aligned multi-source video stream is input into a lightweight object detection model deployed on an edge computing node; the lightweight object detection model extracts multi-scale features of the image through a convolutional neural network and outputs a prediction tensor containing target location information, category probability and confidence score using a detection head; S22. Analyze the prediction tensor and extract the initial category vector of the security event and its corresponding confidence score; S23. Set a safety threshold and an alarm threshold, where the safety threshold is less than the alarm threshold; S24. If the confidence score is greater than or equal to the alarm threshold, it is determined to be a confirmed abnormal event and step S5 is triggered directly; if the confidence score is less than or equal to the safety threshold, it is determined to be safe, the frame data is discarded and the edge computing power is released; if the safety threshold is less than the confidence score and less than the alarm threshold, it is determined to be a suspected abnormal event and step S3 is triggered. Preferably, the lightweight edge target detection model in S21 also has an adaptive frame rate adjustment function: S211. Edge computing nodes calculate the average amplitude of the global motion vector of the image by calculating the dense optical flow field between the current video frame and the previous video frame. S212. If the average amplitude is less than the preset static threshold, the edge computing node will reduce the sampling inference frame rate of the video stream to the first sampling inference frame rate. If the average amplitude is greater than or equal to the preset static threshold, the edge computing node immediately restores the sampling inference frame rate to the second sampling inference frame rate. Preferably, step S3 includes the following steps: S31. Extract the center pixel coordinates of the target identified as a suspected abnormal event in the two-dimensional image of the first video source to obtain the center pixel coordinates of the suspected abnormal event target. S32. Based on the pixel coordinates of the center point of the suspected abnormal event target, combined with the camera intrinsic parameter matrix, rotation matrix and translation vector of the first video source, and the depth estimate of the suspected abnormal event target in the image, the three-dimensional spatial coordinates of the target in the three-dimensional physical coordinate system of the factory are calculated. S33. Using the three-dimensional spatial coordinates as the center of a sphere and the preset scheduling radius as the radius, retrieve all available candidate video source sets in the factory's three-dimensional BIM model; S34. Construct a cooperative scheduling cost function; calculate the time cost, distance cost, and line-of-sight occlusion cost required for each candidate video source in the candidate video source set to move / rotate to the alignment coordinates, and calculate the comprehensive cost value through the cooperative scheduling cost function; S35. Select the candidate video source with the lowest comprehensive cost value as the second video source, issue control commands to make its gimbal turn or move to the best observation position, and continuously perform cross-view verification shooting of the three-dimensional spatial coordinates for a set duration to obtain cross-view verification video slices. Preferably, step S4 includes the following steps: S41. Extract N frames of images before and after the suspected abnormal event is triggered by the first video source as the first video slice, and extract M frames of images captured by the second video source as the second video slice; synchronously upload the first video slice and the second video slice together with the corresponding camera pose parameters to the cloud server. S42. The cloud server uses a multi-view feature fusion network to extract the depth spatial feature maps of the first video slice and the second video slice respectively. S43. Calculate the homography matrix using the camera pose parameters, and reproject the feature map of the second video slice onto the view space of the first video slice using the homography matrix to achieve spatial alignment of the feature map. S44. The two sets of aligned feature maps are concatenated along the channel dimension and input into a cloud-based high-precision visual large model based on the Transformer architecture. The multi-head attention mechanism is used to perform global context interaction on the concatenated features, and the final judgment result of the security event is output. S45. Dynamically update the large visual model and the lightweight object detection model based on the final judgment result; Preferably, S45 includes the following steps: S451. The cloud server compares the final judgment result with the initial category output by the edge computing node; if the two are inconsistent, the first video slice, the second video slice and the final judgment result are stored as difficult case samples in the cloud difficult case database. S452. When the number of samples in the cloud-based difficult example database reaches the preset update threshold, the difficult example samples are used to fine-tune and train the high-precision visual large model in the cloud. After the high-precision visual model in the cloud is fine-tuned, new lightweight object detection model weights are generated through knowledge distillation technology, and these lightweight object detection model weights are then distributed to edge computing nodes for hot updates. Preferably, the steps in S452, including fine-tuning the high-precision visual model in the cloud and generating new lightweight object detection model weights through knowledge distillation, include the following: S4521. When the number of difficult sample cases in the cloud-based difficult sample database reaches the preset update threshold Nupdate, extract the image data of the difficult sample cases and the final judgment result as pseudo-labels to construct a fine-tuning training set. S4522: Freeze the parameters of the bottom feature extraction layer of the high-precision visual large model in the cloud, and only fine-tune the top attention encoder and classification head using the fine-tuning training set to update the weights of the large model. S4523. The fine-tuned high-precision visual model in the cloud is used as the teacher model, and the lightweight object detection model is used as the student model. Using unlabeled real-world video streams from factories as input, knowledge distillation is performed by calculating the KL divergence loss between the output feature maps of the teacher model and the student model. New lightweight model weights are generated and distributed to edge nodes for hot updates. Preferably, step S5 includes the following steps: S51. Receive the final judgment result output by S4 and confirm the type of security event and the three-dimensional spatial coordinates of the target; S52. In the factory's 3D BIM model, retrieve the coordinates of the preset high-risk source equipment that is closest to the 3D spatial coordinates; the preset high-risk source equipment includes high-temperature furnaces and high-pressure gas cylinders. S53. Calculate the static risk index based on the three-dimensional spatial coordinates and the coordinates of the nearest preset high-risk source equipment using the static risk index formula. S54. Based on the calculated static risk index, match the preset emergency response strategy library, trigger the corresponding audible and visual alarm level, and send emergency stop or power cut command to the associated automated equipment in the accident area. S55. Push the event type, coordinates, and static risk index to the factory's digital twin 3D screen via the WebSocket protocol; generate dynamic warning signs at the corresponding coordinates on the 3D screen, and automatically pop up the real-time monitoring screens of the first and second video sources.

[0008] Preferably, the factory safety inspection system based on multi-source video AI analysis implements the factory safety inspection method based on multi-source video AI analysis, and the system includes: Multi-source spatiotemporal alignment module: used to access multi-source video streams from fixed cameras, AGVs and drones, and to achieve spatiotemporal alignment and coordinate system mapping of video data using global timestamps and 3D BIM models; Edge-end adaptive initial screening module: Deployed on edge computing nodes, it uses a lightweight model to perform inference on video streams and outputs initial screening results of certain anomalies, safe or suspected anomalies based on a two-layer confidence threshold. Spatial perspective dynamic collaboration module: In response to suspected abnormal results, it dynamically finds and schedules an unobstructed second video source in the 3D BIM space for cross-perspective verification by calculating the collaborative scheduling cost function; Cloud-based multi-source feature fusion module: Receives video slices from multiple perspectives, aligns feature maps using homography matrix, performs secondary inference through a high-precision visual large model, and outputs the final judgment result; Digital twin and closed-loop linkage module: Calculates the static risk index based on the final judgment result and spatial distance, executes equipment linkage power-off and sound and light alarm, and performs three-dimensional visualization rendering on the digital twin screen.

[0009] (III) Beneficial Effects The present invention has the following beneficial effects: This invention introduces a spatial perspective dynamic collaborative verification mechanism based on a 3D BIM model, realizing spatial collaboration between fixed monitoring and mobile AGVs. When a suspected alarm is generated by the fixed monitoring at the edge due to glass reflection or physical equipment obstruction, the system can automatically convert two-dimensional pixels into three-dimensional physical coordinates and dynamically schedule surrounding mobile AGVs or PTZ cameras to perform unobstructed optimal cross-view shooting based on a cost function. Furthermore, it uses multi-view geometric constraints to correct depth errors, eliminating blind spots in single-view recognition and 3D positioning distortion, achieving a leap from single-point passive capture to multi-source active collaborative judgment, and solving the machine vision recognition problem caused by optical interference in complex heavy industrial scenarios.

[0010] This invention designs an edge-cloud collaborative architecture that combines edge initial screening with cloud-based secondary review. At the edge, a lightweight model, combined with optical flow adaptive frame rate adjustment and a dual-layer confidence threshold mechanism for security and alarms, filters out a massive amount of static and supposedly safe video footage, uploading only a small number of suspected abnormal video clips to the cloud. This not only saves network bandwidth and avoids wasting cloud computing power, but also, through closed-loop feedback of cloud-based difficult example mining and knowledge distillation, enables the edge model to continuously and automatically learn and optimize factory-specific characteristics, improving the system's long-term operational efficiency and generalization ability.

[0011] This invention transforms AI identification into a quantifiable risk score by combining environmental factors and spatial decay into a quantitative risk model. By introducing multi-source consistency logic judgment, task priority preemption, and manual secondary confirmation mechanisms, it ensures the safety and reliability of PLC linkage control. It realizes an intelligent closed loop from identification to handling, avoids the risk of production interruption caused by AI false alarms, and conforms to the design principles of industrial safety.

[0012] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of the invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For those skilled in the art, the drawings can be obtained from these drawings without creative effort.

[0014] Figure 1 This is a flowchart illustrating the factory safety inspection method based on multi-source video AI analysis according to the present invention. Figure 2 This is a schematic diagram of the modules of the factory safety inspection system based on multi-source video AI analysis of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the invention, and not all embodiments. Based on the embodiments of the invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the invention.

[0016] Furthermore, to address the technical problems raised in the background section, please refer to... Figure 1 This invention provides a factory safety inspection system and method based on multi-source video AI analysis, including: Example 1:

[0017] S1. Add a global timestamp to the multi-source video streams within the factory, and map the spatial perspective of the multi-source video streams to a unified three-dimensional physical coordinate system of the factory through the factory's 3D BIM model to obtain spatiotemporally aligned multi-source video data. S2. Use a lightweight object detection model to perform real-time preliminary screening of spatiotemporally aligned multi-source video streams to obtain the initial category and confidence score of security events; Set safety thresholds and alarm thresholds; if the confidence score is between the safety threshold and the alarm threshold, it is judged as a suspected abnormal event. S3. Extract the two-dimensional pixel coordinates of the suspected abnormal event in the first video source, and convert them into three-dimensional spatial coordinates in the three-dimensional physical coordinate system of the factory using a perspective transformation matrix; Based on the occlusion relationships in the three-dimensional spatial coordinates and the three-dimensional BIM model, the collaborative scheduling cost function of the surrounding candidate video sources is calculated; the second video source with the smallest scheduling cost function performs cross-view verification shooting on the three-dimensional spatial coordinates. S4. Spatially align and stitch together the suspected abnormal video slice from the first video source and the cross-view verification video slice from the second video source, and input them into the visual big model for secondary reasoning, and output the final judgment result of the security event. S5. Based on the final judgment result, calculate the static risk index by combining the three-dimensional spatial coordinates of the safety event with the distance to the high-risk area of ​​the factory, and trigger the corresponding audible and visual alarm and equipment linkage command. The above embodiments achieve strict spatiotemporal alignment of multi-source videos through timestamps and BIM models, utilize dual-layer threshold filtering of massive security data at the edge, and propose a "spatial perspective dynamic collaborative verification" mechanism based on a three-dimensional cost function to automatically schedule mobile devices such as AGVs to eliminate visual blind spots in fixed monitoring. Combined with a cloud-based multi-view feature fusion model, it solves the problem of high false alarms / missed alarms caused by reflections, obstructions, high temperatures, and moisture in complex industrial scenarios such as glass factories. By combining the recognition results with BIM spatial distance to calculate and quantify the risk index, it realizes equipment linkage and digital twin display. It overcomes the pain points of traditional AI monitoring such as wasted computing power, data silos, and lack of spatial perception capabilities, and constructs an intelligent factory security defense line from single-point passive capture to multi-source active collaborative judgment.

[0018] The above embodiment S1 includes the following steps: S11. Connect to multi-source video acquisition equipment within the factory; the multi-source video acquisition equipment includes fixed PTZ cameras, cameras built into mobile inspection AGVs, and cameras mounted on drones; The clocks of all multi-source video acquisition devices are synchronized using the Network Time Protocol (NTP), and a global timestamp is assigned to each frame of the image. In specific implementation, the above embodiment S11 is as follows: Taking the application in a large glass factory as an example, the present invention connects to fixed pan-tilt-zoom (PTZ) cameras deployed in the glass melting furnace area, annealing furnace area, and cold end cutting area; simultaneously connects to the forward-looking wide-angle cameras of the AGV trolleys that transport glass sheets in the factory area; and sends synchronous clock signals to all IPCs (network cameras) and AGV main control boards through an NTP time server deployed in the local area network to ensure that the fixed cameras... A The image captured at 10:05:00.123 shows the AGV trolley. B The image captured at 10:05:00.123 is perfectly aligned on the timeline; S12. Obtain the factory's 3D BIM model and extract the geometric topology data and spatial coordinate data from the factory's 3D BIM model; In specific implementation, the above embodiment S12 specifically involves: importing the Revit 3D BIM model of the glass factory; the Revit 3D BIM model includes the specific 3D dimensions of the melting furnace with a temperature of up to 1600 degrees Celsius, the position of the steel structure columns, and the elevation of the annealing furnace conveyor belt; extracting the absolute coordinates of all physical components in the Revit 3D BIM model (setting the center of the factory gate as the origin (0,0,0)). S13. Calibrate the camera intrinsic parameter matrix and initial extrinsic parameter matrix of each multi-source video acquisition device; based on the camera intrinsic parameter matrix, extrinsic parameter matrix and the factory 3D BIM model, establish the mapping relationship matrix between the two-dimensional image pixel coordinate system and the three-dimensional physical coordinate system to obtain spatiotemporally aligned multi-source video data. In specific implementation, the above embodiment S13 is as follows: for a fixed camera, the intrinsic parameter matrix K of the multi-source video acquisition device, such as focal length and optical center, is obtained by Zhang Zhengyou calibration method; the extrinsic parameter rotation matrix R and translation vector T of the video acquisition device relative to the BIM origin are obtained by measuring the actual height and angle of the video acquisition device installed on the steel column. The real-time extrinsic parameter matrix of the mobile AGV camera is output in real time and synchronized to the system by the laser SLAM navigation system built into the AGV; through the above operations, any pixel in any video frame establishes a strict mathematical mapping channel with the actual physical space of the glass factory. This embodiment breaks down the data silos between fixed monitoring and mobile AGVs within the glass factory by integrating a unified NTP time synchronization protocol with the Revit 3D BIM model. It maps the discrete video streams of the traditional two-dimensional plane into structured spatiotemporal data with global timestamps and three-dimensional spatial depth. It eliminates the time difference misalignment between multiple source devices, laying a spatiotemporal digital foundation for subsequent calculation of the real physical coordinates of target objects, determination of occlusion relationships, and realization of cross-view spatial linkage scheduling, thereby improving the dimensionality and usability of the factory's underlying monitoring data.

[0019] The above embodiment S2 includes the following steps: S21. The spatiotemporally aligned multi-source video stream is input into a lightweight object detection model deployed on an edge computing node; the lightweight object detection model extracts multi-scale features of the image through a convolutional neural network and outputs a prediction tensor containing target location information, category probability and confidence score using a detection head; The lightweight object detection model is based on the YOLO architecture and has undergone knowledge distillation and compression. In specific implementation, the above embodiment S21 is as follows: an edge computing box (Jetson Orin Nano in this embodiment) is installed in the weak current box near the glass melting furnace area, and a YOLOv8 model that has been compressed by cloud-based large model knowledge distillation is run; the knowledge distillation process is implemented through the following steps: a YOLOv8-based model is built on a cloud server and pre-trained using the full historical dataset of the glass factory; the full historical dataset of the glass factory includes image data and tag data of people not wearing heat-insulating safety helmets, not wearing reflective heat-insulating clothing, falling down, and glass molten material leakage; A lightweight object detection model based on the YOLOv8 architecture is constructed. The backbone network includes a C2f module to enhance gradient flow during feature extraction. The number of model parameters is compressed to less than 3 million (3M), and the computational complexity is reduced to 8.7 GFLOPs, making it suitable for the limited computing power of edge computing boxes. The same image is input into the YOLOv8 model and the lightweight object detection model. The difference between the output probability distributions of the two models is obtained as the first loss value. The error between the lightweight object detection model and the true label is combined to obtain the second loss value. The first and second loss values ​​are used for joint training, and the weight formula of the lightweight object detection model is adjusted to transfer the complex feature extraction capability and implicit association knowledge of the YOLOv8 model to the lightweight object detection model. The lightweight edge target detection model in embodiment S21 above also has an adaptive frame rate adjustment function: S211. Edge computing nodes calculate the average amplitude of the global motion vector of the image by calculating the dense optical flow field between the current video frame and the previous video frame. In specific implementation, the above embodiment S211 is as follows: the edge computing node uses the Lucas-Kanade optical flow algorithm to calculate the motion vector of all pixels in two adjacent frames, sums the motion vector amplitudes of all pixels and divides them by the total number of pixels to obtain the average amplitude of the global motion vector of the image. S212. If the average amplitude is less than the preset static threshold, it indicates that there is no obvious dynamic target in the picture. The edge computing node will reduce the sampling inference frame rate of the video stream to the first sampling inference frame rate (1 frame / second). If the average amplitude is greater than or equal to the preset static threshold, the edge computing node immediately restores the sampling inference frame rate to the second sampling inference frame rate (full frame rate 30 frames / second); thereby realizing the dynamic allocation of edge computing power and energy saving. Specifically, in the above embodiment S212: the preset static threshold is obtained through environmental noise baseline testing; during a purely static period at night in a glass factory where no one is present and no handling equipment is operating, 1000 frames are continuously captured and their average amplitude is calculated. The maximum amplitude within this period is taken and multiplied by a safety factor of 1.2 as the preset static threshold (in this embodiment, the preset static threshold is calculated to be 2.5 pixels / frame, representing the noise caused by slight vibration of the equipment or light flicker); when the average amplitude is <2.5, the image is determined to be substantially static, and the frequency is reduced to 1 frame / second; when the average amplitude is ≥2.5, it is determined that personnel or equipment have entered the image, and the frequency is immediately restored to 30 frames / second; S22. Analyze the prediction tensor and extract the initial category vector of the security event and its corresponding confidence score; In specific implementation, the above embodiment S21 is as follows: Lightweight object detection models divide the input image into S × SThe model uses a grid, with each grid cell responsible for predicting the target. The model outputs a multidimensional prediction tensor, which contains the center coordinates, width, height, and object presence probability of each prediction box. The object presence probability is output by the sigmoid function of the detection head, representing the confidence that the grid cell contains the target. The class conditional probability is obtained by normalization using the softmax function, representing the probability distribution of belonging to each category when the target is present. Confidence score C The calculation formula is: C = P O × IoU tp ; where, indicates P O Category conditional probability; IoU tp This represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. The model compares the values ​​in the category conditional probability vector and selects the index with the highest probability value as the initial category. If the probability value corresponding to "not wearing reflective heat-insulating clothing" is the highest in the vector, the system marks the detection box as this category and extracts its associated confidence score. Through this tensor-based parsing method, the system realizes the parsing from raw pixels to semantic information and the automatic qualitative and quantitative assessment of safety events. S23. Set a safety threshold T safe and alarm threshold T alarm And the safety threshold T safe <Alarm threshold T alarm ; In specific implementation, the above embodiment S23 is as follows: For the event category of "not wearing a heat-resistant safety helmet", instead of setting a single threshold of 0.5, it identifies the confidence interval where false alarms are concentrated by statistically analyzing the false alarm distribution of edge computing boxes in the glass factory over the past 30 days (in this embodiment, the calculated confidence interval for false alarms is between 0.42 and 0.8). A safety redundancy is then added to the upper and lower limits of this interval, setting them as dual-layer thresholds. Therefore, a safety threshold is set. T safe =0.40, alarm threshold T alarm =0.85; S24. If the confidence score C ≥ alarm threshold T alarm If it is determined to be a confirmed abnormal event, step S5 is triggered directly; if the confidence score is... C ≤ safety threshold T safeIf deemed safe, discard the frame data and release edge computing power; if the safety threshold is not met... T safe Confidence score C Alarm threshold T alarm If the event is identified as a suspected abnormal event, step S3 is triggered. In specific implementation, the above embodiment S24 is as follows: In the cold-end cutting area of ​​a glass factory, a worker is standing behind an iron rack piled with large-sized glass sheets; edge camera A The system detected the worker's head, but due to the strong reflection from the glass in front and the partial obstruction from the metal frame, the lightweight target detection model output a confidence score of C=0.65 for "not wearing a heat-insulating safety helmet". Since 0.40 < 0.65 < 0.85, the system neither blindly sounds an alarm (to avoid worker resentment and production interruption) nor ignores it directly (to avoid missing real violations). Instead, it marks it as a "suspected abnormal event" and proceeds to the next stage of dynamic collaborative verification. This embodiment optimizes the allocation of computing resources by deploying a lightweight YOLO model based on optical flow adaptive frame rate adjustment on edge computing nodes and introducing a dual-layer confidence threshold mechanism for security and alarms. The adaptive frame rate effectively reduces the power consumption of invalid inference for static images. The dual-layer threshold separates the data at both ends of the confidence-safe and confidence-dangerous ends, only retaining suspected abnormal events in the ambiguity zone. This not only avoids the black-and-white false alarms and false alarms caused by a single threshold, but also intercepts more than 90% of worthless video streams at the source, avoiding the bandwidth pressure caused by importing all video to the cloud and reducing the risk of bandwidth paralysis. The above embodiment S3 includes the following steps: S31. Extract the center pixel coordinates of the target identified as a suspected abnormal event in the two-dimensional image of the first video source. u , v ), thus obtaining the pixel coordinates of the center point of the suspected abnormal event target; In specific implementation, the above embodiment S31 specifically involves: continuing the application scenario of S2, extracting the worker's head behind the reflective glass from the camera. A Pixel coordinates in the image ( u =1024, v =512); S32. Based on the pixel coordinates of the center point of the suspected abnormal event target, combined with the camera intrinsic parameter matrix of the first video source. K Rotation matrix R Translation vector T And depth estimates of targets suspected of being anomalous events in the image. d The three-dimensional spatial coordinates of the target in the factory's three-dimensional physical coordinate system are calculated. P (X , Y , Z ); In specific implementation, the above embodiment S32 specifically involves estimating the depth based on the bounding box size output by the edge model. d =8.5 meters; Substituting into the formula: [ X , Y , Z ] T = R -1 *( K -1 *[ u d , v d , d ] T - T );in, u d , v d Representing pixel coordinates u , v With depth d The product of, i.e. u d = u × d , v d = v × d The calculated physical coordinates of the suspected helmetless individual are as follows: P ( X =45.2, Y =12.5, Z =1.7); Furthermore, considering the strong refractive environment of glass factories, a ray projection correction algorithm based on the BIM physical plane is introduced; from the camera optical center to the target pixel ( u , v )Emit a virtual probe ray and calculate the intersection point of the ray with the cutting table surface in the BIM model; if the depth estimate is... d If the depth error of the intersection point with the ray exceeds 15%, the position is recalibrated using the BIM physical plane intersection point coordinates as the reference; finally, the corresponding cold end No. 2 cutting table rear passage in the BIM model is obtained; S33, using the aforementioned three-dimensional spatial coordinates P ( X , Y , Z ( ) is the center of the sphere, with a preset scheduling radius. R search Using a radius, retrieve the set of all available candidate video sources in the factory's 3D BIM model; In specific implementation, the above embodiment S33 specifically involves: setting the scheduling radius. R search =20 meters; the system uses coordinates in the BIM model P Using a circular scan, three candidate video sources were found within a 20-meter radius: Candidate Source 1 (a fixed PTZ camera mounted on the ceiling). B (15 meters away), Candidate Source 2 (an AGV trolley transporting broken glass). C (8 meters away), Candidate Source 3 (a fixed camera located on the side steel column) D (12 meters away) S34. Construct a cooperative scheduling cost function; for each candidate video source in the candidate video source set, calculate the time cost, distance cost, and line-of-sight occlusion cost required for it to move / rotate to the alignment coordinate P, and calculate the comprehensive cost value through the cooperative scheduling cost function; In specific implementation, the above embodiment S34 is as follows: the formula for the cooperative scheduling cost function is: Cost = w 1* D norm + w 2* T norm + w 3* O penalty +w task ;in, D norm This represents the normalized distance cost; T norm Normalization time cost; O penalty This indicates the cost of obstructed vision. O penalty The ratio of the portion of the target observed by the camera to the three-dimensional dimensions of the target in the BIM model is obtained; w task This indicates the task weighting factor. If the AGV is performing a high-priority task such as emergency transfer of molten glass, w task Set to infinity to ensure it is not scheduled; if low-priority tasks such as broken glass cleaning are performed, preemption logic is triggered; the weight setting rule is: in security inspections, visibility is the most important factor, therefore the occlusion weight is set accordingly. w 3 is set to the highest ( w 3=0.6); secondly, response speed, time weight. w 2 = 0.3; Distance weight w 1 = 0.1; For candidate source 1 (ceiling camera B), the system retrieves the data from the camera in the BIM model.B Optical center coordinates P When a virtual ray was emitted, it was found that the ray was completely blocked by the "large overhead crane" in the BIM model. O penalty If the value is assigned to infinity, Cost = infinity, and candidate source 1 is directly eliminated; For candidate source 2 (AGV vehicle C): emit virtual ray without obstruction ( O penalty =0); the car's current front-end angle only needs to rotate 15 degrees in place to align with coordinate P, taking 0.5 seconds. T norm =0.1; distance D 2 is 8 meters. D norm = D 2 / R search =0.4; Cost=0.1*0.4+0.3*0.1+0.6*0=0.07; For candidate source 3 (side camera) D ), firing virtual rays to detect unobstructed ( O penalty =0); however, candidate source 3 is currently monitoring in the opposite direction, and it takes 3 seconds for the PTZ to rotate 180 degrees. T norm The calculated value is 0.8; distance D 3 is 12 meters. D norm = D 3 / R search =0.6; Cost=0.1*0.6+0.3*0.8+0.6*0=0.30; S35. Select the candidate video source with the lowest comprehensive cost value as the second video source, issue control commands to make its gimbal turn or move to the best observation position, and continuously perform cross-view verification shooting for a set duration on the three-dimensional spatial coordinates P to obtain cross-view verification video slices. In specific implementation, the above embodiment S35 specifically involves: after calculating the cost function, the AGV cart... C The overall cost is minimized; the system immediately transmits data to the AGV cart via the factory's 5G private network. C Issue high-priority control commands; AGV (Automated Guided Vehicle) C Pause the current broken glass transport task, rotate 15 degrees in place, and point the built-in high-definition camera at the coordinates. P ( X =45.2, Y =12.5, Z=1.7); At this time, the AGV C's perspective avoided the reflective glass in front and captured a clear 5-second video slice of the worker's side (as a second video source); Furthermore, during the process of moving to the best observation position, the AGV calls up its own LiDAR data and merges it with the BIM model path map in real time, and uses the A* algorithm and dynamic window method (DWA) to perform obstacle avoidance planning to ensure that it does not collide with personnel or other handling equipment in the complex workshop; In this embodiment, when a "suspected anomaly" caused by reflection or obstruction is detected at the edge, the system no longer waits passively, but actively converts its two-dimensional pixels into three-dimensional physical coordinates. Taking into account line obstruction, response time and spatial distance, the system dynamically schedules the optimal mobile AGV to perform a secondary scan. This leap from passive blind capture to active collaborative verification eliminates visual blind spots in the complex environment of glass factories and achieves the capture of safety hazards without increasing the cost of manual inspection.

[0020] The above embodiment S4 steps are as follows: S41. Extract N frames of images before and after the suspected abnormal event is triggered by the first video source as the first video slice, and extract M frames of images captured by the second video source as the second video slice; synchronously upload the first video slice and the second video slice together with the corresponding camera pose parameters to the cloud server. In specific implementation, the above embodiment S41 specifically involves: packaging 30 frames of video from camera A (the first video source, with reflective obstruction), 30 frames of video from AGV vehicle C (the second video source, with a clear side view), and their respective intrinsic and extrinsic parameter matrices, and sending them to the factory's private cloud server via a 5G uplink; at this time, only a small amount of questionable data is uploaded, which reduces bandwidth usage compared to the traditional full-scale cloud upload solution; S42. The cloud server uses a multi-view feature fusion network to extract the depth spatial feature maps of the first video slice and the second video slice respectively. In specific implementation, the above embodiment S42 is as follows: The multi-view feature fusion network adopts a multi-stream convolutional neural network based on the ResNet-50 backbone, and removes the original fully connected layers, retaining the feature map with an output dimension of 2048×H×W as a spatial feature extractor; the cloud server inputs the first video slice of camera A and the second video slice of AGV car C into two independent branches of the multi-view feature fusion network, and extracts deep spatial feature maps FA and FC containing rich spatial texture and semantic information through multi-layer convolution and pooling operations, providing high-dimensional feature representation for subsequent spatial alignment; S43. Calculate the homography matrix using the camera pose parameters, and reproject the feature map of the second video slice onto the view space of the first video slice using the homography matrix to achieve spatial alignment of the feature map. In specific implementation, the above embodiment S43 specifically refers to: due to the camera A and AGV trolley C The significant difference in perspectives means that directly stitching feature maps together can lead to model inconsistencies; the cloud-based approach utilizes the known extrinsic parameters of both maps to calculate the homography matrix. H = R C * R A -1 -[(( T C - R C * R A -1 * T A )*n T) / d 1]; among which, R C Indicates AGV (Automated Guided Vehicle) C The rotation matrix represents the rotation transformation from the world coordinate system to the camera C coordinate system; R A Indicates camera A The rotation matrix represents the rotation transformation from the world coordinate system to the camera A coordinate system; R C Indicates camera C The rotation matrix represents the rotation from the world coordinate system to the camera. C Rotational transformation of the coordinate system; T C Indicates AGV (Automated Guided Vehicle) C The translation vector, i.e., from the world coordinate system to... C Translation of the coordinate system; T A Indicates camera A Translation vector, i.e., from the world coordinate system to the camera. A Translation of the coordinate system; n represents the target plane (such as the ground or the plane where the molten glass leaks); d 1 represents a camera A Distance to plane n; Calculate the homography matrix using the extrinsic parameter matrix. H Perform perspective transformation calculations on the AGV vehicle. C The extracted side view feature vectors are twisted in mathematical space and projected onto the camera. A In the front view coordinate system, the feature pixels of the two are perfectly aligned in spatial semantics; Furthermore, to eliminate coordinate drift caused by optical interference, the system utilizes the first video source (camera). A) and the second video source (AGV cart) C The stereoscopic visual baseline is formed, and the disparity of corresponding feature points (such as the center of the worker's head) is extracted. d 2; Using multi-source triangulation formulas Z d =( f m * B m ) / d 2; Perform multi-source deep fusion computation; among which, Z d This represents the depth value of the target point after fusion; f m For camera focal length, B m For camera A With AGV trolley C The physical distance between the baselines; d 2 represents the parallax of the same feature point (such as the center of the worker's head); the fused depth value is fed back to the system to strictly limit the three-dimensional coordinate accuracy error in strong reflective environment to within ±5cm, providing a positional basis for subsequent static risk index calculation; S44. The two sets of aligned feature maps are concatenated along the channel dimension and input into a cloud-based high-precision visual large model based on the Transformer architecture. The multi-head attention mechanism is used to perform global context interaction on the concatenated features, and the final judgment result of the security event is output. In specific implementation, the above embodiment S44 specifically involves: inputting the aligned feature map into a cloud-based visual large model with tens of billions of parameters (this embodiment uses a model based on Swin-Transformer); the Swin-Transformer-based model contains approximately 88 million (88M) parameters and adopts a hierarchical architecture and a moving window self-attention mechanism; by dividing the stitched feature map into non-overlapping patches and performing local and global feature interactions, the scale variation and long-distance dependency problems during cross-view feature fusion are effectively solved; Swin-Transformer, through a cross-attention mechanism, discovered that in the features of camera A, the worker's head area appears as blurry silver pixels (due to reflection interference); however, in the AGV (Automated Guided Vehicle)... C Among the features, it is clearly shown that the silver pixel is actually an "aluminum foil heat-insulating and radiation-proof safety helmet" that meets national standards and is worn by the worker (it looks very different from a regular yellow engineering helmet, which caused the edge lightweight model to misjudge); the large model makes a comprehensive judgment: the worker has worn the safety helmet in accordance with regulations, and the final judgment result is "safe", correcting the suspected alarm at the edge; S45. Dynamically update the large visual model and the lightweight object detection model based on the final judgment result; The above embodiment S45 includes the following steps: S451. The cloud server compares the final judgment result with the initial category output by the edge computing node; if the two are inconsistent, the first video slice, the second video slice and the final judgment result are stored as difficult case samples in the cloud difficult case database. S452. When the number of samples in the cloud-based difficult example database reaches the preset update threshold, the difficult example samples are used to fine-tune and train the high-precision visual large model in the cloud. After the high-precision visual model in the cloud is fine-tuned, new lightweight object detection model weights are generated through knowledge distillation technology, and these lightweight object detection model weights are then distributed to edge computing nodes for hot updates. The fine-tuning training of the high-precision visual model in the cloud and the generation of new lightweight object detection model weights through knowledge distillation in embodiment S452 above include the following steps: S4521. When the number of difficult sample cases in the cloud-based difficult sample database reaches the preset update threshold Nupdate, extract the image data of the difficult sample cases and the final judgment result as pseudo-labels to construct a fine-tuning training set. In specific implementation, the above embodiment S4521 is as follows: set a preset update threshold Nupdate=500, and after the system collects 500 difficult case samples such as "aluminum foil heat insulation cap is misjudged as not wearing a safety helmet" during operation, automatically use the final judgment result "safe" output by the large model as the correct pseudo label to construct a fine-tuning dataset for model iteration. S4522: Freeze the parameters of the bottom feature extraction layer of the high-precision visual large model in the cloud, and only fine-tune the top attention encoder and classification head using the fine-tuning training set to update the weights of the large model. In specific implementation, the above embodiment S4522 is as follows: a small learning rate of 1e-5 is used, and the AdamW optimizer is used for fine-tuning training for 50 epochs; the operation of freezing the bottom layer enables the large model to quickly adapt to the unique light and shadow features of the glass factory, such as reflection and occlusion, while preventing the model from memorizing the original general features. S4523. The fine-tuned high-precision visual model in the cloud is used as the teacher model, and the lightweight object detection model is used as the student model. Using unlabeled real-world video streams from factories as input, knowledge distillation is performed by calculating the KL divergence loss between the output feature maps of the teacher model and the student model. New lightweight model weights are generated and distributed to edge nodes for hot updates. In specific implementation, the above embodiment S4523 is as follows: through soft tag knowledge distillation, the edge YOLOv8 model is forced to learn the feature response pattern of the cloud big model to the specific target "aluminum foil heat insulation cap"; after the update is issued, when the edge computing node encounters a similar aluminum foil heat insulation cap under reflective light again, it will directly output a high-confidence security result. This embodiment utilizes homography matrices to align cross-view feature maps with significant differences in mathematical space, and leverages a large-scale Swin-Transformer model with numerous parameters for global attention interaction, enabling the system to analyze information from multiple perspectives. This embodiment also designs a dynamic feedback update closed loop based on hard example mining and knowledge distillation. The large cloud model not only handles high-precision arbitration but also periodically compresses and distributes newly learned knowledge to the lightweight edge model. This mechanism of mutual promotion and optimization between cloud and edge nodes allows the entire intelligent inspection system to be continuously updated, improving the model's generalization ability and lifespan in complex industrial scenarios.

[0021] The above embodiment S5 includes the following steps: S51. Receive the final judgment result output by S4, and confirm the type of security event and the three-dimensional spatial coordinates of the target. P ( X =45.2, Y =12.5, Z =1.7); In specific implementation, the above embodiment S51 is as follows: In another scenario, the cloud fusion determination result is a confirmed "glass melt leakage (temperature up to 1500 degrees)", and the coordinates are confirmed to be in... P 2(X=80.5, Y=45.2, Z =0.5); S52. In the factory's 3D BIM model, retrieve the coordinates of the preset high-risk source equipment that is closest to the 3D spatial coordinate P. H ( X h , Y h , Z h The pre-set high-risk source equipment includes a high-temperature furnace and a high-pressure gas cylinder. In specific implementation, the above embodiment S52 specifically involves: the system searching and finding in the BIM model the distance from the material leakage point. P 2. The nearest high-risk source is the natural gas main pipeline valve, whose coordinates are: H ( X h =82.0, Y h =46.5, Z h =1.0); S53, Based on three-dimensional spatial coordinates and the coordinates of the nearest preset high-risk source equipment. H ( X h , Y h , Z h The static risk index is calculated using the static risk index formula. R isk ; In specific implementation, the above embodiment S53 is as follows: The static risk index calculation formula is: R isk = Base score *exp(- λ * L )* k env *k area ;in, Base score The basic risk score is set based on the type of the final judgment result; L coordinates P With coordinates H The Euclidean distance between them; λ Spatial attenuation coefficient k env This is an environmental correction factor (1.5 if located downstream of a vent or in an area where flammable release agents are stored, otherwise 1.0). k area The regional importance coefficient is 2.0 for the core area of ​​the melting furnace and 0.8 for the general finished product warehouse. Glass molten metal leakage is inherently extremely dangerous, so a basic hazard score is set. Base score =100; Calculate leakage points P 2. Natural gas valve H Euclidean distance Set the spatial attenuation coefficient λ = 0.2 (the spatial attenuation coefficient represents the rate at which the degree of danger decreases with increasing distance), and substitute it into the formula to calculate the static risk index. R isk =100*exp(-0.2*2.05)≈100*0.663=66.3 points; Based on the statistics of the damage radius and thermodynamic laws of historical safety accidents, it is calculated that when λ=0.2, the radiant heat caused by the leakage of molten glass (1500℃) propagates in the air, and its destructive force at a distance of 5 meters from the source decreases to about 36% of the dangerous critical value (i.e., the mathematical calculation exp(-0.25)≈0.367), which is completely consistent with the thermal radiation index decay curve law in factory safety engineering; If the leak point is 20 meters away from the valve R isk The value will drop to approximately 1.8 points; this invention uses this formula to transform a simple material leakage alarm into a quantitative risk assessment that incorporates spatial location. S54. Static risk index based on calculation R isk Combined with the multi-source consistency logic protection strategy, it matches the preset emergency response strategy library, triggers the corresponding audible and visual alarm level, and sends emergency stop or power cut command to the associated automated equipment in the accident area. In specific implementation, the above embodiment S54 is as follows: In order to prevent unplanned shutdowns triggered by false alarms from a single source AI, the system executes the following logic and verification: only when the first video source (visible light) and the second video source (such as infrared thermal imaging) simultaneously detect abnormal temperatures, or when the consistency probability of the AI ​​judgment results from two different perspectives exceeds 90%, is it allowed to perform irreversible actions such as closing the main natural gas valve. The system according to R isk Values ​​are executed in a hierarchical linkage manner; R isk When the time is less than 60, only sound and light warnings and digital twin pop-up windows will be activated; R isk When the response time is ≥60, the system enters a pre-linkage state, automatically locks the associated PLC control loop, and pushes a one-click confirmation request to the operator's handheld terminal and the central control screen; it achieves second-level interaction through WebSocket technology, allowing the operator to perform secondary confirmation through real-time multi-view feeds; if there is no response within 30 seconds and R isk If the value continues to rise to 80, the system will activate fail-safe mode and force a safe shutdown, balancing production continuity and safety. S55. Use the WebSocket protocol to transmit the event type, coordinate P, and static risk index. R isk Push the data to the factory's digital twin 3D screen; generate dynamic warning signs at the corresponding coordinates on the 3D screen, and automatically pop up real-time monitoring screens from the first and second video sources; In specific implementation, the above embodiment S55 is as follows: On the digital twin 3D screen in the factory manager's office, a red 3D light column instantly rises at the position (X=80.5, Y=45.2, Z=0.5) in the BIM model, and a floating display above the light column reads "..." R iskThe screen displays the message "66.3, high temperature material leakage approaching natural gas valve"; simultaneously, the right side of the screen automatically displays the footage from the fixed camera that detected the leakage and the infrared thermal imaging footage from the AGV camera that was dispatched to confirm it; without going to the site, the location, severity, and multi-angle on-site situation of the accident can be fully grasped through the 3D large screen, realizing an efficient emergency command closed loop; S56. Construct a secure closed-loop system with manual review and failover protection. In specific implementation, the above embodiment S56 specifically involves: setting two models: automatic linkage and manual confirmation; for R isk In extreme high-risk events (e.g., large-scale molten salt leaks) with a score >80, the system skips manual intervention and directly triggers a PLC emergency stop; for events with a score <40... R isk For events with a score of ≤80, the system will display a confirmation window on the digital twin screen within 60 seconds. If the duty officer does not click to eliminate the false alarm within 60 seconds, the system will automatically execute a safety-oriented controlled shutdown procedure according to the preset early warning response rules, ensuring a safety redundancy balance between AI false alarms and real disasters.

[0022] This embodiment combines AI visual recognition results with the depth of 3D BIM physical space and calculates a static risk index based on Euclidean distance and an exponential decay model. It transforms qualitative material leakage alarms into quantitative hazard scores, enabling the system to perceive the real threat of safety events to surrounding high-risk facilities (such as natural gas valves). The tiered emergency response triggered by the quantitative score and the hard linkage with PLC equipment enable the factory to achieve automatic circuit breaker protection when facing sudden crises. Combined with the intuitive 3D rendering of the digital twin screen, it improves the intelligence and closed-loop handling efficiency of factory safety management. Example 2:

[0023] Please see Figure 2 A factory safety inspection system based on multi-source video AI analysis, implementing the factory safety inspection method based on multi-source video AI analysis, the system comprising: Multi-source spatiotemporal alignment module: used to access multi-source video streams from fixed cameras, AGVs and drones, and to achieve spatiotemporal alignment and coordinate system mapping of video data using global timestamps and 3D BIM models; Edge-end adaptive initial screening module: Deployed on edge computing nodes, it uses a lightweight model to perform inference on video streams and outputs initial screening results of certain anomalies, safe or suspected anomalies based on a two-layer confidence threshold. Spatial perspective dynamic collaboration module: In response to suspected abnormal results, it dynamically finds and schedules an unobstructed second video source in the 3D BIM space for cross-perspective verification by calculating the collaborative scheduling cost function; Cloud-based multi-source feature fusion module: Receives video slices from multiple perspectives, aligns feature maps using homography matrix, performs secondary inference through a high-precision visual large model, and outputs the final judgment result; Digital twin and closed-loop linkage module: Calculates the static risk index based on the final judgment result and spatial distance, executes equipment linkage power-off and sound and light alarm, and performs three-dimensional visualization rendering on the digital twin screen.

[0024] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0025] The preferred embodiments of the invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention.

Claims

1. A factory safety inspection method based on multi-source video AI analysis, characterized in that, Includes the following steps: S1. Add a global timestamp to the multi-source video streams within the factory, and map the spatial perspective of the multi-source video streams to a unified three-dimensional physical coordinate system of the factory through the factory's 3D BIM model to obtain spatiotemporally aligned multi-source video data. S2. Use a lightweight object detection model to perform real-time preliminary screening of spatiotemporally aligned multi-source video streams to obtain the initial category and confidence score of security events; Set safety thresholds and alarm thresholds; if the confidence score is between the safety threshold and the alarm threshold, it is judged as a suspected abnormal event. S3. Extract the two-dimensional pixel coordinates of the suspected abnormal event in the first video source, and convert them into three-dimensional spatial coordinates in the three-dimensional physical coordinate system of the factory using a perspective transformation matrix; Based on the occlusion relationships in the three-dimensional spatial coordinates and the three-dimensional BIM model, the collaborative scheduling cost function of the surrounding candidate video sources is calculated; the second video source with the smallest scheduling cost function performs cross-view verification shooting on the three-dimensional spatial coordinates. S4. Spatially align and stitch together the suspected abnormal video slice from the first video source and the cross-view verification video slice from the second video source, and input them into the visual big model for secondary reasoning, and output the final judgment result of the security event. S5. Based on the final judgment result, calculate the static risk index by combining the three-dimensional spatial coordinates of the safety event with the distance to the high-risk area of ​​the factory, and trigger the corresponding audible and visual alarm and equipment linkage command.

2. The factory safety inspection method based on multi-source video AI analysis according to claim 1, characterized in that, S1 includes the following steps: S11. Connect to multi-source video acquisition equipment within the factory; The clocks of all multi-source video acquisition devices are synchronized using the Network Time Protocol (NTP), and a global timestamp is assigned to each frame of the image. S12. Obtain the factory's 3D BIM model and extract the geometric topology data and spatial coordinate data from the factory's 3D BIM model; S13. Calibrate the camera intrinsic parameter matrix and initial extrinsic parameter matrix of each multi-source video acquisition device; based on the camera intrinsic parameter matrix, extrinsic parameter matrix and the factory 3D BIM model, establish the mapping relationship matrix between the two-dimensional image pixel coordinate system and the three-dimensional physical coordinate system to obtain spatiotemporally aligned multi-source video data.

3. The factory safety inspection method based on multi-source video AI analysis according to claim 1, characterized in that, S2 includes the following steps: S21. The spatiotemporally aligned multi-source video stream is input into a lightweight object detection model deployed on an edge computing node; the lightweight object detection model extracts multi-scale features of the image through a convolutional neural network and outputs a prediction tensor containing target location information, category probability and confidence score using a detection head; S22. Analyze the prediction tensor and extract the initial category vector of the security event and its corresponding confidence score; S23. Set a safety threshold and an alarm threshold, where the safety threshold is less than the alarm threshold; S24. If the confidence score is greater than or equal to the alarm threshold, it is determined to be a confirmed abnormal event and step S5 is triggered directly; if the confidence score is less than or equal to the safety threshold, it is determined to be safe, the frame data corresponding to the safety event is discarded and the edge computing power is released; if the safety threshold is less than the confidence score and less than the alarm threshold, it is determined to be a suspected abnormal event and step S3 is triggered.

4. The factory safety inspection method based on multi-source video AI analysis according to claim 3, characterized in that, The lightweight edge target detection model in S21 also has an adaptive frame rate adjustment function: S211. Edge computing nodes calculate the average amplitude of the global motion vector of the image by calculating the dense optical flow field between the current video frame and the previous video frame. S212. If the average amplitude is less than the preset static threshold, the edge computing node will reduce the sampling inference frame rate of the video stream to the first sampling inference frame rate. If the average amplitude is greater than or equal to the preset static threshold, the edge computing node immediately restores the sampling inference frame rate to the second sampling inference frame rate.

5. The factory safety inspection method based on multi-source video AI analysis according to claim 3, characterized in that, S3 includes the following steps: S31. Extract the center pixel coordinates of the target identified as a suspected abnormal event in the two-dimensional image of the first video source to obtain the center pixel coordinates of the suspected abnormal event target. S32. Based on the pixel coordinates of the center point of the suspected abnormal event target, combined with the camera intrinsic parameter matrix, rotation matrix and translation vector of the first video source, and the depth estimate of the suspected abnormal event target in the image, the three-dimensional spatial coordinates of the target in the three-dimensional physical coordinate system of the factory are calculated. S33. Using the three-dimensional spatial coordinates as the center of a sphere and the preset scheduling radius as the radius, retrieve all available candidate video source sets in the factory's three-dimensional BIM model; S34. Construct a cooperative scheduling cost function; calculate the time cost, distance cost, and line-of-sight occlusion cost required for each candidate video source in the candidate video source set to move / rotate to the alignment coordinates, and calculate the comprehensive cost value through the cooperative scheduling cost function; S35. Select the candidate video source with the lowest comprehensive cost value as the second video source, issue control commands to make its gimbal turn or move to the best observation position, and continuously perform cross-view verification shooting on the three-dimensional spatial coordinates for a set duration to obtain cross-view verification video slices.

6. The factory safety inspection method based on multi-source video AI analysis according to claim 1, characterized in that, The S4 step is as follows: S41. Extract N frames of images before and after the suspected abnormal event is triggered by the first video source as the first video slice, and extract M frames of images captured by the second video source as the second video slice; synchronously upload the first video slice and the second video slice together with the corresponding camera pose parameters to the cloud server. S42. The cloud server uses a multi-view feature fusion network to extract the depth spatial feature maps of the first video slice and the second video slice respectively. S43. Calculate the homography matrix using the camera pose parameters, and reproject the feature map of the second video slice onto the view space of the first video slice using the homography matrix to achieve spatial alignment of the feature map. S44. The two sets of aligned feature maps are concatenated along the channel dimension and input into a cloud-based high-precision visual large model based on the Transformer architecture. A multi-head attention mechanism is used to perform global contextual interaction on the concatenated features, and the final judgment result of the security event is output. S45. Based on the final judgment result, dynamically update the large visual model and the lightweight object detection model.

7. The factory safety inspection method based on multi-source video AI analysis according to claim 6, characterized in that, S45 includes the following steps: S451. The cloud server compares the final judgment result with the initial category output by the edge computing node; if the two are inconsistent, the corresponding first video slice, second video slice and final judgment result are stored as difficult case samples in the cloud difficult case database. S452. When the number of samples in the cloud-based difficult example database reaches the preset update threshold, the difficult example samples are used to fine-tune and train the high-precision visual large model in the cloud. After the high-precision visual model in the cloud is fine-tuned, new lightweight object detection model weights are generated through knowledge distillation technology, and these lightweight object detection model weights are then distributed to edge computing nodes for hot updates.

8. The factory safety inspection method based on multi-source video AI analysis according to claim 7, characterized in that, The steps in S452, including fine-tuning the high-precision visual model in the cloud and generating new lightweight object detection model weights through knowledge distillation, include the following: S4521. When the number of difficult sample cases in the cloud-based difficult sample database reaches the preset update threshold Nupdate, extract the image data of the difficult sample cases and the final judgment result as pseudo-labels to construct a fine-tuning training set. S4522: Freeze the parameters of the bottom feature extraction layer of the high-precision visual large model in the cloud, and only fine-tune the top attention encoder and classification head using the fine-tuning training set to update the weights of the large model. S4523. The finely tuned high-precision visual large model in the cloud is used as the teacher model, and the lightweight object detection model is used as the student model. Using unlabeled real-world video streams of factories as input, knowledge distillation is performed by calculating the KL divergence loss between the output feature maps of the teacher model and the student model. New lightweight model weights are generated and distributed to edge nodes for hot updates.

9. The factory safety inspection method based on multi-source video AI analysis according to claim 1, characterized in that, S5 includes the following steps: S51. Receive the final judgment result output by S4 and confirm the type of security event and the three-dimensional spatial coordinates of the target; S52. In the factory's 3D BIM model, retrieve the coordinates of the preset high-risk source equipment that is closest to the 3D spatial coordinates; S53. Calculate the static risk index based on the three-dimensional spatial coordinates and the coordinates of the nearest preset high-risk source equipment using the static risk index formula. S54. Based on the calculated static risk index, match the preset emergency response strategy library, trigger the corresponding audible and visual alarm level, and send emergency stop or power cut command to the associated automated equipment in the accident area. S55. Push the event type, coordinates, and static risk index to the factory's digital twin 3D screen via the WebSocket protocol; generate dynamic warning signs at the corresponding coordinates on the 3D screen, and automatically pop up the real-time monitoring screens of the first and second video sources.

10. A factory safety inspection system based on multi-source video AI analysis, characterized in that, The system implementing the factory safety inspection method based on multi-source video AI analysis as described in any one of claims 1-9 includes: Multi-source spatiotemporal alignment module: used to access multi-source video streams from fixed cameras, AGVs and drones, and to achieve spatiotemporal alignment and coordinate system mapping of video data using global timestamps and 3D BIM models; Edge-end adaptive initial screening module: Deployed on edge computing nodes, it uses a lightweight model to perform inference on video streams and outputs initial screening results of certain anomalies, safe or suspected anomalies based on a two-layer confidence threshold. Spatial perspective dynamic collaboration module: In response to suspected abnormal results, it dynamically finds and schedules an unobstructed second video source in the 3D BIM space for cross-perspective verification by calculating the collaborative scheduling cost function; Cloud-based multi-source feature fusion module: Receives video slices from multiple perspectives, aligns feature maps using homography matrix, performs secondary inference through a high-precision visual large model, and outputs the final judgment result; Digital twin and closed-loop linkage module: Calculates the static risk index based on the final judgment result and spatial distance, executes equipment linkage power-off and sound and light alarm, and performs three-dimensional visualization rendering on the digital twin screen.

Images