A loto information-based intelligent integrated safety management method

Abstract

This invention relates to a LOTO information-based and intelligent integrated security management method, comprising the following steps: 1) Optimizing the YOLO human body recognition algorithm under high-definition images, dividing the image into multiple regions with the same resolution as the standard image of the algorithm, and recognizing them separately; 2) Using OpenCV for image stitching, dividing, matching, and stitching these regional images so that the segmented object remains intact after the fusion of multiple images, avoiding duplicate recognition; 3) Adding motion foreground detection, using OpenCV for foreground detection; 4) Visual recognition of the number of locks; 5) Label recognition, using OCR to visually recognize security warning words; 6) Visual recognition of equipment status, developing algorithms for on-site equipment scenarios to identify the operating status of the equipment; This invention can reduce the risks caused by subjective factors of personnel in the traditional LOTO operation process, greatly improving LOTO security.

Description

[Technical Field]

[0001] This invention relates to the field of LOTO security technology, specifically a LOTO information-based and intelligent integrated security management method. [Background Technology]

[0002] LOTO (Lock Out and Tag Out) is an existing management technique for preventing workplace injuries caused by energy release. It's a method of preventing personal injury by isolating and locking certain hazardous energy sources. It includes two key aspects: First, before authorizing relevant personnel to work, equip the energy isolation device (such as a power switch or compressed air switch) with a locking device to ensure that the energy isolation device and the controlled equipment cannot be operated or powered until the locking device is removed (lock out). Second, install dedicated safety warning signs (tag out) to remind employees not to power the isolated equipment.

[0003] Current LOTO (Location-to-Operate) management technology relies on authorized personnel to switch on / off power supplies and / or padlock and tag entrances / exits. This depends on the personnel's fully conscious awareness. Safety risks are concentrated in situations where workers fail to strictly follow LOTO management rules, such as failing to lock or tag isolation devices before entering the work area, or discrepancies between the number of people and the number of locks due to tailgating. Theoretically, if this step is error-free, the safety of the LOTO design is guaranteed. However, employees' daily routines, including long, repetitive procedures, lack of tangible incentives, inactivity, fatigue, or apathy, as well as weak safety awareness, can lead to these violations and potentially cause accidents. [Summary of the Invention]

[0004] The purpose of this invention is to address the aforementioned shortcomings by providing an integrated information and intelligent security management method for LOTO, which can reduce the risks caused by subjective factors of personnel during the traditional LOTO operation process and greatly improve the security of LOTO.

[0005] To achieve the above objectives, a LOTO-based integrated information and intelligent security management method is designed, comprising the following steps:

[0006] 1) Optimize the YOLO human recognition algorithm for high-resolution images by segmenting the image into multiple regions with the same resolution as the algorithm's standard image and recognizing them separately;

[0007] 2) Use OpenCV for image stitching, divide, match and stitch these regional images so that the segmented objects remain intact after the fusion of multiple images, avoiding duplicate recognition;

[0008] 3) Add motion foreground detection. Use OpenCV for foreground detection. Use the presence of motion foreground to determine if there is a person, and then determine if there is a non-stationary human body.

[0009] 4) Visual recognition of the number of locks, training a custom YOLO model, developing algorithms for locks and latches, and using real-time video of locks and latches with sufficient resolution as input;

[0010] 5) Label recognition: Use OCR to visually recognize the safety warning words used on hazard and DANGER labels;

[0011] 6) Visual recognition of equipment status: Develop algorithms for on-site equipment scenarios to identify the operating status of the equipment.

[0012] Furthermore, step 1) specifically includes the following steps:

[0013] 1.1) Preprocessing: Load the high-resolution image, calculate the size of smaller segments based on the input size of the YOLO model, segment the original image into different resolutions, and split the high-resolution image into non-overlapping segments of each pixel; to ensure that objects near the segment boundaries are not missed, overlapping segments are used by defining the overlap percentage.

[0014] 1.2) Image segmentation: Traverse the high-resolution image, extract smaller segments based on the calculated size and overlap percentage, track the coordinates of each segment relative to the original high-resolution image, and then reconstruct the position of the detected object in the original image;

[0015] 1.3) Object detection: For each extracted segment, perform the preprocessing required by the YOLO model, input each preprocessed segment into the YOLO model to detect objects, and calculate the position of each detected object relative to the original high-resolution image based on the coordinates of the segment to which it belongs.

[0016] 1.4) Post-processing: Merge all detected objects in all segments, handle duplicate detections caused by overlapping areas, and identify them separately.

[0017] Furthermore, in step 2), image stitching using OpenCV includes the following steps:

[0018] 2.1) Load Image: Read the input images to be stitched together;

[0019] 2.2) Feature detection and description: For each input image, feature points are detected using a feature detector including but not limited to SIFT, SURF, or ORB; a feature descriptor for each keypoint in each image is computed. The descriptor is a compact representation of the local image region around the keypoint, which helps to match keypoints between images.

[0020] 2.3) Feature matching: Compare the feature descriptors of key points in different images to find potential matches. Use brute-force matching or FLANN-based matching methods to find the best match between key points in different images. Apply matching methods including but not limited to RANSAC to filter out incorrect outlier matches and retain correct inlier matches.

[0021] 2.4) Homography Matrix Estimation: For a matched image pair, calculate the homography matrix describing the relationship between the image planes;

[0022] 2.5) Image warping: Warp the input image according to the calculated homography matrix to align it on a common plane;

[0023] 2.6) Composing the panorama: Combining distorted images to create the final stitched panorama; finding the optimal seam between overlapping images to minimize visible artifacts; blending the images together; and ensuring smooth transitions in overlapping areas.

[0024] 2.7) Crop and adjust the panorama: Finally, crop out any unwanted black areas or artifacts from the panorama and adjust the overall exposure or color balance to create a visually appealing result.

[0025] Furthermore, in step 3), foreground detection using OpenCV includes the following steps:

[0026] 3.1) Reading video frames: Use the video capture object to read the input video frames;

[0027] 3.2) Choosing a background subtraction method: OpenCV provides background subtraction algorithms including, but not limited to, MOG, MOG2, and KNN;

[0028] 3.3) Initialize the background subtractor: Create an instance of the selected background subtraction method, and customize the parameters according to the selected algorithm, which include history length, threshold, shadow detection and learning rate;

[0029] 3.4) Apply background subtraction: Iterate through the video frames and apply the apply method of the background subtractor to each frame to generate a foreground mask;

[0030] 3.5) Post-processing: Since the generated foreground mask may contain noise, shadows or holes, post-processing techniques are applied to clean up the mask, using thresholding and contour detection to separate individual objects and filter out unwanted artifacts.

[0031] 3.6) Extracting foreground objects: Extracting foreground objects from the original video frames using the cleaned foreground mask;

[0032] 3.7) Analyze or track objects: Once the foreground objects are extracted, analyze or track them according to the specific application of object tracking, motion analysis, or activity recognition;

[0033] 3.8) Display or store results: Visualize the original video frames of the detected foreground objects, or store the results in a new video file or image sequence.

[0034] Furthermore, in step 4), the specific steps for training the custom YOLO model are as follows:

[0035] 4.1) Collect and prepare the dataset: Collect an image dataset containing the objects detected by the model, including locks and labels, and use annotation tools to annotate the bounding boxes and category labels for each object;

[0036] 4.2) Split the dataset: Split the dataset into a training set, a validation set, and a test set. The validation set is used to evaluate the model's performance, while the test set is used to evaluate the model's generalization ability after training.

[0037] 4.3) Convert comments: Convert comments to a YOLO-compatible format;

[0038] 4.4) Configure YOLO: Select a pre-trained YOLO architecture as a starting point, modify the configuration file to match the number of classes in the custom dataset, and adjust other parameters according to the hardware and dataset size;

[0039] 4.5) Preprocessing the dataset: Apply data augmentation techniques to increase the size and diversity of the dataset. Augmentation methods include flipping, rotating, scaling, and changing brightness and contrast.

[0040] 4.6) Initialize weights: You can start training from scratch, or you can use pre-trained weights from a similar task as a starting point for transfer learning;

[0041] 4.7) Training the model: Train a custom YOLO model using the dataset, configuration, and initialized weights. Monitor the training process and pay attention to the loss and average accuracy metrics. Stop training when the performance on the validation set reaches a plateau or begins to decline.

[0042] 4.8) Fine-tuning the model: If the model's performance is unsatisfactory, fine-tune the model by adjusting hyperparameters, changing the architecture, or adding more training data;

[0043] 4.9) Evaluate the model: Use the test set to evaluate the performance of the trained model and calculate the evaluation metrics;

[0044] 4.10) Deployment Model: Once the custom YOLO model achieves satisfactory performance, deploy it to the target environment;

[0045] 4.11) Monitoring and updating: In practical applications, continuously monitor the performance of the model, collect and label new data to address the problems discovered.

[0046] Furthermore, in step 4), the visual recognition of the lock includes the following steps:

[0047] 4.1) Preprocessing: In terms of hardware, the colors of the lock body, lock hook, and buckle plate are distinguished and selected from colors not found in the image environment; in terms of software, the object area is searched and focused on based on color characteristics and then calculated on the entire image screen.

[0048] 4.2) Apply a color that contrasts sharply with the color of the latch plate to the outer ring of the latch plate opening. When the latch plate is locked, the opening will appear as a "C" shape with the above color; when it is not locked, the opening will appear as an "O" shape.

[0049] 4.3) Visual recognition: The number of locks is determined by recognizing the number of "C" and "O" shapes on the latch plate; if the sum of the number of "C" and "O" shapes is not equal to the number of latch plate holes, an alarm "visual algorithm error" is triggered; if the number of "C" shapes is less than the number of people in the room, an alarm "LOTO violation" is triggered.

[0050] Furthermore, in step 4), during the visual identification of the lock, the lock latch plate is first marked with a prominent colored line, and then the number of locks is determined by identifying the number of segments separated by the prominent colored line.

[0051] Furthermore, in step 4), during the visual recognition of the lock, the snap plate is selected with color A, which has high saturation and brightness, the lock hook above the lock is selected as black, and the lock body is selected as a non-color A. The number of convex defects in the color A area of ​​the snap plate is identified and calculated, which is the number of locks.

[0052] Furthermore, in step 6), the visual recognition of the equipment status is performed from two aspects: ① when all locks are open or the number of locks on site is 0, the equipment is in operation; ② when there is at least one lock on site, the equipment is in operation.

[0053] Compared with the prior art, the present invention has the following advantages:

[0054] (1) This invention can reduce the risk caused by subjective factors of personnel in the traditional LOTO operation process. Through objective, highly reliable and tireless software algorithm means, it enhances the compliance of LOTO rules, improves the security of LOTO management, increases the convenience of daily operations related to LOTO, and forms corresponding program records.

[0055] (2) This invention improves security by installing cameras in the LOTO management area, applying machine vision algorithms to identify personnel and their permissions, the number of locks and the number of people in the activity area, discovering defects in the execution of LOTO rules, and triggering corresponding linkage measures.

[0056] (3) This invention retains the original LOTO management rules by default, which have been verified by global practice over many years, safeguarding the bottom line of safety, and uses the linkage between machine vision algorithms and LOTO rules to achieve prevention, recording and alarm functions;

[0057] (4) This invention can effectively eliminate the premise of human subjective consciousness in the traditional LOTO operation rules, and monitor and eliminate it through machine vision technology, which greatly improves the safety of LOTO.

[0058] (5) This invention provides a sufficient and computable historical record of compliance with and violation of rules, including photos and events labeled by visual algorithms, which is beneficial for management inspection, tracking, education and prevention.

[0059] (6) Based on full verification and extensive experience, users can consider a fully electronic LOTO, that is, directly binding the power supply switch and the detection logic for whether someone is present, as a verification and audit before powering the device, to ensure safety. [Image Description]

[0060] Figure 1 This is a schematic diagram of the invention where the outer ring of the lock latch plate opening is coated with yellow.

[0061] Figure 2 This is a schematic diagram of the invention with yellow lines marking the area around the latch plate of the lock;

[0062] Figure 3 This is a schematic diagram of the invention where the lock hook on the top of the lock is black;

[0063] Figure 4 This is a schematic diagram illustrating the identification of the label of this invention;

[0064] Figure 5 This is a flowchart illustrating the present invention. [Detailed Implementation]

[0065] The present invention provides an LOTO information-based intelligent integrated security management method, including the following steps:

[0066] 1) Optimize the YOLO human recognition algorithm under high-definition images, divide the images into multiple regions with the same resolution as the standard images of the algorithm, and perform recognition separately;

[0067] 2) Use OpenCV for image stitching, divide, match, and splice these regional images, so that the segmented objects remain complete after the fusion of multiple images, avoiding repeated recognition;

[0068] 3) Increase the detection of moving foregrounds, use OpenCV for foreground detection, and determine whether there is a person based on whether there is a moving foreground, and then determine whether there is a non-static human body;

[0069] 4) Visual recognition of the number of locks, customize the YOLO model training, develop algorithms for locks and fastening plates, and use the real-time video of on-site locks and fastening plates with sufficient resolution as input;

[0070] 5) Recognition of labels, use OCR to visually recognize the safety warning words used for danger and DANGER labels;

[0071] 6) Visual recognition of the device status, develop algorithms for on-site device scenarios, and recognize the operating status of devices.

[0072] The following further illustrates the present invention in conjunction with specific embodiments:

[0073] The present invention relates to an integrated solution for information automation and intelligence in LOTO management, hereinafter referred to as ALT (Augmented Lock Out & Tag Out). Its purpose is to reduce the risks caused by human subjective factors during the operation of traditional LOTO, enhance the compliance of LOTO rules, improve the safety of LOTO management, increase the convenience of daily operations related to LOTO, and form corresponding program records through objective, highly reliable, and tireless software algorithm means.

[0074] The working principle of the present invention is: Install cameras in the LOTO management area, apply machine vision algorithms to identify personnel and their permissions, the number of locks, and the number of people in the activity area, discover defects in the implementation of LOTO rules, and trigger corresponding linkage measures. Its main technical target functions include static lock number recognition in machine vision, dynamic employee number judgment, duplicate removal of overlapping recognition ranges of multiple cameras, and improvement of recognition accuracy. To achieve the above objectives, the specific technical solutions of the present invention are as follows:

[0075] 1) Machine vision recognizes people, locks, labels, and device status

[0076] 1.1) Optimize the YOLO human recognition algorithm for high-resolution images to improve recognition accuracy.

[0077] Because the images used by the YOLO algorithm are high-resolution images captured by high-definition cameras, they are standardized to a resolution of approximately 400*400 during facial recognition, significantly reducing the utilization of image information. Therefore, this invention improves and optimizes the algorithm by segmenting the image into multiple regions with the same resolution as the algorithm's standard image, and then recognizing each region separately, thereby effectively improving the recognition performance. The specific implementation steps are as follows:

[0078] 1.1.1) Preprocessing: Load the high-resolution image. Calculate the size of smaller segments based on the input size of the YOLO model. In this embodiment, the YOLO model requires an input size of 416x416 pixels, so the high-resolution image can be split into non-overlapping segments of 416x416 pixels. In the future, different versions of YOLO can be selected to segment the original image into different resolutions. Optionally, if you want to ensure that objects near the segment boundaries are not missed, overlapping segments can be used by defining an overlap percentage (e.g., 10-20% overlap).

[0079] 1.1.2) Image Segmentation: Traverse the high-resolution image and extract smaller segments based on the calculated size and overlap percentage. This information is crucial when reconstructing the position of detected objects in the original image by tracking the coordinates of each segment relative to the original high-resolution image.

[0080] 1.1.3) Object Detection: For each extracted segment, perform the preprocessing required by the YOLO model (e.g., normalization, resizing, etc.). Input each preprocessed segment into the YOLO model to detect objects. For each detected object, calculate its position relative to the original high-resolution image based on the coordinates of its corresponding segment.

[0081] 1.1.4) Post-processing: Merge the detected objects from all segments and handle duplicate detections due to overlapping regions. Non-maximum suppression (NMS) or other techniques can be used to handle duplicate detections. First, a list of detected objects containing their positions relative to the original high-resolution image is generated. These detections can then be visualized on the original image for further analysis or to perform other tasks as needed.

[0082] 1.2) Furthermore, when multiple cameras are installed in the same area, a single object may be captured by multiple cameras during image capture, resulting in the object being segmented into multiple images and reducing recognition accuracy. Therefore, this invention requires the division, matching, and merging of these area images so that the segmented object remains essentially intact after the fusion of multiple images, avoiding duplicate recognition and improving recognition accuracy.

[0083] The following are the main steps for image stitching using OpenCV:

[0084] 1.2.1) Load Image: Read the input images to be stitched together.

[0085] 1.2.2) Feature Detection and Description: For each input image, feature points (keypoints) are detected using a feature detector such as SIFT, SURF, or ORB. A feature descriptor is computed for each keypoint in each image. The descriptor is a compact representation of the local image region surrounding the keypoint, which helps in matching keypoints between images.

[0086] 1.2.3) Feature Matching: Compare feature descriptors of keypoints in different images to find potential matches. Use brute-force matching, FLANN-based matching, or other methods to find the best match among keypoints in different images. Apply robust matching methods, such as RANSAC, to filter out incorrect matches (outliers) and retain correct matches (inliers).

[0087] 1.2.4) Homography Matrix Estimation: For a matched image pair, the homography matrix (transformation matrix) describing the relationship between the image planes is calculated, usually using the inlier matching obtained in the previous step.

[0088] 1.2.5) Image warping: The input images are warped according to the calculated homography matrix so that they are aligned on a common plane. This step is crucial for creating seamless panoramas.

[0089] 1.2.6) Composing the Panorama: Distorted images are combined to create the final stitched panorama. This process includes: finding the optimal seam between overlapping images to minimize visible artifacts; and blending the images using techniques such as multi-band blending to ensure smooth transitions in overlapping areas.

[0090] 1.2.7) Cropping and Adjusting the Panorama: Finally, crop out any unwanted black areas or artifacts from the panorama and adjust the overall exposure or color balance to create a visually appealing result.

[0091] 2) Increase the detection of moving foreground elements.

[0092] Foreground detection is much more stable than human detection. Therefore, using the presence of moving foreground objects to determine the presence of people is a safer approach when identifying non-stationary human figures. Foreground detection (also known as background subtraction) is a technique in computer vision and image processing used to separate moving objects from static backgrounds in video sequences. The following are the steps for foreground detection using OpenCV:

[0093] 2.1) Read video frames: Use the video capture object to read the input video frames, and process each frame to detect foreground objects.

[0094] 2.2) Choosing a background subtraction method: OpenCV provides various background subtraction algorithms, such as MOG (Gaussian Mixture), MOG2, and KNN (K-Nearest Neighbors). Choose the appropriate method based on application requirements and constraints.

[0095] 2.3) Initialize background subtractor: Create an instance of the selected background subtraction method. You can customize parameters such as history length, threshold, shadow detection and learning rate according to the selected algorithm.

[0096] 2.4) Apply background subtraction: Iterate through the video frames and apply the apply() method of the background subtractor to each frame. This operation will generate a foreground mask, which is a binary image that highlights moving objects.

[0097] 2.5) Post-processing: The generated foreground mask may contain noise, shadows, or holes; apply post-processing techniques (such as morphological operations (erosion, dilation, opening, or closing operations)) to clean up the mask and make it more accurate; thresholding and contour detection can also be used to separate individual objects and filter out unwanted artifacts.

[0098] 2.6) Extracting Foreground Objects: Foreground objects are extracted from the original video frame using a cleaned foreground mask. This can be achieved by performing a bitwise AND operation between the video frame and the foreground mask.

[0099] 2.7) Analyze or track objects: Once the foreground objects are extracted, they can be analyzed or tracked according to specific applications (such as object tracking, motion analysis, or activity recognition).

[0100] 2.8) Display or store results: Visualize the original video frames of the detected foreground objects, or store the results in a new video file or image sequence.

[0101] 3) Visual recognition of the number of locks

[0102] Specifically, algorithms are developed for the customer's locks and latches, using real-time video of the locks and latches at sufficient resolution as input.

[0103] 3.1) Option 1: Custom YOLO model training. The following are the specific steps of this process:

[0104] 3.1.1) Collect and prepare the dataset: Collect an image dataset containing the objects that the model wants to detect, including locks and labels; the dataset should be diverse and represent various scenes, lighting conditions and object orientations; use annotation tools such as Labelbox, VGG Image Annotator (VIA) or labelImg to annotate the bounding boxes and class labels for each object.

[0105] 3.1.2) Splitting the dataset: Split the dataset into a training set, a validation set, and a test set; a typical split ratio is 70% for training, 20% for validation, and 10% for testing; during training, the validation set is used to evaluate the model's performance, while the test set is used to evaluate the model's generalization ability after training.

[0106] 3.1.3) Convert annotations: Convert annotations to a YOLO-compatible format, typically consisting of a text file for each image containing the bounding box's category, x, y, width, and height, all normalized to the range [0,1].

[0107] 3.1.4) Configure YOLO: Choose a pre-trained YOLO architecture (e.g., YOLOv3, YOLOv4, YOLOv5) as a starting point, modify the configuration file to match the number of classes in the custom dataset, and adjust other parameters such as learning rate, batch size, and sub-regions according to the hardware and dataset size.

[0108] 3.1.5) Preprocessing the dataset: Apply data augmentation techniques to increase the size and diversity of the dataset, which helps improve the performance of the model; common augmentation methods include flipping, rotating, scaling, and changing brightness and contrast.

[0109] 3.1.6) Initialize weights: You can start training from scratch or use pre-trained weights from a similar task as a starting point (transfer learning); it is recommended to use transfer learning because it can save time and often achieve better performance.

[0110] 3.1.7) Training the model: Train a custom YOLO model using the dataset, configuration, and initialization of weights; monitor the training process and pay attention to the loss and average precision (AP) metrics; stop training when the performance on the validation set reaches a plateau or begins to decline.

[0111] 3.1.8) Fine-tuning the model: If the model's performance is unsatisfactory, it can be fine-tuned by adjusting hyperparameters, changing the architecture, or adding more training data.

[0112] 3.1.9) Evaluate the model: Use the test set to evaluate the performance of the trained model and calculate evaluation metrics such as precision, recall, F1 score, and mean precision (mAP). If necessary, the model can be further optimized to improve these metrics.

[0113] 3.1.10) Deployment Model: Once the custom YOLO model achieves satisfactory performance, it can be deployed to the target environment, such as mobile devices, embedded systems, or cloud servers, ensuring that the optimized model meets the performance and resource requirements of the target platform.

[0114] 3.1.11) Monitoring and updating: In practical applications, continuously monitor the performance of the model, collect and label new data to solve the problems found; if necessary, the model can be updated regularly to maintain its performance and accuracy.

[0115] The above is the entire process of training a custom YOLO model. In practice, you can iterate and adjust multiple times to obtain the best performance until the effect meets your requirements.

[0116] 3.2) Option Two

[0117] 3.2.1) Preprocessing: On the hardware side, this invention incorporates relevant designs to help achieve stable recognition results. Specifically, the colors of the lock body, lock hook, and latch plate need to be differentiated. Furthermore, the selected colors are not present in the image environment. On the software side, the entire image is searched for and the target area is selected based on features such as color, and then calculations are performed.

[0118] 3.2.2) Implementation method: ① Apply yellow (or another color with a strong contrast to the snap plate color) to the outer ring of the snap plate opening. When the snap plate is used as a padlock, if attached... Figure 1 As shown, the opening appears as a yellow "C" shape; when not locked, the opening appears as an "O" shape. ② Visual Recognition: The number of locks is determined by identifying the number of "C" and "O" shapes on the latch plate (e.g., for a 6-hole latch plate, if there are 4 "C" shapes and 2 "O" shapes, the number of locks is 4). If the sum of the number of "C" shapes and "O" shapes is not equal to the number of latch plate holes, an alarm "Visual Algorithm Error" is triggered. If the number of "C" shapes is less than the number of people in the area, an alarm "LOTO Violation" is triggered.

[0119] 3.3) Option 3

[0120] 3.3.1) Pre-treatment: Mark the area around the latch plate with yellow lines.

[0121] 3.3.2) Visual Recognition: By identifying the number of segments separated by the yellow lines, as shown in the attached diagram. Figure 2 As shown in ① and ②, determine the number of locks (if the number of segments is 2, then the number of locks is 2).

[0122] 3.4) Option Four

[0123] The snap plate is selected in color A (yellow or red, or other colors with high saturation and brightness), the lock hook on top of the lock is selected in black, and the lock body is selected in a non-color A. Identify and count the number of protruding defects in the color A area of ​​the snap plate, as shown in the attached diagram. Figure 3 As shown, this is the number of locks.

[0124] 4) Tag Recognition: OCR is used to visually recognize safety warning words such as "Danger" and "DANGER" on the tags. Technically, this is achieved by calling Baidu's PaddlePaddle module.

[0125] 5) Visual recognition of equipment status: Algorithm development is carried out specifically for on-site equipment scenarios, mainly from two aspects: ① The operating status of the equipment when all locks are open or the number of locks on site is 0; ② The operating status of the equipment when there is at least one lock on site.

[0126] This invention may also include aspects such as: a) a backend application relational database system; b) a frontend BS architecture user interface; c) accounts, logins, default tags, and labels for relevant personnel; d) records related to manual processes: login, approval, entry / exit, door opening, and door closing; e) equipment process events: events indicating that the equipment can accept energy and events indicating that it cannot accept energy. Optional automated interlocking may also be included: a) communication with relevant equipment to obtain equipment status; b) communication and control implementation of electromagnetic door locks and relays in the management area; c) alarms for violations of LOTO rules, such as SMS notifications, audible and visual alarms, and preventative power supply.

[0127] This invention is not limited to the above-described embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of this invention shall be considered equivalent substitutions and shall be included within the scope of protection of this invention.

Claims

1. A LOTO information-based intelligent integrated safety management method, characterized in that, The method comprises the following steps: 1) optimizing the yolo human recognition algorithm under high-definition pictures, dividing the pictures into multiple areas with the same standard picture resolution as the algorithm, and identifying them respectively; 2) using OpenCV to perform image stitching, dividing, matching, and splicing the area pictures to make the segmented objects remain complete after the fusion of multiple pictures, avoiding repeated identification; 3) increasing the detection of moving foreground, using OpenCV for foreground detection, using whether there is a moving foreground to determine whether there is a person, and then determining whether there is a non-stationary human body; 4) visual identification of the number of locks, self-defining YOLO model training, developing an algorithm for locks and hasps, and using real-time video of the on-site lock and hasp with sufficient resolution as input; 5) label recognition, using OCR to visually identify the safety warning words on the danger and DANGER labels; 6) visual identification of the state of the equipment, developing an algorithm for the on-site equipment scene to identify the running state of the equipment; In step 4), lock visual identification comprises the following steps: 4.1) preprocessing: on the hardware, i.e. in the color selection of the lock body, lock hook and hasp, each is distinguished, and the selected color is a color that does not exist in the image environment; in terms of software, on the entire image screen, search and focus on the object area according to color characteristics and then perform calculation; 4.2) coat a color that is greatly different from the color of the hasp on the outer circle of the hole of the hasp hasp, when the hasp is hung with a lock, the hole presents a "C" shape; when there is no lock, the hole presents an "O" shape; 4.3) visual identification: determine the number of locks by identifying the number of "C" shapes and "O" shapes on the hasp; if the sum of the number of "C" shapes and "O" shapes is not equal to the number of hasp holes, an alarm "visual algorithm error" is given; if the number of "C" shapes is less than the number of people in the field, an alarm "LOTO violation" is given.

2. The LOTO informationized intelligentized integrated safety management method of claim 1, wherein, Step 1) comprises the following steps: 1.1) preprocessing: load a high-resolution image, calculate the size of a smaller segment according to the input size of the YOLO model, divide the original image into different resolutions, and split the high-resolution image into non-overlapping segments for each pixel; if you want to ensure that no objects near the segment boundary are missed, use overlapping segments by defining an overlap percentage; 1.2) split the image: traverse the high-resolution image, extract smaller segments according to the calculated size and overlap percentage, track the coordinates of each segment relative to the original high-resolution image, and then reconstruct the position of the detected object in the original image; 1.3) object detection: for each extracted segment, perform the preprocessing required by the YOLO model, input each preprocessed segment into the YOLO model to detect objects, and for each detected object, calculate its position relative to the original high-resolution image according to the coordinates of the segment it belongs to; 1.4) post-processing: merge all the objects detected in the segments, handle the repeated detection due to overlapping areas, and identify them respectively.

3. The LOTO informationized intelligentized integrated safety management method of claim 1, wherein, In step 2), using OpenCV for image stitching comprises the following steps: 2.1) Load images: Read the input images to be stitched together; 2.2) Feature detection and description: For each input image, detect feature points using a feature detector including but not limited to SIFT, SURF, or ORB; compute a feature descriptor for each keypoint in each image, which is a compact representation of the local image region around the keypoint, helping to match keypoints between images; 2.3) Feature matching: Compare the feature descriptors of key points in different images to find potential matches, find the best matches between key points of different images using brute-force matching or FLANN-based matching methods, apply matching methods including but not limited to RANSAC, filter out incorrect outlier matches and keep correct inlier matches; 2.4) Homography matrix estimation: For matched image pairs, compute a homography matrix that describes the relationship between the image planes; 2.5) Image warping: Warp the input images according to the computed homography matrix to align them on a common plane; 2.6) Assemble the panorama: Combine the warped images together to create the final stitched panorama; find the best seam between overlapping images to minimize visible artifacts, blend the images together, ensuring smooth transitions in overlapping areas; 2.7) Crop and adjust the panorama: Finally, crop any unwanted black areas or artifacts from the panorama and adjust the overall exposure or color balance to create a visually appealing result.

4. The LOTO informationized intelligentized integrated safety management method of claim 1, wherein, In step 3), foreground detection using OpenCV includes the following steps: 3.1) Read video frames: Read input video frames using a video capture object; 3.2) Choose background subtraction method: OpenCV provides background subtraction algorithms including but not limited to MOG, MOG2, and KNN; 3.3) Initialize background subtractor: Create an instance of the chosen background subtraction method, customize parameters according to the selected algorithm, including history length, threshold, shadow detection, and learning rate; 3.4) Apply background subtraction: Iterate through video frames, apply the apply method of the background subtractor to each frame to generate a foreground mask; 3.5) Post-processing: Since the generated foreground mask may contain noise, shadows, or holes, apply post-processing techniques to clean up the mask, use thresholding and contour detection to separate individual objects and filter out unwanted artifacts; 3.6) Extract foreground objects: Use the cleaned foreground mask to extract foreground objects from the original video frames; 3.7) Analyze or track objects: Once the foreground objects are extracted, analyze or track them according to the specific application of object tracking, motion analysis, or activity recognition; 3.8) Display or store results: Visualize the original video frames with detected foreground objects, or store the results in a new video file or image sequence.

5. The LOTO informationized intelligentized integrated safety management method of claim 1, wherein, In step 4), the specific steps for customizing YOLO model training are: 4.1) Collect and prepare dataset: Collect an image dataset containing objects to be detected by the model, including locks and labels, use annotation tools to label each object with a bounding box and class label; 4.2) Split the dataset: Split the dataset into training, validation, and test sets, with the validation set used to evaluate the model's performance and the test set used to assess the model's generalization ability after training; 4.3) Convert annotations: Convert annotations to a format compatible with YOLO; 4.4) Configure YOLO: Choose a pre-trained YOLO architecture as a starting point, modify the configuration file to match the number of classes in the custom dataset, and adjust other parameters based on hardware and dataset size; 4.5) Preprocess the dataset: Apply data augmentation techniques to increase the size and diversity of the dataset, including flipping, rotation, scaling, and changing brightness and contrast; 4.6) Initialize weights: You can either train from scratch or use pre-trained weights from a similar task as a starting point for transfer learning; 4.7) Train the model: Train the customized YOLO model using the dataset, configuration, and initialized weights, monitor the training process, and focus on loss and average precision indicators, stop training when the performance on the validation set reaches a plateau or starts to decline; 4.8) Fine-tune the model: If the model's performance is not satisfactory, fine-tune the model by adjusting hyperparameters, changing the architecture, or adding more training data; 4.9) Evaluate the model: Evaluate the trained model's performance using the test set and calculate evaluation metrics; 4.10) Deploy the model: Once the custom YOLO model achieves satisfactory performance, deploy it to the target environment; 4.11) Monitor and update: In practical applications, continuously monitor the model's performance, collect and label new data to address issues found.

6. The LOTO informationized intelligentized integrated safety management method of claim 1, wherein: In step 4, when the lock visual recognition is performed, first mark the lock with a distinctive color line around the buckle plate, then identify the number of segments separated by the distinctive color line to determine the number of locks.

7. The LOTO informationized intelligentized integrated safety management method of claim 1, wherein: In step 4, when the lock visual recognition is performed, the buckle plate is selected with a high saturation and brightness A color, the lock hook above the lock body is selected with black, and the lock body is selected with a non-A color. The number of convex defects in the A color area of the buckle plate is calculated, which is the number of locks.

8. The LOTO informationized intelligentized integrated safety management method of claim 1, wherein: In step 6, when the device status is visually recognized, it is identified from two aspects: ① When all locks are open or there are no locks on site, the running status of the device; ② When there is at least one lock on site, the running status of the device.

Images