An artificial intelligence-based operation and maintenance video abnormal behavior automatic auditing method

Abstract

The application discloses an operation and maintenance video abnormal behavior automatic auditing method based on artificial intelligence, and specifically comprises the following steps: collecting operation and maintenance field videos in real time through proxy software, and carrying out pretreatment such as denoising, background removal and image enhancement to improve the accuracy of target detection and behavior recognition; the behavior target of operation and maintenance personnel in the video is identified by using a YOLO target detection model to obtain boundary box coordinates, category labels and confidence scores; the detection result is transmitted to a Kalman filter for continuous tracking of the target to generate motion trajectory data; 3D convolutional neural network (3D-CNN) is used to analyze space-time features, judge and label abnormal behaviors; finally, the labeled information is integrated with the video data, video metadata is embedded through FFmpeg, the quick jump function of the labeled information is realized, the audit personnel are helped to quickly locate and examine abnormal behaviors, and the auditing efficiency and the intelligent level of operation and maintenance management are greatly improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence and computer vision, and specifically relates to an operation and maintenance video abnormal behavior automatic auditing method based on artificial intelligence. BACKGROUND

[0002] With the continuous development of informationization and intelligent technology, the operation and maintenance management of modern enterprises, especially large industrial enterprises and data centers, is facing more and more challenges. Traditional operation and maintenance management methods mostly rely on manual monitoring and inspection, which can ensure the stability of daily operation, but have many limitations. Manual review not only requires a large number of human resources, but is also easily affected by human fatigue, subjectivity and negligence, etc., which leads to the inability to timely discover potential safety hazards or abnormal behaviors. These abnormal behaviors may include improper operation, equipment failure, data anomalies, rule violations, etc., which pose a threat to the safety of enterprises, the stability of equipment, and work efficiency, etc.

[0003] In recent years, video monitoring technology has been widely used in operation and maintenance management. Through real-time video monitoring of the workplace, the on-site situation can be fully grasped, and potential safety hazards and operation abnormalities can be discovered in a timely manner. However, traditional video monitoring-based abnormal behavior detection methods mostly rely on manual analysis or rule setting, lack intelligent and automated processing capabilities, and are difficult to respond to complex and variable operation and maintenance environments in real time and efficiently. Especially when dealing with a large amount of video data, manual review is inefficient and prone to missing details.

[0004] Therefore, developing an operation and maintenance video abnormal behavior automatic auditing method based on artificial intelligence can automatically analyze video data using deep learning technology, accurately identify abnormal behaviors and provide timely warnings, which is an important direction to improve the intelligent level of operation and maintenance management and ensure the safe operation of enterprises. SUMMARY

[0005] The purpose of the application is to use artificial intelligence technology as the core, combine it with video target detection and behavior recognition methods, and apply it to the automatic auditing of abnormal behaviors in operation and maintenance management. Through the efficient feature extraction capability of the deep learning model, the motion trajectory and behavior characteristics of the target object are captured from the operation and maintenance video, and the potential abnormal behaviors are intelligently identified and classified. The method further utilizes metadata embedding technology to combine the identification results of abnormal behaviors with video data, achieving significant labeling and information recording of abnormal behaviors. Auditing personnel can quickly locate key behaviors with the help of embedded labeling information, skip redundant video content, effectively improve work efficiency and auditing accuracy, and thus realize efficient monitoring, automatic analysis and intelligent auditing of operation and maintenance behaviors.

[0006] In order to achieve the above purpose, the technical solutions provided by the application are as follows:

[0007] Step 1: Collect on-site operation and maintenance video data in real time through agent software and perform corresponding preprocessing.

[0008] Step 2: Use the YOLO object detection model to identify target objects in the video and determine their positions. At the same time, combine the Kalman filter to track the target in real time and generate the target's continuous motion trajectory.

[0009] Step 3: Use a 3D convolutional neural network (3D-CNN) to identify anomalies in real-time behavioral data.

[0010] Step 4: Integrate the recognition results with the real-time video file to generate a video file with anomaly annotations.

[0011] Furthermore, since video capture is performed at a fixed frequency to rapidly acquire video frames, a filtering mechanism needs to be introduced to remove videos that do not have recognition value in order to avoid processing meaningless redundant data, thereby reducing the computational burden on the subsequent 3D-CNN recognition module.

[0012] Furthermore, the specific steps for target detection and real-time tracking include:

[0013] Step 21: Use the collected video as input to provide the pre-trained YOLO (You Only Look Once) object detection model for the detection of behavioral objects by maintenance personnel.

[0014] Step 22: The YOLO model processes each frame of the video, automatically identifying the operational personnel's behavior targets in the scene and locating the target's specific position (bounding box) in the image. Simultaneously, the YOLO model passes the bounding box coordinates, class label, and confidence score of the identified target to the Kalman filter, providing data support for subsequent target tracking tasks.

[0015] Step 23: After receiving the target information of the operation and maintenance personnel's behavior from the YOLO model, the Kalman filter combines the target's historical motion trajectory and real-time detection information to predict the target's position at the next moment, thereby achieving continuous tracking of the target and generating the target's motion trajectory data in the video.

[0016] Furthermore, the training methods for the YOLO model include:

[0017] The YOLO model is trained using classic object detection datasets (such as COCO or PASCAL VOC datasets) as the training set to achieve accurate identification and localization of target objects in operation and maintenance scenarios. Simultaneously, based on practical application needs, operation and maintenance-related video and image data are collected and labeled to construct a test set for complex scenarios, used to evaluate the model's performance in real-world applications.

[0018] To achieve a balance between detection accuracy and processing speed, the YOLO model employs a multi-scale feature extraction network structure, capturing the spatial features and contextual information of targets through multi-layer convolutional networks. During training, the intersection-over-union (IoU) ratio between predefined anchor boxes and ground truth bounding boxes is used as part of the loss function to optimize the loss for target class prediction and bounding box location, thereby improving detection accuracy. Data augmentation techniques (such as random cropping, horizontal flipping, and color dithering) are widely used in training to enhance the model's adaptability to diverse scenes.

[0019] The activation function used is Leaky ReLU, ensuring that negative values ​​are not completely ignored and improving gradient flow to enhance training efficiency. To avoid overfitting, regularization methods such as Dropout and weight decay are introduced to enhance the network's generalization ability and stability, enabling it to adapt to changing operational environments. These training strategies effectively improve the performance and reliability of the YOLO model in operational object detection tasks.

[0020] Furthermore, the specific steps for identifying abnormal behavior include:

[0021] Step 31: The Kalman filter takes the generated motion trajectory of the operation and maintenance personnel and the corresponding video frames as input data and passes them to the pre-trained 3D convolutional neural network (3D-CNN) to provide basic data for the spatiotemporal analysis of behavioral features.

[0022] Step 32: The 3D-CNN receives and processes the motion trajectory of the operation and maintenance personnel and its associated continuous video frames generated by the Kalman filter. It extracts the motion and behavior features of the target through spatiotemporal data analysis and determines whether the operation and maintenance personnel's behavior is abnormal, thereby achieving accurate abnormal behavior identification.

[0023] Furthermore, training methods for 3D-CNN models include:

[0024] The 3D-CNN model is pre-trained using classic video datasets (such as UCF101 or Kinetics-400) as training sets to extract spatiotemporal features from videos and identify behavioral patterns. Simultaneously, for specific operational scenarios, abnormal behavior data from operational videos is collected and labeled to construct a dedicated test set for validating the model's performance in real-world applications.

[0025] To balance recognition accuracy and computational efficiency, the 3D-CNN network employs a 5-layer convolutional structure with kernel sizes of 16, 32, 64, 128, and 256, respectively, to achieve multi-level spatiotemporal feature extraction from video frames. The 3D convolutional layers capture the temporal relationships and spatial information between video frames, while pooling layers reduce feature dimensionality, lowering computational complexity while preserving key features. Furthermore, an attention mechanism is introduced, using sigmoid and softmax functions to generate weights that highlight spatiotemporal regions more correlated with abnormal behavior, thereby significantly improving the model's ability to recognize abnormal behavior.

[0026] Convolutional and fully connected layers employ the ReLU activation function to improve gradient flow and computational efficiency while avoiding the vanishing gradient problem. To enhance the model's generalization ability, Dropout is introduced during training, randomly deactivating some neuron connections with a certain probability, thereby strengthening the network's robustness and reducing the risk of overfitting. The Adam optimization algorithm is used, accelerating the convergence of model parameters through an adaptive learning rate. The loss function combines classification loss and time series consistency loss to improve the model's accuracy and consistency in action recognition tasks.

[0027] Furthermore, the specific steps for integrating the annotation results with the generation of video files with anomaly annotations include:

[0028] Step 41: Integrate the identification results with relevant information from the current video frame (including video ID, timestamp, etc.), and use FFmpeg to embed the annotation information into the video file's metadata to ensure the video file can completely store the behavioral analysis results. Simultaneously, remind auditors to use a video playback tool to play the annotated video file for auditing.

[0029] Step 42: Use a video playback tool to play the labeled video file, parse the embedded metadata to obtain the labeling information, and display the labeling results at the corresponding time points. Depending on the type of labeled behavior, a quick jump function can be supported to help auditors quickly locate and efficiently review abnormal operational behaviors.

[0030] Beneficial Effects: This invention proposes an automated auditing method for abnormal behavior in operation and maintenance videos based on artificial intelligence. It integrates YOLO object detection, Kalman filters, and 3D-CNN technologies to achieve accurate identification and automatic annotation of abnormal behaviors in operation and maintenance videos. Compared to traditional abnormal behavior detection methods that rely on manual monitoring or rule-based settings, this invention features high intelligence and real-time capabilities. It can automatically and efficiently process massive amounts of video data, significantly reducing the workload of manual review while improving the accuracy and timeliness of anomaly detection. Furthermore, through the quick jump function of annotation information, auditors can quickly locate the key positions of abnormal behaviors, thereby significantly shortening audit time and improving work efficiency and management reliability. Through the innovative integration of the above technologies, this invention provides an efficient and secure intelligent auditing solution for complex and ever-changing operation and maintenance scenarios. Attached Figure Description

[0031] The accompanying drawings used in this invention will be described below.

[0032] Figure 1 Overview diagram of automated auditing methods for abnormal behavior

[0033] Figure 2 A flowchart illustrating the methodology of an automated auditing system for abnormal behavior. Detailed Implementation

[0034] To provide a clearer explanation of the technical solution of this invention, a detailed description of the invention will be given below based on the accompanying drawings. Furthermore, this description is for illustrative purposes only, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention.

[0035] This invention discloses an automatic auditing method for abnormal behavior in operation and maintenance videos based on artificial intelligence. The specific steps include: First, real-time acquisition of video data from the operation and maintenance site via agent software, followed by preprocessing such as noise reduction, removal of irrelevant backgrounds, and image enhancement to improve the accuracy of subsequent target detection and behavior recognition. Then, the YOLO target detection model is used to process each frame of video image, identifying and labeling the behavioral targets of the operation and maintenance personnel, and obtaining the bounding box coordinates, category labels, and confidence scores of the targets. Next, the detection results are passed to a Kalman filter for continuous target tracking, generating target motion trajectory data. Subsequently, a trained 3D convolutional neural network (3D-CNN) is used to analyze the spatiotemporal characteristics of the targets, determine whether abnormal behavior exists, and label it. Finally, the labeled information is integrated with the video data, and the labeled information is embedded into the video metadata using FFmpeg. When playing the video using a video playback tool, the labeled information supports quick jump functionality, helping auditors quickly locate and review abnormal behavior, thereby significantly improving auditing efficiency and the level of intelligence in operation and maintenance management.

[0036] Figure 1 This paper presents an outline of the automatic auditing method for abnormal behavior according to the present invention.

[0037] Figure 2 A schematic diagram of the automatic abnormal behavior auditing system method of the present invention is shown. The automatic abnormal behavior auditing process specifically includes the following steps:

[0038] Step 1: Real-time video data is collected from the site using proxy software to ensure the collected video quality meets certain standards for subsequent processing. Video frames are collected at a rate of 30 frames per second (30 FPS) and stored as video sequences to ensure sufficient spatiotemporal resolution.

[0039] Step 2: Preprocess the acquired video, including noise reduction, removal of irrelevant background, and image enhancement, to improve the accuracy of subsequent target detection and behavior recognition.

[0040] Step 3: Use the collected video as input to provide the pre-trained YOLO (You Only Look Once) object detection model for the detection of behavioral objects by maintenance personnel.

[0041] Step 4: The YOLO model processes each frame of the video, automatically identifying the operational personnel's actions within the frame and locating the target's specific position (bounding box) within the image. Simultaneously, the YOLO model passes the identified target's bounding box coordinates, class label, and confidence score to the Kalman filter, providing data support for subsequent target tracking tasks.

[0042] Step 5: After receiving the target information of the operation and maintenance personnel's behavior from the YOLO model, the Kalman filter combines the target's historical motion trajectory and real-time detection information to predict the target's position at the next moment, thereby achieving continuous tracking of the target and generating the target's motion trajectory data in the video.

[0043] Step 6: The Kalman filter takes the generated motion trajectory of the operation and maintenance personnel and the corresponding video frames as input data and passes them to the pre-trained 3D convolutional neural network (3D-CNN) to provide basic data for the spatiotemporal analysis of behavioral features.

[0044] Step 7: The 3D-CNN receives and processes the motion trajectory of the operation and maintenance personnel and its associated continuous video frames generated by the Kalman filter. It extracts the motion and behavior features of the target through spatiotemporal data analysis and determines whether the operation and maintenance personnel's behavior is abnormal, thereby achieving accurate abnormal behavior identification.

[0045] Step 8: Integrate the identification results with relevant information from the current video frame (including video ID, timestamp, etc.), and use FFmpeg to embed the annotation information into the video file's metadata to ensure the video file can completely store the behavioral analysis results. Simultaneously, remind auditors to use a video playback tool to play the annotated video file for auditing.

[0046] Step 9: Use a video player to play the labeled video file, parse the embedded metadata to obtain the annotation information, and display the annotation results at the corresponding time points. Depending on the type of annotation behavior, a quick jump function can be supported to help auditors quickly locate and efficiently review abnormal operational behaviors.

[0047] This invention discloses an automated auditing method for abnormal behavior in operation and maintenance videos based on artificial intelligence. This method combines four main technical modules: the YOLO object detection model, a Kalman filter, a 3D convolutional neural network (3D-CNN), and the FFmpeg tool. Compared with traditional manual review methods, this invention, through automation and intelligence, can efficiently identify and label abnormal behaviors in operation and maintenance videos, significantly improving the efficiency and accuracy of operation and maintenance management.

[0048] The above description is merely a description of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the protection scope of the present invention.

Claims

1. An automatic auditing method for abnormal behavior in operation and maintenance videos based on artificial intelligence, characterized in that, Includes the following steps: Step 1: Collect on-site operation and maintenance video data in real time through agent software and perform corresponding preprocessing. Step 2: Use the YOLO object detection model to identify target objects in the video and determine their positions. At the same time, combine the Kalman filter to track the target in real time and generate the target's continuous motion trajectory. Step 3: Use a 3D convolutional neural network (3D-CNN) to identify anomalies in real-time behavioral data. Step 4: Integrate the recognition results with the real-time video file to generate a video file with anomaly annotations.

2. The method for automatic auditing of abnormal behavior in operation and maintenance videos based on artificial intelligence according to claim 1, characterized in that, In step 1, to avoid processing meaningless redundant data, a filtering mechanism needs to be introduced to remove videos that have no recognition value, thereby reducing the computational burden on the subsequent 3D-CNN recognition module.

3. The method for automatic auditing of abnormal behavior in operation and maintenance videos based on artificial intelligence according to claim 1, characterized in that, The specific steps for target detection and real-time tracking in step 2 include: Step 21: Use the collected video as input to provide the pre-trained YOLO (You Only Look Once) object detection model for the detection of behavioral objects by maintenance personnel. Step 22: The YOLO model processes each frame of the video, automatically identifying the operational personnel's behavior targets in the scene and locating the target's specific position (bounding box) in the image. Simultaneously, the YOLO model passes the bounding box coordinates, class label, and confidence score of the identified target to the Kalman filter, providing data support for subsequent target tracking tasks. Step 23: After receiving the target information of the operation and maintenance personnel's behavior from the YOLO model, the Kalman filter combines the target's historical motion trajectory and real-time detection information to predict the target's position at the next moment, thereby achieving continuous tracking of the target and generating the target's motion trajectory data in the video.

4. The method for automatic auditing of abnormal behavior in operation and maintenance videos based on artificial intelligence according to claim 1, characterized in that, The specific steps for abnormal behavior identification in step 3 include: Step 31: The Kalman filter takes the generated motion trajectory of the operation and maintenance personnel and the corresponding video frames as input data and passes them to the pre-trained 3D convolutional neural network (3D-CNN) to provide basic data for the spatiotemporal analysis of behavioral features. Step 32: The 3D-CNN receives and processes the motion trajectory of the operation and maintenance personnel and its associated continuous video frames generated by the Kalman filter. It extracts the motion and behavior features of the target through spatiotemporal data analysis and determines whether the operation and maintenance personnel's behavior is abnormal, thereby achieving accurate abnormal behavior identification.

5. The method for automatic auditing of abnormal behavior in operation and maintenance videos based on artificial intelligence according to claim 1, characterized in that, The specific steps for integrating the annotation results and generating a video file with anomaly annotations in step 4 include: Step 41: Integrate the identification results with relevant information from the current video frame (including video ID, timestamp, etc.), and use FFmpeg to embed the annotation information into the video file's metadata to ensure the video file can completely store the behavioral analysis results. Simultaneously, remind auditors to use a video playback tool to play the annotated video file for auditing. Step 42: Use a video playback tool to play the labeled video file, parse the embedded metadata to obtain the labeling information, and display the labeling results at the corresponding time points. Depending on the type of labeled behavior, a quick jump function can be supported to help auditors quickly locate and efficiently review abnormal operational behaviors.

6. The YOLO model according to claim 2, characterized in that, The training methods for the YOLO model include: The YOLO model is trained using classic object detection datasets (such as COCO or PASCAL VOC datasets) as the training set to achieve accurate identification and localization of target objects in operation and maintenance scenarios. Simultaneously, based on practical application needs, operation and maintenance-related video and image data are collected and labeled to construct a test set for complex scenarios, used to evaluate the model's performance in real-world applications. To achieve a balance between detection accuracy and processing speed, the YOLO model employs a multi-scale feature extraction network structure, capturing the spatial features and contextual information of targets through multi-layer convolutional networks. During training, the intersection-over-union (IoU) ratio between predefined anchor boxes and ground truth bounding boxes is used as part of the loss function to optimize the loss for target class prediction and bounding box location, thereby improving detection accuracy. Data augmentation techniques (such as random cropping, horizontal flipping, and color dithering) are widely used in training to enhance the model's adaptability to diverse scenes. The activation function used is Leaky ReLU, ensuring that negative values ​​are not completely ignored and improving gradient flow to enhance training efficiency. To avoid overfitting, regularization methods such as Dropout and weight decay are introduced to enhance the network's generalization ability and stability, enabling it to adapt to changing operational environments. These training strategies effectively improve the performance and reliability of the YOLO model in operational object detection tasks.

7. The 3D-CNN model according to claim 3, characterized in that, The training methods for the 3D-CNN model include: The 3D-CNN model is pre-trained using classic video datasets (such as UCF101 or Kinetics-400) as training sets to extract spatiotemporal features from videos and identify behavioral patterns. Simultaneously, for specific operational scenarios, abnormal behavior data from operational videos is collected and labeled to construct a dedicated test set for validating the model's performance in real-world applications. To balance recognition accuracy and computational efficiency, the 3D-CNN network employs a 5-layer convolutional structure with kernel sizes of 16, 32, 64, 128, and 256, respectively, to achieve multi-level spatiotemporal feature extraction from video frames. The 3D convolutional layers capture the temporal relationships and spatial information between video frames, while pooling layers reduce feature dimensionality, lowering computational complexity while preserving key features. Furthermore, an attention mechanism is introduced, using sigmoid and softmax functions to generate weights that highlight spatiotemporal regions more correlated with abnormal behavior, thereby significantly improving the model's ability to recognize abnormal behavior. Convolutional and fully connected layers employ the ReLU activation function to improve gradient flow and computational efficiency while avoiding the vanishing gradient problem. To enhance the model's generalization ability, Dropout is introduced during training, randomly deactivating some neuron connections with a certain probability, thereby strengthening the network's robustness and reducing the risk of overfitting. The Adam optimization algorithm is used, accelerating the convergence of model parameters through an adaptive learning rate. The loss function combines classification loss and time series consistency loss to improve the model's accuracy and consistency in action recognition tasks.

Images