Factory nonstandard operation identification and early warning technology based on deep learning and target tracking

Abstract

The invention belongs to the field of intelligent video monitoring and industrial safety management, and particularly relates to a factory nonstandard operation identification and early warning technology based on deep learning and target tracking. The method mainly comprises four parts of data acquisition and preprocessing, data annotation and classification, model construction and training and target tracking and early warning. According to the technology, a large amount of monitoring video data is collected and processed from a key monitoring area of an enterprise, data labeling and classification are performed by using professionals, a behavior detection model based on a deep learning architecture (such as an improved YOLO model) is constructed, training is performed in combination with a data enhancement technology and a multi-loss function, and after a non-standard operation behavior is detected, the non-standard operation behavior is detected. A DeepSORT improved target tracking algorithm is used for continuously tracking involved personnel, and meanwhile an intelligent early warning mechanism is constructed to give an alarm in time. According to the technology, through a large number of actual scene tests, compared with a traditional manual inspection and simple video analysis method, the recognition accuracy and the early warning timeliness of non-standard operation of a factory are improved, and the defects of the traditional method in the aspects of accuracy, real-time performance and robustness are effectively overcome; and a more efficient and reliable solution is provided for safe production management of enterprises.

Description

Technical Field

[0001] The present invention relates to the fields of intelligent video surveillance technology and industrial safety management, and in particular to the application of deep learning-based target detection and tracking technology in monitoring non-standard operations of factory workers. Background Art

[0002] In modern industrial production, many industries have high-intensity and high-risk working environments. Due to factors such as worker fatigue and distraction, non-standard operations often occur, which not only reduces production efficiency, but also easily leads to safety accidents, becoming a key cause of safety accidents and causing serious losses to corporate production. The functions of traditional monitoring systems are limited to simple collection and display, and cannot intelligently analyze monitoring content. Manual monitoring is prone to fatigue and it is difficult to detect non-standard operations and issue warnings in a timely manner.

[0003] With the rapid development of artificial intelligence technology, achievements in pattern recognition, machine vision and other fields have been widely used in many industries, promoting the progress of intelligent monitoring technology in industrial sites. The maturity of video surveillance technology and the popularity of network cameras have made intelligent video surveillance possible, which uses computer vision technology to detect, track, identify and understand targets, and can actively record alarms when abnormal events occur, assisting monitoring work.

[0004] In the field of target detection, deep learning has become the mainstream. Traditional methods manually create features, which have poor robustness, are time-consuming and labor-intensive, and have insufficient detection efficiency and accuracy. Deep learning target detection technologies include two-stage and single-stage algorithms. Two-stage algorithms such as the R-CNN proposed by Girshick et al. and its subsequent improved algorithms have improved performance, but there is a problem of slow speed due to non-shared calculations; single-stage algorithms such as YOLO and SSD series have outstanding performance in real-time detection. The YOLO algorithm has undergone many improvements and continuously improved detection accuracy and speed. In the multi-target tracking algorithm, the detection-based method uses the detector to obtain moving targets. For example, the SORT algorithm uses Faster R-CNN detection, combined with Kalman filtering and Hungarian algorithm to achieve target tracking and identity recognition, providing an effective idea for multi-target tracking.

[0005] At present, the research on monitoring technology for non-standard operations of factory workers is constantly advancing at home and abroad. Although China started later, it has received increasing attention. Enterprises and research institutions are actively developing related technologies, and some cities have begun to build real-time monitoring systems. Some developed countries abroad have relatively mature technologies, and have developed and widely used advanced products such as smart helmets and sensors. However, existing technologies generally have problems such as high cost, difficulty in use, poor accuracy and stability, and data protection. Sensor detection methods have low applicability and are susceptible to electromagnetic interference. Traditional image detection methods have poor robustness and rely on manual features. Deep learning detection methods need to balance detection speed and accuracy. Therefore, it is crucial and challenging to develop an efficient and accurate detection technology for non-standard operations in factories. Summary of the invention

[0006] In order to effectively deal with the problem that non-standard operations of workers in the factory area are difficult to be discovered and warned in time, an efficient deep learning system with high recognition accuracy is designed and implemented. This technology mainly covers two key modules. One is used to detect the behavior of operators in the monitoring video images to determine whether there are non-standard operations. The other is to use advanced target tracking algorithms and intelligent judgment mechanisms to ensure continuous monitoring and accurate warning of non-standard operation behaviors when complex environmental factors interfere or target tracking deviates. This technology can provide reliable monitoring and warning results, helping enterprise safety managers to timely discover and deal with workers' non-standard operation behaviors, effectively prevent safety accidents, and ensure the safe and orderly conduct of production activities.

[0007] Step 1: Data collection and preprocessing

[0008] A large amount of surveillance video data is collected from the company's key monitoring areas using professional equipment, and the video is decomposed into image frame sequences. Each sequence is screened based on factors such as different time periods, different production links, different weather conditions, and different population densities to ensure that all possible work scenarios are covered. The image size is then unified to a size suitable for model processing and standardized, laying a solid foundation for subsequent model training.

[0009] Step 2: Data labeling and classification

[0010] Corresponding to the collected image data, professional personnel are organized to perform manual annotation based on safety regulations and actual production experience, clearly marking the key features and areas of normal operating behaviors and various non-standard operating behaviors. At the same time, the labeled data is classified and organized according to dimensions such as operation type and degree of danger to build a rich and representative training sample library.

[0011] Step 3: Model building and training

[0012] First, data enhancement techniques such as random cropping, rotation, flipping, and adding simulated noise are used to expand the labeled data to increase the diversity of samples and the generalization ability of the model. Then, the enhanced image data is input into the behavior detection model built based on the deep learning architecture (such as the improved YOLO model). The model can efficiently extract the behavior characteristics of operators by integrating lightweight network structure, attention mechanism and multi-scale feature extraction module, and use the designed multi-loss function to constrain the training process, including behavior classification loss based on category cross entropy, positioning loss based on bounding box regression, and contrast loss based on feature similarity, etc., to comprehensively calculate the final training loss. By continuously iteratively optimizing the model parameters, the model can accurately identify and distinguish normal and non-standard operation behaviors, and can accurately locate the location and range of non-standard operations. Due to the use of efficient network architecture and optimization strategy, the model has high operating efficiency while ensuring accuracy, which can meet the needs of real-time monitoring.

[0013] Step 4: Target tracking and early warning

[0014] After the model detects suspected non-standard operating behavior, it uses advanced target tracking algorithms (such as the improved version of DeepSORT) to continuously track the personnel involved, and can maintain stable tracking even in complex situations such as personnel movement, partial occlusion, or light changes. At the same time, an intelligent early warning mechanism is established to make a comprehensive judgment based on historical behavior data and current detection results, and set reasonable early warning thresholds and rules to avoid false alarms and missed reports. For example, when the same type of non-standard operating behavior of the same person is detected in multiple consecutive frames of images and his or her behavior characteristics meet the preset danger standards, the system immediately triggers multiple forms of alarms, including popping up eye-catching text prompts and warning icons in the monitoring center, sending text message notifications to relevant managers, and activating voice alarm equipment at the work site to clearly inform the type, location and information of the personnel involved in the non-standard operation, ensuring that it can be paid attention to and take corresponding measures in time.

[0015] Compared with traditional manual inspections and simple video surveillance methods, the technology of the present invention has significant advantages: First, the behavior detection model adopts an advanced lightweight design concept and integrates efficient feature extraction and classification technology, which greatly improves the detection speed while ensuring recognition accuracy, and can meet the needs of enterprises for real-time processing of large-scale video data, and realize the immediate discovery of non-standard operating behaviors. Second, by using powerful target tracking and intelligent early warning mechanisms, it can effectively respond to various interference factors in complex and changeable industrial environments, ensure continuous monitoring and reliable early warning of non-standard operating behaviors, significantly enhance the robustness and practicality of the system, and provide strong guarantees for enterprise safety production. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments:

[0017] Figure 1 This is the structure of the improved CBAM attention mechanism. Figure 2 Improved matching process of DeepSORT algorithm. DETAILED DESCRIPTION

[0018] The following will be combined with the relevant drawings in the embodiments of the present invention, and a detailed and comprehensive description will be given in the technical scheme of the embodiments of the present invention. Obviously, the embodiments shown here are only part of many feasible embodiments and do not cover all situations. Without involving creative work, other related embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention are also within the scope of protection of the present invention.

[0019] Example:

[0020] Referring to the accompanying drawings, the present invention provides a plant non-standard operation identification and early warning technology based on deep learning and target tracking, comprising the following steps:

[0021] (1) Data collection and organization: First, a large number of surveillance videos were obtained from the key operating areas of a company in Nanjing using high-definition surveillance cameras, and the videos were decomposed into frame-by-frame image sequences. Professional personnel were organized to manually annotate the images, clearly marking the characteristic areas and related attribute information of normal operating behaviors and various types of non-standard operating behaviors, and then the training set, validation set, and test set were constructed in a ratio of 7:2:1.

[0022] (2) Data expansion and enhancement: Given that some non-standard work scenes are relatively rare in the original data, images containing non-standard work are processed to increase the number of such samples. At the same time, random occlusion processing is performed to simulate the situation where objects in actual scenes occlude workers or work tools to enhance the model's adaptability to complex environments; Gaussian noise is added to simulate different degrees of light interference, equipment electromagnetic interference and other environmental factors, so that the model can maintain stable performance under complex and changeable actual working conditions. In addition, key elements such as workers and tools in the image are segmented, and sub-parts are divided into independent samples according to a certain proportion to further enrich the data set and improve the generalization ability of the model.

[0023] Through the above-mentioned data enhancement operation, the number of samples can be effectively expanded. On the one hand, it makes the trained model more accurate, and on the other hand, it enhances the robustness of the model in complex environments, so that it can better cope with various situations in the actual factory.

[0024] (3) Model construction and training: Referring to the relevant model architecture diagram, the collected images are input into the network architecture built based on the improved YOLO model for training. Before training, the input images are resized to a suitable size and sent to a model that integrates a lightweight network structure such as MobileOne or MobileNetV3 for feature extraction. This model innovatively combines dilated convolution and multi-scale convolution, and introduces the CBAM attention mechanism and a specially designed small object detection layer.

[0025] At the same time, a set of multi-loss functions is designed to constrain the network training process. Let the classification loss be L cls , the cross entropy loss function is used to calculate the difference between the predicted category and the true category; the positioning loss is L loc , the smooth L1 loss function is used to measure the deviation between the predicted position and the actual position; the feature similarity loss for non-standard operation behavior is set to L sim , calculated by comparing the similarity between the predicted features and the standard feature template. The final training loss L total The calculation formula is as follows:

[0026] L total =αL cls +βL loc +γL sim

[0027] Among them, α, β, and γ are weight coefficients set according to experience, which are used to balance the importance of different losses in the training process.

[0028] During the training process, the model parameters are continuously adjusted, and the network parameters are updated according to the calculated loss values ​​based on the reverse gradient propagation algorithm until the model converges, and the final trained model is saved for future use.

[0029] (4) Target tracking and early warning: After the model detects suspected non-standard operating behavior, it uses the DeepSORT improved target tracking algorithm to continuously track the person involved. Even in complex situations such as frequent movement of people, partial obstruction of the body by equipment, or changes in light brightness, the target can still be stably tracked.

[0030] At the same time, an intelligent early warning mechanism is constructed. A warning credibility threshold is set. When the model detects the same type of non-standard operation behavior of the same person in consecutive frames, and the warning credibility C calculated by comprehensively evaluating their behavior characteristics, historical records, and current environmental factors is greater than T, the system immediately triggers multiple forms of alarms. For example, eye-catching text prompts and warning icons pop up in the monitoring center, SMS notifications containing detailed operation locations and behavior types are sent to relevant managers, and voice alarm equipment is activated at the operation site to clearly inform the specific situation of non-standard operations, ensuring that it can attract attention in time and prompt relevant personnel to take corresponding measures.

[0031] The Kalman filter (if used in the target tracking process) time update equation is as follows:

[0032]

[0033] The Kalman filter state update equation is as follows:

[0034]

[0035] The present invention has undergone a large number of tests in various actual scenarios such as different workshops, different time periods, and different production tasks of the enterprise. Compared with traditional manual inspections and simple video analysis methods, the average accuracy of identifying non-standard operations has been significantly improved, effectively making up for the shortcomings of traditional methods in accuracy, real-time and robustness. It does not require complex additional training parameter settings, and can efficiently provide strong guarantees for enterprise safety production.

[0036] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the technical solutions described in the above embodiments may be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A plant non-standard operation identification and early warning technology based on deep learning and target tracking, characterized in that: The following steps are involved: Step 1: Data collection and preprocessing: Use the monitoring equipment on site to collect video data containing workers' work scenes from key monitoring areas, screen and clean these video data, remove blurry, low-light and other poor quality parts, and then decompose them into frame-by-frame image sequences. Each sequence extracts the corresponding number of frames according to different work process stages (such as startup stage, processing stage, closing stage, etc.), and a total of 30 frames are selected. The data sets are grouped according to factors such as different workshops, different work shifts, and different equipment operating conditions, and uniformly scaled to 640*480 size to adapt to subsequent network training. Assuming the original image size is (w0, h0) and the target size is (w1, h1), the scaling process can be calculated by the following formula: Where (x′, y′) is the coordinate in the original image, and (x′, y′) is the coordinate in the scaled image. Step 2: Data labeling: Organize professionals to manually label the collected image data, and label the key feature points and areas of normal and non-standard operating behaviors of workers. For non-standard operating behaviors, such as failure to operate tools according to regulations, crossing safety lines illegally, and not wearing protective equipment, the location and range of their specific behaviors are detailed. Assuming that the labeled area is a rectangle, the upper left corner coordinate (x min ,y min ) and the lower right corner coordinate (x max ,y max ) is expressed as follows: its area can be expressed by the formula (x max -y min )(x max -y min )calculate. Step 3: Behavior detection and feature extraction. The labeled data is sent to the deep learning network model for training, and finally 20 feature maps are obtained, of which 8 are data-enhanced feature maps. By analyzing the feature information related to the operation behavior in different feature maps, the key feature representation of non-standard operations is extracted, and then it is determined whether there is non-standard operation behavior. Step 4: Target tracking and abnormal warning. When complex situations such as personnel movement, ambient light changes, and equipment occlusion occur in the operation scene, the target tracking algorithm is used to continuously track the identified non-standard operators, and their behavior is continuously evaluated through intelligent algorithms. When it is judged that non-standard operation behavior persists or has an escalating trend, the corresponding warning strategy is used to issue an alarm.

2. According to the non-standard operation identification and early warning technology for factory areas based on deep learning and target tracking according to claim 1, the data set preparation method described in step 1 includes the following steps: Step 1: First, use the high-definition surveillance camera on the enterprise site to obtain the operation video and proceed to the next step; Step 2: Decompose each video into a 30-frame image sequence based on the time interval (such as one frame per second), and scale the image to a uniform size of 640*480 by bilinear interpolation and other methods. The bilinear interpolation formula is as follows: f(x,y)=(1-x)(1-y)f(0,0)+x(1-y)f(1,0)+(1-x)yf(0,1)+xyf(1,1) where x and y are the positions within the unit square, and f(0,0), f(1,0), f(0,1), and f(1,1) are the values ​​of the four nearest neighbor pixels.

3. According to the non-standard operation identification and early warning technology for factory areas based on deep learning and target tracking according to claim 1, the data labeling method described in step 2 includes the following steps: Step 1: Label the key behavioral elements related to the worker's operation in the image (such as operating tools, human posture, safety protection equipment, etc.), clearly define the area of ​​non-standard operation behavior, mark its boundary coordinates, and then proceed to the next step; Step 2: Use the image dataset to construct training set, validation set, and test set in a ratio of 6:2:

2. Step 3: During the annotation process, for the overlapping parts between different annotated areas, the degree of overlap can be measured by the intersection over union (IoU), the formula is: The Area of ​​Overlap is the area of ​​the overlapping parts of the two labeled areas, and the Area of ​​Union is the area of ​​the union of the two labeled areas.

4. According to the non-standard operation identification and early warning technology for factory areas based on deep learning and target tracking according to claim 1, the behavior detection and feature extraction described in step 3 includes the following steps: Step 1: The collected images are sent to a deep learning-based network model for training. Finally, 20 feature maps are obtained, of which 8 are feature maps after data enhancement. The key feature information is extracted by analyzing the distribution and correlation of the operation behavior features in different feature maps. During the training process, the input image is first normalized in size and then fed into our lightweight network structure (such as MobileNetV2) for feature extraction. An optimized convolutional neural network architecture is adopted to combine depthwise separable convolution and grouped convolution. At the same time, the SE attention module is introduced to improve the pertinence and effectiveness of feature extraction, achieving lightweight while ensuring performance. Step 2: Design multiple loss functions to constrain the network. Use category cross entropy loss as the behavior classification loss, and its formula is: where y i is the true category label (0 or 1), p i is the probability predicted for this category, which is used to distinguish normal and non-standard operation behaviors; smooth L2 loss is used as the position offset loss, and its formula is: Where (x i ,y i ) is the real location, is the predicted position, which is used to train the position information of operators and key behavioral elements. The association loss considers the coherence between actions by calculating the similarity between the predicted action trajectory and the standard trajectory. We first calculate the Manhattan distance between each predicted action trajectory endpoint and the standard trajectory endpoint using the formula: d=|x1-x2|+|y1-y2| These distances are then used to match the predicted action trajectory with the true trajectory, provided that their distance is below a threshold δ. Finally, the Huber loss is used for the association loss, which aims to minimize the geometric distance of the matching trajectory relative to the starting point, the end point, and the middle point, as follows: Finally, we get the final loss of training: L total =αL cls +βL loc +γL hub Among them, α, β, and γ are weight coefficients, which can be adjusted according to actual conditions. The sum of losses is used for reverse gradient propagation to update the parameters of the feature extraction network until the loss reaches the minimum and the model converges. The final model is saved for subsequent use.

5. According to the deep learning and target tracking-based non-standard operation identification and early warning technology for factory areas according to claim 1, the target tracking and abnormal early warning described in step 4 includes the following steps: Step 1: The target tracking algorithm is divided into two main steps, trajectory prediction and state update. Trajectory prediction is to estimate the current state based on the position and speed information of the personnel at the previous moment, while the state update is to combine the estimated state and the observed state at the current moment to determine the optimal position and behavior state of the personnel. Therefore, it can be used to stably track the operating personnel and provide timely abnormal warnings. Step 2: During the target tracking process, the target tracking algorithm used will continuously estimate and update the movement trajectory of the operator based on the position and speed information of the operator at different times to ensure accurate tracking of the operator. In terms of early warning, the system will judge the detected non-standard operation behavior based on the set rules and standards. When the characteristics of non-standard operation behavior meet the early warning conditions, the system will trigger a variety of alarm methods, including displaying eye-catching warning information on the screen of the monitoring center, sending notification messages to relevant managers, and activating the sound and light alarm on site. At the same time, different types of non-standard operation behaviors will be classified and recorded for subsequent statistical analysis and management improvement.

Images