A method for monitoring column inspection and entry specification operation based on personnel stay time and path tracking

Abstract

The application discloses a kind of based on personnel stay time and its path tracking's column inspection up way specification operation monitoring method, this method uses pedestrian detection algorithm in up way operation stay judgment to detect whether there is personnel in stay area in real time, if personnel enters based on pedestrian tracking algorithm tracking operation personnel, and whether its up way operation process is compliant according to pedestrian motion trajectory judgment.This application is based on pedestrian detection algorithm real-time detection whether there is personnel in stay area, based on pedestrian tracking algorithm tracking operation personnel motion trajectory, statistics pedestrian stay time, then according to pedestrian motion trajectory judgment the personnel is in or completes operation, comprehensive identification, whether the pedestrian up way operation is standard.The standard automatic identification of up way operation is realized, and efficiency is high, and strong anti-interference.

Description

Technical Field

[0001] This invention relates to the field of railway operations, specifically to a method for detecting standard track inspection procedures based on personnel dwell time and path tracking. Background Technology

[0002] Work safety is always a fundamental requirement that frontline railway workers cannot ignore. Before starting work, workers must stop and observe whether any trains are passing by to ensure safety. If any worker violates the regulations, an alarm will be triggered, and video and images of the incident will be recorded before and after the violation. On the other hand, workers who have completed their work and are returning will also pass by the surveillance cameras; these workers do not need to stop for observation. To ensure that workers on the track comply with these regulations, analyzing the positions and trajectories of workers in the camera footage is a simple and feasible method.

[0003] Using manual methods to constantly monitor track workers via cameras is labor-intensive and inefficient. Furthermore, in the face of unexpected situations, simultaneous multi-step actions by humans can easily lead to errors. However, with the rapid development of computer vision and machine learning technologies, image acquisition and object detection / tracking technologies have become capable of replacing manual methods to assess the work status of track workers. At night, track workers wear reflective clothing, which negatively impacts machine learning-based detection methods and can even lead to missed detections. Therefore, it is crucial to use pedestrian detection algorithms to determine the location of track workers after acquiring camera images, and then use pedestrian tracking algorithms to track their movements. Combining these methods to distinguish between workers entering and leaving the station and to determine whether track workers have violated regulations is of great significance. Summary of the Invention

[0004] This application proposes a monitoring method for standardized train inspection operations based on personnel dwell time and path tracking, addressing the problems existing in the background technology.

[0005] Technical solution:

[0006] This method uses a pedestrian detection algorithm to detect whether there are people in the stopping area in real time during the on-track operation stop determination. If people enter, a pedestrian tracking algorithm is used to track the workers, and the on-track operation process is judged based on the pedestrian's movement trajectory. The specific workflow is as follows:

[0007] (1) Acquire real-time footage from the camera;

[0008] (2) Mark the stopping area and the position of the railway track in the camera view;

[0009] (3) Real-time detection of whether people are present in the stopping area based on pedestrian detection algorithm:

[0010] (3-A) If a person is detected entering the camera's view, proceed to step (4);

[0011] (3-B) If no person is detected entering the camera frame, proceed to step (3);

[0012] (4) After detecting that a person has entered the camera's view, the movement trajectory of the person is tracked based on a pedestrian tracking algorithm;

[0013] (5) The time spent by personnel in the designated area;

[0014] (6) Detect whether the person has disappeared from the camera's view:

[0015] (6-A) If the person disappears from the screen, proceed to step (7);

[0016] (6-B) If the person does not disappear from the screen, proceed to step (4);

[0017] (7) Detect whether the person's stay time in the designated area exceeds the preset value n0 seconds:

[0018] (7-A) If the person stays for more than n0 seconds, proceed to step (3);

[0019] (7-B) If the person stays for no more than n0 seconds, then the person's trajectory is determined to be either entering the station for work or exiting the station after completing the work, based on the pedestrian's movement trajectory:

[0020] (7-B-1) If the personnel's trajectory indicates entry into the station for work, an alarm will be triggered;

[0021] (7-B-2) If the personnel trajectory is that they have completed their work and left the station, proceed to step (3).

[0022] Specifically, the steps for real-time detection of whether people are present in the designated stopping area based on pedestrian detection algorithms are as follows:

[0023] (3-1) Obtain the RGB image of the camera view using OpenCV;

[0024] (3-2) Perform convolution processing on the RGB image to obtain RGB image convolution data;

[0025] (3-3) Obtain the frame difference image of the camera image;

[0026] (3-4) Perform convolution processing on the frame difference image to obtain image motion convolution data;

[0027] (3-5) The RGB image convolutional data and the image motion convolutional data are fused and input into the improved YOLOV5 object detection network;

[0028] (3-6) Obtain the target boundary rectangle and calculate the intersection-union ratio between the dwelling area and the target boundary rectangle;

[0029] (3-7) Determine if the intersection-union ratio of the dwell area and the target boundary rectangle is greater than a set threshold:

[0030] (3-7-1) If the intersection-union ratio is greater than the set threshold, the detection result is that a person has been detected entering the camera screen, the target boundary rectangle is output and the timing starts;

[0031] (3-7-2) If the crossover ratio is not greater than the set threshold, the detection result is that no one enters the camera screen;

[0032] (3-8) Output the detection results.

[0033] Specifically, after detecting personnel, the movement trajectory of the workers is tracked based on a pedestrian tracking algorithm. The specific process is as follows:

[0034] (4-1) Obtain the bounding boxes of the personnel in the previous and current frames of the camera image;

[0035] (4-2) Calculate the movement speed of personnel by measuring the change of the center point of the target rectangle;

[0036] (4-3) Predict the position and speed of people in the picture using a Kalman filter;

[0037] (4-4) Calculate the cosine similarity between the image of the person in the bounding rectangle of the previous frame and the image of the person in the bounding rectangle of the current frame;

[0038] (4-5) Determine if the cosine similarity is greater than the set threshold:

[0039] (4-5-1) If the cosine similarity is greater than the set threshold, the judgment result is that the two bounding rectangles are related.

[0040] (4-5-2) If the cosine similarity is not greater than the set threshold, end the tracking;

[0041] (4-6) The cosine similarity between the position of the person in the frame predicted by the Kalman filter and the image of the person in the bounding rectangle of the previous frame and the image of the person in the bounding rectangle of the current frame is input into the position predicted by the Hungarian algorithm for target matching and tracking.

[0042] (4-7) During the target tracking process, save the personnel's location information and image information;

[0043] (4-8) After the target is lost, the movement trajectory of the personnel working on the track is obtained by tracking the trajectory of the target's center point;

[0044] (4-9) Output the movement trajectory of the workers.

[0045] Specifically, the process of determining whether a person is entering or leaving the station based on their movement trajectory is as follows:

[0046] (7-B-1) Mark the line segment l where the railway track is located in the video image and the rectangle h where the workers should stay;

[0047] (7-B-2) Obtain the movement trajectory s of the operator;

[0048] (7-B-3) Determine whether the worker is entering the station or leaving the station after completing their work based on whether the worker's movement trajectory s first intersects with the rectangle h of the area where the worker should stay, and then intersects with the rectangle l where the railway track is located:

[0049] (7-B-3-1) The movement trajectory s of the worker first intersects with the rectangle h of the area where the worker should stay, and then intersects with the rectangle l where the rail is located. It is determined that the worker is entering the station for work, and proceeds to the next step.

[0050] (7-B-3-2) The movement trajectory s of the worker first intersects with the rectangle l where the rail is located, and then intersects with the rectangle h where the worker should stay. It is determined that the worker has completed the work and left the station, and then proceeds to the next step.

[0051] (7-B-4) Output the judgment result.

[0052] Specifically, the target bounding boxes of workers are obtained based on the improved YOLOv5 network for target detection. The specific process is as follows:

[0053] (3-5-1) Extract video frames containing workers from the monitoring videos captured by the monitoring cameras at the work site, and establish a worker image dataset;

[0054] (3-5-2) Data augmentation is performed on the acquired image dataset of workers by randomly adding noise to the images, or by randomly cropping, scaling, rotating, or stitching the images to expand the dataset and improve its generalization ability.

[0055] (3-5-3) Use the LabelImg tool to label the people in the image to obtain the corresponding XML format dataset file;

[0056] (3-5-4) Convert the XML format dataset into a txt format dataset suitable for the YOLOv5 object detection network;

[0057] (3-5-5) Build a YOLOv5 object detection network using the Darknet deep learning framework;

[0058] (3-5-6) The DIoU_NMS method is used in the YOLOv5 object detection network to obtain the coordinate parameters of the bounding boxes and to filter the bounding boxes;

[0059] (3-5-7) Output the target rectangle.

[0060] Specifically, the YOLOv5 object detection network is built using the Darknet deep learning framework, and the specific method is as follows:

[0061] (3-5-5-1) The YOLOv5 object detection network backbone is built using the CSPDarknet53 network structure, including the Focus module, Conv module, C3 module, SPPF module, and CSP module. A channel attention mechanism, SENet, is added between layers 5 and 6, 7 and 8, and 9 and 10 of the backbone network (to improve the relevance of object information across different channels of the feature maps). The backbone network uses the SiLU activation function, with the specific formula as follows:

[0062]

[0063] This activation function has a smooth curve, which can better handle the problems of vanishing and exploding gradients, and can also better preserve input information to improve model accuracy and accelerate model convergence.

[0064] (3-5-5-2) The Neck part of the YOLOv5 target detection network is built using the FPN and PAN structure;

[0065] (3-5-5-3) The YOLOv5 object detection network initially uses the GIOU_LOSS loss function for bounding box regression. To accelerate network convergence, the loss function is changed to the CIOU_LOSS loss function, whose formula is as follows:

[0066]

[0067]

[0068] Where IOU is the intersection-union ratio between the predicted bounding box and the ground truth bounding box, ρ represents the Euclidean distance, and b and b gt Let w and w' represent the center points of the detection box and the target box, respectively. Let C be the slope distance of the smallest rectangle covering the area between the detection box and the target box. gt H and h represent the widths of the detection box and the target box, respectively. gt These represent the heights of the detection box and the target box, respectively.

[0069] YOLOv5 consists of three parts: Backbone, Neck, and YOLO Head. The backbone network is CSPDarknet53, and the Neck consists of FPN and PANet.

[0070] (3-5-5-4) Change the activation function of the YOLO head module to the Mish activation function. The specific formula is as follows:

[0071] Mish(x) = x * tanh(ln(1+e)) x )).

[0072] Specifically, the YOLOv5 object detection network uses the DIoU_NMS bounding box filtering method, and the specific process is as follows:

[0073] (3-5-6-1) Import the YOLOv5 object detection network pre-trained using the COCO image dataset as the pre-trained model. The COCO image dataset is an image dataset containing more than 330K images, including 1.5 million objects and 80 object categories. It is used for object detection and segmentation, key point detection of people, and filling segmentation in the field of machine vision.

[0074] (3-5-6-2) The generated dataset of images of on-site workers is fed into a pre-trained network for training, resulting in a YOLOv5 network model that can be used for on-site worker detection.

[0075] (3-5-6-3) Input the video frame into the YOLOv5 object detection network to detect the credibility of the person and the coordinate parameters of the person's bounding box;

[0076] (3-5-6-4) Output the coordinate parameters of the target rectangle.

[0077] Beneficial effects:

[0078] 1) This invention uses a pedestrian detection algorithm to detect the presence of people in the designated area in real time, and a pedestrian tracking algorithm to track the movement trajectory of workers, calculate the dwell time of pedestrians, and then determine whether the person is entering or leaving the station based on the movement trajectory. It comprehensively identifies and judges whether the pedestrian's work on the track is in accordance with regulations. This achieves automatic identification of the standardization of work on the track, and is highly efficient and resistant to interference.

[0079] 2) This invention is based on YOLOv5 as a pedestrian detection algorithm and improves upon it. It can quickly and accurately detect targets in real-time video and has high accuracy and anti-interference ability. Furthermore, by modifying the single input to a dual input, that is, modifying the input to RGB image and frame difference, the accuracy of pedestrian detection algorithm in detecting pedestrians on the road at night is further improved.

[0080] 3) For the improved YOLOv5 object detection network, this invention adds the channel attention mechanism SENet to its backbone network, which effectively improves the relevance representation of object information between different channels of the feature map; the activation function used in the backbone network is the SiLU activation function, which has a smooth function curve, can better handle the problems of gradient vanishing and gradient exploding, and can also better retain input information to improve model accuracy and accelerate model convergence speed.

[0081] 4) The activation function of the YOLO head module was changed to the Mish activation function. The advantages are: (1) The unbounded nature of this function avoids gradient saturation that leads to a sharp drop in training speed, thus speeding up the training process. Its lower bounded nature helps to achieve strong regularization; (2) This function is a non-monotonic function, which helps to keep the negative value small and stabilize the gradient flow of the network.

[0082] (3) This function is smooth, which allows more input information to penetrate the neural network, improving the accuracy and generalization ability of the network.

[0083] 5) This invention combines Kalman filtering and the Hungarian algorithm in a pedestrian tracking algorithm to achieve target tracking. Kalman filtering estimates the target's position, velocity, acceleration, and other state variables, and makes predictions and corrections based on these variables, reducing the error in the target's position; while the Hungarian algorithm matches the target tracking by minimizing the tracking error. Attached Figure Description

[0084] Figure 1 The present invention provides a method for monitoring standardized operations of train inspection based on personnel dwell time and path tracking.

[0085] Figure 2 The pedestrian detection algorithm flow for on-track operations of this invention

[0086] Figure 3 The pedestrian tracking algorithm flow of this invention

[0087] Figure 4 The present invention provides a process for determining entry or exit procedures based on pedestrian movement trajectories.

[0088] Figure 5 This invention provides a schematic diagram illustrating the process of determining whether a worker is entering or leaving the station based on their pedestrian movement trajectory. Detailed Implementation

[0089] A method for monitoring standardized operations on train tracks based on personnel dwell time and path tracking is proposed. In the determination of dwell time during track operations, a pedestrian detection algorithm is used to detect in real time whether there are personnel in the dwell area. If personnel enter, a pedestrian tracking algorithm is used to track the personnel, and the compliance of their track operation process is determined based on the pedestrian's movement trajectory. Figure 1 The specific workflow is as follows:

[0090] (1) Acquire real-time footage from the camera;

[0091] (2) Mark the stopping area and the position of the railway track in the camera view;

[0092] (3) Real-time detection of whether people are present in the stopping area based on pedestrian detection algorithm:

[0093] (3-A) If a person is detected entering the camera's field of view, proceed to the next step;

[0094] (3-B) If no person is detected entering the camera's view, proceed to step (3).

[0095] (4) After detecting that a person has entered the camera's view, the movement trajectory of the person is tracked based on a pedestrian tracking algorithm;

[0096] (5) The time spent by personnel in the designated area;

[0097] (6) Detect whether the person has disappeared from the camera's view:

[0098] (6-A) If the person disappears from the screen, proceed to the next step;

[0099] (6-B) If the person does not disappear from the screen, proceed to step (4).

[0100] (7) Detect whether the person's stay time in the designated area exceeds the preset value n0 seconds:

[0101] (7-A) If the person stays for more than n0 seconds, proceed to step (3);

[0102] (7-B) If the person stays for no more than n0 seconds, then the person's trajectory is determined to be either entering the station for work or exiting the station after completing the work, based on the pedestrian's movement trajectory:

[0103] (7-B-1) If the personnel's trajectory indicates entry into the station for work, an alarm will be triggered;

[0104] (7-B-2) If the personnel trajectory is that they have completed their work and left the station, proceed to step (3).

[0105] In a preferred embodiment, combined with Figure 2This method acquires images using OpenCV and performs pedestrian detection based on an improved YOLOv5 object detection network. The steps include:

[0106] (3-1) Obtain the RGB image of the camera view using OpenCV;

[0107] (3-2) Perform convolution processing on the RGB image to obtain RGB image convolution data;

[0108] (3-3) Obtain the frame difference image of the camera image;

[0109] (3-4) Perform convolution processing on the frame difference image to obtain image motion convolution data;

[0110] (3-5) The RGB image convolutional data and the image motion convolutional data are fused and input into the improved YOLOV5 object detection network;

[0111] (3-6) Obtain the target boundary rectangle and calculate the intersection-union ratio between the dwelling area and the target boundary rectangle;

[0112] (3-7) Determine if the intersection-union ratio of the dwell area and the target boundary rectangle is greater than a set threshold:

[0113] (3-7-1) If the intersection-union ratio is greater than the set threshold, the detection result is that a person has been detected entering the camera screen, the target boundary rectangle is output and the timing starts;

[0114] (3-7-2) If the crossover ratio is not greater than the set threshold, the detection result is that no person enters the camera screen; (3-8) Output the detection result.

[0115] In a preferred embodiment, combined with Figure 3 After detecting personnel, the movement trajectory of the workers is tracked based on a pedestrian tracking algorithm. The specific process is as follows:

[0116] (4-1) Obtain the bounding boxes of the personnel in the previous and current frames of the camera image;

[0117] (4-2) Calculate the movement speed of personnel by measuring the change of the center point of the target rectangle;

[0118] (4-3) Predict the position and speed of people in the picture using a Kalman filter;

[0119] (4-4) Calculate the cosine similarity between the image of the person in the bounding rectangle of the previous frame and the image of the person in the bounding rectangle of the current frame;

[0120] (4-5) Determine if the cosine similarity is greater than the set threshold:

[0121] (4-5-1) If the cosine similarity is greater than the set threshold, the judgment result is that the two bounding rectangles are related.

[0122] (4-5-2) If the cosine similarity is not greater than the set threshold, end the tracking.

[0123] (4-6) The cosine similarity between the position of the person in the frame predicted by the Kalman filter and the image of the person in the bounding rectangle of the previous frame and the image of the person in the bounding rectangle of the current frame is input into the position predicted by the Hungarian algorithm for target matching and tracking.

[0124] (4-7) During the target tracking process, save the personnel's location information and image information;

[0125] (4-8) After the target is lost, the movement trajectory of the personnel working on the track is obtained by tracking the trajectory of the target's center point;

[0126] (4-9) Output the movement trajectory of the workers.

[0127] Example 1 illustrates the pedestrian tracking process:

[0128] Based on the center point of the target rectangle, obtain the position of this point in the previous frame as p0 and the position of this point in the current frame as p1; calculate the personnel movement speed as v0 based on p0, p1 and the time interval Δt between the two frames.

[0129] Given the moving speed v0, the position of this point in the previous frame p0, and the position of this point in the current frame p1, the predicted velocity and acceleration values ​​can be obtained by substituting the known quantities into the Kalman filter formula. The specific formula is as follows:

[0130]

[0131] Wherein, the velocities to be predicted are v1 and u t The acceleration metric for the target. Based on the target position p1 in the current frame, the predicted velocity v1, and the target acceleration metric u. t Predict the target location.

[0132] Calculate the cosine similarity S(A,B) between the image vector A of the person in the bounding rectangle of the previous frame and the image vector B of the person in the bounding rectangle of the current frame. The specific formula is as follows:

[0133]

[0134] Among them, A i B i Let be the i-th element of vectors A and B respectively, and n be the dimension of vectors A and B.

[0135] Next, it is determined whether the cosine similarity is greater than a set threshold of 0.35.

[0136] If the cosine similarity is greater than the set threshold, the result is that the two bounding rectangles are related; if the cosine similarity is not greater than the set threshold, the tracking ends.

[0137] The cosine similarity between the position of the person in the frame predicted by the Kalman filter and the image of the person in the bounding rectangle of the previous frame and the image of the person in the bounding rectangle of the current frame is input into the position predicted by the Hungarian algorithm for target matching and tracking.

[0138] The specific process is as follows:

[0139] The cosine similarity between the image of a person within the bounding rectangle of the previous frame and the image of a person within the bounding rectangle of the current frame is input into the Hungarian algorithm to process the cost matrix and perform target matching. The detailed steps are as follows:

[0140] 1) Obtain the estimated state variables for the next frame through Kalman filtering, and obtain the cost matrix of motion information based on the estimation results and the actual detection results of the next frame;

[0141] 2) Perform the following operation on each row of the cost matrix: find the element with the smallest value in each row, and then subtract this minimum value from all elements in that row;

[0142] 3) Perform operations on each column of the cost matrix: find the element with the smallest value in each column, and then subtract this minimum value from all elements in that column;

[0143] 4) Cover all the zeros in the matrix with as few horizontal or vertical lines as possible;

[0144] 5) Find the minimum value among the elements that were not covered in the previous step, then subtract this minimum value from all these elements and add this minimum value to the elements at the intersection of the lines;

[0145] 6) Repeat steps 4) and 5) until all targets (the images of people in the bounding rectangle of the previous frame and the images of people in the bounding rectangle of the current frame) are matched (match tracking). After matching, the target position p1' of the current frame is obtained.

[0146] The target position p1' in the current frame is obtained through the above steps. The target position p1 predicted by the Kalman filter is then fused with the target position p1' obtained by the Hungarian algorithm. The specific fusion formula is as follows:

[0147] p=α*p1+(1-α)*p1′

[0148] Where α is the fusion ratio and p is the final predicted position of the target.

[0149] The process of calculating the intersection-union ratio is as follows:

[0150] Calculate the area S of the rectangular frame of target A. A The area S of the target rectangle B B Then calculate the area S of the intersection of the rectangles A and B. AB Then the intersection-union ratio (I) of target A and target B is I. The specific formula for calculating the intersection-union ratio is as follows:

[0151]

[0152] In a preferred embodiment, combined with Figure 4 Based on the pedestrian's movement trajectory, the process of determining whether the person is entering or leaving the station after completing their work is as follows:

[0153] (7-B-1) Mark the rectangle l where the railway track is located in the video frame and the rectangle h where the workers should stay;

[0154] (7-B-2) Obtain the movement trajectory s of the operator;

[0155] (7-B-3) Determine whether the worker is entering the station or leaving the station after completing the work based on whether the worker's movement trajectory s first intersects with the rectangle h of the area where the worker should stay, and then intersects with the rectangle l where the rail is located:

[0156] (7-B-3-1) The movement trajectory s of the worker first intersects with the rectangle h of the area where the worker should stay, and then intersects with the rectangle l where the rail is located. It is determined that the worker is entering the station for work, and proceeds to the next step.

[0157] (7-B-3-2) The movement trajectory s of the worker first intersects with the rectangle l where the rail is located, and then intersects with the rectangle h where the worker should stay. It is determined that the worker has completed the work and left the station, and then proceeds to the next step.

[0158] (7-B-4) Output the judgment result.

[0159] Example 2 illustrates how to determine whether a worker is entering or leaving the station based on their movement trajectory:

[0160] In one instance, the process of determining whether a worker was entering or leaving the station based on the movement trajectory of a pedestrian was as follows: Figure 5 As shown:

[0161] The rectangle l representing the location of the railway track in the video frame is compared with the rectangle h representing the area where the worker O should stay; the movement trajectory s of the worker O is obtained as follows: Figure 5 As shown;

[0162] If the movement trajectory s of the worker O first intersects the rectangle h where the worker O should stay, and then intersects the rectangle l where the rail is located, it is determined that the worker O is entering the station; if the movement trajectory s of the worker O first intersects the rectangle l where the rail is located, and then intersects the rectangle h where the worker O should stay, it is determined that the worker O is leaving the station; finally, the judgment result is output.

[0163] In a preferred embodiment, the bounding boxes of people are obtained based on the improved target detection YOLOv5 network, and the specific process is as follows:

[0164] (3-5-1) Extract video frames containing workers from the monitoring videos captured by the monitoring cameras at the work site, and establish a worker image dataset;

[0165] (3-5-2) Data augmentation is performed on the acquired image dataset of workers by randomly adding noise to the images, or by randomly cropping, scaling, rotating, or stitching the images to expand the dataset and improve its generalization ability.

[0166] (3-5-3) Use the LabelImg tool to label the people in the image to obtain the corresponding XML format dataset file;

[0167] (3-5-4) Convert the XML format dataset into a txt format dataset suitable for the YOLOv5 object detection network;

[0168] (3-5-5) Build a YOLOv5 object detection network using the Darknet deep learning framework;

[0169] (3-5-6) The DIoU_NMS method is used in the YOLOv5 object detection network to obtain the coordinate parameters of the bounding boxes and to filter the bounding boxes;

[0170] (3-5-7) Output the target rectangle.

[0171] In a preferred embodiment, the YOLOv5 object detection network is built using the Darknet deep learning framework, as detailed below:

[0172] (3-5-5-1) The YOLOv5 object detection network backbone is constructed using the CSPDarknet53 network structure. It mainly consists of the Focus module, Conv module, C3 module, SPPF module, and CSP module. Furthermore, a channel attention mechanism, SENet, is added between layers 5 and 6, 7 and 8, and 9 and 10 of the backbone network (to improve the relevance of object information across different channels of the feature map). The backbone network uses the SiLU activation function, with the specific formula as follows:

[0173]

[0174] This activation function has a smooth curve, which can better handle the problems of vanishing and exploding gradients, and can also better preserve input information to improve model accuracy and accelerate model convergence.

[0175] (3-5-5-2) The Neck part of the YOLOv5 target detection network is built using the FPN and PAN structure;

[0176] (3-5-5-3) The YOLOv5 object detection network initially uses the GIOU_LOSS loss function for bounding box regression. To accelerate network convergence, the loss function is changed to the CIOU_LOSS loss function, whose formula is as follows:

[0177]

[0178]

[0179] Where IOU is the intersection-union ratio between the predicted bounding box and the ground truth bounding box, ρ represents the Euclidean distance, and b and b gt Let w and w' represent the center points of the detection box and the target box, respectively. Let C be the slope distance of the smallest rectangle covering the area between the detection box and the target box. gt H and h represent the widths of the detection box and the target box, respectively. gt These represent the heights of the detection box and the target box, respectively.

[0180] YOLOv5 consists of three parts: Backbone, Neck, and YOLO Head. The backbone network is CSPDarknet53, and the Neck consists of FPN and PANet.

[0181] (3-5-5-4) Change the activation function of the YOLO head module to the Mish activation function. The specific formula is as follows:

[0182] Mish(x) = x * tanh(ln(1+e)) x ))

[0183] In a preferred embodiment, the YOLOv5 object detection network uses the DIoU_NMS object bounding box filtering method, the specific process of which is as follows:

[0184] (3-5-6-1) Import the YOLOv5 object detection network pre-trained using the COCO image dataset as the pre-trained model; (The COCO image dataset is an image dataset containing more than 330K images, including 1.5 million objects and 80 object categories, which is used for object detection and segmentation, person keypoint detection, and padding segmentation in the field of machine vision)

[0185] (3-5-6-2) The generated dataset of images of on-site workers is fed into a pre-trained network for training, resulting in a YOLOv5 network model that can be used for on-site worker detection.

[0186] (3-5-6-3) Input the video frame into the YOLOv5 object detection network to detect the credibility of the person and the coordinate parameters of the person's bounding box;

[0187] (3-5-6-4) Output the coordinate parameters of the target rectangle.

[0188] Example 3: Illustrate the dataset creation and network training process with an example:

[0189] The process of obtaining the dataset images is as follows: Download the pre-trained YOLOv5 model file from Github. This model is pre-trained using the COCO dataset (the COCO image dataset contains over 330K images, including 1.5 million targets and 80 target categories, and is used for object detection and segmentation, person keypoint detection, and fill segmentation in the field of machine vision). It can recognize 80 target categories, including people. Import the model file into PyTorch to build the pre-trained neural network. Use this neural network to determine whether there are human targets in each frame of the monitoring video captured by the monitoring camera at the work site. If they exist, save the current frame image to be used to build the dataset; if they do not exist, continue to judge the next frame until all frames have been judged.

[0190] The network training process is as follows: Download the YOLOv5 source code from GitHub. Create a folder named "data" within the "datasets" folder under the "yolov5-master" folder. Inside the "data" folder, create folders named "train" and "val," as well as a "data.yaml" file. The "train" folder stores training data, and the "val" folder stores validation data. Within both the "train" and "val" folders, create folders named "images" and "labels," respectively, to store image data and label data. The "data.yaml" file records the names of the object detection categories and the locations of the training and test datasets. The acquired images of the workers are stored in the "images" folder within the "train" and "val" folders at a 9:1 ratio for training and testing. The "LabelImg" tool is used to label the training and test sets, resulting in VOC format label files with the .xml extension. A script is used to convert the .xml VOC format label files to .txt YOLO format label files, and these are stored in the "labels" folder within the "train" and "val" folders according to the labels of the training and validation files, respectively.

[0191] The SENet channel attention mechanism was added between layers 5 and 6, 7 and 8, and 9 and 10 of the YOLOv5 backbone network. The improved YOLOv5 object detection network was trained for object classification using the COCO image dataset (a dataset containing more than 330K images with 1.5 million objects in 80 object categories, used for object detection and segmentation, person keypoint detection, and fill segmentation in machine vision). Each iteration trained 64 images, and after 200,000 iterations, an improved YOLOv5 network model that could be used for on-site person detection was obtained. Finally, the model was saved locally.

Claims

1. A method for monitoring the operation of a column inspection and entry specification based on the time a person stays and the tracking of their path, characterized by: The method uses a pedestrian detection algorithm to detect whether there is a person in the stopping area in real time in the up-line operation stay judgment, if a person enters, the worker is tracked based on a pedestrian tracking algorithm, and whether the up-line operation process is compliant is judged according to the pedestrian motion trajectory, the specific working process is: (1) Obtain the real-time picture of the camera; (2) Calibrate the position of the stopping area and the rail in the camera picture; (3) Detect whether there is a person in the stopping area in real time based on the pedestrian detection algorithm: (3-A) If a person is detected to enter the camera picture, go to step (4); (3-B) If no person is detected to enter the camera picture, go to step (3); (4) After detecting that a person enters the camera picture, track the motion trajectory of the person based on the pedestrian tracking algorithm; (5) Count the time the person stays in the stopping area; (6) Detect whether the person disappears from the camera picture: (6-A) If the person disappears from the picture, go to step (7); (6-B) If the person does not disappear from the picture, go to step (4); (7) Detect whether the person stays in the stopping area for more than a preset value n0 seconds: (7-A) If the person stays for more than n0 seconds, go to step (3); (7-B) If the person stays for no more than n0 seconds, judge whether the person's trajectory is entering the station for work or completing the work and leaving according to the pedestrian motion trajectory: judge whether the person is entering the station for work or completing the work and leaving according to the pedestrian motion trajectory, the specific process is as follows: Calibrate the position of the rail in the video picture Line segment l and the area rectangle h where the worker should stay; Obtain the motion trajectory s of the worker; According to whether the worker's motion trajectory s intersects the area rectangle h where the worker should stay first and then intersects the rail position rectangle l, judge whether the worker is entering the station for work or completing the work and leaving: (i) The worker's motion trajectory s intersects the area rectangle h where the worker should stay first and then intersects the rail position rectangle l, judging that the worker is entering the station for work; (ii) The worker's motion trajectory s intersects the rail position rectangle l first and then intersects the area rectangle h where the worker should stay, judging that the worker is completing the work and leaving; Output the judgment result; If the person's trajectory is entering the station for work, alarm; If the person's trajectory is completing the work and leaving, go to step (3).

2. The method of claim 1, wherein Detect whether there is a person in the stopping area in real time based on the pedestrian detection algorithm, the specific steps are: (3-1) Obtain the RGB image of the camera picture through OpenCV; (3-2) Perform convolution processing on the RGB image to obtain RGB image convolution data; (3-3) Obtain the frame difference image of the camera picture; (3-4) Perform convolution processing on the frame difference image to obtain image motion convolution data; (3-5) Fuse the RGB image convolution data and the image motion convolution data into the improved YOLOV5 target detection network; (3-6) Obtain the target bounding rectangle box, calculate the intersection ratio of the stopping area and the target bounding rectangle box; (3-7) Judge whether the intersection ratio of the stopping area and the target bounding rectangle box is greater than a set threshold: (3-7-1) If the intersection over union is greater than the set threshold, the detection result is that a person enters the camera picture, and the target bounding rectangle frame is output and the timing starts; (3-7-2) If the intersection over union is not greater than the set threshold, the detection result is that no person enters the camera picture; (3-8) Output the detection result.

3. The method of claim 1, wherein After detecting the person, the motion trajectory of the working personnel is tracked based on the pedestrian tracking algorithm, and the specific process is as follows: (4-1) Obtain the personnel target rectangle frame in the last frame and the current frame in the camera picture; (4-2) Calculate the personnel moving speed through the center point change of the target rectangle frame; (4-3) Predict the position of the personnel in the picture through the Kalman filter; (4-4) Calculate the cosine similarity between the personnel image in the last frame bounding rectangle frame and the personnel image in the current frame bounding rectangle frame; (4-5) Judge whether the cosine similarity is greater than the set threshold: (4-5-1) If the cosine similarity is greater than the set threshold, the result of the judgment is that the above two bounding rectangle frames are associated; (4-5-2) If the cosine similarity is not greater than the set threshold, end the tracking; (4-6) Fuse the position predicted by the Kalman filter and the cosine similarity between the personnel image in the last frame bounding rectangle frame and the personnel image in the current frame bounding rectangle frame to match and track the target; (4-7) During the tracking of the target, save the position information and image information of the personnel; (4-8) After the target is lost, obtain the action trajectory of the working personnel through the tracked target center point trajectory; (4-9) Output the action trajectory of the working personnel.

4. The method of claim 2, wherein Obtain the working personnel target rectangle frame based on the improved target detection YOLOV5 network, and the specific process is as follows: (3-5-1) Extract the video frame containing the working personnel from the monitoring video shot by the working site monitoring camera, and establish the working personnel image data set; (3-5-2) Data augmentation is performed on the obtained working personnel image data set, random noise is added in the image, or the image is randomly cropped, randomly scaled, randomly rotated, and randomly spliced to expand the data set and improve the generalization ability of the data set; (3-5-3) Label the personnel in the image using LabelImg tool to obtain the corresponding XML format data set file; (3-5-4) Convert the XML format data set into a txt format data set suitable for YOLOv5 target detection network; (3-5-5) Build YOLOv5 target detection network using darknet deep learning framework; (3-5-6) Use DIoU_NMS method in YOLOv5 target detection network to obtain target frame coordinate parameters and screen the target frame; (3-5-7) Output the target rectangle frame.

5. The method of claim 4, wherein Build YOLOv5 target detection network using darknet deep learning framework, and the specific method is as follows: (3-5-5-1)YOLOv5 target detection network BackBone part is built by adopting CSPDarknet53 network structure, including Focus module, Conv module, C3 module, SPPF module and CSP module, and channel attention mechanism SENet is added between the fifth layer and the sixth layer, the seventh layer and the eighth layer and the ninth layer and the tenth layer of the backbone network; the activation function used by the backbone network is SiLU activation function, which has smooth function curve and can better handle the problems of gradient disappearance and gradient explosion, and can better preserve input information to improve model accuracy and accelerate model convergence speed; (3-5-5-2)YOLOv5 target detection network Neck part is built by adopting FPN and PAN structure; (3-5-5-3)YOLOv5 target detection network target frame regression loss function adopts GIOU_LOSS loss function, in order to accelerate the network convergence speed, the loss function is changed to CIOU_LOSS loss function, and its formula is: where IOU is the intersection over union of the target detection prediction box and the real box, p represents the Euclidean distance, b and b gt respectively represent the center points of the detection box and the target box, C is the slant distance of the minimum rectangle covering the detection box and the target box, w and w gt respectively represent the widths of the detection box and the target box, h and h gt respectively represent the heights of the detection box and the target box; YOLOv5 is composed of Backbone, Neck and YOLO Head, the Backbone backbone network is CSPDarknet53, the Neck is composed of FPN and PANet; (3-5-5-4)change the activation function of YOLO head module to Mish activation function.

6. The method of claim 4, wherein YOLOv5 target detection network adopts DIoU_NMS target frame screening method, the specific process is as follows: (3-5-6-1)import the pre-trained YOLOv5 target detection network using COCO image dataset as pre-training model, COCO image dataset is an image dataset containing more than 330K images, containing 1.5 million targets, 80 target categories, which is used for object detection and segmentation, human key point detection, filling segmentation in machine vision field; (3-5-6-2)send the prepared on-site operator image dataset into the pre-trained network for training to get YOLOv5 network model which can be used for on-site personnel detection; (3-5-6-3)input video frame into YOLOv5 target detection network to detect the confidence of personnel and the coordinate parameters of personnel target frame; (3-5-6-4)output the coordinate parameters of target rectangular frame.

Images