A retrieval method for intelligent violation identification of railway locomotive video

Abstract

The application discloses a search method for railway locomotive video intelligent violation identification, and comprises the following steps: image acquisition; target detection on the acquired picture according to a deep learning network Faster-Rcnn to acquire whether the current picture contains a locomotive attendant and a violation behavior target; feature extraction on the acquired image to acquire the current locomotive running speed and the duration of the target behavior; conditional screening on the detected violation behavior target according to the acquired locomotive running speed and the duration of the target behavior to eliminate the violation behaviors not meeting the conditions. The application has low cost: the user only needs to run the analysis program, without manual video monitoring, thereby reducing the labor cost; the application has high speed: based on single-frame image analysis, without analyzing all image frames in a video segment, thereby reducing the number of analyzed pictures and improving the analysis speed; and the application has high accuracy: with duration and space as constraints, interference results are eliminated, thereby improving the accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of rail transit technology, and particularly relates to a retrieval method for intelligent violation identification of railway locomotive video. BACKGROUND

[0002] In daily analysis of a railway locomotive system, a user needs to watch a large number of locomotive videos to find out violation problems of a driver in a whole operation process of operating a locomotive. A locomotive depot normally has 300 locomotives, and a train runs for 8 hours on average. A locomotive depot generates 2400 hours of locomotive video in a day. An analysis personnel needs to watch all the 2400 hours of video to complete the whole analysis work. There are the following problems:

[0003] 1) High labor cost: the user needs to arrange multiple analysis personnel to complete daily analysis work, and the labor cost is high.

[0004] 2) High work intensity: the analysis personnel needs to watch the video for a long time to find out violation problems of a driver in a whole operation process of operating a locomotive.

[0005] A current mainstream intelligent violation identification method of a locomotive video is based on deep learning technology. Video clips are input into a network framework to extract time sequence features and image features, and identify and retrieve violation behaviors. The analysis method based on video clips is time-consuming, and cannot meet the daily video analysis speed requirement.

[0006] The existing intelligent violation identification retrieval method of a video can be divided into two categories. One category takes a whole picture sequence of a video as input, and the other category takes a single picture as input. Each category has the following shortcomings:

[0007] 1) Taking a whole picture sequence of a video as input is time-consuming: the locomotive video has 8-16 frames per second, and each frame of picture is analyzed, which needs to consume a large amount of time.

[0008] 2) Taking a single picture as input is inaccurate in identification result: behaviors and actions in the locomotive video have time sequence and spatiality, and a single picture feature information cannot accurately identify. SUMMARY

[0009] To solve the above problems, a retrieval method for intelligent violation identification of a railway locomotive video is provided.

[0010] The purpose of the present application is achieved in the following manner:

[0011] A retrieval method for intelligent violation identification of a railway locomotive video, the method comprising the following steps:

[0012] S1: image acquisition: reading a locomotive monitoring video by using ffmpeg to capture one picture per second;

[0013] S2: target detection: target detection is performed on the picture obtained in S1 according to a deep learning network Faster-Rcnn to obtain whether a locomotive attendant and a violation behavior target are contained in the current picture;

[0014] S3: feature extraction: the image in S1 is processed to obtain the current locomotive running speed; and the duration of the target behavior is obtained according to the number of video frames extracted per second as a time calculation basis;

[0015] S4: comprehensive analysis: the violation behavior target detected in S2 is conditionally screened according to the locomotive running speed and the duration of the target behavior obtained in S3, and the violation behavior not meeting the condition is eliminated;

[0016] S5: operation analysis: the violation picture containing the violation behavior target and the occurrence time after screening are compared with the LKJ train data, the action that should be made but not made by the attendant is analyzed by comparing the gesture and action made by the locomotive attendant during the locomotive running time recorded by the LKJ train data and the position of the locomotive during the locomotive speed recorded by the LKJ train data and corresponding time, the violation action meeting the condition is further screened out, and the image corresponding to the violation behavior of the attendant is saved in real time;

[0017] S6: statistics and output: the violation behavior is statistically analyzed in terms of the occurrence time period and the occurrence frequency.

[0018] The deep learning network Faster-Rcnn includes model training; the model training includes the following steps:

[0019] An image is extracted from a locomotive monitoring video to make a data set;

[0020] The detection targets in the data set are labeled and grouped according to the violation behavior action categories;

[0021] The pictures with classification labels are scaled to M*N and sent to the deep learning network Faster-Rcnn for model training to obtain a final target detection model.

[0022] The model training of the deep learning network Faster-Rcnn includes: 1) first, the images in the data set are extracted by backboon, and ResNet_50 network is used in this layer; after 4 pooling layers, the feature maps are C2, C3, C4 and C5 layers, and the shapes of the feature maps are respectively: [1, M / 2, N / 2, 256], [1, M / 8, N / 8, 512], [1, M / 16, N / 16, 1024] and [1, M / 32, N / 32, 2048];

[0023] 2) 4 feature maps obtained through backbone, after FPN layer, M5, M4, M3, M2 feature maps are obtained; M2, M3, M4, M5 feature maps are obtained through a 3*3 convolution kernel to obtain P2, P3, P4, P5 feature maps; P6 feature map is obtained by downsampling P5 feature map; the shape of the 5 feature maps is: [1, M / 2, N / 2, 256], [1, M / 4, N / 4, 512], [1, M / 8, N / 8, 1024], [1, M / 16, N / 16, 2048], [1, M / 32, N / 32, 256];

[0024] 3) RPN network processes P2, P3, P4, P5, P6 feature maps through a series of fixed reference frames anchors, judges whether the anchor contains the target through softmax classification, calculates the offset of the target frame to the anchor, and obtains the accurate candidate region; finally, the Proposal layer processes the anchors containing the target and the corresponding bbox regression offset to obtain the candidate region, and eliminates the candidate regions that are too small and exceed the boundary;

[0025] 4) ROIPooling layer collects the candidate regions generated by RPN network, maps them to feature maps and fixes the dimensions for classification and regression; after ROIPooling layer, the feature map with uniform size is obtained; the feature map is classified through subsequent full connection layer and softmax to calculate which behavior category the candidate region belongs to, and the probability value (cls_prob) is output; at the same time, the position offset of each candidate region is obtained by regression method, and the more accurate target detection frame (bbox_preb) is obtained by regression.

[0026] S3 specifically includes: based on the acquired locomotive monitoring image, firstly, the image is converted to grayscale using cv2.cvtcolor(); then, the image is binarized using cv2.threshold(); then, the image is eroded and dilated using cv2.erode() and cv2.dilate() to eliminate interference from some thin lines in the image; then, the contour values ​​on the image are found using cv2.findcontours(); the area of ​​the image enclosed by the contour values ​​is calculated using cv2.contourArea() based on the contour values; the area ratio of the detected contour values ​​is used to determine whether there are obstructions in front of the camera that hinder video analysis, and if so, the determination of the violation item is obtained simultaneously; the approximate location of the locomotive speed in the monitoring image is extracted, and the current locomotive speed is obtained by using cv2.matahTemplate() to match the digital template created based on the locomotive video display, as a judgment condition for the speed limit of the violation item.

[0027] S4 specifically includes the following steps: According to the railway bureau's requirements, some violations have speed limits. The locomotive speed detected in S3 is used to filter the violations detected in S2. When the locomotive is running, i.e., when the locomotive speed is greater than 0, talking with one's head turned to the side, making a phone call, or playing on a mobile phone are violations. Smoking is required when the speed is less than a specified value, and smoking is not allowed when passing through designated sections with speed limits. Lying down to sleep is required when the train is stopped, and sleeping is not allowed when the locomotive is entering a train station or waiting for a train in a designated section. Otherwise, they are all violations.

[0028] Based on the duration of the identified action and the locomotive's operating speed, the locomotive's forward distance is calculated. When the locomotive passes a fixed distance, the co-driver is required to stand up to determine whether the locomotive crew has violated regulations by not standing up.

[0029] By comparing the duration of the identified action with the required duration of the violation action item, violations that do not meet the criteria are further eliminated.

[0030] The violation targets in S2 are determined based on the category of violation actions, including actions that crew members should perform and violations. The actions that crew members should perform include leaning forward, gesturing with their hands, inputting confirmation, clenching their fists, standing in front of their seats, and using the locomotive's mobile phone. The violations include smoking, talking with their head tilted to the side, making phone calls, playing on their mobile phones, riding alone, and lying down to sleep.

[0031] The beneficial effects of this invention are: 1) Low cost: Users only need to run this analysis program, without the need for manual monitoring of the video, thus reducing labor costs;

[0032] 2) Fast speed: based on single-frame image analysis, no need to analyze all image frames in the video segment, reduce the amount of analysis pictures, improve the analysis speed;

[0033] 3) High accuracy: fusion image processing feature point statistics and edge information, with duration and space as constraints, eliminate interference results, improve accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is the method flowchart of the present application.

[0035] Figure 2 is the processing flowchart of the deep learning network Faster-Rcnn of the present application. DETAILED DESCRIPTION

[0036] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0037] It should be pointed out that the following detailed description is exemplary, and is intended to provide further description of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same technical meaning as understood by ordinary skilled persons in the art to which the present application belongs.

[0038] A retrieval method for intelligent violation identification of railway locomotive video, the method comprising the following steps:

[0039] S1: image acquisition: reading the locomotive monitoring video through ffmpeg, grabbing one image per second;

[0040] S2: target detection: target detection is performed on the image obtained in S1 according to the deep learning network Faster-Rcnn, to obtain whether the current image contains a locomotive attendant and a violation behavior target;

[0041] S3: feature extraction: processing the image in S1 to obtain the current locomotive running speed; according to the number of video frames extracted per second as the time calculation basis, the duration of the target behavior is obtained;

[0042] S4: comprehensive analysis: condition screening is performed on the violation behavior target detected in S2 according to the locomotive running speed and the duration of the target behavior obtained in S3, to eliminate the violation behaviors that do not meet the conditions;

[0043] S5: Operation Analysis: Compare the selected violation images and their occurrence times with LKJ train data. Based on the locomotive running time recorded in the LKJ train data, analyze the actions that the crew should have performed but did not, such as leaning forward, gesturing with their hands, confirming, clenching their fists, and standing. Based on the locomotive speed and the locomotive's position at the corresponding time recorded in the LKJ train data, further filter out violations that meet the criteria, such as smoking, chatting with their head turned to the side, making phone calls, playing on their mobile phones, and lying down to sleep. Save the images corresponding to the crew's violations in real time.

[0044] S6: Statistics and Output: Perform probabilistic statistics on the time periods and frequency of violations. Based on the locomotive running time recorded by LKJ train data, calculate the frequency of violations by train crew members; based on the analysis data of all locomotive videos from recent times, analyze the frequency of violations by crew members and the most frequent violations in different time periods (early morning, morning, afternoon, and night); calculate the frequency of each violation; and identify crew members with a high number of violations.

[0045] The deep learning network Faster-R-CNN includes model training; the model training includes the following steps:

[0046] Datasets were created by extracting images from locomotive monitoring videos and labeling them with targets.

[0047] The detected targets in the dataset are grouped by label according to the category of the violation behavior;

[0048] The images with classification labels are scaled to size M*N and fed into the Faster-R-cnn deep learning network for model training to obtain the final object detection model.

[0049] The training of the deep learning network Faster-R-CNN includes: 1) First, the images in the dataset are used for feature extraction through backbone, which uses the ResNet_50 network; after 4 pooling layers, the resulting feature maps are denoted as layers C2, C3, C4, and C5, with shapes of [1, M / 2, N / 2, 256], [1, M / 8, N / 8, 512], [1, M / 16, N / 16, 1024], and [1, M / 32, N / 32, 2048], respectively;

[0050] 2) 4 feature maps obtained through backbone, after FPN layer (up-sampling from top to bottom and then added to the same layer), M5, M4, M3, M2 feature maps are obtained; M2, M3, M4, M5 feature maps are obtained through a 3*3 convolution kernel, P2, P3, P4, P5 feature maps are obtained; P6 feature map is obtained by down-sampling P5 feature map; the shape of the 5 feature maps is: [1, M / 2, N / 2, 256], [1, M / 4, N / 4, 512], [1, M / 8, N / 8, 1024], [1, M / 16, N / 16, 2048], [1, M / 32, N / 32, 256];

[0051] 3) The RPN network processes P2, P3, P4, P5, P6 feature maps through a series of fixed reference frames anchors, judges whether the anchor contains the target through softmax classification, calculates the offset of the target frame to the anchor, and obtains the accurate candidate region; finally, the Proposal layer processes the anchors containing the target and the corresponding bbox regression offset to obtain the candidate region, and eliminates the candidate regions that are too small and exceed the boundary;

[0052] 4) The ROIPooling layer maps the candidate regions generated by the RPN network to the feature map and fixes the dimension for classification and regression. After the ROIPooling layer, a feature map of uniform size is obtained; the feature map is classified through the subsequent full connection layer and softmax to calculate which behavior category the candidate region belongs to, and the probability value (cls_prob) is output; at the same time, the position offset of each candidate region is obtained by regression, and a more accurate target detection frame (bbox_preb) is obtained by regression.

[0053] Further, in the training process, a weighted image strategy is adopted to solve the problem of uneven sample quantity; a label smoothing method is adopted to reduce model training overfitting; Mosaic data enhancement is adopted to splice four pictures into one picture, and random scaling, random cropping and random arrangement are used for splicing to enrich the data set; cutout data enhancement is adopted, a part of the image is randomly reduced in proportion during the training process to improve the robustness of the model; enhanced images are adopted to randomly adjust the color chart, saturation and brightness.

[0054] The S3 specifically includes: according to the obtained locomotive monitoring image, first, in order to improve the picture quality, make the display result of the image clearer, use cv2.cvtcolor() to convert the image into a gray image; then in order to highlight the outline of the target, use cv2.threshold() to binarize the image; then use cv2.erode() and cv2.dilate() to erode and dilate the image, eliminate the interference of part of the fine lines in the image; then use cv2.findcontours() to find the contour value on the image; according to the contour value, use cv2.contourArea() to calculate the area of the image surrounded by the contour value. According to the area ratio of the detected contour value (obstructions generally block the camera, and the proportion of the whole image area is larger), it is judged whether there is an obstruction in front of the camera (the obstruction area is darker than the unobstructed area) to hinder video analysis, and the judgment of the camera obstruction violation item point is obtained. Extract the approximate position of the locomotive speed in the monitoring image (the locomotive running speed is displayed in the upper right position of the video), and use the digital template (use the existing locomotive image to extract the speed display number in the image to make a template) produced according to the locomotive video display, use cv2.matahTemplate() for template matching, get the current locomotive running speed, as the judgment condition of the speed limit of the violation item point.

[0055] The S4 specifically includes the following steps: according to the requirements, using the locomotive running speed detected in S3 to conditionally screen the violation actions detected in S2 (according to the requirements of the railway bureau, some violation actions have speed limit requirements, such as chatting, making a phone call, playing a mobile phone while the locomotive is running (speed greater than 0), smoking when the speed is less than a specified value (when passing through a specified road section with speed limit), and sleeping while lying down (the locomotive cannot sleep when entering a train station or running in a specified road section during the running period).

[0056] According to the duration of the identified action (the image can be obtained from the video in seconds by using the ffmpeg method in S1, and the duration of the action can be calculated according to the number of detections before and after), and the approximate calculation of the locomotive running distance (the locomotive requires the assistant driver to stand up when passing through a specified road section (fixed distance)), it is judged whether the locomotive crew is in violation.

[0057] According to the comparison of the duration of the identified action and the required duration of the violation behavior item point, further eliminate the occurrence of the condition. Among them, the required duration of the head leaning item point is more than 5 seconds. (Different behavior actions in the railway locomotive have their own duration and activity range in time and space.)

[0058] The required duration of action includes the short duration (1-5 seconds) of actions such as hand gestures, reaching out, clenched fists, input confirmation, and head deviation; the long duration (10 seconds or even 30 seconds or more) of actions such as standing in front of the seat, using the locomotive seat, leaving the post, making a phone call, simultaneously changing ends by two people, interrupting the lookout, looking at the mobile phone, chatting during shunting operation, lying down to sleep, and single-person operation; the long duration (5-10 seconds) of actions such as dozing off, smoking, not holding the brake, and false flag.

[0059] According to the detected illegal picture and the occurrence time (the video name records the time when the video starts to occur, and according to the video frame rate and the image frame number, the occurrence time of the illegal picture can be calculated), the LKJ train data (LKJ data is a train operation monitoring record setting, which is a new generation of train overspeed protection equipment in China, which can accurately record the train running condition, signal equipment condition and operation condition of the crew. It records the time when the locomotive passes through the signal light and the action that the crew needs to take. The intelligent illegal action recognition of the locomotive video can know whether the crew has taken the action at the time point by comparing the LKJ data with the corresponding action and obtaining the corresponding occurrence time. According to the locomotive running time recorded by the LKJ train data, the actions that the crew should take but do not take, such as reaching out, hand gestures, input confirmation, clenched fists, and standing in front of the seat, are analyzed; and according to the locomotive speed recorded by the LKJ train data, the actions that meet the conditions, such as smoking, chatting with the head, making a phone call, playing a mobile phone, and lying down to sleep, are further screened out. (According to the requirements of the railway bureau, some illegal actions have speed limit requirements. The locomotive running time, chatting with the head, making a phone call, and playing a mobile phone are illegal actions. The requirement for smoking is that the speed is less than a specified value (when passing through a specified road section with speed limit, smoking is not allowed). The requirement for lying down to sleep is that the locomotive is in a stationary state.)

[0060] The target of the illegal behavior in S2 is to determine the target detection type according to the action category of the illegal behavior, including the normal points for the locomotive operation personnel such as hand gestures, reaching out, clenched fists, input confirmation, standing in front of the seat, and using the locomotive seat; and the illegal points such as leaving the post, making a phone call, dozing off, smoking, single-person operation, head deviation, not holding the brake, lying down to sleep, simultaneously changing ends by two people, interrupting the lookout, looking at the mobile phone, chatting during shunting operation, and false flag.

[0061] The present application has low cost: the user only needs to run the analysis program, without manual video monitoring, reducing labor cost; fast speed: based on single-frame image analysis, without analyzing all image frames in the video segment, reducing the number of analyzed images, and improving the analysis speed; high accuracy: fusion of image processing feature point statistical value and edge information, with duration and space as constraints, to eliminate interference results and improve accuracy.

[0062] The above merely describes the preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, several changes and improvements can be made without departing from the overall concept of the present application, and these should also be considered as the protection scope of the present application.

Claims

1. A retrieval method for intelligent video violation recognition of railway locomotives, characterized in that: The method includes the following steps: S1: Image Acquisition: Read locomotive monitoring video using ffmpeg, capturing one image per second; S2: Object detection: Perform object detection on the image obtained in S1 using the deep learning network Faster-Rcnn to determine whether the current image contains locomotive crew members and objects of illegal behavior; S3: Feature Extraction: Process the image in S1 to obtain the current locomotive speed; based on the number of video frames extracted per second as the time estimation basis, obtain the duration of the target behavior; S4: Comprehensive Analysis: Based on the locomotive speed and duration of the target behavior obtained in S3, the violation targets detected in S2 are filtered according to the conditions, and the violations that do not meet the conditions are eliminated. S5: Operation Analysis: Compare the selected images of violations containing the target violations and their occurrence times with LKJ train data. Based on the locomotive running time recorded in the LKJ train data and the gestures and actions made by the locomotive crew during that running time, analyze the actions that the crew should have performed but did not. Based on the locomotive speed recorded in the LKJ train data and the locomotive's position at the corresponding time, further filter out violations that meet the criteria. Save the images corresponding to the crew's violations in real time. S6: Statistics and Output: Perform probabilistic statistics on the time periods and frequency of violations; The deep learning network Faster-R-CNN includes model training; the model training includes the following steps: Datasets were created by extracting images from locomotive monitoring videos and labeling them with targets. The detected targets in the dataset are grouped by label according to the category of the violation behavior; The images with classification labels are scaled to M*N and fed into the deep learning network Faster-Rcnn for model training to obtain the final object detection model. The training of the deep learning network Faster-R-CNN includes: 1) First, the images in the dataset are used to extract features through the backbone, using a ResNet_50 network; after 4 pooling layers, the resulting feature maps are denoted as layers C2, C3, C4, and C5, with shapes of [1, M / 2, N / 2, 256], [1, M / 8, N / 8, 512], [1, M / 16, N / 16, 1024], and [1, M / 32, N / 32, 2048], respectively; 2) 4 feature maps obtained through backbone, after FPN layer, M5, M4, M3, M2 feature maps are obtained; M2, M3, M4, M5 feature maps are obtained through a 3*3 convolution kernel, P2, P3, P4, P5 feature maps are obtained; P6 feature map is obtained by downsampling P5 feature map; the shape of the 5 feature maps is: [1, M / 2, N / 2, 256], [1, M / 4, N / 4, 512], [1, M / 8, N / 8, 1024], [1, M / 16, N / 16, 2048], [1, M / 32, N / 32, 256]; 3) RPN network processes P2, P3, P4, P5, P6 feature maps through a series of fixed reference frames anchors, judges whether the anchor contains the target through softmax classification, calculates the offset of the target frame to the anchor, and obtains the accurate candidate region; finally, the Proposal layer processes the anchors containing the target and the corresponding bbox regression offset to obtain the candidate region, and eliminates the candidate regions that are too small and exceed the boundary; 4) ROIPooling layer maps the candidate regions generated by RPN network to the feature map and fixes the dimension for classification and regression; after ROIPooling layer, the feature map with uniform size is obtained; the feature map is classified through the subsequent full connection layer and softmax calculation to obtain which behavior category the candidate region belongs to, and the probability value is output; at the same time, the position offset of each candidate region is obtained by regression method, and the more accurate target detection frame is obtained by regression; The S3 specifically comprises: according to the obtained locomotive monitoring image, first, the image is converted into a gray image using cv2.cvtcolor(); then the image is binarized using cv2.threshold(); then the image is eroded and dilated using cv2.erode() and cv2.dilate() to eliminate the interference of part of the fine lines in the image; then the contour value on the image is found using cv2.findcontours(); the contour value is calculated using cv2.contourArea() according to the contour value; whether there is an obstruction in front of the camera to hinder video analysis is judged according to the area ratio of the detected contour value, if there is, the judgment of the camera obstruction violation point is obtained at the same time; the position of the locomotive speed in the monitoring image is extracted, the digital template is matched using cv2.matchTemplate() according to the digital template produced by the locomotive video display, and the current locomotive running speed is obtained as the judgment condition of the speed limit of the violation point.

2. The method of claim 1, wherein: The S4 specifically includes the following steps: according to the requirements of the railway bureau, part of the illegal actions have speed limit requirements, using the locomotive running speed detected in S3 to conditionally screen the illegal actions detected in S2; when the locomotive is running, i.e. the locomotive speed is greater than 0, leaning head to chat, making a phone call, playing a mobile phone are illegal behaviors; smoking requires the speed to be less than a specified value, and smoking is not allowed when passing through a specified route with speed limit, lying down to sleep requires a stop state, and the locomotive cannot sleep during the running period in the train station or the specified route, otherwise it is an illegal behavior; According to the duration of the identified action and the locomotive running speed, the locomotive forward mileage is calculated, the locomotive is required to stand when passing through a fixed mileage, and it is judged whether the locomotive crew has an illegal behavior of not standing; According to the comparison of the duration of the identified action and the required duration of the illegal behavior action item, further eliminate the illegal behaviors that do not meet the conditions.

3. The method of claim 1, wherein: The S2 illegal behavior target is to determine the target detection type according to the illegal behavior action category, including the actions that the crew should take and illegal actions; the actions that the crew should take include reaching out, hand comparison, input confirmation, clenched fist, standing in front of the seat, and using the locomotive mobile phone; the illegal actions include smoking, leaning head to chat, making a phone call, playing a mobile phone, single person on duty, and lying down to sleep.

Images