A method and system for detecting driving actions of a subway driver

Abstract

The application provides a subway driver driving action detection method and system, comprising: obtaining video data to be detected; using a pre-trained action detection model to perform target detection on a target object in the video data to be detected, and obtaining a region where the target object is located; and based on the region where the target object is located, performing action recognition on the target object to obtain an action category. The application can not only quickly and accurately realize region positioning, but also realize action recognition, saves the cost of manual determination, has high automation and intelligence, and thus can meet the increasing demand for operation safety.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a subway driver driving action detection method and system. BACKGROUND

[0002] With the rapid development of urban rail transit, higher requirements are put forward for train operation safety. Train drivers are an important role to ensure the safe operation of trains. Drivers need to confirm with their fingers according to the actual steps completed to ensure that no step is missed. These gestures indicate that the current equipment is in a normal operating state. The category of the driver's action is crucial to determine whether the corresponding equipment is currently in a normal operating state. Therefore, it is required to detect the driver's action category in real time to provide a basis for judging the normal operation of the equipment and ensure the safe and reliable operation of the train.

[0003] Currently, the detection of subway driver actions mainly relies on manual review of surveillance videos, which is labor-intensive, low-efficiency, and has poor real-time performance and low automation and intelligence, making it difficult to meet the growing demand for operational safety. SUMMARY

[0004] Embodiments of the present application provide a subway driver driving action detection method and system to overcome the shortcomings of the prior art.

[0005] To achieve the above-mentioned purpose, the present application adopts the following technical solutions.

[0006] In a first aspect, the present application provides a subway driver driving action detection method, comprising:

[0007] obtaining video data to be detected;

[0008] using a pre-trained action detection model to perform target detection on a target object in the video data to be detected, obtaining a region where the target object is located; and based on the region where the target object is located, performing action recognition on the target object to obtain an action category.

[0009] Optionally, the pre-trained action detection model includes a target detection network and an action recognition network.

[0010] The pre-trained action detection model is used to perform target detection on a target object in the video data to be detected, obtain a region where the target object is located; and based on the region where the target object is located, perform action recognition on the target object to obtain an action category, comprising:

[0011] using the target detection network to perform target detection on the target object in the video data to be detected, and obtaining a region where the target object is located;

[0012] perform action recognition on the target object based on the region where the target object is located by using the action recognition network to obtain an action category.

[0013] Optionally, the action recognition network comprises a first convolution module, a spatial enhancement module, a plurality of permutation units and a second convolution module connected in sequence.

[0014] The action recognition on the target object based on the region where the target object is located by using the action recognition network to obtain an action category comprises:

[0015] performing convolution processing on the to-be-detected video data by using the first convolution module to obtain a first convolution feature map;

[0016] performing spatial enhancement on the first convolution feature map by using the spatial enhancement module to obtain an enhanced feature map;

[0017] extracting spatio-temporal features from the enhanced feature map by using the plurality of permutation units to obtain spatio-temporal features;

[0018] performing convolution processing on the spatio-temporal features by using the second convolution module to obtain a second convolution feature map;

[0019] performing ROI alignment and ROI pooling on the second convolution feature map and an anchor box corresponding to the region where the target object is located to obtain feature information corresponding to the anchor box;

[0020] performing action recognition on the feature information corresponding to the anchor box by using a fully connected layer to obtain an action category.

[0021] Optionally, the plurality of permutation units comprise a first permutation unit, a second permutation unit and a third permutation unit connected in sequence.

[0022] The first permutation unit comprises a first permutation module, a second permutation module and a permutation attention module connected in sequence.

[0023] The second permutation unit comprises a first permutation module, a second permutation module and a permutation attention module connected in sequence.

[0024] The third permutation unit comprises a second permutation module and a permutation attention module connected in sequence.

[0025] The first permutation module comprises a first branch, a second branch, a third branch, a fourth branch, a concatenation layer and a channel permutation layer of the first permutation module.

[0026] The second permutation module comprises a channel division layer, a first branch, a second branch, a third branch, a concatenation layer and a channel permutation layer of the second permutation module.

[0027] the third branch comprises a 1x1x1 convolutional layer, a 5x5x5 depth separable convolutional layer and a 1x1x1 convolutional layer connected in sequence;

[0028] the fourth branch comprises a 5x5x5 depth separable convolutional layer and a 1x1x1 convolutional layer connected in sequence.

[0029] Optionally, the method further comprises:

[0030] performing channel grouping on the feature map output by the first permutation module or the second permutation module by using the permutation attention module to obtain a plurality of groups of feature maps;

[0031] respectively processing the plurality of groups of feature maps by using the channel attention mechanism and the spatial attention mechanism in the permutation attention module to correspondingly obtain a channel importance coefficient and a spatial importance coefficient of each group of feature maps;

[0032] based on the channel importance coefficient and the spatial importance coefficient, performing splicing and fusion on the plurality of groups of feature maps by using the permutation attention module, and performing inter-group communication on the fused feature map by using channel permutation to obtain a feature map output by the permutation attention module.

[0033] Optionally, the spatial enhancement module is used to perform spatial enhancement on the first convolutional feature map to obtain an enhanced feature map, comprising:

[0034] performing global average pooling and global maximum pooling on the first convolutional feature map along the channel dimension by using the spatial enhancement module, and splicing the average pooled feature map and the maximum pooled feature map to obtain a spliced pooled feature map;

[0035] performing feature extraction on the spliced pooled feature map by using the 3D convolutional layer in the spatial enhancement module, and performing activation on the feature map output by the 3D convolutional layer by using the activation function in the spatial enhancement module to obtain an activated feature map;

[0036] multiplying the activated feature map and the first convolutional feature map to obtain a multiplied feature map as the enhanced feature map.

[0037] Optionally, the target detection network is MobileNetV2-SSDLite.

[0038] Optionally, the target detection network comprises a two-dimensional standard convolutional layer, a plurality of first bottleneck layers, a two-dimensional standard convolutional layer and a plurality of second bottleneck layers connected in sequence.

[0039] Optionally, the pre-trained action detection model is trained based on training video data and corresponding label data thereof, and the training video data is action video data of a driver in a cab.

[0040] In a second aspect, the present application further provides a subway driver driving action detection system, comprising:

[0041] The region detection and action recognition module is configured to perform target detection on the target object in the to-be-detected video data by using the pre-trained action detection model, to obtain a region where the target object is located, and perform action recognition on the target object based on the region where the target object is located, to obtain an action category.

[0042] The subway driver driving action detection method and system provided by the present application can quickly and accurately realize region positioning and action recognition, save the cost of manual determination, and have high automation and intelligence, thereby meeting the increasing demand for operation safety.

[0043] Additional aspects and advantages of the present application will be given in part in the following description, which will become apparent from the following description, or will be learned by practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0045] Figure 1 One of the flowcharts of the subway driver driving action detection method provided by the embodiment of the present application;

[0046] Figure 2 The structural diagram of the action detection model provided by the embodiment of the present application;

[0047] Figure 3 The structural diagram of the first permutation module and the second permutation module provided by the embodiment of the present application;

[0048] Figure 4a The flowchart of the channel permutation layer data processing provided by the embodiment of the present application;

[0049] Figure 4b ForFigure 4a grayscale image;

[0050] Figure 5 This is a schematic diagram of the process of the displacement attention module provided in an embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram of the spatial enhancement module processing flow provided in an embodiment of the present invention;

[0052] Figure 7 A schematic diagram of the bottleneck layer provided in an embodiment of the present invention;

[0053] Figure 8 This is a second flowchart illustrating a method for detecting driving actions of a subway driver, provided as an embodiment of the present invention.

[0054] Figure 9 This is a schematic diagram of motion detection results provided in an embodiment of the present invention;

[0055] Figure 10 This is a schematic diagram of action categories provided for an embodiment of the present invention. Detailed Implementation

[0056] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0057] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0058] Those skilled in the art will understand that, unless otherwise defined, all terms used herein have the same meaning as commonly understood by one of ordinary skill in the art in the field of the invention. It will also be appreciated that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0059] Current research on subway driver action detection can be divided into two categories: image processing technology-based and deep learning technology-based. Among them, the action recognition method based on image processing technology can be divided into three types: template-based, space-time interest point-based and motion trajectory-based. The template-based method is suitable for simple actions, and has high requirements for the selection of templates; the space-time interest point-based method needs to detect the interest points in the video, i.e. the points with the most dramatic changes in space-time dimensions, which can greatly improve the action recognition accuracy, but the interest point detection calculation is large and cannot achieve real-time detection; the motion trajectory-based method uses the motion trajectory of the human body key points to represent the action, which has the characteristics of good robustness and strong anti-interference ability, but it is severely dependent on the accurate estimation of the key points and the precise tracking of the key points, and the calculation amount of the model is large, which cannot realize real-time detection. The method based on deep learning technology uses deep convolutional neural network to extract the space-time features of the action video, which can be further divided into three types: three-dimensional convolutional neural network-based, double-flow convolutional neural network-based and long short-term memory network-based. Compared with the action recognition method based on traditional image processing technology, the method based on deep learning technology greatly improves the accuracy and detection speed of action recognition, but due to the large model size and calculation amount, it still cannot meet the requirements of real-time action detection.

[0060] Among them, the driver action recognition based on deep learning specifically includes the following steps:

[0061] Step 1, cutting and labeling of various action videos: from the original long video, using video editing software, the segments containing each type of action in the video are cut out, so that each action segment contains only one action. After cutting, the video segments of each type of action are labeled with the same number.

[0062] Step 2, video frame extraction and sampling: for each video segment, it is extracted into a frame sequence at a fixed frame rate, and an equal interval sampling strategy is adopted, i.e. 1 frame is sampled every 8 frames, a total of 16 frames of images are sampled from the frame sequence.

[0063] Step 3, data preprocessing: the 16 frames of images after sampling are first scaled to 342 (width) x 256 (height), and then the images are cropped to 224 x 224 through random cropping operation, and then the images are standardized (normalized).

[0064] Step 4, model training: using a convolutional neural network to train the input 16-frame image, and finally obtain the trained model weight.

[0065] Step 5, model inference: input a certain action video segment into the model, perform action recognition, and the model outputs the action category and its predicted score of the video.

[0066] However, the driver action recognition based on deep learning has the following disadvantages: the current deep learning-based subway driver action recognition method mainly recognizes the cropped action segments, and each action segment only contains one action category, which cannot detect multiple action categories and perform action recognition from real-time video streams. In addition, due to the large size and large amount of calculation of the three-dimensional convolutional network model used, the model is slow in actual action recognition. In addition, the existing method is for video-level action recognition, i.e. video action classification, which cannot locate the driver area.

[0067] In view of the fact that there is no real-time high-precision and efficient subway driver driving action detection method at present, an improved 3D ShuffleNetV2 network-based subway driver driving action detection method is provided by using deep learning technology, which fully considers the action types of the driver in the real driving environment and the problem of multiple action labels of the driver. The method described in the present application can not only quickly and accurately realize region positioning, but also realize action recognition. In addition, based on this method, a subway driver action detection system is further built to read video data from the monitoring video in real time and perform real-time driver action detection. The automatic driver driving action detection method provided by the monitoring department can not only reduce labor costs, but also improve detection efficiency, improve the safety of urban rail transit, and improve the automation level of urban rail transit. A subway driver driving action detection method provided in the present application will be described below in conjunction with the accompanying drawings.

[0068] Embodiment 1

[0069] Figure 1 Fig. 1 is a flowchart of a subway driver driving action detection method provided in an embodiment of the present application; as shown in Figure 1 The subway driver driving action detection method comprises the following steps:

[0070] S101, obtaining video data to be detected.

[0071] In this step, the video data to be detected including the target is obtained, and the target can be a person, object or the like in a dynamic state, such as a vehicle, a driver, a pedestrian, etc.

[0072] S102, target detection is performed on the target object in the to-be-detected video data by using the pre-trained action detection model to obtain a region where the target object is located; and action recognition is performed on the target object based on the region where the target object is located to obtain an action category.

[0073] In this step, one three-dimensional convolutional neural network can be used to simultaneously perform target object region positioning and action recognition on the to-be-detected video data, or two subnetworks (a target detection subnetwork and an action recognition subnetwork) can be used to perform region positioning and action recognition, respectively. Through the above manner, the region where the target object is located and the action category corresponding to the target object can be simultaneously obtained.

[0074] According to the subway driver driving action detection method provided in the embodiment of the application, target detection is performed on the target object in the to-be-detected video data by using the pre-trained action detection model to obtain a region where the target object is located; and action recognition is performed on the target object based on the region where the target object is located to obtain an action category, which can not only quickly and accurately realize region positioning, but also realize action recognition, saves the cost of manual determination, and has high automation and intelligence, thereby meeting the increasing demand for operation safety.

[0075] Figure 2 As shown in a structure diagram of the action detection model provided in the embodiment of the application, Figure 2 The pre-trained action detection model includes a target detection network and an action recognition network.

[0076] The target detection is performed on the target object in the to-be-detected video data by using the pre-trained action detection model to obtain the region where the target object is located; and the action recognition is performed on the target object based on the region where the target object is located to obtain the action category, including:

[0077] The target detection network is used to perform target detection on the target object in the to-be-detected video data to obtain the region where the target object is located. It should be noted that the region where the target object is located directly output by the target detection network is a plurality of candidate positioning regions, and each target object corresponds to only one positioning region after non-maximum suppression processing.

[0078] The action recognition network is used to perform action recognition on the target object based on the region where the target object is located to obtain the action category. It should be noted that the action recognition is performed based on the positioning region after the non-maximum suppression processing.

[0079] In the embodiment, the target detection network and the action recognition network are respectively used for region positioning and action recognition. The target detection network can be an SSD series, a YOLO series, an R-CNN series, etc. In order to further improve the real-time performance of target detection, preferably, the target detection network can be a lightweight target detection network, such as NanoDet, MobileNetV2-SSDLite, etc. The action recognition network can be a 3D CNN, or a lightweight 3D convolution network, such as a ShuffleNet series, a MobileNet series, a GhostNet, etc. The use of a lightweight 3D convolution network can further improve the real-time performance of action recognition.

[0080] Further, as shown in Figure 2 The action recognition network is an improved 3D ShuffleNetV2, which includes a first convolution module, a spatial enhancement module, a plurality of group permutation units (3 groups in the embodiment, which can be adjusted according to the model performance in other embodiments, and the present application does not limit this), and a second convolution module connected in sequence.

[0081] The action recognition network is an improved 3D ShuffleNetV2, which includes a first convolution module, a spatial enhancement module, a plurality of group permutation units (3 groups in the embodiment, which can be adjusted according to the model performance in other embodiments, and the present application does not limit this), and a second convolution module connected in sequence.

[0082] The first convolution module is used for convolution processing on the to-be-detected video data, to obtain a first convolution feature map.

[0083] The spatial enhancement module is used for spatial enhancement on the first convolution feature map, to obtain an enhanced feature map.

[0084] The plurality of group permutation units are used for spatio-temporal feature extraction on the enhanced feature map, to obtain spatio-temporal features.

[0085] The second convolution module is used for convolution processing on the spatio-temporal features, to obtain a second convolution feature map.

[0086] According to the second convolution feature map and the anchor box corresponding to the region where the target object is located, ROI alignment and ROI pooling are performed, to obtain feature information corresponding to the anchor box.

[0087] The full connection layer is used for action recognition on the feature information corresponding to the anchor box, to obtain an action category.

[0088] In the embodiment, the first convolution module in the improved 3D ShuffleNetV2 is used to down-sample the input video data to be detected, so that the feature map size is reduced, and then the spatial enhancement module is used for feature extraction. The spatial enhancement module is used to reduce the influence of spatial information loss caused by the down-sampling process of the first convolution module, so as to improve the spatial information representation ability of the action recognition network. A plurality of permutation units are used to enhance the receptive field of the enhanced feature map output by the spatial enhancement module, and the second convolution module is used to perform convolution processing on the spatio-temporal features output by the last permutation unit to obtain the second convolution feature map. The positioning region anchor frame obtained in the target detection network is mapped on the second convolution feature map, and ROI alignment and ROI pooling are performed, so that each anchor frame can generate a fixed size feature, and finally the action recognition is performed through the full connection layer. Finally, the target object region and the action category and its confidence in the video are displayed.

[0089] Further, the plurality of permutation units include a first permutation unit, a second permutation unit and a third permutation unit connected in sequence;

[0090] The first permutation unit includes a first permutation module, a plurality of second permutation modules and a permutation attention module connected in sequence;

[0091] The second permutation unit includes a first permutation module, a plurality of second permutation modules and a permutation attention module connected in sequence;

[0092] The third permutation unit includes a plurality of second permutation modules and a permutation attention module connected in sequence;

[0093] The first permutation module includes a first branch, a second branch, a third branch, a fourth branch, a concatenation layer and a channel permutation layer of the first permutation module;

[0094] The second permutation module includes a channel division layer, a first branch, a second branch, a third branch, a concatenation layer and a channel permutation layer of the second permutation module;

[0095] The third branch includes a 1x1x1 convolution layer, a 5x5x5 depth separable convolution layer and a 1x1x1 convolution layer connected in sequence;

[0096] The fourth branch includes a 5x5x5 depth separable convolution layer and a 1x1x1 convolution layer connected in sequence. It should be noted that the first branch of the first permutation module and the first branch of the second permutation module are not the same, and the specific structure is shown in Figure 3 Structure diagram.

[0097] In this embodiment, in order to further improve the receptive field of the action recognition network, and because a large convolution kernel size can greatly improve the receptive field of the model, and the receptive field of the action recognition network is very important for subsequent recognition tasks, therefore, an improved permutation unit is proposed, which adds a branch with a convolution kernel size of 5x5x5 on the basis of the original permutation module (the existing permutation module only includes the first branch and the second branch), as shown in Figure 3 .

[0098] When the step size is 1 (i.e. the second permutation module), as shown in the left part of Figure 3 , on the basis of the 3x3x3 convolution kernel of the original branch 1 and branch 2 (i.e. the second branch), a branch 3 (i.e. the third branch) is added, and the convolution kernel size of the depth separable convolution is 5x5x5, so as to obtain a larger receptive field, and the other parts of the branch are consistent with branch 2, and the output channel number of the 1x1x1 convolution at the end of branch 2 and branch 3 is set to 1 / 4 of the input channel number, so as to ensure that the channel number of the final splicing is unchanged. Then branch 1, branch 2 and branch 3 are spliced together to obtain a feature map with the same channel number as the original, and then a channel permutation layer (as shown in Figure 4a and Figure 4b ) is introduced.

[0099] When the step size is 2, as shown in the right part of Figure 3 , on the basis of the original branch 1 (i.e. the first branch of the first permutation module) and branch 2, branch 3 and branch 4 (i.e. the fourth branch) are added, wherein the overall structure of branch 1 and branch 4 remains unchanged, and the difference is that the convolution kernel size of the depth separable convolution of branch 1 is 3x3x3, and the convolution kernel size of the depth separable convolution of branch 4 is 5x5x5; similarly, the overall structure of branch 3 and branch 2 remains unchanged, and the difference is that the convolution kernel size of the depth separable convolution of branch 2 is 3x3x3, and the convolution kernel size of the depth separable convolution of branch 3 is 5x5x5, in order to ensure that the channel number of the final splicing is unchanged, the output channel of the 1x1x1 convolution at the end of all branches is set to 1 / 4 of the original channel number. Then branch 1, branch 2, branch 3 and branch are spliced together to obtain a feature map with the same channel number as the original, and then a channel permutation layer (as shown in Figure 4a and Figure 4b ) is introduced to group the feature map into G groups and then process the feature map.

[0100] Figure 5 is a flowchart of the processing of the permutation attention module provided by the embodiment of the present application, as shown in Figure 5 .

[0101] The permutation attention module is used to process the feature map output by the first permutation module or the second permutation module (i.e. Figure 5channel grouping is performed on the original feature map in the first convolutional layer to obtain a plurality of groups of feature maps;

[0102] The channel attention mechanism and the spatial attention mechanism in the permutation attention module are used to process the plurality of groups of feature maps respectively, so as to obtain a channel importance coefficient and a spatial importance coefficient of each group of feature maps;

[0103] Based on the channel importance coefficient and the spatial importance coefficient, the plurality of groups of feature maps are spliced and fused by using the permutation attention module, and inter-group communication is performed on the fused feature maps by using channel permutation, so as to obtain a feature map output by the permutation attention module (i.e. Figure 5 an improved feature map in the first convolutional layer).

[0104] In this embodiment, in order to enable the action recognition network to pay more attention to important channels and spatial positions, a permutation attention module is introduced. By introducing the attention module, the action recognition network can learn the importance weight of the channel and the spatial position, multiply the initial feature map, and obtain an improved feature map.

[0105] Specifically, the feature map output by the first permutation module or the second permutation module is first divided into g groups along the channel, and each group is further divided into a channel attention branch and a spatial attention branch. After the feature map of each group is processed by the channel attention branch and the spatial attention branch respectively, a channel importance coefficient and a spatial importance coefficient are obtained. The channel attention branch adopts a combination of global average pooling, scaling and Sigmoid activation. The spatial attention branch uses group regularization (GN) to obtain statistical information of the spatial dimension, and then uses Fc(.) for enhancement.

[0106] After obtaining the channel importance coefficient and the spatial importance coefficient, the grouped feature maps are integrated according to the channel importance coefficient and the spatial importance coefficient, that is, splicing and fusion are first performed, and then inter-group communication is performed by using channel permutation, so as to obtain a final feature map output by the permutation attention module.

[0107] Figure 6 A spatial enhancement module processing flow diagram provided by the embodiment of the present application is shown in FIG. Figure 6 As shown in FIG.

[0108] The spatial enhancement module is used to perform global average pooling and global maximum pooling on the first convolutional feature map along the channel dimension respectively, and the average pooled feature map and the maximum pooled feature map are spliced to obtain a spliced pooled feature map;

[0109] The 3D convolution layer in the spatial enhancement module is used to extract features of the pooled feature map after splicing, and the activation function in the spatial enhancement module is used to activate the feature map output by the 3D convolution layer, to obtain an activated feature map.

[0110] The activated feature map is multiplied by the first convolution feature map to obtain a multiplied feature map as an enhanced feature map.

[0111] In this embodiment, the spatial enhancement module performs global average pooling and global maximum pooling on the first convolution feature map along the channel dimension respectively, splices the two feature maps obtained after pooling, and then performs feature extraction through a 3D convolution with a convolution kernel size of 7x7x7. Then, the activation function is used to improve the expression ability of the network, and finally the original feature map is multiplied to obtain an enhanced feature map.

[0112] Based on the above description, in this embodiment, the action recognition network structure and parameters are shown in Table 1:

[0113] Table 1 Action recognition network structure and parameters

[0114]

[0115]

[0116] As shown in Table 1, the first permutation unit includes a first permutation module with a step size of 2, three second permutation modules with a step size of 1, and a permutation attention module; the second permutation unit includes a first permutation module with a step size of 2, seven second permutation modules with a step size of 1, and a permutation attention module; and the third permutation unit includes four second permutation modules with a step size of 1 and a permutation attention module.

[0117] Further, the target detection network is MobileNetV2-SSDLite. The target detection network includes two-dimensional standard convolution layers (i.e., 2D convolution), multiple first bottleneck layers (i.e., bottleneck layer 1), two-dimensional standard convolution layers (i.e., 2D convolution), and multiple second bottleneck layers (i.e., bottleneck layer 2) connected in sequence.

[0118] In this embodiment, the target detection network adopts MobileNetV2-SSDLite, and the structure and parameters of the target detection network are shown in Table 2. MobileNetV2-SSDLite has the advantages of high precision, good real-time performance, and small model parameter quantity and calculation quantity for target detection. As shown in Table 2, MobileNetV2-SSDLite is composed of two standard 2D convolutions, 17 bottleneck layers 1, and 4 bottleneck layers 2.

[0119] In the process of processing the video data to be detected, the input of the MobileNetV2-SSDLite can be set to 320*320, and the IoU (intersection over union) threshold during non-maximum suppression is set to 0.45, that is, when the IoU of two anchor boxes is greater than 0.45, it is considered to be the same object. The confidence threshold is set to 0.5, that is, when the confidence of the detected anchor box is greater than 0.5, it is considered that the target object is detected.

[0120] Table 2 Driver detection network structure

[0121]

[0122]

[0123] The network structure of the bottleneck layer 1 and the network structure of the bottleneck layer 2 are as shown in FIG. 2. Figure 7 When the step is 1, the network structure of the bottleneck layer 1 includes the input (i.e., the output of the previous network layer), 1x1 convolution + ReLU6 activation, 3x3 depth separable convolution + ReLU6 activation, and 1x1 convolution + linear layer connected in sequence, and the input is connected with the 1x1 convolution + linear layer, so that the input and the result of the 1x1 convolution + linear layer are added.

[0124] When the step is 2, the network structure of the bottleneck layer 1 includes the input, 1x1 convolution + ReLU6 activation, 3x3 depth separable convolution + ReLU6 activation, and 1x1 convolution + linear layer connected in sequence.

[0125] The network structure of the bottleneck layer 2 includes the input, 1x1 convolution + ReLU6 activation (the number of channels here = C / 2), 3x3 depth separable convolution + ReLU6 activation (the number of channels here = C), and the output (the number of channels here = C) connected in sequence.

[0126] Further, the pre-trained action detection model is trained based on training video data and corresponding label data thereof, and the training video data is action video data of a driver in a cab.

[0127] Based on the aforementioned target detection network and action recognition network, in the training process, the pre-trained target detection network is directly selected for the target detection network, and only the action recognition network needs to be trained in the action detection model.

[0128] The network training parameters of the action recognition network are as follows: the adopted optimizer is a stochastic gradient descent (SGD), the initial learning rate is 0.01, the momentum weight is set to 0.9, the weight decay is set to 0.00003, the learning rate updating strategy is cosine decay, the loss function is a cross-entropy loss function, the batch size during training is set to 32, and the remaining parameters use default parameters.

[0129] After the action detection model is trained, the generated weight file is saved in.pth format, and on this basis, inference prediction can be performed only by inputting a video.

[0130] According to the subway driver driving action detection method provided by the embodiment of the application, the driver region can be automatically and accurately detected in real time and the action type can be recognized through the action detection model. Compared with the existing action recognition method, the region positioning and the multi-label action category are considered, and the action detection is realized by reading the video data from the monitoring camera in real time. The method is easy to program and implement, convenient and practical, and has high economic and social benefits. In addition, due to the use of the lightweight network, the overall model parameters and the calculation amount are small, and the method is easy to deploy and use on embedded devices and mobile devices with limited computing resources. The driver action detection system based on the model realizes the action detection by directly reading the video data from the monitoring camera, has the advantages of real-time accuracy and high efficiency, and the system algorithm is simple to implement and easy to apply.

[0131] Figure 8 As shown in FIG. 2, a flowchart of a subway driver driving action detection method provided by an embodiment of the application is shown, taking the action recognition of a subway driver as an example. Figure 8 The subway driver driving action detection method includes the following steps.

[0132] S801, reading a subway driver monitoring video, which can be recorded by two monitoring cameras installed in the driver's room (located at the lower left corner and the upper right corner of the driver's room, respectively), and the specific resolution can be 1280*720.

[0133] S802, video frame reading. In this step, since the action detection model is trained by inputting a 10-second action video segment, a 10-second action video segment is cropped from the monitoring room video (in other embodiments, the length of the video segment can be adjusted according to actual conditions, and the application does not limit this). Then, a frame sequence is extracted from each video segment at a frame rate of 30 FPS to obtain a frame sequence constituting each video segment.

[0134] S803, video frame sampling, in this step, since the action is represented by a certain number of video frames arranged in time sequence, inputting all frame sequences into the model makes the model calculation amount surge, a certain number of frames can be sampled from the video, such as 8 frames, 16 frames, 32 frames to reduce the calculation amount. Preferably, 8 frames are used in the embodiment. At the same time, the sampling interval is set to 8, that is, 1 frame of image is sampled every 8 frames of image.

[0135] S804, video frame preprocessing; in this step, standardization operation is needed for the 8 frames of image after sampling, that is, standardization operation is performed on the pixel value, and the pixel value is standardized from [0, 255] to a distribution with a mean value of 0 and a standard deviation of 1. The specific formula is as follows:

[0136]

[0137] In the formula, x is the original pixel value;

[0138] μ is the average value of the current channel pixel value in the image;

[0139] σ is the standard deviation of the current channel pixel value in the image;

[0140] x new is the new pixel value after standardization.

[0141] S805, driver area positioning, in this step, the driver area positioning is performed on the preprocessed video frame by using the target detection network mentioned above.

[0142] S806, driver action recognition, in this step, the driver action recognition is performed on the preprocessed video frame by using the action recognition network mentioned above.

[0143] S807, output action detection result as shown in Figure 9 , including action category and driver area. Among them, the action category of the driver is as shown in Figure 10 , including sitting + pointing to the front window, sitting + no other action, standing in the driver room + pressing the open / close door button, standing outside the driver room, walking from outside to inside, walking from inside to outside, sitting + pointing to the instrument and screen, sitting + pointing to the left lower instrument, sitting + pushing the instrument, etc.

[0144] The subway driver driving action detection method based on the improved 3D ShuffleNetV2 network is provided, and automatic real-time accurate detection of the driver area and identification of the action type are realized. 2) Compared with the existing action recognition method, the driver area positioning and the driver action category multi-label situation are considered. 3) The model is mainly based on a deep method, and the driver body area is positioned and the current driver action type is identified. 4) A driver action real-time detection system is built, and action detection is realized from real-time reading of video data of a monitoring camera. 5) It is easy to program and realize, convenient and practical, and has high economic and social benefits.

[0145] Embodiment 2

[0146] Based on embodiment 1, the subway driver driving action detection system provided in embodiment 2 corresponds to the above-mentioned action detection, and the subway driver driving action detection system comprises:

[0147] A data acquisition module is configured to acquire video data to be detected.

[0148] A region detection and action recognition module is configured to perform target detection on a target object in the video data to be detected by using a pre-trained action detection model, to acquire a region where the target object is located; and perform action recognition on the target object based on the region where the target object is located, to obtain an action category.

[0149] In actual application, the subway driver driving action detection system further comprises a result display module, which is configured to draw the action detection result on an image, including the action category and the corresponding confidence, synthesize a video at a fixed frame rate, and display the video in front of a screen.

[0150] For specific details, refer to the description of the subway driver driving action detection method part, which will not be repeated here.

[0151] In summary, the subway driver driving action detection method and system provided in the embodiments of the present application use a pre-trained action detection model to perform target detection on a target object in the video data to be detected, to acquire a region where the target object is located; and perform action recognition on the target object based on the region where the target object is located, to obtain an action category. Not only can the region positioning be quickly and accurately realized, but also the action recognition can be realized, the cost of manual determination is saved, and the degree of automation and intelligence is high, so as to meet the increasing demand for operation safety.

[0152] Those skilled in the art can understand that the drawings are only schematic diagrams of an embodiment, and the modules or flows in the drawings are not necessarily necessary for implementing the present application.

[0153] The various embodiments described in this specification are described with reference to a particular sequence or order, but the order of the various steps comprising the embodiments can be modified so that particular sequences or orders make no significant contribution to the progress of the embodiments, but the order of steps is not limiting. It is also possible in some embodiments that some steps can be omitted, reordered, or performed concurrently, without departing from the method described herein. As used in this specification, the singular forms "a," "an," and "the" include plural referents unless the content clearly dictates otherwise. Thus, for example, reference to "a" entity includes both single and plural forms. As used herein, the term "exemplary" is used interchangeably with "by way of example," to mean "serving as an example, instance, or illustration." Any implementation of the methods and systems described herein that is not expressly described as "exemplary" should not be construe as preferred or necessary. The detailed description is not to be regarded as limiting the described embodiments to the materials, manufacturers, and processes expressly identified. Other wide ly similar materials and methods can be used without departing from the spirit and scope of the described embodiments.

[0154] The foregoing is merely illustrative of the principles of the application, and various modifications can be made by those skilled in the art without departing from the scope and spirit of the application. The illustrated embodiments are exemplary only, and not limiting, of the scope of the inventive arrangements. Thus, there are many variations to this inventive technology. Visual representations of the variations, or individual elements thereof, can have different colors, shapes, or relative sizes in different drawings and are not intended to be anatomically accurate. Certain actions can have been described as being taken by the user, but in actual practice, the user can not perform the action.

Claims

1. A method of detecting a driving action of a subway driver, characterized by, The method comprises: acquiring to-be-detected video data; performing target detection on a target object in the to-be-detected video data by using a target detection network in a pre-trained action detection model, to obtain a region where the target object is located; and performing action recognition on the target object based on the region where the target object is located by using an action recognition network in the pre-trained action detection model, to obtain an action category; wherein the action recognition network comprises a first convolution module, a spatial enhancement module, a plurality of permutation units, and a second convolution module connected in sequence; the action recognition network comprises a first convolution module, a spatial enhancement module, a plurality of permutation units, and a second convolution module connected in sequence; the action recognition network comprises a first convolution module, a spatial enhancement module, a plurality of permutation units, and a second convolution module connected in sequence; the action recognition network comprises a first convolution module, a spatial enhancement module, a plurality of permutation units, and a second convolution module connected in sequence; the action recognition network comprises a first convolution module, a spatial enhancement module, a plurality of permutation units, and a second convolution module connected in sequence; the action recognition network comprises a first convolution module, a spatial enhancement module, a plurality of permutation units, and a second convolution module connected in sequence; the action recognition network comprises a first convolution module, a spatial enhancement module, a plurality of permutation units, and a second convolution module connected in sequence; the action recognition network comprises a first convolution module, a spatial enhancement module, a plurality of permutation units, and a second convolution module connected in sequence; the action recognition network comprises a first convolution module, a spatial enhancement module, a plurality of permutation units, and a second convolution module connected in sequence; the action recognition network comprises a first convolution module, a spatial enhancement module, a plurality of permutation units, and a second convolution module connected in sequence; the action recognition network comprises a first convolution module, a spatial enhancement module, a plurality of permutation units, and a second convolution module connected in sequence; the action recognition network comprises a first convolution module, a spatial enhancement module, a plurality of permutation units, and a second convolution module connected in sequence; the action recognition network comprises a first convolution module, a spatial enhancement module, a plurality of permutation units, and a second convolution module connected in sequence; the action recognition network comprises a first convolution module, a spatial enhancement module, a plurality of permutation units, and a second convolution module connected in sequence; the action recognition network comprises a first convolution module, a spatial enhancement module, a plurality of permutation units, and a second convolution module connected in sequence. The method further comprises:

2. The method of claim 1, wherein grouping channels of a feature map output by the first permutation module or the second permutation module by using the permutation attention module, to obtain a plurality of groups of feature maps; processing the plurality of groups of feature maps by using a channel attention mechanism and a spatial attention mechanism in the permutation attention module, respectively, to correspondingly obtain a channel importance coefficient and a spatial importance coefficient of each group of feature maps; ​ Based on the channel importance coefficient and the spatial importance coefficient, the plurality of groups of feature maps are spliced and fused by using the permutation attention module, and channel permutation is used for inter-group communication of the fused feature maps to obtain feature maps output by the permutation attention module.

3. The method of claim 1, wherein The spatial enhancement module is used for spatial enhancement on the first convolution feature map to obtain an enhanced feature map, including: The spatial enhancement module is used for global average pooling and global maximum pooling on the first convolution feature map along the channel dimension respectively, and the average pooled feature map and the maximum pooled feature map are spliced to obtain a spliced pooled feature map; The 3D convolution layer in the spatial enhancement module is used for feature extraction on the spliced pooled feature map, and the activation function in the spatial enhancement module is used for activation on the feature map output by the 3D convolution layer to obtain an activated feature map; The activated feature map and the first convolution feature map are multiplied to obtain a multiplied feature map as the enhanced feature map.

4. The method of claim 1 to 3, wherein The target detection network is MobileNetV2-SSDLite.

5. The method of claim 4, wherein The target detection network includes a two-dimensional standard convolution layer, a plurality of first bottleneck layers, a two-dimensional standard convolution layer, and a plurality of second bottleneck layers connected in sequence.

6. The method of claim 1, wherein The pre-trained action detection model is trained based on training video data and corresponding label data, and the training video data is driver action video data in a cab.

7. A subway driver driving motion detection system characterized by comprising: It includes: a data acquisition module for acquiring video data to be detected; a region detection and action recognition module for target detection on target objects in the video data to be detected by using a target detection network in a pre-trained action detection model to obtain a region where the target objects are located; and based on the region where the target objects are located, an action recognition network in the pre-trained action detection model is used to recognize the actions of the target objects to obtain an action category; The action recognition network includes a first convolution module, a spatial enhancement module, a plurality of permutation units, and a second convolution module connected in sequence. The action recognition network includes a first convolution module, a spatial enhancement module, a plurality of permutation units, and a second convolution module connected in sequence. The first convolution module is used for convolution processing on the video data to be detected to obtain a first convolution feature map; The spatial enhancement module is used for spatial enhancement on the first convolution feature map to obtain an enhanced feature map; The plurality of permutation units are used for spatiotemporal feature extraction on the enhanced feature map to obtain spatiotemporal features; The second convolution module is used for convolution processing on the spatiotemporal features to obtain a second convolution feature map; According to the second convolution feature map and the anchor box corresponding to the region where the target objects are located, ROI alignment and ROI pooling are performed to obtain feature information corresponding to the anchor box; The full connection layer is used for action recognition on the feature information corresponding to the anchor box to obtain an action category; The plurality of permutation units include a first permutation unit, a second permutation unit, and a third permutation unit connected in sequence. The first permutation unit comprises a first permutation module, a second permutation module and a permutation attention module connected in sequence; The second permutation unit comprises a first permutation module, a second permutation module and a permutation attention module connected in sequence; The third permutation unit comprises a second permutation module and a permutation attention module connected in sequence; The first permutation module comprises a first branch, a second branch, a third branch, a fourth branch, a splicing layer and a channel permutation layer of the first permutation module; The second permutation module comprises a channel division layer, a first branch, a second branch, a third branch, a splicing layer and a channel permutation layer of the second permutation module; The third branch comprises a 1x1x1 convolution layer, a 5x5x5 depth separable convolution layer and a 1x1x1 convolution layer connected in sequence; The fourth branch comprises a 5x5x5 depth separable convolution layer and a 1x1x1 convolution layer connected in sequence.

Images