A locomotive, a vehicle terminal and a driver behavior recognition method

Abstract

The embodiment of the application provides a locomotive, a vehicle-mounted terminal and a driver behavior identification method, and relates to the technical field of computers. The method can acquire a first monitoring video frame and a second monitoring video frame, wherein the first monitoring video frame comprises a face region of a driver, and the second monitoring video frame comprises the face region and a body region of the driver, then determines a to-be-identified region in the first monitoring video frame based on the position of a face frame and a face key point in the first monitoring video frame, and determines a first identification result according to the to-be-identified region, and determines a second identification result based on the position of a skeleton point in the second monitoring video frame, and finally determines whether the driver has a target driving behavior within a set time according to the first identification result and the second identification result. Since the first identification result and the second identification result are combined to jointly determine whether the driver has the target driving behavior, the identification accuracy of identifying the driving behavior of the driver is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a locomotive, an on-board terminal, and a method for driver behavior recognition. Background Technology

[0002] As a core component of road transportation, drivers' poor driving behavior poses a significant threat to the lives and property of passengers and other road users. Therefore, effectively identifying and monitoring drivers' poor driving behavior is crucial for correcting such behavior and ensuring safe vehicle operation.

[0003] Existing driver behavior recognition methods often rely on other auxiliary modules such as infrared camera acquisition modules, thermal sensing data acquisition modules, and binocular 3D camera acquisition modules to identify driver behavior. However, these methods suffer from low recognition accuracy. Summary of the Invention

[0004] To address the technical problems existing in related technologies, embodiments of this application provide a method for recognizing a locomotive, an on-board terminal, and a driver's behavior, which can improve the recognition accuracy of driver behavior identification.

[0005] To achieve the above objectives, the technical solution of this application embodiment is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a locomotive, comprising:

[0007] A video frame acquisition component is used to acquire a first monitoring video frame and a second monitoring video frame; the first monitoring video frame includes the driver's face area, and the second monitoring video frame includes the driver's face area and body area;

[0008] A memory for storing program code and data generated during the operation of the locomotive;

[0009] The processor is configured to execute the program code to implement the following process: determining the region to be identified in the first monitoring video frame based on the position of the face frame and facial key points in the first monitoring video frame, and determining a first identification result based on the region to be identified; and determining a second identification result based on the position of the skeletal points in the second monitoring video frame; and determining whether the driver has engaged in the target driving behavior within a set time period based on the first identification result and the second identification result.

[0010] The locomotive provided in this application embodiment can first acquire a first monitoring video frame and a second monitoring video frame. The first monitoring video frame includes the driver's face area, and the second monitoring video frame includes the driver's face area and body area. Then, based on the positions of the face bounding box and facial key points in the first monitoring video frame, the region to be identified in the first monitoring video frame is determined, and a first identification result is determined based on the region to be identified. A second identification result is determined based on the positions of skeletal points in the second monitoring video frame. Finally, based on the first and second identification results, it is determined whether the driver exhibits the target driving behavior within a set time period. Because the first and second identification results are combined to jointly determine whether the driver exhibits the target driving behavior, the accuracy of identifying the driver's driving behavior is improved.

[0011] In one possible implementation, the processor is specifically used for:

[0012] The first monitoring video frame is input into a multi-task convolutional neural network, and the face bounding box and facial key points contained in the first monitoring video frame are determined through the multi-task convolutional neural network.

[0013] Based on the positions of the face frame and the facial key points in the first monitoring video frame, the region to be identified in the first monitoring video frame is determined.

[0014] The aforementioned locomotive can input a first monitoring video frame into a multi-task convolutional neural network (CNN). The CNN then determines the face bounding boxes and facial key points within the first monitoring video frame. Based on the positions of these boxes and key points, the region to be identified within the first monitoring video frame is determined. Because the face bounding boxes and key points can be extracted more accurately from the first monitoring video frame, the required region to be identified can be determined within that frame, reducing the impact of background pixels on target behavior recognition.

[0015] In one possible implementation, the processor is further configured to:

[0016] Feature extraction is performed on the region to be identified in the first monitoring video frame to obtain the region features corresponding to the region to be identified.

[0017] The driving behavior of the driver in the first monitoring video frame is identified by using a machine learning model based on the regional features corresponding to the region to be identified.

[0018] If, within a set time period, the number of first monitoring video frames identifying preliminary target driving behavior exceeds a first set threshold, then the first monitoring video frames identifying preliminary target driving behavior are extracted, and a first identification result is determined based on the region to be identified in the extracted first monitoring video frames identifying preliminary target driving behavior using a deep learning model.

[0019] The aforementioned locomotive can extract features from the region to be identified in the first monitoring video frame, obtaining the corresponding region features. Then, using a machine learning model, it identifies the driver's driving behavior in the first monitoring video frame based on these region features. Within a set time period, if the number of first monitoring video frames identifying preliminary target driving behavior exceeds a first set threshold, these frames can be extracted. A deep learning model then determines the first identification result based on the region to be identified within the extracted frames. Since the machine learning model identifies the first monitoring video frames with preliminary target driving behavior, further confirmation using a deep learning model is needed, thereby improving the accuracy of driver behavior identification.

[0020] In one possible implementation, the processor is further configured to:

[0021] Extract the directional gradient histogram features, local binary pattern features, and Haar features of the region to be identified, respectively.

[0022] The directional gradient histogram features, the local binary pattern features, and the Haar features are fused to obtain the region features corresponding to the region to be identified.

[0023] The aforementioned locomotive can extract the directional gradient histogram (HGD) features, local binary pattern features, and Haar features of the region to be identified, and then fuse these features to obtain the corresponding region features. Because three features are extracted separately and fused into a single region feature, the influence of external factors such as lighting on the recognition results can be reduced, improving the accuracy of the machine learning model in recognizing the driver's driving behavior.

[0024] In one possible implementation, the processor is further configured to:

[0025] The skeleton point detection model is used to detect skeleton points in the second monitoring video frame to determine the position of skeleton points in the second monitoring video frame; based on the position of the skeleton points, the driving behavior of the driver in the second monitoring video frame is identified.

[0026] Within a set time period, the second identification result is determined based on whether the number of second monitoring video frames identifying target driving behavior is greater than a second set threshold.

[0027] The aforementioned locomotive can use a skeletal point detection model to detect skeletal points in the second monitoring video frame, determine the positions of skeletal points in the second monitoring video frame, and identify the driver's driving behavior in the second monitoring video frame based on the skeletal point positions. Within a set time, the second identification result can be determined by whether the number of second monitoring video frames in which the target driving behavior is identified exceeds a second set threshold. Because the skeletal point positions in the second monitoring video frame can be determined relatively accurately, the identification accuracy of the driver's driving behavior in the second monitoring video frame based on skeletal point positions can be improved.

[0028] In one possible implementation, the processor is further configured to:

[0029] If it is determined that the driver is engaging in the target driving behavior within the set time period, an alarm message will be issued.

[0030] If the locomotive is found to be engaging in targeted driving behavior within a set time period, an alarm will be issued. Because an alarm can be issued upon detection of targeted driving behavior, the driver's actions can be communicated to relevant personnel for correction.

[0031] Secondly, embodiments of this application provide a driver behavior recognition method, including:

[0032] Acquire a first monitoring video frame and a second monitoring video frame; the first monitoring video frame includes the driver's face area, and the second monitoring video frame includes the driver's face area and body area;

[0033] Based on the positions of the face bounding box and facial key points in the first surveillance video frame, the region to be identified in the first surveillance video frame is determined, and a first identification result is determined based on the region to be identified; and

[0034] The second recognition result is determined based on the skeletal point positions in the second monitoring video frame;

[0035] Based on the first identification result and the second identification result, it is determined whether the driver exhibits the target driving behavior within a set time period.

[0036] In one possible implementation, determining the region to be identified in the first monitoring video frame based on the positions of the face bounding box and facial key points in the first monitoring video frame includes:

[0037] The first monitoring video frame is input into a multi-task convolutional neural network, and the face bounding box and facial key points contained in the first monitoring video frame are determined through the multi-task convolutional neural network.

[0038] Based on the positions of the face frame and the facial key points in the first monitoring video frame, the region to be identified in the first monitoring video frame is determined.

[0039] In one possible implementation, determining the first identification result based on the region to be identified includes:

[0040] Feature extraction is performed on the region to be identified in the first monitoring video frame to obtain the region features corresponding to the region to be identified.

[0041] The driving behavior of the driver in the first monitoring video frame is identified by using a machine learning model based on the regional features corresponding to the region to be identified.

[0042] If, within a set time period, the number of first monitoring video frames identifying preliminary target driving behavior exceeds a first set threshold, then the first monitoring video frames identifying preliminary target driving behavior are extracted, and a first identification result is determined based on the region to be identified in the extracted first monitoring video frames identifying preliminary target driving behavior using a deep learning model.

[0043] In one possible implementation, the step of extracting features from the region to be identified in the first surveillance video frame to obtain the region features corresponding to the region to be identified includes:

[0044] Extract the directional gradient histogram features, local binary pattern features, and Haar features of the region to be identified, respectively.

[0045] The directional gradient histogram features, the local binary pattern features, and the Haar features are fused to obtain the region features corresponding to the region to be identified.

[0046] In one possible implementation, determining the second recognition result based on the skeletal point positions in the second monitoring video frame includes:

[0047] The skeleton point detection model is used to detect skeleton points in the second monitoring video frame to determine the position of skeleton points in the second monitoring video frame.

[0048] Based on the location of the skeletal points, the driver's driving behavior in the second monitoring video frame is identified;

[0049] Within a set time period, the second identification result is determined based on whether the number of second monitoring video frames identifying target driving behavior is greater than a second set threshold.

[0050] In one possible implementation, the method further includes:

[0051] If it is determined that the driver is engaging in the target driving behavior within the set time period, an alarm message will be issued.

[0052] Thirdly, embodiments of this application provide a vehicle-mounted terminal, including:

[0053] A video frame acquisition component is used to acquire a first monitoring video frame and a second monitoring video frame; the first monitoring video frame includes the driver's face area, and the second monitoring video frame includes the driver's face area and body area;

[0054] The memory is used to store program code and data generated during the operation of the vehicle-mounted terminal.

[0055] The processor is configured to execute the program code to implement the following process: determining the region to be identified in the first monitoring video frame based on the position of the face frame and facial key points in the first monitoring video frame, and determining a first identification result based on the region to be identified; and determining a second identification result based on the position of the skeletal points in the second monitoring video frame; and determining whether the driver has engaged in the target driving behavior within a set time period based on the first identification result and the second identification result.

[0056] In one possible implementation, the processor is specifically used for:

[0057] The first monitoring video frame is input into a multi-task convolutional neural network, and the face bounding box and facial key points contained in the first monitoring video frame are determined through the multi-task convolutional neural network.

[0058] Based on the positions of the face frame and the facial key points in the first monitoring video frame, the region to be identified in the first monitoring video frame is determined.

[0059] In one possible implementation, the processor is further configured to:

[0060] Feature extraction is performed on the region to be identified in the first monitoring video frame to obtain the region features corresponding to the region to be identified.

[0061] The driving behavior of the driver in the first monitoring video frame is identified by using a machine learning model based on the regional features corresponding to the region to be identified.

[0062] If, within a set time period, the number of first monitoring video frames identifying preliminary target driving behavior exceeds a first set threshold, then the first monitoring video frames identifying preliminary target driving behavior are extracted, and a first identification result is determined based on the region to be identified in the extracted first monitoring video frames identifying preliminary target driving behavior using a deep learning model.

[0063] In one possible implementation, the processor is further configured to:

[0064] Extract the directional gradient histogram features, local binary pattern features, and Haar features of the region to be identified, respectively.

[0065] The directional gradient histogram features, the local binary pattern features, and the Haar features are fused to obtain the region features corresponding to the region to be identified.

[0066] In one possible implementation, the processor is further configured to:

[0067] The skeleton point detection model is used to detect skeleton points in the second monitoring video frame to determine the position of skeleton points in the second monitoring video frame; based on the position of the skeleton points, the driving behavior of the driver in the second monitoring video frame is identified.

[0068] Within a set time period, the second identification result is determined based on whether the number of second monitoring video frames identifying target driving behavior is greater than a second set threshold.

[0069] In one possible implementation, the processor is further configured to:

[0070] If it is determined that the driver is engaging in the target driving behavior within the set time period, an alarm message will be issued.

[0071] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the driver behavior recognition method of the second aspect described above.

[0072] Fifthly, embodiments of this application also provide a driver behavior recognition device, the device comprising:

[0073] A video frame acquisition unit is used to acquire a first monitoring video frame and a second monitoring video frame; the first monitoring video frame includes the driver's face area, and the second monitoring video frame includes the driver's face area and body area;

[0074] The recognition result determination unit is used to determine the region to be recognized in the first monitoring video frame based on the position of the face frame and facial key points in the first monitoring video frame, and to determine a first recognition result based on the region to be recognized; and to determine a second recognition result based on the position of the skeletal points in the second monitoring video frame.

[0075] The driving behavior determination unit is used to determine, based on the first identification result and the second identification result, whether the driver has engaged in the target driving behavior within a set time period.

[0076] In one possible implementation, the identification result determination unit is specifically used for:

[0077] The first monitoring video frame is input into a multi-task convolutional neural network, and the face bounding box and facial key points contained in the first monitoring video frame are determined through the multi-task convolutional neural network.

[0078] Based on the positions of the face frame and the facial key points in the first monitoring video frame, the region to be identified in the first monitoring video frame is determined.

[0079] In one possible implementation, the identification result determination unit is further configured to:

[0080] Feature extraction is performed on the region to be identified in the first monitoring video frame to obtain the region features corresponding to the region to be identified.

[0081] The driving behavior of the driver in the first monitoring video frame is identified by using a machine learning model based on the regional features corresponding to the region to be identified.

[0082] If, within a set time period, the number of first monitoring video frames identifying preliminary target driving behavior exceeds a first set threshold, then the first monitoring video frames identifying preliminary target driving behavior are extracted, and a first identification result is determined based on the region to be identified in the extracted first monitoring video frames identifying preliminary target driving behavior using a deep learning model.

[0083] In one possible implementation, the identification result determination unit is further configured to:

[0084] Extract the directional gradient histogram features, local binary pattern features, and Haar features of the region to be identified, respectively.

[0085] The directional gradient histogram features, the local binary pattern features, and the Haar features are fused to obtain the region features corresponding to the region to be identified.

[0086] In one possible implementation, the identification result determination unit is further configured to:

[0087] The skeleton point detection model is used to detect skeleton points in the second monitoring video frame to determine the position of skeleton points in the second monitoring video frame.

[0088] Based on the location of the skeletal points, the driver's driving behavior in the second monitoring video frame is identified;

[0089] Within a set time period, the second identification result is determined based on whether the number of second monitoring video frames identifying target driving behavior is greater than a second set threshold.

[0090] In one possible implementation, the device further includes an alarm notification unit for:

[0091] If it is determined that the driver is engaging in the target driving behavior within the set time period, an alarm message will be issued.

[0092] The technical effects of any of the implementation methods in the second, third, fourth, or fifth aspects can be found in the technical effects of the implementation method in the first aspect, and will not be repeated here. Attached Figure Description

[0093] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0094] Figure 1 A schematic diagram of the structure of a locomotive provided in an embodiment of this application;

[0095] Figure 2 A flowchart illustrating a driver behavior recognition method provided in an embodiment of this application;

[0096] Figure 3 This is a schematic diagram of a depthwise separable convolution structure provided in an embodiment of this application;

[0097] Figure 4 A schematic diagram of an inverted residual structure provided in an embodiment of this application;

[0098] Figure 5 This is a schematic diagram of the structure of a multi-spectral channel attention model provided in an embodiment of this application;

[0099] Figure 6 A schematic diagram of a lightweight inverse residual structure combining a multispectral channel attention model provided in an embodiment of this application;

[0100] Figure 7A schematic diagram of a lightweight convolutional neural network provided in an embodiment of this application;

[0101] Figure 8 A flowchart illustrating another driver behavior recognition method provided in this application embodiment;

[0102] Figure 9 A schematic diagram illustrating the process of quantization porting of a network model provided in an embodiment of this application;

[0103] Figure 10 This is a schematic diagram of the structure of a driver behavior recognition device provided in an embodiment of this application;

[0104] Figure 11 This is a schematic diagram of another driver behavior recognition device provided in an embodiment of this application. Detailed Implementation

[0105] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0106] Figure 1 The diagram illustrates a structural schematic of a locomotive according to an embodiment of this application. Figure 1 As shown, the locomotive provided in this application embodiment includes a processor 103, a memory 102, and a video frame acquisition component 101.

[0107] The video frame acquisition component 101 is used to acquire a first monitoring video frame and a second monitoring video frame. The first monitoring video frame includes the driver's face area, and the second monitoring video frame includes the driver's face area and body area.

[0108] The memory 102 is used to store data information generated during locomotive operation and program code used by the processor 103 during operation, such as the program code of the driver behavior recognition method provided in the embodiments of this application, wherein the program code can be executed by the processor 103.

[0109] The processor 103 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 103 is used to call program code stored in memory 102 to implement the following processes: determining the region to be identified in the first monitoring video frame based on the positions of the face bounding box and facial key points in the first monitoring video frame, and determining a first recognition result based on the region to be identified; determining a second recognition result based on the positions of skeletal points in the second monitoring video frame; and determining whether the driver exhibits the target driving behavior within a set time period based on the first and second recognition results.

[0110] In one embodiment, this application provides a vehicle-mounted terminal, which may include a processor, a memory, and a video frame acquisition component. Furthermore, the vehicle-mounted terminal can be integrated into a vehicle for recognizing driver behavior.

[0111] In some embodiments, the vehicle terminal provided in this application may be an intelligent bus terminal, and the intelligent bus terminal may acquire, through a video frame acquisition component, a first monitoring video frame in a first monitoring video located directly in front of the bus driver and a second monitoring video frame in a second monitoring video located above and to the left of the bus driver's head.

[0112] This application embodiment does not limit the specific connection medium between the video frame acquisition component 101, the memory 102, and the processor 103. This application embodiment... Figure 1 In this diagram, the video frame acquisition component 101, memory 102, and processor 103 are connected via bus 104. The connections between other components are merely illustrative and not intended to be limiting. Bus 104 can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 1 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0113] As a core component of road transportation, drivers' poor driving behavior poses a significant threat to the lives and property of passengers and other road users. Poor driving behavior can include behaviors such as fatigue, yawning, making phone calls, smoking, looking around distractedly, eating, and drinking. Therefore, effectively identifying and monitoring drivers' poor driving behavior is crucial for protecting the lives and property of drivers and passengers. Furthermore, identifying and correcting poor driving behavior is of great importance in ensuring safe vehicle operation.

[0114] In related technologies, driver behavior recognition can be achieved using auxiliary modules such as infrared camera acquisition modules, thermal sensing data acquisition modules, and binocular 3D camera acquisition modules. To achieve high detection accuracy and system cost-effectiveness, analyzing video from in-vehicle surveillance cameras is the optimal choice. However, this approach also faces a series of practical difficulties. Taking the analysis of video from surveillance cameras inside a bus as an example, the above approach may encounter the following difficulties:

[0115] (1) There are many types of buses, and the backgrounds of different carriages vary greatly. Background pixels have a significant impact on driver behavior recognition.

[0116] (2) Buses operate for a long time, and the video footage from the surveillance cameras inside the buses is severely affected by changes in lighting.

[0117] (3) When the light intensity is low at night, the camera switches to infrared mode, and the video image of the monitoring camera inside the bus will change from a color image to a gray image.

[0118] Therefore, the identification results obtained by using the above method to identify poor driving behavior are greatly affected by external factors such as lighting, and are prone to false alarms, resulting in a low accuracy rate for identifying driving behavior.

[0119] To improve the accuracy of identifying driver behavior, this application provides a driver behavior recognition method. This method acquires a first monitoring video frame and a second monitoring video frame. The first monitoring video frame includes the driver's face area, and the second monitoring video frame includes the driver's face area and body area. Based on the positions of the face bounding box and facial key points in the first monitoring video frame, a region to be identified is determined. A first recognition result is determined based on this region. A second recognition result is determined based on the positions of skeletal points in the second monitoring video frame. Finally, based on the first and second recognition results, it is determined whether the driver exhibits the target driving behavior within a set time period.

[0120] In some embodiments, a flowchart of the driver behavior recognition method performed by the locomotive described above may be found. Figure 2 As shown, it includes the following steps:

[0121] Step S201: Obtain the first monitoring video frame and the second monitoring video frame.

[0122] The first monitoring video frame may include the driver's face area, and the second monitoring video frame may include the driver's face area and body area.

[0123] Step S202: Based on the positions of the face bounding box and facial key points in the first monitoring video frame, determine the region to be identified in the first monitoring video frame, and determine the first recognition result based on the region to be identified; and determine the second recognition result based on the positions of the skeletal points in the second monitoring video frame.

[0124] After acquiring the first and second surveillance video frames, the first surveillance video frame can be input into a Multi-Task Convolutional Neural Network (MTCNN). MTCNN then identifies the face bounding boxes and facial landmarks within the first surveillance video frame. Based on the positions of the face bounding boxes and facial landmarks within the first surveillance video frame, the region to be identified can be determined.

[0125] The facial key points included in the first monitoring video frame can be the center points of the left and right eyes, the tip of the nose, and the corners of the mouth. Each region to be identified corresponds to at least one driving behavior to be identified. For example, driving behaviors to be identified may include fatigue, yawning, making a phone call, smoking, looking around, eating, drinking, and not wearing a mask. The region to be identified for fatigue is the part of the face above the tip of the nose; the regions to be identified for yawning, smoking, not wearing a mask, eating, and drinking are the part of the face below the tip of the nose; and there are two regions to be identified for making a phone call, namely the area near the left and right ears.

[0126] In one embodiment, MTCNN mainly employs three cascaded networks: a proposal network (PNet) for quickly generating candidate windows, a refinement network (RNet) for high-precision candidate window filtering and selection, and an output network (ONet) for generating the final bounding boxes and facial key points.

[0127] After identifying the region to be identified in the first surveillance video frame, the Histogram of Oriented Gradient (HOG), Local Binary Pattern (LBP), and Haar features of the region can be extracted. These features are then fused to obtain the region features corresponding to the region to be identified. Finally, a machine learning model is used to identify the driver's driving behavior in the first surveillance video frame based on these region features.

[0128] In one embodiment, after extracting HOG features, LBP features, and Haar features respectively, the HOG features, LBP features, and Haar features can be sequentially concatenated to obtain region features. For example, assuming that the HOG features are 1*f1 dimensional features, the LBP features are 1*f2 dimensional features, and the Haar features are 1*f3 dimensional features, then sequentially concatenating the HOG features, LBP features, and Haar features can yield 1*(f1+f2+f3) dimensional region features.

[0129] In another embodiment, the machine learning model can be a Support Vector Machine (SVM) model. After identifying the region to be identified in the first monitoring video frame and obtaining the corresponding region features, the region features can be input into the SVM model. Based on the SVM model, it can be determined whether the driver exhibits any unsafe driving behavior in the first monitoring video frame. For example, the SVM model can be used to identify the region to be identified in the first monitoring video frame to determine whether there is a region corresponding to unsafe driving behaviors such as fatigue, yawning, making a phone call, smoking, looking around, eating, drinking water, and not wearing a mask. If such regions exist, it can be determined that the driver exhibits unsafe driving behavior in the first monitoring video frame.

[0130] If the number of first monitoring video frames with initial target driving behavior identified within a set time period is greater than a first set threshold, the first monitoring video frames with initial target driving behavior can be extracted, and a first identification result can be determined based on the region to be identified in the extracted first monitoring video frames with initial target driving behavior using a deep learning model.

[0131] Specifically, a deep learning model can be used to determine whether the driver exhibits the target driving behavior in the first surveillance video frame that has been extracted and shows the initial target driving behavior. If the driver exhibits the target driving behavior, the first identification result can be determined as: the driver exhibits the target driving behavior; if the driver does not exhibit the target driving behavior, the first identification result can be determined as: the driver does not exhibit the target driving behavior.

[0132] In one embodiment, the deep learning model can be a lightweight convolutional neural network composed of depthwise separable convolution, an inverted residual block, and a multispectral channel attention model. The structure of the depthwise separable convolution can be as follows: Figure 3As shown, one convolutional kernel is responsible for one channel, and each channel is convolved by only one kernel. Compared to conventional convolution operations, depthwise separable convolution has a lower number of parameters and lower computational cost. For the same image input, the number of parameters calculated is about 1 / 3 of that of conventional convolution, which can significantly reduce the floating-point computation of the network and reduce hardware resource consumption. The inverse residual structure can be as follows: Figure 4 As shown, the number of channels is first increased using a 1x1 convolutional layer, followed by a depthwise 3x3 spatial convolution, and finally a 1x1 convolutional layer is used to reduce the dimensionality. This approach maintains accuracy while reducing computational cost. The structure of the multispectral channel attention model can be as follows: Figure 5 As shown, the feature map of w*h*c is first preprocessed through a two-dimensional discrete cosine transform (2D-DCT) network layer to obtain a 1*1*c output representing the global information of each channel. Then, it passes through a fully connected (FC) layer, a rectified linear unit (ReLU) network layer, another FC layer, and finally a sigmoid function network layer to obtain a 1*1*c output representing the weight values ​​of each channel. The input feature map of w*h*c is then weighted using the 1*1*c output representing the weight values ​​of each channel, and the weighted w*h*c feature map is output.

[0133] By combining the inverse residual structure with a multispectral channel attention model, a lightweight inverse residual structure incorporating the multispectral channel attention model can be obtained, as shown below. Figure 6 As shown. The final lightweight convolutional neural network, composed of depthwise separable convolutions, inverted residual structures, and a multispectral channel attention model, can be implemented as follows: Figure 7 As shown.

[0134] Meanwhile, the skeleton point detection model can be used to detect skeleton points in the second monitoring video frame, determine the position of skeleton points in the second monitoring video frame, and identify the driver's driving behavior in the second monitoring video frame based on the position of skeleton points.

[0135] In one embodiment, the skeleton point detection model can be a high-resolution network (HRNet). Based on the aforementioned depthwise separable convolution, inverse residual structure, and multispectral channel attention model, the basic block structure and bottleneck structure in the feature extraction stage of HRNet can be improved by replacing the conventional convolution and residual structures and adding a channel attention model. This can improve the feature extraction capability of the skeleton point detection model while reducing the floating-point computation of the network.

[0136] Within a set time period, the second identification result can be determined based on whether the number of second monitoring video frames identifying target driving behavior is greater than a second set threshold.

[0137] Specifically, if the number of second monitoring video frames identifying target driving behavior within a set time period is not greater than a second set threshold, the second identification result can be determined as: the driver does not exhibit target driving behavior. If the number of second monitoring video frames identifying target driving behavior within a set time period is greater than the second set threshold, the second identification result can be determined as: the driver exhibits target driving behavior.

[0138] Step S203: Based on the first identification result and the second identification result, determine whether the driver has engaged in the target driving behavior within the set time period.

[0139] Based on the first and second identification results, it is determined whether the driver exhibited the target driving behavior within a set time period.

[0140] In one embodiment, when the target driving behavior includes driving behaviors such as fatigue or yawning, if the first identification result is determined by the deep learning model to be that the driver is engaging in the target driving behavior, then the driver can be determined to be engaging in the target driving behavior within a set time period directly based on the first identification result without the assistance of the second identification result.

[0141] When the target driving behavior includes driving behaviors such as looking left and right, if the second identification result is determined to be that the driver is engaging in the target driving behavior, then it is not necessary to combine the first identification result with the second identification result to determine that the driver is engaging in the target driving behavior within the set time.

[0142] When the target driving behavior includes driving behaviors such as making a phone call or smoking, if the first identification result is determined by the deep learning model as: the driver is engaging in the target driving behavior, then a second identification result is also needed to determine: the driver is engaging in the target driving behavior, before it can be determined that the driver is engaging in the target driving behavior within the set time period.

[0143] If it is determined that the driver is engaging in the target driving behavior within the set time period, an alarm message will be issued.

[0144] The driver behavior recognition method provided in this application first acquires a first monitoring video frame and a second monitoring video frame. The first monitoring video frame includes the driver's face area, and the second monitoring video frame includes the driver's face area and body area. Then, based on the positions of the face bounding box and facial key points in the first monitoring video frame, the region to be recognized in the first monitoring video frame is determined. A first recognition result is determined based on the region to be recognized, and a second recognition result is determined based on the positions of skeletal points in the second monitoring video frame. Finally, based on the first and second recognition results, it is determined whether the driver exhibits the target driving behavior within a set time period. By combining the first and second recognition results to jointly determine whether the driver exhibits the target driving behavior, the recognition accuracy of driver behavior recognition is improved.

[0145] In other embodiments, a flowchart of the driver behavior recognition method performed by the locomotive described above can be found. Figure 8 As shown, it includes the following steps:

[0146] Step S801: Obtain the first surveillance video.

[0147] The first surveillance video frame may include the driver's face. For example, the first surveillance video frame may be obtained from surveillance video directly in front of the bus driver.

[0148] Step S802: Perform face detection on the first surveillance video frame using MTCNN to determine the face bounding boxes and facial key points contained in the first surveillance video frame.

[0149] The first surveillance video frame can be input into MTCNN, and face detection can be performed on the first surveillance video frame based on MTCNN to determine the face bounding boxes and facial key points contained in the first surveillance video frame.

[0150] In one embodiment, MTCNN can be trained using a labeled face dataset. Specifically, a face dataset can be constructed first. In this embodiment, a bus vehicle face dataset is used as an example to illustrate the construction process.

[0151] First, surveillance videos directly in front of drivers on buses across all provinces and cities nationwide can be extracted according to four time periods: 5:00–7:00, 11:00–13:00, 17:00–19:00, and 21:00–23:00. The extracted videos must cover various weather conditions, including sunny, cloudy, rainy, snowy, and foggy days, as well as the four seasons: spring, summer, autumn, and winter. After extracting the surveillance videos, frames can be extracted every 5 seconds to obtain the corresponding video frames. These video frames can then be used as face sample images in a face dataset.

[0152] Secondly, the face dataset can be expanded using data augmentation methods. This involves transforming some of the face sample images included in the dataset by adjusting their brightness, contrast, chroma, saturation, and noise levels. This increases the variability of the input images, allowing for scaling, flipping, and rotation. Consequently, the MTCNN trained on this face dataset exhibits greater robustness to images acquired in different scenes.

[0153] Specifically, the brightness and contrast of an image can be adjusted using the following formula:

[0154] g(i,j)=a*f(i,j)+b

[0155] Where i and j represent the image pixels located in the i-th row and j-th column, respectively, f(i,j) represents the original image pixels, g(i,j) represents the adjusted image pixels, a is the gain used to control the image contrast and must satisfy a>0, and b is the bias used to control the image brightness.

[0156] When adjusting the chroma and saturation of an image, you can first normalize the RGB image values ​​to [0,1], then perform color space conversion to convert the RGB image to an HLS image, then use linear transformation to adjust the chroma and saturation components, and finally convert the adjusted image back to an RGB image.

[0157] In addition to transforming a subset of face sample images included in the face dataset, salt-and-pepper noise and Gaussian noise can be added to these images to increase the variability of the input images. Furthermore, x-axis flipping and affine transformations can be applied to the face sample images.

[0158] After constructing the face dataset, the annotation tool LableImg can be used to annotate the face sample images in the face dataset to obtain the bounding box of the face target and the annotations of five key facial points, including the center point of the left and right eyes, the tip of the nose, and the corners of the left and right mouths.

[0159] After obtaining a labeled face dataset, the MTCNN can be trained using this face dataset.

[0160] Step S803: Based on the positions of the face bounding box and facial key points in the first monitoring video frame, determine the region to be identified in the first monitoring video frame.

[0161] Among them, facial key points are determined based on face bounding boxes, and these key points can be the center points of the left and right eyes, the tip of the nose, and the corners of the left and right mouths. Each region to be identified corresponds to at least one driving behavior to be identified.

[0162] Step S804: Extract the HOG features, LBP features and Haar features of the region to be identified, and fuse the HOG features, LBP features and Haar features to obtain the region features.

[0163] Among them, HOG features are obtained based on the gradient orientation histogram of local regions of an image, while LBP features can be used to describe local texture features of an image and have advantages such as rotation invariance and gray-level invariance. Haar features can be divided into edge features, linear features, center features, and diagonal features.

[0164] Step S805: Using a machine learning model, identify the driver's driving behavior in the first monitoring video frame based on the regional features corresponding to the area to be identified.

[0165] Using a machine learning model, based on the regional characteristics corresponding to the area to be identified, it is determined whether the driver exhibits preliminary target driving behavior in the first monitoring video frame.

[0166] Step S806: If, within a set time period, the number of first monitoring video frames identifying preliminary target driving behavior is greater than a first set threshold, the first monitoring video frames identifying preliminary target driving behavior are extracted, and a first identification result is determined based on the region to be identified in the extracted first monitoring video frames identifying preliminary target driving behavior using a deep learning model.

[0167] When it is identified that the driver has preliminary target driving behavior in the first monitoring video frame, the first monitoring video frame can be saved. If the number of saved first monitoring video frames is greater than a first set threshold, the saved first monitoring video frames can be extracted, and the first recognition result can be determined by using a deep learning model based on the region to be identified in the extracted first monitoring video frames.

[0168] For example, the first monitoring video frame can be acquired in real time. After acquiring each frame, the region to be identified within that frame can be determined. Using an SVM model, the region features corresponding to this region are identified. If it is determined that the driver exhibits initial target driving behavior in that frame, the frame can be saved. Assuming a first threshold of 8 frames, if 10 frames are saved within 3 seconds (more than 8 frames), then 3 frames can be extracted. A lightweight convolutional neural network can then be used to identify the region to be identified within these 3 frames. If it is determined that the driver exhibits target driving behavior in each of these 3 frames, then the first identification result is: the driver exhibits target driving behavior.

[0169] In one embodiment, the machine learning model and the deep learning model can be trained using a behavior recognition dataset. This behavior recognition dataset can be a dataset consisting of face bounding boxes and facial key points within face sample images determined by MTCNN, and the identified regions to be recognized. Furthermore, driver behavior recognition can be performed using image binary classification, meaning each driving behavior has a corresponding set of positive and negative samples.

[0170] After obtaining the behavior recognition dataset, it can be expanded using data augmentation methods, similar to those used for the face dataset. Simultaneously, to ensure consistency in image feature dimensions, all sample images in the behavior recognition dataset need to be resized to a fixed size. Specifically, when training the SVM model, the sample images in the positive and negative driving behavior sample sets can be resized to 40*60 pixels, and when training the deep learning model, they can be resized to 224*224 pixels.

[0171] Step S807: Obtain the second surveillance video.

[0172] The second surveillance video frame may include the driver's face and body area. For example, the second surveillance video may be obtained from a surveillance video located above and to the left of the bus driver's head. Furthermore, steps S807 and S801 are performed simultaneously.

[0173] Step S808: Perform skeletal point detection on the second monitoring video frame using the skeletal point detection model to determine the positions of skeletal points in the second monitoring video frame.

[0174] After obtaining the second monitoring video frame, the second monitoring video frame can be input into the skeleton point detection model. Based on the skeleton point detection model, skeleton point detection is performed on the second monitoring video frame to determine the position of the skeleton points in the second monitoring video frame.

[0175] In one embodiment, the skeletal point detection model can be trained using a labeled skeletal point detection dataset. Specifically, taking a bus as an example, the skeletal point detection dataset can include images of bus drivers under various lighting conditions, weather conditions, and in all four seasons. Furthermore, the dataset is obtained from surveillance video taken from the upper left corner above the driver's head inside the bus. After obtaining the skeletal point detection dataset, it can be expanded using data augmentation methods, similar to those used for face datasets.

[0176] Then, the skeletal keypoints in the sample images included in the skeletal point dataset were labeled using the labeling tool LableImg. These skeletal keypoints can include the nose, left and right ears, left and right shoulders, left and right elbows, left and right wrists, and left and right hips.

[0177] After obtaining the labeled skeleton point dataset, the skeleton point detection model can be trained using the skeleton point dataset.

[0178] Step S809: Based on the skeletal point positions, identify the driver's driving behavior in the second monitoring video frame.

[0179] Based on the relative positions between the skeleton points, it can be determined whether the driver exhibits the target driving behavior in the second monitoring video frame.

[0180] Step S810: Within a set time period, determine the second identification result based on whether the number of second monitoring video frames identifying target driving behavior is greater than a second set threshold.

[0181] If it is determined that the driver exhibits the target driving behavior in the second monitoring video frame, the second monitoring video frame can be saved. Within a set time, when the number of saved second monitoring video frames exceeds a second set threshold, the second identification result can be determined as: the driver exhibits the target driving behavior.

[0182] For example, second monitoring video frames can be acquired in real time. After acquiring each frame, the positions of the skeletal points within that frame can be determined. Based on the relative positions of the skeletal points, it can be determined that the driver exhibits the target driving behavior within that frame, and the second monitoring video frame can then be saved. Assuming a second threshold of 12 frames, if 15 frames are saved within 5 seconds, which is greater than 12 frames, then the second identification result can be determined as: the driver exhibits the target driving behavior.

[0183] Step S811: Based on the first identification result and the second identification result, determine whether the driver has engaged in the target driving behavior within the set time period.

[0184] Based on the first and second identification results, it can be determined whether the driver exhibited the target driving behavior within the set time period.

[0185] Target driving behaviors can include fatigue, yawning, making phone calls, smoking, looking around, eating, drinking, and not wearing a mask. If, based on the first identification result, the driver exhibits driving behaviors such as fatigue, yawning, or not wearing a mask in the first monitoring video frame that do not require identification based on changes in the driver's body movements, then the driver can directly determine the presence of the target driving behavior without the assistance of the second identification result. For example, the first and second monitoring video frames can be acquired in real time. If an SVM model determines that the driver is yawning in the first monitoring video frame, that first monitoring video frame can be saved. If, within 3 seconds, 7 first monitoring video frames are saved (meaning the driver yawns in all 7 saved frames, exceeding the first set threshold of 5 frames), then 3 first monitoring video frames can be extracted from the 7 saved frames. A lightweight convolutional neural network can then be used to determine that the driver yawns in each of the extracted 3 frames, thus ultimately confirming that the driver yawned within 3 seconds.

[0186] When the second recognition result determines that the driver exhibits driving behavior such as looking left and right in the second monitoring video frame—a behavior that does not require consideration of changes in the image features of the area to be identified at the time the driving behavior occurs—the driver can directly determine the presence of the target driving behavior without combining it with the first recognition result. For example, the first and second monitoring video frames can be acquired in real time. If the positions of the skeleton points in the second monitoring video frame are determined using a skeleton point detection model, and the driver is found to be looking left and right in the second monitoring video frame based on the relative positions between the skeleton points, the second monitoring video frame can be saved. If nine second monitoring video frames are saved within 4 seconds, meaning that the driver exhibits looking left and right behavior in all nine saved frames, exceeding the second set threshold of six frames, then it can be finally determined that the driver exhibited looking left and right behavior within 4 seconds.

[0187] When the first recognition result determines that the driver is engaging in driving behavior such as making a phone call, smoking, or eating in the first monitoring video frame—a behavior requiring both changes in the driver's body language and changes in the image features of the area to be identified at the time of the driving behavior—the second recognition result is also required. If the second recognition result indicates that the driver is engaging in the target driving behavior in the second monitoring video frame, then the driver is confirmed to be engaging in the target driving behavior. For example, the first and second monitoring video frames can be acquired in real time. If the SVM model determines that the driver is making a phone call in the first monitoring video frame, that first monitoring video frame can be saved. If 18 first monitoring video frames are saved within 8 seconds (meaning the driver is making a phone call in all 18 saved frames, exceeding the first set threshold of 15 frames), then 5 first monitoring video frames can be extracted from the 18 saved frames. A lightweight convolutional neural network can then be used to determine that the driver is making a phone call in each of the extracted 5 frames. Simultaneously, the skeletal point detection model determines the position of the skeletal points in the second monitoring video frame. Based on the relative positions between the skeletal points, it is also determined that the driver made a phone call in the second monitoring video frame. The second monitoring video frame is then saved. Within 8 seconds, 15 second monitoring video frames are saved, meaning that the driver made a phone call in all 15 saved second monitoring video frames. This is greater than the second set threshold of 10 frames, so it can be finally determined that the driver made a phone call within 8 seconds.

[0188] Step S812: If it is determined that the driver has engaged in the target driving behavior within the set time period, an alarm message is issued.

[0189] When it is determined that a driver is engaging in the target driving behavior within a set time period, an alarm message can be issued, and the driver's target driving behavior can be uploaded to the monitoring platform to remind relevant personnel that the driver is engaging in bad driving behavior and to correct the bad driving behavior.

[0190] The driver behavior recognition method provided in this application analyzes surveillance videos installed in different locations inside the vehicle in real time based on machine learning and deep learning models. It can identify undesirable driving behaviors such as driver fatigue, yawning, making phone calls, smoking, not wearing a mask, looking around, eating, and drinking. Compared to related technologies, the driver behavior recognition method provided in this application combines changes in image features of the area to be recognized and changes in the driver's body movements when the driving behavior occurs, comprehensively recognizing the driver's behavior. This improves the accuracy of driving behavior recognition, reduces false alarms, thereby ensuring vehicle operational safety and correcting undesirable driving behaviors.

[0191] In one embodiment, the network model can be quantized using Int8. A flowchart illustrating the specific process of quantizing and porting the network model can be shown below. Figure 9 As shown, this application demonstrates that a bus driver behavior recognition network model can be ported to an embedded device based on a neural network feedforward computing framework (ncnn). First, PyTorch models of MTCNN, SVM, lightweight neural networks, and skeletal point detection models can be trained, converted into ONNX models, and then converted back into ncnn models using the ncnn feedforward computing framework. By performing Int8 quantization on the network model, its size can be reduced to 30% of its original size, reducing hardware resource consumption while maintaining essentially the same accuracy in recognizing driver behavior, thus improving the speed of driver behavior recognition.

[0192] and Figure 2 The driver behavior recognition method shown is based on the same inventive concept. This application also provides a driver behavior recognition device, which can be installed in a locomotive. Since this device corresponds to the driver behavior recognition method of this application, and the principle by which this device solves the problem is similar to that of the method, the implementation of this device can refer to the implementation of the above method; repeated details will not be elaborated further.

[0193] Figure 10 This application provides a schematic diagram of the structure of a driver behavior recognition device according to an embodiment of the present application. Figure 10 As shown, the driver behavior recognition device includes a video frame acquisition unit 1001, a recognition result determination unit 1002, and a driving behavior determination unit 1003.

[0194] The video frame acquisition unit 1001 is used to acquire a first monitoring video frame and a second monitoring video frame; the first monitoring video frame includes the driver's face area, and the second monitoring video frame includes the driver's face area and body area.

[0195] The recognition result determination unit 1002 is used to determine the region to be recognized in the first monitoring video frame based on the position of the face frame and facial key points in the first monitoring video frame, and to determine the first recognition result based on the region to be recognized; and to determine the second recognition result based on the position of the skeleton points in the second monitoring video frame.

[0196] The driving behavior determination unit 1003 is used to determine whether the driver has the target driving behavior within a set time period based on the first identification result and the second identification result.

[0197] In one possible implementation, the identification result determination unit 1002 is specifically used for:

[0198] The first surveillance video frame is input into a multi-task convolutional neural network, and the face bounding box and facial key points contained in the first surveillance video frame are determined through the multi-task convolutional neural network.

[0199] Based on the positions of the face bounding box and facial key points in the first monitoring video frame, the region to be identified in the first monitoring video frame is determined.

[0200] In one possible implementation, the identification result determination unit 1002 is further configured to:

[0201] Feature extraction is performed on the region to be identified in the first monitoring video frame to obtain the region features corresponding to the region to be identified.

[0202] Using a machine learning model, the driver's driving behavior in the first monitoring video frame is identified based on the regional features corresponding to the area to be identified.

[0203] If the number of first monitoring video frames identifying preliminary target driving behavior exceeds a first set threshold within a set time period, the first monitoring video frames with preliminary target driving behavior are extracted, and a first identification result is determined based on the region to be identified in the extracted first monitoring video frames with preliminary target driving behavior using a deep learning model.

[0204] In one possible implementation, the identification result determination unit 1002 is further configured to:

[0205] Extract the directional gradient histogram features, local binary pattern features, and Haar features of the region to be identified, respectively.

[0206] The directional gradient histogram features, local binary pattern features, and Haar features are fused to obtain the region features corresponding to the region to be identified.

[0207] In one possible implementation, the identification result determination unit 1002 is further configured to:

[0208] The skeleton point detection model is used to detect skeleton points in the second monitoring video frame to determine the location of skeleton points in the second monitoring video frame.

[0209] Based on the location of the skeleton points, identify the driver's driving behavior in the second monitoring video frame;

[0210] Within a set time period, the second identification result is determined based on whether the number of second monitoring video frames identifying target driving behavior is greater than a second set threshold.

[0211] In one possible implementation, such as Figure 11 As shown, the above-mentioned device may further include an alarm notification unit 1101, used for:

[0212] If it is determined that the driver is engaging in the target driving behavior within the set time period, an alarm message will be issued.

[0213] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0214] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0215] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0216] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0217] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A locomotive, characterized in that, include: A video frame acquisition component is used to acquire a first monitoring video frame and a second monitoring video frame; the first monitoring video frame includes the driver's face area and comes from a first camera deployed directly in front of the driver, and the second monitoring video frame includes the driver's face area and body area and comes from a second camera deployed to the upper left of the driver; A memory for storing program code and data generated during the operation of the locomotive; A processor is configured to execute the program code to perform the following process: Based on the positions of the face bounding box and facial key points in the first monitoring video frame, the region to be identified in the first monitoring video frame is determined; and the driver's driving behavior in the first monitoring video frame is identified by a machine learning model according to the regional features corresponding to the region to be identified. If the number of first monitoring video frames with preliminary target driving behavior identified within a set time is greater than a first set threshold, the first monitoring video frames with preliminary target driving behavior are extracted, and a first identification result is determined based on the region to be identified in the extracted first monitoring video frames with preliminary target driving behavior using a deep learning model. The skeleton point detection model is used to detect skeleton points in the second monitoring video frame to determine the position of skeleton points in the second monitoring video frame; based on the position of the skeleton points, the driving behavior of the driver in the second monitoring video frame is identified. Within the set time period, the second identification result is determined based on whether the number of second monitoring video frames identifying target driving behavior is greater than a second set threshold. Based on the first identification result, the second identification result, and the type of target driving behavior, determine whether the driver exhibits target driving behavior within the set time period; Specifically, for the first type of target driving behavior, it is determined directly based on the first identification result. The first type of target driving behavior is used to characterize driving behaviors that do not require identification in conjunction with changes in the driver's body movements. For the second type of target driving behavior, it is determined directly based on the second recognition result. The second type of target driving behavior is used to characterize driving behaviors that do not need to be identified by combining the changes in the image features of the region to be identified when the driving behavior occurs. For the third type of target driving behavior, it is determined by combining the first recognition result and the second recognition result. The third type of target driving behavior is used to characterize driving behavior that needs to be identified by combining the changes in the driver's body movements and the changes in the image features of the area to be identified when the driving behavior occurs.

2. The locomotive according to claim 1, characterized in that, The processor is specifically used for: The first monitoring video frame is input into a multi-task convolutional neural network, and the face bounding box and facial key points contained in the first monitoring video frame are determined through the multi-task convolutional neural network. Based on the positions of the face frame and the facial key points in the first monitoring video frame, the region to be identified in the first monitoring video frame is determined.

3. The locomotive according to claim 1, characterized in that, The processor is also used for: Feature extraction is performed on the region to be identified in the first monitoring video frame to obtain the region features corresponding to the region to be identified.

4. The locomotive according to claim 3, characterized in that, The processor is also used for: Extract the directional gradient histogram features, local binary pattern features, and Haar features of the region to be identified, respectively. The directional gradient histogram features, the local binary pattern features, and the Haar features are fused to obtain the region features corresponding to the region to be identified.

5. The locomotive according to any one of claims 1 to 4, characterized in that, The processor is also used for: If it is determined that the driver is engaging in the target driving behavior within the set time period, an alarm message will be issued.

6. A driver behavior recognition method, characterized in that, The method includes: Acquire a first monitoring video frame and a second monitoring video frame; the first monitoring video frame includes the driver's face area and comes from a first camera deployed directly in front of the driver, and the second monitoring video frame includes the driver's face area and body area and comes from a second camera deployed to the upper left of the driver; Based on the positions of the face bounding box and facial key points in the first monitoring video frame, the region to be identified in the first monitoring video frame is determined; using a machine learning model, the driver's driving behavior in the first monitoring video frame is identified according to the regional features corresponding to the region to be identified; if the number of first monitoring video frames with preliminary target driving behavior identified within a set time is greater than a first set threshold, the first monitoring video frames with preliminary target driving behavior are extracted, and a first identification result is determined based on the region to be identified in the extracted first monitoring video frames with preliminary target driving behavior using a deep learning model; The skeleton point detection model is used to detect skeleton points in the second monitoring video frame to determine the position of skeleton points in the second monitoring video frame; based on the position of the skeleton points, the driving behavior of the driver in the second monitoring video frame is identified; within the set time, the second identification result is determined based on whether the number of second monitoring video frames in which the target driving behavior is identified is greater than a second set number threshold. Based on the first identification result, the second identification result, and the type of target driving behavior, determine whether the driver exhibits target driving behavior within the set time period; Specifically, the first type of target driving behavior is determined based on the first identification result. The first type of target driving behavior is used to characterize driving behaviors that do not require identification in conjunction with changes in the driver's body movements. For the second type of target driving behavior, it is determined based on the second recognition result. The second type of target driving behavior is used to characterize driving behaviors that do not need to be identified by combining the changes in the image features of the region to be identified when the driving behavior occurs. For the third type of target driving behavior, it is determined by combining the first recognition result and the second recognition result. The third type of target driving behavior is used to characterize driving behavior that needs to be identified by combining the changes in the driver's body movements and the changes in the image features of the area to be identified when the driving behavior occurs.

7. The method according to claim 6, characterized in that, The step of determining the region to be identified in the first monitoring video frame based on the positions of the face bounding box and facial key points in the first monitoring video frame includes: The first monitoring video frame is input into a multi-task convolutional neural network, and the face bounding box and facial key points contained in the first monitoring video frame are determined through the multi-task convolutional neural network. Based on the positions of the face frame and the facial key points in the first monitoring video frame, the region to be identified in the first monitoring video frame is determined.

8. The method according to claim 6, characterized in that, The region features corresponding to the region to be identified are determined in the following way: Feature extraction is performed on the region to be identified in the first monitoring video frame to obtain the region features corresponding to the region to be identified.

9. A vehicle-mounted terminal, characterized in that, include: A video frame acquisition component is used to acquire a first monitoring video frame and a second monitoring video frame; the first monitoring video frame includes the driver's face area and comes from a first camera deployed directly in front of the driver, and the second monitoring video frame includes the driver's face area and body area and comes from a second camera deployed to the upper left of the driver; The memory is used to store program code and data generated during the operation of the vehicle-mounted terminal. A processor is configured to execute the program code to perform the following process: Based on the positions of the face bounding box and facial key points in the first monitoring video frame, the region to be identified in the first monitoring video frame is determined; and the driver's driving behavior in the first monitoring video frame is identified by a machine learning model according to the regional features corresponding to the region to be identified. If the number of first monitoring video frames with preliminary target driving behavior identified within a set time is greater than a first set threshold, the first monitoring video frames with preliminary target driving behavior are extracted, and a first identification result is determined based on the region to be identified in the extracted first monitoring video frames with preliminary target driving behavior using a deep learning model. The skeleton point detection model is used to detect skeleton points in the second monitoring video frame to determine the position of skeleton points in the second monitoring video frame; based on the position of the skeleton points, the driving behavior of the driver in the second monitoring video frame is identified. Within the set time period, the second identification result is determined based on whether the number of second monitoring video frames identifying target driving behavior is greater than a second set threshold. Based on the first identification result, the second identification result, and the type of target driving behavior, determine whether the driver exhibits target driving behavior within the set time period; Specifically, the first type of target driving behavior is determined based on the first identification result. The first type of target driving behavior is used to characterize driving behaviors that do not require identification in conjunction with changes in the driver's body movements. For the second type of target driving behavior, it is determined based on the second recognition result. The second type of target driving behavior is used to characterize driving behaviors that do not need to be identified by combining the changes in the image features of the region to be identified when the driving behavior occurs. For the third type of target driving behavior, it is determined by combining the first recognition result and the second recognition result. The third type of target driving behavior is used to characterize driving behavior that needs to be identified by combining the changes in the driver's body movements and the changes in the image features of the area to be identified when the driving behavior occurs.

Images