Driver motion recognition system and method

Abstract

This invention provides a driver action recognition system and method. The system includes: an image acquisition module for acquiring in-vehicle images, which serve as input images for the action recognition module; the in-vehicle images include images of the overall in-vehicle space; the action recognition module for: extracting a feature matrix from the input image through a convolution submodule; performing a convolution process on the feature matrix to obtain key features; inputting the key features into two channels; in the first channel, processing the key features sequentially through a max pooling layer and a first detection module; and in the second channel, processing the key features through a second detection module. The first detection module detects the driver's behavior, and the second detection module detects the behavior of other occupants in the vehicle; and a warning module for issuing an alarm when the action recognition module detects one or more of the driver's behaviors, including smoking, drinking water, wearing sunglasses, making a phone call, looking away from the road, closing eyes, or taking both hands off the steering wheel.

Description

Technical Field

[0001] The embodiments of the present invention generally relate to the field of intelligent driving technology, and more specifically, to a driver action recognition system and method. Background Technology

[0002] Driving safety is one of the most important issues in the automotive industry. Guiding drivers to drive correctly and safely is the mission of car manufacturers. Being able to identify various driving behaviors in a timely manner, especially dangerous driving behaviors (such as taking both hands off the steering wheel, looking away from the road, driving while fatigued, and abnormal emotions), can help remind drivers to operate safely in a timely manner and avoid accidents.

[0003] Currently, driver behavior detection functions applied to vehicles mainly judge the driver's state based on single conditions such as driving duration and vehicle speed. This has drawbacks such as limited scenarios, limited conditions, inability to recognize complex scenarios, and lack of support for expansion. Meanwhile, car owners are placing increasing demands on the safety performance and intelligence of their vehicles.

[0004] With the development of vehicle networking technology and the rapid changes in the automotive industry, it is imperative to understand users' driving behavior from multiple dimensions, conditions, and scenarios in order to provide users with more accurate services.

[0005] Current publicly available solutions for driver action recognition suffer from drawbacks such as limited application scenarios, simplistic detection methods, and unreliable accuracy. First, the limited application scenarios primarily refer to the focus on simple scenarios like driver fatigue, steering wheel detachment, and dangerous driving behaviors, a limitation stemming from the simplistic detection methods. Second, the limited detection methods rely mainly on vehicle data, such as speed, driving duration, and location information. For example, detecting whether a user is holding the steering wheel typically involves reading data from a pressure sensor mounted on the steering wheel; determining driver fatigue is done by calculating continuous driving time; and abnormal driving behavior is primarily assessed based on vehicle speed. This lack of data makes these solutions indirect in their assessments of user actions. Finally, due to the limited and unreliable data (e.g., collected speed data differs significantly from actual speed), the assessment of the user's state is also inaccurate. Summary of the Invention

[0006] To address the aforementioned problems in the prior art, embodiments of the present invention provide a driver action recognition system, comprising: an image acquisition module, an action recognition module, and an early warning module. The image acquisition module acquires in-vehicle images as input images to the action recognition module, wherein the in-vehicle images include images of the overall vehicle interior space. The action recognition module extracts a feature matrix from the input image using a convolution submodule, performs a convolution process on the feature matrix to obtain key features, and inputs these key features into two channels for processing. In the first channel, the key features are processed sequentially through a max pooling layer and a first detection module; in the second channel, the key features are processed by a second detection module. The first detection module detects the driver's behavior, and the second detection module detects the behavior of other occupants in the vehicle. The early warning module issues an alarm when the action recognition module detects one or more of the driver's behaviors, including smoking, drinking water, wearing sunglasses, making a phone call, looking away from the vehicle, closing their eyes, or taking their hands off the steering wheel.

[0007] In some implementations, the first detection module is further configured to: process the features to be detected input to the first detection module in two processes; in the first process, the features to be detected are passed through a 3x3 convolutional layer and then processed by a rectified linear unit layer, which processes the linearly convolved features of the features to be detected to fit non-linear features; in the second process, the features to be detected are processed by a wide-area information processing module, which includes four convolutional layers and four rectified linear unit layers, and the number of channels to be detected is halved; the features to be detected after the two processes are fused by a fusion module to obtain fused features; and the fused features are input into two 1x1 convolutional layers to obtain action classification and the driver's position in the image.

[0008] In some implementations, the convolutional submodule includes 17 processing layers, namely 13 convolutional layers with a kernel size of 3*3 and 4 max pooling layers with a stride of 2, wherein the 4 max pooling layers are located at the 3rd, 6th, 10th and 14th layers of the 17 processing layers.

[0009] In some embodiments, the image acquisition module is further configured to compress the in-vehicle image through a compression submodule and use the compressed image as the input image of the action recognition module, wherein the compression submodule includes a three-layer neural network.

[0010] In some implementations, the triggering conditions for the warning module to issue an alarm include the duration, frequency, and type of one or more actions such as smoking, drinking water, wearing sunglasses, making a phone call, looking away from the road, closing one's eyes, and taking both hands off the steering wheel.

[0011] In some implementations, the warning module is also used to issue one or more of the following warning information based on the action recognition result: the duration of the current action; the type of the current action; the danger level of the current action; and the current vehicle status, including vehicle speed, window status, mileage, and travel time.

[0012] In some embodiments, the system further includes an image preprocessing module for: detecting key points in the in-vehicle image, the key points including one or more of the driver's eyes, ears, mouth, nose, mouth, shoulders, and hands; and cropping the input image according to the position of the key points, wherein the cropping method is as follows: with the midpoint of the shoulders as the center, extending horizontally outward to the image boundary, and taking twice the shortest extension distance as the image width; extending vertically outward to the image boundary, and taking twice the shortest extension distance as the image height.

[0013] In some implementations, the training process of the first detection module includes: using an initialized neural network to recognize sample images, and locating the recognition loss by comparing the recognition result with a predefined annotation error; adjusting network parameters according to the loss to reduce the recognition error in the next iteration; and adjusting the error to a stable minimum value, thereby obtaining the first detection module that can be used for action recognition.

[0014] In some implementations, the warning module is also used to classify different types of driving behaviors into levels and to give different reminders to users according to different levels of danger. For actions at the "dangerous" level, the module uses voice reminders to remind the user, and for actions at the "warning" level, the module uses warning text reminders to remind the user.

[0015] The present invention also proposes a driver action recognition method, the method comprising: acquiring an in-vehicle image as an input image, wherein the in-vehicle image includes an image of the overall in-vehicle space; extracting a feature matrix of the input image through a convolutional submodule; performing a convolutional process on the feature matrix to obtain key features; inputting the key features into two channels for processing; in the first channel, processing the key features sequentially through a max pooling layer and a first detection module; and in the second channel, processing the key features through a second detection module. The first detection module is used to detect the driver's behavior, and the second detection module is used to detect the behavior of other occupants in the vehicle; and issuing an alarm when the driver's behavior is detected to include one or more of the following: smoking, drinking water, wearing sunglasses, making a phone call, looking away from the vehicle, closing their eyes, or taking their hands off the steering wheel.

[0016] This invention provides a driver action recognition scheme, including dangerous driving behaviors such as leaving the steering wheel, deviating from the line of sight, fatigued driving, abnormal emotions, and answering the phone, thereby promptly judging the user's behavior and reminding the user to drive safely. This invention has advantages such as fast recognition speed, high accuracy, high aggregation, and scalability to various scenarios.

[0017] Compared with solutions currently known in the field, the advantages of this invention are as follows:

[0018] It has a wide detection range and can detect all parts of the body above the driver's knees;

[0019] The detection speed is fast; it can detect a user's action if they hold it for more than 2 seconds.

[0020] Offline operation: The model has been specially optimized to run offline and maintain service throughout the entire process;

[0021] Multiple types of identification can be used to identify dangerous driving behaviors, including smoking, drinking water, driving while fatigued, deviating from the line of sight, and wearing sunglasses. Attached Figure Description

[0022] The above and other objects, features, and advantages of embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example and not limitation, wherein:

[0023] Figure 1 A block diagram of a driver action recognition system according to an embodiment of the present invention is shown;

[0024] Figure 2 A schematic diagram of the processing procedure of a driver action recognition system according to an embodiment of the present invention is shown;

[0025] Figure 3A system operation flowchart of a driver action recognition system according to an embodiment of the present invention is shown;

[0026] Figure 4 A schematic diagram illustrating the acquisition angle and field of view of an in-vehicle camera in the prior art is shown.

[0027] Figure 5 A schematic diagram illustrating the camera angle and field of view according to an embodiment of the present invention is shown.

[0028] Figure 6 A network structure diagram of an action recognition module according to an embodiment of the present invention is shown;

[0029] Figure 7 A comparison diagram is shown between the actions identified according to embodiments of the present invention and those of the prior art;

[0030] Figure 8 A network structure diagram of a convolutional submodule according to an embodiment of the present invention is shown.

[0031] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0032] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way.

[0033] In one aspect, embodiments of the present invention provide a driver action recognition system. (See reference...) Figures 1-3 ,in Figure 1 A block diagram of a driver action recognition system according to an embodiment of the present invention is shown. Figure 2 A schematic diagram of the processing procedure of a driver action recognition system according to an embodiment of the present invention is shown. Figure 3 A system operation flowchart of a driver action recognition system according to an embodiment of the present invention is shown. Figure 1 As shown, the driver action recognition system includes an image acquisition module, an action recognition module, and a warning module.

[0034] The image acquisition module is used to capture images of the vehicle interior, which serve as input images for the motion recognition module. These images include the overall interior space of the vehicle. The image acquisition module can be a camera. It can be mounted on the roof directly in front of the driver and is responsible for transmitting real-time images of the driver back to the motion recognition module.

[0035] refer to Figures 4-5 ,in, Figure 4The diagram illustrates the acquisition angle and field of view of an in-vehicle camera in the prior art, with the field of view including only the image of the driver; Figure 5 A schematic diagram of the camera angle and field of view according to an embodiment of the present invention is shown, the field of view including an image of the entire interior space of the vehicle.

[0036] The image acquisition device's acquisition range is optimized for various vehicle types (MPVs, SUVs, sedans), improving the range of image data acquisition. Firstly, verification showed that to ensure the image acquisition device's field of view, its installation position was adjusted to be slightly off-center from the driver's side of the rearview mirror. This allows the image acquisition depth to cover the rear seats of the vehicle, maximizing the acquisition of in-vehicle information. Secondly, while common in-vehicle motion recognition solutions on the market have image acquisition components with a field of view of 45° or 90°, this invention uses a device with a 120° field of view, covering most areas, including the passenger side.

[0037] Compared to commonly used image acquisition devices on the market, this device expands the image data acquisition range from primarily the driver's seat to the entire vehicle interior. Data acquisition can cover the area above the driver's waist and the rear seat area, providing the image processing unit with more data features. Therefore, the image processing unit can identify the number of people and their status within the vehicle, enabling it to handle more scenarios, such as recognizing the number of people, their gender, and the travel scenario.

[0038] As one embodiment of the present invention, such as Figures 2-3 As shown, the image acquisition module also compresses the in-vehicle images through a compression submodule, and uses the compressed images as input images for the action recognition module. The compression submodule includes a three-layer neural network. This three-layer neural network extracts video information and generates compressed features. The processed data retains important information from the original image and reduces its size, facilitating subsequent recognition. For example, in the three-layer neural network, the first layer is a convolutional (Conv) + rectified linear unit (ReLU) layer, the second layer is a convolutional (Conv) + batch normalization (BN) + rectified linear unit (ReLU) layer, and the third layer is a convolutional (Conv) layer.

[0039] In driving scenarios, the efficiency of driver action recognition determines whether the system can alert the driver to dangerous driving behaviors more quickly. Generally, due to the large size of image data, the hardware processing power of the vehicle's own system cannot quickly process large amounts of video stream data. Therefore, reducing the data storage capacity can more quickly recognize the driver's actions for the system to make further judgments. Currently, the pre-processing of images in in-vehicle image recognition systems on the market mainly involves two aspects: first, reducing the frame rate of the video stream to reduce the processing pressure on the device; and second, compressing each frame of the image to improve the system's processing capabilities.

[0040] Based on the two steps above, this invention first compresses the image data using a video compression module before performing feature recognition. The compressed features are then directly input into the action recognition module for recognition.

[0041] Image compression technology can improve image processing capabilities and image recognition efficiency, thereby increasing the speed and real-time performance of driver action recognition.

[0042] As one embodiment of the present invention, the system may further include an image preprocessing module for: detecting key points in the in-vehicle image, the key points including one or more of the driver's eyes, ears, mouth, nose, mouth, shoulders, and hands; cropping the input image according to the position of the key points, the cropping method being to extend horizontally outward from the midpoint of the shoulders to the image boundary, and take twice the shortest extension distance as the image width, and extend vertically outward to the image boundary, and take twice the shortest extension distance as the image height.

[0043] The action recognition module is used to: extract the feature matrix of the input image through a convolutional submodule; perform a single convolutional process on the feature matrix to obtain key features; and input these key features into two channels for processing. In the first channel, the key features are processed sequentially through a max pooling layer and a first detection module. In the second channel, the key features are processed by a second detection module. The first detection module detects the driver's behavior, and the second detection module detects the behavior of other occupants in the vehicle. When the driver exhibits dangerous driving behavior, the action recognition module identifies the driving behavior based on human characteristics and notifies the warning module of the recognition result.

[0044] refer to Figure 6 The diagram illustrates the network structure of an action recognition module according to an embodiment of the present invention. For example, the input image of the system is a 1200*800 image. After the image enters the network, the network first uses a convolutional module to extract key information (features) from the image. After processing by the above module, a 512-channel feature matrix is ​​obtained. The features after performing a convolutional process on this feature matrix are processed by two modules: a first detection module (M1 module) and a second detection module (M2 module). The function of the M1 module is to detect the driver's actions, and the M2 module is used to detect other people in the vehicle.

[0045] Since the driver is always at the forefront of the image and occupies the largest area, a max-pooling layer with a stride of 2 is used before the M1 module to process the features. The purpose of this step is to increase the network's receptive field, making it easier to identify larger targets (humans). Meanwhile, since other people in the car are located at the rear and have a smaller display area in the image, no additional processing is needed when using the M2 module to identify people in the car.

[0046] As one embodiment of the present invention, such as Figure 2 As shown, the training process of the first detection module includes: using an initialized neural network to recognize sample images, and determining the recognition loss based on the error between the recognition result and the predefined annotation; adjusting the network parameters according to the loss to reduce the recognition error in the next iteration; and adjusting the error until it reaches a stable minimum value, thus obtaining the first detection module that can be used for action recognition. The neural network is trained using seven types of labeled driver action data, enabling it to recognize the seven preset actions. When the driver performs any of these seven actions during driving, the system automatically recognizes them and informs the system's warning module of the recognition result.

[0047] The warning module is used to issue an alarm when the action recognition module detects one or more of the driver's behaviors, including smoking, drinking water, wearing sunglasses, making a phone call, looking away from the road, closing the eyes, or taking both hands off the steering wheel.

[0048] The warning module generates corresponding warning information based on the recognition results of the action recognition module. The warning information includes three parts: action type (the seven actions mentioned above), duration (the duration of the action, given by the action recognition module), and warning level (the danger level of the corresponding action, preset by the system). The warning module provides appropriate warnings based on the danger level of the driving behavior, prompting the driver to drive safely.

[0049] In one embodiment of the present invention, the first detection module is further configured to: process the features to be detected input to the first detection module in two processes; in the first process, the features to be detected are processed by passing through a 3x3 convolutional layer and then by a rectified linear unit layer, which processes the linearly convolved features in the features to be detected to fit nonlinear features; in the second process, the features to be detected are processed by a wide-area information processing module, which includes four convolutional layers and four rectified linear unit layers, and the number of channels to be detected is halved; the features to be detected after the two processes are fused by a fusion module to obtain fused features; the fused features are then input into two 1x1 convolutional layers to obtain action classification and the driver's position in the image.

[0050] As an example, the feature matrix is ​​processed in two steps after being input into the first detection module. The first step involves processing the features through a 3x3 convolutional layer followed by a ReLU (Rectified Linear Unit) layer. This layer primarily processes the linear convolutional features to fit non-linear features. The second step uses a wide-area information processing module to process the features. This module contains four convolutional and ReLU layers and halves the number of channels in the features. This layer processes broader information in the features to improve the accuracy of driver feature recognition. After these two processing steps, a feature fusion module fuses the outputs of the two steps. This step completes the detection of key driver features in the image. The next step is to adjust the image based on these key features.

[0051] refer to Figure 7 The diagram shows a comparison between actions recognized according to an embodiment of the present invention and those of the prior art. The action recognition module can recognize seven actions, including smoking, drinking water, wearing sunglasses, making a phone call, looking away, closing eyes, and taking both hands off the steering wheel, and can also identify the duration of the corresponding actions.

[0052] Currently, common driver action recognition modules only support the recognition of single actions, such as fatigued driving, due to limitations in data and insufficient device processing power. The embodiments of this invention reduce the requirements for device processing power by using techniques such as data compression and feature extraction; furthermore, by training on different types of driving behaviors, it can support the recognition of multiple driving actions. This significantly improves data utilization and the efficiency of the recognition module.

[0053] As one embodiment of the present invention, reference is made to... Figure 8 This illustrates a network structure diagram of a convolutional submodule according to an embodiment of the present invention. Figure 8 As shown, the convolutional submodule includes 17 processing layers, namely 13 convolutional layers with a kernel size of 3*3 and 4 max pooling layers with a stride of 2. The 4 max pooling layers are located at layers 3, 6, 10 and 14 of the 17 processing layers.

[0054] This module has a total of 17 layers, consisting of 13 convolutional layers with a kernel size of 3*3 and 4 max pooling layers with a stride of 2. The function of the convolutional layers is to extract key information from the image. The input of the first convolutional layer is the original image information, and the input of the subsequent convolutional layers is the result calculated by the previous convolutional layer. Because there are thirteen convolutional layers, it can extract information in thirteen dimensions. The higher the dimension, the higher the degree of feature abstraction.

[0055] There are four pooling layers. Their function is to aggregate the features extracted by the convolutional layers, making the key information more apparent and thus facilitating processing by the subsequent feature recognition module. The final output is a 512-channel feature matrix.

[0056] As one embodiment of the present invention, the triggering conditions for the warning module to issue an alarm include the duration, frequency and type of one or more of the following actions: smoking, drinking water, wearing sunglasses, making a phone call, looking away from the camera, closing one's eyes, and taking one's hands off the steering wheel.

[0057] As one embodiment of the present invention, the warning module is also used to issue one or more of the following warning information based on the action recognition result: the duration of the current action; the type of the current action; the danger level of the current action; and the current vehicle status, including vehicle speed, window status, mileage, and travel time.

[0058] As one embodiment of the present invention, the warning module is also used to classify different types of driving behaviors into levels and to give different reminders to users according to different danger levels. For actions of the "danger" level, the module uses voice reminders to remind users, and for actions of the "warning" level, the module uses warning text reminders to remind users.

[0059] The following text combines Figure 2 and Figure 3 The system's operation process is described using examples.

[0060] A camera is installed at the front of the vehicle interior. During driving, the real-time video stream captured by the camera is transmitted back to the processing module. The processing module first parses the video stream and processes it into the data type required for image detection. The second step is that the image detection module extracts convolutional features from the image to identify the driver's body features, including hands, shoulders, neck, mouth, nose, eyes, and ears. The third step is that the system locates the driver's position in the video based on the driver's body features. The fourth step is that the system uses a deep neural network classification model to identify the driver's actions based on the driver's posture, such as looking away, driving while fatigued, smoking, drinking water, and taking hands off the keyboard. The fifth step is to classify different types of dangerous driving behaviors into levels, such as reminder, warning, and danger, and provide different levels of reminders to the user according to different levels of danger. For example, actions at the "danger" level are reminded to the user via voice, while actions at the "warning" level are reminded to the user via warning text.

[0061] An example of how the system runs can be described as follows:

[0062] (1) The vehicle-mounted camera acquires in-vehicle video stream information and transmits the video information back to the vehicle-mounted image processing equipment.

[0063] (2) The image processing equipment uses H.264 technology to decompress the video stream data.

[0064] (3) The image processing device extracts images frame by frame from the decompressed video stream.

[0065] (4) Detect key points in the image

[0066] The system takes a 1200*800 image as input. After the image enters the network, the network first uses a convolutional module to extract key information (features) from the image.

[0067] This module has a total of 17 layers, consisting of 13 convolutional layers with a kernel size of 3*3 and 4 max pooling layers with a stride of 2. The function of the convolutional layers is to extract key information from the image. The input of the first convolutional layer is the original image information, and the input of the subsequent convolutional layers is the result calculated by the previous convolutional layer. Because there are thirteen convolutional layers, it can extract information in thirteen dimensions. The higher the dimension, the higher the degree of feature abstraction.

[0068] There are four pooling layers. Their function is to aggregate the features extracted by the convolutional layers, making the key information more apparent and thus facilitating processing by the subsequent feature recognition module. The final output is a 512-channel feature matrix.

[0069] After processing by the above modules, a 512-channel feature matrix is ​​obtained. The features after one convolution process are processed by two modules, namely the M1 module and the M2 module. The M1 module is used to detect the driver's actions, and the M2 module is used to detect other people in the vehicle.

[0070] Since the driver is always at the forefront of the image and occupies the largest area, a max-pooling layer with a stride of 2 is used before the M1 module to process the features. The purpose of this step is to increase the network's receptive field, making it easier to identify larger targets (humans). Meanwhile, since other people in the car are located at the rear and have a smaller display area in the image, no additional processing is needed when using the M2 module to identify people in the car.

[0071] (5) Detect key features

[0072] The M1 module described in the previous step is used to detect the driver. After the feature matrix is ​​input into this module, it is processed in two steps. The first step involves processing the features through a 3x3 convolutional layer followed by a ReLU (Rectified Linear Unit) layer. This layer's main function is to process the linear convolutional features to fit non-linear features. The second step uses a wide-area information processing module to process the features. This module contains four convolutional and ReLU layers, and it halves the number of channels in the features. This layer's function is to process broader information in the features to improve the accuracy of driver feature recognition. After these two processing steps, a feature fusion module fuses the outputs of the two steps. This step has completed the detection of the driver's key features in the image. The next step is to adjust the image based on these key features.

[0073] (6) After determining the target key points, the image is cropped according to the position of the key points and the noise part is removed. The cropping method is to extend horizontally outward to the image boundary with the midpoint of the shoulders as the center, and take twice the shortest extension distance as the image width. Vertically extend outward to the image boundary, and take twice the shortest extension distance as the image height.

[0074] (7) Perform convolution processing on the image and use a neural network to extract the convolution features of the target image.

[0075] Two 1x1 convolutional layers are used as output processing layers, representing the classification of each pixel and the identification of the driver's position in the image, respectively.

[0076] (8) Classify the convolutional features to determine whether the target object performs a preset action. The preset actions include seven actions: smoking, drinking water, wearing sunglasses, making a phone call, looking away, closing eyes, and taking both hands off the steering wheel. If the same action is identified continuously, the duration of the action is recorded as the number of consecutively identified image frames multiplied by the interval between each frame.

[0077] (9) The aforementioned neural network is defined as a deep neural network trained under supervision using pre-labeled sample image data, wherein the sample image data includes the seven types of action images mentioned above as well as in-vehicle driving image data under normal conditions.

[0078] (10) The training process of a neural network is as follows:

[0079] The above sample images are identified using an initialized neural network. The identification results are compared with the predefined annotations to determine the identification loss.

[0080] The network parameters are adjusted according to the above loss to reduce the recognition error in the next step;

[0081] After sufficient adjustments, the error tends to a stable minimum value, thus obtaining a neural network that can be used for action recognition.

[0082] (11) After the image processing module identifies the presence of seven preset actions in the image, it will send the identification results to the warning module. The conditions for triggering the warning module include the duration, frequency and type of the action.

[0083] (12) The early warning module issues the following early warning information based on the action recognition results:

[0084] a) The duration of the current action;

[0085] b) The type of the current action;

[0086] c) The danger level of the current action;

[0087] d) Current vehicle status, including vehicle speed, window status, mileage, and travel time.

[0088] In embodiments of the present invention, the image processing module is integrated into the vehicle's infotainment system, utilizing the system's computing resources.

[0089] The present invention also proposes a driver action recognition method, which includes: acquiring an in-vehicle image as an input image, wherein the in-vehicle image includes an image of the overall in-vehicle space; extracting a feature matrix of the input image through a convolutional submodule, performing a convolutional process on the feature matrix to obtain key features, and inputting the key features into two channels for processing. In the first channel, the key features are processed sequentially through a max pooling layer and a first detection module. In the second channel, the key features are processed through a second detection module. The first detection module is used to detect the driver's behavior, and the second detection module is used to detect the behavior of other people in the vehicle. When the driver's behavior is detected to include one or more of the following: smoking, drinking water, wearing sunglasses, making a phone call, looking away from the vehicle, closing eyes, or taking both hands off the steering wheel, an alarm is issued.

[0090] This invention provides a driver action recognition scheme, including dangerous driving behaviors such as leaving the steering wheel, deviating from the line of sight, fatigued driving, abnormal emotions, and answering the phone, thereby promptly judging the user's behavior and reminding the user to drive safely. This invention has advantages such as fast recognition speed, high accuracy, high aggregation, and scalability to various scenarios.

[0091] Compared with solutions currently known in the field, the advantages of this invention are as follows:

[0092] It has a wide detection range and can detect all parts of the body above the driver's knees;

[0093] The detection speed is fast; it can detect a user's action if they hold it for more than 2 seconds.

[0094] Offline operation: The model has been specially optimized to run offline and maintain service throughout the entire process;

[0095] Multiple types of identification can be used to identify dangerous driving behaviors, including smoking, drinking water, driving while fatigued, deviating from the line of sight, and wearing sunglasses.

[0096] The foregoing description of embodiments of the invention has been given for illustrative purposes and is not exhaustive, nor is it intended to limit the invention to the exact forms disclosed. Those skilled in the art will understand that various changes can be made without departing from the scope of the invention, and elements therein can be substituted with equivalents. Furthermore, many modifications can be made to adapt particular situations or materials to the teachings of the invention without departing from the basic scope of the invention. Therefore, the invention is not intended to be limited to the specific embodiments disclosed as the best mode contemplated for carrying out the invention; the invention will include all embodiments falling within the scope of the appended claims.

Claims

1. A driver action recognition system, characterized in that, The system includes: an image acquisition module, a motion recognition module, and an early warning module. The image acquisition module is used to acquire images inside the vehicle as input images for the action recognition module, wherein the images inside the vehicle include images of the overall interior space of the vehicle. The action recognition module is used to: extract the feature matrix of the input image through a convolution submodule, perform a convolution process on the feature matrix to obtain key features, and input the key features into two channels for processing. In the first channel, the key features are processed by a max pooling layer and a first detection module. In the second channel, the key features are processed by a second detection module. The first detection module is used to detect the driver's behavior, and the second detection module is used to detect the behavior of other people in the vehicle. The warning module is used to issue an alarm when the action recognition module detects one or more of the driver's behaviors, including smoking, drinking water, wearing sunglasses, making a phone call, looking away from the road, closing eyes, or taking both hands off the steering wheel. Wherein, the first detection module is further used for: The features to be detected input to the first detection module are processed in two steps. In the first step, the features are passed through a 3x3 convolutional layer and then processed by a rectified linear unit (RLU) layer. The RLU layer processes the linearly convolved features to fit non-linear features. In the second step, the features are processed by a wide-area information processing module, which includes four convolutional layers and four RLU layers, and the number of channels to be detected is halved. The fusion module fuses the features to be detected after the two processes to obtain fused features. The fused features are input into two 1x1 convolutional layers to obtain action classification and the driver's position in the image.

2. The system according to claim 1, characterized in that, The convolutional submodule includes 17 processing layers, namely 13 convolutional layers with a kernel size of 3*3 and 4 max pooling layers with a stride of 2. The 4 max pooling layers are located at the 3rd, 6th, 10th and 14th layers of the 17 processing layers.

3. The system according to claim 1, characterized in that, The image acquisition module is also used to compress the in-vehicle image through the compression submodule, and use the compressed image as the input image of the action recognition module, wherein the compression submodule includes a three-layer neural network.

4. The system according to claim 1, characterized in that, The alarm triggering conditions of the warning module include the duration, frequency and type of one or more of the following actions: smoking, drinking water, wearing sunglasses, making a phone call, looking away from the road, closing one's eyes, and taking one's hands off the steering wheel.

5. The system according to claim 1, characterized in that, The early warning module is also used to issue one or more of the following early warning messages based on the action recognition results: The duration of the current action; The type of current action; The danger level of the current action; Current vehicle status, including vehicle speed, window status, mileage, and travel time.

6. The system according to claim 1, characterized in that, The system also includes an image preprocessing module for: Detect key points in the in-vehicle image, including one or more of the driver's eyes, ears, mouth, nose, mouth, shoulders, and hands; The input image is cropped based on the location of the key points. The cropping method is as follows: with the midpoint of the shoulders as the center, the image width is extended horizontally outward to the image boundary, and twice the shortest extension distance is taken as the image width. The image height is also extended vertically outward to the image boundary, and twice the shortest extension distance is taken as the image height.

7. The system according to claim 1, characterized in that, The training process of the first detection module includes: The sample image is identified using an initialized neural network, and the recognition result is compared with the predefined annotation to determine the recognition loss. The network parameters are adjusted based on the loss to reduce the recognition error in the next iteration; After adjustment, the error tends to a stable minimum value, thus obtaining the first detection module that can be used for action recognition.

8. The system according to claim 1, characterized in that, The warning module is also used to classify different types of driving behaviors into levels and provide different reminders to users according to different levels of danger. For actions at the "danger" level, voice reminders are used to remind users, and for actions at the "warning" level, warning text reminders are used to remind users.

9. A driver action recognition method, characterized in that, The method includes: The in-vehicle images are acquired as input images, wherein the in-vehicle images include images of the overall in-vehicle space; The feature matrix of the input image is extracted by the convolution submodule. The feature matrix is ​​then subjected to a convolution process to obtain key features. The key features are then input into two channels for processing. In the first channel, the key features are processed sequentially through a max pooling layer and a first detection module. In the second channel, the key features are processed by a second detection module. The first detection module is used to detect the driver's behavior, and the second detection module is used to detect the behavior of other people in the vehicle. An alarm will be triggered when the driver's behavior is detected to include one or more of the following: smoking, drinking water, wearing sunglasses, making a phone call, looking away from the road, closing their eyes, or taking their hands off the steering wheel. Wherein, the first detection module is further used for: The features to be detected input to the first detection module are processed in two steps. In the first step, the features are passed through a 3x3 convolutional layer and then processed by a rectified linear unit (RLU) layer. The RLU layer processes the linearly convolved features to fit non-linear features. In the second step, the features are processed by a wide-area information processing module, which includes four convolutional layers and four RLU layers, and the number of channels to be detected is halved. The fusion module fuses the features to be detected after the two processes to obtain fused features. The fused features are input into two 1x1 convolutional layers to obtain action classification and the driver's position in the image.

Images