A driving behavior recognition method and system based on depth image

Abstract

A method and system for identifying driving behavior based on depth images, including: collecting a driver's depth image sequence to form an image training set containing the driver's upper body skeletal point information; performing depth image preprocessing; analyzing image position information, obtaining a depth image time series and performing normalization processing; analyzing image skeletal information, calculating a skeletal node matrix and performing normalization processing; using an improved VGG‑16 network to process the obtained data, fusing the training results of the depth image sequence and skeletal point information, and outputting a driving behavior classification result. This method and system adopts a computer vision processing method that combines skeletal tracking with convolutional neural networks and fatigue detection, and designs a lightweight and low-complexity edge system that meets the requirements of easy deployment and high stability in a real driving environment. It can efficiently, accurately, and timely detect the driver's violations and give warnings, thereby reducing the possibility of traffic accidents and ensuring road safety.

Description

Technical Field

[0001] The present invention relates to a computer vision method in the field of artificial intelligence, and in particular to a driving behavior recognition method and system based on depth images. Background Art

[0002] As the popularity of cars continues to increase, the issue of safe driving has repeatedly made headlines. Drivers' decisions and behaviors are the main factors affecting driving safety. The number of accidents caused by failure to strictly abide by existing regulations accounts for 60% of the total number of traffic accidents. According to the World Health Organization, if effective measures are not taken, road traffic accidents will become the seventh leading cause of death by 2030. Convolutional neural networks are currently the most powerful network structure in the field of image and video processing, and their superiority in related tasks is unparalleled. At the same time, the recognition effect of CNN on the large-scale dataset ImageNet also makes driver behavior recognition a feasible direction. Generally speaking, driving behavior is sitting, and the image background collected by the interactive system will also be relatively simple.

[0003] At present, the most common method in the field of human posture recognition is principal component analysis (PCA). This method can identify 8 typical sitting postures and analyze facial movements by changing the number of pixels of facial skin. In terms of the application of Kinect's skeletal tracking function, the OpenPose human posture recognition project is an open source library developed by Carnegie Mellon University (CMU) in the United States based on convolutional neural networks and supervised learning and using caffe as the framework. It can realize posture estimation such as human movements, facial expressions, and finger movements. It is the world's first real-time multi-person two-dimensional posture estimation application based on deep learning. In 2019, the Y.Xing team designed a driver activity recognition system based on a deep convolutional neural network (CNN). First, the Gaussian mixture model (GMM) was used to segment the original image and separate the driver image from the background environment, and then the CNN model was trained using three different pre-trained neural networks (AlexNet, GoogLeNet, ResNet50). The above-mentioned prior art does not well integrate the Kinect-based deep image method with the convolutional neural network method to solve the problem of detecting bad driving behavior of drivers. Summary of the invention

[0004] In view of the problems existing in the above-mentioned background technology, a driving behavior recognition method and system based on depth image is proposed. Jetson Nano is used as the edge computing device, which reduces the complexity of the system while improving the accuracy of posture recognition, making it easy to deploy and highly reliable.

[0005] A driving behavior recognition method based on depth image comprises the following steps:

[0006] S1, collecting a depth image sequence of the driver to form an image training set containing the skeleton point information of the driver's upper body;

[0007] S2, performing deep image preprocessing on the training set;

[0008] S3, further analyzing the image position information, obtaining the depth image time series, and normalizing the depth image time series;

[0009] S4, further analyzing the image bone information, calculating the bone node matrix, and normalizing the bone node three-dimensional matrix;

[0010] S5. Use the improved VGG-16 network to process the obtained data, fuse the training results of the depth image sequence and the skeleton point information, and output the driving behavior classification results.

[0011] Furthermore, in step S1, a depth image sequence of the driver is collected by Microsoft Azure Kinect DK, wherein each frame of the image includes RGB information and distance information; the distance information refers to the distance (in millimeters) from each pixel point (x, y) to the plane closest to the camera plane, which directly reflects the geometric shape of the visible surface of the scene.

[0012] Furthermore, in step S2, the training set depth image preprocessing includes repairing pixels where depth information is lost for image occlusion problems caused by the complex environment inside the vehicle.

[0013] Furthermore, in step S2, the depth information is repaired specifically by setting the mean bone length L and the error parameter σ for the bones at different positions of the driver's upper body when sitting. When the bone length at the corresponding position of the acquired depth image is between [L-σ, L+σ], the image information is considered complete, and the position with insufficient length is smoothly filled to obtain the bone joint feature vector.

[0014] Furthermore, step S3 includes the following sub-steps:

[0015] Step S3-1, projecting each frame of the driver posture depth image sequence onto three orthogonal Cartesian plane coordinate systems to extract the three-dimensional structural spatial features of the driver's behavior in the sequence;

[0016] Step S3-2, calculate the absolute difference of the three-dimensional spatial features of every two consecutive frames, and superimpose them to obtain a depth motion map as a neural network input, that is, a depth image time series.

[0017] Furthermore, step S4 includes the following sub-steps:

[0018] Step S4-1, using the distance information in the driver's posture depth image to detect the behavior edge of the upper body, segmenting the target and extracting the behavior outline from the background, wherein only the main skeleton points of the key parts of the human body in the driving posture are identified, including: head, center of both shoulders, left shoulder, left elbow joint, left wrist joint, left hand, right shoulder, right elbow joint, right wrist joint, right hand, middle part of the spine, and they are numbered in the order of 1 to 11;

[0019] Step S4-2, converting the skeletal motion sequence based on the Microsoft Azure Kinect DK coordinate system (x, y, z) into a skeletal node three-dimensional matrix, where the x, y, z directions correspond to the R, G, B channels of the RGB image one by one; the skeletal motion sequence is skeletal data arranged in chronological order, and each group of skeletal data contains 11 main skeletal point three-dimensional coordinates, and is labeled with numbers 1 to 11;

[0020] Step S4-3, normalizing the three-dimensional matrix of the skeleton nodes, setting the target frame number to N frames, selecting [0, N] frames as the normalized skeleton motion sequence, intercepting and discarding the sequence that exceeds the target frame, and filling the sequence that is less than the target frame with zeros;

[0021] Step S4-4, the size of the processed bone node matrix is ​​N×11×3, and the bone data used as the neural network input is obtained.

[0022] Further, in step S5, an improved VGG-16 model based on the idea of ​​a two-stream convolutional neural network is used, and a convolutional neural network including five channels is adopted, wherein the first three channels are used to train the deep motion graph, and the last two channels are used to train the skeleton node matrix and the skeleton joint feature vector;

[0023] Using the late fusion method, the three-dimensional spatial features and time information of the driver's posture depth image, the three-dimensional motion coordinates of the sitting upper body skeleton data and the classification results of the topological shape are integrated and output for decision making.

[0024] A driving behavior recognition system based on depth images, comprising:

[0025] The built-in infrared transmitter and infrared depth camera of Microsoft Azure Kinect DK are used to collect the position information of the driver's upper body. After decoding and calculation by the CMOS sensor, a depth image of the driver's posture containing distance information, i.e., an RGB-D image, is obtained.

[0026] Nvidia Jetson Nano is used as the edge computing device to analyze the acquired images and output the final recognition results.

[0027] Furthermore, the deep learning method integrated in the Nvidia Jetson Nano development board is used to analyze and extract image information and identify 15 different driving behaviors, including:

[0028] Normal driving, making a phone call with the left hand, looking at the phone with the left hand, making a phone call with the right hand, looking at the phone with the right hand, talking to the passenger seat, adjusting the center instrument, drinking water, eating, turning the head back, bending down, taking hands off the steering wheel, squinting, yawning, and nodding.

[0029] The beneficial effects achieved by the present invention are as follows: the depth image collected by replacing the traditional camera with a depth camera adds distance information compared to the RGB image, that is, the RGB-D image. For the dim lighting and cluttered background that may occur in the real driving environment, the use of depth images can better extract the driver's behavior characteristics. At the same time, the skeleton tracking algorithm can better complete the missing image information in the data preprocessing stage, providing a good foundation for subsequent training. In addition, 15 common driving behaviors are considered, which basically include most of the bad driving behaviors that are prone to dangerous accidents, and have high practical value. The present invention adopts a computer vision processing method that integrates skeleton tracking, convolutional neural networks and fatigue detection, and designs a set of lightweight and low-complexity edge systems, which meet the requirements of easy deployment and high stability in real driving environments, and can efficiently, accurately and timely detect the driver's violations and give warnings, thereby reducing the possibility of traffic safety accidents and safeguarding road safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flowchart of a driving behavior recognition method based on depth images in an embodiment of the present invention.

[0031] Figure 2 It is a schematic diagram of the improved VGG-16 network structure in the driving behavior recognition method based on depth images described in the present invention.

[0032] Figure 3 It is a schematic diagram of the five-channel convolutional neural network architecture in the driving behavior recognition method based on depth images described in the present invention.

[0033] Figure 4 It is a schematic diagram of the structure of Microsoft AzureKinect SD in the driving behavior recognition system based on depth images in an embodiment of the present invention.

[0034] Figure 5It is a schematic diagram of the structure of Nvidia JetsonNano in the driving behavior recognition system based on depth image in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0036] Please refer to Figure 1 The present invention provides a method for identifying driving behavior based on a depth image, comprising:

[0037] S100: Collect a depth image sequence of the driver to form an image training set containing skeleton point information of the driver's upper body.

[0038] In this embodiment, the depth image sequence of the driver is collected by Microsoft Azure Kinect DK. Among them, the measured object in this embodiment is mainly a private car driver as an example, and the depth image sequence of the private car driver is collected by the depth camera. It should be noted that the measured object in this embodiment can also be a public transportation driver.

[0039] S200: performing depth image preprocessing on the training set.

[0040] Due to the influence of the in-car environment, the driver image captured by the depth camera may be partially blocked, making the image information incomplete. Therefore, it is necessary to preprocess the depth image to repair the missing pixels.

[0041] It can be understood that the human skeleton structurally satisfies certain length constraints. Based on this premise, the present invention fills in missing pixels by limiting the length of the extracted bone information in the captured image. For the bones at different positions of the driver's upper body in the sitting position, the bone length mean L and the error parameter σ are set respectively. When the bone length at the corresponding position of the captured depth image is between [L-σ, L+σ], the image information is considered complete, and the position with insufficient length is smoothly filled to obtain the bone joint feature vector.

[0042] S300: further analyzing the image position information to obtain a depth image time series.

[0043] After step S200 is completed, the collected driver depth image data is a set of images containing complete position information and skeleton information. Based on this, this link further processes the obtained image data, aiming to obtain a depth image time series from it.

[0044] First, each frame of the driver posture depth image is projected onto three orthogonal Cartesian plane coordinate systems to extract the three-dimensional structural spatial features of the driver's behavior in the sequence. Specifically, through the projection algorithm, each frame of the depth image sequence Mapi (i represents the i-th frame) will get three components, namely the front view Side View Top view Right now (v∈(f,s,t)).

[0045] After that, the absolute difference of the three-dimensional spatial features of every two consecutive frames is calculated and superimposed to obtain a depth image time series that can be used as a neural network input. Specifically, the present invention adopts the deep motion map (DMM) method. v The calculation method of (v∈(f,s,t)) is:

[0046]

[0047] S301 : normalize the depth image time series.

[0048] In order to make it easier for the neural network to learn image features, the input data needs to be normalized. In this embodiment, the depth image time series after step S210 needs to remove the DMM v The boundary part with zero pixel value is uniformly scaled to a size of 224*224.

[0049] S400, further analyzing the image skeleton information and calculating the skeleton node matrix.

[0050] After step S200 is completed, the collected driver depth image data is a set of images containing complete position information and skeleton information. Based on this, this link further processes the obtained image data, with the goal of calculating the skeleton node matrix from it.

[0051] First, the distance information in the driver's posture depth image is used to detect the behavioral edge of the upper body, and the target is segmented and the behavioral contour is extracted from the background. Only the main skeleton points of the key parts of the human body in the driving posture are identified, including: head, center of shoulders, left shoulder, left elbow joint, left wrist joint, left hand, right shoulder, right elbow joint, right wrist joint, right hand, and middle section of the spine. They are numbered in the order of 1 to 11.

[0052] After that, the skeletal motion sequence based on the Microsoft Azure Kinect DK coordinate system (x, y, z) is converted into a 3D matrix of skeletal nodes, where the x, y, and z directions correspond to the R, G, and B channels of the RGB image. The skeletal motion sequence is skeletal data arranged in chronological order. Each set of skeletal data contains 11 3D coordinates of the main skeletal points, which are labeled with numbers from 1 to 11. The number of matrix rows represents the number of frames N of the skeletal motion sequence, and the number of matrix columns represents the number of skeletal nodes in each frame 11. Finally, each skeletal motion sequence is output as a 3D matrix of N×11×3.

[0053] S401, normalize the three-dimensional matrix of the skeleton node.

[0054] In order to make it easier for the neural network to learn image features, the input data needs to be normalized. In this embodiment, the bone node matrix obtained in step S220 is normalized by intercepting and supplementing. Specifically, the target number of frames is set to 54 frames. Select [0,54] frames as the normalized bone motion sequence, intercept and discard the sequence that is more than the target frame, and fill the sequence that is less than the target frame with zeros. The size of the processed bone node matrix is ​​54*11*3, and the bone data used as the neural network input is obtained.

[0055] S500, using the improved VGG-16 network to process the obtained data, fusing the training results of the depth image sequence and the skeleton point information, and outputting the driver behavior classification result.

[0056] The present invention adopts a convolutional neural network with five channels, wherein the first three channels are used to train the deep motion graph, and the last two channels are used to train the skeleton node matrix and the skeleton joint feature vector. By using the late fusion method, the classification results of the three-dimensional spatial features and time information of the driver's posture depth image, the three-dimensional motion coordinates of the sitting upper body skeleton data and the topological shape are integrated and output for decision making. Each single-channel convolutional neural network adopts an improved VGG-16 network, deleting 9 convolutional layers, 1 fully connected layer, and 1 pooling layer of the classic VGG-16 model, simplifying the model complexity.

[0057] Please refer to Figure 2, the present invention provides a schematic diagram of the improved VGG-16 network structure in a driving behavior recognition system based on a depth image. The single-channel convolutional neural network constructed by the present invention has a total of 10 training layers, including 4 convolutional layers, 4 pooling layers and 2 fully connected layers, and does not include a classification layer. The stride of the convolutional layer is 1 and a padding algorithm is used, which can retain the feature map information in the convolution process. The stride of the pooling layer is 2 and a max-pooling algorithm is used to avoid the blurring effect of average pooling. The first fully connected layer generates 1024 neuron nodes (FC-1024) output, and the second fully connected layer generates 15 neuron nodes (FC-15) output, corresponding to 15 different driving behavior categories.

[0058] There are many algorithms available for late fusion, the most common ones are the average method, the maximum method, and the product method. This embodiment uses the average method, adds the output results of the five classifiers and takes the average value, which can average the impact of each data on the result, eliminate the contribution of prominent data, and obtain a smoother result. Specifically, the calculation method is as follows:

[0059]

[0060] Where n represents the number of classifiers, P i Represents the probability vector for each classifier.

[0061] In this embodiment, a Softmax classifier is used after each convolutional neural network output to obtain the classification probability of each category.

[0062] Please refer to Figure 3 The present invention provides a schematic diagram of a five-channel convolutional neural network architecture in a driving behavior recognition system based on a depth image. The CNNs of the five channels all adopt the above-mentioned improved VGG-16 network structure and the same training parameters, are independent of each other, and are trained simultaneously during the operation of the system. After integrated decision-making, the corresponding classification structures are fused and output. The five-channel convolutional neural network described in the present invention realizes parallel training of multimodal data, and can simultaneously process multi-feature information of the driver's posture, thereby more fully considering the spatiotemporal details of human behavior and greatly improving the model accuracy.

[0063] Please refer to Figure 4 The present invention provides a schematic diagram of the Microsoft AzureKinect DK structure in a driving behavior recognition system based on a depth image. The present invention mainly relies on a depth camera to obtain image information, and the specific working principle includes:

[0064] The depth camera implements the Amplitude Modulated Continuous Wave (AMCW) Time of Flight (ToF) principle. The camera projects modulated light in the near infrared (NIR) spectrum into the scene. It then records indirect measurements of the time it takes for the light to travel from the camera to the scene and then back from the scene to the camera. Processing these measurements generates a depth map. A depth map is a set of Z coordinate values ​​for each pixel of the image, measured in millimeters.

[0065] The use of a depth camera based on the ToF principle can overcome the effects of illumination changes and object texture. In this embodiment, considering the travel time factor, there may be direct sunlight or dim light at night; considering the in-car environment factor, the driver may drive in a background with different decorative styles. The present invention uses Microsoft Azure KinectDK as a depth camera to overcome the above problems.

[0066] Please refer to Figure 5 , the present invention provides a schematic diagram of the structure of Nvidia JetsonNano in a driving behavior recognition system based on depth images.

[0067] The Jetson Nano development kit uses a microSD card as a boot device and main storage. In this embodiment, a 32GB UHS-1 card is used to provide sufficient system memory. The present invention mainly uses it to process image classification tasks, and can run multiple neural networks in parallel in an easy-to-use platform, with only 5 watts of power required for operation. On the basis of providing sufficient computing power support, the power consumption is extremely low, which is well adapted to the needs of system layout in a relatively small environment in the car, and simplifies the complexity of the edge computing system.

[0068] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A driving behavior recognition method based on depth image, characterized in that: The identification method comprises the following steps: S1, collecting a depth image sequence of the driver to form an image training set containing the skeleton point information of the driver's upper body; S2, performing deep image preprocessing on the training set; In step S2, the training set depth image preprocessing includes repairing the pixel points where the depth information is lost for the image occlusion problem caused by the complex environment inside the car; In step S2, the depth information repair is specifically to set the mean bone length L and the error parameter y for the bones at different positions of the driver's upper body. , when the bone length at the corresponding position of the acquired depth image is The image information is considered complete when the length is between 0 and 1. The insufficient length is filled smoothly to obtain the bone joint feature vector. S3, further analyzing the image position information, obtaining the depth image time series, and normalizing the depth image time series; Step S3 includes the following sub-steps: Step S3-1, projecting each frame of the driver posture depth image sequence onto three orthogonal Cartesian plane coordinate systems to extract the three-dimensional structural spatial features of the driver's behavior in the sequence; Step S3-2, calculating the absolute difference of the three-dimensional spatial features of every two consecutive frames, and superimposing them to obtain a depth motion map as an input of the neural network, that is, a depth image time series; S4, further analyzing the image bone information, calculating the bone node matrix, and normalizing the bone node three-dimensional matrix; Step S4 includes the following sub-steps: Step S4-1, using the distance information in the driver's posture depth image to detect the behavior edge of the upper body, segmenting the target and extracting the behavior outline from the background, wherein only the main skeleton points of the key parts of the human body in the driving posture are identified, including: head, center of shoulders, left shoulder, left elbow joint, left wrist joint, left hand, right shoulder, right elbow joint, right wrist joint, right hand, middle part of the spine, and they are numbered in the order of 1 to 11; Step S4-2, converting the bone joint feature vector based on the Microsoft Azure Kinect DK coordinate system (x, y, z) into a three-dimensional matrix of bone nodes, where the x, y, and z directions correspond to the R, G, and B channels of the RGB image one by one; Step S4-3, normalizing the three-dimensional matrix of the skeleton nodes, setting the target frame number to N frames, selecting [0, N] frames as the normalized skeleton motion sequence, intercepting and discarding the sequence that exceeds the target frame, and filling the sequence that is less than the target frame with zeros; Step S4-4, the size of the processed skeleton node matrix is ​​N×11×3, and the skeleton data used as the neural network input is obtained; S5. Using the improved VGG-16 network to process the obtained data, the training results of the depth image sequence and the skeleton point information are integrated to output the driving behavior classification results; In step S5, an improved VGG-16 model based on the idea of ​​a two-stream convolutional neural network is used, and a convolutional neural network including five channels is adopted, wherein the first three channels are used to train the deep motion graph, and the last two channels are used to train the skeleton node matrix and the skeleton joint feature vector; Using the late fusion method, the three-dimensional spatial features and time information of the driver's posture depth image, the three-dimensional motion coordinates of the sitting upper body skeleton data and the classification results of the topological shape are integrated and output for decision making.

2. The method for identifying driving behavior based on depth image according to claim 1, characterized in that: In step S1, a driver posture depth image sequence is collected through Microsoft Azure Kinect DK, where each frame of the image includes RGB information and distance information. The distance information refers to the distance from each pixel point (x, y) to the plane closest to the camera plane, reflecting the geometric shape of the visible surface of the scene.

3. A system using the driving behavior recognition method based on depth image according to any one of claims 1 to 2, characterized in that: The system comprises: The built-in infrared transmitter and infrared depth camera of Microsoft Azure Kinect DK are used to collect the position information of the driver's upper body. After decoding and calculation by the CMOS sensor, a depth image of the driver's posture containing distance information, i.e., an RGB-D image, is obtained. Nvidia Jetson Nano is used as the edge computing device to analyze the acquired images and output the final recognition results.

4. The system using the driving behavior recognition method based on depth image according to claim 3, characterized in that: The deep learning method integrated into the Nvidia Jetson Nano development board is used to analyze and extract image information and identify 15 different driving behaviors, including: Normal driving, making a phone call with the left hand, looking at the phone with the left hand, making a phone call with the right hand, looking at the phone with the right hand, talking to the passenger seat, adjusting the center instrument, drinking water, eating, turning the head back, bending down, taking hands off the steering wheel, squinting, yawning, and nodding.

Images