A distracted driving behavior recognition method based on gradient controlled federated learning

Abstract

The present invention discloses a distracted driving behavior identification method based on gradient-controlled federated learning. First, each vehicle node collects image data while the vehicle is driving; then, a convolutional neural network model is jointly trained using gradient-controlled federated learning; finally, each node uses the model to identify distracted driving behavior. In the driving behavior identification scenario, the present invention uses a decentralized method based on edge computing to implement model training and reasoning, which has the advantages of protecting user privacy, good real-time performance, and convenient upgrades compared to existing centralized training and reasoning methods. The proposed gradient-controlled federated learning algorithm can effectively improve the convergence speed of the model compared to the traditional federated learning algorithm without increasing any computing overhead.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and edge computing, and in particular to a distracted driving behavior recognition method based on gradient controlled federated learning. Background Art

[0002] According to data from China's Ministry of Public Security, by the end of 2020, China had 372 million motor vehicles, of which 281 million were cars; China had 456 million motor vehicle drivers, of which 418 million were car drivers. The increasing number of cars and drivers has indirectly led to an increase in the number of traffic accidents year by year, and many of these accidents are caused by distracted driving behaviors of drivers, such as using mobile phones, eating, chatting with passengers, adjusting the central control, etc. Therefore, identifying distracted driving behaviors of drivers is of great significance to advanced driver assistance systems (ADAS).

[0003] With the development of deep learning technology in recent years, image classification models based on deep convolutional neural networks such as VGG, GoogleNet, and RestNet have shown strong advantages on the ImageNet dataset. More and more distracted driving behavior recognition methods based on deep learning have been proposed. Most of these methods are improvements on deep learning models, and their training and reasoning are centralized, that is, the model training and reasoning process will only be performed locally or in the cloud.

[0004] Local training and reasoning of distracted driving behavior recognition models require the pre-collection of a large number of samples, and subsequent model upgrades require redeployment, which is rather troublesome. Training and reasoning of distracted driving behavior recognition models in the cloud require extremely high network bandwidth, latency, and stability. In addition, the uploaded driving images may involve the driver's privacy.

[0005] Based on the above-mentioned problems with model training and reasoning locally and in the cloud, if the model training and reasoning are performed at the edge, the edge is any computing resources and network resources between the data source and the cloud. Applying edge computing in the Internet of Vehicles can effectively solve the above problems.

[0006] Federated learning is a training strategy for edge computing. In federated learning, multiple edge nodes and an edge server form a cluster to jointly train a global model. The edge server does not have any training data. All training data is distributed on each edge node. After each training, the edge node uploads the updated local model to the edge server. The edge server aggregates the local models uploaded by each node and finally realizes the training of the global model. In this process, each node only uploads model parameters but not data, so the privacy of node users is also protected. Summary of the invention

[0007] In view of the above-mentioned deficiencies in the prior art, the present invention provides a distracted driving behavior identification method based on gradient controlled federated learning.

[0008] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0009] A distracted driving behavior recognition method based on gradient controlled federated learning includes the following steps:

[0010] S1, obtaining images of distracted driving behaviors of the driver during driving and preprocessing them to obtain a training data set;

[0011] S2. Use a gradient-controlled federated learning algorithm to train the training data set obtained in step S1 to obtain a distracted driving behavior recognition model;

[0012] S3. Using the recognition model obtained in step S2, the driver's distracted driving behavior is recognized.

[0013] Furthermore, the step S1 specifically includes:

[0014] S11, using a vehicle-mounted camera to obtain an image of the driver's distracted driving behavior;

[0015] S12, detecting and cropping the collected distracted driving behavior images, performing driver area detection based on HOG features and SVM algorithm model, and intercepting and saving the corresponding areas to obtain a driver area image sequence;

[0016] S13, labeling the driver area image sequence obtained in step S12 to obtain a training data set.

[0017] Furthermore, the step S2 specifically includes:

[0018] S21, using a vehicle as a node, using a vehicle-mounted ad hoc network to build a vehicle-to-vehicle communication network including multiple nodes, and selecting one of the nodes as an edge server;

[0019] S22: Initialize the global model using the edge server, where the global model in the tth round of cycles is denoted as ω t , and distribute the global model to each node;

[0020] S23, each node trains the global model distributed to itself, where the local model of the kth node after training is recorded as And send the trained model to the edge server;

[0021] S24. The edge server aggregates the global models sent back by each node, uses the aggregated new model as the global model of the next cycle, and determines whether the global model converges. If not, it returns to step S22. If so, it outputs the global model.

[0022] Furthermore, in step S23, the training data set is enhanced using the mixup method, which is expressed as:

[0023] mixed_image x =λ×image x1 +(1-λ)×image x2 ;

[0024] mixed_image y =λ×image y1 +(1-λ)×mage y2 ;

[0025] Among them, mixed_image x Represents the image after mixup, mixed_image y Indicates the image label after mixup; image x1 、image x2 Indicates superposition to form mixed_image x Two pictures of image y1 、image y2 Represents the labels corresponding to the two pictures; λ is the mixing coefficient calculated by Beta distribution, where λ=Beta(α,β), α, β are Beta distribution parameters.

[0026] Furthermore, the loss function of the global model in step S24 is expressed as:

[0027]

[0028] Among them, ω t+1 is the global model of the federated learning round t+1, is the local model of node k after the t+1th round of training in federated learning, is the multi-classification cross entropy loss function of node k, γ is the constraint coefficient, cosine_distance is used to measure the cosine distance between two gradients, 0 means the gradient directions are the same, 1 means the gradient directions are orthogonal, and 2 means the gradient directions are opposite.

[0029] Furthermore, the step S24 specifically includes:

[0030] S241. Calculate the weight of the model returned by each node to the edge server, and the calculation method is:

[0031]

[0032] S242, using the weights obtained in step S241 to calculate the global model of the next cycle, expressed as:

[0033]

[0034] where ω t+1 is the global model of the federated learning round t+1, It is the local model of node k after the t+1th round of training in federated learning.

[0035] Furthermore, the step S3 specifically includes:

[0036] S31, collecting images while the vehicle is driving and preprocessing the images to be recognized, and using the vehicle-mounted distracted driving behavior recognition model to classify the preprocessed driver area image sequence;

[0037] S32: If three consecutive frames in the image sequence have the same image classification and their confidence levels are greater than a set threshold, it is determined that the driver has made a distracted driving action, and a corresponding voice reminder is issued to the driver.

[0038] Furthermore, the step S31 specifically includes:

[0039] S311, using the Lucas–Kanade optical flow algorithm to determine whether there is a moving object in the image collected by the vehicle, and if so, taking images in which the moving object exists in multiple consecutive frames as images to be identified;

[0040] S312, detecting and cropping the image to be identified based on the HOG feature and the SVM algorithm model to obtain a driver area image sequence.

[0041] The present invention has the following beneficial effects:

[0042] 1. The present invention adopts edge computing to train and infer the model, which has the advantages of protecting user privacy, good real-time performance, and convenient upgrading.

[0043] 2. The present invention pre-processes the image frames during data collection and recognition, and only performs subsequent operations when certain conditions are met, thereby reducing the computational overhead of the system.

[0044] 3. The present invention adopts the mixup data enhancement method before node training. Since each vehicle node reserves a part of data, the use of mixup technology can generate a variety of samples, which greatly enhances the generalization of the model after training.

[0045] 4. The present invention improves the existing federated learning method. Since most distributed data are not independent and identically distributed, each node only performs gradient descent on local data, and the gradient direction of each node is necessarily different. The gradients of each node may offset each other after aggregation, thereby slowing down the model convergence. The present invention adds a gradient direction control constraint term to the loss function of each node, requiring the gradient direction of each node to be as close as possible to the gradient direction of the previous global model. The advantage of this is that the gradients of each node do not interfere with each other after aggregation, thereby improving the convergence speed of the model without increasing any computational overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The figure is a schematic flow chart of a method for identifying distracted driving behavior based on gradient controlled federated learning provided in an embodiment of the present invention.

[0047] Figure 2 Shown is a schematic diagram of gradient controlled federated learning of multiple vehicle nodes provided by an embodiment of the present invention.

[0048] Figure 3 The figure shows the image effect after using the mixup data enhancement method provided by an embodiment of the present invention.

[0049] Figure 4 Shown is a data distribution diagram of 10 nodes provided in an embodiment of the present invention.

[0050] Figure 5 The figure is a schematic diagram comparing the effects of the gradient-controlled federated learning and the federated learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0052] A distracted driving behavior recognition method based on gradient controlled federated learning, such as Figure 1 As shown, the following steps are included:

[0053] S1. Obtaining images of distracted driving behaviors of the driver during driving and preprocessing them to obtain a training data set, specifically including the following steps:

[0054] S11. Use the vehicle camera to obtain the driver's distracted driving behavior image. In this embodiment, the public StateFarm distracted driving behavior dataset is used to simulate the data collected by 10 vehicle nodes. The data distribution of the nodes is as follows: Figure 4 As shown;

[0055] S12, detecting and cropping the collected distracted driving behavior images, performing driver area detection based on HOG features and SVM algorithm model, and intercepting and saving the corresponding areas to obtain a driver area image sequence;

[0056] S13, labeling the driver area image sequence obtained in step S12 to obtain a training data set.

[0057] S2. Using a gradient-controlled federated learning algorithm to train the training data set obtained in step S1 to obtain a distracted driving behavior recognition model, specifically, comprising the following steps:

[0058] S21, using a vehicle as a node, using a vehicle-mounted ad hoc network to build a vehicle-to-vehicle communication network including multiple nodes, and selecting one of the nodes as an edge server;

[0059] S22. Initialize the global model using the edge server, where the global model at the tth cycle is denoted as ω t , and distribute the global model to each node;

[0060] S23. Each node performs data enhancement on the acquired image and trains the global model distributed to itself.

[0061] In this embodiment, the edge server distributes the global model to 10 nodes. The global model uses MobileNetV2, which is a lightweight model with fewer parameters and powerful performance. The input image size is 224*224*3. Each node trains the global model distributed to it, and each node trains for 5 epochs each time. The optimizer is SGD, and the learning rate is 0.1. Each node sends the trained model to the edge server within the specified time.

[0062] 100 samples of each type of distracted driving behavior are reserved in each node training set. Mixup technology is used for data enhancement before training. The principle formula of mixup technology is as follows:

[0063] λ=Beta(α,β)

[0064] Beta refers to the Beta distribution, and λ is the mixing coefficient calculated by the Beta distribution with parameters α = 2, β = 2. After using the mixup technique, a mixed-up image will be formed by superimposing the other two images.

[0065] mixed_image x =λ×image x1 +(1-λ)×image x2 ;

[0066] mixed_image y =λ×image y1 +(1-λ)×image y2 ;

[0067] Among them, mixed_image x is the image after mixup, mixed_image y is the image label after mixup. The image effect after using the mixup data enhancement method is as follows Figure 3 shown.

[0068] S24. The edge server aggregates the global models sent back by each node, uses the aggregated new model as the global model of the next cycle, and determines whether the global model converges. If not, it returns to step S22. If so, it outputs the global model.

[0069] In this embodiment, the loss function of node k is An additional gradient direction control constraint will be added, and the calculation formula is:

[0070]

[0071] where ω t+1 is the global model of the federated learning round t+1, is the local model of node k after the t+1th round of training in federated learning, is the multi-classification cross entropy loss function of node k, and γ is the constraint coefficient. cosine_distance is used to measure the cosine distance between two gradients, where 0 means the gradients are in the same direction, 1 means the gradients are orthogonal, and 2 means the gradients are in opposite directions.

[0072] The edge server aggregates the model and uses the new model obtained by aggregation as the global model of the next cycle t+1 , the specific method is:

[0073] S241. Calculate the weight of the model returned by each node to the edge server, and the calculation method is:

[0074]

[0075] S242, using the weights obtained in step S241 to calculate the global model of the next cycle, expressed as:

[0076]

[0077] where ω t+1 is the global model of the federated learning round t+1, It is the local model of node k after the t+1th round of training in federated learning.

[0078] S3, using the recognition model obtained in step S2 to identify the driver's distracted driving behavior, specifically, including the following steps:

[0079] S31, collecting images while the vehicle is driving and preprocessing the images to be recognized, and using the vehicle-mounted distracted driving behavior recognition model to classify the preprocessed driver area image sequence;

[0080] In this scheme, the Lucas–Kanade optical flow algorithm is used to determine whether there is a moving object in the image collected by the vehicle. If there is, the image with moving objects in multiple consecutive frames is used as the image to be identified, and the image to be identified is detected and cropped based on the HOG feature and SVM algorithm model to obtain the driver area image sequence.

[0081] S32: If three consecutive frames in the image sequence have the same image classification and their confidence levels are greater than a set threshold, it is determined that the driver has made a distracted driving action, and a corresponding voice reminder is issued to the driver.

[0082] In order to verify the superiority of the present invention in the federated learning algorithm, a part of the StateFarm data set is taken as a test set. When each cycle reaches step 24, the accuracy and loss value of the global model on the test set are tested, and the gradient-controlled federated learning proposed in the present invention is compared with the federated learning in terms of model performance. Figure 5 As shown, by comparing the gradient-controlled federated learning (dashed line) and the federated learning (solid line) proposed in the present invention, it is found that the accuracy of the model trained by the gradient-controlled federated learning algorithm is on average about 14% higher than the accuracy of the model trained by the federated learning algorithm, and the loss is about 14% lower. Therefore, this method can effectively improve the convergence speed of the model.

[0083] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0084] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0086] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0087] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A distracted driving behavior recognition method based on gradient controlled federated learning, characterized in that: The steps include: S1, obtaining images of distracted driving behaviors of the driver during driving and preprocessing them to obtain a training data set; S2. Use the gradient-controlled federated learning algorithm to train the training data set obtained in step S1 to obtain a distracted driving behavior recognition model, specifically including: S21, using a vehicle as a node, using a vehicle-mounted ad hoc network to build a vehicle-to-vehicle communication network including multiple nodes, and selecting one of the nodes as an edge server; S22, using the edge server to initialize the global model, where the global model in the tth round of cycles is recorded as ω t , and distribute the global model to each node; S23, each node performs data enhancement on the acquired image and trains the global model distributed to itself, where the local model after training of the kth node is recorded as , and send the trained model to the edge server. Specifically, the obtained image is enhanced using the mixup method, which is expressed as: ； ； Among them, mixed_image x Represents the image after mixup, mixed_image y Indicates the image label after mixup; , Indicates superposition Two pictures of , Indicates the labels corresponding to the two images; the mixing coefficient calculated by the λ Beta distribution, where , is the Beta distribution parameter; S24, the edge server aggregates the global models sent back by each node, and uses the aggregated new model as the global model of the next cycle to determine whether the global model converges. If not, it returns to step S22. If so, it outputs the global model. The loss function of the global model is expressed as: ； Among them, is the global model of the t+1th round of federated learning, is the local model of node k after the t+1th round of training in federated learning, is the multi-classification cross entropy loss function of node k, γ is the constraint coefficient, cosine_distance is used to measure the cosine distance between two gradients, 0 means the gradient directions are the same, 1 means the gradient directions are orthogonal, and 2 means the gradient directions are opposite; S3. Using the recognition model obtained in step S2, the driver's distracted driving behavior is recognized.

2. The method for identifying distracted driving behavior based on gradient controlled federated learning according to claim 1, characterized in that: The step S1 specifically includes: S11, using a vehicle-mounted camera to obtain an image of the driver's distracted driving behavior; S12, detecting and cropping the collected distracted driving behavior images, performing driver area detection based on HOG features and SVM algorithm model, and intercepting and saving the corresponding areas to obtain a driver area image sequence; S13, labeling the driver area image sequence obtained in step S12 to obtain a training data set.

3. The method for identifying distracted driving behavior based on gradient controlled federated learning according to claim 1, characterized in that: The step S24 specifically includes: S241. Calculate the weight of the model returned by each node to the edge server, and the calculation method is: ； S242, using the weights obtained in step S241 to calculate the global model of the next cycle, expressed as: ； in is the global model of the federated learning round t+1, It is the local model of node k after the t+1th round of training in federated learning.

4. The method for identifying distracted driving behavior based on gradient controlled federated learning according to claim 3, characterized in that: The step S3 specifically includes: S31, collecting images while the vehicle is driving and preprocessing the images to be recognized, and using the vehicle-mounted distracted driving behavior recognition model to classify the preprocessed driver area image sequence; S32: If three consecutive frames in the image sequence have the same image classification and their confidence levels are greater than a set threshold, it is determined that the driver has made a distracted driving action, and a corresponding voice reminder is issued to the driver.

5. The method for identifying distracted driving behavior based on gradient controlled federated learning according to claim 4, characterized in that: The step S31 specifically includes: S311, using the Lucas–Kanade optical flow algorithm to determine whether there is a moving object in the image collected by the vehicle, and if so, taking images in which the moving object exists in multiple consecutive frames as images to be identified; S312, detecting and cropping the image to be identified based on the HOG feature and the SVM algorithm model to obtain a driver area image sequence.

Images