A distracted driving behavior recognition method based on personalized federated learning

Abstract

The present invention discloses a method for identifying distracted driving behaviors based on personalized federated learning. First, each vehicle node collects image data while the vehicle is driving; then, a convolutional neural network model is jointly trained using personalized federated learning; finally, each node uses the model to identify distracted driving behaviors. In the driving behavior recognition scenario, the present invention uses a decentralized method based on edge computing to implement model training and reasoning, which has the advantages of protecting user privacy, good real-time performance, and convenient upgrades compared to existing centralized training and reasoning methods. Compared with traditional federated learning algorithms, the proposed personalized federated learning algorithm can make the trained model better adapt to the data of each node.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and edge computing, and in particular to a method for identifying distracted driving behavior based on personalized federated learning. Background Art

[0002] Distracted driving behavior causes a large number of traffic accidents. Currently, most of the distracted driving behavior methods based on computer vision are to improve the deep learning model, and its training and reasoning are centralized, that is, the model training and reasoning process will only be performed locally or in the cloud.

[0003] Local training and reasoning of distracted driving behavior recognition models require the pre-collection of a large number of samples, and subsequent model upgrades require redeployment, which is rather troublesome. Training and reasoning of distracted driving behavior recognition models in the cloud require extremely high network bandwidth, latency, and stability. In addition, the uploaded driving images may involve the driver's privacy.

[0004] Based on the above-mentioned problems with model training and reasoning locally and in the cloud, if the model training and reasoning are performed at the edge, that is, any computing resources and network resources between the data source and the cloud, the application of edge computing in the Internet of Vehicles can effectively solve the above problems.

[0005] Federated learning is a training strategy for edge computing. In federated learning, multiple edge nodes and an edge server form a cluster to jointly train a global model. The edge server does not have any training data. All training data is distributed on each edge node. After each training, the edge node uploads the updated local model to the edge server. The edge server aggregates the local models uploaded by each node and finally realizes the training of the global model. In this process, each node only uploads the model but not the data, so the privacy of the node user is also protected. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention provides a distracted driving behavior identification method based on personalized federated learning.

[0007] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0008] A distracted driving behavior recognition method based on personalized federated learning includes the following steps:

[0009] S1, obtaining image data collected by the vehicle node when the vehicle is traveling;

[0010] S2, pre-installing a vehicle-mounted distracted driving behavior recognition model and training it using the image data obtained in step S1;

[0011] S3. Using the model trained in step S2, identify distracted driving behavior at the vehicle node.

[0012] Furthermore, the step S1 specifically includes:

[0013] S11, obtaining distracted driving behavior images during vehicle driving, and extracting the hash fingerprint of the current frame;

[0014] S12, analyzing the driving behavior of the current frame. If the Hamming distance between the hash fingerprint of the current frame and the hash fingerprint of the previous frame is greater than a set threshold, it is determined that the driver has made a distracted behavior, and the current frame is data enhanced and saved, wherein the data enhancement includes horizontal flipping, brightness transformation, and random translation of the current frame;

[0015] S13. Label the corresponding driving behavior category for the images stored locally and use them as a training data set.

[0016] Furthermore, the step S13 specifically includes:

[0017] S131, sending the image saved in step S12 to the pre-trained distracted driving behavior recognition model, and outputting the image category;

[0018] S132: Annotate images whose confidence is greater than or equal to a set threshold, and discard images whose confidence is less than the set threshold.

[0019] Furthermore, the step S2 specifically includes:

[0020] S21, using the vehicle as a node, using the vehicle-mounted ad hoc network to build a vehicle-to-vehicle communication network, and selecting one of the nodes as an edge server;

[0021] S22: Initialize the global model using the edge server, where the global model in the tth round of cycles is denoted as ω t , and distribute the global model to each node;

[0022] S23, each node trains the global model distributed to itself, where the local model of the kth node after training is recorded as And send the trained model to the edge server;

[0023] S24, the edge server performs model aggregation, uses the aggregated new model as the global model of the next cycle, and determines whether the output global model is converged. If so, it proceeds to step S25, otherwise it returns to step S22;

[0024] S25. Each node uses local data to fine-tune the converged global model. The convolutional layer and the fully connected layer of the converged global model received by each node are adjusted using the loss function, and the remaining layers are frozen. The loss function is expressed as:

[0025]

[0026] Where L′(ω) is the new loss function, L(ω) is the multi-classification cross entropy loss function, ω is the parameter to be optimized for the last convolutional layer and the fully connected layer of the model, and ω L To fine-tune the last convolutional layer and fully connected layer of the model before fine-tuning, and ω i is the i-th element in the corresponding parameter, b i is the difficulty of changing each element, and γ is the constraint coefficient.

[0027] Furthermore, the step S24 specifically includes:

[0028] S241. Calculate the weight of the model returned by each node to the edge server, and the calculation method is:

[0029]

[0030] S242, using the weights obtained in step S241 to calculate the global model of the next cycle, expressed as:

[0031]

[0032] where ω t+1 is the global model of the federated learning round t+1, It is the local model of node k after the t+1th round of training in federated learning.

[0033] Furthermore, the step S3 specifically includes:

[0034] The step S3 specifically includes:

[0035] S31, collecting distracted driving behavior images of the driver during driving every n frames, and extracting the hash fingerprint of the current frame. If the Hamming distance between the hash fingerprint of the current frame and the hash fingerprint of the previous frame is greater than a set threshold, the current frame is added to the queue;

[0036] S32, setting a sliding window of a fixed size in the queue to be identified, and classifying the images in the sliding window using the distracted driving behavior recognition model;

[0037] S33: If the image classification results in the sliding window are the same and the confidence levels are greater than a set threshold, it is determined that the driver has made a distracted driving action and a corresponding voice prompt is given.

[0038] The present invention has the following beneficial effects:

[0039] The present invention adds additional constraints to the loss function of the model during fine-tuning. Using traditional fine-tuning methods to fine-tune the model will cause the model to "forget" data types that do not appear in the fine-tuning training data, thereby reducing the recognition rate of these data types. The constraints in the present invention constrain the difficulty of changing each element in the parameter, thereby avoiding "forgetting".

[0040] The present invention uses a personalized federated learning algorithm. The traditional federated learning algorithm focuses on obtaining a high-quality global model through distributed learning of the local data of all participating nodes. Therefore, it cannot capture the personal information of each node, resulting in a decrease in the performance of reasoning or classification. In addition, traditional federated learning requires all participating nodes to agree on a common model for collaborative training, which is unrealistic in actual complex applications. The present invention adopts a personalized federated learning algorithm. First, each node jointly trains a global model with the help of other nodes, and then each node uses local data to personalize the model to obtain the local optimal model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The figure shows the overall framework diagram of the distracted driving behavior identification method based on personalized federated learning provided by an embodiment of the present invention.

[0042] Figure 2 Shown is a schematic diagram of personalized federated learning for multiple vehicle nodes provided by an embodiment of the present invention.

[0043] Figure 3 FIG. 4 is a schematic diagram of a sliding window used in the model recognition stage provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0045] A distracted driving behavior recognition method based on personalized federated learning includes a training data collection stage, an on-board distracted driving behavior recognition model training stage, and a model recognition stage. Figure 1 As shown, the specific steps include:

[0046] S1, obtaining image data collected by the vehicle node when the vehicle is traveling;

[0047] In this embodiment, the following steps are included:

[0048] S11. When the vehicle is driving, use the on-board camera to collect images of the driver's distracted driving behavior every 3 frames, and extract the hash fingerprint of the current frame.

[0049] S12. Data enhancement is performed on the current frame that meets the conditions and saved locally. The specific conditions are: if the Hamming distance between the hash fingerprint of the current frame and the hash fingerprint of the previous frame is greater than 2, it is considered that the driver has made some action at this time. The current frame image that meets the conditions is horizontally flipped, brightness transformed, and randomly translated to complete data enhancement.

[0050] S13. When the vehicle is idle, the images stored locally are labeled with the corresponding driving behavior categories as training sets. The specific method is as follows:

[0051] S131, sending the locally stored image to a pre-trained distracted driving behavior recognition model, and outputting the image category;

[0052] S132: Annotate images whose confidence is greater than or equal to a set threshold, and discard images whose confidence is less than the set threshold.

[0053] In this embodiment, the threshold is set to 0.95. For images with a confidence level greater than or equal to the threshold, a label, i.e., a classification result, is annotated thereon; and for images with a confidence level less than the threshold, the images are discarded.

[0054] S2, pre-installing the vehicle-mounted distracted driving behavior recognition model and training it using the image data obtained in step S1, specifically, Figure 2 As shown in FIG. 1 , the training phase of the vehicle-mounted distracted driving behavior recognition model includes the following steps:

[0055] S21. When the vehicle is idle, use the vehicle as a node, use the vehicle-to-vehicle ad hoc network (VANET) to build an inter-vehicle communication network, and select one of the nodes as an edge server;

[0056] S22: Initialize the global model using the edge server, where the global model in the tth round of cycles is denoted as ω t , and distribute the global model to each node;

[0057] S23, each node trains the global model distributed to itself, where the local model of the kth node after training is recorded as And send the trained model to the edge server;

[0058] S24, the edge server performs model aggregation, uses the aggregated new model as the global model of the next cycle, and determines whether the output global model converges. If so, it proceeds to step S25, and if not, it returns to step S22, wherein using the aggregated new model as the global model of the next cycle specifically includes:

[0059] S241. Calculate the weight of the model returned by each node to the edge server, and the calculation method is:

[0060]

[0061] S242, using the weights obtained in step S241 to calculate the global model of the next cycle, expressed as:

[0062]

[0063] where ω t+1 is the global model of the federated learning round t+1, It is the local model of node k after the t+1th round of training in federated learning.

[0064] S25. Each node uses local data to fine-tune the converged global model. The convolutional layer and the fully connected layer of the converged global model received by each node are adjusted using the loss function, and the remaining layers are frozen. The loss function is expressed as:

[0065]

[0066] Where L′(ω) is the new loss function, L(ω) is the multi-classification cross entropy loss function, ω is the parameter to be optimized for the last convolutional layer and the fully connected layer of the model, and ω L To fine-tune the last convolutional layer and fully connected layer of the model before fine-tuning, and ω i is the i-th element in the corresponding parameter, b i is the difficulty of changing each element, and γ is the constraint coefficient. i The calculation of is the second-order derivative of the loss function for each element in the parameter, but the calculation amount of the second-order derivative is large, so the present invention uses the Fisher information matrix to approximate the second-order derivative.

[0067] S3, using the model trained in step S2 to identify distracted driving behavior at the vehicle node. The model identification stage includes the following steps:

[0068] S31. When the vehicle is driving, a queue of images to be identified is maintained, and images of the driver's distracted driving behavior are collected every n frames, and the hash fingerprint of the current frame is extracted. If the Hamming distance between the hash fingerprint of the previous frame and the hash fingerprint of the previous frame is greater than 2, it is considered that the driver has made some action at this time, and the current frame is added to the queue.

[0069] S32, setting a sliding window of size 3 in the queue image to be identified, and using the vehicle-mounted distracted driving behavior recognition model to classify the image in the sliding window, such as Figure 3 shown.

[0070] S33: If the image classification results in the sliding window are the same and the confidence levels are greater than a set threshold, it is determined that the driver has made a distracted driving action and a corresponding voice prompt is given.

[0071] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0072] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0073] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0074] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0075] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A distracted driving behavior recognition method based on personalized federated learning, characterized in that: The steps include: S1, obtaining image data collected by the vehicle node when the vehicle is traveling; S2, pre-installing a vehicle-mounted distracted driving behavior recognition model and training it using the image data obtained in step S1, specifically including: S21, using the vehicle as a node, using the vehicle-mounted ad hoc network to build a vehicle-to-vehicle communication network, and selecting one of the nodes as an edge server; S22, using the edge server to initialize the global model, where the global model in the tth round of cycles is recorded as ω t , and distribute the global model to each node; S23, each node trains the global model distributed to itself, where the local model of the kth node after training is recorded as , and send the trained model to the edge server; S24, the edge server aggregates the global models sent back by each node, uses the aggregated new model as the global model of the next cycle, and determines whether the output global model converges. If so, it proceeds to step S25, otherwise it returns to step S22, which specifically includes: S241. Calculate the weight of the model returned by each node to the edge server, and the calculation method is: ； S242, using the weights obtained in step S241 to calculate the global model of the next cycle, expressed as: ； in is the global model of the federated learning round t+1, is the local model of node k after the t+1th round of training in federated learning; S25. Each node uses local data to fine-tune the converged global model. The convolutional layer and the fully connected layer of the converged global model received by each node are adjusted using the loss function, and the remaining layers are frozen. The loss function is expressed as: in is the new loss function, is the multi-classification cross entropy loss function, The parameters to be optimized for the last convolutional layer and the fully connected layer of the model, To fine-tune the last convolutional layer and fully connected layer of the model before fine-tuning, and is the i-th element in the corresponding parameter, is the difficulty of changing each element, and γ is the constraint coefficient; S3. Using the model trained in step S2 to identify distracted driving behavior at the vehicle node, specifically including: S31, collecting distracted driving behavior images of the driver during driving every n frames, and extracting the hash fingerprint of the current frame. If the Hamming distance between the hash fingerprint of the current frame and the hash fingerprint of the previous frame is greater than a set threshold, the current frame is added to the queue; S32, setting a sliding window of a fixed size in the queue to be identified, and classifying the images in the sliding window using the distracted driving behavior recognition model; S33: If the image classification results in the sliding window are the same and the confidence levels are greater than a set threshold, it is determined that the driver has made a distracted driving action and a corresponding voice prompt is given.

2. The method for identifying distracted driving behavior based on personalized federated learning according to claim 1, characterized in that: The step S1 specifically includes: S11, obtaining distracted driving behavior images during vehicle driving, and extracting the hash fingerprint of the current frame; S12, analyzing the driving behavior of the current frame. If the Hamming distance between the hash fingerprint of the current frame and the hash fingerprint of the previous frame is greater than a set threshold, it is determined that the driver has made a distracted behavior, and the current frame is data enhanced and saved, wherein the data enhancement includes horizontal flipping, brightness transformation, and random translation of the current frame; S13, labeling the corresponding driving behavior categories for the images stored locally, and using them as training data sets.

3. The method for identifying distracted driving behavior based on personalized federated learning according to claim 2, characterized in that: The step S13 specifically includes: S131, sending the image saved in step S12 to the pre-trained distracted driving behavior recognition model, and outputting the image category; S132: Annotate images whose confidence is greater than or equal to a set threshold, and discard images whose confidence is less than the set threshold.

Images