Fatigue driving monitoring method, system and device based on federated learning and storage medium

Abstract

The invention discloses a federated learning-based fatigue driving monitoring method, system and device and a storage medium, and the method comprises the steps: training a cloud teacher model according to a fatigue detection data set, carrying out the knowledge distillation, obtaining a cloud student model, and transmitting the cloud student model to a vehicle-mounted terminal; the method comprises the following steps: acquiring vehicle end sample data through a vehicle-mounted terminal, dividing the vehicle end sample data into a positive sample and a negative sample through a cloud student model, and training the cloud student model according to the positive sample to obtain a vehicle end student model; extracting a negative sample feature map through the vehicle-side student model, and uploading the negative sample feature map and model parameters of the vehicle-side student model to the cloud; performing parameter aggregation on the cloud student model according to the model parameters to obtain a global student model, performing knowledge distillation on the global student model according to the negative sample feature map and the cloud teacher model to obtain a vehicle end fatigue detection model, and issuing the vehicle end fatigue detection model to the vehicle-mounted terminal. The method improves the accuracy and timeliness of fatigue driving monitoring, and can be applied to the technical field of vehicle monitoring.

Description

Technical Field

[0001] The present invention relates to the field of vehicle monitoring technology, and in particular to a method, system, device and storage medium for monitoring fatigue driving based on federated learning. Background Art

[0002] In vehicle driving scenarios, driver fatigue detection is crucial. Most vehicle-side systems use trained fatigue detection models to detect and identify the driver's facial images to obtain fatigue detection results. Existing fatigue driving monitoring solutions are generally as follows: The driver's facial images and behavioral data are captured in real time by cameras installed in the vehicle; the video stream data is framed and converted into a series of static image frames that are transmitted back to the data center; a deep learning model is used to automatically extract features from the video frames, including the driver's blinking frequency, eye movement trajectory, and head tilt angle; the model is trained using a large number of annotated datasets, and the model parameters are continuously adjusted to enable it to accurately detect the driver's fatigue state; the model's performance is evaluated on a validation set, and metrics such as accuracy, recall, and F1 score are calculated to ensure the model's reliability and effectiveness; the trained model is deployed in the vehicle system to monitor the driver's fatigue state in real time. Once fatigue is detected, the system will issue an alarm through sound, light, or seat vibration to remind the driver to rest.

[0003] However, existing fatigue driving monitoring solutions have the following defects:

[0004] 1) Security issues of data processing and storage: Data collected by on-board cameras usually needs to be uploaded to the cloud for processing, but this data transmission and storage process is vulnerable to cyber attacks or data leakage risks.

[0005] 2) Uncontrollability of data sharing: Companies may use the collected private data for commercial purposes (such as advertising recommendations, algorithm optimization, etc.), and users lack control over the use of this data.

[0006] 3) Insufficient accuracy: Using single image data for fatigue detection results in insufficient detection accuracy of the fatigue detection model, which affects the accuracy of fatigue driving monitoring.

[0007] 3) Insufficient timeliness: The computing power on the vehicle side is limited. It takes a certain amount of time to use the trained model to perform fatigue detection on the vehicle side, and there is also a certain lag in generating fatigue driving warnings. Summary of the Invention

[0008] The purpose of the present invention is to solve one of the technical problems existing in the prior art to at least a certain extent.

[0009] To this end, one purpose of an embodiment of the present invention is to provide a fatigue driving monitoring method based on federated learning, which improves the accuracy and timeliness of fatigue driving monitoring while protecting user privacy data, thereby improving driving safety.

[0010] Another object of an embodiment of the present invention is to provide a fatigue driving monitoring system based on federated learning.

[0011] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present invention include:

[0012] In a first aspect, an embodiment of the present invention provides a method for monitoring driving fatigue based on federated learning, comprising the following steps:

[0013] A cloud-based teacher model is trained based on a preset fatigue detection dataset, and knowledge distillation is performed on the cloud-based teacher model to obtain a cloud-based student model, which is then sent to the vehicle terminal.

[0014] Acquiring vehicle-side sample data through the vehicle-mounted terminal, dividing the vehicle-side sample data into positive samples and negative samples through the cloud-based student model, and training the cloud-based student model based on the positive samples to obtain a vehicle-side student model;

[0015] Extracting features of the negative samples using the vehicle-side student model to obtain a negative sample feature map, and uploading the negative sample feature map and model parameters of the vehicle-side student model to the cloud;

[0016] Performing parameter aggregation on the cloud-based student model according to the model parameters to obtain a global student model, and performing knowledge distillation on the global student model according to the negative sample feature map and the cloud-based teacher model to obtain a vehicle-side fatigue detection model, and then delivering the vehicle-side fatigue detection model to the vehicle-mounted terminal;

[0017] Real-time monitoring data is acquired through the vehicle-mounted terminal, and the real-time monitoring data is input into the vehicle-side fatigue detection model to obtain a fatigue driving detection result.

[0018] Furthermore, in one embodiment of the present invention, the cloud teacher model is trained based on a preset fatigue detection dataset, and knowledge distillation is performed on the cloud teacher model to obtain a cloud student model, which specifically includes:

[0019] Acquire the fatigue detection dataset, wherein the fatigue detection dataset includes a plurality of fatigue detection samples and corresponding true labels, wherein the fatigue detection samples include facial image sample data and physiological state sample data of the driver;

[0020] Training the initialized fatigue detection model according to the fatigue detection dataset to obtain the cloud-based teacher model;

[0021] constructing an initial student model based on the cloud-based teacher model, and determining a first soft label for each fatigue detection sample based on the cloud-based model;

[0022] Inputting the fatigue detection sample into the initial student model to obtain a first inference result;

[0023] Determine a first loss value according to the first inference result, the true label, and the first soft label;

[0024] The parameters of the initial student model are updated according to the first loss value to obtain the cloud-based student model.

[0025] Furthermore, in one embodiment of the present invention, the vehicle-side sample data is obtained through the vehicle-mounted terminal, the vehicle-side sample data is divided into positive samples and negative samples through the cloud-based student model, and the cloud-based student model is trained according to the positive samples to obtain the vehicle-side student model, which specifically includes:

[0026] Acquiring the vehicle-side sample data through the vehicle-mounted terminal, wherein the vehicle-side sample data includes the driver's facial image sample data and physiological state sample data;

[0027] Inputting the vehicle-side sample data into the cloud-based student model to obtain a second inference result and a sample confidence of the vehicle-side sample data;

[0028] Dividing the vehicle-side sample data into positive samples and negative samples according to the sample confidence, and generating pseudo labels for the positive samples according to the second inference result;

[0029] The cloud-side student model is trained according to the positive samples and the pseudo labels to obtain the vehicle-side student model.

[0030] Furthermore, in one embodiment of the present invention, extracting features from the negative samples using the vehicle-side student model to obtain a negative sample feature map, and uploading the negative sample feature map and the model parameters of the vehicle-side student model to the cloud, specifically includes:

[0031] Obtaining model parameters of the vehicle-side student model;

[0032] Inputting the negative sample into the vehicle-side student model, and extracting the negative sample feature map through the feature extraction network of the vehicle-side student model;

[0033] The negative sample feature map and the model parameters are encrypted and uploaded to the cloud.

[0034] Furthermore, in one embodiment of the present invention, the cloud-based student model is aggregated according to the model parameters to obtain a global student model, which specifically includes:

[0035] Performing weighted averaging processing on the model parameters uploaded by the plurality of vehicle-mounted terminals to obtain aggregated model parameters;

[0036] The cloud-based student model is updated with parameters according to the aggregated model parameters to obtain the global student model.

[0037] Furthermore, in one embodiment of the present invention, the knowledge distillation of the global student model is performed based on the negative sample feature map and the cloud-based teacher model to obtain a vehicle-side fatigue detection model, which specifically includes:

[0038] Determining a second soft label of the negative sample feature map according to the cloud teacher model;

[0039] Inputting the negative sample feature map into the global student model to obtain a third inference result;

[0040] Determine a second loss value according to the third inference result and the second soft label;

[0041] The parameters of the global student model are updated according to the second loss value to obtain the vehicle-side fatigue detection model.

[0042] Furthermore, in one embodiment of the present invention, the real-time monitoring data is acquired through the vehicle-mounted terminal, and the real-time monitoring data is input into the vehicle-side fatigue detection model to obtain the fatigue driving detection result, which specifically includes:

[0043] Acquiring the real-time monitoring data through the vehicle-mounted terminal, wherein the real-time monitoring data includes the driver's real-time facial image data and real-time physiological status data;

[0044] Inputting the real-time monitoring data into the vehicle-side fatigue detection model to obtain the fatigue driving detection result;

[0045] The corresponding early warning measures are triggered according to the fatigue driving detection result.

[0046] In a second aspect, an embodiment of the present invention provides a fatigue driving monitoring system based on federated learning, including:

[0047] A cloud model training module is used to train a cloud teacher model based on a preset fatigue detection dataset, perform knowledge distillation on the cloud teacher model to obtain a cloud student model, and then send the cloud student model to the vehicle terminal;

[0048] a vehicle-side model training module, configured to obtain vehicle-side sample data through the vehicle-mounted terminal, divide the vehicle-side sample data into positive samples and negative samples through the cloud-based student model, and train the cloud-based student model based on the positive samples to obtain a vehicle-side student model;

[0049] A vehicle-side model extraction module is used to extract features of the negative samples using the vehicle-side student model to obtain a negative sample feature map, and upload the negative sample feature map and the model parameters of the vehicle-side student model to the cloud;

[0050] a cloud-based model updating module, configured to aggregate the cloud-based student model parameters according to the model parameters to obtain a global student model, perform knowledge distillation on the global student model according to the negative sample feature map and the cloud-based teacher model to obtain a vehicle-side fatigue detection model, and then transmit the vehicle-side fatigue detection model to the vehicle-mounted terminal;

[0051] The vehicle-side fatigue detection module is used to obtain real-time monitoring data through the vehicle-mounted terminal, input the real-time monitoring data into the vehicle-side fatigue detection model, and obtain fatigue driving detection results.

[0052] In a third aspect, an embodiment of the present invention provides a fatigue driving monitoring device based on federated learning, comprising:

[0053] at least one processor;

[0054] at least one memory for storing at least one program;

[0055] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned fatigue driving monitoring method based on federated learning.

[0056] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium storing a program executable by a processor, wherein the program executable by the processor is used to execute the above-mentioned method for fatigue driving monitoring based on federated learning when executed by the processor.

[0057] The advantages and benefits of the present invention will be described in part in the following description and will become apparent from the following description or learned through practice of the present invention:

[0058] The embodiment of the present invention trains a cloud-based teacher model based on a preset fatigue detection data set, performs knowledge distillation on the cloud-based teacher model to obtain a cloud-based student model, and then sends the cloud-based student model to the vehicle-mounted terminal, obtains vehicle-side sample data through the vehicle-mounted terminal, divides the vehicle-side sample data into positive samples and negative samples through the cloud-based student model, and trains the cloud-based student model based on the positive samples to obtain a vehicle-side student model, performs feature extraction on negative samples through the vehicle-side student model to obtain a negative sample feature map, and uploads the negative sample feature map and the model parameters of the vehicle-side student model to the cloud, performs parameter aggregation on the cloud-based student model based on the model parameters to obtain a global student model, and performs knowledge distillation on the global student model based on the negative sample feature map and the cloud-based teacher model to obtain a vehicle-side fatigue detection model, and then sends the vehicle-side fatigue detection model to the vehicle-mounted terminal, obtains real-time monitoring data through the vehicle-mounted terminal, and inputs the real-time monitoring data into the vehicle-side fatigue detection model to obtain fatigue driving detection results. The embodiment of the present invention obtains a high-precision cloud-based teacher model through training of a high-quality fatigue detection data set in the cloud, obtains a cloud-based student model through lightweight processing through knowledge distillation and sends it to multiple vehicle-mounted terminals, and uses the cloud-based student model to divide the collected vehicle-side sample data into positive and negative samples at the vehicle-mounted terminal, obtains a vehicle-side student model through unsupervised learning training, and simultaneously uses the vehicle-side student model to extract a negative sample feature map, uploads the negative sample feature map and the model parameters of the vehicle-side student model to the cloud, and performs parameter aggregation on the cloud-based student model according to the model parameters uploaded by multiple vehicle-mounted terminals in the cloud to obtain a global student model, and then uses the negative sample feature maps uploaded by multiple vehicle-mounted terminals and the high-precision cloud-based teacher model to perform knowledge distillation on the global student model again, so that it learns the discriminant knowledge of the cloud-based teacher model for the negative sample feature maps, and finally obtains a high-precision and lightweight vehicle-side fatigue detection model. The vehicle-side fatigue detection model can be deployed on the vehicle-mounted terminal to realize real-time monitoring of fatigue driving. It can be recognized that the embodiment of the present invention obtains a lightweight cloud-side student model and a vehicle-side fatigue detection model through knowledge distillation, which reduces the requirements for vehicle-side computing power in both the model training and model application stages, and improves the timeliness of fatigue driving monitoring; the cloud-side student model is trained in an unsupervised learning manner on the vehicle side, which not only avoids manual labeling of vehicle-side sample data, but also can screen out negative samples that are difficult for the cloud-side student model to distinguish for subsequent cloud-side model updates, and the vehicle side only uploads the negative sample feature map to the cloud, avoiding the sharing of the original vehicle-side data and ensuring the security of user privacy data; the cloud-side student model is aggregated using the model parameters uploaded by the on-board terminal on the cloud, and the global student model is again subjected to knowledge distillation using the negative sample feature map and the cloud-side teacher model, and finally a vehicle-side fatigue detection model that can distinguish various types of vehicle-side samples is obtained, thereby improving the accuracy of fatigue driving monitoring and the user's driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following introduction is made to the drawings required for use in the embodiments of the present invention. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0060] Figure 1 A flowchart of a method for monitoring driver fatigue based on federated learning provided in an embodiment of the present invention;

[0061] Figure 2 A schematic diagram of the training process of the vehicle-side fatigue detection model provided in an embodiment of the present invention;

[0062] Figure 3 A structural block diagram of a fatigue driving monitoring system based on federated learning provided in an embodiment of the present invention;

[0063] Figure 4 This is a structural block diagram of a federated learning-based driving fatigue monitoring device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided for ease of explanation only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0065] In the description of the present invention, "a plurality" means two or more. The terms "first" and "second" are used solely to distinguish technical features and are not to be construed as indicating or implying relative importance, or as implicitly indicating the number of the indicated technical features, or as implicitly indicating the order of the indicated technical features. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art.

[0066] Reference Figure 1 The embodiment of the present invention provides a method for monitoring driving fatigue based on federated learning, which specifically includes the following steps:

[0067] S101: Training a cloud-based teacher model based on a preset fatigue detection dataset, performing knowledge distillation on the cloud-based teacher model to obtain a cloud-based student model, and then delivering the cloud-based student model to the in-vehicle terminal;

[0068] S102, obtaining vehicle-side sample data through the vehicle terminal, dividing the vehicle-side sample data into positive samples and negative samples using the cloud-based student model, and training the cloud-based student model based on the positive samples to obtain the vehicle-side student model;

[0069] S103: extract features of the negative samples using the vehicle-side student model to obtain a negative sample feature map, and upload the negative sample feature map and model parameters of the vehicle-side student model to the cloud;

[0070] S104: Aggregate the cloud-based student model based on the model parameters to obtain a global student model. Perform knowledge distillation on the global student model based on the negative sample feature map and the cloud-based teacher model to obtain a vehicle-side fatigue detection model. The vehicle-side fatigue detection model is then distributed to the vehicle terminal.

[0071] S105. Acquire real-time monitoring data through the vehicle-mounted terminal, input the real-time monitoring data into the vehicle-side fatigue detection model, and obtain fatigue driving detection results.

[0072] like Figure 2 FIG. 1 is a schematic diagram of the training process of the vehicle-side fatigue detection model provided by an embodiment of the present invention, which is as follows:

[0073] 1) A high-precision cloud teacher model is trained on a high-quality fatigue detection dataset in the cloud. This model is then lightweighted through knowledge distillation to obtain a cloud student model, which is then distributed to multiple in-vehicle terminals (1 to N).

[0074] 2) Using the cloud-based student model on the vehicle terminal (1 to N), the collected vehicle-side sample data (1 to N) is divided into positive and negative samples, and the vehicle-side student model (1 to N) is obtained through unsupervised learning training. At the same time, the vehicle-side student model (1 to N) is used to extract the negative sample feature map (1 to N), and the negative sample feature map (1 to N) and the model parameters (1 to N) of the vehicle-side student model are uploaded to the cloud;

[0075] 3) In the cloud, the cloud student model is aggregated based on the model parameters (1 to N) uploaded by multiple vehicle terminals to obtain a global student model. Then, the global student model is further subjected to knowledge distillation using the negative sample feature maps (1 to N) uploaded by multiple vehicle terminals and the high-precision cloud teacher model. This allows the global student model to learn the discriminative knowledge of the cloud teacher model for the negative sample feature maps (1 to N). The iteratively optimized cloud student model is then distributed to each vehicle terminal (1 to N).

[0076] 4) Repeat steps 2) and 3) above until the preset convergence conditions are reached (the number of iterations reaches the preset value, the loss value is lower than the preset value, etc.), and output the cloud-based student model after the last iterative optimization to obtain a high-precision, lightweight vehicle-side fatigue detection model. Deploying this vehicle-side fatigue detection model on the vehicle terminal can realize real-time monitoring of fatigue driving.

[0077] It can be recognized that the embodiment of the present invention obtains a lightweight cloud-side student model and a vehicle-side fatigue detection model through knowledge distillation, which reduces the requirements for vehicle-side computing power in both the model training and model application stages, and improves the timeliness of fatigue driving monitoring; the cloud-side student model is trained in an unsupervised learning manner on the vehicle side, which not only avoids manual labeling of vehicle-side sample data, but also can screen out negative samples that are difficult for the cloud-side student model to distinguish for subsequent cloud-side model updates, and the vehicle side only uploads the negative sample feature map to the cloud, avoiding the sharing of the original vehicle-side data and ensuring the security of user privacy data; the cloud-side student model is aggregated using the model parameters uploaded by the on-board terminal on the cloud, and the global student model is again subjected to knowledge distillation using the negative sample feature map and the cloud-side teacher model, and finally a vehicle-side fatigue detection model that can distinguish various types of vehicle-side samples is obtained, thereby improving the accuracy of fatigue driving monitoring and the user's driving safety.

[0078] As an optional implementation, a cloud teacher model is trained based on a preset fatigue detection dataset, and knowledge distillation is performed on the cloud teacher model to obtain a cloud student model, which specifically includes:

[0079] S1011. Obtain a fatigue detection dataset, where the fatigue detection dataset includes multiple fatigue detection samples and corresponding true labels, and the fatigue detection samples include driver's facial image sample data and physiological state sample data;

[0080] S1012. Train the initialized fatigue detection model according to the fatigue detection dataset to obtain a cloud-based teacher model.

[0081] S1013. Construct an initial student model based on the cloud teacher model, and determine a first soft label for each fatigue detection sample based on the cloud model;

[0082] S1014, inputting the fatigue detection sample into the initial student model to obtain a first inference result;

[0083] S1015. Determine a first loss value according to the first inference result, the true label, and the first soft label;

[0084] S1016. Update the parameters of the initial student model according to the first loss value to obtain a cloud-based student model.

[0085] Specifically, a high-quality fatigue detection dataset is obtained, and a multimodal fatigue detection model is constructed in the cloud. The fatigue detection dataset is input into the fatigue detection model for training to obtain a high-precision cloud-based teacher model. The cloud-based teacher model has many network parameters and complex operations, so it needs to be lightweight before it can be deployed on the terminal; an initial student model is constructed based on the cloud-based teacher model, and the first soft label of each fatigue detection sample is determined based on the cloud-based model. It should be noted that, unlike the true label that points to a unique prediction result, the soft label corresponds to the probability distribution information of the prediction result of the fatigue detection sample; the fatigue detection sample is input into the initial student model to obtain a first inference result, and the cross entropy loss between the first inference result and the true label, as well as the KL divergence loss between the first inference result and the first soft label are calculated. Then, a weighted sum is taken to obtain a first loss value. The parameters of the initial student model are updated with the goal of minimizing the first loss value, and the teacher knowledge of the cloud-based teacher model and the supervision of the true label are balanced to obtain a cloud-based student model.

[0086] As an optional implementation method, vehicle-side sample data is obtained through the vehicle terminal, the vehicle-side sample data is divided into positive samples and negative samples through the cloud-based student model, and the cloud-based student model is trained based on the positive samples to obtain the vehicle-side student model, which specifically includes:

[0087] S1021. Obtain vehicle-side sample data through the vehicle-mounted terminal, where the vehicle-side sample data includes driver's facial image sample data and physiological status sample data;

[0088] S1022. Input the vehicle-side sample data into the cloud-based student model to obtain a second inference result and sample confidence of the vehicle-side sample data;

[0089] S1023. Divide the vehicle-side sample data into positive samples and negative samples according to the sample confidence, and generate pseudo labels for the positive samples according to the second inference result;

[0090] S1024. Train the cloud-side student model based on the positive samples and pseudo labels to obtain the vehicle-side student model.

[0091] Specifically, vehicle-side sample data, including facial image sample data and physiological status sample data of the driver, is collected at the vehicle terminal. The cloud-based learning model is then subjected to unsupervised learning using the vehicle-side sample data. Specifically, the vehicle-side sample data is input into the cloud-based student model to obtain the second inference result and sample confidence of the vehicle-side sample data. Vehicle-side sample data with a sample confidence not lower than a preset threshold are selected as positive samples. The pseudo-label of the positive sample is determined based on the corresponding second inference result. The cloud-based student model is then trained to obtain the vehicle-side student model. It should be noted that, unlike the aforementioned soft labels, pseudo-labels, like true labels, point to unique prediction results. However, they are essentially labels inferred by the model rather than manually labeled true labels, hence the name pseudo-label.

[0092] As an optional implementation, feature extraction of negative samples is performed on the vehicle-side student model to obtain a negative sample feature map, and the negative sample feature map and the model parameters of the vehicle-side student model are uploaded to the cloud, which specifically includes:

[0093] S1031. Obtain model parameters of the vehicle-side student model;

[0094] S1032. Input the negative sample into the vehicle-side student model, and extract the negative sample feature map through the feature extraction network of the vehicle-side student model;

[0095] S1033: Encrypt the negative sample feature map and model parameters and upload them to the cloud.

[0096] Specifically, after obtaining the vehicle-side student model, the corresponding model parameters are extracted, and the feature extraction network of the vehicle-side student model is used to extract the negative sample feature map of the negative sample (vehicle-side sample data whose sample confidence is lower than the preset threshold). The negative sample feature map and model parameters are encrypted and uploaded to the cloud. The model parameters will be used to perform parameter aggregation on the cloud-side student model in the cloud, and the negative sample feature map will serve as sample data that is difficult for the cloud-side student model to distinguish. After the cloud-side teacher model makes the distinction, the corresponding knowledge will be passed to the cloud-side student model through knowledge distillation to improve the prediction accuracy of the cloud-side student model.

[0097] As an optional implementation, the cloud-based student model is aggregated according to the model parameters to obtain a global student model, which specifically includes:

[0098] S1041. Perform weighted averaging on the model parameters uploaded by multiple vehicle-mounted terminals to obtain aggregated model parameters.

[0099] S1042. Update the parameters of the cloud-based student model according to the aggregated model parameters to obtain a global student model.

[0100] Specifically, after the cloud obtains the model parameters uploaded by each participating vehicle terminal, it performs a weighted average of the various parameters to obtain the aggregated model parameters. These aggregated model parameters are then used to update the cloud-based student model to obtain the global student model. It can be seen that while this global student model is capable of distinguishing positive samples from each vehicle terminal, it is still unable to distinguish negative samples from each vehicle terminal, and therefore requires further optimization.

[0101] As an optional implementation, knowledge distillation is performed on the global student model based on the negative sample feature map and the cloud-based teacher model to obtain a vehicle-side fatigue detection model, which specifically includes:

[0102] S1043. Determine a second soft label of the negative sample feature map according to the cloud teacher model;

[0103] S1044. Input the negative sample feature map into the global student model to obtain a third inference result;

[0104] S1045. Determine a second loss value based on the third inference result and the second soft label;

[0105] S1046. Update the parameters of the global student model according to the second loss value to obtain a vehicle-side fatigue detection model.

[0106] Specifically, the negative sample feature map is directly input into the cloud-based teacher model for discrimination, and the second soft label (probability distribution of the prediction result) corresponding to the negative sample feature map can be obtained. The second soft label contains the cloud-based teacher model's discriminative knowledge of the negative sample; the negative sample feature map is input into the global student model to obtain the third inference result, and the KL divergence loss of the third inference result and the second soft label can be calculated to obtain the second loss value. The parameters of the global student model are updated with the goal of minimizing the second loss value, and the iteratively optimized cloud-based student model with discriminative knowledge of negative samples can be obtained. If this iteration reaches the preset convergence condition, the vehicle-side fatigue detection model can be output.

[0107] As an optional implementation, real-time monitoring data is obtained through the vehicle terminal, and the real-time monitoring data is input into the vehicle-side fatigue detection model to obtain fatigue driving detection results, which specifically includes:

[0108] S1051. Acquire real-time monitoring data through the vehicle-mounted terminal, where the real-time monitoring data includes real-time facial image data and real-time physiological status data of the driver;

[0109] S1052: Input the real-time monitoring data into the vehicle-side fatigue detection model to obtain fatigue driving detection results;

[0110] S1053. Trigger corresponding warning measures based on the fatigue driving detection result.

[0111] Specifically, after obtaining the vehicle-side fatigue detection model, it can be deployed on the vehicle terminal for fatigue driving monitoring. While having high precision, it also protects user privacy data, and the lightweight model also improves the timeliness of fatigue driving monitoring.

[0112] The above describes the method steps of the embodiment of the present invention. In the embodiment of the present invention, a high-precision cloud teacher model is obtained by training a high-quality fatigue detection data set in the cloud, and a cloud student model is obtained by lightweight processing it through knowledge distillation and sent to multiple vehicle-mounted terminals. The cloud student model is used at the vehicle-mounted terminal to divide the collected vehicle-side sample data into positive and negative samples, and the vehicle-side student model is obtained through unsupervised learning training. At the same time, the vehicle-side student model is used to extract the negative sample feature map, and the negative sample feature map and the model parameters of the vehicle-side student model are uploaded to the cloud. In the cloud, the cloud student model is aggregated according to the model parameters uploaded by multiple vehicle-mounted terminals to obtain a global student model, and then the negative sample feature maps uploaded by multiple vehicle-mounted terminals and the high-precision cloud teacher model are used to perform knowledge distillation on the global student model again, so that it learns the discriminant knowledge of the cloud teacher model for the negative sample feature map, and finally a high-precision and lightweight vehicle-side fatigue detection model can be obtained. The vehicle-side fatigue detection model can be deployed on the vehicle terminal to realize real-time monitoring of fatigue driving. It can be recognized that the embodiment of the present invention obtains a lightweight cloud-side student model and a vehicle-side fatigue detection model through knowledge distillation, which reduces the requirements for vehicle-side computing power in both the model training and model application stages, and improves the timeliness of fatigue driving monitoring; the cloud-side student model is trained in an unsupervised learning manner on the vehicle side, which not only avoids manual labeling of vehicle-side sample data, but also can screen out negative samples that are difficult for the cloud-side student model to distinguish for subsequent cloud-side model updates, and the vehicle side only uploads the negative sample feature map to the cloud, avoiding the sharing of the original vehicle-side data and ensuring the security of user privacy data; the cloud-side student model is aggregated using the model parameters uploaded by the on-board terminal on the cloud, and the global student model is again subjected to knowledge distillation using the negative sample feature map and the cloud-side teacher model, and finally a vehicle-side fatigue detection model that can distinguish various types of vehicle-side samples is obtained, thereby improving the accuracy of fatigue driving monitoring and the user's driving safety.

[0113] Reference Figure 3 , an embodiment of the present invention provides a fatigue driving monitoring system based on federated learning, comprising:

[0114] The cloud model training module is used to train a cloud teacher model based on a preset fatigue detection dataset, perform knowledge distillation on the cloud teacher model, obtain a cloud student model, and then send the cloud student model to the vehicle terminal;

[0115] The vehicle-side model training module is used to obtain vehicle-side sample data through the vehicle terminal, divide the vehicle-side sample data into positive samples and negative samples using the cloud-based student model, and train the cloud-based student model based on the positive samples to obtain the vehicle-side student model;

[0116] The vehicle-side model extraction module is used to extract features of negative samples using the vehicle-side student model to obtain negative sample feature maps, and upload the negative sample feature maps and model parameters of the vehicle-side student model to the cloud;

[0117] The cloud model update module is used to aggregate the cloud student model parameters based on the model parameters to obtain a global student model. It then performs knowledge distillation on the global student model based on the negative sample feature map and the cloud teacher model to obtain a vehicle-side fatigue detection model, which is then distributed to the vehicle terminal.

[0118] The vehicle-side fatigue detection module is used to obtain real-time monitoring data through the vehicle-mounted terminal, input the real-time monitoring data into the vehicle-side fatigue detection model, and obtain fatigue driving detection results.

[0119] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0120] Reference Figure 4 , an embodiment of the present invention provides a fatigue driving monitoring device based on federated learning, comprising:

[0121] at least one processor;

[0122] at least one memory for storing at least one program;

[0123] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned method for monitoring fatigue driving based on federated learning.

[0124] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0125] An embodiment of the present invention also provides a computer-readable storage medium, which stores a program executable by a processor. When the program is executed by the processor, it is used to execute the above-mentioned fatigue driving monitoring method based on federated learning.

[0126] A computer-readable storage medium in an embodiment of the present invention can execute a fatigue driving monitoring method based on federated learning provided by an embodiment of the method of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0127] The embodiment of the present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs Figure 1 The method shown.

[0128] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the above-mentioned boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0129] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the above-mentioned functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the routine skills of an engineer. Therefore, a person skilled in the art can implement the present invention set forth in the claims using ordinary skills without undue experimentation. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0130] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the above methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0131] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0132] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable media on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0133] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0134] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0135] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0136] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for monitoring driving fatigue based on federated learning, characterized in that: The following steps are involved: A cloud-based teacher model is trained based on a preset fatigue detection dataset, and knowledge distillation is performed on the cloud-based teacher model to obtain a cloud-based student model, which is then sent to the vehicle terminal. Acquiring vehicle-side sample data through the vehicle-mounted terminal, dividing the vehicle-side sample data into positive samples and negative samples through the cloud-based student model, and training the cloud-based student model based on the positive samples to obtain a vehicle-side student model; Extracting features of the negative samples using the vehicle-side student model to obtain a negative sample feature map, and uploading the negative sample feature map and model parameters of the vehicle-side student model to the cloud; Performing parameter aggregation on the cloud-based student model according to the model parameters to obtain a global student model, and performing knowledge distillation on the global student model according to the negative sample feature map and the cloud-based teacher model to obtain a vehicle-side fatigue detection model, and then delivering the vehicle-side fatigue detection model to the vehicle-mounted terminal; Real-time monitoring data is acquired through the vehicle-mounted terminal, and the real-time monitoring data is input into the vehicle-side fatigue detection model to obtain a fatigue driving detection result.

2. The method for monitoring driving fatigue based on federated learning according to claim 1, characterized in that: The cloud teacher model is trained according to a preset fatigue detection dataset, and knowledge distillation is performed on the cloud teacher model to obtain a cloud student model, which specifically includes: Acquire the fatigue detection dataset, wherein the fatigue detection dataset includes a plurality of fatigue detection samples and corresponding true labels, wherein the fatigue detection samples include facial image sample data and physiological state sample data of the driver; Training the initialized fatigue detection model according to the fatigue detection dataset to obtain the cloud-based teacher model; constructing an initial student model based on the cloud-based teacher model, and determining a first soft label for each fatigue detection sample based on the cloud-based model; Inputting the fatigue detection sample into the initial student model to obtain a first inference result; Determine a first loss value according to the first inference result, the true label, and the first soft label; The parameters of the initial student model are updated according to the first loss value to obtain the cloud-based student model.

3. The method for monitoring driving fatigue based on federated learning according to claim 1, characterized in that: The vehicle-side sample data is obtained through the vehicle-mounted terminal, the vehicle-side sample data is divided into positive samples and negative samples through the cloud-based student model, and the cloud-based student model is trained according to the positive samples to obtain the vehicle-side student model, which specifically includes: Acquiring the vehicle-side sample data through the vehicle-mounted terminal, wherein the vehicle-side sample data includes the driver's facial image sample data and physiological state sample data; Inputting the vehicle-side sample data into the cloud-based student model to obtain a second inference result and a sample confidence of the vehicle-side sample data; Dividing the vehicle-side sample data into positive samples and negative samples according to the sample confidence, and generating pseudo labels for the positive samples according to the second inference result; The cloud-side student model is trained according to the positive samples and the pseudo labels to obtain the vehicle-side student model.

4. The method for monitoring driving fatigue based on federated learning according to claim 1, characterized in that: The extracting features of the negative samples by the vehicle-side student model to obtain a negative sample feature map, and uploading the negative sample feature map and the model parameters of the vehicle-side student model to the cloud specifically includes: Obtaining model parameters of the vehicle-side student model; Inputting the negative sample into the vehicle-side student model, and extracting the negative sample feature map through the feature extraction network of the vehicle-side student model; The negative sample feature map and the model parameters are encrypted and uploaded to the cloud.

5. The method for monitoring driving fatigue based on federated learning according to claim 1, characterized in that: The step of aggregating the parameters of the cloud-based student model according to the model parameters to obtain a global student model specifically includes: Performing weighted averaging processing on the model parameters uploaded by the plurality of vehicle-mounted terminals to obtain aggregated model parameters; The cloud-based student model is updated with parameters according to the aggregated model parameters to obtain the global student model.

6. The method for monitoring driving fatigue based on federated learning according to claim 1, characterized in that: The method of performing knowledge distillation on the global student model based on the negative sample feature map and the cloud-based teacher model to obtain a vehicle-side fatigue detection model specifically includes: Determining a second soft label of the negative sample feature map according to the cloud teacher model; Inputting the negative sample feature map into the global student model to obtain a third inference result; Determine a second loss value according to the third inference result and the second soft label; The parameters of the global student model are updated according to the second loss value to obtain the vehicle-side fatigue detection model.

7. A method for monitoring driving fatigue based on federated learning according to any one of claims 1 to 6, characterized in that: The step of acquiring real-time monitoring data through the vehicle-mounted terminal and inputting the real-time monitoring data into the vehicle-side fatigue detection model to obtain fatigue driving detection results specifically includes: Acquiring the real-time monitoring data through the vehicle-mounted terminal, wherein the real-time monitoring data includes the driver's real-time facial image data and real-time physiological status data; Inputting the real-time monitoring data into the vehicle-side fatigue detection model to obtain the fatigue driving detection result; The corresponding early warning measures are triggered according to the fatigue driving detection result.

8. A fatigue driving monitoring system based on federated learning, characterized in that: include: A cloud model training module is used to train a cloud teacher model based on a preset fatigue detection dataset, perform knowledge distillation on the cloud teacher model to obtain a cloud student model, and then send the cloud student model to the vehicle terminal; a vehicle-side model training module, configured to obtain vehicle-side sample data through the vehicle-mounted terminal, divide the vehicle-side sample data into positive samples and negative samples through the cloud-based student model, and train the cloud-based student model based on the positive samples to obtain a vehicle-side student model; A vehicle-side model extraction module is used to extract features of the negative samples using the vehicle-side student model to obtain a negative sample feature map, and upload the negative sample feature map and the model parameters of the vehicle-side student model to the cloud; a cloud-based model updating module, configured to aggregate the cloud-based student model parameters according to the model parameters to obtain a global student model, perform knowledge distillation on the global student model according to the negative sample feature map and the cloud-based teacher model to obtain a vehicle-side fatigue detection model, and then transmit the vehicle-side fatigue detection model to the vehicle-mounted terminal; The vehicle-side fatigue detection module is used to obtain real-time monitoring data through the vehicle-mounted terminal, input the real-time monitoring data into the vehicle-side fatigue detection model, and obtain fatigue driving detection results.

9. A fatigue driving monitoring device based on federated learning, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the fatigue driving monitoring method based on federated learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to execute the fatigue driving monitoring method based on federated learning as described in any one of claims 1 to 7 when executed by the processor.

Images