Driver fatigue detection method and device and electronic equipment

Abstract

The invention provides a driver fatigue detection method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining driving data of a vehicle and physiological data of a driver of the vehicle; performing fatigue detection processing through a fatigue detection model according to the driving data and the physiological data to obtain a fatigue state of the driver; wherein the model parameters of the fatigue detection model are obtained by performing federal learning according to local fatigue data samples of a plurality of sample vehicles, and the local fatigue data samples comprise sample driving data, sample physiological data and sample fatigue states. According to the invention, the fatigue detection precision and the data security in the fatigue detection process can be improved.

Description

Technical Field

[0001] The present application relates to vehicle technology, and in particular to a driver fatigue detection method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] With the development of computer technology, the research on driver fatigue detection has become more and more in-depth. One of the hot topics nowadays is to realize fatigue detection based on machine learning (ML). Machine learning is the core of artificial intelligence, which specializes in studying how computers simulate or realize human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance.

[0003] In the solutions provided by related technologies, a centralized training method is usually used to train machine learning models, and fatigue detection is achieved through the trained machine learning models. Centralized training refers to concentrating training data obtained from different channels on a server to train the machine learning model, and finally deploying the trained machine learning model to each vehicle for use. However, this easily leads to difficulties in collecting some sensitive data, as well as user privacy leakage caused by data leakage during transmission, that is, the data security of the solutions provided by related technologies is low. Summary of the invention

[0004] The embodiments of the present application provide a driver fatigue detection method, device, electronic device, computer-readable storage medium and computer program product, which can improve data security.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The embodiment of the present application provides a driver fatigue detection method, which is applied to a vehicle, and the method includes:

[0007] Acquiring driving data of the vehicle and physiological data of the driver of the vehicle;

[0008] Performing fatigue detection processing according to the driving data and the physiological data through a fatigue detection model to obtain the fatigue state of the driver;

[0009] The model parameters of the fatigue detection model are obtained by federated learning based on fatigue data samples local to a plurality of sample vehicles, wherein the local fatigue data samples include sample driving data, sample physiological data and sample fatigue status.

[0010] The embodiment of the present application provides a driver fatigue detection method, which is applied to a server, and the method includes:

[0011] Performing federated learning based on fatigue data samples of multiple sample vehicles to obtain model parameters of the fatigue detection model; wherein the local fatigue data samples include sample driving data, sample physiological data, and sample fatigue status;

[0012] The model parameters of the fatigue detection model are sent to the target vehicle so that the target vehicle obtains the driving data of the target vehicle and the physiological data of the driver of the target vehicle, and the fatigue detection model performs fatigue detection processing according to the driving data and the physiological data to obtain the fatigue state of the driver.

[0013] The embodiment of the present application provides a driver fatigue detection device, which is applied to a vehicle, and the device includes:

[0014] An acquisition module, used to acquire the driving data of the vehicle and the physiological data of the driver of the vehicle;

[0015] A fatigue detection module, used to perform fatigue detection processing according to the driving data and the physiological data through a fatigue detection model to obtain the fatigue state of the driver;

[0016] The model parameters of the fatigue detection model are obtained by federated learning based on fatigue data samples local to a plurality of sample vehicles, wherein the local fatigue data samples include sample driving data, sample physiological data and sample fatigue status.

[0017] The embodiment of the present application provides a driver fatigue detection device, which is applied to a server, and the device includes:

[0018] A learning module, used for performing federated learning based on fatigue data samples of multiple sample vehicles to obtain model parameters of a fatigue detection model; wherein the local fatigue data samples include sample driving data, sample physiological data and sample fatigue status;

[0019] A sending module is used to send the model parameters of the fatigue detection model to a target vehicle so that the target vehicle obtains the driving data of the target vehicle and the physiological data of the driver of the target vehicle, and performs fatigue detection processing according to the driving data and the physiological data through the fatigue detection model to obtain the fatigue status of the driver.

[0020] An embodiment of the present application provides an electronic device, including:

[0021] A memory for storing executable instructions;

[0022] The processor is used to implement the driver fatigue detection method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0023] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the driver fatigue detection method provided in the embodiment of the present application.

[0024] An embodiment of the present application provides a computer program product, which includes executable instructions for causing a processor to execute and implement the driver fatigue detection method provided in the embodiment of the present application.

[0025] The embodiments of the present application have the following beneficial effects:

[0026] The server performs federated learning based on the local fatigue data samples of multiple sample vehicles to obtain the model parameters of the fatigue detection model. In this way, based on the federated learning method, the server can achieve learning without obtaining the local fatigue data samples of the sample vehicles, effectively improving data security. At the same time, in the process of federated learning and fatigue detection, both driving data and physiological data are utilized, which can improve the richness of feature information, thereby improving the learning effect of federated learning and the accuracy of fatigue detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 It is a schematic diagram of the architecture of a driver fatigue detection system provided in an embodiment of the present application;

[0029] Figure 2A is a structural diagram of a terminal device provided in an embodiment of the present application;

[0030] Figure 2B It is a structural diagram of a server provided in an embodiment of the present application;

[0031] Figure 3A It is a flowchart of a driver fatigue detection method provided in an embodiment of the present application;

[0032] Figure 3B is another flowchart of a driver fatigue detection method provided in an embodiment of the present application;

[0033] Figure 3C is another flowchart of a driver fatigue detection method provided in an embodiment of the present application;

[0034] Figure 3Dis another flowchart of a driver fatigue detection method provided in an embodiment of the present application;

[0035] Figure 4 This is another flowchart of the driver fatigue detection method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0037] In the following description, reference is made to "some embodiments", which describes a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. In the following description, reference is made to the term "multiple" to mean at least two.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0039] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0040] 1) Fatigue detection model: A machine learning model used to implement fatigue detection. The input of the fatigue detection model is driving data and physiological data, and the output is the fatigue status. The fatigue detection model can be a classification model, that is, the output fatigue status is a discrete value, such as "fatigue" or "awake"; the fatigue detection model can also be a regression model, that is, the output fatigue status is a continuous value, such as a value in the numerical interval [1, 10]. The larger the value, the more tired (or awake) the driver is.

[0041] 2) Driving data: refers to the data collected from the vehicle driven by the driver, which is used to indicate the driving conditions of the vehicle.

[0042] 3) Physiological data: refers to the data obtained by collecting data from the driver, which is used to represent the driver's physiological condition.

[0043] 4) Model parameters: refers to the configuration variables inside the fatigue detection model, which can be updated through training. Model parameters have the following characteristics: model parameters are required for model prediction; model parameters can define model functions; model parameters can be obtained through data estimation or data learning (i.e. training); model parameters are usually saved as part of the model. Taking the case where the fatigue detection model is a neural network model, the model parameters can refer to the weight parameters of the network layer.

[0044] In the embodiment of the present application, the model parameters can be divided into global model parameters and local model parameters. The global model parameters are generated on the server side, and the local model parameters are generated locally in the vehicle.

[0045] 5) Standardization: It is used to remove the influence of dimension and convert dimensional values ​​into dimensionless values. For example, standardization can transform data into a distribution with a mean of 0 and a standard deviation of 1, while retaining the original data distribution after the transformation.

[0046] The embodiments of the present application provide a driver fatigue detection method, device, electronic device, computer-readable storage medium, and computer program product, which can improve data security and fatigue detection accuracy. The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as various types of terminal devices or as a server.

[0047] See also Figure 1 , Figure 1 It is an architectural diagram of a driver fatigue detection system 100 provided in an embodiment of the present application. Vehicles 400-1 and 400-2 are connected to a server 200 via a network 300, wherein the network 300 may be a wide area network or a local area network, or a combination of the two.

[0048] In some embodiments, the driver fatigue detection method provided in the embodiments of the present application can be applied to vehicles. Taking vehicle 400-1 as an example, vehicle 400-1 receives the model parameters of the fatigue detection model sent by server 200 as the service end; obtains the driving data of vehicle 400-1 itself and the physiological data of the driver of vehicle 400-1; performs fatigue detection processing according to the driving data and the physiological data through the fatigue detection model to obtain the fatigue state of the driver of vehicle 400-1. In some embodiments, vehicle 400-1 can send the driving data and the physiological data to the corresponding client, and the client performs fatigue detection processing according to the driving data and the physiological data through the fatigue detection model to obtain the fatigue state of the driver of vehicle 400-1, and sends the fatigue state to vehicle 400-1, wherein one client can correspond to one or more vehicles.

[0049] It is worth noting that the above description about vehicle 400 - 1 is also applicable to vehicle 400 - 2.

[0050] In some embodiments, the driver's fatigue detection method provided in the embodiment of the present application can be applied to the server. For example, the server 200 as the server performs federated learning based on the fatigue data samples of multiple sample vehicles to obtain the model parameters of the fatigue detection model; and sends the model parameters of the fatigue detection model to the target vehicle (referring to vehicle 400-1 or vehicle 400-2), so that the target vehicle obtains the driving data of the target vehicle and the physiological data of the driver of the target vehicle, and performs fatigue detection processing according to the driving data and the physiological data through the fatigue detection model to obtain the fatigue state of the driver of the target vehicle. In some embodiments, the server 200 can also transmit data with the clients corresponding to the multiple sample vehicles respectively to realize federated learning, wherein the client corresponding to the sample vehicle stores the fatigue data samples of the local sample vehicle, and one client can have a corresponding relationship with one or more vehicles. In some embodiments, the server 200 can also send the model parameters of the fatigue detection model to the client corresponding to the target vehicle. After the target vehicle collects the driving data and the physiological data, the driving data and the physiological data are sent to the client, and the client performs fatigue detection processing according to the driving data and the physiological data through the fatigue detection model to obtain the fatigue state of the driver of the target vehicle, and sends the fatigue state to the target vehicle.

[0051] In some embodiments, a vehicle (such as vehicle 400-1 or vehicle 400-2) or a server (such as server 200) can implement the driver fatigue detection method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0052] In some embodiments, the server (such as server 200) can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The vehicle and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the embodiments of the present application.

[0053] The electronic device provided in the embodiment of the present application is a terminal device as an example. Figure 2A , Figure 2A The vehicle provided in the embodiment of the present application (with Figure 1 A schematic diagram of the structure of the vehicle 400-1 in FIG. 1 (here, a schematic diagram of the structure of the electronic equipment inside the vehicle such as the vehicle-mounted terminal) is shown. Figure 2A The vehicle 400-1 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the vehicle 400-1 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2A Various buses are labeled as bus system 440 .

[0054] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0055] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as microphones, touch screen displays, cameras, other input buttons and controls.

[0056] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0057] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0058] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0059] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0060] A network communication module 452, for reaching other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB);

[0061] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0062] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0063] In some embodiments, the driver fatigue detection device provided in the embodiments of the present application can be implemented in a software manner. Figure 2A The driver's fatigue detection device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: an acquisition module 4551 and a fatigue detection module 4552. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0064] See also Figure 2B , Figure 2B is a schematic diagram of the structure of the server 200 provided in an embodiment of the present application, Figure 2BThe server 200 shown includes: at least one processor 210, a memory 250 and at least one network interface 220. The various components in the server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2B Various buses are labeled as bus system 240 .

[0065] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0066] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0067] The memory 250 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0068] In some embodiments, memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0069] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0070] A network communication module 252, used to reach other computing devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0071] In some embodiments, the driver fatigue detection device provided in the embodiments of the present application can be implemented in a software manner. Figure 2B The driver's fatigue detection device 255 stored in the memory 250 is shown, which can be software in the form of a program and a plug-in, including the following software modules: a learning module 2551 and a sending module 2552. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0072] The driver fatigue detection method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the electronic device provided in the embodiment of the present application.

[0073] See also Figure 3A , Figure 3A is a flowchart of a driver fatigue detection method provided by an embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0074] In step 101, the server performs federated learning based on fatigue data samples of multiple sample vehicles to obtain model parameters of the fatigue detection model; wherein the local fatigue data samples include sample driving data, sample physiological data and sample fatigue status.

[0075] In traditional centralized machine learning, training data is usually stored in a centralized server. However, this easily leads to difficulties in collecting some sensitive data, as well as user privacy leakage caused by data leakage during transmission, that is, low data security. Therefore, in the embodiment of the present application, the fatigue detection model is trained by federated learning. In essence, a fatigue detection model is trained by multiple sample vehicles together, and the training process does not require the exchange of fatigue data samples of each sample vehicle. In this way, the privacy leakage risk caused by the centralized storage of data on the server can be avoided, and the decentralized data resources can be better utilized.

[0076] For example, the server performs federated learning based on fatigue data samples from multiple sample vehicles to obtain model parameters of the fatigue detection model. Local fatigue data samples include sample driving data, sample physiological data, and sample fatigue status. The sample driving data and sample physiological data are collected by the sample vehicle at a certain point in history or time period. The richness and comprehensiveness of features are improved through both the driver and the vehicle, thereby improving the effect of federated learning and the accuracy of subsequent fatigue detection. The sample fatigue status is labeled and identified as the correct fatigue status. Figure 3A The sample vehicles include Figure 1 The vehicle 400-1 and the vehicle 400-2 are shown, and the server is Figure 1The server 200 is shown by way of example.

[0077] In step 102, the server sends the model parameters of the fatigue detection model to the target vehicle.

[0078] Here, the target vehicle may be a sample vehicle or may not be a sample vehicle. Figure 3A In the example, the target vehicle is Figure 1 Vehicle 400 - 1 is shown as an example.

[0079] It is worth noting that the server can also send the model architecture of the fatigue detection model to the target vehicle so that the target vehicle uses the fatigue detection model based on the model architecture and model parameters. For example, in the TensorFlow machine learning framework, the TensorFlow model file can include a model architecture file (.pbtxt or .pb format) for storing the model architecture and a model parameter file (.ckpt or .saved_model format) for storing the model parameters.

[0080] In step 103 , the target vehicle acquires driving data of the target vehicle and physiological data of the driver of the target vehicle.

[0081] Here, the target vehicle obtains the driving data of the target vehicle itself and simultaneously obtains the physiological data of the driver of the target vehicle.

[0082] In some embodiments, step 103 may be performed periodically to implement periodic fatigue detection of the driver of the target vehicle.

[0083] In some embodiments, the driving data includes longitudinal speed, lateral speed, longitudinal acceleration, lateral acceleration, accelerator pedal opening and closing degree, brake pedal opening and closing degree, and steering wheel angle; the physiological data includes heart rate variability characteristics of the electrocardiogram signal.

[0084] Here, the physiological data includes the heart rate variability characteristics of the ECG signal. For example, the ECG signal of the driver operating the steering wheel can be collected through a steering wheel with an ECG sensor, and the approximate entropy of the RR interval of the ECG signal can be extracted as the heart rate variability feature. Among them, considering that the ECG signal is a non-stationary signal, the frequency and amplitude of the ECG signal will change with the passage of time, and the ECG signal is relatively weak, the signal amplitude is small, and it is easily affected by the noise in the environment. Therefore, the ECG signal can be pre-processed (for example, the ECG signal can be filtered through a filter), and then the approximate entropy of the RR interval of the pre-processed ECG signal can be extracted, so that the adverse effects of noise can be reduced. In addition, in the ECG signal, the ventricular depolarization waveform (or QRS complex) occupies the largest energy and has the most obvious waveform characteristics. By detecting the R wave in the QRS complex, various information such as the heart rate characteristics and conduction velocity of the ECG signal can be obtained. In addition, the peak of the R wave in the QRS complex is the largest and most obvious. Therefore, the RR interval of the ECG signal is extracted. On this basis, since the linear characteristics cannot reflect all the information contained in the RR interval, the approximate entropy algorithm is used to calculate the approximate entropy of the RR interval. In this way, the obtained approximate entropy can better reflect the mental and physiological state of the driver.

[0085] As for vehicles, sensors can be deployed on various vehicle components (such as steering wheels, accelerator pedals, etc.) to collect corresponding data. Since vehicle speed, vehicle acceleration, accelerator pedal, brake pedal and steering wheel angle can effectively reflect the driver's ability to control the vehicle, for example, compared with normal driving, the frequency of steering wheel correction and the stability of vehicle speed during fatigue will be significantly reduced. Therefore, in the embodiment of the present application, the longitudinal speed, lateral speed, longitudinal acceleration, lateral acceleration, accelerator pedal opening and closing, brake pedal opening and closing and steering wheel angle of the vehicle are collected to construct driving data.

[0086] Through the above method, the feature richness and comprehensiveness of the data can be improved, and different features can verify and complement each other, thereby improving the accuracy of fatigue detection.

[0087] In step 104, the target vehicle performs fatigue detection processing according to the driving data and physiological data through a fatigue detection model to obtain the fatigue state of the driver of the target vehicle.

[0088] Here, the target vehicle performs fatigue detection processing based on the driving data and physiological data through the fatigue detection model to obtain the fatigue status of the driver of the target vehicle. That is, the driving data and physiological data are used as the input of the fatigue detection model. After processing by the fatigue detection model, the output of the fatigue detection model is the fatigue status of the driver of the target vehicle.

[0089] Further applications can be made based on the obtained fatigue status. For example, the fatigue status can be displayed on the on-board display screen of the target vehicle; or an alarm (such as a voice alarm) can be issued when the fatigue status is a preset state (such as "fatigue") to remind the driver to slow down or take a rest, thereby reducing the occurrence of traffic accidents.

[0090] It is worth noting that the operations performed by the vehicle involved in the embodiments of the present application can also be performed by the client corresponding to the vehicle, wherein the client refers to the identity in federated learning (federated learning is implemented by the server and multiple clients), and the client can be implemented by a terminal device and / or a server. For example, the client corresponding to the sample vehicle can conduct federated learning with the server, and the client corresponding to the sample vehicle stores the fatigue data sample of the sample vehicle; the server can send the model parameters of the fatigue detection model to the client corresponding to the target vehicle, so that when the client obtains the driving data of the target vehicle and the physiological data of the driver of the target vehicle, the fatigue detection processing is performed according to the driving data and the physiological data through the fatigue detection model to obtain the fatigue state of the driver of the target vehicle. Among them, a client can have a corresponding relationship with one or more vehicles. For example, if a company manages multiple vehicles, a corresponding client can be uniformly configured for the multiple vehicles, and of course, a corresponding client can be configured for each vehicle separately.

[0091] like Figure 3A As shown, in an embodiment of the present application, the server performs federated learning based on fatigue data samples local to multiple sample vehicles to obtain model parameters of the fatigue detection model. In this way, based on the method of federated learning, it is ensured that the server can achieve learning without obtaining fatigue data samples local to the sample vehicles, thereby effectively improving data security. At the same time, in the process of federated learning and fatigue detection processing, both driving data and physiological data are utilized, which can improve the richness of feature information, thereby improving the learning effect of federated learning and the accuracy of fatigue detection.

[0092] In some embodiments, see Figure 3B , Figure 3B is a flow chart of a driver fatigue detection method provided by an embodiment of the present application. Figure 3B middle, Figure 3A The illustrated step 101 can be implemented through steps 201 to 204 , which will be described in conjunction with each step.

[0093] In step 201 , during the i-th iteration, the server sends the global model parameters to the sample vehicles.

[0094] In the embodiment of the present application, the process of federated learning includes multiple iterations. The following is an example of the i-th iteration process, where i is an integer greater than zero. In the i-th iteration process, the server sends the global model parameters of the fatigue detection model to each sample vehicle, such as Figure 3B As shown, the server sends the global model parameters of the fatigue detection model to vehicle 400-1 and vehicle 400-2. It is worth noting that the server can also send the model architecture of the fatigue detection model to each sample vehicle, so that each sample vehicle uses the fatigue detection model based on the model architecture and the global model parameters.

[0095] In step 202, during the i-th iteration, the sample vehicle trains the fatigue detection model according to the local fatigue data sample, and updates the global model parameters of the fatigue detection model to the local model parameters.

[0096] Here, the sample vehicle obtains fatigue data samples stored locally, and trains the fatigue detection model using global model parameters based on the local fatigue data samples, so as to update the global model parameters to local model parameters during the training process. In this way, the fatigue detection model is trained locally by the sample vehicle, ensuring that the fatigue data samples of the sample vehicle are not out of the sample vehicle, thereby improving security and privacy.

[0097] It is worth noting that in order to improve the training effect of the fatigue detection model, multiple fatigue data samples of the sample vehicle can be obtained, a local training data set for the fatigue detection model can be constructed based on the multiple local fatigue data samples, and the fatigue detection model can be trained based on the local training data set. Among them, the fatigue detection model can be trained based on all fatigue data samples in the local training data set, or some fatigue data samples can be selected (such as randomly selected) from the local training data set to train the fatigue detection model, thereby improving the randomness of the training data and the robustness of the model.

[0098] In some embodiments, the above-mentioned training of the fatigue detection model based on local fatigue data samples and updating the global model parameters of the fatigue detection model to local model parameters can be achieved in the following manner: fatigue detection processing is performed according to sample driving data and sample physiological data through the fatigue detection model to obtain the fatigue state to be compared; the error between the sample fatigue state and the fatigue state to be compared is determined; the global model parameters are updated according to the error to obtain the local model parameters.

[0099] Here is an example of training a fatigue detection model locally in a sample vehicle, that is, implementing training through back propagation.

[0100] For example, for a local fatigue data sample, a fatigue detection model that applies global model parameters is used to perform fatigue detection processing based on the sample driving data and sample physiological data in the fatigue data sample to obtain a fatigue state. In order to facilitate distinction, the fatigue state obtained here is named the fatigue state to be compared; the error between the sample fatigue state in the fatigue data sample and the fatigue state to be compared is determined; and backpropagation is performed in the fatigue detection model that applies global model parameters based on the error to update the global model parameters to obtain local model parameters.

[0101] The purpose of model training is to minimize the error through continuous iteration. The error can also be understood as a loss value. The embodiment of the present application does not limit the method of calculating the loss value. For example, it can be calculated by a cross entropy loss function, an L1 norm loss function, or a mean square error loss function. It is worth noting that the fatigue detection model in the embodiment of the present application can be a classification model. In this case, the fatigue state is a discrete value, such as "awake" and "fatigue". The classification loss function can be used to calculate the loss value; the fatigue detection model can also be a regression model. In this case, the fatigue state is a continuous value, such as a value in the numerical interval [1, 10]. The regression loss function can be used to calculate the loss value.

[0102] Through the above method, effective training of the fatigue detection model using global model parameters can be achieved locally in the sample vehicle.

[0103] In step 203, during the i-th iteration, the sample vehicle sends the local model parameters to the server.

[0104] After the sample vehicle completes the training, it sends the local model parameters to the server. Figure 3B As shown, vehicle 400-1 sends local model parameters to the server, and vehicle 400-2 sends local model parameters to the server. Considering that the local fatigue data samples of vehicle 400-1 are usually different from the local fatigue data samples of vehicle 400-2, the local model parameters of vehicle 400-1 and the local model parameters of vehicle 400-2 are usually different. Of course, the possibility of being the same cannot be ruled out.

[0105] In step 204, during the i-th iteration, the server performs a fusion process on the local model parameters respectively sent by multiple sample vehicles to obtain the global model parameters for the next iteration.

[0106] Here, the server performs a fusion process on the local model parameters of all received sample vehicles to obtain global model parameters for the next iteration, i.e., the i+1th iteration. The embodiment of the present application does not limit the method of fusion processing, for example, it can be direct averaging or weighted averaging. For example, the server can calculate the average value of the local model parameters of vehicle 400-1 and the local model parameters of vehicle 400-2 to obtain the global model parameters for the next iteration; it can also weight the local model parameters of vehicle 400-1 according to the weight of vehicle 400-1, and weight the local model parameters of vehicle 400-2 according to the weight of vehicle 400-2, and then calculate the average value of the weighted local model parameters of vehicle 400-1 and the weighted local model parameters of vehicle 400-2 to obtain the global model parameters for the next iteration.

[0107] In this way, the global model parameters are updated at the global level where the server is located, and the server will not obtain the fatigue data samples of the sample vehicle locally.

[0108] It is worth noting that when the calculation method is weighted average, the weight of each sample vehicle can be set according to the actual application scenario. For example, if the local fatigue data sample of a sample vehicle is more representative, the weight of the sample vehicle can be set larger than the weights of other sample vehicles, thereby increasing the proportion of the local model parameters of the sample vehicle in the fusion processing process.

[0109] It is worth noting that a termination condition can be set for the iterative process of federated learning. When the termination condition is met, the iteration is terminated. The termination condition may be reaching a preset number of iterations or the error is less than a certain threshold, etc., and there is no limitation on this.

[0110] exist Figure 3B middle, Figure 3A The illustrated step 102 may be updated to step 205. In step 205, the server uses the global model parameters obtained after multiple iterations as the model parameters of the fatigue detection model for fatigue detection processing, and sends them to the target vehicle.

[0111] Here, when the termination condition is met, the server uses the global model parameters obtained after multiple iterations as the model parameters of the fatigue detection model used by the target vehicle for fatigue detection processing, and sends them to the target vehicle.

[0112] like Figure 3BAs shown, in the embodiment of the present application, the interaction between the sample vehicle and the server only involves the model parameters, and does not involve the local fatigue data samples of each sample vehicle, which can ensure that the local fatigue data samples of each sample vehicle do not leave the sample vehicle, thereby improving the data security of the training data; at the same time, the global model parameters need to be obtained by referring to the local model parameters of multiple sample vehicles, and the global model parameters are updated through multiple iterations, which can ensure the training effect and fatigue detection accuracy.

[0113] In some embodiments, see Figure 3C , Figure 3C is a flowchart of a driver fatigue detection method provided by an embodiment of the present application, based on Figure 3B Before step 201, in step 301, the server may construct a fatigue detection model according to the initial model parameters.

[0114] Here, the server may initialize the fatigue detection model. For easy distinction, the model parameters in the initialized fatigue detection model are named as initial model parameters.

[0115] In step 302, the server trains the fatigue detection model according to the preset fatigue data samples, and updates the initial model parameters of the fatigue detection model to the global model parameters.

[0116] Here, the server obtains preset fatigue data samples, which are similar to the local fatigue data samples of the sample vehicle. The preset fatigue data samples include sample driving data, sample physiological data and sample fatigue status. The difference between the two is that the local fatigue data samples of the sample vehicle are sensitive data and cannot be known to the server; while the preset fatigue data samples can be non-sensitive data obtained by the server from other channels, such as public data.

[0117] The server trains the fatigue detection model using the initial model parameters according to the preset fatigue data samples, so as to update the initial model parameters to the global model parameters during the training process. The server trains the fatigue detection model in a similar way to the above-mentioned sample vehicle training fatigue detection model. It is worth noting that the global model parameters obtained here are the global model parameters used for the first iteration in federated learning.

[0118] In some embodiments, in order to improve the training effect of the fatigue detection model on the server, a preset plurality of fatigue data samples may be obtained to construct a preset training data set for the fatigue detection model, and the fatigue detection model may be trained according to the preset training data set.

[0119] like Figure 3CAs shown, the embodiment of the present application can pre-train the fatigue detection model on the server side, so that the fatigue detection model can learn the general data rules and ensure the accuracy of the global model parameters to a certain extent. In this way, each sample vehicle only needs to fine-tune the global model parameters, which can improve the model training effect as a whole. At the same time, it can reduce the training cost for the sample vehicles, that is, only a small amount of training is required based on the global model parameters, which greatly reduces the consumption of computing resources and time.

[0120] In some embodiments, see Figure 3D , Figure 3D is a flowchart of a driver fatigue detection method provided by an embodiment of the present application, based on Figure 3A After step 103, in step 401, the target vehicle may perform standardization processing on the test data corresponding to the multiple time periods to obtain standardized test data corresponding to the multiple time periods; wherein the test data includes driving data and physiological data.

[0121] Here, the data to be measured include driving data and physiological data. Since the data to be measured include actual values ​​collected, these values ​​are affected by dimensions and are difficult to compare with each other. If they are directly input into the fatigue detection model for fatigue detection processing, it is difficult to ensure accuracy. Therefore, in the embodiment of the present application, the influence of dimensions can be removed in a standardized manner.

[0122] First, the target vehicle obtains the test data corresponding to multiple time periods. For example, the test data for each second in the latest 30 seconds may be obtained.

[0123] Then, the target vehicle performs standardization processing on the test data corresponding to the multiple moments to obtain the standardized test data corresponding to the multiple moments. The embodiment of the present application does not limit the method of standardization processing. Taking the characteristics of any dimension (named as the target dimension) in the test data as an example, the following function can be used:

[0124]

[0125] Among them, x represents the characteristics of the target dimension in the data to be tested at any moment, such as driving data or physiological data; μ represents the average value of the characteristics of the target dimension in the data to be tested corresponding to multiple moments, σ ​​represents the standard deviation of the characteristics of the target dimension in the data to be tested corresponding to multiple moments, and x' represents the standardized characteristics of the target dimension at any moment (the same moment as x). In this way, the standardized characteristics corresponding to multiple moments obey the data distribution with an average value of 0 and a standard deviation of 1, and the standardized characteristics of all dimensions constitute the standardized data to be tested.

[0126] Alternatively, the following function can be used:

[0127]

[0128] Among them, x represents the characteristics of the target dimension in the test data at any time, x' represents the standardized characteristics of the target dimension at any time (corresponding to the same time as x), min represents the minimum value of the characteristics of the target dimension in the test data corresponding to multiple times, and max represents the maximum value of the characteristics of the target dimension in the test data corresponding to multiple times.

[0129] It is worth noting that the dimension is used to represent the type of features in the data to be tested. For example, in the data to be tested, driving data represents one dimension and physiological data represents another dimension. Of course, driving data or physiological data can also be subdivided to obtain more subdivided dimensions. The above method performs standardization processing on each dimension separately, which can improve the effectiveness and accuracy of the standardization processing.

[0130] In some embodiments, the standardized processing of the data to be measured may refer to the standardized processing of the features of some dimensions in the data to be measured. For the data to be measured, it may be that only the features of some dimensions are greatly affected by the dimension. Therefore, only the features of this part of the dimensions can be standardized, thereby improving the efficiency of the standardized processing and avoiding waste of resources. For example, the data to be measured includes driving data and physiological data. The physiological data is less or not affected by the dimension, and the driving data is greatly affected by the dimension. In this case, only the driving data can be standardized, and the physiological data can be kept as it is (that is, the physiological data in the data to be measured is the same as the physiological data in the standardized data to be measured). The dimensions that need to be standardized in the data to be measured can be pre-set. Of course, the features of each dimension in the data to be measured can also be standardized.

[0131] In some embodiments, driving data or physiological data can be further subdivided into lower-level dimensions. For example, driving data includes features of seven dimensions: longitudinal speed, lateral speed, longitudinal acceleration, lateral acceleration, accelerator pedal opening, brake pedal opening, and steering wheel angle; physiological data includes features of one dimension: heart rate variability characteristics of electrocardiogram signals. In summary, standardization of the data to be measured may refer to standardization of some or all of the features of the eight dimensions: longitudinal speed, lateral speed, longitudinal acceleration, lateral acceleration, accelerator pedal opening, brake pedal opening, steering wheel angle, and heart rate variability characteristics.

[0132] exist Figure 3D middle, Figure 3AStep 104 shown can be updated to step 402. In step 402, the target vehicle performs fatigue detection processing according to the standardized test data corresponding to the target time through the fatigue detection model to obtain the fatigue state of the driver of the target vehicle at the target time; wherein the target time represents any one of multiple times.

[0133] After completing the standardization processing, for any one of the multiple moments (named as the target moment), the target vehicle performs fatigue detection processing on the standardized test data corresponding to the target moment through the fatigue detection model to obtain the fatigue state of the driver of the target vehicle at the target moment.

[0134] It is worth noting that in addition to standardizing the data to be tested, the sample data to be tested is also standardized during the federated learning process. The standardized sample data to be tested is used to train the fatigue detection model. This can effectively speed up the model fitting process, that is, improve the training efficiency. The sample data to be tested includes sample driving data and sample physiological data.

[0135] like Figure 3D As shown, the embodiment of the present application removes the dimensionality effect through standardization processing, so that features of different dimensions can be compared with each other, thereby improving the accuracy of fatigue detection; the standardization processing is also applicable to sample driving data, which helps to improve the efficiency of training fatigue detection models locally in sample vehicles.

[0136] The following will describe an exemplary application of the present application embodiment in a practical application scenario. Figure 4 A flowchart of a driver fatigue detection method is shown in FIG. Figure 4 , which is explained in the following three steps.

[0137] 1) Data collection and processing.

[0138] ①Physiological data collection.

[0139] The human electrocardiogram (ECG) contains a lot of information related to the physiological and mental state of the human body. The ECG reflects the electrical excitation process of the heart that triggers the heart beat, and records the curve of the change of the electrical activity signal sequence over time during the electrical excitation process. In the embodiment of the present application, the heart rate variability (HRV) feature in the ECG signal is extracted for analysis. The steps are as follows:

[0140] a) The driver's ECG signal is collected through a steering wheel with an ECG sensor. For example, the ECG sensor uses a conductive leather sensor to collect ECG signals in a contact detection manner, and the collected ECG signals are transmitted to a central control unit (CCU) through a local interconnect network (LIN) of the vehicle.

[0141] b) Preprocess the collected ECG signals. Since ECG signals are non-stationary signals, their frequency and amplitude will change over time. ECG signals are relatively weak and have a small amplitude, which is easily affected by noise in the environment. Therefore, the collected ECG signals are preprocessed accordingly, such as filtering, to reduce the impact of noise on feature extraction.

[0142] c) Extraction of heart rate variability feature p. The QRS complex in the ECG signal has the largest energy and the most obvious waveform characteristics, belonging to the medium and high frequency. In addition, the peak of the R wave in the QRS complex is the largest and most obvious. Therefore, the RR interval in the ECG signal is extracted. On this basis, since the linear feature cannot reflect all the information contained in the RR interval, after extracting the RR interval of the ECG signal, the approximate entropy of the RR interval is calculated using the approximate entropy algorithm as the extracted heart rate variability feature, which can effectively reflect the physiological and mental state of the driver.

[0143] ② Driving data collection.

[0144] In the embodiment of the present application, the driving data includes the longitudinal speed v and the lateral speed v of the vehicle. h , longitudinal acceleration a, lateral acceleration a h , accelerator pedal opening and closing degree y, brake pedal opening and closing degree s, steering wheel angle f and other vehicle characteristics. These vehicle characteristics can be collected by deploying sensors on corresponding vehicle components (such as steering wheel, accelerator pedal, etc.).

[0145] The above vehicle characteristics can reflect the driver's ability to control the vehicle. For example, the frequency of steering wheel corrections and the smoothness of vehicle speed will be significantly reduced during fatigue compared to normal driving.

[0146] Considering that driving data includes vehicle characteristics in multiple dimensions, driving data is standardized so that vehicle characteristics in different dimensions can be compared with each other. For example, driving data for every second within 30 seconds can be obtained and standardized according to the following formula:

[0147]

[0148] Among them, x represents the vehicle characteristics in the target dimension (target dimension refers to any dimension) at any time, μ represents the average value of the vehicle characteristics in the target dimension corresponding to multiple time periods, σ represents the standard deviation of the vehicle characteristics in the target dimension corresponding to multiple time periods, and x' represents the standardized vehicle characteristics in the target dimension at any time (the same time corresponding to x).

[0149] It is worth mentioning that physiological data can also be normalized.

[0150] 2) Integrate physiological data and driving data.

[0151] ①Set the input and output of the fatigue detection model.

[0152] Since a single feature can extract insufficient information, the accuracy of fatigue detection based on a single feature is relatively low. In the embodiment of the present application, different features can verify and complement each other, and by integrating a variety of different features, more comprehensive and accurate information can be provided for fatigue detection, thereby ensuring the accuracy of detection. Therefore, in the embodiment of the present application, the driver's fatigue state is comprehensively judged based on physiological data and driving data.

[0153] For example, the fatigue detection model can be a binary classification model, with the input being the standardized heart rate variability feature p, longitudinal velocity v, and lateral velocity v h , longitudinal acceleration a, lateral acceleration a h , accelerator pedal opening and closing degree y, brake pedal opening and closing degree s and steering wheel angle f, a total of 8 dimensions of features; the output is the secondary classification of fatigue state, that is, awake (suitable for driving) or tired (unsuitable for driving).

[0154] ② Collection of the initial training data set (corresponding to the preset training data set mentioned above).

[0155] For example, drivers of different ages, genders, and driving experience can be selected to drive at different times to obtain the drivers' physiological data, driving data, and labeled fatigue status (corresponding to the sample fatigue status above), and further generate the corresponding initial training data set.

[0156] ③Pre-training of fatigue detection model.

[0157] Pre-training refers to training using a centralized training method, the steps are as follows:

[0158] a) Construct a fatigue detection model and initialize model parameters. The number of input layers is 8 (corresponding to 8 dimensions of features), the number of output layers is 2 (corresponding to 2 categories of fatigue status), and the number of intermediate layers is set to 10 (can be optimized and adjusted, not limited to 10);

[0159] b) selecting training data from the initial training data set constructed above;

[0160] c) For each piece of training data, calculate the output of the fatigue detection model;

[0161] d) Calculate the error and update the model parameters of the fatigue detection model.

[0162] e) Repeat steps b) to d) until the training termination condition is reached (such as reaching a preset number of iterations, or the error is less than a certain threshold, etc.), and use the model parameters applied by the fatigue detection model at this time as global model parameters.

[0163] 3) Decentralized training.

[0164] In the embodiment of the present application, decentralized training can be implemented through K (K is an integer greater than 1) clients and a server, wherein each client has a local training data set, the fatigue detection model of the client and the fatigue detection model of the server have the same model architecture, and the fatigue detection model is trained locally on the client based on the local training data set, and the client does not pass the local training data set to the server. Each client has a corresponding relationship with one or more sample vehicles, that is, each client stores fatigue data samples of one or more sample vehicles locally.

[0165] The training process of decentralized training involves multiple cycles (iterations). In each cycle, the server sends the global model parameters of this round to each client. After receiving the global model parameters from the server, the client trains the fatigue detection model that applies the global model parameters according to the local training data set to update the global model parameters to the local model parameters. After the client training is completed, the local model parameters are uploaded to the server. After receiving the local model parameters from each client, the server performs fusion processing to obtain the new global model parameters for the next cycle. This cycle continues until the set number of training times is reached or the model converges. The operations performed by the server and the client are as follows.

[0166] ①Server side.

[0167] The server performs the following operations in each loop:

[0168] a) Set the global model parameter θ of this round t Delivered to each client;

[0169] b) Receive local model parameters θ uploaded by each client k ,θ k represents the local model parameters of the kth client;

[0170] c) Based on the local model parameters θ uploaded by each client k , for the global model parameter θ t Update, the calculation formula is as follows:

[0171]

[0172] Where n represents the number of clients, n k Represents the weight of the k-th client.

[0173] ②Client.

[0174] The kth client performs the following operations in each loop:

[0175] a) Receive the global model parameters θ sent by the server t , and as the local model parameter θ k ;

[0176] b) Apply local model parameters θ based on the local training dataset k The fatigue detection model is trained, that is, the local model parameters θ k Make updates;

[0177] c) Update the local model parameters θ k Send to the server.

[0178] The embodiments of the present application can at least achieve the following technical effects: 1) using physiological data and driving data to comprehensively judge the driver's fatigue state can improve the accuracy of fatigue detection; 2) completing model training while ensuring that the data does not leave the local client, which can protect the data privacy of the driver and the vehicle.

[0179] The following is a description of an exemplary structure of a driver fatigue detection device 455 provided in an embodiment of the present application implemented as a software module. In some embodiments, for example, Figure 2A As shown, the software modules stored in the driver's fatigue detection device 455 of the memory 450 may include: an acquisition module 4551, used to acquire the vehicle's driving data and the physiological data of the vehicle's driver; a fatigue detection module 4552, used to perform fatigue detection processing based on the driving data and the physiological data through a fatigue detection model to obtain the driver's fatigue status; wherein, the model parameters of the fatigue detection model are obtained by federated learning based on fatigue data samples of multiple sample vehicles, and the local fatigue data samples include sample driving data, sample physiological data and sample fatigue status.

[0180] In some embodiments, the driver's fatigue detection device 455 also includes a parameter determination module, which is used to: perform the following processing in each iteration of federated learning: obtain the global model parameters of the fatigue detection model; train the fatigue detection model according to the local fatigue data sample, and update the global model parameters of the fatigue detection model to the local model parameters; send the local model parameters to the server, so that the server can fuse the local model parameters sent by multiple sample vehicles respectively, and obtain the global model parameters for the next iteration; use the global model parameters obtained after multiple iterations as the model parameters of the fatigue detection model used for fatigue detection processing.

[0181] In some embodiments, the parameter determination module is also used to: perform fatigue detection processing based on sample driving data and sample physiological data through a fatigue detection model to obtain a fatigue state to be compared; determine the error between the sample fatigue state and the fatigue state to be compared; and update the global model parameters based on the error to obtain local model parameters.

[0182] In some embodiments, the driver's fatigue detection device 455 also includes a standardization module, which is used to standardize the test data corresponding to multiple moments, and obtain standardized test data corresponding to multiple moments; wherein the test data includes driving data and physiological data; the fatigue detection module 4552 is also used to perform fatigue detection processing according to the standardized test data corresponding to the target moment through a fatigue detection model, and obtain the driver's fatigue state at the target moment; wherein the target moment represents any one of multiple moments.

[0183] In some embodiments, the driving data includes longitudinal speed, lateral speed, longitudinal acceleration, lateral acceleration, accelerator pedal opening and closing degree, brake pedal opening and closing degree, and steering wheel angle; the physiological data includes heart rate variability characteristics of the electrocardiogram signal.

[0184] The following is an exemplary structure of the driver fatigue detection device 255 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2B As shown, the software modules stored in the driver fatigue detection device 255 of the memory 250 may include: a learning module 2551, used to perform federated learning based on fatigue data samples local to multiple sample vehicles to obtain model parameters of the fatigue detection model; wherein the local fatigue data samples include sample driving data, sample physiological data and sample fatigue status; a sending module 2552, used to send the model parameters of the fatigue detection model to the target vehicle, so that the target vehicle obtains the driving data of the target vehicle and the physiological data of the driver of the target vehicle, and performs fatigue detection processing based on the driving data and the physiological data through the fatigue detection model to obtain the fatigue status of the driver.

[0185] The embodiment of the present application provides a computer program product or a computer program, which includes executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the executable instructions from the computer-readable storage medium, and the processor executes the executable instructions, so that the electronic device executes the driver fatigue detection method described in the embodiment of the present application.

[0186] An embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the driver fatigue detection method provided in the embodiment of the present application.

[0187] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0188] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0189] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0190] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0191] The above are only embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for detecting driver fatigue, characterized in that: Applied to a vehicle, the method comprises: Acquiring driving data of the vehicle and physiological data of the driver of the vehicle; Performing fatigue detection processing according to the driving data and the physiological data through a fatigue detection model to obtain the fatigue state of the driver; The model parameters of the fatigue detection model are obtained by federated learning based on fatigue data samples local to a plurality of sample vehicles, wherein the local fatigue data samples include sample driving data, sample physiological data and sample fatigue status.

2. The method according to claim 1, characterized in that The vehicle is a sample vehicle; the process of federated learning includes multiple iterations; and the method further includes: The following processing is performed in each iteration of the federated learning: Obtaining global model parameters of the fatigue detection model; Training the fatigue detection model according to local fatigue data samples, and updating the global model parameters of the fatigue detection model to local model parameters; Sending the local model parameters to a server, so that the server performs fusion processing on the local model parameters respectively sent by multiple sample vehicles to obtain global model parameters for the next iteration; Before performing fatigue detection processing according to the driving data and the physiological data by using the fatigue detection model, the method further includes: The global model parameters obtained through the multiple iterations are used as model parameters of the fatigue detection model used for fatigue detection processing.

3. The method according to claim 2, characterized in that The step of training the fatigue detection model according to the local fatigue data sample and updating the global model parameters of the fatigue detection model to the local model parameters includes: Performing fatigue detection processing according to the sample driving data and the sample physiological data through the fatigue detection model to obtain a fatigue state to be compared; determining an error between the sample fatigue state and the fatigue state to be compared; The global model parameters are updated according to the error to obtain local model parameters.

4. The method according to claim 1, characterized in that After acquiring the driving data of the vehicle and the physiological data of the driver of the vehicle, the method further includes: Performing standardization processing on the data to be tested corresponding to the multiple moments respectively, to obtain the standardized data to be tested corresponding to the multiple moments respectively; wherein the data to be tested includes driving data and physiological data; The step of performing fatigue detection processing according to the driving data and the physiological data by using a fatigue detection model to obtain the fatigue state of the driver includes: Performing fatigue detection processing according to the standardized test data corresponding to the target time by the fatigue detection model to obtain the fatigue state of the driver at the target time; The target time represents any one of the multiple time periods.

5. The method according to any one of claims 1 to 4, characterized in that: The driving data includes longitudinal speed, lateral speed, longitudinal acceleration, lateral acceleration, accelerator pedal opening and closing degree, brake pedal opening and closing degree and steering wheel angle; the physiological data includes heart rate variability characteristics of electrocardiogram signals.

6. A method for detecting driver fatigue, characterized in that: Applied to the server, the method includes: Performing federated learning based on fatigue data samples of multiple sample vehicles to obtain model parameters of the fatigue detection model; wherein the local fatigue data samples include sample driving data, sample physiological data, and sample fatigue status; The model parameters of the fatigue detection model are sent to the target vehicle so that the target vehicle obtains the driving data of the target vehicle and the physiological data of the driver of the target vehicle, and the fatigue detection model performs fatigue detection processing according to the driving data and the physiological data to obtain the fatigue state of the driver.

7. A driver fatigue detection device, characterized in that: Applied to a vehicle, the device comprises: An acquisition module, used to acquire the driving data of the vehicle and the physiological data of the driver of the vehicle; A fatigue detection module, used to perform fatigue detection processing according to the driving data and the physiological data through a fatigue detection model to obtain the fatigue state of the driver; The model parameters of the fatigue detection model are obtained by federated learning based on fatigue data samples local to a plurality of sample vehicles, wherein the local fatigue data samples include sample driving data, sample physiological data and sample fatigue status.

8. A driver fatigue detection device, characterized in that: Applied to the server, the device comprises: A learning module, used for performing federated learning based on fatigue data samples of multiple sample vehicles to obtain model parameters of a fatigue detection model; wherein the local fatigue data samples include sample driving data, sample physiological data and sample fatigue status; A sending module is used to send the model parameters of the fatigue detection model to a target vehicle so that the target vehicle obtains the driving data of the target vehicle and the physiological data of the driver of the target vehicle, and performs fatigue detection processing according to the driving data and the physiological data through the fatigue detection model to obtain the fatigue status of the driver.

9. An electronic device, characterized in that: include: A memory for storing executable instructions; The processor is used to implement the driver fatigue detection method described in any one of claims 1 to 5, or the driver fatigue detection method described in claim 6 when executing the executable instructions stored in the memory.

10. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to implement the driver fatigue detection method described in any one of claims 1 to 5, or the driver fatigue detection method described in claim 6 when executed by a processor.

Images