Multi-modal early warning intervention method and system for fatigue driving of passenger and freight vehicles

Abstract

The present application relates to the technical field of fatigue driving early warning, in particular to a passenger and freight vehicle fatigue driving multi-modal early warning intervention method and system, which obtains clustered driver image data by clustering processing driver image data, narrows the matching range, and further optimizes the matching process and reduces the calculation amount; combines driver fatigue image feature data to accurately analyze the real-time driving state of the driver, if the analysis result is fatigue, obtains driver physiological feature data and vehicle state feature data, combines a fatigue degree mapping model to scientifically analyze the fatigue degree of the driver, and simultaneously selects corresponding instructions according to the fatigue degree of the driver to perform fatigue driving early warning operation, thereby maximally protecting the safety of the driver and realizing passenger and freight vehicle fatigue driving multi-modal timely early warning and accurate intervention.

Description

Technical Field

[0001] This invention relates to the field of fatigue driving early warning technology, specifically to a multimodal early warning and intervention method and system for fatigue driving in passenger and freight vehicles. Background Technology

[0002] Fatigue driving is one of the major causes of passenger and freight vehicle accidents. Traditional fatigue driving detection methods are often based on single-modal data, resulting in poor accuracy of detection results. At the same time, the computational load when analyzing driver status is large, resulting in a long calculation process and errors in the calculation results, which in turn leads to poor real-time response and poor accuracy of analysis results. Summary of the Invention

[0003] To address the problems in related technologies, this invention provides a multimodal early warning and intervention method and system for fatigue driving in passenger and freight vehicles, thereby overcoming the aforementioned technical problems existing in the prior art.

[0004] To address the aforementioned technical problems, this invention provides the following technical solution: a multimodal early warning and intervention method for fatigue driving in passenger and freight vehicles, comprising the following steps:

[0005] S1. Acquire driver image data during real-time driving of passenger and freight vehicles, and perform clustering processing on the driver image data to obtain clustered driver image data.

[0006] S2. Analyze the driver's real-time driving status based on the clustered driver image data and driver fatigue image feature data to generate real-time driver driving status data.

[0007] If it is for safety reasons, then end this fatigue driving warning operation;

[0008] If fatigue occurs, proceed to S3;

[0009] S3. Collect driver physiological characteristic data and vehicle status characteristic data during real-time driving of passenger and freight vehicles;

[0010] S4. Obtain historical data to construct a fatigue level mapping model, and analyze the driver's fatigue level based on the driver image data, the driver's physiological characteristic data, the vehicle state characteristic data, and the fatigue level mapping model to obtain driver fatigue level data.

[0011] S5. Select the appropriate instruction based on the driver's fatigue level data to execute the fatigue driving warning operation.

[0012] Preferably, the specific steps for acquiring driver image data during real-time driving of passenger and freight vehicles and performing clustering processing on the driver image data to obtain driver image clustering data are as follows:

[0013] S11. By using DMS cameras installed inside passenger and freight vehicles to collect real-time image data of the driver during the driving process, a driver image dataset is obtained. ,in, This indicates the first data point collected during the driver's real-time driving process. Image data, This indicates the total number of driver image data;

[0014] The image data includes, but is not limited to, driver's facial information, driver's eye information, and driver's head information;

[0015] S12. Perform clustering processing on each driver image data in the driver image dataset to obtain driver image clustering data;

[0016] S121. Treat each driver image data in the driver image dataset as a cluster, calculate the Euclidean distance between clusters, and obtain a distance matrix.

[0017] S122. Based on the distance matrix, find the two closest clusters and perform clustering to obtain a new cluster. Recalculate the Euclidean distance between the new cluster and other clusters, and update the distance matrix accordingly. The Euclidean distance calculation formula is as follows:

[0018]

[0019] in, express and The Euclidean distance between them and Represents any two cluster centers 3D feature vectors For vectors In the Values ​​in dimensions For vectors In the Values ​​in a dimension;

[0020] S123. Set the Euclidean distance threshold, and repeat the steps in S121 to S122 until the Euclidean distance between any two clusters is greater than the Euclidean distance threshold, and obtain the clustered driver image feature data.

[0021] By clustering the acquired driver image data, the matching range is narrowed, thereby optimizing the matching process, reducing the amount of computation, ensuring the accuracy of the matching results, and improving the efficiency of the image matching process.

[0022] Preferably, the driver's real-time driving status is analyzed based on the driver image clustering data and driver fatigue image feature data to generate real-time driver driving status data. If the driving is safe, the current fatigue driving warning operation ends; if the driver is fatigued, the specific steps of S3 are as follows:

[0023] S21. Establish a dataset of driver fatigue image features. ,in, Indicates the first Driver fatigue image feature data, This represents the total number of image feature data points related to driver fatigue.

[0024] S22, Set the first cost value to And the second cost value ;

[0025] If the clustered driver image data belongs to one class, then any clustered driver image data is selected and matched one-to-one with the driver fatigue image feature data in the driver fatigue image feature dataset;

[0026] If a driver fatigue image feature data is successfully matched with any driver fatigue image feature data, it means that all driver image data in the driver image dataset are successfully matched with the driver fatigue image feature data, and the real-time driving status data of the driver is output as fatigue.

[0027] Otherwise, it indicates that all driver image data in the driver image dataset fails to match the driver fatigue image feature data, and the driver's real-time driving status data is output as safe.

[0028] If the clustered driver image data is of multiple classes, then any one of the clustered driver image data in each class is selected and matched one by one with the driver fatigue image feature data in the driver fatigue image feature dataset.

[0029] When the selected clustered driver image data successfully matches the driver fatigue image feature data in the driver fatigue image feature dataset, it means that all the clustered driver image data in the corresponding class of the selected clustered driver image data successfully match the driver fatigue image feature data in the driver fatigue image feature dataset.

[0030] If the selected clustered driver image data fails to match the driver fatigue image feature data in the driver fatigue image feature dataset, it means that all clustered driver image data in the corresponding class of the selected clustered driver image data fails to match the driver fatigue image feature data in the driver fatigue image feature dataset.

[0031] If all the selected clustered driver image data fail to match the driver fatigue image feature data in the driver fatigue image feature dataset, it means that all driver image data in the driver image dataset fails to match the driver fatigue image feature data in the driver fatigue image feature dataset, and the real-time driving status data of the driver is output as safe.

[0032] Otherwise, count the total number of clustered driver image data in all successfully matched classes, and analyze the sequence number corresponding to all clustered driver image data in all successfully matched classes;

[0033] When the serial numbers are consecutive, the total quantity is compared with the value of the first generation. Perform numerical comparisons;

[0034] If the total number is greater than or equal to the value of the first generation, then the driver's real-time driving status data is output as fatigue.

[0035] Otherwise, output the driver's real-time driving status data as safe;

[0036] When the serial numbers are not consecutive, the total quantity is compared with the second generation value. Perform numerical comparisons;

[0037] If the total number is greater than or equal to the value of the second generation, then the driver's real-time driving status data is output as fatigue.

[0038] Otherwise, output the driver's real-time driving status data as safe;

[0039] The step of selecting any clustered driver image data and matching it one-to-one with the driver fatigue image feature data in the driver fatigue image feature dataset includes the following steps:

[0040] S221. Construct an image feature search set, and set the current iteration number to . The maximum number of iterations is And the search space dimension of driver fatigue image feature data is ;

[0041] The driver fatigue image feature dataset is used as the driver fatigue image feature data search space, and random generation is performed within the driver fatigue image feature data search space. Each driver fatigue image feature data point corresponds to the initial position of an image feature search data point in the image feature search set, thus obtaining the initial position set of the image feature search set. ,in, Indicates the first element in the image feature search set. The initial position of the image feature search data;

[0042] S222. Calculate the fitness value between each image feature search data in the image feature search set and the selected clustered driver image data according to the fitness function formula. Arrange each image feature search data in the image feature search set in descending order of fitness value, and select the image feature search data with the highest fitness value as the current optimal solution. The fitness function formula is as follows:

[0043] ,

[0044] in, Indicates the first element in the image feature search set. Fitness values ​​for each image feature search data Indicates the first element in the image feature search set. The Euclidean distance between the driver fatigue image feature data corresponding to each image feature search data and the selected clustered driver image data. Indicates the correction value;

[0045] S223. Calculate the spatial distance between each image feature search data in the image feature search set; the spatial distance calculation formula is as follows:

[0046] ,

[0047] in, Indicates the first element in the image feature search set. Image feature search data and the first image feature search data and the first Spatial distance between image feature search data and These respectively represent the first in the image feature search set. Image feature search data and the first image feature search data Image feature search data in the th ... Position in the 3D search space;

[0048] S224. Calculate the attraction between each image feature search data in the image feature search set based on the spatial distance between each image feature search data in the image feature search set;

[0049] ,

[0050] in, Indicates the first element in the image feature search set. Image feature search data and the first image feature search data and the first The relative attractiveness between image feature search data, Indicates initial attraction level. Indicates the attraction coefficient;

[0051] S225. The position of each image feature search data in the image feature search set within the driver fatigue image feature data search space is updated based on its attractiveness to other image feature search data; the position update formula is as follows:

[0052] ,

[0053] in, Indicates the first element in the image feature search set. The position after updating the location using image feature search data. and These respectively represent the first in the image feature search set. Image feature search data and the first image feature search data and the first The current position of the image feature search data. This represents a random number that follows a uniform distribution between (-0.5, 0.5). Indicates the first element in the image feature search set. The adaptive step size factor corresponding to each image feature search data, and ,in, Represents the initial step size factor. and These represent the maximum and minimum fitness values, respectively.

[0054] S226. Calculate the fitness value of each image feature search data in the image feature search set after position update. If the fitness value of the image feature search data after position update is greater than the original fitness value, replace the original position with the new position of the image feature search data; otherwise, retain the original position.

[0055] S227. Determine the current iteration number. Is it less than the maximum number of iterations? If the current iteration number Less than the maximum number of iterations Then the current iteration number Increment by 1 and return to S3; otherwise, use the image feature search data with the highest fitness value as the global optimal solution.

[0056] S228. Set a fitness threshold. If the fitness value of the global optimal solution is greater than or equal to the fitness threshold, it means that the selected clustered driver image data is successfully matched with the driver fatigue image feature data in the driver fatigue image feature dataset; otherwise, it means that the selected clustered driver image data is not successfully matched with the driver fatigue image feature data in the driver fatigue image feature dataset.

[0057] S23. When the driver's real-time driving status data is safe, the driver's real-time driving status data is pushed to the fatigue driving warning platform through the wireless communication network, and the current fatigue driving warning operation ends; when the driver's real-time driving status data is fatigued, proceed to S3.

[0058] When analyzing the matching results, the influence of the continuity of driver image data is considered. The discrimination conditions are dynamically adjusted according to the continuity of the sequence numbers of all clustered driver image data in the successfully matched classes, making the discrimination results of the driver's real-time state more accurate and reliable. By combining the clustered driver image data with intelligent optimization algorithms and driver fatigue image feature data, image feature matching is performed. Through multiple iterative searches, driver fatigue image feature data that matches the selected driver image data is accurately identified, realizing the scientific analysis of driver image data. At the same time, an adaptive step size factor is introduced into the algorithm process. The search step size is dynamically adjusted according to the difference in the fitness value of the search data, which improves the search performance of the algorithm and accelerates the convergence speed.

[0059] Preferably, the specific steps for collecting driver physiological characteristic data and vehicle status characteristic data during real-time driving of passenger and freight vehicles are as follows:

[0060] S31. Collect the driver's physiological characteristic data in real time during driving using biometric sensing devices to obtain a driver physiological characteristic dataset. ,in, This indicates the first data point collected during the driver's real-time driving process. Physiological characteristic data The total number of categories representing the driver's physiological characteristics data;

[0061] The biometric sensing device refers to a type of smart bracelet and smart watch;

[0062] The physiological characteristic data includes, but is not limited to, heart rate information, blood pressure information, and respiratory rate;

[0063] By collecting real-time vehicle state feature data from drivers during driving using multi-source sensors mounted on passenger and freight vehicles, a vehicle state feature dataset is obtained. ,in, This indicates the first data point collected during the driver's real-time driving process. Vehicle status characteristic data, The total number of categories representing vehicle status feature data;

[0064] The vehicle status characteristic data includes, but is not limited to, driving speed, acceleration, steering angle, and braking frequency;

[0065] The multi-source sensors include, but are not limited to, vehicle speed sensors, acceleration sensors, angle sensors, pressure sensors, and gyroscope sensors.

[0066] Preferably, the specific steps for acquiring historical data to construct a fatigue level mapping model, and analyzing the driver's fatigue level based on the driver's image data, the driver's physiological characteristic data, the vehicle state characteristic data, and the fatigue level mapping model to obtain driver fatigue level data are as follows:

[0067] S41. Obtain several sets of image data, physiological characteristic data, and vehicle status characteristic data of drivers in a fatigued driving state through the fatigue driving early warning platform to obtain a historical driver image data matrix. Historical driver physiological characteristic data matrix and historical vehicle state feature data matrix as follows:

[0068] , ,

[0069] ,

[0070] in, Indicates the first The first group of drivers collected data while in a state of fatigued driving. Wheel image data, Indicates the first The first group of drivers collected data while in a state of fatigued driving. Physiological characteristic data Indicates the first The first group of drivers collected data while in a state of fatigued driving. Vehicle status characteristic data, This indicates the total number of sets of image data, physiological feature data, and vehicle status feature data acquired when the driver is in a state of fatigued driving.

[0071] S42. A pre-trained multimodal adversarial network is used to fuse the historical driver image data in the historical driver image data matrix, the historical driver physiological feature data in the historical driver physiological feature data matrix, and the historical driver physiological feature data in the historical vehicle state feature data matrix to obtain a historical fatigue driving dataset. ,in, Indicates the first Historical fatigue driving data is obtained by fusing image data, physiological characteristic data, and vehicle status characteristic data collected when the drivers are in a state of fatigue driving.

[0072] S43. Collect the driver fatigue level corresponding to each historical fatigue driving data point in the historical fatigue driving early warning platform online to obtain the historical driver fatigue level dataset. ,in, Indicates the first Data on driver fatigue levels when drivers are in a state of fatigued driving;

[0073] S44. Construct a fatigue level mapping model based on the historical fatigue driving dataset and the historical driver fatigue level dataset;

[0074] S45. Input the driver image dataset, the driver physiological feature dataset, and the vehicle state feature dataset into the fatigue level mapping model to map the driver's fatigue level during real-time driving, generating driver fatigue level data. .

[0075] By using a pre-trained multimodal adversarial network to fuse various historical data, and then training the model with the resulting multimodal data, the model can accurately analyze driver fatigue from multiple dimensions, providing a reliable tool for further analysis of driver fatigue.

[0076] Preferably, the specific steps for selecting the appropriate instruction to execute the fatigue driving warning operation based on driver fatigue level data are as follows:

[0077] S51. Transmit the driver fatigue data via a wireless communication network. The data is pushed to the driver fatigue warning platform, which then uses the driver fatigue level data to... Select the appropriate operation command;

[0078] If the driver fatigue level data If the driver is experiencing mild fatigue, the fatigue driving warning platform will display a fatigue warning sign on the vehicle's screen, sound an alarm through the vehicle's audio system, and adjust the seat's tilt angle to provide vibration feedback to the driver.

[0079] If the driver fatigue level data For moderate fatigue, the fatigue driving warning platform works in concert with the vehicle power system and chassis control system of the passenger and freight vehicles to automatically adjust the vehicle speed to maintain a safe distance, fine-tune the steering assist, increase the feedback of driving operation, and at the same time control the display screen to pop up a fatigue warning sign, control the in-vehicle audio to sound an alarm, and control the seat tilt angle and provide vibration feedback to the driver.

[0080] If the driver fatigue level data In cases of severe fatigue, the fatigue driving warning platform automatically takes over the braking and steering systems of the passenger and freight vehicles through the mechanical exoskeleton-assisted driving equipment mounted on them. According to the preset safety strategy, it controls the alarm to sound an alarm and horn to remind pedestrians and other vehicles to give way. At the same time, it controls the passenger and freight vehicles to slow down, stabilize their driving trajectory, and guide them to the emergency lane, service area, or safe parking spot for parking.

[0081] Select appropriate instructions based on driver fatigue data to maximize driver safety.

[0082] The present invention also includes a multimodal early warning and intervention system for fatigue driving of passenger and freight vehicles, comprising a driver image data clustering module, a driver real-time driving status analysis module, a data acquisition module, a driver fatigue level analysis module, and a fatigue driving early warning operation execution module;

[0083] The driver image data clustering module collects real-time image data of the driver during the driving process online through DMS cameras installed in the passenger and freight vehicles, and obtains driver image data. The module then performs clustering processing on each driver image data in the driver image dataset to obtain driver image clustering data.

[0084] The driver real-time driving status analysis module analyzes the driver's real-time driving status based on the driver image clustering data and preset driver fatigue image feature data, and generates driver real-time driving status data. If the driver is safe, the current fatigue driving warning operation ends; if the driver is fatigued, the process proceeds to S3.

[0085] The data acquisition module collects the driver's physiological characteristics data online during real-time driving through a biometric sensing device, and obtains the driver's physiological characteristics data online through a multi-source sensor mounted on the passenger and freight vehicle to collect the vehicle's status characteristics data online during real-time driving.

[0086] The driver fatigue analysis module acquires several sets of image data, physiological feature data, and vehicle status feature data of drivers in a fatigued driving state through a fatigue driving early warning platform, and fuses these data with a pre-trained multimodal adversarial network to obtain historical fatigue driving data. The fatigue driving early warning platform then collects the driver fatigue level corresponding to each historical fatigue driving data point in the historical fatigue driving dataset online to obtain historical driver fatigue level data. A fatigue level mapping model is constructed based on the historical fatigue driving dataset and the historical driver fatigue level dataset. The driver image data, driver physiological feature data, and vehicle status feature data are input into the fatigue level mapping model to map the driver's fatigue level during real-time driving, generating driver fatigue level data.

[0087] The fatigue driving warning operation execution module pushes the driver's fatigue level data to the fatigue driving warning platform through a wireless communication network, and the fatigue driving warning platform selects the corresponding operation instructions based on the driver's fatigue level data.

[0088] By employing the above technical solution, the present invention provides a multimodal early warning and intervention method and system for fatigue driving of passenger and freight vehicles, which has at least the following beneficial effects:

[0089] 1. This invention clusters driver image data to obtain clustered driver image data, and combines it with driver fatigue image feature data to accurately analyze the driver's real-time driving state. If the analysis result indicates fatigue, it acquires driver physiological feature data and vehicle state feature data, and scientifically analyzes the driver's fatigue level using a fatigue degree mapping model. At the same time, it selects corresponding instructions to execute fatigue driving warning operations based on the driver's fatigue level, realizing timely warning and accurate intervention for multimodal fatigue driving of passenger and freight vehicles.

[0090] 2. This invention optimizes the matching process by clustering the acquired driver image data, narrowing the matching range, reducing computational load, ensuring the accuracy of the matching results, and improving the efficiency of the image matching process. When analyzing the matching results, the influence of the continuity of driver image data is considered. The discrimination conditions are dynamically adjusted according to the continuity of the sequence numbers of all clustered driver image data in the successfully matched classes, making the discrimination results of the driver's real-time status more accurate and reliable.

[0091] 3. This invention combines clustered driver image data with intelligent optimization algorithms and driver fatigue image feature data for image feature matching. Through multiple iterative searches, it accurately identifies driver fatigue image feature data that matches the selected driver image data, thus achieving scientific analysis of driver image data. At the same time, an adaptive step size factor is introduced into the algorithm process to dynamically adjust the search step size according to the difference in fitness values ​​of the search data, thereby improving the search performance of the algorithm and accelerating the convergence speed of the algorithm.

[0092] 4. This invention uses a pre-trained multimodal adversarial network to perform data fusion processing on the acquired historical data. The model is trained using the multimodal data obtained after data fusion, enabling the model to accurately analyze driver fatigue from multiple dimensions, providing a reliable tool for further analysis of driver fatigue. Appropriate instructions are selected based on driver fatigue data to maximize driver safety. Attached Figure Description

[0093] To more clearly illustrate the technical solutions of the embodiments of the invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For those skilled in the art, the drawings can be obtained from these drawings without creative effort.

[0094] Figure 1 A flowchart of the multimodal early warning and intervention method for fatigue driving of passenger and freight vehicles provided by the present invention;

[0095] Figure 2 This is a schematic diagram of the modules of the multimodal early warning and intervention system for fatigue driving of passenger and freight vehicles provided by the present invention. Detailed Implementation

[0096] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0097] Example 1 is as follows:

[0098] To address the issues of poor accuracy and real-time performance in existing fatigue driving detection technologies, this embodiment proposes a multimodal early warning and intervention method for fatigue driving in passenger and freight vehicles. This method accurately analyzes the driver's real-time driving state by combining clustered driver image data with driver fatigue image feature data. If the analysis indicates fatigue, it acquires driver physiological characteristic data and vehicle state characteristic data, and scientifically analyzes the driver's fatigue level using a fatigue degree mapping model. Simultaneously, based on the driver's fatigue level, it selects appropriate instructions to execute fatigue driving early warning operations, achieving timely and accurate multimodal early warning and intervention for fatigue driving in passenger and freight vehicles. Figure 1 As shown, the method includes the following steps:

[0099] S1. Acquire driver image data during real-time driving of passenger and freight vehicles, and perform clustering processing on the driver image data to obtain clustered driver image data. As a specific implementation plan of this method, the detailed plan for this step is as follows:

[0100] S11. By using DMS cameras installed inside passenger and freight vehicles to collect real-time image data of the driver during the driving process, a driver image dataset is obtained. ,in, This indicates the first data point collected during the driver's real-time driving process. Wheel image data, This indicates the total number of driver image data;

[0101] Image data includes, but is not limited to, driver's facial information, driver's eye information, and driver's head information;

[0102] S12. Perform clustering processing on each driver image data in the driver image dataset to obtain driver image clustering data;

[0103] S121. Treat each driver image data in the driver image dataset as a cluster, calculate the Euclidean distance between clusters, and obtain a distance matrix.

[0104] S122. Based on the distance matrix, find the two closest clusters and perform clustering to obtain a new cluster. Recalculate the Euclidean distance between the new cluster and other clusters, and update the distance matrix accordingly. The Euclidean distance calculation formula is as follows:

[0105]

[0106] in, express and The Euclidean distance between them and Represents any two cluster centers 3D feature vectors For vectors In the Values ​​in dimensions For vectors In the Values ​​in a dimension;

[0107] S123. Set the Euclidean distance threshold and repeat steps S121 to S122 until the Euclidean distance between any two clusters is greater than the Euclidean distance threshold, and then obtain the clustered driver image feature data.

[0108] S2. Analyze the driver's real-time driving status based on the clustered driver image data and driver fatigue image feature data to generate real-time driver driving status data. If the driver is safe, end the fatigue driving warning operation; if the driver is fatigued, proceed to S3. The detailed implementation plan for this step is as follows:

[0109] S21. Establish a dataset of driver fatigue image features. ,in, Indicates the first Driver fatigue image feature data, This represents the total number of image feature data points related to driver fatigue.

[0110] S22, Set the first cost value to And the second cost value ;

[0111] If the clustered driver image data belongs to one class, then any clustered driver image data is selected and matched one-to-one with the driver fatigue image feature data in the driver fatigue image feature dataset;

[0112] If a driver fatigue image feature is successfully matched with any driver fatigue image feature data, it means that all driver image data in the driver image dataset are successfully matched with the driver fatigue image feature data, and the real-time driving status data of the driver is output as fatigue.

[0113] Otherwise, it means that all driver image data in the driver image dataset cannot be matched with the driver fatigue image feature data, and the real-time driving status data of the driver is output as safe;

[0114] If the clustered driver image data is of multiple classes, then any clustered driver image data in each class is selected and matched one-to-one with the driver fatigue image feature data in the driver fatigue image feature dataset.

[0115] When the selected clustered driver image data successfully matches the driver fatigue image feature data in the driver fatigue image feature dataset, it means that all the clustered driver image data in the corresponding class of the selected clustered driver image data successfully match the driver fatigue image feature data in the driver fatigue image feature dataset.

[0116] If the selected clustered driver image data fails to match the driver fatigue image feature data in the driver fatigue image feature dataset, it means that all clustered driver image data in the corresponding class of the selected clustered driver image data fails to match the driver fatigue image feature data in the driver fatigue image feature dataset.

[0117] If all the selected clustered driver image data fail to match the driver fatigue image feature data in the driver fatigue image feature dataset, it means that all driver image data in the driver image dataset fails to match the driver fatigue image feature data in the driver fatigue image feature dataset, and the output driver real-time driving status data is safe.

[0118] Otherwise, count the total number of clustered driver image data in all successfully matched classes, and analyze the sequence number corresponding to all clustered driver image data in all successfully matched classes;

[0119] When the serial numbers are consecutive, the total quantity is compared with the value of the first generation. Perform numerical comparisons;

[0120] If the total number is greater than or equal to the value of the first generation, then output the driver's real-time driving status data as fatigue;

[0121] Otherwise, output the driver's real-time driving status data as safe;

[0122] When the serial numbers are not consecutive, combine the total quantity with the second-generation value. Perform numerical comparisons;

[0123] If the total number is greater than or equal to the value of the second generation, then output the driver's real-time driving status data as fatigue;

[0124] Otherwise, output the driver's real-time driving status data as safe;

[0125] The process of selecting any clustered driver image data and matching it one-to-one with driver fatigue image feature data in the driver fatigue image feature dataset includes the following steps:

[0126] S221. Construct an image feature search set, and set the current iteration number to . The maximum number of iterations is And the search space dimension of driver fatigue image feature data is ;

[0127] The driver fatigue image feature dataset is used as the driver fatigue image feature data search space, and features are randomly generated within the driver fatigue image feature data search space. Each driver fatigue image feature data point corresponds to an initial position of an image feature search data point in the image feature search set, thus obtaining the initial position set of the image feature search set. ,in, Represents the first element in the image feature search set. The initial position of the image feature search data;

[0128] S222. Calculate the fitness value between each image feature search data in the image feature search set and the selected clustered driver image data according to the fitness function formula. Sort each image feature search data in the image feature search set from largest to smallest fitness value, and select the image feature search data with the highest fitness value as the current optimal solution. The fitness function formula is as follows:

[0129] ,

[0130] in, Represents the first element in the image feature search set. Fitness values ​​for each image feature search data Represents the first element in the image feature search set. The Euclidean distance between the driver fatigue image feature data corresponding to each image feature search data and the selected clustered driver image data. Indicates the correction value;

[0131] S223. Calculate the spatial distance between each image feature search data in the image feature search set; the formula for calculating the spatial distance is as follows:

[0132] ,

[0133] in, Represents the first element in the image feature search set. Image feature search data and the first image feature search data and the first Spatial distance between image feature search data and These represent the first and second elements in the image feature search set, respectively. Image feature search data and the first image feature search data Image feature search data in the th ... Position in the 3D search space;

[0134] S224. Calculate the attraction between each image feature search data in the image feature search set based on the spatial distance between each image feature search data in the image feature search set;

[0135] ,

[0136] in, Represents the first element in the image feature search set. Image feature search data and the first image feature search data and the first The relative attractiveness between image feature search data, Indicates initial attraction level. Indicates the attraction coefficient;

[0137] S225. In the image feature search set, each image feature search data point will have its position updated in the driver fatigue image feature data search space based on its attractiveness to other image feature search data points; the position update formula is as follows:

[0138] ,

[0139] in, Represents the first element in the image feature search set. The position after updating the location using image feature search data. and These represent the first and second elements in the image feature search set, respectively. Image feature search data and the first image feature search data and the first The current position of the image feature search data. This represents a random number that follows a uniform distribution between (-0.5, 0.5). Represents the first element in the image feature search set. The adaptive step size factor corresponding to each image feature search data, and ,in, Represents the initial step size factor. and These represent the maximum and minimum fitness values, respectively.

[0140] S226. Calculate the fitness value of each image feature search data in the image feature search set after position update. If the fitness value of the image feature search data after position update is greater than the original fitness value, replace the original position with the new position of the image feature search data; otherwise, retain the original position.

[0141] S227. Determine the current iteration number. Is it less than the maximum number of iterations? If the current iteration number Less than the maximum number of iterations Then the current iteration number Increment by 1 and return to S3; otherwise, use the image feature search data with the highest fitness value as the global optimal solution.

[0142] S228. Set a fitness threshold. If the fitness value of the global optimal solution is greater than or equal to the fitness threshold, it means that the selected clustered driver image data is successfully matched with the driver fatigue image feature data in the driver fatigue image feature dataset; otherwise, it means that the selected clustered driver image data is not successfully matched with the driver fatigue image feature data in the driver fatigue image feature dataset.

[0143] S23. When the driver's real-time driving status data is safe, the driver's real-time driving status data is pushed to the fatigue driving warning platform through the wireless communication network, and the current fatigue driving warning operation ends; when the driver's real-time driving status data is fatigued, proceed to S3.

[0144] S3. Collect driver physiological characteristic data and vehicle status characteristic data during real-time driving of passenger and freight vehicles. S31. Collect driver physiological characteristic data online during real-time driving using biometric sensing devices to obtain a driver physiological characteristic dataset. ,in, This indicates the first data point collected during the driver's real-time driving process. Physiological characteristic data The total number of categories representing the driver's physiological characteristics data;

[0145] Biometric sensing devices refer to a type of smart bracelet and smartwatch.

[0146] Physiological characteristic data includes, but is not limited to, heart rate information, blood pressure information, and respiratory rate;

[0147] By collecting real-time vehicle state feature data from drivers during driving using multi-source sensors mounted on passenger and freight vehicles, a vehicle state feature dataset is obtained. ,in, This indicates the first data point collected during the driver's real-time driving process. Vehicle status characteristic data, The total number of categories representing vehicle status feature data;

[0148] Vehicle status characteristic data includes, but is not limited to, driving speed, acceleration, steering angle, and braking frequency;

[0149] Multi-source sensors include, but are not limited to, vehicle speed sensors, acceleration sensors, angle sensors, pressure sensors, and gyroscope sensors.

[0150] S4. Obtain historical data to construct a fatigue level mapping model, and analyze the driver's fatigue level based on driver image data, driver physiological characteristic data, vehicle state characteristic data, and the fatigue level mapping model to obtain driver fatigue level data. As a specific implementation plan of this method, the detailed plan for this step is as follows:

[0151] S41. Obtain several sets of image data, physiological characteristic data, and vehicle status characteristic data of drivers in a fatigued driving state through the fatigue driving early warning platform to obtain a historical driver image data matrix. Historical driver physiological characteristic data matrix and historical vehicle state feature data matrix as follows:

[0152] , ,

[0153] ,

[0154] in, Indicates the first The first group of drivers collected data while in a state of fatigued driving. Wheel image data, Indicates the first The first group of drivers collected data while in a state of fatigued driving. Physiological characteristic data Indicates the first The first group of drivers collected data while in a state of fatigued driving. Vehicle status characteristic data, This indicates the total number of sets of image data, physiological feature data, and vehicle status feature data acquired when the driver is in a state of fatigued driving.

[0155] S42. By using a pre-trained multimodal adversarial network, the historical driver image data in the historical driver image data matrix, the historical driver physiological feature data in the historical driver physiological feature data matrix, and the historical driver physiological feature data in the historical vehicle state feature data matrix are fused to obtain the historical fatigue driving dataset. ,in, Indicates the first Historical fatigue driving data is obtained by fusing image data, physiological characteristic data, and vehicle status characteristic data collected when the drivers are in a state of fatigue driving.

[0156] S43. Collect the driver fatigue level corresponding to each historical fatigue driving data point in the historical fatigue driving early warning platform online to obtain the historical driver fatigue level dataset. ,in, Indicates the first Data on driver fatigue levels when drivers are in a state of fatigued driving;

[0157] S44. Construct a fatigue level mapping model based on historical fatigue driving datasets and historical driver fatigue level datasets;

[0158] S441. Construct the initial Transformer model;

[0159] S442. Set the training data ratio, and divide the historical fatigue driving dataset and the historical driver fatigue level dataset according to the training data ratio to obtain the historical fatigue driving training dataset, the historical driver fatigue level training dataset, the historical fatigue driving test dataset, and the historical driver fatigue level test dataset.

[0160] S443. Set the training error threshold and the maximum number of training iterations. Input the historical fatigue driving training dataset as training data and the historical driver fatigue level training dataset as training label data into the initial Transformer model for training. Continuously adjust the parameters of the initial Transformer model according to the training results until the training error is less than the training error threshold or the number of training iterations is greater than the maximum number of training iterations, and obtain the trained Transformer model.

[0161] S444. Set an accuracy threshold. Input the historical fatigue driving test dataset as test data and the historical driver fatigue level test dataset as test label data into the trained Transformer model for testing. Calculate the accuracy of the test results. If the accuracy of the test results is greater than or equal to the accuracy threshold, the fatigue level mapping model is obtained; otherwise, return to S443 until the accuracy of the test results is greater than or equal to the accuracy threshold.

[0162] S45. Input the driver image dataset, driver physiological feature dataset, and vehicle state feature dataset into the fatigue level mapping model to map the driver's fatigue level during real-time driving, generating driver fatigue level data. .

[0163] S5. Based on the driver fatigue level data, select the appropriate instruction to execute the fatigue driving warning operation. As a specific implementation plan of this method, the detailed plan for this step is as follows:

[0164] S51. Transmit driver fatigue data via wireless communication network The data is pushed to the driver fatigue warning platform, which then analyzes the driver's fatigue level data. Select the appropriate operation command;

[0165] If driver fatigue level data If the driver is experiencing mild fatigue, the fatigue driving warning platform will display a fatigue warning sign on the vehicle's screen, sound an alarm through the vehicle's audio system, and adjust the seat's tilt angle to provide vibration feedback to the driver.

[0166] If driver fatigue level data For moderate fatigue, the fatigue driving warning platform works in concert with the vehicle power system and chassis control system of the passenger and freight vehicles to automatically adjust the vehicle speed to maintain a safe distance, fine-tune the steering assist, increase the feedback of driving operation, and at the same time control the display screen to pop up a fatigue warning sign, control the in-vehicle audio to sound an alarm, and control the seat tilt angle and provide vibration feedback to the driver.

[0167] If driver fatigue level data In cases of severe fatigue, the fatigue driving warning platform automatically takes over the braking and steering systems of the passenger and freight vehicles through the mechanical exoskeleton-assisted driving equipment mounted on them. According to the preset safety strategy, it controls the alarm to sound an alarm and horn to remind pedestrians and other vehicles to give way. At the same time, it controls the passenger and freight vehicles to slow down, stabilize their driving trajectory, and guide them to the emergency lane, service area, or safe parking spot for parking.

[0168] Example 2 is as follows:

[0169] Please see Figure 2 A multimodal early warning and intervention system for fatigue driving of passenger and freight vehicles includes a driver image data clustering module, a driver real-time driving status analysis module, a data acquisition module, a driver fatigue level analysis module, and a fatigue driving early warning operation execution module.

[0170] The driver image data clustering module collects real-time image data of drivers during driving through DMS cameras installed in passenger and freight vehicles, obtains driver image data, and performs clustering processing on each driver image data in the driver image dataset to obtain driver image clustering data.

[0171] The driver real-time driving status analysis module analyzes the driver's real-time driving status based on driver image clustering data and preset driver fatigue image feature data, and generates driver real-time driving status data. If the driver is safe, the current fatigue driving warning operation ends; if the driver is fatigued, the process proceeds to S3.

[0172] The data acquisition module collects the driver's physiological characteristics data online in real time through biometric sensing devices to obtain the driver's physiological characteristics data; and collects the vehicle status characteristics data online in real time through multi-source sensors mounted on the passenger and freight vehicles to obtain the vehicle status characteristics data.

[0173] The driver fatigue analysis module acquires several sets of image data, physiological feature data, and vehicle status feature data of drivers in a fatigued driving state through a fatigue driving early warning platform, and fuses this data with a pre-trained multimodal adversarial network to obtain historical fatigue driving data. It then collects the driver fatigue level corresponding to each historical fatigue driving data point from the fatigue driving early warning platform online to obtain historical driver fatigue level data. Based on the historical fatigue driving dataset and the historical driver fatigue level dataset, a fatigue level mapping model is constructed. Driver image data, driver physiological feature data, and vehicle status feature data are input into the fatigue level mapping model to map the driver's fatigue level during real-time driving, generating driver fatigue level data.

[0174] The fatigue driving warning operation execution module pushes driver fatigue level data to the fatigue driving warning platform through a wireless communication network. The fatigue driving warning platform selects the corresponding operation instructions based on the driver fatigue level data.

[0175] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0176] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0177] The preferred embodiments of the invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention.

Claims

1. A multimodal early warning and intervention method for fatigue driving in passenger and freight vehicles, characterized in that, Includes the following steps: S1. Acquire driver image data during real-time driving of passenger and freight vehicles, and perform clustering processing on the driver image data to obtain clustered driver image data. S2. Analyze the driver's real-time driving status based on the clustered driver image data and driver fatigue image feature data to generate real-time driver driving status data. If it is for safety reasons, then end this fatigue driving warning operation; If fatigue occurs, proceed to S3; S3. Collect driver physiological characteristic data and vehicle status characteristic data during real-time driving of passenger and freight vehicles; S4. Obtain historical data to construct a fatigue level mapping model, and analyze the driver's fatigue level based on the driver image data, the driver's physiological characteristic data, the vehicle state characteristic data, and the fatigue level mapping model to obtain driver fatigue level data. S5. Based on the driver fatigue level data, select the appropriate instruction to execute the fatigue driving warning operation, including the following steps: The driver fatigue level data is transmitted via wireless communication network. The data is pushed to the driver fatigue warning platform, which then uses the driver fatigue level data to... Select the appropriate operation command; If the driver fatigue level data If the driver is experiencing mild fatigue, the fatigue driving warning platform will display a fatigue warning sign on the vehicle's screen, sound an alarm through the vehicle's audio system, and adjust the seat's tilt angle to provide vibration feedback to the driver. If the driver fatigue level data For moderate fatigue, the fatigue driving warning platform works in concert with the vehicle power system and chassis control system of the passenger and freight vehicles to automatically adjust the vehicle speed to maintain a safe distance, fine-tune the steering assist, increase the feedback of driving operation, and at the same time control the display screen to pop up a fatigue warning sign, control the in-vehicle audio to sound an alarm, and control the seat tilt angle and provide vibration feedback to the driver. If the driver fatigue level data In cases of severe fatigue, the fatigue driving warning platform automatically takes over the braking and steering systems of the passenger and freight vehicles through the mechanical exoskeleton-assisted driving equipment mounted on them. According to the preset safety strategy, it controls the alarm to sound an alarm and horn to remind pedestrians and other vehicles to give way. At the same time, it controls the passenger and freight vehicles to slow down, stabilize their driving trajectory, and guide them to the emergency lane, service area, or safe parking spot for parking.

2. The multimodal early warning and intervention method for fatigue driving of passenger and freight vehicles according to claim 1, characterized in that, S1 includes the following steps: S11. By using DMS cameras installed inside passenger and freight vehicles to collect real-time image data of the driver during the driving process, a driver image dataset is obtained. ,in, This indicates the first data point collected during the driver's real-time driving process. Wheel image data, This indicates the total number of driver image data; S12. Perform clustering processing on each driver image data in the driver image dataset to obtain driver image clustering data; S121. Treat each driver image data in the driver image dataset as a cluster, calculate the Euclidean distance between clusters, and obtain a distance matrix. S122. Based on the distance matrix, find the two closest clusters and perform clustering to obtain a new cluster. Recalculate the Euclidean distance between the new cluster and other clusters, and update the distance matrix. S123. Set the Euclidean distance threshold, and repeat the steps in S121 to S122 until the Euclidean distance between any two clusters is greater than the Euclidean distance threshold, and obtain the clustered driver image feature data.

3. The multimodal early warning and intervention method for fatigue driving of passenger and freight vehicles according to claim 2, characterized in that, S2 includes the following steps: S21. Establish a dataset of driver fatigue image features. ,in, Indicates the first Driver fatigue image feature data, This represents the total number of image feature data points related to driver fatigue. S22, Set the first cost value to And the second cost value ; If the clustered driver image data belongs to one class, then any clustered driver image data is selected and matched one-to-one with the driver fatigue image feature data in the driver fatigue image feature dataset; If a driver fatigue image feature data is successfully matched with any driver fatigue image feature data, it means that all driver image data in the driver image dataset are successfully matched with the driver fatigue image feature data, and the real-time driving status data of the driver is output as fatigue. Otherwise, it indicates that all driver image data in the driver image dataset fails to match the driver fatigue image feature data, and the driver's real-time driving status data is output as safe. If the clustered driver image data is of multiple classes, then any one of the clustered driver image data in each class is selected and matched one by one with the driver fatigue image feature data in the driver fatigue image feature dataset. When the selected clustered driver image data successfully matches the driver fatigue image feature data in the driver fatigue image feature dataset, it means that all the clustered driver image data in the corresponding class of the selected clustered driver image data successfully match the driver fatigue image feature data in the driver fatigue image feature dataset. If the selected clustered driver image data fails to match the driver fatigue image feature data in the driver fatigue image feature dataset, it means that all clustered driver image data in the corresponding class of the selected clustered driver image data fails to match the driver fatigue image feature data in the driver fatigue image feature dataset. If all the selected clustered driver image data fail to match the driver fatigue image feature data in the driver fatigue image feature dataset, it means that all driver image data in the driver image dataset fails to match the driver fatigue image feature data in the driver fatigue image feature dataset, and the real-time driving status data of the driver is output as safe. Otherwise, count the total number of clustered driver image data in all successfully matched classes, and analyze the sequence number corresponding to all clustered driver image data in all successfully matched classes; When the serial numbers are consecutive, the total quantity is compared with the value of the first generation. Perform numerical comparisons; If the total number is greater than or equal to the value of the first generation, then the driver's real-time driving status data is output as fatigue. Otherwise, output the driver's real-time driving status data as safe; When the serial numbers are not consecutive, the total quantity is compared with the second generation value. Perform numerical comparisons; If the total number is greater than or equal to the value of the second generation, then the driver's real-time driving status data is output as fatigue. Otherwise, output the driver's real-time driving status data as safe; S23. When the driver's real-time driving status data is safe, the driver's real-time driving status data is pushed to the fatigue driving warning platform through the wireless communication network, and the current fatigue driving warning operation ends; when the driver's real-time driving status data is fatigued, proceed to S3.

4. The multimodal early warning and intervention method for fatigue driving of passenger and freight vehicles according to claim 3, characterized in that, The step of selecting any clustered driver image data and matching it one-to-one with the driver fatigue image feature data in the driver fatigue image feature dataset includes the following steps: S221. Construct an image feature search set, and set the current iteration number to . The maximum number of iterations is And the search space dimension of driver fatigue image feature data is ; The driver fatigue image feature dataset is used as the driver fatigue image feature data search space, and random generation is performed within the driver fatigue image feature data search space. Each driver fatigue image feature data point corresponds to the initial position of an image feature search data point in the image feature search set, thus obtaining the initial position set of the image feature search set. ,in, Indicates the first element in the image feature search set. The initial position of the image feature search data; S222. Calculate the fitness value between each image feature search data in the image feature search set and the selected clustered driver image data according to the fitness function formula. Arrange each image feature search data in the image feature search set in descending order of fitness value, and select the image feature search data with the highest fitness value as the current optimal solution. The fitness function formula is as follows: ， in, Indicates the first element in the image feature search set. Fitness values ​​for each image feature search data Indicates the first element in the image feature search set. The Euclidean distance between the driver fatigue image feature data corresponding to each image feature search data and the selected clustered driver image data. Indicates the correction value; S223. Calculate the spatial distance between each image feature search data in the image feature search set; the spatial distance calculation formula is as follows: ， in, Indicates the first element in the image feature search set. Image feature search data and the first image feature search data and the first Spatial distance between image feature search data and These respectively represent the first in the image feature search set. Image feature search data and the first image feature search data Image feature search data in the th ... Position in the 3D search space; S224. Calculate the attraction between each image feature search data in the image feature search set based on the spatial distance between each image feature search data in the image feature search set; ， in, Indicates the first element in the image feature search set. Image feature search data and the first image feature search data and the first The relative attractiveness between image feature search data, Indicates initial attraction level. Indicates the attraction coefficient; S225. The position of each image feature search data in the image feature search set within the driver fatigue image feature data search space is updated based on its attractiveness to other image feature search data; the position update formula is as follows: ， in, Indicates the first element in the image feature search set. The position after updating the location using image feature search data. and These respectively represent the first in the image feature search set. Image feature search data and the first image feature search data and the first The current position of the image feature search data. This represents a random number that follows a uniform distribution between (-0.5, 0.5). Indicates the first element in the image feature search set. The adaptive step size factor corresponding to each image feature search data, and ,in, Represents the initial step size factor. and These represent the maximum and minimum fitness values, respectively. S226. Calculate the fitness value of each image feature search data in the image feature search set after position update. If the fitness value of the image feature search data after position update is greater than the original fitness value, replace the original position with the new position of the image feature search data; otherwise, retain the original position. S227. Determine the current iteration number. Is it less than the maximum number of iterations? If the current iteration number Less than the maximum number of iterations Then the current iteration number Increment by 1 and return to S3; otherwise, use the image feature search data with the highest fitness value as the global optimal solution. S228. Set a fitness threshold. If the fitness value of the global optimal solution is greater than or equal to the fitness threshold, it means that the selected clustered driver image data is successfully matched with the driver fatigue image feature data in the driver fatigue image feature dataset; otherwise, it means that the selected clustered driver image data is not successfully matched with the driver fatigue image feature data in the driver fatigue image feature dataset.

5. The multimodal early warning and intervention method for fatigue driving of passenger and freight vehicles according to claim 4, characterized in that, S3 includes the following steps: S31. Collect the driver's physiological characteristic data in real time during driving using biometric sensing devices to obtain a driver physiological characteristic dataset. ,in, This indicates the first data point collected during the driver's real-time driving process. Physiological characteristic data The total number of categories representing the driver's physiological characteristics data; By collecting real-time vehicle state feature data from drivers during driving using multi-source sensors mounted on passenger and freight vehicles, a vehicle state feature dataset is obtained. ,in, This indicates the first data point collected during the driver's real-time driving process. Vehicle status characteristic data, This represents the total number of categories of vehicle status feature data.

6. The multimodal early warning and intervention method for fatigue driving of passenger and freight vehicles according to claim 5, characterized in that, S4 includes the following steps: S41. Obtain several sets of image data, physiological characteristic data, and vehicle status characteristic data of drivers in a fatigued driving state through the fatigue driving early warning platform to obtain a historical driver image data matrix. Historical driver physiological characteristic data matrix and historical vehicle state feature data matrix ; S42. A pre-trained multimodal adversarial network is used to fuse the historical driver image data in the historical driver image data matrix, the historical driver physiological feature data in the historical driver physiological feature data matrix, and the historical vehicle state feature data in the historical vehicle state feature data matrix to obtain a historical fatigue driving dataset. ,in, Indicates the first Historical fatigue driving data is obtained by fusing image data, physiological characteristic data, and vehicle status characteristic data collected when the drivers are in a state of fatigue driving. S43. Collect the driver fatigue level corresponding to each historical fatigue driving data point in the historical fatigue driving early warning platform online to obtain the historical driver fatigue level dataset. ,in, Indicates the first Data on driver fatigue levels when drivers are in a state of fatigued driving; S44. Construct a fatigue level mapping model based on the historical fatigue driving dataset and the historical driver fatigue level dataset; S45. Input the driver image dataset, the driver physiological feature dataset, and the vehicle state feature dataset into the fatigue level mapping model to map the driver's fatigue level during real-time driving, generating driver fatigue level data. .

7. The multimodal early warning and intervention method for fatigue driving of passenger and freight vehicles according to claim 6, characterized in that, The fatigue level mapping model adopts the Transformer model.

8. A multimodal early warning and intervention system for fatigue driving in passenger and freight vehicles, characterized in that, This method is used to implement the multimodal early warning and intervention method for fatigue driving of passenger and freight vehicles as described in any one of claims 1-7.

Images