Multi-mode early warning intervention method and system for fatigue driving of van

Abstract

The invention relates to the technical field of fatigue driving early warning, in particular to a multi-mode early warning intervention method and system for fatigue driving of a van, and the method comprises the steps: carrying out the clustering processing of driver image data, obtaining the clustered driver image data, reducing the matching range, optimizing the matching process, and reducing the calculation amount; the real-time driving state of the driver is accurately analyzed in combination with driver fatigue image feature data, if the analysis result is fatigue, driver physiological feature data and vehicle state feature data are obtained, and the fatigue degree of the driver is scientifically analyzed in combination with a fatigue degree mapping model. Meanwhile, a corresponding instruction is selected according to the fatigue degree of the driver to execute fatigue driving early warning operation, the safety of the driver is protected to the maximum degree, and multi-mode timely early warning and accurate intervention of fatigue driving of the van are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of fatigue driving warning technology, and in particular to a multimodal warning intervention method and system for fatigue driving of passenger and freight vehicles. Background Art

[0002] Fatigue driving is one of the main causes of passenger and freight vehicle traffic accidents. Traditional fatigue driving detection methods are often based on single-modal data, resulting in poor detection accuracy. At the same time, the amount of calculation required to analyze the driver's status is large, resulting in a long calculation process and prone to errors in the calculation results, which in turn makes the response results less real-time and the analysis results less accurate. Summary of the Invention

[0003] In response to the problems in the related art, the present invention provides a multimodal warning intervention method and system for fatigue driving of passenger and freight vehicles to overcome the above-mentioned technical problems existing in the existing related art.

[0004] To solve the above technical problems, the present invention provides the following technical solution: a multimodal warning intervention method for fatigue driving of passenger and freight vehicles, comprising the following steps: S1. Acquire driver image data of a passenger or freight vehicle during real-time driving, and perform clustering processing on the driver image data to obtain clustered driver image data; S2. Analyze the driver's real-time driving status based on the clustered driver image data and the driver fatigue image feature data to generate real-time driving status data of the driver; If it is safe, then end the fatigue driving warning operation; If it is fatigue, enter S3; S3, collecting driver physiological characteristic data and vehicle status characteristic data during real-time driving of passenger and freight vehicles; S4. Acquire historical data to construct a fatigue level mapping model, and analyze the driver's fatigue level based on the driver image data, the driver's physiological characteristic data, the vehicle state characteristic data, and the fatigue level mapping model to obtain driver fatigue level data; S5. Select corresponding instructions to execute fatigue driving warning operations based on the driver's fatigue level data.

[0005] Preferably, the specific steps of obtaining driver image data during the real-time driving process of a passenger or freight vehicle and performing clustering processing on the driver image data to obtain driver image cluster data are as follows: S11. Collect the image data of the driver during real-time driving online through the DMS camera installed in the passenger and freight vehicle to obtain the driver image dataset ,in, Indicates the first Image data, Indicates the total number of driver image data; The image data includes but is not limited to driver's facial information, driver's eye information and driver's head information; S12, performing clustering processing on each driver image data in the driver image dataset to obtain driver image cluster data; S121, treating each driver image data in the driver image dataset as a cluster, calculating the Euclidean distance between clusters, and obtaining a distance matrix; S122. Find the two closest clusters according to the distance matrix and perform clustering to obtain a new cluster. Recalculate the Euclidean distance between the new cluster and the other clusters and update the distance matrix. The Euclidean distance calculation formula is as follows: in, express and The Euclidean distance between and represents the center of any two clusters dimensional feature vector, is a vector In the The value of the dimension, is a vector In the The value of the dimension; S123 , setting a Euclidean distance threshold, repeating steps S121 to S122 until the Euclidean distance between any two clusters is greater than the Euclidean distance threshold, and obtaining clustered driver image feature data.

[0006] By clustering the acquired driver image data, the matching range is narrowed, thereby optimizing the matching process and reducing the amount of calculation. This ensures the accuracy of the matching results while improving the efficiency of the image matching process.

[0007] Preferably, the real-time driving state of the driver is analyzed based on the driver image clustering data and the driver fatigue image feature data to generate real-time driving state data of the driver. If the driver is safe, the fatigue driving warning operation is terminated; if the driver is fatigued, the specific steps of S3 are as follows: S21. Establishing a driver fatigue image feature dataset ,in, Indicates the Driver fatigue image feature data, Indicates the total number of driver fatigue image feature data; S22, set the first cost value And the second cost is ; If the clustered driver image data belongs to one category, selecting any clustered driver image data and performing one-to-one matching with the driver fatigue image feature data in the driver fatigue image feature dataset; If the driver fatigue image feature data is successfully matched with any one of the driver fatigue image feature data, it means that all the driver image data in the driver image data set are successfully matched with the driver fatigue image feature data, and the driver's real-time driving status data is output as fatigue; Otherwise, it indicates that all driver image data in the driver image data set are unsuccessfully matched with the driver fatigue image feature data, and the driver's real-time driving status data is output as safe; If the clustered driver image data is multi-class, then any one of the clustered driver image data is selected in each class to be matched one-to-one with the driver fatigue image feature data in the driver fatigue image feature dataset; When the selected clustered driver image data successfully matches the driver fatigue image feature data in the driver fatigue image feature dataset, it means that all clustered driver image data in the class corresponding to the selected clustered driver image data successfully matches the driver fatigue image feature data in the driver fatigue image feature dataset; When the selected clustered driver image data fails to match the driver fatigue image feature data in the driver fatigue image feature dataset, it means that all clustered driver image data in the class corresponding to the selected clustered driver image data fail to match the driver fatigue image feature data in the driver fatigue image feature dataset; If all the selected clustered driver image data fail to match the driver fatigue image feature data in the driver fatigue image feature dataset, it means that all the driver image data in the driver image dataset fail to match the driver fatigue image feature data in the driver fatigue image feature dataset, and the real-time driving status data of the driver is output as safe; Otherwise, the total number of all clustered driver image data in the successfully matched class is counted, and the serial numbers corresponding to all clustered driver image data in the successfully matched class are analyzed; When the sequence numbers are continuous, the total quantity is added to the first generation value. Perform numerical comparisons; If the total number is greater than or equal to the first generation value, outputting the driver's real-time driving status data as fatigue; Otherwise, outputting the driver's real-time driving status data is safe; When the sequence number is not continuous, the total quantity is added to the second generation value Perform numerical comparisons; If the total amount is greater than or equal to the second-generation value, outputting the driver's real-time driving status data as fatigue; Otherwise, outputting the driver's real-time driving status data is safe; The selecting of any clustered driver image data and performing one-to-one matching with the driver fatigue image feature data in the driver fatigue image feature dataset comprises the following steps: S221, build image feature search set, set the current number of iterations to , the maximum number of iterations is And the search space dimension of driver fatigue image feature data is ; The driver fatigue image feature data set is used as a driver fatigue image feature data search space, and a random Driver fatigue image feature data, each driver fatigue image feature data corresponds to the initial position of an image feature search data in the image feature search set, and the initial position set of the image feature search set is obtained. ,in, Indicates the first The initial position of the image feature search data; S222. Calculate the fitness value between each image feature search data in the image feature search set and the selected clustered driver image data according to a fitness function formula, arrange each image feature search data in the image feature search set from largest to smallest according to the fitness value, and select the image feature search data with the highest fitness value as the current optimal solution; the fitness function formula is as follows: , in, Indicates the first The fitness value of image feature search data, Indicates the first The Euclidean distance between the driver fatigue image feature data corresponding to the image feature search data and the selected clustered driver image data, Indicates the correction value; S223. Calculate the spatial distance between each image feature search data in the image feature search set. The spatial distance calculation formula is as follows: , in, Indicates the first image feature search data and The spatial distance between image feature search data, and Respectively represent the first Image feature search data and Image feature search data in Position in the dimensional search space; S224, calculating the attractiveness between each image feature search data in the image feature search set according to the spatial distance between each image feature search data in the image feature search set; , in, Indicates the first image feature search data and The relative attractiveness between image feature search data, represents the initial attraction, represents the attraction coefficient; S225. Each image feature search data in the image feature search set is updated in the driver fatigue image feature data search space according to its attraction to other image feature search data. The position update formula is as follows: , in, Indicates the first The position after the image feature search data is updated, and Respectively represent the first image feature search data and The current position of the image feature search data, Indicates a random number that is uniformly distributed between (-0.5, 0.5). Indicates the first The adaptive step size factor corresponding to the image feature search data, and ,in, represents the initial step size factor, and Represent the maximum and minimum fitness values ​​respectively; S226, calculating the fitness value of each image feature search data in the image feature search set after the position is updated, and if the fitness value of the image feature search data after the position is updated is greater than the original fitness value, replacing the original position with the new position of the image feature search data; otherwise, retaining the original position; S227, determine the current number of iterations Is it less than the maximum number of iterations? , if the current number of iterations Less than the maximum number of iterations , then the current number of iterations Add 1 and return to S3; otherwise, the image feature search data with the highest fitness value is used as the global optimal solution; S228. Setting a fitness threshold. If the fitness value of the global optimal solution is greater than or equal to the fitness threshold, it indicates that the selected clustered driver image data successfully matches the driver fatigue image feature data in the driver fatigue image feature dataset; otherwise, it indicates that the selected clustered driver image data fails to match the driver fatigue image feature data in the driver fatigue image feature dataset. S23. When the real-time driving status data of the driver is safe, the real-time driving status data of the driver is pushed to the fatigue driving warning platform through the wireless communication network, and the fatigue driving warning operation is ended; when the real-time driving status data of the driver is fatigue, enter S3.

[0008] When analyzing the matching results, the influence of the continuity of the driver image data is taken into account, and the discrimination conditions are dynamically adjusted according to the continuity of the serial numbers corresponding to all clustered driver image data in the successfully matched class, so that the discrimination results of the driver's real-time status are more accurate and reliable; image feature matching is performed by combining the clustered driver image data with the intelligent optimization algorithm and the driver fatigue image feature data. Through multiple iterative searches, the driver fatigue image feature data that matches the selected driver image data is accurately identified, and scientific analysis of the driver image data is achieved. At the same time, an adaptive step size factor is introduced in the algorithm process, and the search step size is dynamically adjusted according to the difference in the fitness value of the search data, which improves the search performance of the algorithm and accelerates the convergence speed of the algorithm.

[0009] Preferably, the specific steps for collecting the driver's physiological characteristic data and vehicle status characteristic data during the real-time driving process of a passenger or freight vehicle are as follows: S31. Collect the driver's physiological characteristic data during real-time driving online through the biometric sensing device to obtain the driver's physiological characteristic data set ,in, Indicates the first Physiological characteristic data, Indicates the total number of categories of driver physiological characteristic data; The biometric sensing device refers to a smart bracelet and a smart watch; The physiological characteristic data include but are not limited to heart rate information, blood pressure information and respiratory rate; The vehicle status feature data set is obtained by collecting the vehicle status feature data of the driver in real time during driving through the multi-source sensors installed on the passenger and freight vehicles. ,in, Indicates the first Vehicle status characteristic data, Indicates the total number of categories of vehicle status feature data; The vehicle status characteristic data includes but is not limited to driving speed, acceleration, steering angle and braking frequency; The multi-source sensor includes but is not limited to a vehicle speed sensor, an acceleration sensor, an angle sensor, a pressure sensor, and a gyroscope sensor.

[0010] Preferably, the specific steps of acquiring historical data to construct a fatigue level mapping model, and analyzing the driver's fatigue level based on the driver image data, the driver's physiological characteristic data, the vehicle state characteristic data, and the fatigue level mapping model to obtain the driver's fatigue level data are as follows: S41, obtain image data, physiological characteristic data and vehicle status characteristic data of several groups of drivers in fatigue driving state through the fatigue driving warning platform, and obtain historical driver image data matrix , Historical driver physiological characteristics data matrix And the historical vehicle status feature data matrix as follows: , , , in, Indicates the The first data collected when the group of drivers were in fatigue driving state wheel image data, Indicates the The first data collected when the group of drivers were in fatigue driving state Physiological characteristic data, Indicates the The first data collected when the group of drivers were in fatigue driving state Vehicle status characteristic data, Indicates the total number of sets of image data, physiological characteristic data, and vehicle status characteristic data obtained when the driver is in a fatigue driving state; S42, using a pre-trained multimodal adversarial network, data fusion is performed on the historical driver image data in the historical driver image data matrix, the historical driver physiological feature data in the historical driver physiological feature data matrix, and the historical driver physiological feature data in the historical vehicle state feature data matrix to obtain a historical fatigue driving data set. ,in, Indicates the The historical fatigue driving data is obtained by data fusion of image data, physiological characteristic data and vehicle status characteristic data collected when the group of drivers are in a fatigue driving state; S43, collecting the driver fatigue level corresponding to each historical fatigue driving data in the historical fatigue driving data set online through the fatigue driving warning platform to obtain a historical driver fatigue level data set ,in, Indicates the Driver fatigue level data of a group of drivers when they are in a fatigue driving state; S44, constructing a fatigue level mapping model based on the historical fatigue driving dataset and the historical driver fatigue level dataset; S45: Input the driver image data set, the driver physiological characteristic data set, and the vehicle state characteristic data set into a fatigue degree mapping model to map the driver's fatigue degree during real-time driving, and generate driver fatigue degree data. .

[0011] The pre-trained multimodal adversarial network is used to perform data fusion processing on the various historical data obtained. The multimodal data training model obtained after data fusion enables the model to accurately analyze the driver's fatigue level from multiple dimensions, providing a reliable tool for further analysis of the driver's fatigue level.

[0012] Preferably, the specific steps of selecting corresponding instructions to execute fatigue driving warning operation according to the driver fatigue level data are as follows: S51, transmitting the driver fatigue level data via a wireless communication network Push to the fatigue driving warning platform, the fatigue driving warning platform will calculate the driver's fatigue level data Select the corresponding operation instruction; If the driver fatigue level data If the driver is experiencing mild fatigue, the fatigue driving warning platform will control the fatigue warning sign to pop up on the vehicle display, control the in-vehicle audio system to sound an alarm, and control the seat tilt angle and provide vibration feedback to the driver. If the driver fatigue level data If the driver is moderately fatigued, the fatigue driving warning platform works in conjunction with the vehicle's powertrain and chassis control systems to automatically adjust the vehicle speed to maintain a safe distance, fine-tune the steering assist, and increase feedback on driving operations. It also controls the fatigue warning icon to pop up on the onboard display, the in-car audio system to sound an alarm, and the seat's tilt angle to provide vibration feedback to the driver. If the driver fatigue level data If the fatigue is severe, the fatigue driving warning platform will automatically take over the braking system and steering system of the passenger and freight vehicle through the mechanical exoskeleton assisted driving equipment installed on the passenger and freight vehicle. According to the preset safety strategy, the alarm will be controlled to sound the horn to remind pedestrians and moving vehicles to pay attention and avoid. At the same time, the passenger and freight vehicle will be controlled to slow down, stabilize its driving trajectory, and guide it to the emergency lane, service area or safe parking spot for parking.

[0013] Select corresponding instructions based on the driver's fatigue data to maximize the driver's safety.

[0014] The present invention also includes a multimodal warning intervention system for fatigue driving of passenger and freight vehicles, comprising a driver image data clustering module, a driver real-time driving status analysis module, a data acquisition module, a driver fatigue level analysis module, and a fatigue driving warning operation execution module; The driver image data clustering module collects image data of the driver in real time during driving online through a DMS camera installed in a passenger or freight vehicle to obtain driver image data, and performs clustering processing on each driver image data in the driver image data set to obtain driver image cluster data; The driver real-time driving state analysis module analyzes the driver's real-time driving state according to the driver image clustering data and the preset driver fatigue image feature data to generate the driver's real-time driving state data. If the driver is safe, the fatigue driving warning operation is terminated; if the driver is fatigued, the process proceeds to S3; The data acquisition module collects the driver's physiological characteristic data in real time during driving online through the biometric sensing device to obtain the driver's physiological characteristic data; and collects the vehicle status characteristic data in real time during driving online through the multi-source sensors installed on the passenger and freight vehicles to obtain the vehicle status characteristic data; The driver fatigue level analysis module obtains image data, physiological characteristic data, and vehicle status characteristic data of several groups of drivers in a fatigue driving state through a fatigue driving warning platform, and performs data fusion in combination with a pre-trained multimodal adversarial network to obtain historical fatigue driving data; collects the driver fatigue level corresponding to each historical fatigue driving data in the historical fatigue driving data set online through the fatigue driving warning platform to obtain historical driver fatigue level data; constructs a fatigue level mapping model based on the historical fatigue driving data set and the historical driver fatigue level data set; inputs the driver image data, the driver physiological characteristic data, and the vehicle status characteristic data into the fatigue level mapping model to map the driver's fatigue level during real-time driving, thereby generating driver fatigue level data; The fatigue driving warning operation execution module pushes the driver fatigue level data to the fatigue driving warning platform through a wireless communication network, and the fatigue driving warning platform selects corresponding operation instructions according to the driver fatigue level data.

[0015] By means of the above technical solution, the present invention provides a multi-modal warning intervention method and system for fatigue driving of passenger and freight vehicles, which has at least the following beneficial effects: 1. The present invention clusters driver image data to obtain clustered driver image data, and combines the driver's fatigue image feature data to accurately analyze the driver's real-time driving status. If the analysis result is fatigue, the driver's physiological feature data and vehicle status feature data are obtained, and the driver's fatigue level is scientifically analyzed in combination with a fatigue level mapping model. At the same time, corresponding instructions are selected according to the driver's fatigue level to execute fatigue driving warning operations, thereby realizing multi-modal timely warning and accurate intervention of fatigue driving in passenger and freight vehicles.

[0016] 2. The present invention clusters the acquired driver image data to narrow the matching range, thereby optimizing the matching process, reducing the amount of calculation, ensuring the accuracy of the matching results, and improving the efficiency of the image matching process; when analyzing the matching results, the influence of the continuity of the driver image data is taken into account, and the judgment conditions are dynamically adjusted according to the continuity of the serial numbers corresponding to all clustered driver image data in the successfully matched class, so that the judgment results of the driver's real-time status are more accurate and reliable.

[0017] 3. The present invention performs image feature matching by combining clustered driver image data with an intelligent optimization algorithm and driver fatigue image feature data. Through multiple iterative searches, the present invention accurately identifies the driver fatigue image feature data that matches the selected driver image data, thereby achieving scientific analysis of the driver image data. At the same time, an adaptive step size factor is introduced into the algorithm process, and the search step size is dynamically adjusted according to the difference in fitness values ​​of the search data, thereby improving the search performance of the algorithm and accelerating the convergence speed of the algorithm.

[0018] 4. The present invention uses a pre-trained multimodal adversarial network to perform data fusion processing on the acquired historical data. The multimodal data training model obtained after data fusion enables the model to accurately analyze the driver's fatigue level from multiple dimensions, providing a reliable tool for further analysis of the driver's fatigue level; and selects corresponding instructions based on the driver's fatigue level data to maximize the driver's safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the invention. For ordinary technicians in this field, they can also obtain drawings based on these drawings without paying any creative work.

[0020] Figure 1 A flowchart of the multimodal warning intervention method for fatigue driving of passenger and freight vehicles provided by the present invention; Figure 2 This is a module schematic diagram of the multimodal warning intervention system for fatigue driving of passenger and freight vehicles provided by the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0022] Example 1 is as follows: Aiming at the problem of poor accuracy and real-time performance of existing fatigue driving detection technology. This embodiment proposes a multimodal warning intervention method for fatigue driving of passenger and freight vehicles. The real-time driving status of the driver is accurately analyzed by combining clustered driver image data with driver fatigue image feature data. If the analysis result is fatigue, the driver's physiological feature data and vehicle status feature data are obtained, and the driver's fatigue level is scientifically analyzed in combination with the fatigue level mapping model. At the same time, the corresponding instructions are selected according to the driver's fatigue level to execute fatigue driving warning operations, thereby realizing multimodal timely warning and accurate intervention of fatigue driving of passenger and freight vehicles. Figure 1 As shown, the method includes the following steps: S1. Acquire driver image data of a passenger or freight vehicle during real-time driving, and perform clustering processing on the driver image data to obtain clustered driver image data. As a specific implementation plan of this method, the detailed plan of this step is as follows: S11. Collect the image data of the driver during real-time driving online through the DMS camera installed in the passenger and freight vehicle to obtain the driver image dataset ,in, Indicates the first wheel image data, Indicates the total number of driver image data; Image data includes but is not limited to driver's facial information, driver's eye information, and driver's head information; S12, performing clustering processing on each driver image data in the driver image dataset to obtain driver image cluster data; S121, treating each driver image data in the driver image dataset as a cluster, calculating the Euclidean distance between clusters, and obtaining a distance matrix; S122. Find the two closest clusters according to the distance matrix and perform clustering to obtain a new cluster. Recalculate the Euclidean distance between the new cluster and the other clusters and update the distance matrix. The Euclidean distance calculation formula is as follows: in, express and The Euclidean distance between and represents the center of any two clusters dimensional feature vector, is a vector In the The value of the dimension, is a vector In the The value of the dimension; S123. Set a Euclidean distance threshold, and repeat steps S121 to S122 until the Euclidean distance between any two clusters is greater than the Euclidean distance threshold, thereby obtaining clustered driver image feature data.

[0023] S2. Analyze the driver's real-time driving status based on the clustered driver image data and the driver fatigue image feature data to generate real-time driver driving status data. If the driver is safe, end the fatigue driving warning operation; if the driver is fatigued, proceed to S3. As a specific implementation plan of this method, the detailed plan of this step is as follows: S21. Establishing a driver fatigue image feature dataset ,in, Indicates the Driver fatigue image feature data, Indicates the total number of driver fatigue image feature data; S22, set the first cost value And the second cost is ; If the clustered driver image data is of one class, any clustered driver image data is selected for one-to-one matching with the driver fatigue image feature data in the driver fatigue image feature dataset; If the matching is successful with any driver fatigue image feature data, it means that all driver image data in the driver image dataset are successfully matched with the driver fatigue image feature data, and the output driver real-time driving status data is fatigue; Otherwise, it means that all driver image data in the driver image dataset are not successfully matched with the driver fatigue image feature data, and the driver's real-time driving status data is output as safe; If the clustered driver image data is multi-class, then any clustered driver image data is selected from each class to be matched one-to-one with the driver fatigue image feature data in the driver fatigue image feature dataset; When the selected clustered driver image data successfully matches the driver fatigue image feature data in the driver fatigue image feature dataset, it means that all clustered driver image data in the class corresponding to the selected clustered driver image data successfully matches the driver fatigue image feature data in the driver fatigue image feature dataset; When the selected clustered driver image data fails to match the driver fatigue image feature data in the driver fatigue image feature dataset, it means that all clustered driver image data in the class corresponding to the selected clustered driver image data fail to match the driver fatigue image feature data in the driver fatigue image feature dataset; If all the selected clustered driver image data fail to match the driver fatigue image feature data in the driver fatigue image feature dataset, it means that all the driver image data in the driver image dataset fail to match the driver fatigue image feature data in the driver fatigue image feature dataset, and the output driver real-time driving status data is safe; Otherwise, the total number of all clustered driver image data in the successfully matched class is counted, and the serial numbers corresponding to all clustered driver image data in the successfully matched class are analyzed; When the sequence numbers are continuous, add the total quantity to the first generation value Perform numerical comparisons; If the total number is greater than or equal to the first generation value, the driver's real-time driving status data is output as fatigue; Otherwise, outputting the driver's real-time driving status data is safe; When the sequence number is not continuous, the total quantity is combined with the second generation value Perform numerical comparisons; If the total number is greater than or equal to the second-generation value, the driver's real-time driving status data is output as fatigue; Otherwise, outputting the driver's real-time driving status data is safe; Selecting any clustered driver image data and matching it one-to-one with the driver fatigue image feature data in the driver fatigue image feature dataset includes the following steps: S221, build image feature search set, set the current number of iterations to , the maximum number of iterations is And the search space dimension of driver fatigue image feature data is ; The driver fatigue image feature dataset is used as the driver fatigue image feature data search space, and the driver fatigue image feature data is randomly generated in the driver fatigue image feature data search space. Driver fatigue image feature data, each driver fatigue image feature data corresponds to the initial position of an image feature search data in the image feature search set, and the initial position set of the image feature search set is obtained. ,in, Represents the first The initial position of the image feature search data; S222. Calculate the fitness value between each image feature search data in the image feature search set and the selected clustered driver image data according to the fitness function formula, arrange each image feature search data in the image feature search set from largest to smallest according to the fitness value, and select the image feature search data with the highest fitness value as the current optimal solution; the fitness function formula is as follows: , in, Represents the first The fitness value of image feature search data, Represents the first The Euclidean distance between the driver fatigue image feature data corresponding to the image feature search data and the selected clustered driver image data, Indicates the correction value; S223. Calculate the spatial distance between each image feature search data in the image feature search set. The spatial distance calculation formula is as follows: , in, Represents the first image feature search data and The spatial distance between image feature search data, and Respectively represent the first Image feature search data and Image feature search data in Position in the dimensional search space; S224, calculating the attractiveness between each image feature search data in the image feature search set according to the spatial distance between each image feature search data in the image feature search set; , in, Represents the first image feature search data and The relative attractiveness between image feature search data, represents the initial attraction, represents the attraction coefficient; S225. Each image feature search data in the image feature search set is updated in the driver fatigue image feature data search space based on its attraction to other image feature search data. The position update formula is as follows: , in, Represents the first The position after the image feature search data is updated, and Respectively represent the first image feature search data and The current position of the image feature search data, Indicates a random number that is uniformly distributed between (-0.5, 0.5). Represents the first The adaptive step size factor corresponding to the image feature search data, and ,in, represents the initial step size factor, and Represent the maximum and minimum fitness values ​​respectively; S226, calculating the fitness value of each image feature search data in the image feature search set after the position is updated. If the fitness value of the image feature search data after the position is updated is greater than the original fitness value, the new position of the image feature search data is used to replace the original position; otherwise, the original position is retained; S227, determine the current number of iterations Is it less than the maximum number of iterations? , if the current number of iterations Less than the maximum number of iterations , then the current number of iterations Add 1 and return to S3; otherwise, the image feature search data with the highest fitness value is used as the global optimal solution; S228. Setting a fitness threshold. If the fitness value of the global optimal solution is greater than or equal to the fitness threshold, it indicates that the selected clustered driver image data successfully matches the driver fatigue image feature data in the driver fatigue image feature dataset; otherwise, it indicates that the selected clustered driver image data successfully matches the driver fatigue image feature data in the driver fatigue image feature dataset. S23. When the real-time driving status data of the driver is safe, the real-time driving status data of the driver is pushed to the fatigue driving warning platform through the wireless communication network, and the fatigue driving warning operation is ended; when the real-time driving status data of the driver is fatigued, enter S3.

[0024] S3, collect the driver's physiological characteristics data and vehicle status characteristics data during the real-time driving process of the passenger and freight vehicle. S31, collect the driver's physiological characteristics data during the real-time driving process online through the biometric sensing device, and obtain the driver's physiological characteristics data set ,in, Indicates the first Physiological characteristic data, Indicates the total number of categories of driver physiological characteristic data; Biometric sensing device refers to a type of smart bracelet and smart watch; Physiological characteristic data includes but is not limited to heart rate information, blood pressure information, and respiratory rate; The vehicle status feature data set is obtained by collecting the vehicle status feature data of the driver in real time during driving through the multi-source sensors installed on the passenger and freight vehicles. ,in, Indicates the first Vehicle status characteristic data, Indicates the total number of categories of vehicle status feature data; Vehicle status characteristic data includes but is not limited to driving speed, acceleration, steering angle, and braking frequency; The multi-source sensors include but are not limited to vehicle speed sensors, acceleration sensors, angle sensors, pressure sensors, and gyroscope sensors.

[0025] S4. Acquire historical data to construct a fatigue level mapping model, and analyze the driver's fatigue level based on the driver image data, the driver's physiological characteristic data, the vehicle status characteristic data, and the fatigue level mapping model to obtain driver fatigue level data. As a specific implementation plan of this method, the detailed plan of this step is as follows: S41, obtain image data, physiological characteristic data and vehicle status characteristic data of several groups of drivers in fatigue driving state through the fatigue driving warning platform, and obtain historical driver image data matrix , Historical driver physiological characteristics data matrix And the historical vehicle status feature data matrix as follows: , , , in, Indicates the The first data collected when the group of drivers were in fatigue driving state wheel image data, Indicates the The first data collected when the group of drivers were in fatigue driving state Physiological characteristic data, Indicates the The first data collected when the group of drivers were in fatigue driving state Vehicle status characteristic data, Indicates the total number of sets of image data, physiological characteristic data, and vehicle status characteristic data obtained when the driver is in a fatigue driving state; S42. Using a pre-trained multimodal adversarial network, the historical driver image data in the historical driver image data matrix, the historical driver physiological feature data in the historical driver physiological feature data matrix, and the historical driver physiological feature data in the historical vehicle state feature data matrix are fused to obtain a historical fatigue driving dataset. ,in, Indicates the The historical fatigue driving data is obtained by data fusion of image data, physiological characteristic data and vehicle status characteristic data collected when the group of drivers are in a fatigue driving state; S43, collecting the driver fatigue level corresponding to each historical fatigue driving data in the historical fatigue driving data set online through the fatigue driving warning platform, and obtaining the historical driver fatigue level data set ,in, Indicates the Driver fatigue level data of a group of drivers when they are in a fatigue driving state; S44, constructing a fatigue level mapping model based on a historical fatigue driving dataset and a historical driver fatigue level dataset; S441, build the initial Transformer model; S442: Setting a training data ratio, and dividing the historical fatigue driving dataset and the historical driver fatigue level dataset according to the training data ratio to obtain a historical fatigue driving training dataset, a historical driver fatigue level training dataset, a historical fatigue driving test dataset, and a historical driver fatigue level test dataset; S443: Setting a training error threshold and a maximum number of training times, inputting the historical fatigue driving training dataset as training data and the historical driver fatigue level training dataset as training label data into the initial Transformer model for training, and continuously adjusting the parameters of the initial Transformer model according to the training results until the training error is less than the training error threshold or the number of training times is greater than the maximum number of training times, thereby obtaining a trained Transformer model; S444: Set an accuracy threshold, input the historical fatigue driving test dataset as test data and the historical driver fatigue level test dataset as test label data into the trained Transformer model for testing, calculate the accuracy of the test results, and if the accuracy of the test results is greater than or equal to the accuracy threshold, obtain a fatigue level mapping model; otherwise, return to S443 until the accuracy of the test results is greater than or equal to the accuracy threshold; S45: Input the driver image data set, the driver physiological characteristic data set, and the vehicle state characteristic data set into the fatigue degree mapping model to map the driver's fatigue degree during real-time driving, and generate driver fatigue degree data. .

[0026] S5. Selecting corresponding instructions to execute fatigue driving warning operations based on the driver fatigue level data. As a specific implementation plan of this method, the detailed plan of this step is as follows: S51, transmit the driver fatigue level data via wireless communication network Push to the fatigue driving warning platform, the fatigue driving warning platform based on the driver's fatigue level data Select the corresponding operation instruction; If the driver fatigue data If the driver is experiencing mild fatigue, the fatigue driving warning platform will control the fatigue warning sign to pop up on the vehicle display, control the in-vehicle audio system to sound an alarm, and control the seat tilt angle and provide vibration feedback to the driver. If the driver fatigue data If the driver is moderately fatigued, the fatigue driving warning platform works in conjunction with the vehicle's powertrain and chassis control systems to automatically adjust the vehicle speed to maintain a safe distance, fine-tune the steering assist, and increase feedback on driving operations. It also controls the fatigue warning icon to pop up on the onboard display, the in-car audio system to sound an alarm, and the seat's tilt angle to provide vibration feedback to the driver. If the driver fatigue data If the fatigue is severe, the fatigue driving warning platform will automatically take over the braking system and steering system of the passenger and freight vehicle through the mechanical exoskeleton assisted driving equipment installed on the passenger and freight vehicle. According to the preset safety strategy, the alarm will be controlled to sound the horn to remind pedestrians and moving vehicles to pay attention and avoid. At the same time, the passenger and freight vehicle will be controlled to slow down, stabilize its driving trajectory, and guide it to the emergency lane, service area or safe parking spot for parking.

[0027] The second embodiment is as follows: See also Figure 2 , a multimodal warning and intervention system for fatigue driving of passenger and freight vehicles, including a driver image data clustering module, a driver real-time driving status analysis module, a data acquisition module, a driver fatigue level analysis module and a fatigue driving warning operation execution module; The driver image data clustering module collects the image data of the driver in real time during driving online through the DMS camera installed in the passenger and freight vehicle to obtain the driver image data, and performs clustering processing on the individual driver image data in the driver image dataset to obtain the driver image cluster data; The driver's real-time driving status analysis module analyzes the driver's real-time driving status based on the driver image clustering data and the preset driver fatigue image feature data to generate the driver's real-time driving status data. If it is safe, the fatigue driving warning operation is terminated; if it is fatigue, the process proceeds to S3; The data acquisition module collects the driver's physiological characteristic data during real-time driving online through biometric sensing equipment to obtain the driver's physiological characteristic data; the multi-source sensors installed on the passenger and freight vehicles collect the driver's vehicle status characteristic data during real-time driving online to obtain vehicle status characteristic data; The driver fatigue analysis module obtains image data, physiological characteristic data, and vehicle status characteristic data of several groups of drivers in fatigued driving states through the fatigue driving warning platform, and combines this data with a pre-trained multimodal adversarial network to fuse the data to obtain historical fatigue driving data. The fatigue driving warning platform collects the driver fatigue level corresponding to each historical fatigue driving data in the historical fatigue driving dataset online to obtain historical driver fatigue level data. A fatigue level mapping model is constructed based on the historical fatigue driving dataset and the historical driver fatigue level dataset. The driver image data, driver physiological characteristic data, and vehicle status characteristic data are input into the fatigue level mapping model to map the driver's fatigue level during real-time driving to generate driver fatigue level data. The fatigue driving warning operation execution module pushes the driver's fatigue level data to the fatigue driving warning platform through the wireless communication network. The fatigue driving warning platform selects corresponding operation instructions based on the driver's fatigue level data.

[0028] Those skilled in the art will appreciate that all or part of the steps in the above-mentioned embodiment methods can be accomplished by instructing the relevant hardware through a program. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0029] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0030] The preferred embodiments of the invention disclosed above are intended only to help illustrate the invention. These preferred embodiments do not exhaust all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention.

Claims

1. A multi-modal warning intervention method for fatigue driving of passenger and freight vehicles, characterized in that: The steps include: S1. Acquire driver image data of a passenger or freight vehicle during real-time driving, and perform clustering processing on the driver image data to obtain clustered driver image data; S2. Analyze the driver's real-time driving status based on the clustered driver image data and the driver fatigue image feature data to generate real-time driving status data of the driver; If it is safe, then end the fatigue driving warning operation; If it is fatigue, enter S3; S3, collecting driver physiological characteristic data and vehicle status characteristic data during real-time driving of passenger and freight vehicles; S4. Acquire historical data to construct a fatigue level mapping model, and analyze the driver's fatigue level based on the driver image data, the driver's physiological characteristic data, the vehicle state characteristic data, and the fatigue level mapping model to obtain driver fatigue level data; S5. Select corresponding instructions to execute fatigue driving warning operations based on the driver's fatigue level data.

2. The multimodal warning intervention method for fatigue driving of passenger and freight vehicles according to claim 1 is characterized in that: Said S1 comprises the following steps: S11. Collect the image data of the driver during real-time driving online through the DMS camera installed in the passenger and freight vehicle to obtain the driver image dataset ,in, Indicates the first wheel image data, Indicates the total number of driver image data; S12, performing clustering processing on each driver image data in the driver image dataset to obtain driver image cluster data; S121, treating each driver image data in the driver image dataset as a cluster, calculating the Euclidean distance between clusters, and obtaining a distance matrix; S122. Find the two closest clusters according to the distance matrix and perform clustering to obtain a new cluster. Recalculate the Euclidean distance between the new cluster and other clusters to update the distance matrix. S123 , setting a Euclidean distance threshold, repeating steps S121 to S122 until the Euclidean distance between any two clusters is greater than the Euclidean distance threshold, and obtaining clustered driver image feature data.

3. The multimodal warning intervention method for fatigue driving of passenger and freight vehicles according to claim 2 is characterized in that: The S2 comprises the following steps: S21. Establishing a driver fatigue image feature dataset ,in, Indicates the Driver fatigue image feature data, Indicates the total number of driver fatigue image feature data; S22, set the first cost value And the second cost is ; If the clustered driver image data are of one category, any one of the clustered driver image data is selected for one-to-one matching with the driver fatigue image feature data in the driver fatigue image feature dataset; If the driver fatigue image feature data is successfully matched with any one of the driver fatigue image feature data, it means that all the driver image data in the driver image dataset are successfully matched with the driver fatigue image feature data, and the driver's real-time driving status data is output as fatigue; Otherwise, it indicates that all driver image data in the driver image data set are unsuccessfully matched with the driver fatigue image feature data, and the driver's real-time driving status data is output as safe; If the clustered driver image data is multi-class, then any one of the clustered driver image data is selected in each class to be matched one-to-one with the driver fatigue image feature data in the driver fatigue image feature dataset; When the selected clustered driver image data successfully matches the driver fatigue image feature data in the driver fatigue image feature dataset, it means that all clustered driver image data in the class corresponding to the selected clustered driver image data successfully matches the driver fatigue image feature data in the driver fatigue image feature dataset; When the selected clustered driver image data fails to match the driver fatigue image feature data in the driver fatigue image feature dataset, it means that all clustered driver image data in the class corresponding to the selected clustered driver image data fail to match the driver fatigue image feature data in the driver fatigue image feature dataset; If all the selected clustered driver image data fail to match the driver fatigue image feature data in the driver fatigue image feature dataset, it means that all the driver image data in the driver image dataset fail to match the driver fatigue image feature data in the driver fatigue image feature dataset, and the real-time driving status data of the driver is output as safe; Otherwise, the total number of all clustered driver image data in the successfully matched class is counted, and the serial numbers corresponding to all clustered driver image data in the successfully matched class are analyzed; When the sequence numbers are continuous, the total quantity is added to the first generation value. Perform numerical comparisons; If the total number is greater than or equal to the first generation value, outputting the driver's real-time driving status data as fatigue; Otherwise, outputting the driver's real-time driving status data is safe; When the sequence number is not continuous, the total quantity is added to the second generation value Perform numerical comparisons; If the total amount is greater than or equal to the second-generation value, outputting the driver's real-time driving status data as fatigue; Otherwise, outputting the driver's real-time driving status data is safe; S23. When the real-time driving status data of the driver is safe, the real-time driving status data of the driver is pushed to the fatigue driving warning platform through the wireless communication network, and the fatigue driving warning operation is ended; when the real-time driving status data of the driver is fatigue, enter S3.

4. The multimodal warning intervention method for fatigue driving of passenger and freight vehicles according to claim 3 is characterized in that: The selecting of any clustered driver image data and performing one-to-one matching with the driver fatigue image feature data in the driver fatigue image feature dataset comprises the following steps: S221, build image feature search set, set the current number of iterations to , the maximum number of iterations is And the search space dimension of driver fatigue image feature data is ; The driver fatigue image feature data set is used as a driver fatigue image feature data search space, and a random Driver fatigue image feature data, each driver fatigue image feature data corresponds to the initial position of an image feature search data in the image feature search set, and the initial position set of the image feature search set is obtained. ,in, Indicates the first The initial position of the image feature search data; S222. Calculate the fitness value between each image feature search data in the image feature search set and the selected clustered driver image data according to a fitness function formula, arrange each image feature search data in the image feature search set from largest to smallest according to the fitness value, and select the image feature search data with the highest fitness value as the current optimal solution; the fitness function formula is as follows: ， in, Indicates the first The fitness value of image feature search data, Indicates the first The Euclidean distance between the driver fatigue image feature data corresponding to the image feature search data and the selected clustered driver image data, Indicates the correction value; S223. Calculate the spatial distance between each image feature search data in the image feature search set. The spatial distance calculation formula is as follows: ， in, Indicates the first image feature search data and the The spatial distance between image feature search data, and Respectively represent the first Image feature search data and Image feature search data in Position in the dimensional search space; S224, calculating the attractiveness between each image feature search data in the image feature search set according to the spatial distance between each image feature search data in the image feature search set; ， in, Indicates the first image feature search data and the The relative attractiveness between image feature search data, represents the initial attraction, represents the attraction coefficient; S225. Each image feature search data in the image feature search set is updated in the driver fatigue image feature data search space according to its attraction to other image feature search data. The position update formula is as follows: ， in, Indicates the first The position after the image feature search data is updated, and Respectively represent the first image feature search data and the The current position of the image feature search data, Indicates a random number that is uniformly distributed between (-0.5, 0.5). Indicates the first The adaptive step size factor corresponding to the image feature search data, and ,in, represents the initial step size factor, and Represent the maximum and minimum fitness values ​​respectively; S226, calculating the fitness value of each image feature search data in the image feature search set after the position is updated, and if the fitness value of the image feature search data after the position is updated is greater than the original fitness value, replacing the original position with the new position of the image feature search data; otherwise, retaining the original position; S227, determine the current number of iterations Is it less than the maximum number of iterations? , if the current number of iterations Less than the maximum number of iterations , then the current number of iterations Add 1 and return to S3; otherwise, the image feature search data with the highest fitness value is used as the global optimal solution; S228. Set a fitness threshold. If the fitness value of the global optimal solution is greater than or equal to the fitness threshold, it means that the selected clustered driver image data and the driver fatigue image feature data in the driver fatigue image feature data set are successfully matched; otherwise, it means that the selected clustered driver image data and the driver fatigue image feature data in the driver fatigue image feature data set are unsuccessful.

5. The multi-modal warning intervention method for fatigue driving of passenger and freight vehicles according to claim 4 is characterized in that: The S3 includes the following steps: S31. Collect the driver's physiological characteristic data during real-time driving online through the biometric sensing device to obtain the driver's physiological characteristic data set ,in, Indicates the first Physiological characteristic data, Indicates the total number of categories of driver physiological characteristic data; The vehicle status feature data set is obtained by collecting the vehicle status feature data of the driver in real time during driving through the multi-source sensors installed on the passenger and freight vehicles. ,in, Indicates the first Vehicle status characteristic data, Indicates the total number of classes of vehicle status feature data.

6. The multimodal warning intervention method for fatigue driving of passenger and freight vehicles according to claim 5 is characterized in that: The S4 comprises the following steps: S41, obtain image data, physiological characteristic data and vehicle status characteristic data of several groups of drivers in fatigue driving state through the fatigue driving warning platform, and obtain historical driver image data matrix , Historical driver physiological characteristics data matrix And the historical vehicle status feature data matrix ; S42, using a pre-trained multimodal adversarial network, data fusion is performed on the historical driver image data in the historical driver image data matrix, the historical driver physiological feature data in the historical driver physiological feature data matrix, and the historical driver physiological feature data in the historical vehicle state feature data matrix to obtain a historical fatigue driving data set. ,in, Indicates the The historical fatigue driving data is obtained by data fusion of image data, physiological characteristic data and vehicle status characteristic data collected when the group of drivers are in a fatigue driving state; S43, collecting the driver fatigue level corresponding to each historical fatigue driving data in the historical fatigue driving data set online through the fatigue driving warning platform to obtain a historical driver fatigue level data set ,in, Indicates the Driver fatigue level data of a group of drivers when they are in a fatigue driving state; S44, constructing a fatigue level mapping model based on the historical fatigue driving dataset and the historical driver fatigue level dataset; S45: Input the driver image data set, the driver physiological characteristic data set, and the vehicle state characteristic data set into a fatigue degree mapping model to map the driver's fatigue degree during real-time driving, and generate driver fatigue degree data. .

7. The multimodal warning intervention method for fatigue driving of passenger and freight vehicles according to claim 6 is characterized in that: The fatigue degree mapping model adopts the Transformer model.

8. The multimodal warning intervention method for fatigue driving of passenger and freight vehicles according to claim 7 is characterized in that: The S5 comprises the following steps: S51, transmitting the driver fatigue level data via a wireless communication network Push to the fatigue driving warning platform, the fatigue driving warning platform will calculate the driver's fatigue level data Select the corresponding operation instruction; If the driver fatigue level data If the driver is experiencing mild fatigue, the fatigue driving warning platform will control the fatigue warning sign to pop up on the vehicle display, the in-vehicle audio system to sound an alarm, and the seat tilt angle to provide vibration feedback to the driver. If the driver fatigue level data If the driver is moderately fatigued, the fatigue driving warning platform works in conjunction with the vehicle's powertrain and chassis control systems to automatically adjust the vehicle speed to maintain a safe distance, fine-tune the steering assist, and increase feedback on driving operations. It also controls the fatigue warning icon to pop up on the onboard display, the in-car audio system to sound an alarm, and the seat's tilt angle to provide vibration feedback to the driver. If the driver fatigue level data If the fatigue is severe, the fatigue driving warning platform will automatically take over the braking system and steering system of the passenger and freight vehicle through the mechanical exoskeleton assisted driving equipment installed on the passenger and freight vehicle. According to the preset safety strategy, the alarm will be controlled to sound the horn to remind pedestrians and moving vehicles to pay attention and avoid. At the same time, the passenger and freight vehicle will be controlled to slow down, stabilize its driving trajectory, and guide it to the emergency lane, service area or safe parking spot for parking.

9. A system for implementing the multimodal warning intervention method for fatigue driving of passenger and freight vehicles according to any one of claims 1 to 8.

Images