Driving behavior identification method based on multi-modal data

Abstract

The invention relates to a driving behavior identification method based on multi-modal data. The method comprises the following steps: S1, acquiring fixation data of a driver and running state data of a vehicle in a driving process; s2, performing density estimation processing on the gaze data to obtain a gaze density dispersion coefficient and a gaze hidden danger coefficient; performing weighted average and smoothing processing on the running state data to obtain a vehicle stationary system; s3, based on the driving evaluation model, the driving behavior of the driver in the driving process is comprehensively analyzed in combination with the gaze density dispersion coefficient, the gaze hidden danger coefficient and the vehicle stability coefficient, so that a driving evaluation coefficient is obtained; and S4, identifying the driving behavior of the driver based on the driving evaluation coefficient, matching a threshold value of the driving evaluation coefficient according to a driving scene, comparing the driving evaluation coefficient with the threshold value, and judging a comparison result to prompt the driving behavior of the driver. The driving behavior recognition precision can be improved.

Description

Technical Field

[0001] The present invention relates to the field of driving behavior recognition, and in particular to a driving behavior recognition method based on multimodal data. Background Art

[0002] In modern transportation systems, driving behavior recognition technology plays a vital role. Driving behavior recognition technology is based on the real-time collection of the driver's physiological and behavioral data. It collects physiological data such as the driver's facial expression, eye gaze direction, heart rate, and dynamic data such as the vehicle's speed, acceleration, and steering angle through on-board sensors and information systems. Subsequently, signal processing technology is used to pre-process the raw data, extract key features, and classify and analyze driving behavior through pattern recognition and machine learning algorithms.

[0003] With the rapid development of intelligent driving technology and the Internet of Vehicles, driver behavior monitoring and safety reminders during driving have become increasingly important. The driver's concentration and the vehicle's operating status are the core factors affecting driving safety. However, in existing technologies, most systems can only monitor the driver's attention or the vehicle's operating status separately, and few systems can comprehensively consider these two aspects of data to conduct real-time evaluation of driving behavior and issue warnings. In addition, in different driving scenarios, the driver's behavior standards and risk assessments need to be dynamically adjusted to improve the accuracy of the assessment. Summary of the invention

[0004] The purpose of the present invention is to provide a driving behavior recognition method based on multimodal data to solve the problem of inaccurate driving behavior recognition caused by failure to comprehensively consider the driver's concentration and the operating status of the vehicle in the prior art.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A driving behavior recognition method based on multimodal data comprises the following steps:

[0007] S1: Collect the driver's gaze data and vehicle operation status data during driving;

[0008] S2: performing density estimation processing on the gaze data to obtain a gaze density dispersion coefficient and a gaze concealment risk coefficient, wherein the gaze density dispersion coefficient and the gaze concealment risk coefficient are used to represent the driver's gaze information;

[0009] Performing weighted averaging and smoothing processing on the running status data to obtain a vehicle stability coefficient, wherein the vehicle stability coefficient is used to represent the vehicle status;

[0010] S3: Based on the driving evaluation model, the driver's driving behavior is comprehensively analyzed during driving by combining the gaze density dispersion coefficient, gaze concealment risk coefficient and vehicle stability coefficient to obtain the driving evaluation coefficient;

[0011] S4: matching a threshold of the driving evaluation coefficient according to a driving scenario, comparing the driving evaluation coefficient with the threshold, and determining a comparison result to provide a driving behavior prompt to the driver.

[0012] According to the above technical means, the gaze data of the driver and the operating status data of the vehicle are collected, and the characteristics of driving behavior are captured from multiple dimensions at the same time, thereby improving the comprehensiveness and reliability of the analysis; the driving behavior of the driver is identified from the driver behavior and the vehicle status, so that the driving behavior can be identified more accurately, thereby effectively improving the accuracy of driving behavior identification; a driving evaluation model is constructed through the driver information and vehicle status data in historical driving scenarios, and the evaluation criteria can be dynamically adjusted according to different driving scenarios during the evaluation process, thereby improving the adaptability and accuracy of the evaluation; the threshold of the driving evaluation coefficient can be matched according to the driving scenario, and the driving evaluation coefficient can be compared with the threshold of the driving evaluation coefficient matched with the same driving scenario, and the driver can be prompted with driving behavior according to the judgment result, which helps the driver to adjust the driving behavior in time and reduce the occurrence of traffic accidents.

[0013] Furthermore, it also includes S0: through different historical driving scenarios, obtaining the driver's gaze data and the vehicle's operating status data in the historical driving scenarios, and weighting the gaze density dispersion coefficient, gaze hidden danger coefficient and vehicle stability coefficient in the historical driving scenarios to construct a driving evaluation model.

[0014] According to the above technical means, by utilizing the driver's gaze data and the vehicle's operating status data in different historical driving scenarios, a more comprehensive and accurate driving evaluation model can be constructed, thereby improving the generalization ability of the driving model and the ability to adapt to different driving environments; weighted processing of the gaze density dispersion coefficient, gaze hidden danger coefficient and vehicle stability coefficient in historical driving scenarios can more accurately reflect the impact of different factors on driving behavior, thereby improving the accuracy of driving evaluation, and the robustness of model recognition can be improved according to the historical driving scenarios in different historical scenarios, so that it can work stably under various actual driving conditions.

[0015] Furthermore, the S0 also includes: evaluating the driving evaluation models of different historical driving scenarios respectively to obtain a balanced F score and a driving evaluation coefficient under each historical driving scenario; and determining a threshold of the driving evaluation coefficient under the driving scenario through each balanced F score and each driving evaluation coefficient.

[0016] According to the above technical means, the driving evaluation model of different historical driving scenarios is evaluated respectively, which can optimize the performance of the driving evaluation model and ensure that the best recognition effect of the driver's behavior can be achieved in various driving scenarios; by determining the threshold of the driving evaluation coefficient in each driving scenario, the driving evaluation model can adapt to different driving environments and conditions, thereby improving the adaptability and flexibility of the driving evaluation model.

[0017] Further, the obtaining of the balanced F-score and the driving evaluation coefficients under each historical driving scenario specifically includes performing a plurality of driving tests through each historical driving scenario to obtain the driving evaluation coefficient and the balanced F-score consistent with the number of driving tests, and each balanced F-score corresponds one-to-one to each driving evaluation coefficient.

[0018] According to the above technical means, a driving evaluation coefficient and an F1 value consistent with the number of driving tests are obtained through multiple driving tests, and the F1 value and the driving evaluation coefficient are derived based on historical driving behaviors under historical scenarios, rather than based on theory or simulation, and can reliably evaluate driving behaviors, thereby improving the accuracy and reliability of threshold matching; through multiple tests and data analysis, the deviations that may occur in the model under specific conditions can be identified and reduced, thereby improving the robustness of the model.

[0019] Furthermore, determining the threshold of the driving evaluation coefficient in the driving scenario through each of the balanced F scores and each of the driving evaluation coefficients specifically includes plotting a curve of the balanced F scores and each of the driving evaluation coefficients in each of the historical driving scenarios, obtaining the driving evaluation coefficient corresponding to the maximum balanced F score in the curve, and using the driving evaluation coefficient as the threshold of the driving evaluation coefficient in the historical driving scenario.

[0020] According to the above technical means, the F1 value and driving evaluation coefficient under each historical driving scenario are obtained, and a curve is drawn to determine the driving evaluation coefficient corresponding to the maximum F1 value as a threshold value. The threshold value of the driving evaluation coefficient under the driving scenario can be accurately determined, thereby improving the accuracy of driving behavior recognition; through accurate threshold values ​​and real-time driving behavior evaluation, driving safety can be significantly improved, helping drivers avoid potential dangerous situations.

[0021] Furthermore, the gaze density dispersion coefficient is obtained by the following sub-steps:

[0022] S21: acquiring the gaze point coordinate data of the driver at a first time interval during driving, and taking all the gaze point data of the driver as a first set, wherein each element in the first set is the gaze point data of the driver;

[0023] S22: smoothing the influence range of each element in the first set by using a kernel function, thereby obtaining the density distribution of each element in the first set in space;

[0024] S23: obtaining a total density estimation value of all the elements according to the density distribution, dividing the total density estimation value according to the number of the first time intervals in the first set, and thereby obtaining a mean value and a standard deviation of the total density estimation value;

[0025] S24: Obtain the coefficient of variation of the total density estimate according to the mean value and the standard deviation, and calculate the gaze density dispersion coefficient according to the coefficient of variation.

[0026] According to the above technical means, by obtaining the coordinate data of the driver's gaze point during driving and performing kernel function smoothing on it, the driver's gaze behavior can be analyzed more finely, thereby more accurately capturing the driver's attention distribution; by calculating the gaze density dispersion coefficient, the driver's gaze behavior can be quantified to provide objective data support for driving behavior analysis; by collecting and analyzing gaze data at different time intervals, the driver's gaze behavior can be monitored in real time, abnormal behavior can be discovered in time and responded to.

[0027] Furthermore, the gaze concealment risk coefficient is obtained through the following sub-steps:

[0028] S25: determining the driver's region of interest, matching all the driver's gaze points in the region of interest as a second set in each of the first time intervals, wherein each element in the second set is the driver's gaze point in the region of interest, performing kernel density estimation on each element in the second set to obtain a density estimation value, and then obtaining a function of the density estimation value changing over time;

[0029] S26: Obtain a density estimation threshold value according to the variation function, obtain a deviation function through the density estimation threshold value, and further obtain the gaze concealment risk coefficient.

[0030] According to the above technical means, by determining the driver's region of interest (R), the driver's gaze behavior in a specific area can be analyzed more accurately, which is crucial for understanding the driver's attention allocation and predicting potential risk behaviors; by obtaining the density estimation value change function of the driver's region of interest over time through the kernel density estimation formula, the driver's gaze behavior can be dynamically monitored to promptly detect distraction or inattention; by real-time monitoring of the driver's gaze hidden danger coefficient, timely reminders can be given when the driver is distracted or inattentive, thereby improving driving safety; by obtaining the deviation function and the gaze hidden danger coefficient, the system can evaluate the deviation between the driver's gaze behavior and the safe driving requirements, issue warnings in time, and reduce traffic accidents caused by inattention.

[0031] Furthermore, the vehicle stability coefficient is obtained through the following sub-steps:

[0032] S21': Acquire vehicle status data during driving, and obtain a smoothed value of the vehicle status data by exponentially weighted moving average;

[0033] S22': Obtaining a running smoothness coefficient through the smoothed value of the vehicle state data, and representing the vehicle running state through the running smoothness coefficient.

[0034] According to the above technical means, by acquiring the vehicle status data during the driving process and using the exponentially weighted moving average method to obtain the smoothing value, the running smoothness of the vehicle is quantified by the running smoothness coefficient, which provides an objective quantitative indicator for driving behavior identification and helps to more accurately evaluate driving behavior. The introduction of the vehicle smoothness coefficient enables the driving evaluation model to take into account not only the driver's gaze behavior, but also the actual operating status of the vehicle, thereby improving the accuracy and comprehensiveness of the driving evaluation.

[0035] Furthermore, the driving evaluation coefficient calculation formula is as follows:

[0036] where β 1 represents the proportionality coefficient of the gaze density dispersion coefficient, FS represents the gaze density dispersion coefficient, β 2 represents the proportionality coefficient of the gaze concealment risk coefficient, ZS represents the gaze concealment risk coefficient, β 3 represents the proportionality coefficient of the running smoothness coefficient, and PW represents the running smoothness coefficient.

[0037] According to the above technical means, a driving evaluation coefficient is obtained by combining the proportional coefficient of the gaze density dispersion coefficient, the gaze hidden danger coefficient and the vehicle stability coefficient. The driving evaluation coefficient can comprehensively reflect the driver's driving behavior and vehicle status, thereby improving the accuracy of driving behavior recognition; the proportional coefficient can be adjusted according to specific circumstances, and the driving evaluation coefficient allows personalized driving evaluation for different drivers or different driving environments, thereby enhancing the adaptability and flexibility of the evaluation system.

[0038] Furthermore, the S4 specifically includes: matching a threshold of the driving evaluation coefficient according to a driving scenario, comparing the driving evaluation coefficient with the threshold, determining a comparison result, and if the threshold of the driving evaluation coefficient is greater than the driving evaluation coefficient, generating an abnormal signal to remind the driver to adjust his driving behavior, otherwise no abnormal signal is generated.

[0039] Based on the above-mentioned technical means, timely abnormal signal reminders can help reduce traffic accidents caused by driver inattention, fatigue or other unsafe behaviors, and improve road safety; by real-time monitoring of the driving evaluation coefficient and comparing it with the threshold, the system can continuously evaluate the driving safety status and provide real-time safety protection for the driver.

[0040] Beneficial effects of the present invention:

[0041] The driver's gaze data and the vehicle's operating status data are collected, and the characteristics of driving behavior are captured from multiple dimensions, thereby improving the comprehensiveness and reliability of the analysis; the driver's driving behavior is identified from the driver's behavior and the vehicle's status, so that the driving behavior can be identified more accurately, thereby effectively improving the accuracy of driving behavior identification; a driving evaluation model is constructed through the driver information and vehicle status data in historical driving scenarios, and the evaluation criteria can be dynamically adjusted according to different driving scenarios during the evaluation process, thereby improving the adaptability and accuracy of the evaluation; the threshold of the driving evaluation coefficient can be matched according to the driving scenario, and the driving evaluation coefficient can be compared with the threshold of the driving evaluation coefficient matched with the same driving scenario, and the driver can be prompted with driving behavior according to the judgment result, which helps the driver to adjust the driving behavior in time and reduce the occurrence of traffic accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of a specific implementation process of the present invention;

[0043] Figure 2 It is a schematic diagram of the calculation process of the gaze density dispersion coefficient and the gaze concealment risk coefficient of the present invention;

[0044] Figure 3 The figure is a schematic diagram of the calculation flow of the vehicle stability coefficient of the present invention. DETAILED DESCRIPTION

[0045] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0046] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0047] like Figure 1 As shown, a driving behavior recognition method based on multimodal data includes the following steps:

[0048] S1: Collect the driver's gaze data and vehicle operation status data during driving;

[0049] S2: performing density estimation processing on the gaze data to obtain a gaze density dispersion coefficient and a gaze concealment risk coefficient, which are used to represent the driver's gaze information;

[0050] The running status data is weighted averaged and smoothed to obtain the vehicle stability coefficient, which is used to represent the vehicle status;

[0051] S3: Based on the driving evaluation model, the driver's driving behavior is comprehensively analyzed during driving by combining the gaze density dispersion coefficient, gaze concealment risk coefficient and vehicle stability coefficient to obtain the driving evaluation coefficient;

[0052] S4: Identify the driver's driving behavior based on the driving evaluation coefficient, match the threshold of the driving evaluation coefficient according to the driving scenario, compare the driving evaluation coefficient with the threshold, and determine the comparison result to provide the driver with driving behavior prompts.

[0053] The driver's gaze data and vehicle operating status data are collected, and the characteristics of driving behavior are captured from multiple dimensions at the same time, which improves the comprehensiveness and reliability of the analysis; the driver's driving behavior is identified from the driver's behavior and vehicle status, which can more accurately identify driving behavior, thereby improving the accuracy of driving behavior identification; a driving evaluation model is constructed through driver information and vehicle status data in historical driving scenarios, and the evaluation criteria can be dynamically adjusted according to different driving scenarios during the judgment process, thereby improving the adaptability and accuracy of the evaluation; the threshold of the driving evaluation coefficient can be matched according to the driving scenario, and the driving evaluation coefficient can be compared with the threshold of the driving evaluation coefficient matched in the same driving scenario, and the driver can be prompted with driving behavior according to the judgment result, which helps the driver to adjust his driving behavior in time and reduce the occurrence of traffic accidents.

[0054] In this embodiment, the collected driver's gaze data and vehicle operation status data are obtained through implementation collection.

[0055] Preferably, collecting the driver's gaze data during driving is specifically implemented by selecting a desktop eye tracker, installing the eye tracker on the dashboard or front windshield of the vehicle, ensuring that it is aimed at the driver's eyes, and capturing the driver's eye movements; calibrating the eye tracker, using multiple gaze points to cover the driver's entire field of view, including the front and side areas, to ensure that the system can accurately record data when the driver looks at different positions; ensuring time synchronization between eye movement data and other sensor data, so as to accurately analyze the relationship between eye movement changes and vehicle behavior, and combining eye movement data with driving behavior data to ensure consistency between different data.

[0056] Preferably, the specific implementation of collecting vehicle status data during vehicle operation includes, where the vehicle status data includes speed, acceleration, steering wheel angle, brake status, accelerator pedal position, vehicle position, etc., and real-time monitoring of abnormal conditions in the vehicle status data. Combined with the driver's eye movement data, it is possible to identify potential dangerous driving behaviors and promptly alert the driver.

[0057] like Figure 1 As shown, in this embodiment, it also includes S0: through different historical driving scenarios, the driver's gaze data and the vehicle's operating status data in the historical driving scenarios are obtained, and the gaze density dispersion coefficient, gaze hidden danger coefficient and vehicle stability coefficient in the historical driving scenarios are weighted to construct a driving evaluation model.

[0058] By utilizing the driver's gaze data and the vehicle's operating status data in different historical driving scenarios, a more comprehensive and accurate driving evaluation model can be constructed, which improves the generalization ability of the driving model and its ability to adapt to different driving environments; weighted processing of the gaze density dispersion coefficient, gaze hidden danger coefficient and vehicle stability coefficient in historical driving scenarios can more accurately reflect the impact of different factors on driving behavior, thereby improving the accuracy of driving evaluation, and the robustness of model recognition can be improved according to the historical driving scenarios in different historical scenarios, so that it can work stably under various actual driving conditions.

[0059] In this embodiment, S0 also includes: evaluating the driving evaluation models of different historical driving scenarios respectively to obtain the balanced F scores and driving evaluation coefficients in each historical driving scenario; and determining the threshold of the driving evaluation coefficient in the driving scenario through each balanced F score and each driving evaluation coefficient.

[0060] The balanced F score, also known as the F1 value, is obtained by the recall and precision of the driving evaluation model for different driving scenarios. By evaluating the driving evaluation model for different historical driving scenarios separately, the performance of the driving evaluation model can be optimized to ensure the best recognition effect of driver behavior in various driving scenarios; by determining the threshold of the driving evaluation coefficient in each driving scenario, the driving evaluation model can adapt to different driving environments and conditions, improving the adaptability and flexibility of the driving evaluation model.

[0061] In this embodiment, obtaining the F1 value and each driving evaluation coefficient under each historical driving scenario specifically includes, by performing multiple driving tests under each historical driving scenario, obtaining a driving evaluation coefficient and an F1 value consistent with the number of driving tests, and each F1 value corresponds to each driving evaluation coefficient. The driving evaluation coefficient and the F1 value consistent with the number of driving tests are obtained through multiple driving tests, and the F1 value and the driving evaluation coefficient are derived based on the historical driving behavior under the historical scenario, rather than based on theory or simulation, and can reliably evaluate the driving behavior, thereby improving the accuracy and reliability of the threshold matching; through multiple tests and data analysis, the deviations that may occur in the model under specific conditions can be identified and reduced, thereby improving the robustness of the model.

[0062] In this embodiment, determining the threshold of the driving evaluation coefficient in the driving scenario through each F1 value and each driving evaluation coefficient specifically includes plotting a curve of the F1 value and each driving evaluation coefficient in each historical driving scenario, obtaining the driving evaluation coefficient corresponding to the maximum F1 value in the curve, and using the driving evaluation coefficient as the threshold of the driving evaluation coefficient in the historical driving scenario.

[0063] By obtaining the F1 value and driving evaluation coefficient under each historical driving scenario and drawing a curve to determine the driving evaluation coefficient corresponding to the maximum F1 value as the threshold, the threshold of the driving evaluation coefficient under the driving scenario can be accurately determined, thereby improving the accuracy of driving behavior recognition; through precise thresholds and real-time driving behavior evaluation, driving safety can be significantly improved and drivers can be helped to avoid potential dangerous situations.

[0064] Preferably, different driving scenarios (such as cities, highways, rural roads, etc.) have different environmental characteristics, such as traffic flow, road conditions, weather conditions, etc. These factors will directly affect driving behavior and attention. Therefore, the threshold of the driving evaluation coefficient should be adjusted according to the specific situation.

[0065] like Figure 2 As shown, in this embodiment, the gaze density dispersion coefficient is obtained through the following sub-steps:

[0066] S21: acquiring the gaze point coordinate data of the driver at a first time interval during driving, and taking the gaze point data of all the drivers as a first set, wherein each element in the first set is the gaze point data of the driver;

[0067] S22: smoothing the influence range of each element in the first set by a kernel function, thereby obtaining the density distribution of each element in the first set in space;

[0068] S23: Obtaining total density estimation values ​​of all elements according to the density distribution, dividing the total density estimation values ​​according to the number of first time intervals in the first set, and then obtaining a mean value and a standard deviation of the total density estimation values;

[0069] S24: According to the mean value and the standard deviation, the coefficient of variation of the total density estimate is obtained, and the dispersion coefficient of the gaze density is calculated according to the coefficient of variation.

[0070] By obtaining the coordinate data of the driver's gaze point during driving and performing kernel function smoothing on it, the driver's gaze behavior can be analyzed more finely, thereby capturing the driver's attention distribution more accurately; by calculating the gaze density dispersion coefficient, the driver's gaze behavior can be quantified to provide objective data support for driving behavior analysis; by collecting and analyzing gaze data at different time intervals, the driver's gaze behavior can be monitored in real time, abnormal behavior can be discovered in time and responded to.

[0071] Preferably, the density estimation value of each fixation point relative to other fixation points in the first time interval is calculated as follows:

[0072] in represents the density estimation value of the i-th fixation point relative to other fixation points in the first time interval, K is the Gaussian kernel function, h is the bandwidth, I represents the number of fixation point coordinate data in the first time interval, and i represents the number of the calculated fixation point coordinate data;

[0073] Preferably, the calculation formula for obtaining the total density estimation value of the fixation points in the first time interval is as follows:

[0074] Among them, MD n is the total density estimate of fixations in different first time intervals during driving, and n is the number of the time interval during driving;

[0075] Preferably, the calculation formula for the coefficient of variation of the total density estimate at different time intervals during driving is:

[0076] Among them, BY md is the coefficient of variation of the total density estimate at different first time intervals during driving, MD std is the standard deviation of the total density estimate of fixations in different first time intervals, MD avg is the average of the total density estimates of fixations in different first time intervals;

[0077] Preferably, the gaze density dispersion coefficient is calculated using the following formula:

[0078] FS md =BY md ×ln(BY md +1), where FS md is the gaze density dispersion coefficient, BY md is the coefficient of variation of the total density estimate at different first time intervals during the driving process.

[0079] Preferably, the driving process is a specific time period, usually a longer time period, and the driving process can be divided into multiple time intervals. By monitoring the driving process, it is helpful to analyze the driver's concentration level. The driving process is set by professional staff.

[0080] like Figure 2 As shown, in this embodiment, the gaze concealment risk coefficient is obtained through the following sub-steps:

[0081] S25: determining the driver's region of interest, matching all the driver's gaze points in the region of interest as a second set in each first time interval, wherein each element in the second set is the driver's gaze point in the region of interest, performing kernel density estimation on each element in the second set to obtain a density estimation value, and then obtaining a function of the density estimation value changing over time;

[0082] S26: According to the variation function, a density estimation threshold is obtained, and a deviation function is obtained through the density estimation threshold, thereby obtaining a gaze concealment risk coefficient.

[0083] By determining the driver's region of interest (R), the driver's gaze behavior in a specific area can be analyzed more accurately, which is crucial for understanding the driver's attention allocation and predicting potential risk behaviors; by obtaining the density estimate value of the driver's region of interest over time through the kernel density estimation formula, the driver's gaze behavior can be dynamically monitored to promptly detect distracted or inattentive situations; by real-time monitoring of the driver's gaze hidden risk coefficient, timely reminders can be given when the driver is distracted or inattentive, thereby improving driving safety; by obtaining the deviation function and the gaze hidden risk coefficient, the system can evaluate the deviation between the driver's gaze behavior and safe driving requirements, issue warnings in a timely manner, and reduce traffic accidents caused by inattention.

[0084] Preferably, the driver's area of ​​interest is calculated as follows:

[0085] R = [X min , X max ]×[Y min , Y max ], where R is the driver’s area of ​​interest, X min is the minimum value of the horizontal coordinate of the driver’s gaze point, X max is the maximum value of the horizontal coordinate of the driver's gaze point, Y min is the minimum ordinate value of the driver’s gaze point, Y max is the maximum value of the vertical coordinate of the driver’s gaze point;

[0086] Preferably, the deviation function calculation formula is as follows:

[0087] Where P (t) represents the deviation function in the time period t, YZ represents the density estimation threshold in the driver's area of ​​interest, and f (t) represents a function of the change of the density estimation value in the driver's region of interest over time in a time period t, and N is the number of elements in the second data set;

[0088] Preferably, the calculation formula of the observation concealment risk coefficient is as follows:

[0089] Where ZS represents the risk factor of gaze concealment, t 1 ~t 2 It is represented as a time period during driving in which the density estimation value in the driver's area of ​​interest is less than a density estimation value threshold in the driver's area of ​​interest.

[0090] Preferably, if the coordinates of the driver's gaze points within the time interval are all within the region of interest R, the larger the density estimate within the region of interest; if the coordinates of the driver's gaze points within the time interval are not within the region of interest R, the density estimate within the region of interest is close to 0; if the driver's gaze points are distributed in the region of interest R and other regions within the time interval w, the density estimate within the region of interest will be between 0 and a higher value, depending on the proportion of the gaze point distribution; the more gaze points fall within the region of interest R, the larger the density estimate within the region of interest.

[0091] More preferably, the region of interest is an area that the driver should pay priority attention to during driving, such as the road ahead, rearview mirror, instrument panel, etc. The scope of the region of interest is adjusted based on the actual driving scenario, and the region of interest is set by professional staff.

[0092] like Figure 3 As shown, in this embodiment, the vehicle stability coefficient is obtained through the following sub-steps:

[0093] S21': Acquire vehicle status data during driving, and obtain a smoothed value of the vehicle status data by exponentially weighted moving average;

[0094] S22': Obtaining a running smoothness coefficient through a smoothed value of the vehicle state data, and representing the vehicle running state through the running smoothness coefficient.

[0095] By acquiring the vehicle status data during the driving process and using the exponentially weighted moving average method to obtain the smoothed value, the running smoothness of the vehicle is quantified by the running smoothness coefficient, which provides an objective quantitative indicator for driving behavior identification and helps to evaluate driving behavior more accurately. The introduction of the vehicle smoothness coefficient enables the driving evaluation model to take into account not only the driver's gaze behavior, but also the actual operating status of the vehicle, thereby improving the accuracy and comprehensiveness of the driving evaluation.

[0096] Preferably, during the driving of the vehicle, the second version of the on-board diagnostic system (OBD-II) interface is connected to the on-board computer system to obtain the vehicle status data during the driving process, and an exponentially weighted moving average is used to smooth the data and extract the trend of the vehicle status data, including preprocessing of the vehicle operation data. Since different sensors of the vehicle may collect data at different sampling frequencies, it is necessary to align the timestamps to ensure that different sensor data can be analyzed at the same time point, and time alignment is achieved through linear interpolation or resampling; rule-based detection of vehicle status data, for example, the vehicle speed cannot exceed a reasonable upper limit, and the steering wheel angle also has physical limitations.

[0097] Preferably, the smoothed value of the vehicle state data is expressed as:

[0098] PH t =α·YS t +(1-α)·PH t-1 , where PH t is the smoothed value of the vehicle status data at time t, YS t is the original value of the vehicle status data, α is the smoothing coefficient, PH t-1 is the smoothed value of the vehicle status data at time t-1;

[0099] Preferably, the calculation formula of the running stability coefficient is as follows:

[0100] Among them, PW yx is the running smoothness coefficient, M is the time length of the driving process, and K is the type of vehicle status data.

[0101] In this embodiment, the driving evaluation coefficient calculation formula is as follows:

[0102] where β 1 represents the proportionality coefficient of the gaze density dispersion coefficient, FS represents the gaze density dispersion coefficient, β 2 represents the proportional coefficient of the gaze concealment risk coefficient, ZS represents the gaze concealment risk coefficient, β 3 It represents the proportionality coefficient of the running smoothness coefficient, and PW represents the running smoothness coefficient.

[0103] The driving evaluation coefficient is obtained by combining the proportional coefficient of the gaze density dispersion coefficient, the gaze hidden danger coefficient and the vehicle stability coefficient. The driving evaluation coefficient can comprehensively reflect the driver's driving behavior and vehicle status, and improve the accuracy of driving behavior recognition. The proportional coefficient can be adjusted according to the specific situation. The driving evaluation coefficient allows personalized driving evaluation for different drivers or different driving environments, which enhances the adaptability and flexibility of the evaluation system.

[0104] Preferably, according to the driving evaluation model, an optimal weight distribution among the gaze density dispersion coefficient, the gaze concealment risk coefficient and the running smoothness coefficient is determined using an index-based reference point adaptive multi-objective evolutionary algorithm, including:

[0105] Initial weight allocation: randomly generate a set of initial weight allocations (weights of gaze density dispersion coefficient, gaze concealment risk coefficient, and running stability coefficient);

[0106] Calculate the driving evaluation coefficient under each set of weight combinations. The fitness function is measured based on the driving evaluation coefficient generated by this set of weights. The weight combinations with good performance will be given priority to enter the next generation.

[0107] Select parents: Use common selection strategies, such as roulette wheel selection or tournament selection, to select the parent weight combinations for the next generation. These strategies ensure that better performing weight combinations have a greater probability of being selected.

[0108] Mutation and crossover: Select two weight combinations from the parent generation and cross them to generate new offspring weights. Apply small mutations to the newly generated weight combinations to further explore the solution space.

[0109] Fitness evaluation: Compare the generated driving evaluation coefficient with the predetermined standard, that is, the actual driving behavior performance data or historical evaluation data, and evaluate the performance of the new weight combination through comparison. The fitness function is used to reflect that the closer the generated driving evaluation coefficient is to the actual driving performance, the higher the fitness;

[0110] Reference point update: For each new weight combination in a generation, the reference point in the new generation will replace the reference point of the previous generation and become the new optimization target. After the reference point is updated, the algorithm will perform mutation and crossover operations based on the new reference point to guide the evolution direction of the next generation.

[0111] Environmental selection: In each generation, the newly generated weight combinations and the current solution set are evaluated for fitness, and the combinations with poor performance will be eliminated. The weight combinations with better performance evaluated by the fitness function generated by the driving evaluation coefficient will be retained to the next generation as the basis for the next generation search;

[0112] Termination condition: The evolution process ends when a certain number of generations is reached or the quality of the solution is no longer significantly improved.

[0113] In this embodiment, S4 specifically includes: matching the threshold of the driving evaluation coefficient according to the driving scenario, comparing the driving evaluation coefficient with the threshold, determining the comparison result, and generating an abnormal signal to remind the driver to adjust his driving behavior if the threshold of the driving evaluation coefficient is greater than the driving evaluation coefficient, otherwise no abnormal signal is generated. Timely abnormal signal reminders help reduce traffic accidents caused by driver inattention, fatigue or other unsafe behaviors, and improve road safety; by real-time monitoring of the driving evaluation coefficient and comparing it with the threshold, the system can continuously evaluate the driving safety status and provide real-time safety protection for the driver.

[0114] In summary, a driving behavior recognition method based on multimodal data has the following specific steps:

[0115] S0: Through different historical driving scenarios, the driver's gaze data and vehicle operation status data in the historical driving scenarios are obtained, and the gaze density dispersion coefficient, gaze hidden danger coefficient and vehicle stability coefficient in the historical driving scenarios are weighted to construct a driving evaluation model.

[0116] According to multiple tests in different historical scenarios, the threshold of the driving coefficient in the historical scenario is obtained;

[0117] S1: Collect the driver's gaze data and vehicle operation status data during driving;

[0118] S2: performing density estimation processing on the gaze data to obtain a gaze density dispersion coefficient and a gaze concealment risk coefficient, which are used to represent the driver's gaze information;

[0119] Perform weighted averaging and smoothing on the running status data to obtain the vehicle stability coefficient. The vehicle stability coefficient is used to represent the vehicle status;

[0120] S3: Based on the driving evaluation model, the driver's driving behavior is comprehensively analyzed during driving by combining the gaze density dispersion coefficient, gaze concealment risk coefficient and vehicle stability coefficient to obtain the driving evaluation coefficient;

[0121] S4: Identify the driver's driving behavior based on the driving evaluation coefficient, match the threshold of the driving evaluation coefficient according to the driving scenario, compare the driving evaluation coefficient with the threshold, and determine the comparison result to give the driver a driving behavior prompt.

[0122] The serial numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above are only preferred embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A driving behavior recognition method based on multimodal data, characterized in that: The following steps are involved: S1: Collect the driver's gaze data and vehicle operation status data during driving; S2: performing density estimation processing on the gaze data to obtain a gaze density dispersion coefficient and a gaze concealment risk coefficient, wherein the gaze density dispersion coefficient and the gaze concealment risk coefficient are used to represent the driver's gaze information; Performing weighted averaging and smoothing processing on the running status data to obtain a vehicle stability coefficient, wherein the vehicle stability coefficient is used to represent the vehicle status; S3: Based on the driving evaluation model, the driver's driving behavior is comprehensively analyzed during driving by combining the gaze density dispersion coefficient, gaze concealment risk coefficient and vehicle stability coefficient to obtain the driving evaluation coefficient; S4: Identify the driver's driving behavior based on the driving evaluation coefficient, match the threshold of the driving evaluation coefficient according to the driving scenario, compare the driving evaluation coefficient with the threshold, and determine the comparison result to provide driving behavior prompts to the driver.

2. The driving behavior recognition method based on multimodal data according to claim 1, characterized in that: Also includes: S0: Through different historical driving scenarios, the driver's gaze data and the vehicle's operating status data in the historical driving scenarios are obtained, and the gaze density dispersion coefficient, the gaze hidden danger coefficient and the vehicle stability coefficient in the historical driving scenarios are weighted to construct the driving evaluation model.

3. The driving behavior recognition method based on multimodal data according to claim 2 is characterized in that: The S0 also includes: evaluating the driving evaluation models of different historical driving scenarios respectively to obtain a balanced F score and a driving evaluation coefficient under each historical driving scenario; and determining a threshold of the driving evaluation coefficient under the driving scenario through each balanced F score and each driving evaluation coefficient.

4. The driving behavior recognition method based on multimodal data according to claim 3 is characterized in that: The obtaining of the balanced F-score and the driving evaluation coefficients under each historical driving scenario specifically includes performing a plurality of driving tests through each historical driving scenario to obtain the driving evaluation coefficient and the balanced F-score consistent with the number of driving tests, wherein each balanced F-score corresponds to each driving evaluation coefficient one by one.

5. The driving behavior recognition method based on multimodal data according to claim 4 is characterized in that: Determining the threshold of the driving evaluation coefficient in the driving scenario through each of the balanced F scores and each of the driving evaluation coefficients specifically includes plotting a curve of the balanced F scores and each of the driving evaluation coefficients in each historical driving scenario, obtaining the driving evaluation coefficient corresponding to the maximum balanced F score in the curve, and using the driving evaluation coefficient as the threshold of the driving evaluation coefficient in the historical driving scenario.

6. The driving behavior recognition method based on multimodal data according to claim 1, characterized in that: The gaze density dispersion coefficient is obtained by the following sub-steps: S21: acquiring the gaze point coordinate data of the driver at a first time interval during driving, and taking all the gaze point data of the driver as a first set, wherein each element in the first set is the gaze point data of the driver; S22: smoothing the influence range of each element in the first set by using a kernel function, thereby obtaining the density distribution of each element in the first set in space; S23: obtaining a total density estimation value of all the elements according to the density distribution, dividing the total density estimation value according to the number of the first time intervals in the first set, and thereby obtaining a mean value and a standard deviation of the total density estimation value; S24: Obtain the coefficient of variation of the total density estimate according to the mean value and the standard deviation, and calculate the gaze density dispersion coefficient according to the coefficient of variation.

7. The driving behavior recognition method based on multimodal data according to claim 6, characterized in that: The gaze concealment risk coefficient is obtained by the following sub-steps: S25: determining the driver's region of interest, matching all the driver's gaze points in the region of interest as a second set in each of the first time intervals, wherein each element in the second set is the driver's gaze point in the region of interest, performing kernel density estimation on each element in the second set to obtain a density estimation value, and then obtaining a function of the density estimation value changing over time; S26: According to the variation function, a density estimation threshold is obtained, and a deviation function is obtained through the density estimation threshold to obtain the gaze concealment risk coefficient.

8. The driving behavior recognition method based on multimodal data according to claim 7, characterized in that: The vehicle stability coefficient is obtained by the following sub-steps: S21': Acquire vehicle status data during driving, and obtain a smoothed value of the vehicle status data by exponentially weighted moving average; S22': obtaining a running smoothness coefficient through a smoothing value of the vehicle state data, and representing the vehicle running state through the running smoothness coefficient.

9. The driving behavior recognition method based on multimodal data according to claim 8, characterized in that: The driving evaluation coefficient calculation formula is as follows: Wherein, β1 represents the proportional coefficient of the gaze density dispersion coefficient, FS represents the gaze density dispersion coefficient, β2 represents the proportional coefficient of the gaze concealment risk coefficient, ZS represents the gaze concealment risk coefficient, β3 represents the proportional coefficient of the operation smoothness coefficient, and PW represents the operation smoothness coefficient.

10. The driving behavior recognition method based on multimodal data according to claim 1, characterized in that: The S4 specifically includes: matching the threshold of the driving evaluation coefficient according to the driving scenario, comparing the driving evaluation coefficient with the threshold, determining the comparison result, and if the threshold of the driving evaluation coefficient is greater than the driving evaluation coefficient, generating an abnormal signal to remind the driver to adjust his driving behavior, otherwise no abnormal signal is generated.

Images