Bus driving behavior early warning evaluation method and system based on multi-modal data

Abstract

The invention discloses a bus driving behavior early warning evaluation method and system based on multi-modal data, and relates to the technical field of driving behavior safety analysis. By introducing and processing the bus operation environment data, it can be accurately identified that a vehicle is in a specific scene such as entering, stopping or leaving a station; on the basis, the safety early warning evaluation model dynamically loads a special scoring rule corresponding to the scene; according to the method, quantitative evaluation of specific operation of the bus is realized, misjudgment caused by lack of scene perception in a conventional method is fundamentally solved, and the pertinence and accuracy of early warning are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of driving behavior safety analysis technology, specifically to a method and system for early warning and evaluation of bus driving behavior based on multimodal data. Background Technology

[0002] In the field of road traffic safety management, driver behavior early warning and analysis systems have become an important technological tool. These systems are typically deployed in vehicles and collect data such as driver facial images, vehicle motion status, and external road environment. Using computer vision, sensor fusion, and pattern recognition algorithms, they monitor and issue real-time warnings for risky behaviors such as fatigued driving, distracted driving, rapid acceleration, rapid deceleration, and lane departure. These general methods are primarily designed and optimized based on the driving patterns of passenger cars or long-haul trucks, and their warning logic relies on preset fixed thresholds or general behavioral models.

[0003] However, when the aforementioned conventional early warning analysis methods are directly applied to urban bus driving scenarios, their effectiveness and applicability are significantly insufficient. As special operating vehicles undertaking public passenger transport tasks, buses exhibit high complexity and uniqueness in their driving process in terms of operating scenarios, work tasks, and vehicle status. This makes it difficult for general models to accurately assess the safety and standardization of their driving behavior. Existing technologies mainly have the following limitations: First, in terms of operational scenarios, bus driving involves frequent and standardized procedures for entering and exiting bus stops. This process involves precise control of the distance to the side of the road, accurate alignment of the bus stop position, safe linkage between the doors and the platform screen doors, and observation and avoidance of pedestrians and non-motorized vehicles during passenger boarding and alighting. Conventional warning systems usually treat lane departure as a risk, but cannot distinguish between a bus's reasonable lane-keeping action for entering the stop and a genuine unintentional deviation. At the same time, for high-risk violations unique to buses, such as opening doors before the bus has come to a complete stop or starting the bus before the doors have closed, general systems lack targeted monitoring rules and sensor data support, thus creating critical safety monitoring blind spots.

[0004] Secondly, regarding the in-vehicle environment and driver tasks, bus drivers need to focus on driving while simultaneously monitoring for unusual events inside the vehicle, including disputes between passengers, sudden illnesses or falls, and suspected dangerous items. Traditional driver-facing DMS (Driver Monitoring System) cameras have limited fields of view, while road-facing ADAS (Advanced Driver Assistance Systems) cameras completely ignore the in-vehicle area. Therefore, existing technologies lack the ability to effectively perceive public safety conditions inside the vehicle and cannot correlate unusual events with the driver's response responsibilities, resulting in serious deficiencies in assessing the driver's overall safety performance.

[0005] Third, regarding vehicle status, the passenger load of buses dynamically and drastically changes between empty and fully loaded, which fundamentally affects the vehicle's braking distance and handling stability. When fully loaded, the vehicle's mass increases significantly, and its inertia increases substantially, resulting in a longer braking distance with the same brake pedal force, and a more sluggish steering response. However, conventional warning systems generally use fixed acceleration thresholds to evaluate driving behavior; this inevitably leads to two types of misjudgments: when the vehicle is fully loaded, the system may misreport normal strong braking necessary for the driver to avoid an accident as emergency braking; while when the vehicle is empty, the system may miss some truly dangerous braking operations where the deceleration has not reached the fixed threshold.

[0006] In summary, due to the unique safety regulations governing operations such as entering and exiting stations and stopping at stops, the complex tasks of handling abnormal events within the passenger compartment, and the different impacts of empty and fully loaded states on braking distance and handling characteristics during bus driving, driver behavior early warning analysis methods designed based on general scenarios cannot accurately evaluate the safety and compliance of bus-specific operations, nor can they comprehensively reflect the driver's overall safety performance in complex environments. Therefore, a new early warning analysis method is needed that can integrate multi-dimensional information on bus operating characteristics, vehicle load status, and driving tasks to achieve a safe, accurate, and comprehensive evaluation of bus driver behavior. Summary of the Invention

[0007] To address the challenges of bus driving, which involves specific safety regulations for operations such as entering and exiting stations and stopping at bus stops, the complex task of handling abnormal events within the passenger compartment, and the different impacts of empty and fully loaded states on braking distance and handling characteristics, current driver behavior warning analysis methods based on general scenarios are unable to accurately evaluate the safety compliance of bus-specific operations or comprehensively reflect the driver's overall safety performance in complex environments. This invention provides a bus driving behavior warning evaluation method and system based on multimodal data. By introducing and processing bus operating environment data, it can accurately identify specific scenarios such as the vehicle entering, stopping, or exiting a station. Based on this, the safety warning evaluation model dynamically loads specific scoring rules corresponding to that scenario. For example, in the station entry scenario, the focus is on evaluating the smoothness and accuracy of the distance to the curb, rather than triggering general lane departure warnings. This allows for quantifiable evaluation of bus-specific operations (such as standardized stopping and safe door opening and closing), fundamentally solving the misjudgment caused by the lack of scenario awareness in conventional methods and significantly improving the targeting and accuracy of warnings.

[0008] The above-mentioned technical objective of the present invention is achieved through the following technical solution: This invention provides a method for early warning and evaluation of bus driving behavior based on multimodal data, the method comprising: The system simultaneously collects first and second data related to the operation of the bus. The first data includes basic data reflecting the vehicle's status, road conditions, and driver status. The second data includes bus operating environment data and bus vehicle and load data. Feature extraction is performed on the first data and the second data to obtain a first feature vector and a second feature vector, and the first feature vector and the second feature vector are preprocessed. The preprocessed first and second feature vectors are input into the constructed safety warning evaluation model to obtain the driver's safe driving score; the scoring rules of the safety warning evaluation model change dynamically based on the changes in the second data. Safety warnings will be issued based on the safety driving score results.

[0009] A further optimized solution is that the first feature vector includes driver state features, bus operation state features, and road environment features; The driver's state characteristics include: driver fatigue characteristics, driver distraction characteristics, and driver emotional characteristics; The bus operating status characteristics include: bus longitudinal dynamics characteristics, bus lateral dynamics characteristics, and bus speed and acceleration characteristics; the road environment characteristics include lane-related characteristics and traffic sign characteristics; wherein, the bus longitudinal dynamics characteristics are characterized by the bus's acceleration; the bus lateral dynamics characteristics are characterized by the steering wheel angular velocity and lateral acceleration.

[0010] A further optimized solution is that the second feature vector includes operating environment features and vehicle and load features; The operating environment characteristics include road segment scenario type and abnormal events in the carriage; the vehicle and load characteristics include: load coefficient, door status, and dedicated lane compliance status. The road segment scenario type is a discrete categorical variable, and the categorical variables of the road segment scenario type include regular road segments, entry road segments, stop stations, and exit road segments.

[0011] A further optimized solution is that the preprocessing method includes: For all features in the first and second feature vectors, first perform missing value imputation, and then perform smoothing filtering. After normalizing the first and second feature vectors after smoothing filtering, they are concatenated to form the input feature vector.

[0012] A further optimized solution is that the method for scoring driving safety includes: The current scene is matched based on the second feature vector and preset scene variables; M risk indicator functions are preset, and risk indicators are calculated by combining the first feature vector and the second feature vector; the calling of some risk indicator functions is determined by the current scenario. A comprehensive risk score is obtained by dynamically aggregating the results of each risk indicator and the current scenario. During the dynamic aggregation calculation, the weights of related risk indicators are adjusted according to the current scenario.

[0013] A further optimization scheme is that the preset scenario variables include: Scenario C1: Regular road section, empty; Scenario C2: Regular road section, fully loaded; Scenario C3: The section leading to the station is empty; Scene C4: Entrance section, fully loaded; Scenario C5: Exit section, empty; Scene C6: Exit section, fully loaded; Scenario C7: Dock stop, empty; Scene C8: Stop, fully loaded; The no-load and full-load conditions are determined by the load factor. A first load factor threshold range and a second load factor threshold range are preset, wherein the lower limit of the first load factor threshold range is greater than the upper limit of the second load factor threshold range. When the current load factor is within the first load factor threshold range, the bus is determined to be fully loaded; when the current load factor is within the second load factor threshold range, the bus is determined to be empty.

[0014] A further optimized approach is that the calculation method for the comprehensive risk score includes: Obtain the hierarchical structured vector of the current scene; the hierarchical structured vector includes basic state encoding, joint scene encoding, and continuous features; The weights w of each risk indicator are dynamically generated based on the hierarchical structured vector. m (t); m=1,2,...,M; A power function that depends on scenario variables is constructed. A generalized weighted power average is then calculated by combining the power function, the calculation results of each risk indicator, and the weights of each risk indicator to obtain a comprehensive risk score.

[0015] A further optimized solution is that the method for constructing the hierarchical structured vector includes: One-hot encoding is performed on the road segment type and load status to obtain the road segment one-hot vector and the load one-hot vector, respectively. A hierarchical structured vector is constructed based on the road segment unique heat vector, the load unique heat vector, the interaction between the road segment unique heat vector and the load unique heat vector, and the continuous load coefficient.

[0016] A further optimized solution is that the method for calculating the generalized weighted power average includes: The comprehensive risk score is calculated using the following formula: ; in, This represents the overall risk score; This represents the weight of the i-th risk indicator; The exponential function representing the dependency of the scenario variable; Indicates the current scenario; when When =0, =1; when When it is scene C1, scene C2, scene C3 or scene C5, >1; when When it is scene C8, scene C6 or scene C4, <1; r i (t) represents the value of the i-th risk indicator.

[0017] This solution also provides a bus driving behavior early warning and evaluation system based on multimodal data, used to implement the aforementioned bus driving behavior early warning and evaluation method based on multimodal data. The system includes: The data acquisition module is used to simultaneously collect first and second data from the bus driver; the first data includes basic data reflecting the vehicle status, road status, and driver status; the second data includes bus operating environment data and bus vehicle and load data. The preprocessing module is used to extract features from the first data and the second data to obtain a first feature vector and a second feature vector, and to preprocess the first feature vector and the second feature vector. The evaluation module is used to input the preprocessed first feature vector and second feature vector into the pre-constructed safety warning evaluation model to obtain the driver's safe driving score; the scoring rules of the safety warning evaluation model change dynamically based on the changes in the second data. The early warning module is used to issue safety warnings based on the safe driving score results.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention provides a method and system for early warning and evaluation of bus driving behavior based on multimodal data. By introducing and processing bus operating environment data, it can accurately identify specific scenarios such as the vehicle entering, stopping, or leaving a station. On this basis, the safety early warning evaluation model dynamically loads the specific scoring rules corresponding to the scenario (i.e., dynamically generates the weights of each risk indicator according to the current scenario status). Safety warnings are issued based on the safe driving score results, realizing the quantifiable evaluation of bus-specific states (such as entering and leaving stations, stopping at stations, and safely opening and closing doors). This solves the misjudgment caused by the lack of scenario awareness in conventional methods and improves the pertinence and accuracy of the early warning results.

[0019] 2. This invention provides a method and system for early warning and evaluation of bus driving behavior based on multimodal data. It analyzes abnormal event data within the bus compartment as secondary data. By extracting features from data such as bus video, it can detect safety events such as passenger conflicts, abnormal gatherings, and falls. The safety early warning evaluation model can assess the driver's attention to and response to events within the vehicle in a timely manner, achieving a comprehensive evaluation of people, vehicle, road, and task, thus enhancing the comprehensiveness and depth of safety management. Furthermore, this solution dynamically adjusts the evaluation mechanism based on real-time load data and road conditions, avoiding false alarms for necessary braking when fully loaded and enhancing sensitivity to dangerous operations when unloaded, thereby improving the accuracy of the early warning results.

[0020] 3. This invention provides a method and system for early warning and evaluation of bus driving behavior based on multimodal data. The output safe driving score integrates quantitative indicators of multi-dimensional and multi-scenario information. It is not only used for real-time early warning, but also provides operation managers with a structured data analysis basis for driver operating habits, route risk points, and differences in safety performance at different times (such as peak load periods). Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This is a schematic diagram of the early warning and evaluation method for bus driving behavior based on multimodal data; Figure 2 This is a schematic diagram illustrating the principle of early warning and evaluation of bus driving behavior based on multimodal data. Figure 3 This is a schematic diagram of the structure of a bus driving behavior early warning and evaluation system based on multimodal data. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0023] Because bus driving involves unique safety regulations for operations such as entering and exiting stations and stopping at stations, complex tasks that need to take into account abnormal events inside the carriage, and special situations such as the different impacts of empty and full load states on braking distance and handling characteristics, driver behavior early warning analysis methods designed based on general scenarios cannot accurately evaluate the safety compliance of bus-specific operations, nor can they fully reflect the driver's comprehensive safety performance in complex environments. In view of this, this solution provides the following embodiments to solve the above-mentioned technical problems.

[0024] Example 1

[0025] This embodiment provides a method for early warning and evaluation of bus driving behavior based on multimodal data, such as... Figure 1 and Figure 2 As shown, the method includes: Step 1: Simultaneously collect first and second data on bus driving; the first data includes basic data reflecting vehicle status, road status, and driver status; the second data includes bus operating environment data and bus vehicle and load data. Specifically, the primary data includes: raw data on driver status, raw data on vehicle operating status, and raw data on road environment conditions; Step 2: Extract features from the first data and the second data to obtain a first feature vector and a second feature vector, and preprocess the first feature vector and the second feature vector. The first feature vector includes driver status features, bus operation status features, and road environment features. The first feature vector is similar to the feature construction method in the general model. For example, the bus operation status features can be directly obtained from the vehicle's CAN bus. The road environment features can be extracted based on onboard or road cameras, maps, and GPS data. Driver state characteristics are extracted from raw driver state data (such as driver-facing camera video streams); The driver's state characteristics include: driver fatigue characteristics, driver distraction characteristics, and driver emotional characteristics. Specifically, driver fatigue characteristics can be calculated based on the proportion of time the driver's eyelids are closed within a unit time window (e.g., 3 minutes), and a time proportion threshold is set to determine whether the driver is fatigued. Driver distraction characteristics can be calculated based on the angle between the driver's line of sight and the road ahead, and an angle threshold is set to determine whether the driver is distracted. Driver emotional characteristics can be determined based on the amplitude of movement of key points on the driver's face, and a movement amplitude threshold is set to determine the driver's emotional characteristics. The bus operating status characteristics include: bus longitudinal dynamics characteristics, bus lateral dynamics characteristics, and bus speed and acceleration characteristics; the road environment characteristics include lane-related characteristics and traffic sign characteristics; wherein, the bus longitudinal dynamics characteristics are characterized by the bus's acceleration; the bus lateral dynamics characteristics are characterized by the steering wheel angular velocity and lateral acceleration.

[0026] The second feature vector includes operating environment features and vehicle and load features; Specifically, the operating environment features described in this embodiment include road segment scene types and abnormal events in the passenger compartment; the vehicle and load features include: load coefficient, door status, and dedicated lane compliance status; specifically, abnormal events in the passenger compartment can be detected using the video stream from the camera on the top of the passenger compartment and a trained behavior recognition model (such as 3D CNN); the load coefficient can be estimated based on the number of passengers boarding and alighting recorded by the vehicle to determine the load status of the bus at different times, or calculated using the load data collected by the vehicle axle load sensors, with the load coefficient λ(t) ∈ [0,1]; specifically, this embodiment calculates the load coefficient λ(t) = (M) using the load data from the vehicle axle load sensors. c (t) -M e ) / (M max -M e ); where M c (t) represents the total mass of the vehicle at the current moment; M e Indicates the vehicle's unloaded mass; M max Indicates the maximum permissible gross weight of the vehicle; The road segment scenario type is a discrete categorical variable, and the categorical variables of the road segment scenario type include regular road segments, entry road segments, stop stations, and exit road segments.

[0027] The preprocessing method includes: For all features in the first and second feature vectors, missing values ​​are first imputed, and then smoothing filtering is performed; obvious outliers are removed. For short-term missing values, interpolation can be used; for long-term missing values, the time period is ignored or the default value is used. Since the data comes from different sensors, the timestamps may be different, so they also need to be aligned to a unified time axis.

[0028] After normalizing the first and second feature vectors after smoothing filtering, they are concatenated to form the input feature vector.

[0029] Step 3: Input the preprocessed first feature vector and second feature vector into the constructed safety warning evaluation model to obtain the driver's safe driving score; the scoring rules of the safety warning evaluation model change dynamically based on the changes in the second data. In step three, the method for scoring driving safety includes: S31, the current scene C(t) is matched based on the second feature vector and preset scene variables; in this embodiment, the preset scene variables include: Scenario C1: Regular road section, empty; Scenario C2: Regular road section, fully loaded; Scenario C3: The section leading to the station is empty; Scene C4: Entrance section, fully loaded; Scenario C5: Exit section, empty; Scene C6: Exit section, fully loaded; Scenario C7: Dock stop, empty; Scene C8: Stop, fully loaded; The no-load and full-load conditions are determined by the load factor. A first load factor threshold range and a second load factor threshold range are preset. The lower limit of the first load factor threshold range is greater than the upper limit of the second load factor threshold range. If the first load factor threshold range is set to [0.8, 1] and the second load factor threshold range is set to [0.1, 0.2], the bus is determined to be fully loaded when the current load factor is in the first load factor threshold range [0.8, 1], and the bus is determined to be empty when the current load factor is in the second load factor threshold range [0.1, 0.2].

[0030] Specifically, based on the obtained second feature vector and the aforementioned preset scene variables, the current scene can be easily matched, and the bus can be determined to be fully loaded or empty based on the magnitude of the load coefficient. The load coefficient is smaller when the bus is empty and larger when the bus is fully loaded.

[0031] S32, with M preset risk indicator functions, calculates risk indicators by combining the first feature vector and the second feature vector; the triggering of some risk indicator functions is determined by the current scenario; Specifically, common risk indicator functions include: speed violations, following distance being too close, lane departure, sudden acceleration, sudden braking, sharp turns, driver fatigue, and driver inattention. This plan also needs to set specific violation indicators for buses, such as failure to use turn signals when entering or leaving stations, opening doors before the bus has come to a complete stop at the platform, and starting the bus with the doors open. These indicators can be defined as binary events or continuous risk values, or for more complex indicators, risk functions can be defined, such as piecewise functions or distance threshold functions. The specific settings will depend on the actual application.

[0032] In this embodiment, some risk indicator functions mainly refer to specific violation indicators for buses. For example, risk indicator functions based on the risk indicators defined by the station entry regulations are only called and triggered in scenarios related to the station entry section. They are not called in other non-station entry section scenarios, so there is no need to calculate the indicator, thus reducing unnecessary computation in the system.

[0033] S33. A comprehensive risk score is obtained by dynamically aggregating the results of each risk indicator and the current scenario. During the dynamic aggregation calculation, the weights of the associated risk indicators are adjusted according to the current scenario.

[0034] Specifically, in step S33, the method for calculating the comprehensive risk score includes: G31, Obtain the hierarchical structured vector of the current scene; the hierarchical structured vector includes basic state encoding, joint scene encoding, and continuous features; this step specifically includes the following methods: G311, perform one-hot encoding on the road segment type and load status to obtain the road segment one-hot vector and load one-hot vector respectively; specifically, perform one-hot encoding on the road segment type {regular road segment, entrance road segment, stop station, exit road segment} to obtain the road segment one-hot vector r; perform one-hot encoding on the load status {empty, full load} to obtain the load one-hot vector l. G312, based on the segment unique heat vector, load unique heat vector, the interaction between the segment unique heat vector and the load unique heat vector, and continuous load coefficients, constructs a hierarchical structured vector c(t): ; Where ⊗ represents the Kronecker product of vectors; T represents the transpose; λ(t) represents the continuous load factor at time t, which is the average load factor over the time interval from t to t-1; In this system, the road segment hot vector and the load hot vector are used as the basic state encoding, and the Kronecker product of the road segment hot vector and the load hot vector is used as the joint scene encoding. The continuous load coefficient at time t is used as the continuous feature. The hierarchical structured vector construction method combines discrete classification information (scene) and continuous physical information (load). The Kronecker product creates features of the combined scene, enabling the subsequent weight allocation function to directly learn the essential differences between fully loaded and empty entry scenarios. The precise state at any given time can be determined by examining the activation status in the hierarchical structured vector. The location can be determined by the fact that the hierarchical structured vector retains the continuous load coefficient at time t, which allows the model to perceive subtle changes in load within the categories of full load or empty load, such as perceiving 70% or 95% passenger load. Because load and road segment types are discrete and finite, one-hot encoding can clearly distinguish each state and is computationally simple, making it suitable for real-time systems. At the same time, considering the combined effect of load and road segment, it is also necessary to introduce the interaction features between the one-hot vector of the road segment and the one-hot vector of the load, so as to capture the special risks of specific combinations such as full load entering the station.

[0035] G32, dynamically generates the weights of each risk indicator based on the hierarchical structured vector; Specifically, the hierarchical structured vector c(t) is mapped to risk awareness as follows: ; Where Wc represents the weight matrix, which determines how different contextual features affect the weights of each risk indicator; b c Let W(t) represent the bias vector, which is the basic weight preference; W(t) represents the output weight matrix; W(t) = [w1(t), w1(t), ..., w m (t),..., w M (t)];w m (t) represents the weight of the m-th risk indicator, m=1,2,...,M; G33 constructs a power function that depends on scenario variables, and then uses a generalized weighted power average to calculate the comprehensive risk score by combining the power function, the calculation results of each risk indicator, and the weights of each risk indicator.

[0036] The method for calculating the generalized weighted power average includes: The comprehensive risk score is calculated using the following formula: ; in, This represents the overall risk score; This represents the weight of the i-th risk indicator; The exponential function representing the dependency of the scenario variable; Indicates the current scenario; when When =0, =1; when When it is scene C1, scene C2, scene C3 or scene C5, >1; when When it is scene C8, scene C6 or scene C4, <1; r i (t) represents the value of the i-th risk indicator.

[0037] In this embodiment, when When = 1, the comprehensive risk score is a linear weighted sum, suitable for simple scenarios similar to scenario C1 (low-risk indicator); when When >1, it is more sensitive to high-risk indicators similar to those in scenarios C8, C6, or C4; when When <1, it is more sensitive to general risk indicators similar to those in scenarios C2, C3, or C5. This solution allows... As the scenario changes dynamically, the dynamic characteristics of the scenario are more deeply integrated into the risk aggregation process, so that the score not only reflects the instantaneous risk at the current moment, but also reflects the risk evolution characteristics under the scenario transformation and the synergistic effect between different risks.

[0038] Step 4: Issue a safety warning based on the safety driving score results.

[0039] Specifically, a first threshold can be set. and the second threshold The safe driving score results are used to issue tiered warnings. The first-level warning is triggered when: When this occurs, a strong audible and visual alarm is triggered, and details of the risk event (such as "sudden braking upon entering the station while fully loaded") are automatically reported to the fleet safety management center; a level two warning is triggered when... The system will prompt the driver via voice or dashboard icons; under normal circumstances, when... If no warning is triggered, the data will be retained for security analysis.

[0040] This solution introduces and processes bus operating environment data to accurately identify specific scenarios such as vehicle entry, stop, or exit from a station. Based on this, the safety warning evaluation model dynamically loads specific scoring rules corresponding to that scenario. Safety warnings are issued based on the safe driving score, enabling quantifiable evaluation of bus-specific states (such as entering / exiting stations, stopping at stations, and safe door opening / closing). This fundamentally solves the misjudgment caused by the lack of scenario awareness in conventional methods, significantly improving the targeting and accuracy of warnings. Abnormal event data within the passenger compartment is analyzed as secondary data. By extracting features from data such as passenger compartment video, safety events such as passenger conflicts, abnormal gatherings, and falls can be detected. The safety warning evaluation model can then assess whether the driver's attention to and response to in-vehicle events is timely and appropriate, achieving a comprehensive evaluation of people, vehicle, road, and task. This enhances the comprehensiveness and depth of safety management. The evaluation mechanism is dynamically adjusted based on real-time load data and road conditions, avoiding false alarms for necessary braking when fully loaded and strengthening sensitivity to dangerous operations when empty, thereby greatly improving the scientific rigor and fairness of warnings. The output safe driving score integrates quantitative indicators from multiple dimensions and scenarios. It is not only used for real-time early warning, but also provides operation managers with a structured data analysis basis for driver operating habits, route risk points, and differences in safety performance at different times (such as peak load periods).

[0041] Example 2 This embodiment provides a bus driving behavior early warning and evaluation system based on multimodal data, such as... Figure 3 As shown, the system used to implement the bus driving behavior early warning and evaluation method based on multimodal data described in Embodiment 1 includes: The data acquisition module is used to simultaneously collect first and second data from the bus driver; the first data includes basic data reflecting the vehicle status, road status, and driver status; the second data includes bus operating environment data and bus vehicle and load data. The preprocessing module is used to extract features from the first data and the second data to obtain a first feature vector and a second feature vector, and to preprocess the first feature vector and the second feature vector. The evaluation module is used to input the preprocessed first feature vector and second feature vector into the pre-constructed safety warning evaluation model to obtain the driver's safe driving score; the scoring rules of the safety warning evaluation model change dynamically based on the changes in the second data. The early warning module is used to issue safety warnings based on the safe driving score results.

[0042] Specifically, in this embodiment, the safety early warning evaluation model is a two-layer decision-making model consisting of a bottom layer and a top layer. The bottom layer consists of M parallel risk indicator calculators, and the top layer is a dynamic aggregation and decision engine. The system includes preset scenario variables and M risk indicator functions. Each risk indicator calculator calculates one risk indicator function, while some risk indicator calculators are triggered based on the current scenario. The dynamic aggregation and decision engine performs driving safety scoring based on changes in the second data.

[0043] Example 3 This embodiment provides a computer-readable medium storing a computer program. The computer program, when executed by a processor, can implement the bus driving behavior early warning and evaluation method based on multimodal data as described in Embodiment 1. Figure 1 As shown, the specific steps are as follows: Step 1: Simultaneously collect first and second data on bus driving; the first data includes basic data reflecting vehicle status, road status, and driver status; the second data includes bus operating environment data and bus vehicle and load data. Step 2: Extract features from the first data and the second data to obtain a first feature vector and a second feature vector, and preprocess the first feature vector and the second feature vector. Step 3: Input the preprocessed first feature vector and second feature vector into the constructed safety warning evaluation model to obtain the driver's safe driving score; the scoring rules of the safety warning evaluation model change dynamically based on the changes in the second data. Step 4: Issue a safety warning based on the safety driving score results.

[0044] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for early warning and evaluation of bus driving behavior based on multimodal data, characterized in that, The method includes: The system simultaneously collects first and second data related to the operation of the bus. The first data includes basic data reflecting the vehicle's status, road conditions, and driver status. The second data includes bus operating environment data and bus vehicle and load data. Feature extraction is performed on the first data and the second data to obtain a first feature vector and a second feature vector, and the first feature vector and the second feature vector are preprocessed. The preprocessed first and second feature vectors are input into the constructed safety warning evaluation model to obtain the driver's safe driving score; the scoring rules of the safety warning evaluation model change dynamically based on the changes in the second data. Safety warnings will be issued based on the safety driving score results.

2. The bus driving behavior early warning and evaluation method based on multimodal data according to claim 1, characterized in that, The first feature vector includes: driver state features, bus operation state features, and road environment features; The driver's state characteristics include: driver fatigue characteristics, driver distraction characteristics, and driver emotional characteristics; The bus operating status characteristics include: bus longitudinal dynamics characteristics, bus lateral dynamics characteristics, and bus speed and acceleration characteristics; the road environment characteristics include lane-related characteristics and traffic sign characteristics; wherein, the bus longitudinal dynamics characteristics are characterized by the bus's acceleration; the bus lateral dynamics characteristics are characterized by the steering wheel angular velocity and lateral acceleration.

3. The bus driving behavior early warning and evaluation method based on multimodal data according to claim 2, characterized in that, The second feature vector includes operating environment features and vehicle and load features; The operating environment characteristics include road segment scenario types and abnormal events in the carriage; The vehicle and load characteristics include: load factor, door status, and dedicated lane compliance status; The road segment scenario type is a discrete categorical variable, and the categorical variables of the road segment scenario type include: regular road segment, entrance road segment, stop station, and exit road segment.

4. The bus driving behavior early warning and evaluation method based on multimodal data according to claim 3, characterized in that, The preprocessing method includes: For all features in the first and second feature vectors, first perform missing value imputation, and then perform smoothing filtering. After normalizing the first and second feature vectors after smoothing filtering, they are concatenated to form the input feature vector.

5. The bus driving behavior early warning and evaluation method based on multimodal data according to claim 3, characterized in that, The methods for scoring driving safety include: The current scene is matched based on the second feature vector and preset scene variables; M risk indicator functions are preset, and risk indicators are calculated by combining the first feature vector and the second feature vector; the calling of some risk indicator functions is determined by the current scenario. A comprehensive risk score is obtained by dynamically aggregating the results of each risk indicator and the current scenario. During the dynamic aggregation calculation, the weights of related risk indicators are adjusted according to the current scenario.

6. The bus driving behavior early warning and evaluation method based on multimodal data according to claim 5, characterized in that, The preset scenario variables include: Scenario C1: Regular road section, empty; Scenario C2: Regular road section, fully loaded; Scenario C3: Entrance section, empty; Scene C4: Entrance section, fully loaded; Scenario C5: Exit section, empty; Scene C6: Exit section, fully loaded; Scenario C7: Dock stop, empty; Scene C8: Stop, fully loaded; The no-load and full-load conditions are determined by the load factor. A first load factor threshold range and a second load factor threshold range are preset, wherein the lower limit of the first load factor threshold range is greater than the upper limit of the second load factor threshold range. When the current load factor is within the first load factor threshold range, the bus is determined to be fully loaded; when the current load factor is within the second load factor threshold range, the bus is determined to be empty.

7. The bus driving behavior early warning and evaluation method based on multimodal data according to claim 5, characterized in that, The calculation method for the comprehensive risk score includes: Obtain the hierarchical structured vector of the current scene; the hierarchical structured vector includes basic state encoding, joint scene encoding, and continuous features; The weights w of each risk indicator are dynamically generated based on the hierarchical structured vector. m (t); m=1,2,...,M; A power function that depends on scenario variables is constructed. A generalized weighted power average is then calculated by combining the power function, the calculation results of each risk indicator, and the weights of each risk indicator to obtain a comprehensive risk score.

8. The bus driving behavior early warning and evaluation method based on multimodal data according to claim 7, characterized in that, The method for constructing the hierarchical structured vector includes: One-hot encoding is performed on the road segment type and load status to obtain the road segment one-hot vector and the load one-hot vector, respectively. A hierarchical structured vector is constructed based on the road segment unique heat vector, the load unique heat vector, the interaction between the road segment unique heat vector and the load unique heat vector, and the continuous load coefficient.

9. The bus driving behavior early warning and evaluation method based on multimodal data according to claim 7, characterized in that, The method for calculating the generalized weighted power average includes: The comprehensive risk score is calculated using the following formula: ; in, This represents the overall risk score; This represents the weight of the i-th risk indicator; The exponential function representing the dependency of the scenario variable; Indicates the current scenario; when When =0, it means that the current scene does not belong to the preset scene variables. =1; when When it is scene C1, scene C2, scene C3 or scene C5, >1; when When it is scene C8, scene C6 or scene C4, <1; r i (t) represents the value of the i-th risk indicator.

10. A bus driving behavior early warning and evaluation system based on multimodal data, characterized in that, The system is used to implement the bus driving behavior early warning and evaluation method based on multimodal data as described in any one of claims 1-8, the system comprising: The data acquisition module is used to simultaneously collect first and second data from the bus driver; the first data includes basic data reflecting the vehicle status, road status, and driver status; the second data includes bus operating environment data and bus vehicle and load data. The preprocessing module is used to extract features from the first data and the second data to obtain a first feature vector and a second feature vector, and to preprocess the first feature vector and the second feature vector. The evaluation module is used to input the preprocessed first feature vector and second feature vector into the pre-constructed safety warning evaluation model to obtain the driver's safe driving score; the scoring rules of the safety warning evaluation model change dynamically based on the changes in the second data. The early warning module is used to issue safety warnings based on the safe driving score results.

Images