Driver safety risk real-time evaluation system based on multi-source data fusion

Abstract

The application discloses a driver safety risk real-time evaluation system based on multi-source data fusion and relates to the technical field of safety risk evaluation.The driver safety risk real-time evaluation system based on multi-source data fusion comprises a vehicle evaluation module, a behavior evaluation module and a comprehensive evaluation module.The vehicle evaluation module is used for acquiring vehicle state data of a driver to be evaluated in real time and analyzing driving response stability characteristic values of the driver to be evaluated.The behavior evaluation module is used for acquiring vehicle driving image data and driving image data of the driver to be evaluated, combining a pre-trained safety evaluation model and analyzing behavior safety characteristic values of the driver to be evaluated.The comprehensive evaluation module is used for analyzing safety risk characteristic values based on the driving response stability characteristic values and the behavior safety characteristic values.The application evaluates the safety risk of a driver based on the safety risk characteristic values in a risk feedback module, so that the system can accurately reflect the risk level of the driver in different environments, provide corresponding safety warnings and further improve the safety of the driver.

Description

Technical Field

[0001] This invention relates to the field of safety risk assessment technology, specifically to a real-time driver safety risk assessment system based on multi-source data fusion. Background Technology

[0002] Multi-source data fusion refers to the effective integration of information from different sources and types of data using appropriate algorithms and models to arrive at a comprehensive evaluation result. In modern technology fields, especially in intelligent transportation, autonomous driving, and smart driving, multi-source data fusion is widely used. Its main purpose is to extract information from different data sources to improve the reliability and accuracy of data and overcome the defects and limitations that may exist in a single data source.

[0003] In modern intelligent transportation systems, driver behavior and vehicle status are crucial to traffic safety. With the development of autonomous driving technology and intelligent driving assistance systems, traditional driver behavior assessment methods have gradually revealed their limitations, such as the difficulty in fully reflecting the driver's true state based on single data analysis.

[0004] Existing technologies, such as the patent application with publication number CN119558531A, disclose a method, system, and program for assessing the driving safety of official vehicles. This method involves real-time collection of driving data during the use of official vehicles, data processing and feature extraction, followed by risk identification using a corresponding risk identification network model based on the extracted features. This identifies the corresponding risk category and probability, further determines the risk level, risk characteristics, and recommended measures, and finally summarizes all risk information to generate a safety assessment report. This approach enables efficient assessment of the driving safety of official vehicles, improves risk identification capabilities, provides data-driven decision support for safety management personnel, enhances driver safety awareness, and reduces the incidence of accidents involving official vehicles.

[0005] Based on the above findings, the limitations of existing technologies include at least the following issues: Existing technologies lack multi-dimensional safety risk assessment, making it difficult to comprehensively consider the combined impact of driver behavior and the real-time environment. Specifically, existing technologies primarily rely on vehicle status data, neglecting the driver's individual behavior and their adaptability in complex environments. These factors are crucial for driving safety, such as the impact of complex road conditions and unexpected events on driver decision-making. It is difficult to accurately assess the driver's emergency response capabilities under these environmental changes, thus affecting the system's effective improvement of driver safety. Consequently, it fails to provide drivers with accurate safety warnings, thereby increasing the potential risk of traffic accidents. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a real-time driver safety risk assessment system based on multi-source data fusion, which solves the problem that existing technologies lack multi-dimensional safety risk assessment and are difficult to accurately assess safety risks.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a real-time driver safety risk assessment system based on multi-source data fusion, comprising: a vehicle assessment module for acquiring vehicle status data of the driver to be assessed in real time and analyzing the driving response stability characteristic value of the driver to be assessed; a behavior assessment module for acquiring vehicle driving image data and driving image data of the driver to be assessed, and combining them with a pre-trained safety assessment model to analyze the behavioral safety characteristic value of the driver to be assessed; a comprehensive assessment module for analyzing the safety risk characteristic value of the driver to be assessed based on the driving response stability characteristic value and behavioral safety characteristic value of the driver to be assessed; and a risk feedback module for conducting a safety risk assessment of the driver based on the safety risk characteristic value.

[0008] Furthermore, the vehicle state data includes yaw rate, vehicle speed, steering wheel grip entropy, seat pressure gradient, and brake disc thermal radiation gradient. The specific steps for analyzing the driving response stability characteristics of the driver to be evaluated are as follows: Based on the vehicle state data of the driver to be evaluated, analyze the driving response characteristic set of the driver to be evaluated, including vehicle steady-state characteristic values ​​and driving control stability characteristic values; based on the driving response characteristic set of the driver to be evaluated, analyze the driving response stability characteristic values ​​of the driver to be evaluated.

[0009] Further, the specific steps for analyzing the driving response feature set of the driver to be evaluated are as follows: Based on the yaw rate, vehicle speed, and brake disc thermal radiation gradient of the driver to be evaluated, analyze the steady-state characteristic value of the vehicle condition of the driver to be evaluated; based on the steering wheel grip entropy and seat pressure gradient of the driver to be evaluated, analyze the driving control stability characteristic value of the driver to be evaluated.

[0010] Furthermore, the vehicle driving image data specifically refers to the pixel value and two-dimensional coordinates of each pixel in the vehicle driving image, and the driving image data specifically refers to the driving pixel value and driving two-dimensional coordinates of each driving pixel in the driving image.

[0011] Further, the specific steps for analyzing the behavioral safety feature values ​​of the driver to be evaluated are as follows: input the vehicle driving image data and driving image data of the driver to be evaluated into the pre-trained safety assessment model for comprehensive analysis to obtain the visual safety feature set of the driver to be evaluated, including driving visual disturbance feature values ​​and abnormal driving behavior feature values; based on the visual safety feature set of the driver to be evaluated, analyze the behavioral safety feature values ​​of the driver to be evaluated.

[0012] Furthermore, the safety assessment model includes a driving subnetwork and a driving subnetwork. The specific steps for analyzing the visual safety feature set of the driver to be assessed are as follows: In the driving subnetwork of the safety assessment model, the vehicle driving image data of the driver to be assessed is received, and the driving visual disturbance feature value of the driver to be assessed is analyzed; In the driving subnetwork of the safety assessment model, the driving image data of the driver to be assessed is received, and the abnormal driving behavior feature value of the driver to be assessed is analyzed.

[0013] Furthermore, the driving sub-network includes a preprocessing layer, a driving environment feature extraction layer, and a driving fusion output layer. The specific steps for analyzing the driving visual disturbance feature values ​​of the driver to be evaluated are as follows: In the preprocessing layer of the driving sub-network, the vehicle driving image data of the driver to be evaluated is received and preprocessed; in the driving environment feature extraction layer of the driving sub-network, the driving environment feature vector of the driver to be evaluated is extracted based on the preprocessed vehicle driving image data of the driver to be evaluated; in the driving fusion output layer of the driving sub-network, the driving visual disturbance feature values ​​of the driver to be evaluated are analyzed based on the driving environment feature vector of the driver to be evaluated.

[0014] Furthermore, the driving sub-network includes a driving preprocessing layer, a driving behavior feature extraction layer, and a driving fusion output layer. The specific steps for analyzing the abnormal driving behavior feature values ​​of the driver to be evaluated are as follows: In the driving preprocessing layer of the driving sub-network, the driving image data of the driver to be evaluated is received and preprocessed; in the driving behavior feature extraction layer of the driving sub-network, the driving behavior feature vector of the driver to be evaluated is extracted based on the preprocessed driving image data of the driver to be evaluated; in the driving fusion output layer of the driving sub-network, the abnormal driving behavior feature values ​​of the driver to be evaluated are analyzed based on the driving behavior feature vector of the driver to be evaluated.

[0015] Furthermore, the specific formula for calculating the safety risk characteristic value of the driver to be evaluated is as follows: ;in, The safety risk characteristic value of the driver to be evaluated. The driving response stability characteristic value of the driver to be evaluated. The response stability coefficient is stored in the database. The behavioral safety characteristic value of the driver to be evaluated. The safety factor for behaviors stored in the database. These are adjustment coefficients stored in the database. These are the interaction coefficients stored in the database.

[0016] Furthermore, the specific steps for assessing driver safety risks based on safety risk characteristic values ​​are as follows: The safety risk characteristic value of the driver to be assessed is compared with a preset safety risk characteristic value range; if the driver's safety risk characteristic value is lower than the lower limit of the preset safety risk characteristic value range, it is marked as a green risk level; if the driver's safety risk characteristic value is within the preset safety risk characteristic value range, it is marked as a yellow risk level; if the driver's safety risk characteristic value is higher than the upper limit of the preset safety risk characteristic value range, it is marked as a red risk level.

[0017] The present invention has the following beneficial effects:

[0018] (1) The real-time driver safety risk assessment system based on multi-source data fusion can accurately assess the driver's safety risks in complex driving environments by comprehensively analyzing the multi-source data of the driver during the driving process. The system not only acquires and analyzes various status data of the vehicle in real time, but also combines a pre-trained safety assessment model to analyze the driver's behavioral safety characteristics and generate safety risk feature values. This enables the system to accurately reflect the driver's risk level in different environments. Especially in complex road conditions, the system can promptly identify and assess the risks that the driver may face and provide corresponding safety warnings, thereby effectively reducing the probability of accidents and significantly improving the driver's safety.

[0019] (2) The real-time driver safety risk assessment system based on multi-source data fusion combines vehicle driving image data and driving image data, and uses a pre-trained safety assessment model to extract driving visual disturbance feature values ​​and abnormal driving behavior feature values, thereby comprehensively assessing the driver's behavior and the interference of the surrounding environment in actual driving, thereby improving the system's sensitivity to driver safety risks and providing timely safety risk warnings to ensure that the driver always maintains a suitable driving state, thereby reducing the probability of traffic accidents and improving the driver's driving experience.

[0020] (3) The real-time driver safety risk assessment system based on multi-source data fusion divides the safety assessment model into a driving sub-network and a driving sub-network, and receives corresponding image data respectively, so as to conduct more accurate assessment of different images. The driving sub-network is mainly responsible for analyzing the image data of the vehicle interacting with the external environment during driving to assess the driver's reaction ability when facing complex road conditions; while the driving sub-network focuses on the driver's behavior to assess the driver's behavior stability, thereby effectively reducing the safety risks caused by improper driver behavior or environmental complexity, enhancing the overall safety protection capability, and thus ensuring the safety of the driver.

[0021] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0022] Figure 1 This is a block diagram of the real-time driver safety risk assessment system based on multi-source data fusion according to the present invention.

[0023] Figure 2 This is a schematic diagram of the driving response feature set and visual safety feature set data of the driver to be evaluated in the real-time driver safety risk assessment system based on multi-source data fusion of the present invention.

[0024] Figure 3 This is a flowchart illustrating the specific steps involved in analyzing the visual disturbance characteristics of a driver to be evaluated in the real-time driver safety risk assessment system based on multi-source data fusion, as described in this invention.

[0025] Figure 4 This is a flowchart illustrating the specific steps involved in analyzing abnormal driving behavior characteristics of the driver to be evaluated using the real-time driver safety risk assessment system based on multi-source data fusion, as described in this invention. Detailed Implementation

[0026] Please see Figure 1-2 This invention provides a technical solution: a real-time driver safety risk assessment system based on multi-source data fusion, comprising: a vehicle assessment module for acquiring vehicle status data of the driver to be assessed in real time and analyzing the driving response stability characteristic value of the driver to be assessed; a behavior assessment module for acquiring vehicle driving image data and driving image data of the driver to be assessed, and combining them with a pre-trained safety assessment model to analyze the behavioral safety characteristic value of the driver to be assessed; a comprehensive assessment module for analyzing the safety risk characteristic value of the driver to be assessed based on the driving response stability characteristic value and behavioral safety characteristic value of the driver to be assessed; and a risk feedback module for assessing the driver's safety risk based on the safety risk characteristic value and taking corresponding warning measures based on the assessment results.

[0027] The specific steps for assessing driver safety risks based on safety risk characteristic values ​​are as follows: The safety risk characteristic value of the driver to be assessed is compared with a preset safety risk characteristic value range. If the driver's safety risk characteristic value is lower than the lower limit of the preset range, it is marked as a green risk level, and a first warning measure is taken, i.e., a mild reminder, such as an audible alert, indicating that the driver's current driving behavior and vehicle condition are within a safe range, and good driving behavior should continue. If the driver's safety risk characteristic value is within the preset range, it is marked as a yellow risk level, and a second warning measure is taken, i.e., a moderate warning, such as an audible alert and a warning message on the screen, reminding the driver that they are currently in a moderate risk state and need to be more vigilant. If the driver's safety risk characteristic value is higher than the upper limit of the preset range, it is marked as a red risk level, and a third warning measure is taken, i.e., a strong warning, such as a vibration alert, audible alarm, or emergency warning text, informing the driver that there is a high safety risk.

[0028] The specific formula for calculating the safety risk characteristic value of the driver to be evaluated is as follows: ;in, The safety risk characteristic value of the driver to be evaluated. The driving response stability characteristic value of the driver to be evaluated. The response stability coefficient is stored in the database. The behavioral safety characteristic value of the driver to be evaluated. The safety factor for behaviors stored in the database. The adjustment coefficient is stored in the database; in this implementation example, it is set to 2.000. These are the interaction coefficients stored in the database.

[0029] It needs to be explained that the response stability coefficient is stored in the database. The steps are as follows: Read the driving response stability characteristic value and behavioral safety characteristic value of the driver to be evaluated, and analyze them separately, such as 1 / (1+driving response stability characteristic value) to obtain the driving response stability risk characteristic value and behavioral safety risk characteristic value. Then, perform a summation analysis to obtain the risk sum value. Finally, compare the driving response stability risk characteristic value with the risk sum value, and use the result as the response stability coefficient. ;

[0030] Behavioral security factor stored in the database The acquisition steps are as follows: Read the driving response stability characteristic value and behavioral safety characteristic value of the driver to be evaluated, and analyze them separately, such as 1 / (1+driving response stability characteristic value) to obtain the driving response stability risk characteristic value and behavioral safety risk characteristic value. Then, perform a summation analysis to obtain the risk sum value. Finally, compare the behavioral safety risk characteristic value with the risk sum value, and use the result as the behavioral safety coefficient. ;

[0031] Interaction coefficients stored in the database The acquisition steps are as follows: Read the driving response stability risk characteristic value and behavioral safety risk characteristic value of the driver to be evaluated, perform ratio processing, and use the result as the interaction coefficient. .

[0032] The following is a specific implementation example for calculating the safety risk characteristic values ​​of the driver to be evaluated. The available data includes the vehicle steady-state characteristic values, driving control stability characteristic values, driving visual disturbance characteristic values, and abnormal driving behavior characteristic values ​​of the driver to be evaluated, as shown in Table 1.

[0033] Table 1. Examples of driving response feature sets and visual safety feature sets for the drivers to be evaluated.

[0034]

[0035] Steady-state coefficients stored in the database Approximately 0.525;

[0036] Manipulation coefficients stored in the database Approximately 0.475;

[0037] Disturbance coefficients stored in the database Approximately 0.556;

[0038] Anomaly coefficients stored in the database Approximately 0.444;

[0039] Substituting the data from Table 1 and the aforementioned coefficients into the specific formulas for calculating the driving response stability characteristic value and behavioral safety characteristic value of the driver to be evaluated, we obtain:

[0040] The driving response stability characteristic value of the driver to be evaluated is calculated as follows: 1 / (1+exp(-(0.525×0.849+0.475×0.768)))≈0.693;

[0041] Calculate the behavioral safety characteristic value of the driver to be evaluated = 1 / (1+exp(-(0.556×0.374+0.444×0.7268)) -1 ))≈0.681;

[0042] And the response stability coefficient stored in the database Approximately 0.498;

[0043] Behavioral security factor stored in the database Approximately 0.502;

[0044] Interaction coefficients stored in the database Approximately 1.017;

[0045] Substituting the analyzed driving response stability characteristic value, behavioral safety characteristic value, and coefficients of the driver to be evaluated into the specific formula for calculating the safety risk characteristic value of the driver to be evaluated, we obtain:

[0046] The safety risk characteristic value of the driver to be evaluated is calculated as follows: ((1 / (1+0.693))^0.498+(1 / (1+0.681))^0.502)×(1.017 / (1+0.693×0.681))≈0.532.

[0047] Specifically, such as Figure 2 As shown, the vehicle state data includes yaw rate, vehicle speed (obtained through wheel speed sensors), steering wheel grip force entropy, seat pressure gradient, and brake disc thermal radiation gradient. The specific steps for analyzing the driving response stability characteristics of the driver to be evaluated are as follows: Based on the vehicle state data of the driver to be evaluated, analyze the driving response characteristic set of the driver to be evaluated, including vehicle steady-state characteristic values ​​and driving control stability characteristic values; based on the driving response characteristic set of the driver to be evaluated, analyze the driving response stability characteristic values ​​of the driver to be evaluated.

[0048] Among them, the yaw rate is the rate at which the vehicle rotates around the vertical axis, reflecting the vehicle's steering stability. It can be obtained by the IMU sensor to obtain the angular velocity value of the Z-axis and use it as the yaw rate value.

[0049] Steering wheel grip entropy is the dispersion of pressure distribution when the driver's hand grips the steering wheel. It can be obtained by acquiring the pressure value of each area of ​​the steering wheel through a capacitive sensor array (arranged in the grip area of ​​the steering wheel) (covering multiple areas of the steering wheel, such as the sides, top and bottom of the steering wheel), and statistically analyzing the probability values ​​of different pressure values, based on the Shannon entropy formula.

[0050] The seat pressure gradient value is the difference in pressure distribution in different areas of the driver's seat. It can be obtained by installing an array of pressure sensors inside the seat (covering the seat cushion, backrest and other important areas), obtaining the pressure value of each area, and processing the standard deviation to use the result as the seat pressure gradient value.

[0051] The brake disc thermal radiation gradient value is the temperature difference at different locations on the brake disc surface. It can be obtained by acquiring brake disc temperature values ​​within multiple radii using an infrared temperature sensor, performing difference processing on the brake disc temperature values ​​within adjacent radii to obtain several sets of temperature difference values ​​between adjacent radii, and then performing weighted processing to use the result as the brake disc thermal radiation gradient value.

[0052] The specific steps for calculating the driving response stability characteristic value of the driver to be evaluated are as follows: ;in, The driving response stability characteristic value of the driver to be evaluated. The steady-state characteristic value of the vehicle condition for the driver to be evaluated. These are the steady-state coefficients stored in the database. The driving stability characteristic value of the driver to be evaluated. These are the manipulation coefficients stored in the database. .

[0053] It needs to be explained that the steady-state coefficients stored in the database Control coefficient The acquisition steps are as follows: Read the steady-state characteristic values ​​of the vehicle condition and the stable characteristic values ​​of the driving control of the driver to be evaluated, and sum them to obtain a stable sum value. Ratio the steady-state characteristic values ​​of the vehicle condition and the stable characteristic values ​​of the driving control of the driver to be evaluated to the stable sum value, and use the results as the steady-state coefficients. Control coefficient .

[0054] The specific steps for analyzing the driving response feature set of the driver to be evaluated are as follows: Based on the yaw rate, vehicle speed, and brake disc thermal radiation gradient values ​​of the driver to be evaluated, the steady-state characteristic values ​​of the vehicle condition are analyzed. Specifically, the yaw rate, vehicle speed, and brake disc thermal radiation gradient values ​​of the driver to be evaluated are standardized, and the results of the standardization are weighted to obtain the steady-state characteristic values ​​of the vehicle condition, which are used to characterize the stability of the vehicle during driving. Based on the steering wheel grip entropy and seat pressure gradient values ​​of the driver to be evaluated, the driving control stability characteristic values ​​of the driver to be evaluated are analyzed. Specifically, the steering wheel grip entropy and seat pressure gradient values ​​of the driver to be evaluated are standardized, and the results of the standardization are weighted and then ratioed, i.e., 1 + / (1 + weighted result), to obtain the driving control stability characteristic values ​​of the driver to be evaluated, which are used to characterize the driver's control stability.

[0055] In this implementation plan, by comprehensively considering vehicle status data, the stability of the vehicle can be accurately assessed, thereby more comprehensively reflecting the driver's reaction ability in the process of controlling the vehicle. Secondly, the standardization and weighted processing methods enable data from different sources to be scaled uniformly, thereby effectively improving the comparability of the data and enabling a more accurate assessment of the driver's handling stability and vehicle steady state in complex driving environments. Finally, by calculating the steady state coefficient and handling coefficient, the dynamic interaction assessment between the vehicle and the driver is optimized, making the safety assessment more refined, thereby improving the early warning capability of driver behavior and reducing the risk of accidents.

[0056] Specifically, the vehicle driving image data is the pixel value and two-dimensional coordinates of each pixel in the vehicle driving image, and the driving image data is the driving pixel value and driving two-dimensional coordinates of each driving pixel in the driving image.

[0057] The specific steps for analyzing the behavioral safety feature values ​​of the driver to be evaluated are as follows: input the vehicle driving image data and driving image data of the driver to be evaluated into the pre-trained safety assessment model for comprehensive analysis to obtain the visual safety feature set of the driver to be evaluated, including driving visual disturbance feature values ​​and abnormal driving behavior feature values; based on the visual safety feature set of the driver to be evaluated, analyze the behavioral safety feature values ​​of the driver to be evaluated.

[0058] The specific formula for calculating the behavioral safety characteristic value of the driver to be evaluated is as follows: ;in, The behavioral safety characteristic value of the driver to be evaluated. The driving visual disturbance characteristic value of the driver to be evaluated. These are the perturbation coefficients stored in the database. The abnormal driving behavior characteristic value of the driver to be evaluated. The anomaly coefficients stored in the database, .

[0059] It needs to be explained that the perturbation coefficients stored in the database Anomaly coefficient The acquisition steps are as follows: Read the driving visual disturbance feature values ​​and abnormal driving behavior feature values ​​of the driver to be evaluated, and sum them to obtain the behavioral safety sum value. Ratio the driving visual disturbance feature values ​​and abnormal driving behavior feature values ​​of the driver to be evaluated to the behavioral safety sum value, and use the results as the disturbance coefficient. Anomaly coefficient .

[0060] The safety assessment model includes a driving subnetwork and a driving subnetwork. The specific steps for analyzing the visual safety feature set of the driver to be assessed are as follows: In the driving subnetwork of the safety assessment model, the vehicle driving image data of the driver to be assessed is received, and the driving visual disturbance feature value of the driver to be assessed is analyzed; In the driving subnetwork of the safety assessment model, the driving image data of the driver to be assessed is received, and the abnormal driving behavior feature value of the driver to be assessed is analyzed.

[0061] The pre-training steps for the security assessment model are as follows:

[0062] Construct a driver safety assessment dataset. This dataset should include labeled driving image data and driver behavior image data, and accurately label each behavior (such as head turning, posture adjustment, visual occlusion, etc.) to ensure that the dataset covers different road environments (such as complex road conditions, visual occlusion, etc.). The image data should be labeled with corresponding pixel-level labels (such as head tilt, posture deviation, road continuity, etc.) by manual annotation or automatic annotation tools to serve as supervision signals for subsequent model training.

[0063] Initialization of the security assessment model: In the model structure initialization stage, an appropriate convolutional neural network (CNN) architecture is used as the backbone of the security assessment model. For each layer of feature extraction, a reasonable number of output channels is set, and an appropriate loss function (such as cross-entropy loss, MSE loss, etc.) is selected according to the target task of the model.

[0064] Driving subnetwork training: During training, the preprocessing layer of the driving subnetwork performs image input resizing, pixel normalization, and noise removal. Then, the driving environment feature extraction layer extracts the driver's driving visual disturbance features, including features such as road continuity, visual occlusion interference, and visual focus complexity. Finally, the driving fusion output layer generates driving visual disturbance features through weighted mapping and nonlinear transformation, which are used to evaluate the driver's attentional distraction in complex environments.

[0065] Driver Sub-Network Training: In the preprocessing layer of the driver sub-network, the driver's behavior image data will be resized and normalized. After passing through the behavior feature extraction layer, features such as head sway feature value and sitting posture deviation feature value will be extracted. In the driver fusion output layer, these features will be weighted and input into the fully connected layer, where linear weighting and activation function processing (such as sigmoid) will be performed to finally generate abnormal driving behavior feature values, which are used to quantify the degree of abnormal behavior of the driver during the driving process.

[0066] Multi-task joint training: In each round of training, multiple features from the driving sub-network and the driving sub-network (such as driving visual disturbance feature values ​​and driving behavior abnormal feature values) will be concatenated into a joint feature vector and input into the subsequent fully connected layer. Through multi-task joint regression, the system will perform feature mapping and output the driving visual disturbance feature values ​​and driving behavior abnormal feature values ​​for each time period.

[0067] Loss function and error optimization: A weighted combined loss function is adopted, which combines various features of the driving subnetwork and the driving subnetwork. Error optimization is performed using losses such as MSE, cross-entropy, and IoU to improve the performance of the model.

[0068] After each training round, the model is evaluated using a validation set. For various features of driving and driving, the model's prediction accuracy is calculated (such as the error between visual disturbance feature values ​​and true values, the difference between abnormal driving behavior feature values ​​and labeled values, etc.). Based on the validation results, the learning rate and optimizer parameters are dynamically adjusted, and an early stopping mechanism is used during training to prevent overfitting.

[0069] After multiple rounds of training and optimization, the trained model interface package is exported, enabling the model to be used in practical applications in subsequent real-time driver safety risk assessment systems, such as driver safety assessment.

[0070] In this implementation scheme, vehicle driving image data and driving image data are combined, and a pre-trained safety assessment model is used to analyze the driver's visual safety feature set. This enables a comprehensive assessment of the driver's behavior and surrounding environmental interference during driving. Secondly, the structural design of the driving sub-network and the driving sub-network allows the model to conduct in-depth analysis of the driving environment and driver behavior, respectively, thereby improving the assessment accuracy. Furthermore, the multi-task joint training method enhances the model's comprehensive analytical capabilities, enabling the system to make accurate assessments in complex environments. Finally, the system can issue timely safety warnings when the driver faces potential risks, thereby improving driver safety and significantly enhancing the driver's ability to prevent safety risks.

[0071] Specifically, such as Figure 3-4As shown, the driving sub-network includes a preprocessing layer, a driving environment feature extraction layer, and a driving fusion output layer. The specific steps for analyzing the driving visual disturbance feature values ​​of the driver to be evaluated are as follows: In the preprocessing layer of the driving sub-network, the vehicle driving image data of the driver to be evaluated is received and preprocessed. Specifically, the vehicle driving image data is uniformly scaled, such as adjusting the image to a set resolution (e.g., 256×256 pixels), and then image pixel normalization is performed to uniformly compress the RGB channel pixel values ​​to the range of [0, 1]. Brightness enhancement and contrast stretching are performed on the image to improve the separation of road structures, lane lines, obstacles, and other areas in the image. Edge enhancement filtering is performed on the preprocessed image to enhance the structural clarity of traffic signs, lane boundaries, and obstacle edges. In the driving loop of the driving sub-network... In the environmental feature extraction layer, the driving environment feature vector of the driver to be evaluated is extracted based on the preprocessed vehicle driving image data of the driver to be evaluated. In the driving fusion output layer of the driving sub-network, the driving visual disturbance feature value of the driver to be evaluated is analyzed based on the driving environment feature vector of the driver to be evaluated. Specifically, the road continuity feature, visual occlusion interference feature, and visual focus complexity feature in the driving environment feature vector are weighted and mapped and nonlinearly transformed (i.e., input into a set fully connected layer, and linear weight multiplication and addition processing and activation function processing, such as sigmoid, are performed) to generate driving visual disturbance feature value, which is used to evaluate the driver's attention to the surrounding environment. Especially in complex road conditions and high interference environments, the higher the feature value, the greater the visual interference and environmental complexity faced by the driver, which is more likely to cause the driver's attention to be distracted.

[0072] The specific steps for extracting the driving environment feature vector of the driver to be evaluated are as follows: Based on an image segmentation model (such as DeepLabV3), the pixel value and two-dimensional coordinates of each pixel in the vehicle driving image are divided to extract several road pixels in the vehicle driving image, and connected component processing (such as 8-neighborhood) is performed to obtain several road regions. The total number of road pixels and the total number of pixels are counted and the ratio is processed to obtain the road coverage value. The two-dimensional coordinates of each road pixel in each road region are averaged to obtain the centroid coordinates of each road region. The centroid coordinates of adjacent road regions are analyzed (i.e., calculated based on Euclidean distance) to obtain the distance values ​​of multiple adjacent road regions. The standard deviation is processed, and the results are weighted with the road coverage value to extract road coherence features, which are used to evaluate the road coherence faced by the driver during driving.

[0073] Based on object detection models (such as YOLOv5), foreground object recognition is performed on the pixel values ​​of each pixel in the preprocessed vehicle driving image to extract the bounding box coordinates (including the maximum and minimum coordinate values ​​in the horizontal direction and the maximum and minimum coordinate values ​​in the vertical direction) of several foreground object regions (such as vehicles, pedestrians, non-motorized vehicles, traffic facilities, etc.) and the corresponding pixels within the foreground object regions. The pixel area occupied by each foreground object region in the image (i.e., the total number of pixels in each foreground object region) and its center coordinates in the vertical direction of the image (i.e., the sum of the maximum and minimum coordinate values ​​in the vertical direction / 2) are calculated. The center coordinates in the vertical direction of the image are then compared with the vertical coordinates of the image's main view center (i.e., the image height). The deviation value of / 2) is used as a weighting factor for exponential mapping to obtain the weighted occlusion area of ​​each foreground target. The weighted occlusion areas of all foreground targets are accumulated to obtain the total occlusion area. Within the set main view area (such as the horizontal viewing range of ±15° from the center of the image), the total number of pixels occluded by the foreground target (i.e., the total number of pixels in each foreground target area within this viewing range) is counted and compared with the total number of pixels in the main view area (i.e., the total number of pixels within this viewing range). The ratio is then processed to obtain the main view occlusion rate. This rate is then standardized with the total occlusion area. Based on the standardized result, a weighted processing is performed to extract visual occlusion interference features, which are used to evaluate the degree to which the forward field of view is occluded by foreground targets (such as vehicles, pedestrians, non-motorized vehicles, etc.).

[0074] The gradient magnitude of each pixel in the vehicle driving image is calculated using an image gradient algorithm (such as the Sobel operator). This involves extracting the horizontal and vertical gradients of each pixel using the Sobel operator and performing calculations. The gradient value of each pixel is then calculated, and the average gradient value is determined. If the gradient value of each pixel exceeds a set threshold (e.g., 1.2 times the average gradient value), it is marked as a visually high-frequency interest pixel. After connected component processing (e.g., 8-neighborhood), several visually high-frequency interest regions (representing image areas that the driver may focus on) are obtained. The pixel value of each pixel within each visually high-frequency interest region is then processed to obtain its grayscale value, and the grayscale value of each pixel is calculated. The probability of pixel values ​​is calculated, and the structural entropy (measuring the complexity of image information in the region) of each high-frequency visual attention region is analyzed based on Shannon's formula. The gradient direction angle of each pixel in each high-frequency visual attention region is calculated, i.e., arctan (vertical gradient / horizontal gradient). The variance of the gradient direction angle of each high-frequency visual attention region is extracted and averaged with the structural entropy. The results are then weighted to extract complex visual focus features, which are used to evaluate the driver's attention to high-frequency visual areas during driving. Road continuity features, visual occlusion interference features, and complex visual focus features are combined in a preset order into a set of three-dimensional structured feature vectors to construct a driving environment feature vector describing the current image scene state.

[0075] The driving sub-network includes a driving preprocessing layer, a driving behavior feature extraction layer, and a driving fusion output layer. The specific steps for analyzing the abnormal driving behavior feature values ​​of the driver to be evaluated are as follows: In the driving preprocessing layer of the driving sub-network, the driving image data of the driver to be evaluated is received and preprocessed. Specifically, the driving image of the driver to be evaluated is uniformly resized, for example, the image size is standardized to 224×224 pixels. The RGB values ​​of each driving pixel in the image are normalized, mapping the pixel value range from [0, 255] to the [0, 1] interval. Noise in the image is removed using methods such as Gaussian blur or median filtering. The driver's face is improved through brightness enhancement, contrast adjustment, sharpening, and other processing methods. The system assesses the recognizability of the driver's position and driving environment. In the driving behavior feature extraction layer of the driving sub-network, the driving behavior feature vector of the driver to be evaluated is extracted based on the preprocessed driving image data of the driver to be evaluated. In the driving fusion output layer of the driving sub-network, the abnormal driving behavior feature values ​​of the driver to be evaluated are analyzed based on the driving behavior feature vector of the driver to be evaluated. Specifically, the head sway feature value and sitting posture deviation feature value in the driving behavior feature vector are weighted and nonlinearly transformed (i.e., input into a set fully connected layer, and linear weight multiplication and addition processing and activation function processing, such as sigmoid, are performed) to generate abnormal driving behavior feature values, which are used to characterize the degree of abnormal behavior of the driver during the driving process.

[0076] The specific steps for extracting the driving behavior feature vector of the driver to be evaluated are as follows: The driver's face in the driving image is detected using a face detection model (such as MTCNN or OpenFace), facial regions are extracted, and facial key points are located. These key points include, but are not limited to, the central two-dimensional coordinates of key regions such as the eyes, eyebrows, nose, and mouth (e.g., the average of the driving two-dimensional coordinates of each driving pixel within the eye region). Based on the central two-dimensional coordinates of the facial key points, a pose estimation model (such as 3D Head Pose) is used. Estimation is used to obtain the driver's head yaw angle (the rotation of the head on the horizontal plane, i.e., left and right turns) and the driver's upper body reference direction (i.e., based on the body pose estimation model, such as OpenPose, HRNet, etc., extracting upper body key points from the driving image, including: the center two-dimensional coordinates of the left and right shoulders, and performing coordinate difference processing, and performing angle transformation on the results based on the arctangent function to obtain the driver's upper body reference direction). The ratio of the driver's head yaw angle and the upper body reference direction is processed, i.e., |yaw angle - upper body reference direction| / upper body reference direction, to extract head yaw feature values, which are used to quantify the degree of the driver's head rotation, reflecting the amplitude of the driver's head rotation during driving, thereby assessing the driver's attention concentration.

[0077] Based on human pose estimation models (such as OpenPose or HRNet), key points of the upper body are extracted from the driver's image, such as the center 2D coordinates of the left and right shoulders, and the center 2D coordinates of the upper and lower spine. The coordinates of the shoulder center point and the midpoint of the spine are extracted (i.e., the mean of the center 2D coordinates of the left and right shoulders, and the mean of the center 2D coordinates of the upper and lower spine), and then averaged to obtain the coordinates of the upper body center point. Finally, the coordinates of the driver's seat center point are extracted (i.e., based on YOLOv5 or Faster). R-CNN detects objects inside the vehicle, identifies the driver's seat area, and extracts the driver's two-dimensional coordinates of the left, right, top, and bottom edges of the seat based on an edge detection algorithm. It then averages the lateral coordinates of the left and right edges and the lateral coordinates of the top and bottom edges to obtain the coordinates of the driver's seat center point. Furthermore, it performs difference processing (averaging) on ​​the lateral coordinates of the upper body center point and the driver's seat center point to extract posture deviation features. These features are used to assess the stability of the driver's posture. Excessive posture deviation may indicate poor posture, such as excessive forward or side leaning, which could lead to limited visibility or unstable control, affecting driving safety. Finally, the head tilt feature values ​​and posture deviation feature values ​​are combined in a preset order to form a two-dimensional structured feature vector, constructing a driving behavior feature vector.

[0078] In this implementation scheme, the vehicle driving image data and driving image data are deeply analyzed based on the driving subnetwork and the driving subnetwork, respectively, thereby effectively improving the accuracy of driver safety assessment. Secondly, the preprocessing layer in the driving subnetwork performs uniform size adjustment and normalization on the image data, thereby enhancing the image recognizability and providing high-quality data input for subsequent feature extraction. Through the driving environment feature extraction layer, the system can identify and extract key features, thereby assessing the driver's degree of attention distraction in complex environments, which helps to identify potential driving risks in a timely manner. Finally, the driving subnetwork extracts the driver's head tilt and posture deviation features to analyze the driver's attention concentration and postural stability, thereby accurately assessing the driver's abnormal behavior during driving and improving driving safety.

[0079] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0080] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A real-time driver safety risk assessment system based on multi-source data fusion, characterized in that, include: The vehicle evaluation module is used to acquire vehicle status data of the driver to be evaluated in real time and analyze the driving response stability characteristics of the driver to be evaluated. The behavior assessment module is used to acquire vehicle driving image data and driving image data of the driver to be assessed, and combine them with a pre-trained safety assessment model to analyze the behavioral safety feature values ​​of the driver to be assessed. The comprehensive evaluation module is used to analyze the safety risk characteristics of the driver to be evaluated based on the driving response stability characteristics and behavioral safety characteristics. The risk feedback module is used to assess the safety risks of drivers based on safety risk characteristic values. The specific formula for calculating the safety risk characteristic value of the driver to be evaluated is as follows: ； in, The safety risk characteristic value of the driver to be evaluated. The driving response stability characteristic value of the driver to be evaluated. The response stability coefficient is stored in the database. The behavioral safety characteristic value of the driver to be evaluated. The safety factor for behaviors stored in the database. These are adjustment coefficients stored in the database. These are the interaction coefficients stored in the database.

2. The real-time driver safety risk assessment system based on multi-source data fusion according to claim 1, characterized in that, The vehicle state data includes yaw rate, vehicle speed, steering wheel grip force entropy, seat pressure gradient, and brake disc thermal radiation gradient. The specific steps for analyzing the driving response stability characteristics of the driver to be evaluated are as follows: Based on the vehicle status data of the driver to be evaluated, the driving response feature set of the driver to be evaluated is analyzed, including vehicle steady-state feature value and driving control stability feature value. Based on the driving response feature set of the driver to be evaluated, the stable feature values ​​of the driving response of the driver to be evaluated are analyzed.

3. The real-time driver safety risk assessment system based on multi-source data fusion according to claim 2, characterized in that, The specific steps for analyzing the driving response feature set of the driver to be evaluated are as follows: Based on the yaw rate, vehicle speed, and brake disc thermal radiation gradient of the driver to be evaluated, the steady-state characteristics of the vehicle condition of the driver to be evaluated are analyzed. Based on the steering wheel grip force entropy value and seat pressure gradient value of the driver to be evaluated, the driving control stability characteristic value of the driver to be evaluated is analyzed.

4. The real-time driver safety risk assessment system based on multi-source data fusion according to claim 1, characterized in that, The vehicle driving image data specifically refers to the pixel value and two-dimensional coordinates of each pixel in the vehicle driving image, and the driving image data specifically refers to the driving pixel value and driving two-dimensional coordinates of each driving pixel in the driving image.

5. The real-time driver safety risk assessment system based on multi-source data fusion according to claim 4, characterized in that, The specific steps for analyzing the behavioral safety characteristics of the driver to be evaluated are as follows: The vehicle driving image data and driving image data of the driver to be evaluated are input into the pre-trained safety assessment model for comprehensive analysis to obtain the visual safety feature set of the driver to be evaluated, including driving visual disturbance feature values ​​and abnormal driving behavior feature values. Based on the visual safety feature set of the driver to be evaluated, the behavioral safety feature values ​​of the driver to be evaluated are analyzed.

6. The real-time driver safety risk assessment system based on multi-source data fusion according to claim 5, characterized in that, The safety assessment model includes a driving subnetwork and a driver subnetwork. The specific steps for analyzing the visual safety feature set of the driver to be assessed are as follows: In the driving sub-network of the safety assessment model, the vehicle driving image data of the driver to be evaluated is received, and the driving visual disturbance feature value of the driver to be evaluated is analyzed. In the driving sub-network of the safety assessment model, driving image data of the driver to be assessed is received, and abnormal driving behavior feature values ​​of the driver to be assessed are analyzed.

7. The real-time driver safety risk assessment system based on multi-source data fusion according to claim 6, characterized in that, The driving sub-network includes a preprocessing layer, a driving environment feature extraction layer, and a driving fusion output layer. The specific steps for analyzing the driving visual disturbance feature values ​​of the driver to be evaluated are as follows: In the preprocessing layer of the driving sub-network, the vehicle driving image data of the driver to be evaluated is received and preprocessed. In the driving environment feature extraction layer of the driving sub-network, the driving environment feature vector of the driver to be evaluated is extracted based on the preprocessed vehicle driving image data of the driver to be evaluated. In the driving fusion output layer of the driving subnetwork, the driving visual disturbance feature value of the driver to be evaluated is analyzed based on the driving environment feature vector of the driver to be evaluated.

8. The real-time driver safety risk assessment system based on multi-source data fusion according to claim 6, characterized in that, The driving sub-network includes a driving preprocessing layer, a driving behavior feature extraction layer, and a driving fusion output layer. The specific steps for analyzing the abnormal driving behavior feature values ​​of the driver to be evaluated are as follows: In the driving preprocessing layer of the driving subnetwork, driving image data of the driver to be evaluated is received and preprocessed. In the driving behavior feature extraction layer of the driving sub-network, the driving behavior feature vector of the driver to be evaluated is extracted based on the preprocessed driving image data of the driver to be evaluated. In the driving fusion output layer of the driving subnetwork, abnormal feature values ​​of the driving behavior of the driver to be evaluated are analyzed based on the driving behavior feature vector of the driver to be evaluated.

9. The real-time driver safety risk assessment system based on multi-source data fusion according to claim 1, characterized in that, The specific steps for conducting a driver safety risk assessment based on safety risk characteristic values ​​are as follows: The safety risk characteristic values ​​of the driver to be evaluated are compared with the preset safety risk characteristic value range; If the safety risk characteristic value of the driver to be evaluated is lower than the lower limit of the preset safety risk characteristic value range, it will be marked as a green risk level; If the safety risk characteristic value of the driver to be evaluated is within the preset safety risk characteristic value range, it will be marked as a yellow risk level; If the safety risk characteristic value of the driver to be evaluated is higher than the upper limit of the preset safety risk characteristic value range, it will be marked as a red risk level.

Images