Driver safety risk real-time evaluation system based on multi-source data fusion

Abstract

The invention discloses a driver safety risk real-time assessment system based on multi-source data fusion, and relates to the technical field of safety risk assessment. The driver safety risk real-time assessment system based on multi-source data fusion comprises a vehicle assessment module used for acquiring vehicle state data of a to-be-assessed driver in real time and analyzing a driving response stability characteristic value of the to-be-assessed driver; the behavior evaluation module is used for acquiring vehicle driving image data and driving image data of a to-be-evaluated driver, and analyzing a behavior safety characteristic value of the to-be-evaluated driver in combination with a pre-trained safety evaluation model; the comprehensive evaluation module is used for analyzing the safety risk characteristic value based on the driving response stability characteristic value and the behavior safety characteristic value, safety risk evaluation is conducted on the driver based on the safety risk characteristic value in the risk feedback module, and therefore the system can accurately reflect the risk level of the driver in different environments; and corresponding safety warning is provided, so that the safety of the driver is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of safety risk assessment, in particular to a driver safety risk real-time evaluation system based on multi-source data fusion. BACKGROUND

[0002] Multi-source data fusion refers to integrating data from different sources and different types, using appropriate algorithms and models for effective information fusion, and ultimately obtaining a comprehensive evaluation result. In the field of modern technology, especially in the fields of intelligent transportation, autonomous driving and intelligent driving, multi-source data fusion is widely used. Its main purpose is to extract information from different data sources to improve the reliability and accuracy of data and overcome the limitations and limitations of single data sources.

[0003] In modern intelligent transportation systems, the behavior of drivers and the state of vehicles are crucial to traffic safety. With the development of autonomous driving technology and intelligent driving assistance systems, traditional driver behavior evaluation methods have gradually shown limitations, such as difficulty in fully reflecting the true state of drivers based on single data analysis.

[0004] Prior art such as patent application CN119558531A discloses a method, system and program product for evaluating the driving safety of official vehicles. The method collects driving data in real time during the use of official vehicles, processes the data and extracts features, then uses a corresponding risk identification network model based on the extracted features to identify risks, determines the corresponding risk category and probability, further determines the risk level, risk characteristics and recommended measures, and finally outputs a safety evaluation report by summarizing all risk information. This method can efficiently evaluate the driving safety of official vehicles, improve risk identification capabilities, and provide data-driven decision support for safety managers to enhance driver safety awareness and reduce the incidence of official vehicle driving accidents.

[0005] Based on the above scheme, the limitations of the prior art include at least the following problems. The prior art lacks multi-dimensional safety risk assessment, making it difficult to fully consider the comprehensive impact of driver behavior and real-time environment. Specifically, the prior art mainly relies on vehicle state data, ignoring the individual behavior of drivers and their ability to adapt to complex environments. These factors are crucial to driving safety, such as complex road conditions and unexpected events, which affect driver decision-making. It is difficult to accurately assess the emergency response capabilities of drivers in these changing environments, thereby affecting the effective improvement of the system's safety for drivers. As a result, it fails to provide accurate safety warnings for drivers, thereby increasing the potential risk of traffic accidents. SUMMARY

[0006] In view of the deficiencies of the prior art, the present application provides a driver safety risk real-time evaluation system based on multi-source data fusion, which solves the problem of lack of multi-dimensional safety risk evaluation and difficulty in accurately evaluating safety risks in the prior art.

[0007] To achieve the above object, the present application is implemented by the following technical scheme: a driver safety risk real-time evaluation system based on multi-source data fusion, comprising: a vehicle evaluation module, configured to acquire vehicle state data of a driver to be evaluated in real time, and analyze driving response stability characteristic values of the driver to be evaluated; a behavior evaluation module, configured to acquire vehicle driving image data and driving image data of the driver to be evaluated, combine a pre-trained safety evaluation model, and analyze behavior safety characteristic values of the driver to be evaluated; a comprehensive evaluation module, configured to analyze safety risk characteristic values of the driver to be evaluated based on the driving response stability characteristic values and the behavior safety characteristic values of the driver to be evaluated; and a risk feedback module, configured to evaluate the safety risk of the driver based on the safety risk characteristic values.

[0008] Further, the vehicle state data comprises yaw angular velocity values, vehicle speed values, steering wheel grip entropy values, seat pressure gradient values, and brake disc thermal radiation gradient values, and the specific steps of analyzing the driving response stability characteristic values of the driver to be evaluated are as follows: based on the vehicle state data of the driver to be evaluated, analyzing a driving response feature set of the driver to be evaluated, including vehicle condition steady state characteristic values and driving control stability characteristic values; and based on the driving response feature set of the driver to be evaluated, analyzing the driving response stability characteristic values of the driver to be evaluated.

[0009] Further, the specific steps of analyzing the driving response feature set of the driver to be evaluated are as follows: based on the yaw angular velocity values, vehicle speed values, and brake disc thermal radiation gradient values of the driver to be evaluated, analyzing the vehicle condition steady state characteristic values of the driver to be evaluated; and based on the steering wheel grip entropy values and seat pressure gradient values of the driver to be evaluated, analyzing the driving control stability characteristic values of the driver to be evaluated.

[0010] Further, the vehicle driving image data specifically refers to pixel values and two-dimensional coordinates of each pixel point in the vehicle driving image, and the driving image data specifically refers to driving pixel values and driving two-dimensional coordinates of each driving pixel point in the driving image.

[0011] Further, the specific steps of analyzing the behavior safety characteristic values of the driver to be evaluated are as follows: inputting the vehicle driving image data and the driving image data of the driver to be evaluated into the pre-trained safety evaluation model for comprehensive analysis, obtaining a visual safety feature set of the driver to be evaluated, including driving visual disturbance characteristic values and driving behavior abnormality characteristic values; and based on the visual safety feature set of the driver to be evaluated, analyzing the behavior safety characteristic values of the driver to be evaluated.

[0012] Further, the safety evaluation model comprises a driving sub-network and a driving sub-network, and the specific steps of analyzing the visual safety feature set of the to-be-evaluated driver are as follows: in the driving sub-network of the safety evaluation model, vehicle driving image data of the to-be-evaluated driver is received, and driving visual disturbance feature values of the to-be-evaluated driver are analyzed; in the driving sub-network of the safety evaluation model, driving image data of the to-be-evaluated driver is received, and driving behavior abnormality feature values of the to-be-evaluated driver are analyzed.

[0013] Further, the driving sub-network comprises a preprocessing layer, a driving environment feature extraction layer, and a driving fusion output layer, and the specific steps of analyzing the driving visual disturbance feature values of the to-be-evaluated driver are as follows: in the preprocessing layer of the driving sub-network, vehicle driving image data of the to-be-evaluated driver is received and preprocessed; in the driving environment feature extraction layer of the driving sub-network, driving environment feature vectors of the to-be-evaluated driver are extracted based on the preprocessed vehicle driving image data of the to-be-evaluated driver; and in the driving fusion output layer of the driving sub-network, driving visual disturbance feature values of the to-be-evaluated driver are analyzed based on the driving environment feature vectors of the to-be-evaluated driver.

[0014] Further, the driving sub-network comprises a driving preprocessing layer, a driving behavior feature extraction layer, and a driving fusion output layer, and the specific steps of analyzing the driving behavior abnormality feature values of the to-be-evaluated driver are as follows: in the driving preprocessing layer of the driving sub-network, driving image data of the to-be-evaluated driver is received and preprocessed; in the driving behavior feature extraction layer of the driving sub-network, driving behavior feature vectors of the to-be-evaluated driver are extracted based on the preprocessed driving image data of the to-be-evaluated driver; and in the driving fusion output layer of the driving sub-network, driving behavior abnormality feature values of the to-be-evaluated driver are analyzed based on the driving behavior feature vectors of the to-be-evaluated driver.

[0015] Further, the specific formula for calculating the safety risk feature value of the to-be-evaluated driver is as follows: ; wherein, is the safety risk feature value of the to-be-evaluated driver, is the driving response stability feature value of the to-be-evaluated driver, is a response stability coefficient stored in the database, is the behavior safety feature value of the to-be-evaluated driver, is a behavior safety coefficient stored in the database, is an adjustment coefficient stored in the database, is an interaction coefficient stored in the database.

[0016] Further, the specific steps of the safety risk assessment of the driver based on the safety risk characteristic value are as follows: judging and analyzing the safety risk characteristic value of the driver to be evaluated and the preset safety risk characteristic value interval; if the safety risk characteristic value of the driver to be evaluated is lower than the lower limit of the preset safety risk characteristic value interval, marking as a green risk level; if the safety risk characteristic value of the driver to be evaluated is within the preset safety risk characteristic value interval, marking as a yellow risk level; and if the safety risk characteristic value of the driver to be evaluated is higher than the upper limit of the preset safety risk characteristic value interval, marking as a red risk level.

[0017] The present application has the following advantages: (1) The driver safety risk real-time evaluation system based on multi-source data fusion can accurately evaluate the safety risk of the driver in a complex driving environment by comprehensively analyzing the multi-source data of the driver during driving. The system not only obtains and analyzes the state data of the vehicle in real time, but also analyzes the behavior safety characteristics of the driver in combination with the pre-trained safety evaluation model, and generates a safety risk characteristic value, so that the system can accurately reflect the risk level of the driver in different environments, especially in complex road conditions. The system can timely identify and evaluate the risks that the driver may face, and provide corresponding safety warnings, thereby effectively reducing the probability of accidents and significantly improving the safety of the driver.

[0018] (2) The driver safety risk real-time evaluation system based on multi-source data fusion can comprehensively evaluate the behavior performance of the driver in actual driving and the interference of the surrounding environment by combining the vehicle driving image data and the driving image data, and extracting the driving visual disturbance characteristic value and the driving behavior abnormality characteristic value by using the pre-trained safety evaluation model, thereby improving the sensitivity of the system to the safety risk of the driver, and timely giving a safety risk prompt to ensure that the driver always maintains a suitable driving state, thereby reducing the probability of traffic accidents and improving the driving experience of the driver.

[0019] (3) The driver safety risk real-time evaluation system based on multi-source data fusion can more accurately evaluate different images by dividing the safety evaluation model into a driving sub-network and a driving sub-network, and receiving corresponding image data, respectively. The driving sub-network is mainly responsible for analyzing the image data of the vehicle driving process interacting with the external environment to evaluate the reaction ability of the driver when facing complex road conditions. The driving sub-network focuses on the behavior performance of the driver to evaluate the behavior stability of the driver, thereby effectively reducing the safety risk caused by improper behavior of the driver or complexity of the environment, enhancing the overall safety protection capability, and thereby protecting the safety of the driver.

[0020] Of course, implementing any product of the present application does not necessarily require all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 The block diagram of the driver safety risk real-time evaluation system based on multi-source data fusion of the present application.

[0022] Figure 2 The data schematic diagram of the driving response feature set and the visual safety feature set of the driver to be evaluated in the driver safety risk real-time evaluation system based on multi-source data fusion of the present application.

[0023] Figure 3 The specific step flow chart of analyzing the driving visual disturbance feature value of the driver to be evaluated in the driver safety risk real-time evaluation system based on multi-source data fusion of the present application.

[0024] Figure 4 The specific step flow chart of analyzing the driving behavior abnormal feature value of the driver to be evaluated in the driver safety risk real-time evaluation system based on multi-source data fusion of the present application. DETAILED DESCRIPTION

[0025] Please refer to Figures 1-2 The embodiment of the present application provides a technical scheme: a driver safety risk real-time evaluation system based on multi-source data fusion, comprising: a vehicle evaluation module, configured to acquire vehicle state data of a driver to be evaluated in real time, and analyze driving response stability feature values of the driver to be evaluated; a behavior evaluation module, configured to acquire vehicle driving image data and driving image data of the driver to be evaluated, combine a pre-trained safety evaluation model, and analyze behavior safety feature values of the driver to be evaluated; a comprehensive evaluation module, configured to analyze safety risk feature values of the driver to be evaluated based on the driving response stability feature values and the behavior safety feature values of the driver to be evaluated; and a risk feedback module, configured to evaluate the safety risk of the driver based on the safety risk feature values, and take corresponding warning measures based on the evaluation result.

[0026] The specific steps of the safety risk assessment of the driver based on the safety risk characteristic value are as follows: judging and analyzing the safety risk characteristic value of the driver to be evaluated and the preset safety risk characteristic value interval; if the safety risk characteristic value of the driver to be evaluated is lower than the lower limit of the preset safety risk characteristic value interval, marking it as a green risk level, and taking the first warning measure, that is, providing a slight reminder, such as a sound prompt, indicating that the current driving behavior and vehicle condition of the driver are within a safe range, and continuing to maintain good driving behavior; if the safety risk characteristic value of the driver to be evaluated is within the preset safety risk characteristic value interval, marking it as a yellow risk level, and taking the second warning measure, that is, providing a moderate warning, such as a sound prompt and warning information on the screen, reminding the driver that he is currently in a medium risk state and needs to be more alert; if the safety risk characteristic value of the driver to be evaluated is higher than the upper limit of the preset safety risk characteristic value interval, marking it as a red risk level, and taking the third warning measure, that is, providing a strong warning, such as a vibration prompt, a sound alarm, or an emergency warning text, to inform the driver that there is a high safety risk at present.

[0027] The specific formula for calculating the safety risk characteristic value of the driver to be evaluated is as follows: ; wherein, is the safety risk characteristic value of the driver to be evaluated, is the driving response stability characteristic value of the driver to be evaluated, is the response stability coefficient stored in the database, is the behavior safety characteristic value of the driver to be evaluated, is the behavior safety coefficient stored in the database, is the adjustment coefficient stored in the database, which is 2.000 in this embodiment, is the interaction coefficient stored in the database.

[0028] It needs to be explained that the response stability coefficient stored in the database is The acquisition steps are as follows: reading the driving response stability characteristic value and the behavior safety characteristic value of the driver to be evaluated, and analyzing them respectively, such as 1 / (1+driving response stability characteristic value), to obtain the driving response stability risk characteristic value and the behavior safety risk characteristic value, and performing summation analysis to obtain the risk sum value, and performing ratio processing on the driving response stability risk characteristic value and the risk sum value, and taking the result as the response stability coefficient ; The behavior safety coefficient stored in the database is The acquisition steps are as follows: the driving response stability characteristic value and the behavior safety characteristic value of the to-be-evaluated driver are read, analyzed respectively, and the driving response stability risk characteristic value and the behavior safety risk characteristic value are obtained, the risk sum value is obtained by summing analysis, the driving response stability risk characteristic value is processed by ratio, and the result is taken as the behavior safety coefficient ; The interaction coefficient stored in the database The acquisition steps are as follows: the driving response stability risk characteristic value and the behavior safety risk characteristic value of the to-be-evaluated driver are read, analyzed respectively, and the driving response stability risk characteristic value and the behavior safety risk characteristic value are obtained, the risk sum value is obtained by summing analysis, the driving response stability risk characteristic value is processed by ratio, and the result is taken as the behavior safety coefficient .

[0029] The specific implementation example of calculating the safety risk characteristic value of the to-be-evaluated driver is as follows: the existing data includes the vehicle condition steady state characteristic value, the driving control stability characteristic value, the driving visual disturbance characteristic value, and the driving behavior abnormality characteristic value of the to-be-evaluated driver, and the specific data is shown in Table 1: Table 1: Driving response characteristic set and visual safety characteristic set data of the to-be-evaluated driver The steady state coefficient stored in the database is about: 0.525; The control coefficient stored in the database is about: 0.475; The disturbance coefficient stored in the database is about: 0.556; The abnormality coefficient stored in the database is about: 0.444; The data in Table 1 and the above coefficients are respectively substituted into the specific formula for calculating the driving response stability characteristic value and the behavior safety characteristic value of the to-be-evaluated driver to obtain: The driving response stability characteristic value of the to-be-evaluated driver = 1 / (1+exp(- (0.525*0.849+0.475*0.768))) ≈ 0.693; The behavior safety characteristic value of the to-be-evaluated driver = 1 / (1+exp(- (0.556*0.374+0.444*0.7268) -1 )) ≈ 0.681; The response stability coefficient stored in the database is about: 0.498; The behavior safety coefficient stored in the database is about: 0.502; The interaction coefficient stored in the database about: 1.017; The driving response stability characteristic value, behavior safety characteristic value and coefficient of the driver to be evaluated are substituted into the specific formula for calculating the safety risk characteristic value of the driver to be evaluated to obtain: The safety risk characteristic value of the driver to be evaluated = ((1 / (1+0.693))^0.498 + (1 / (1+0.681))^0.502) x (1.017 / (1+0.693*0.681)) ≈ 0.532.

[0030] Specifically, as shown in Figure 2 The vehicle state data includes a yaw rate value, a vehicle speed value (obtained through a wheel speed sensor), a steering wheel grip entropy value, a seat pressure gradient value and a brake disc thermal radiation gradient value. The specific steps for analyzing the driving response stability characteristic value of the driver to be evaluated are as follows: based on the vehicle state data of the driver to be evaluated, the driving response characteristic set of the driver to be evaluated is analyzed, including a vehicle condition steady state characteristic value and a driving control stability characteristic value; and based on the driving response characteristic set of the driver to be evaluated, the driving response stability characteristic value of the driver to be evaluated is analyzed.

[0031] The yaw rate value is the rate of rotation of the vehicle around the vertical axis, reflecting the steering stability of the vehicle, which can be obtained by the angular velocity value of the Z axis of the IMU sensor and taken as the yaw rate value.

[0032] The steering wheel grip entropy value is the pressure distribution dispersion of the driver's hands when holding the steering wheel. The pressure value of each steering wheel region can be obtained by the capacitive sensor array (covering multiple regions of the steering wheel, such as both sides, top and bottom of the steering wheel) arranged in the holding area of the steering wheel, and the probability value of different pressure values is counted to obtain the steering wheel grip entropy value based on the Shannon entropy value formula.

[0033] The seat pressure gradient value is the pressure distribution difference of the driver on different regions of the seat. The pressure value of each region can be obtained by installing a pressure sensor array (covering the seat cushion, backrest and other important regions) inside the seat, and the standard deviation is processed, and the result is taken as the seat pressure gradient value.

[0034] The brake disc thermal radiation gradient value is the temperature difference of different positions on the surface of the brake disc. The temperature values of the brake disc within a plurality of radii can be obtained by an infrared temperature sensor, and the difference values of the brake disc temperature values within adjacent radii are processed to obtain a plurality of temperature difference values of adjacent radii, and the result is taken as the brake disc thermal radiation gradient value.

[0035] The specific steps for calculating the driving response stability characteristic value of the driver to be evaluated are as follows: ; wherein, a driving response stability characteristic value of the driver to be evaluated, a vehicle condition steady state characteristic value of the driver to be evaluated, a steady state coefficient stored in the database, a driving control stability characteristic value of the driver to be evaluated, a control coefficient stored in the database, .

[0036] It needs to be explained that the steady state coefficient stored in the database , the control coefficient The acquisition steps are as follows: reading the vehicle condition steady state characteristic value of the driver to be evaluated, the driving control stability characteristic value, and performing sum processing to obtain a stability sum value, and performing ratio processing of the vehicle condition steady state characteristic value of the driver to be evaluated and the driving control stability characteristic value with the stability sum value, and taking the result as the steady state coefficient , the control coefficient .

[0037] The specific steps of analyzing the driving response characteristic set of the driver to be evaluated are as follows: based on the yaw rate value, the vehicle speed value, and the brake disc thermal radiation gradient value of the driver to be evaluated, the vehicle condition steady state characteristic value of the driver to be evaluated is analyzed, which is specifically: the yaw rate value, the vehicle speed value, and the brake disc thermal radiation gradient value of the driver to be evaluated are standardized, and the standardized results are weighted to obtain the vehicle condition steady state characteristic value of the driver to be evaluated, which is used to represent the stability of the vehicle during driving; based on the steering wheel grip entropy value and the seat pressure gradient value of the driver to be evaluated, the driving control stability characteristic value of the driver to be evaluated is analyzed, which is specifically: the steering wheel grip entropy value and the seat pressure gradient value of the driver to be evaluated are standardized, the standardized results are weighted, and the ratio processing is performed, i.e. 1+ / (1+ weighted result), to obtain the driving control stability characteristic value of the driver to be evaluated, which is used to represent the control stability of the driver.

[0038] In this embodiment, the stability of the vehicle is accurately evaluated by comprehensively considering the vehicle state data, so that the reaction ability of the driver in controlling the vehicle can be more comprehensively reflected, secondly, the standardized and weighted processing makes the data of different sources uniform in scale, so that the comparability of the data is effectively improved, and the control stability and the vehicle condition steady state of the driver in complex driving environment can be more accurately evaluated, finally, through the calculation of the steady state coefficient and the control coefficient, the dynamic interaction evaluation between the vehicle and the driver is optimized, the safety evaluation is more accurate, and the warning ability of the driver behavior is improved, thereby reducing the risk of accidents.

[0039] Specifically, the vehicle driving image data is specifically pixel values and two-dimensional coordinates of each pixel point in the vehicle driving image, and the driving image data is specifically driving pixel values and driving two-dimensional coordinates of each driving pixel point in the driving image.

[0040] The specific steps of analyzing the behavior safety feature value of the to-be-evaluated driver are as follows: inputting the vehicle driving image data and the driving image data of the to-be-evaluated driver into the pre-trained safety evaluation model for comprehensive analysis to obtain a visual safety feature set of the to-be-evaluated driver, including a driving visual disturbance feature value and a driving behavior abnormality feature value; and analyzing the behavior safety feature value of the to-be-evaluated driver based on the visual safety feature set of the to-be-evaluated driver.

[0041] The specific formula for calculating the behavior safety feature value of the to-be-evaluated driver is as follows: ; wherein, is the behavior safety feature value of the to-be-evaluated driver, is the driving visual disturbance feature value of the to-be-evaluated driver, is a disturbance coefficient stored in the database, is the driving behavior abnormality feature value of the to-be-evaluated driver, is an abnormality coefficient stored in the database, .

[0042] It should be explained that the disturbance coefficient and the abnormality coefficient stored in the database are obtained as follows: reading the driving visual disturbance feature value and the driving behavior abnormality feature value of the to-be-evaluated driver, performing summation processing to obtain a behavior safety sum value, and performing ratio processing on the driving visual disturbance feature value and the driving behavior abnormality feature value of the to-be-evaluated driver and the behavior safety sum value respectively, and taking the results as the disturbance coefficient and the abnormality coefficient .

[0043] The safety evaluation model includes a driving sub-network and a driving sub-network, and the specific steps of analyzing the visual safety feature set of the to-be-evaluated driver are as follows: in the driving sub-network of the safety evaluation model, receiving the vehicle driving image data of the to-be-evaluated driver, and analyzing the driving visual disturbance feature value of the to-be-evaluated driver; in the driving sub-network of the safety evaluation model, receiving the driving image data of the to-be-evaluated driver, and analyzing the driving behavior abnormality feature value of the to-be-evaluated driver.

[0044] The pre-training steps of the safety evaluation model are as follows: A driving safety assessment dataset is constructed, which should include annotated driving image data and driver behavior image data, and each behavior (such as head rotation, posture adjustment, visual obstruction, etc.) is accurately annotated to ensure that the dataset covers different road environments (such as complex road conditions, visual obstruction, etc.), and the image data should be annotated by artificial annotation or automatic annotation tool to generate corresponding pixel-level labels (such as head yaw, posture deviation, road continuity, etc.) as the supervision signal for subsequent model training.

[0045] Initialization of safety assessment model: In the model structure initialization stage, a suitable convolutional neural network (CNN) architecture is used as the backbone of the safety assessment model. For each layer of feature extraction, a reasonable number of output channels is set, and according to the target task of the model, a suitable loss function (such as cross-entropy loss, MSE loss, etc.) is selected.

[0046] Driving sub-network training: In the training process, the preprocessing layer of the driving sub-network performs size adjustment, pixel normalization, and noise removal on the image input. Then, the driving environment feature extraction layer extracts the driver's driving visual disturbance feature values, including road continuity, visual obstruction interference, and visual focus complexity. Finally, the driving fusion output layer generates driving visual disturbance feature values through weighted mapping and nonlinear transformation, which are used to evaluate the degree of attention distraction of the driver in complex environments.

[0047] Driving sub-network training: In the preprocessing layer of the driving sub-network, the driver's behavior image data is processed by size adjustment and pixel normalization. The behavior feature extraction layer extracts head yaw feature values, posture deviation feature values, and other features. In the driving fusion output layer, these features are weighted and input to the fully connected layer, which performs linear weighting and activation function processing (such as sigmoid), and finally generates driving behavior abnormality feature values to quantify the degree of abnormal behavior of the driver during driving.

[0048] Multi-task joint training: In each round of training, multiple features (such as driving visual disturbance feature values and driving behavior abnormality feature values) from the driving sub-network and the driving sub-network are concatenated into a joint feature vector and input to the subsequent fully connected layer. Through multi-task joint regression, the system will perform feature mapping and output driving visual disturbance feature values and driving behavior abnormality feature values for each time period.

[0049] Loss function and error optimization: A weighted combination loss function is used to combine the features of the driving sub-network and the driving sub-network, and MSE, cross-entropy, and IoU losses are used for error optimization to improve the performance of the model.

[0050] After each round of training, the model is evaluated using the validation set, and the prediction accuracy of the model is calculated for various features of driving and driving, such as the error between the visual disturbance feature value and the true value, the difference between the driving behavior abnormal feature value and the labeled value, etc. Based on the validation results, the learning rate and optimizer parameters are dynamically adjusted, and the EarlyStopping mechanism is used in the training process to prevent overfitting.

[0051] After multiple rounds of training and optimization, the trained model interface package is derived, so that the model can be used for practical application in the subsequent driver safety risk real-time evaluation system, such as driver safety evaluation.

[0052] In this embodiment, the vehicle driving image data and driving image data are combined, and the pre-trained safety evaluation model is used to analyze the visual safety feature set of the driver, so that the behavior performance and surrounding environment interference of the driver during driving can be comprehensively evaluated. Secondly, the structure design of the driving sub-network and the driving sub-network enables the model to analyze the driving environment and driver behavior in depth, thereby improving the evaluation accuracy. The multi-task joint training mode enhances the comprehensive analysis ability of the model, so that the system can make accurate evaluation in complex environment. Finally, the system can issue a safety warning in time when the driver faces potential risks, thereby improving the safety of the driver, and significantly improving the ability of the driver to prevent safety risks.

[0053] Specifically, as Figures 3-4As shown, the driving sub-network includes a preprocessing layer, a driving environment feature extraction layer, and a driving fusion output layer. The specific steps of analyzing the driving visual disturbance feature value of the to-be-evaluated driver are as follows: in the preprocessing layer of the driving sub-network, the vehicle driving image data of the to-be-evaluated driver is received and preprocessed, which specifically includes: uniformly scaling the vehicle driving image data, for example, adjusting the image to a set resolution (such as 256x256 pixels), then performing image pixel normalization to uniformly compress the RGB channel pixel value to the range of [0, 1], and performing brightness enhancement and contrast stretching on the image to improve the separation degree of road structures, lane lines, obstacles, etc. in the image; performing edge enhancement filtering operation on the preprocessed image to enhance the structural clarity of traffic signs, lane boundaries and obstacle edges; in the driving environment feature extraction layer of the driving sub-network, the driving environment feature vector of the to-be-evaluated driver is extracted based on the preprocessed vehicle driving image data of the to-be-evaluated driver; in the driving fusion output layer of the driving sub-network, the driving visual disturbance feature value of the to-be-evaluated driver is analyzed based on the driving environment feature vector of the to-be-evaluated driver, which specifically includes: weighting and nonlinearly transforming the road continuity feature, visual occlusion interference feature, and visual focus complexity feature in the driving environment feature vector (i.e., inputting into a set fully connected layer, performing linear weight multiplication and addition processing, and activation function processing such as sigmoid), to generate a driving visual disturbance feature value for evaluating the driver's attention to the surrounding environment, especially in complex road conditions and high interference environments. The higher the feature value, the greater the visual interference and environmental complexity faced by the driver, which is likely to cause the driver's attention to be distracted.

[0054] The specific steps of extracting the driving environment feature vector of the to-be-evaluated driver are as follows: based on an image segmentation model (such as DeepLabV3), the pixel value and two-dimensional coordinates of each pixel point in the vehicle driving image are divided to extract a plurality of road pixel points in the vehicle driving image, and connected component processing (such as 8-neighborhood) is performed to obtain a plurality of road regions. The total number of road pixel points and the total number of pixel points are counted and processed by ratio to obtain a road coverage value. The two-dimensional coordinates of each road pixel point in each road region are processed by mean value to obtain the centroid coordinates of each road region. The centroid coordinates of adjacent road regions are analyzed (i.e., calculated based on the Euclidean distance) to obtain the distance values of a plurality of adjacent road regions, and the standard deviation is processed. The result is weighted with the road coverage value to extract the road continuity feature for evaluating the road continuity faced by the driver during driving. Based on the target detection model (such as YOLOv5), the pixel value of each pixel point in the pre-processed vehicle driving image is subjected to foreground target recognition to extract the bounding box coordinates (including the maximum and minimum coordinate values in the horizontal direction, the maximum and minimum coordinate values in the vertical direction) of a plurality of foreground target regions (such as vehicles, pedestrians, non-motor vehicles, traffic facilities, etc.) and the pixel points in the corresponding foreground target regions, and the pixel area (i.e. the total number of pixel points in each foreground target region) occupied by each foreground target region in the image and its center coordinate in the vertical direction of the image (i.e. the sum value of the maximum and minimum coordinate values in the vertical direction / 2) are calculated respectively, the deviation value of the center coordinate of each foreground target in the vertical direction of the image from the vertical coordinate of the main visual center of the image (i.e. the height of the image / 2) is subjected to exponential mapping as a weighting factor to obtain the weighted occlusion area of each foreground target, and the weighted occlusion areas of all foreground targets are accumulated to obtain the total occlusion area value, in the set main view area (such as the horizontal viewing angle range of ±15° in the center of the image), the total number of pixel points occluded by the foreground targets (i.e. the total number of pixel points of each foreground target region in the viewing angle range) and the total number of pixel points in the main view area (i.e. the total number of pixel points in the viewing angle range) are counted and subjected to ratio processing to obtain the main view occlusion rate, and the total occlusion area value is subjected to standardization processing, and the visual occlusion interference feature is extracted based on the standardization processing result for weighted processing to evaluate the degree of occlusion of the front view by the foreground targets (such as vehicles, pedestrians, non-motor vehicles, etc.); The gradient amplitude of each pixel point in the vehicle driving image is calculated based on an image gradient algorithm (such as a Sobel operator) (i.e., the horizontal gradient and the vertical gradient of each pixel point are extracted based on the Sobel operator and calculated), the gradient value of each pixel point is obtained, the average gradient value is counted, it is judged whether the gradient value of each pixel point is higher than a set threshold (such as 1.2 times of the average gradient value), if yes, the pixel point is marked as a visual high-frequency attention pixel point, and connected domain processing (such as 8-neighborhood) is performed to obtain a plurality of visual high-frequency attention regions (representing image regions that the driver may focus on), the pixel value of each pixel point in each visual high-frequency attention region is subjected to gray processing to obtain the gray pixel value of the corresponding pixel point, the probability of each gray pixel value is counted, the structural entropy of each visual high-frequency attention region is analyzed based on the Shannon formula (measuring the complexity of the image information of the region), the gradient direction angle of each pixel point in each visual high-frequency attention region, i.e., arctan (vertical gradient / horizontal gradient), is counted, the gradient direction angle variance of each visual high-frequency attention region is extracted, and the results are subjected to mean value processing and weighted processing to extract visual focus complex features for evaluating the attention degree of the driver to the visual high-frequency region during driving; and the road coherence feature, the visual occlusion interference feature and the visual focus complex feature are combined in a preset order into a set of three-dimensional structured feature vectors to construct a driving environment feature vector for describing the current image scene state.

[0055] The driving sub-network includes a driving preprocessing layer, a driving behavior feature extraction layer, and a driving fusion output layer. The specific steps of analyzing the driving behavior abnormal feature value of the driver to be evaluated are as follows: in the driving preprocessing layer of the driving sub-network, the driving image data of the driver to be evaluated is received and preprocessed, which specifically includes: uniformly adjusting the driving image of the driver to be evaluated, for example, standardizing the image size to 224x224 pixels, performing RGB value normalization processing on each driving pixel point in the image, mapping the pixel value range from [0, 255] to the [0, 1] interval, removing noise in the image by using Gaussian blur or median filtering, and improving the distinguishability of the driver's face and the driving environment by brightness enhancement, contrast adjustment, and sharpening; in the driving behavior feature extraction layer of the driving sub-network, the driving behavior feature vector of the driver to be evaluated is extracted based on the preprocessed driving image data of the driver to be evaluated; in the driving fusion output layer of the driving sub-network, the driving behavior abnormal feature value of the driver to be evaluated is analyzed based on the driving behavior feature vector of the driver to be evaluated, which specifically includes: performing weighted mapping and nonlinear transformation (i.e., inputting into a set fully connected layer, performing linear weight multiplication and addition processing, and activation function processing such as sigmoid) on the head yaw feature value and the sitting posture deviation feature value in the driving behavior feature vector to generate the driving behavior abnormal feature value, which is used to represent the degree of abnormal behavior of the driver during driving.

[0056] wherein the specific steps of extracting the driving behavior feature vector of the driver to be evaluated are as follows: detecting the face of the driver in the driving image based on a face detection model (such as MTCNN or OpenFace), extracting the face region and locating the face key points, including but not limited to the center two-dimensional coordinates of the key regions such as eyes, eyebrows, nose, mouth (such as the average value of the driving two-dimensional coordinates of each driving pixel point in the eye region), and based on the center two-dimensional coordinates of the face key points, obtaining the yaw angle of the driver's head (the rotation of the head in the horizontal plane, i.e. left and right rotation) through a pose estimation model (such as 3D Head Pose Estimation), and obtaining the reference direction of the driver's upper body (i.e. extracting the key points of the upper body from the driving image based on a body pose estimation model such as OpenPose, HRNet, etc., including: the center two-dimensional coordinates of the left shoulder and the right shoulder, and performing coordinate difference processing, and based on the arctangent function, the angle conversion is performed on the result to obtain the reference direction of the driver's upper body), and the yaw angle of the driver's head and the reference direction of the driver's upper body are processed by ratio, i.e. |yaw angle-reference direction of upper body| / reference direction of upper body, to extract the head yaw feature value, which is used to quantify the degree of rotation of the driver's head, reflecting the amplitude of the driver's head rotation during driving, so as to evaluate the driver's concentration; Based on a human pose estimation model (such as OpenPose or HRNet), the key points of the upper body are extracted from the driver image, such as the center two-dimensional coordinates of the left shoulder and the right shoulder, the center two-dimensional coordinates of the upper and lower parts of the spine, and the coordinates of the shoulder center point and the middle point of the spine (i.e. the average value of the center two-dimensional coordinates of the left shoulder and the right shoulder, and the average value of the center two-dimensional coordinates of the upper and lower parts of the spine), and the average value is processed to obtain the coordinates of the center point of the upper body, and the coordinates of the center point of the driver's seat are extracted (i.e. based on YOLOv5 or Faster R-CNN to detect objects in the car and identify the driver's seat area, and based on an edge detection algorithm to extract the driving two-dimensional coordinates of the left edge, right edge, upper edge and lower edge of the seat, and taking the average value of the horizontal coordinate values of the left edge and right edge driving two-dimensional coordinates and the average value of the vertical coordinate values of the upper edge and lower edge driving two-dimensional coordinates to obtain the coordinates of the center point of the driver's seat), and the horizontal coordinate values of the center point of the upper body and the center point of the driver's seat are processed by difference (taking the average value) to extract the sitting posture deviation feature value, which is used to evaluate the stability of the driver's sitting posture. The driver with excessive sitting posture deviation may have an unhealthy posture, such as excessive forward leaning or leaning to one side, which may cause limited vision or unstable control, affecting the safety of driving behavior; and the head yaw feature value and the sitting posture deviation feature value are combined into a set of two-dimensional structured feature vectors in a predetermined order to construct the driving behavior feature vector.

[0057] In the embodiment, the driving image data and the driving image data of the vehicle are analyzed in depth based on the driving sub-network and the driving sub-network respectively, thereby effectively improving the accuracy of the driver safety evaluation. Secondly, the preprocessing layer in the driving sub-network uniformly adjusts and normalizes the image data, thereby enhancing the recognizability of the image, providing high-quality data input for subsequent feature extraction, and through the driving environment feature extraction layer, the system can identify and extract key features, and then evaluate the degree of distraction of the driver in the complex environment, which helps to identify potential driving risks in time. Finally, the driving sub-network extracts the head yaw and sitting posture deviation features of the driver, analyzes the concentration of the driver's attention and the stability of the posture, and accurately evaluates the abnormal behavior of the driver during driving, thereby improving the driving safety.

[0058] Although preferred embodiments of the application have been described herein, changes and modifications can be suggested to one skilled in the art, and it is intended that the application encompass such changes and modifications as fall within the scope of the appended claims.

[0059] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A real-time driver safety risk assessment system based on multi-source data fusion, characterized in that, include: The vehicle evaluation module is used to acquire vehicle status data of the driver to be evaluated in real time and analyze the driving response stability characteristics of the driver to be evaluated. The behavior assessment module is used to acquire vehicle driving image data and driving image data of the driver to be assessed, and combine them with a pre-trained safety assessment model to analyze the behavioral safety feature values ​​of the driver to be assessed. The comprehensive evaluation module is used to analyze the safety risk characteristics of the driver to be evaluated based on the driving response stability characteristics and behavioral safety characteristics. The risk feedback module is used to assess the safety risks of drivers based on safety risk characteristic values.

2. The real-time driver safety risk assessment system based on multi-source data fusion according to claim 1, characterized in that, The vehicle state data includes yaw rate, vehicle speed, steering wheel grip force entropy, seat pressure gradient, and brake disc thermal radiation gradient. The specific steps for analyzing the driving response stability characteristics of the driver to be evaluated are as follows: Based on the vehicle status data of the driver to be evaluated, the driving response feature set of the driver to be evaluated is analyzed, including vehicle steady-state feature value and driving control stability feature value. Based on the driving response feature set of the driver to be evaluated, the stable feature values ​​of the driving response of the driver to be evaluated are analyzed.

3. The real-time driver safety risk assessment system based on multi-source data fusion according to claim 2, characterized in that, The specific steps for analyzing the driving response feature set of the driver to be evaluated are as follows: Based on the yaw rate, vehicle speed, and brake disc thermal radiation gradient of the driver to be evaluated, the steady-state characteristics of the vehicle condition of the driver to be evaluated are analyzed. Based on the steering wheel grip force entropy value and seat pressure gradient value of the driver to be evaluated, the driving control stability characteristic value of the driver to be evaluated is analyzed.

4. The real-time driver safety risk assessment system based on multi-source data fusion according to claim 1, characterized in that, The vehicle driving image data specifically refers to the pixel value and two-dimensional coordinates of each pixel in the vehicle driving image, and the driving image data specifically refers to the driving pixel value and driving two-dimensional coordinates of each driving pixel in the driving image.

5. The real-time driver safety risk assessment system based on multi-source data fusion according to claim 4, characterized in that, The specific steps for analyzing the behavioral safety characteristics of the driver to be evaluated are as follows: The vehicle driving image data and driving image data of the driver to be evaluated are input into the pre-trained safety assessment model for comprehensive analysis to obtain the visual safety feature set of the driver to be evaluated, including driving visual disturbance feature values ​​and abnormal driving behavior feature values. Based on the visual safety feature set of the driver to be evaluated, the behavioral safety feature values ​​of the driver to be evaluated are analyzed.

6. The real-time driver safety risk assessment system based on multi-source data fusion according to claim 5, characterized in that, The safety assessment model includes a driving subnetwork and a driver subnetwork. The specific steps for analyzing the visual safety feature set of the driver to be assessed are as follows: In the driving sub-network of the safety assessment model, the vehicle driving image data of the driver to be evaluated is received, and the driving visual disturbance feature value of the driver to be evaluated is analyzed. In the driving sub-network of the safety assessment model, driving image data of the driver to be assessed is received, and abnormal driving behavior feature values ​​of the driver to be assessed are analyzed.

7. The real-time driver safety risk assessment system based on multi-source data fusion according to claim 6, characterized in that, The driving sub-network includes a preprocessing layer, a driving environment feature extraction layer, and a driving fusion output layer. The specific steps for analyzing the driving visual disturbance feature values ​​of the driver to be evaluated are as follows: In the preprocessing layer of the driving sub-network, the vehicle driving image data of the driver to be evaluated is received and preprocessed. In the driving environment feature extraction layer of the driving sub-network, the driving environment feature vector of the driver to be evaluated is extracted based on the preprocessed vehicle driving image data of the driver to be evaluated. In the driving fusion output layer of the driving subnetwork, the driving visual disturbance feature value of the driver to be evaluated is analyzed based on the driving environment feature vector of the driver to be evaluated.

8. The real-time driver safety risk assessment system based on multi-source data fusion according to claim 6, characterized in that, The driving sub-network includes a driving preprocessing layer, a driving behavior feature extraction layer, and a driving fusion output layer. The specific steps for analyzing the abnormal driving behavior feature values ​​of the driver to be evaluated are as follows: In the driving preprocessing layer of the driving subnetwork, driving image data of the driver to be evaluated is received and preprocessed. In the driving behavior feature extraction layer of the driving sub-network, the driving behavior feature vector of the driver to be evaluated is extracted based on the preprocessed driving image data of the driver to be evaluated. In the driving fusion output layer of the driving subnetwork, abnormal feature values ​​of the driving behavior of the driver to be evaluated are analyzed based on the driving behavior feature vector of the driver to be evaluated.

9. The real-time driver safety risk assessment system based on multi-source data fusion according to claim 1, characterized in that, The specific formula for calculating the safety risk characteristic value of the driver to be evaluated is as follows: ； in, The safety risk characteristic value of the driver to be evaluated. The driving response stability characteristic value of the driver to be evaluated. The response stability coefficients are stored in the database. The behavioral safety characteristic value of the driver to be evaluated. The safety factor for behaviors stored in the database. These are adjustment coefficients stored in the database. These are the interaction coefficients stored in the database.

10. The real-time driver safety risk assessment system based on multi-source data fusion according to claim 1, characterized in that, The specific steps for conducting a driver safety risk assessment based on safety risk characteristic values ​​are as follows: The safety risk characteristic values ​​of the driver to be evaluated are compared with the preset safety risk characteristic value range; If the safety risk characteristic value of the driver to be evaluated is lower than the lower limit of the preset safety risk characteristic value range, it will be marked as a green risk level; If the safety risk characteristic value of the driver to be evaluated is within the preset safety risk characteristic value range, it will be marked as a yellow risk level; If the safety risk characteristic value of the driver to be evaluated is higher than the upper limit of the preset safety risk characteristic value range, it will be marked as a red risk level.

Images