Vehicle auxiliary control method and related device

Abstract

The invention discloses a vehicle auxiliary control method and a related device, and belongs to the technical field of vehicle auxiliary control, and the method comprises the steps: obtaining vehicle driving data and driver physiological data; according to the vehicle driving data and the driver physiological data, carrying out trust degree evaluation on the driver to obtain a trust score of the driver; determining an auxiliary control strategy for assisting the driver in driving the vehicle according to the trust score; and auxiliary control is carried out in the process that the driver drives the vehicle according to the auxiliary control strategy. According to the method, the reliability degree of the driver is converted into the quantifiable trust score through the multi-modal data, the auxiliary control strategy of the vehicle is dynamically adjusted through the trust score of the driver, the vehicle operation risk caused by the poor state of the driver is effectively reduced, and the driving safety is improved.

Description

Technical Field

[0001] This application relates to the field of vehicle auxiliary control technology, specifically to a vehicle auxiliary control method and related devices. Background Technology

[0002] As vehicle automation levels increase, driver condition monitoring plays an increasingly important role in improving driving safety. Driver fatigue, distraction, and emotional fluctuations directly affect driving behavior, thereby increasing the risk of traffic accidents.

[0003] While traditional driver assistance systems have achieved a certain level of perception and warning of the vehicle environment, they still have technical limitations in monitoring the driver's personal state, resulting in rigid control strategies that are difficult to meet higher demands for assisted driving and affect the driver's operating experience. Summary of the Invention

[0004] This application provides a vehicle auxiliary control method and related device to solve the above problems.

[0005] In a first aspect, embodiments of this application provide a vehicle auxiliary control method, the method comprising: Acquire vehicle driving data and driver physiological data; The driver's trust level is assessed based on the vehicle driving data and the driver's physiological data to obtain the driver's trust score; wherein, the trust score characterizes the driver's driving state and the reliability of the driver driving the vehicle in the driving state; The assistance control strategy for assisting the driver in driving the vehicle is determined based on the trust score; wherein, the assistance control strategy includes multiple assistance control actions and corresponding prompt information for the actions; The auxiliary control strategy is used to provide auxiliary control during the driver's operation of the vehicle.

[0006] Secondly, embodiments of this application provide a vehicle auxiliary control device, the device comprising: The data acquisition module is used to acquire vehicle driving data and driver physiological data; The trust scoring module is used to assess the trust level of the driver based on the vehicle driving data and the driver's physiological data, and obtain the driver's trust score; wherein, the trust score represents the driver's driving state and the reliability of the driver driving the vehicle in the driving state; a policy making module, configured to determine an assistance control policy for assisting the driver to drive the vehicle according to the trust score; wherein the assistance control policy comprises a plurality of actions of assistance control and prompt information corresponding to the actions; an assistance control module, configured to perform assistance control on the vehicle during driving of the vehicle according to the assistance control policy.

[0007] In a third aspect, an electronic device is provided, including a processor and a memory, the memory storing a computer program, and the processor calling the computer program to execute the method in the first aspect of the present application.

[0008] In a fourth aspect, a computer readable storage medium is provided, the computer readable storage medium storing a computer program, and when the computer program runs on a computer, the computer program causes the computer to execute the method in the first aspect of the present application.

[0009] The technical solutions provided by some embodiments of the present application have at least the following beneficial effects: The present application evaluates the trust degree of the driver by obtaining vehicle driving data and driver physiological data, and determines the assistance control policy of the vehicle according to the obtained driver trust score, and performs assistance control on the vehicle during driving of the driver.

[0010] The present application can effectively deal with the physiological abnormality of the driver by comprehensively analyzing the driving state of the driver by combining multi-source data such as vehicle driving data and driver physiological data, avoid the limitation of a single data source, and convert the reliability of the driver into a quantifiable trust score to provide an objective basis for subsequent decision-making and reduce the deviation of subjective judgment; the present application can realize dynamic adjustment of the assistance control policy of the vehicle based on the trust score of the driver, thereby realizing dynamic allocation of the control right of the vehicle and greatly improving the operation experience of the driver; finally, the present application performs real-time assistance control on the vehicle during driving by the assistance control policy, effectively reduces the operation risk caused by poor state of the driver, and improves the driving safety. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0012] Figure 1A system architecture example diagram of a vehicle auxiliary control method provided by an embodiment of the present application is shown in FIG. 1. Figure 2 A flowchart of a vehicle auxiliary control method provided by an embodiment of the present application is shown in FIG. 2. Figure 3 Another flowchart of a vehicle auxiliary control method provided by an embodiment of the present application is shown in FIG. 3. Figure 4 Another flowchart of a vehicle auxiliary control method provided by an embodiment of the present application is shown in FIG. 4. Figure 5 Another flowchart of a vehicle auxiliary control method provided by an embodiment of the present application is shown in FIG. 5. Figure 6 A structure diagram of a vehicle auxiliary control device provided by an embodiment of the present application is shown in FIG. 6. Figure 7 A structure diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 7. DETAILED DESCRIPTION

[0013] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0014] The terms “first”, “second”, “third” and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, and are not used to describe a specific order. In addition, the terms “include” and “have” and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed or can optionally include other steps or units inherent to the process, method, product or device.

[0015] Before introducing the technical solutions of the present application, the technical terms involved in the present application are explained: Inertial Measurement Unit (IMU), a device integrating multiple sensors, mainly used for measuring and reporting the three-dimensional spatial state change of an object, mainly including: accelerometer, gyroscope, etc.

[0016] Augmented Reality Head-Up Display (AR-HUD), used to superimpose key driving information (such as navigation guidance, obstacle identification, lane line deviation warning, etc.) directly onto the windshield in front of the driver, so that it dynamically aligns with the real road conditions.

[0017] Precision Time Protocol (PTP) is designed to achieve high-precision clock synchronization over computer networks, suitable for application scenarios with extremely high requirements for time synchronization.

[0018] Bird’s Eye View (BEV) is a perspective of viewing a scene from a high vertical or inclined downward.

[0019] With the development of intelligent transportation systems and autonomous driving technology, the safety and reliability of vehicle driving have become a research focus. However, in the transitional stage before autonomous driving is fully popularized, the driver is still the core subject of vehicle control, and his driving state directly affects road safety. Auxiliary driving technology monitors the vehicle state, environmental information, and driver behavior in real time, provides early warning or active intervention, and enhances driving safety and comfort.

[0020] Currently, the core challenge of auxiliary driving technology is how to improve the reliability and accuracy of environmental perception through multi-sensor fusion. The industry generally uses a combination of cameras, millimeter wave radars, and laser radars to perceive environmental information, but each sensor has inherent defects, such as cameras being susceptible to light and bad weather, millimeter wave radars lacking height information and being sensitive to metal, and laser radars being able to generate high-precision point clouds but being expensive. Existing technologies mostly use hierarchical fusion strategies (such as early fusion, feature-level fusion, or late fusion) to achieve multi-sensor fusion, but face problems such as data heterogeneity, temporal and spatial alignment errors, and computational bottlenecks. For example, early fusion schemes directly integrate multi-modal information at the raw data level, but it is difficult to handle the low correlation between RGB images and sparse point clouds; late fusion can model each sensor feature independently, but its performance is limited in complex dynamic scenarios, especially the sparsity of laser radar point clouds, which leads to poor fusion results. In addition, traditional auxiliary control systems mostly rely on a single sensor (such as Tesla's pure vision solution), which is prone to failure in extreme weather or sudden scenarios, and lacks the ability to dynamically model the driver's state with multiple modalities.

[0021] There are currently some driver state monitoring schemes and risk avoidance technologies based on vision and steering wheel. However, most of them have problems such as single sensor, inability to respond to sudden situations, and rigid control strategies, leading to poor adaptability of risk avoidance strategies. Some other solutions are multi-sensor fusion technology, which can achieve certain effects of vehicle auxiliary control through the complementary advantages of multiple sensors, but cannot adapt to complex scenarios and lacks real-time performance, making it difficult to meet higher auxiliary driving needs.

[0022] In summary, the current technical solutions mainly have the following shortcomings: First, the current technical solutions cannot effectively detect physiological abnormalities of the driver, resulting in no timely emergency response in sudden situations; Secondly, relying on a small amount of sensor data is easy to cause misjudgment; Finally, the current technical solution adopts a "one-size-fits-all" auxiliary control system for different drivers, and directly takes over the control of the vehicle in an emergency state, which leads to rigid control strategy, poor driving experience and increased psychological resistance of the driver, and reduces the utilization rate of the auxiliary driving system.

[0023] The present application aims at the above problems, and proposes a vehicle auxiliary control method, which integrates vehicle driving data and driver physiological data to evaluate the trust degree of the driver, obtains the trust score of the driver, and dynamically adjusts the auxiliary control strategy of the vehicle according to the trust score, greatly improves the driving experience of the driver, and improves the driving safety.

[0024] The present application will be described in detail below in combination with specific embodiments.

[0025] Please refer to Figure 1 , Figure 1 A system architecture example of a vehicle auxiliary control method provided by an embodiment of the present application. As shown in the figure, the system architecture includes a sensor input layer, a data processing layer and an information output layer. Figure 1 The sensor input layer is used to obtain vehicle driving data and driver physiological data based on multi-modal sensors.

[0026] The sensor input layer is used to obtain vehicle driving data and driver physiological data based on multi-modal sensors.

[0027] For example, vehicle driving data and driver physiological data are obtained through multi-modal sensors such as vehicle-mounted cameras, vehicle-mounted radars, vehicle-mounted voice devices, IMUs and wristbands. Among them, the vehicle-mounted radar can be a vehicle-mounted laser radar or a vehicle-mounted millimeter wave radar.

[0028] The data processing layer is used to evaluate the trust degree of the driver according to the vehicle driving data and the driver physiological data, and obtain the trust score of the driver; and determine the auxiliary control strategy for assisting the driver to drive the vehicle according to the trust score.

[0029] In an embodiment of the present application, the trust score represents the driving state of the driver and the reliability of the driver driving the vehicle in the driving state.

[0030] For example, the visual agent can perform semantic segmentation on the images collected by the vehicle-mounted camera to obtain real-time lane lines, pedestrians, traffic signs and other visual information. The point cloud agent can process the point cloud data collected by the vehicle-mounted radar to obtain obstacle distance and speed information, and make up for the lack of information of the visual agent in adverse conditions (rain, fog, dim light, night). The speech agent can process the speech data collected by the vehicle-mounted speech device to monitor abnormal sound information inside and outside the vehicle, and provide driver speech behavior records, which can be used for driver personal portrait incremental learning. The signal agent can process the signals of the IMU and other vehicle-mounted hardware to monitor real-time vehicle information, including vehicle acceleration, steering wheel angle, etc. The physiological agent can process the physiological data (heart rate, blood oxygen, etc.) of the driver obtained by the smart bracelet and other wearable devices to monitor the safety status of the driver and provide driver physiological data records.

[0031] The various agents above form a perception agent cluster. After the real-time collected vehicle driving data and driver physiological data are preprocessed by the perception agent cluster, the driver is evaluated by a decision model to obtain a trust score of the driver, and an auxiliary control strategy for assisting the driver in driving the vehicle is determined according to the trust score.

[0032] An information output layer is configured to perform auxiliary control according to the auxiliary control strategy during the driving of the vehicle by the driver.

[0033] In the embodiments of the present application, the auxiliary control strategy includes a plurality of auxiliary control actions and prompt information corresponding to the actions, wherein the plurality of auxiliary control actions are used for vehicle auxiliary control, and the prompt information corresponding to the actions can be realized by AR-HUD, vehicle-mounted speech device, smart bracelet, etc., such as visual prompting by AR-HUD, voice prompting by vehicle-mounted speech device, vibration prompting by smart bracelet, etc., for reminding the driver to perform corresponding actions, improving the driver assistance system use experience and ensuring safe driving.

[0034] It should be noted that the above system architecture is only exemplary, and the system architecture can be designed according to actual needs, and the present application does not make specific limitations.

[0035] Please refer to Figure 2 , Figure 2A flowchart of a vehicle auxiliary control method provided by an embodiment of the present application is shown. The execution subject of the vehicle auxiliary control method of the present application can be a processor in a vehicle, or a chip loaded in the processor of the vehicle, a vehicle-mounted controller, or a cloud server in communication connection with the vehicle, or jointly executed by the vehicle and the cloud server in communication connection with the vehicle. The embodiment of the present application takes the processor in the vehicle as an example to describe the vehicle auxiliary control method.

[0036] As shown in Figure 2 The vehicle auxiliary control method provided by an embodiment of the present application can include the following steps: S101, obtaining vehicle driving data and driver physiological data.

[0037] Specifically, various vehicle-mounted sensors can be used to obtain the vehicle driving data. For example, the visual image data of the inside and / or outside of the vehicle obtained by the vehicle-mounted camera is used as the inside / outside environment data, the point cloud data obtained by the vehicle-mounted radar is used as another kind of outside environment data, the driver voice data obtained by the vehicle-mounted voice device is used as the inside voice data, and the vehicle acceleration, steering wheel angle and other data obtained by the IMU are used as the vehicle operation data when the driver drives the vehicle. These inside environment data, outside environment data, inside voice data and vehicle operation data all belong to the vehicle driving data.

[0038] The driver physiological data can be collected by wearable devices, such as the heart rate and blood oxygen data collected by the smart bracelet worn by the driver.

[0039] Since the vehicle driving data and the driver physiological data come from different sensors, after the original data is collected, the problem of spatio-temporal alignment of multi-modal sensor data needs to be solved. First, the PTP protocol is used to align the timestamps of the sensor data to control the data frame synchronization error. Then, for the point cloud data and the IMU data, a unified coordinate system can be established based on a joint calibration tool (such as Autoware software) to obtain a projection matrix of the radar point cloud data to the visual image data and an extrinsic matrix between the IMU data and the vehicle chassis coordinate system, which describes the pose relationship of the IMU data relative to the vehicle chassis coordinate system. The spatio-temporal alignment of the vehicle driving data and the driver physiological data collected by the multi-modal sensors is realized through the above method.

[0040] S102, performing trust evaluation on the driver according to the vehicle driving data and the driver physiological data to obtain a trust score of the driver.

[0041] In the embodiment of the present application, the trust score represents the driving state of the driver and the reliability of the driver in driving the vehicle.

[0042] Exemplarily, a deep learning model can be used to evaluate the trust degree of the driver to obtain a trust score of the driver. Specifically, after the in-vehicle and out-of-vehicle environment data, in-vehicle voice data, vehicle operation data and driver physiological data in the vehicle driving data are normalized, denoised, image enhanced and the like, the pre-trained deep learning model is input, and the trust score of the driver is output.

[0043] Exemplarily, the in-vehicle voice data, vehicle operation data and driver physiological data can also be scored according to a preset scoring rule to obtain a comprehensive score as the trust score of the driver. Specifically, different weights are assigned to the in-vehicle voice data, vehicle operation data and driver physiological data, the voice emotion of the driver is recognized by analyzing the in-vehicle voice data through a machine learning model, and the voice emotion of the driver is scored, for example, the corresponding score is deducted for an angry emotion; the operation stability score of the driver is scored based on the vehicle operation data, for example, the corresponding score is deducted for a too large standard deviation between a certain vehicle operation and a standard operation; the physiological state score of the driver is scored based on the change of the driver physiological data, for example, the corresponding score is deducted for a too large heart rate change. Each of the above data can be deducted by a preset score value, for example, 5 points are deducted for an emergency brake.

[0044] Finally, a comprehensive score S is obtained as the trust score of the driver, wherein, respectively, the weight of the i-th data and the corresponding deduction score, and n is the total number of data items participating in the scoring.

[0045] S103, determining an auxiliary control strategy for assisting the driver to drive the vehicle according to the trust score.

[0046] In the embodiments of the present application, the auxiliary control strategy includes a plurality of auxiliary control actions and corresponding prompt information of the actions.

[0047] For example, if the trust score is lower, it indicates that the driver's state is worse, and intervention is needed. A multi-level strategy can be divided according to the trust score, and each level includes corresponding auxiliary control actions and prompt information. For example, if the trust score ranges from 0 to 30, emergency intervention is needed, and the corresponding auxiliary actions can be forced deceleration to a safe speed, automatic parking, unlocking the door (in an emergency), and the like, and the corresponding prompt information can be a danger warning and a voice reminder that the system will take over the vehicle. When the score ranges from 30 to 70, active assistance is needed, and the corresponding auxiliary actions can be a red flashing instrument panel, a limited accelerator pedal response, and the like, and the corresponding prompt information can be a voice reminder for the driver to concentrate and take a break or a reminder based on the vehicle display device. When the score ranges from 70 to 100, passive monitoring is needed, and the corresponding auxiliary actions can be data recording, slight steering wheel haptic feedback, and environmental light adjustment (to alleviate fatigue), and the like. At this time, the driver is relatively reliable, and no prompt is needed.

[0048] In the embodiments of the present application, the basic principle of the auxiliary control strategy is that when the trust score of the driver decreases, the degree of the system's control over the vehicle is increased to perform auxiliary control of the vehicle, and vice versa, the degree of the system's control over the vehicle is reduced, and the diversified auxiliary control strategy of the vehicle can be determined according to the basic principle.

[0049] S104, performing auxiliary control according to the auxiliary control strategy during the process of the driver driving the vehicle.

[0050] During the process of the driver driving the vehicle, auxiliary control is performed according to the auxiliary control strategy determined in S203. After a single auxiliary control is completed, S201 is returned to continuously acquire vehicle driving data and driver physiological data, and the driver's trust score is recalculated. When it is detected that the trust level to which the driver's trust score belongs changes, the auxiliary control strategy of the vehicle is determined again, until the vehicle ends the current trip.

[0051] The present application can effectively deal with the physiological abnormality of the driver by comprehensively analyzing the driving state of the driver by combining vehicle driving data and driver physiological data and other multi-source data, avoid the limitations of a single data source, and convert the reliability of the driver into a quantifiable trust score to provide an objective basis for subsequent decision-making and reduce the deviation of subjective judgment. The present application can dynamically adjust the auxiliary control strategy of the vehicle based on the trust score of the driver, thereby achieving dynamic allocation of vehicle control rights and greatly improving the driver's operation experience. Finally, the present application performs real-time auxiliary control of the vehicle during driving through the auxiliary control strategy, effectively reduces the operation risk caused by the poor state of the driver, and improves the safety of driving.

[0052] Referring to Figure 3 , Figure 3 Another flowchart of a vehicle auxiliary control method provided by an embodiment of the present application. The vehicle auxiliary control method comprises: S201, obtaining vehicle driving data and driver physiological data.

[0053] This step can refer to S101, which will not be described here.

[0054] S202, extracting state feature data representing the current driving state of the driver according to the vehicle driving data and the driver physiological data.

[0055] The vehicle driving data can reflect the comprehensive running state of the vehicle under the driving of the driver, and the driver physiological data directly reflects the physiological state of the driver, so the current state feature data of the driver can be extracted according to the vehicle driving data and the driver physiological data.

[0056] In some possible embodiments, the vehicle driving data at least includes in-vehicle voice data from the driver and vehicle operation data of the driver driving the vehicle; on this basis, S302 can include: extracting behavior feature data related to the driver according to the vehicle driving data at least including the in-vehicle voice data and the vehicle operation data; extracting state feature data representing the current driving state of the driver according to the behavior feature data and the driver physiological data.

[0057] Specifically, the voice change feature of the driver can be extracted through the in-vehicle voice data, which can reflect the emotional state of the driver, such as anxiety and anger, and the operation change feature of the driver can be extracted through the vehicle operation data, which can evaluate the operation level of the driver. The voice change feature and the operation change feature of the driver are spliced to obtain the behavior feature data of the driver when driving the vehicle. According to these behavior feature data, combined with the physiological change feature extracted based on the driver physiological data, the state feature data evaluating the current driving state of the driver can be obtained. The state feature data is a multi-dimensional feature vector obtained by splicing or transforming the voice change feature, the operation change feature and the physiological change feature of the driver.

[0058] The present application can more comprehensively capture the state of the driver such as fatigue, distraction, emotional fluctuation and body change by combining multi-dimensional data such as vehicle operation data, in-vehicle voice data and driver physiological data, avoiding the limitation of a single data source; in night driving or bad weather, the physiological data can make up for the deficiency of the vehicle sensor due to environmental interference, improving the accuracy of the driver state evaluation; the present application can realize dynamic analysis of the driving state of the driver based on the data flow of the vehicle driving data and the driver physiological data, with high real-time performance.

[0059] Exemplarily, the nonlinear correlation between each item of the vehicle driving data and the driver physiological data can be evaluated by mutual information (MI), the key features are screened from the vehicle driving data and the driver physiological data, the feature vectors of the key features are extracted by a machine learning model, the feature vectors of the key features are combined, and the current state feature data of the driver is obtained.

[0060] S203, difference information between the driving state of the driver and the normal driving state is determined according to the state feature data of the driver and the preset personal portrait of the driver.

[0061] In the embodiment of the present application, the personal portrait of the driver includes the normal driving state of the driver driving the vehicle under the historical scene with the trust score higher than the trust score threshold. The personal portrait of the driver is constructed by using incremental learning according to the driver physiological data and the in-vehicle environment data (such as in-vehicle voice data, vehicle operation data, etc.), and the steps of constructing the personal portrait of the driver are as follows: A, heterogeneous data preprocessing and alignment The collected driver physiological data, in-vehicle voice data and IMU data are aligned by using a sliding window mechanism. The driver physiological data includes physiological data such as heart rate and blood oxygen, and the sampling frequency can be set to 1 Hz. The sampling frequency of the in-vehicle voice data can be set to 16 Hz, and the mel spectrum graph is extracted by frame processing. The IMU data includes acceleration and steering wheel angle data, and the sampling frequency can be set to 1 Hz. The window length of the sliding window mechanism can be set to 1 s, and the step length is 0.1 s.

[0062] B, multi-source data feature extraction The heart rate variation feature in the driver physiological data is calculated by time domain standard deviation and frequency domain power ratio, and the blood oxygen variation feature is calculated by an indicator function, to obtain the physiological variation features corresponding to the driver physiological data. The value of the indicator function is 0 or 1, which is 1 if there is a change, otherwise 0.

[0063] The 128-dimensional mel spectrum features of the in-vehicle voice data are extracted by a deep learning network such as MobileNetV2, and the abnormal features such as continuous coughing and rapid breathing are focused on, and the voice variation features corresponding to the in-vehicle voice data are calculated. The vehicle operation data (IMU data) of the driver is recorded by an indicator function, and the operation variation features corresponding to the vehicle operation data are calculated.

[0064] C, portrait construction and incremental learning The continuous feature map of the driver is constructed based on the physiological change feature, the speech change feature and the operation change feature, the feature map is taken as the state feature data of the driver, and the feature map is continuously updated and accumulated according to the subsequent driving behavior to obtain a feature map sequence. The feature map sequence can be taken as the historical state feature data record of the driver. When new driver state feature data appears, loss calculation is performed with the mean value of the state feature data:

[0065] wherein, represents the driver state feature data at time t, represents the normal driving state of the driver driving the vehicle with the trust score higher than the trust score threshold in the historical scene, the normal driving state can be the mean value of the state feature data of the driver driving the vehicle with the trust score higher than the trust score threshold in the historical scene, and is the key information for expressing the personal portrait of the driver. represents the feature loss, which represents the difference information between the driving state of the driver and the normal driving state, The larger the difference information is, the more abnormal the driving state of the driver is.

[0066] The personal portrait of the driver is constructed in the above manner, the driving habit of the driver such as the lane changing frequency and the brake force can be gradually learned, and the normal driving state of the driver driving the vehicle can be updated in real time according to the latest driver state feature data to improve the accuracy of the personal portrait of the driver.

[0067] S204, calculating the trust score of the driver by calculating the vehicle driving data, the driver physiological data and the difference information.

[0068] In the embodiment of the present application, the trust score represents the driving state of the driver and the reliability of the driver driving the vehicle in the driving state, and the trust score of the driver can quantify the necessity of the system taking over the driver state.

[0069] For example, the vehicle driving data, the driver physiological data and the difference information can be respectively assigned weights, and the trust score of the driver can be calculated by weighting. Specifically, the collected driver physiological data includes heart rate h , blood oxygen s, the collected vehicle driving data includes acceleration and steering wheel angle θ of the vehicle, the difference information between the driving state of the driver and the normal driving state is determined by S303 , and then the trust score of the driver is calculated:

[0070] wherein is the trust score of the driver, respectively, are first weights. is an S-shaped function, is a linear rectifier function. It can be understood that the value of the first weight is set as needed, for example, it can be taken as .

[0071] Exemplarily, the trust evaluation model of the driver can also be constructed through a convolutional neural network. After the vehicle driving data, the driver physiological data and the difference information are processed by feature engineering, the trust evaluation model is input, and the trust score of the driver output by the trust evaluation model is obtained.

[0072] In some possible embodiments, the trust score is negatively correlated with the difference degree between the driving state of the driver in the difference information and the normal driving state.

[0073] When the difference between the driving state of the driver and the normal driving state is greater, it means that the driver is in an abnormal condition, and at this time, the trust score of the driver is smaller, so as to accurately quantify the driver state and the necessity of system takeover according to the trust score of the driver.

[0074] S205, determining an auxiliary control strategy for assisting the driver to drive the vehicle according to the trust score.

[0075] This step can refer to S103, which will not be described here.

[0076] S206, performing auxiliary control according to the auxiliary control strategy in the process of driving the vehicle by the driver.

[0077] This step can refer to S104, which will not be described here.

[0078] The present application extracts the state feature data of the driver by integrating the vehicle driving data, the driver physiological data and other multi-modal data, realizes real-time quantitative monitoring of the driver state, and can monitor sudden diseases, abnormal operations and other special conditions of the driver in real time, thereby improving the accuracy of driver state monitoring. Through the individual portrait, an individualized normal driving state baseline is established, and through the difference analysis between the normal driving state and the current driving state, the subtle abnormal behavior of the driver can be recognized, the misjudgment caused by the general standard can be avoided, and the long-term and slow driving state change of the driver can be adapted. The difference information, the vehicle driving data and the driver physiological data are combined to calculate the trust score of the driver, which can effectively improve the credibility of the trust score calculation of the driver.

[0079] Please refer to Figure 4 , Figure 4 is another flowchart of a vehicle auxiliary control method provided by an embodiment of the present application.

[0080] S301, obtaining vehicle driving data and driver physiological data.

[0081] This step can refer to S101, which will not be repeated here.

[0082] S302, according to the vehicle driving data and the driver physiological data, the trust degree of the driver is evaluated, and the trust score of the driver is obtained.

[0083] Among them, the trust score represents the driving state of the driver and the reliability of the driver driving the vehicle in the driving state.

[0084] This step can refer to S102, which will not be repeated here.

[0085] S303, the vehicle driving data and the driver physiological data are fused to obtain the fusion feature data.

[0086] For example, the vehicle driving data can include visual image data collected by the vehicle-mounted camera and point cloud data collected by the vehicle-mounted radar. The visual image data, the point cloud data and the driver physiological data can be fused to obtain the fusion feature data.

[0087] Specifically, the step of multi-modal data fusion can include: a. Extract the BEV feature map of the visual image data and the point cloud data respectively.

[0088] For visual image The multi-scale features of the visual image data can be directly extracted using a residual network (ResNet-50, etc.) model:

[0089] Among them, is the BEV feature map of the visual image data, and 256 is the dimension of the BEV feature map of the visual image data. The height, width, respectively, of the visual image data, represent a multi-dimensional entity space.

[0090] For point cloud data The BEV feature map of the point cloud data can be generated by the PointPillars algorithm:

[0091] Among them, is the BEV feature map of the point cloud data, 64 is the dimension of the BEV feature map of the point cloud data, N is the number of data points in the point cloud data, each data point has 4-dimensional attributes, namely three-dimensional coordinates (x, y, z) and reflection intensity, X and Y represent the width and height of the BEV feature map, respectively.

[0092] b. Feature fusion is performed on the BEV feature map of the visual image data and the BEV feature map of the point cloud data to obtain initial fusion data.

[0093] Exemplarily, the cross-attention mechanism can be used to fuse the BEV feature map of the visual image data and the BEV feature map of the point cloud data to obtain the initial fusion data.

[0094] Specifically, in the attention mechanism query-key value pair definition, the BEV feature map of the point cloud data is taken as the query (Query): wherein is a learnable weight, and d is a dimension representing the input feature vector; the BEV feature map of the visual image data is taken as the key value pair (Key-Value): wherein is a learnable weight.

[0095] Then, the attention weight is calculated as:

[0096] Finally, the initial fusion data is output:

[0097] wherein, is a S-shaped function, is a layer normalization function, is the initial fusion data.

[0098] The present application converts the data collected by multiple sensors into BEV feature maps, fuses the BEV feature maps of the visual image data and the BEV feature maps of the point cloud data through the cross-attention mechanism, facilitates more intuitive understanding and analysis of the environment around the vehicle, and can make the vehicle-mounted radar dynamically focus on the visual high-response area (such as the area of pedestrians, other vehicles, obstacles, etc.), thereby making up for the blind area and defects of a single sensor.

[0099] c. The initial fusion data is fused with the driver physiological data to obtain fusion feature data.

[0100] Exemplarily, the initial fusion data can be directly spliced with the driver physiological data to obtain the fusion feature data.

[0101] Exemplarily, the weight factors corresponding to the initial fusion data and the BEV feature map of the point cloud data can also be determined according to the driver physiological data and the visual image data, and the initial fusion data and the BEV feature map of the point cloud data are weighted and calculated according to the weight factors to obtain the final fusion feature data. For example, the expression of the fusion feature data can be:

[0102] wherein, is the fusion feature data, is the initial feature data, is the BEV feature map of the point cloud data, is a weight factor.

[0103] The weight factor The weight factor can be determined according to the driver physiological data and the visual image data, for example, according to the visual image data to judge the field of view clarity under the current weather condition, in combination with the driver physiological data to judge whether the driver heart rate and other data are normal, when the current field of view is poor and the driver heart rate is too fast, the weight of the visual image data is reduced, and the weight of the point cloud data is increased, so as to enhance the robustness of the environmental perception, solve the problem of decreased visual feature effect in dark light, rain and fog and other bad weather, and obtain the fusion feature data reflecting the comprehensive situation inside and outside the vehicle.

[0104] S304, constructing a safety risk evaluation function according to the fusion feature data.

[0105] In the embodiments of the present application, the safety risk of the vehicle is evaluated by constructing a safety risk evaluation function, for example, the risk factors affecting safety can be determined according to the current fusion feature data, such as possible collision, overspeed driving, slippery road and the like, and the safety risk of the current vehicle driving or the safety risk after the vehicle performs a certain action is evaluated or predicted according to these risk factors through the safety risk evaluation function.

[0106] In some possible embodiments, the establishment of the safety risk evaluation function is related to the driving environment of the vehicle, and the driving environment of the vehicle at least includes one of the following information: whether there is an unyielding obstacle in the driving environment, the remaining time of collision between the vehicle and the obstacle, and the lane deviation degree of the vehicle.

[0107] Specifically, the driving environment related data of the vehicle can be determined according to the vehicle driving data in the fusion feature data or the initial fusion data, for example, the fusion result of the visual image data and the point cloud data is included in the fusion feature data, and whether there is an unyielding obstacle in the driving environment can be confirmed through the fusion feature data, when there is an unyielding obstacle, the remaining time of collision between the vehicle and the obstacle is predicted through the distance between the vehicle and the obstacle, the speed, acceleration and other information of the current vehicle. In addition, according to the visual image data, the lane line can also be determined, so as to determine the current lane deviation degree of the vehicle.

[0108] For example, the driving environment related data of the vehicle can be respectively assigned a weight, and the safety risk evaluation function is established by weighted calculation. For example, the expression of the safety risk evaluation function Risk can be:

[0109] wherein, , , are second weights respectively; Time is the remaining time for the vehicle to collide with the obstacle, whether there is an unyielding obstacle in the driving environment, which takes a value of 0 or 1, a value of 0 indicating that there is no obstacle, a value of 1 indicating that there is an obstacle, is the lane deviation degree of the vehicle.

[0110] It can be understood that, in order to ensure the driving safety of the vehicle, the smaller the function value of the safety risk evaluation function, the higher the safety.

[0111] S305, establishing a comfort evaluation function according to the vehicle driving data.

[0112] In the embodiments of the present application, the comfort of the driver when the current vehicle is operated is evaluated, or the comfort of the driver when the vehicle performs a certain action is predicted, by establishing a comfort evaluation function. For example, factors affecting the comfort of the driver, such as sudden acceleration or deceleration, sudden braking, sudden large steering, etc., can be determined through the vehicle driving data, and the comfort evaluation function can be established according to these factors.

[0113] In some possible embodiments, the establishment of the comfort evaluation function is related to the driving state of the vehicle, and the driving state of the vehicle at least includes one of the following information: the speed change rate of acceleration or deceleration of the vehicle and the steering amplitude change rate.

[0114] For example, the comfort evaluation function Comfort may be expressed as:

[0115] wherein, , are third weights respectively, is the speed change rate of acceleration or deceleration of the vehicle, is the steering amplitude change rate of the vehicle.

[0116] It can be understood that, in order to improve the comfort of the driver, the greater the function value of the comfort evaluation function, the higher the comfort.

[0117] S306, determining the level of the trust score according to the interval range to which the trust score belongs.

[0118] Specifically, the trust score is divided into intervals, for example, if the value range of the trust score is between 0 and 1, the trust score can be divided into four intervals: [0, 0.3), [0.3, 0.7), [0.7, 0.9), [0.9, 1], and each interval corresponds to a level. The level to which the trust score belongs can be determined according to the interval range to which the trust score belongs.

[0119] In the embodiments of the present application, the corresponding function calculation targets of different levels are determined according to the safety risk evaluation function and the comfort evaluation function, and different levels correspond to different function calculation targets. For example, if the trust score is in [0, 0.3), it means that the driver's state is critical, such as the driver's attention is distracted, in a sleep state, or the driver's health has extreme uncontrollable factors, at this time the vehicle driving safety needs to be guaranteed the most, at this time the emergency braking is generally needed. If the trust score is in [0.3, 0.7), it means that the driver's state is abnormal, such as inattention, abnormal heartbeat, etc., at this time the safety risk evaluation function can be used as the main function and the comfort evaluation function can be used as the auxiliary function to determine the function calculation target of the corresponding level. If the trust score is in [0.7, 0.9), it means that the road condition is complex or the driver may be distracted, at this time the comfort evaluation function can be used as the main function and the safety risk evaluation function can be used as the auxiliary function to determine the function calculation target of the corresponding level. If the trust score is in [0.9, 1], it means that the road condition is normal and the driver's state is good, at this time the vehicle auxiliary control can not be needed, or only the comfort evaluation function maximum can be used as the function calculation target of the corresponding level to guarantee the comfort experience of the driver.

[0120] S307, the multiple candidate actions corresponding to the level to which the trust score belongs are calculated as the input of the safety risk evaluation function and the comfort evaluation function, and when the calculation result meets the function calculation target corresponding to the level to which the trust score belongs, the auxiliary control strategy required by the driver to drive the vehicle is determined.

[0121] Specifically, after the level to which the trust score belongs, multiple candidate actions corresponding to different levels can be determined, which are actions taken when it is determined that the driver's state is not good and vehicle auxiliary control is needed, such as emergency braking, steering, etc. These candidate actions can be pre-set for selection when the vehicle is controlled. The related data corresponding to these candidate actions are input into the safety risk evaluation function and the comfort evaluation function for calculation, and when the calculation result meets the function calculation target corresponding to the level to which the trust score belongs, the action of the vehicle required for auxiliary control can be determined, and then the auxiliary control strategy required by the driver to drive the vehicle is obtained by combining the prompt information corresponding to the action of the auxiliary control. In the embodiments of the present application, the auxiliary control strategy includes multiple actions of auxiliary control and prompt information corresponding to the actions.

[0122] For example, according to the example of S306, if the trust score is in [0, 0.3), the function calculation target of the corresponding level is the minimum function value of the safety risk evaluation function. Based on the multiple candidate actions corresponding to the level, the corresponding related data of each candidate action is input into the safety risk evaluation function, and if the function value of the safety risk evaluation function of a certain candidate action is the minimum, the candidate action is selected as the action of the required auxiliary control of the current vehicle. According to the prompt information corresponding to the action of the auxiliary control, such as voice prompt, etc., the auxiliary control strategy required by the auxiliary driver to drive the vehicle under the corresponding level is obtained based on the action of the auxiliary control and the corresponding prompt information determined by the action.

[0123] S308, according to the auxiliary control strategy, auxiliary control is performed in the process of driving the vehicle by the driver.

[0124] This step can refer to S104, which will not be described here.

[0125] The present application enhances the robustness of environmental perception by fusing vehicle driving data and driver physiological data, and establishes a safety risk evaluation function and a comfort evaluation function of the vehicle based thereon. The corresponding function calculation target of different levels is determined according to the safety risk evaluation function and the comfort evaluation function, and the candidate action that meets the function calculation target corresponding to the level to which the trust score belongs is selected as the action of the auxiliary control from the multiple candidate actions of the corresponding level according to the level to which the trust score of the driver belongs. When the trust score is low, the safety of driving is prioritized, and when the trust score is high, the comfort of driving is prioritized, and the dynamic balance between safety and comfort is achieved in vehicle auxiliary control.

[0126] Please refer to Figure 5 , Figure 5 Another flowchart of a vehicle auxiliary control method provided by an embodiment of the present application is shown. The vehicle auxiliary control method comprises: S401, obtaining vehicle driving data and driver physiological data.

[0127] This step can refer to S101, which will not be described here.

[0128] S402, performing trust assessment on the driver according to the vehicle driving data and the driver physiological data, and obtaining a trust score of the driver.

[0129] In an embodiment of the present application, the trust score represents the driving state of the driver and the reliability of the driver in driving the vehicle in the driving state.

[0130] This step can refer to S102, which will not be described here.

[0131] S403, establishing a safety risk evaluation function and a comfort evaluation function of the vehicle according to the vehicle driving data and the driver physiological data.

[0132] This step can refer to S303-S305, which will not be repeated here.

[0133] S404, determining the grade to which the trust score belongs according to the interval range to which the trust score belongs.

[0134] This step can refer to S306, which will not be repeated here.

[0135] S405, when it is detected that the trust score belongs to the first grade, determining that the function calculation target corresponding to the first grade is the maximum function value of the comfort evaluation function.

[0136] Specifically, taking the trust score interval division corresponding to S306 as an example, the trust score is divided into four intervals: [0, 0.3), [0.3, 0.7), [0.7-0.9), and [0.9, 1]. The interval to which the trust score of the first grade belongs can be [0.7-0.9), at which time the driver may be distracted but the overall state is stable. The function calculation target corresponding to the first grade can be the maximum function value of the comfort evaluation function.

[0137] S406, calculating the plurality of candidate actions corresponding to the first grade as the input of the comfort evaluation function, and when the calculation result conforms to the maximum function value of the comfort evaluation function, determining an auxiliary control strategy including at least one action among the plurality of candidate actions corresponding to the first grade and the prompt information corresponding to the action according to the calculation result.

[0138] When it is detected that the trust score belongs to the first grade, the plurality of candidate actions corresponding to the first grade are determined first.

[0139] In some possible embodiments, the candidate actions corresponding to the first grade at least include one of the following actions: deceleration, lane change, and parking.

[0140] For example, the candidate actions corresponding to the first grade can also include lane keeping or correction. For example, when the vehicle deviates from the center line of the lane unexpectedly and there is a risk of collision with the adjacent lane, the lane in which the vehicle travels can be appropriately corrected so that the vehicle travels along the center line of the lane as much as possible to ensure driving safety.

[0141] The plurality of candidate actions corresponding to the first grade are input into the comfort evaluation function for calculation, and at least one candidate action when the function value of the comfort evaluation function is the maximum is screened out. The at least one candidate action is taken as the action of vehicle auxiliary control, and the prompt information corresponding to the action of vehicle auxiliary control under the first grade is determined.

[0142] In some possible embodiments, the auxiliary control strategy corresponding to the first level specifically comprises displaying, by a display device of the vehicle, prompt information corresponding to the action.

[0143] Specifically, the auxiliary control strategy corresponding to the first level comprises not only the action of the vehicle auxiliary control, but also prompt information corresponding to the action of the auxiliary control. Since the driver state is relatively stable at this time, visual prompting can be mainly used, that is, prompt information corresponding to the action is displayed by the display device of the vehicle. For example, when the action of the auxiliary control is lane changing, the driver can be prompted to pay attention to avoidance or lane changing through AR-HUD.

[0144] When it is detected that the trust score belongs to the first level, the application determines an action with the maximum comfort function value from candidate actions such as deceleration, lane changing, and parking waiting, and displays prompt information corresponding to the action by the display device of the vehicle. While the vehicle is controlled in an auxiliary manner, the driver can have the best comfort experience, and the driver can be avoided from being disturbed through visual prompting, thereby improving the driving experience of the driver.

[0145] S407, when it is detected that the trust score belongs to the second level, determining that a function calculation target corresponding to the second level is a function value of a balanced safety risk evaluation function and a function value of a comfort evaluation function.

[0146] Specifically, still taking the trust score interval corresponding to S306 as an example, the trust score interval corresponding to the second level can be [0.3, 0.7). At this time, the driver state is abnormal, and it is necessary to take over the vehicle. The function calculation target corresponding to the second level can be the function value of the balanced safety risk evaluation function and the function value of the comfort evaluation function, that is, the safety and the comfort are balanced.

[0147] The double targets of the balanced safety risk evaluation function and the comfort evaluation function can be balanced through a Pareto Optimality algorithm (Pareto Optimality). The set of candidate actions is the action space of the Pareto algorithm. For example, the function calculation target corresponding to the second level can be:

[0148] wherein, represents the function calculation target of the function value of the balanced safety risk evaluation function and the function value of the comfort evaluation function. The smaller the numerical value of the function calculation target is, the better the action is in terms of safety and comfort. is a safety weight, is a comfort weight, , The values of and are both between 0 and 1.

[0149] safety weight , a trust score for the driver, represents the environmental risk calculated from the fusion feature data, such as heavy rain weather, ; , are correction coefficients, Taking a negative sign means that the higher the trust score, the more reliable the driver, at this time is lower, so as to reduce intervention.

[0150] S408, the plurality of candidate actions corresponding to the second level are taken as inputs of the safety risk evaluation function and the comfort evaluation function for calculation, and when the calculation result meets the function calculation target corresponding to the second level, an auxiliary control strategy including at least one action in the plurality of candidate actions corresponding to the second level and prompt information corresponding to the action is determined according to the calculation result.

[0151] Specifically, based on the function calculation target corresponding to the second level set in S407, the function calculation target corresponding to each candidate action in the candidate action set A is calculated. The function calculation target corresponding to each candidate action The at least one candidate action with the smallest function calculation target corresponding to the second level is the action of vehicle auxiliary control, and the prompt information corresponding to the action is determined to obtain the auxiliary control strategy under the second level.

[0152] In some possible embodiments, the candidate action corresponding to the second level at least includes one of the following actions: deceleration and lane change.

[0153] If necessary, the candidate action corresponding to the second level can also include deceleration, lane change and parking, that is, the candidate action set A .

[0154] In some possible embodiments, the auxiliary control strategy corresponding to the second level specifically includes outputting the prompt information corresponding to the action through the voice device of the vehicle and / or outputting vibration reminder through the wearable device worn by the driver.

[0155] Since the state of the driver is abnormal under the second level, at this time, the driver needs to be reminded to pay attention to driving safety through voice reminder, vibration reminder and other ways that are not easy to ignore, so as to maximize the safety of driving. For example, vibration reminder can be output through the smart bracelet worn by the driver.

[0156] When it is detected that the trust score belongs to the second level, the action of determining the function value of the balanced safety risk evaluation function and the function value of the comfort evaluation function from the deceleration, lane changing and waiting candidate actions is performed, and the prompt information corresponding to the action is output through the voice device of the vehicle and / or the vibration reminder is output through the wearable device worn by the driver, so as to remind the driver to pay attention to driving safety in a non-negligible manner while assisting the control of the vehicle, so as to maximize the driving safety.

[0157] For example, Table 1 shows an example of an auxiliary control strategy under different levels. According to the trust score, when the driver is reliable, no auxiliary control of the vehicle is performed; when the driver is distracted, the action of auxiliary control is selected mainly by reminding intervention and braking appropriately; when the driver's state is abnormal, the vehicle is taken over, the action of auxiliary control is selected with the goal of balancing comfort and safety, and braking is performed when necessary; and when the driver's state is critical, the vehicle is forcibly taken over and emergency parking is performed.

[0158] Table 1: Example of auxiliary control strategy under different levels

[0159] By means of Table 1, the graded risk control based on the driver trust score can be realized, the operation risk caused by the decline of the driver's state can be effectively reduced, and the driving safety can be improved.

[0160] S409, according to the auxiliary control strategy, the auxiliary control is performed in the process of driving the vehicle by the driver.

[0161] This step can refer to S104, which will not be described here.

[0162] According to the interval range to which the trust score belongs, the level to which the trust score belongs is determined, different functions are used to calculate the target under different levels, the auxiliary control action that meets the function calculation target under the corresponding level is determined, and different prompt methods are used for the auxiliary control actions corresponding to different levels. The graded risk control based on the driver trust score is realized, the control right is gradually taken over in the process of the decline of the driver trust score, the control strategy is not rigid, the comfort of the vehicle control right takeover is improved, and the auxiliary driving experience of the driver is improved.

[0163] Please refer to Figure 6 , Figure 6 A structural schematic diagram of a vehicle auxiliary control device provided by an embodiment of the present application. The vehicle auxiliary control device comprises: The data acquisition module 501 is configured to acquire vehicle driving data and driver physiological data. The trust score module 502 is configured to perform trust assessment on the driver according to the vehicle driving data and the driver physiological data, and obtain a trust score of the driver; the trust score represents the driving state of the driver and the reliability of the driver driving the vehicle in the driving state; The strategy making module 503 is configured to determine an auxiliary control strategy for assisting the driver in driving the vehicle according to the trust score; the auxiliary control strategy includes a plurality of auxiliary control actions and prompt information corresponding to the actions; The auxiliary control module 504 is configured to perform auxiliary control according to the auxiliary control strategy during the driving of the vehicle by the driver.

[0164] In some possible embodiments, the trust score module 502 includes: The state extraction unit is configured to extract state feature data representing the current driving state of the driver according to the vehicle driving data and the driver physiological data; The difference determination unit is configured to determine difference information between the driving state of the driver and the normal driving state according to the state feature data of the driver and a preset personal portrait of the driver; the personal portrait includes the normal driving state of the driver driving the vehicle in a historical scenario where the trust score is higher than a trust score threshold; The score calculation unit is configured to calculate the vehicle driving data, the driver physiological data and the difference information to obtain the trust score of the driver.

[0165] In some possible embodiments, the vehicle driving data at least includes in-vehicle voice data from the driver and vehicle operation data of the driver driving the vehicle; The state extraction unit is configured to extract behavior feature data related to the driver according to the vehicle driving data including at least the in-vehicle voice data and the vehicle operation data; and extract state feature data representing the current driving state of the driver according to the behavior feature data and the driver physiological data.

[0166] In some possible embodiments, the trust score is negatively correlated with the difference degree between the driving state of the driver and the normal driving state in the difference information.

[0167] In some possible embodiments, the strategy making module 503 includes: The function establishing unit is configured to establish a safety risk evaluation function and a comfort evaluation function of the vehicle according to the vehicle driving data and the driver physiological data; The level determination unit is configured to determine a level to which the trust score belongs according to an interval range to which the trust score belongs; different levels correspond to function calculation targets; The strategy generation unit is configured to take a plurality of candidate actions corresponding to the level to which the trust score belongs as input to calculate a safety risk evaluation function and a comfort evaluation function, and determine an auxiliary control strategy required by the driver to drive the vehicle when the calculation result meets a function calculation target corresponding to the level to which the trust score belongs.

[0168] In some possible embodiments, the safety risk evaluation function is established in relation to a driving environment of the vehicle, and the driving environment of the vehicle at least includes one of the following information: whether there is an unyielding obstacle in the driving environment, a remaining time of collision between the vehicle and the obstacle, and a lane deviation degree of the vehicle.

[0169] In some possible embodiments, the comfort evaluation function is established in relation to a driving state of the vehicle, and the driving state of the vehicle at least includes one of the following information: a speed change rate of acceleration or deceleration of the vehicle, and a steering amplitude change rate.

[0170] In some possible embodiments, the strategy generation unit is configured to, when it is detected that the trust score belongs to the first level, determine that a function calculation target corresponding to the first level is a maximum function value of the comfort evaluation function, take a plurality of candidate actions corresponding to the first level as input to calculate the comfort evaluation function, and determine, when the calculation result meets the maximum function value of the comfort evaluation function, an auxiliary control strategy including at least one action of the plurality of candidate actions corresponding to the first level and prompt information corresponding to the action according to the calculation result.

[0171] In some possible embodiments, the candidate actions corresponding to the first level at least include one of the following actions: deceleration, lane changing, and parking.

[0172] In some possible embodiments, the auxiliary control strategy corresponding to the first level specifically includes displaying the prompt information corresponding to the action through a display device of the vehicle.

[0173] In some possible embodiments, the strategy generation unit is configured to, when it is detected that the trust score belongs to the second level, determine that a function calculation target corresponding to the second level is a balance between a function value of the safety risk evaluation function and a function value of the comfort evaluation function, take a plurality of candidate actions corresponding to the second level as input to calculate the safety risk evaluation function and the comfort evaluation function, and determine, when the calculation result meets the function calculation target corresponding to the second level, an auxiliary control strategy including at least one action of the plurality of candidate actions corresponding to the second level and prompt information corresponding to the action according to the calculation result.

[0174] In some possible embodiments, the candidate actions corresponding to the second level at least include one of the following actions: deceleration and lane changing.

[0175] In some possible embodiments, the second level corresponds to an auxiliary control strategy specifically including outputting prompt information corresponding to the action through a voice device of the vehicle and / or outputting a vibration reminder through a wearable device worn by the driver.

[0176] In some possible embodiments, the function establishing unit is configured to perform multi-modal data fusion on the vehicle driving data and the driver physiological data to obtain fusion feature data, construct a safety risk evaluation function according to the fusion feature data, and construct a comfort evaluation function according to the vehicle driving data.

[0177] The present application can effectively cope with the physiological abnormal conditions of the driver by comprehensively analyzing the driving state of the driver by combining multi-source data such as vehicle driving data and driver physiological data, avoid the limitations of a single data source, and convert the reliability of the driver into a quantifiable trust score, thereby providing an objective basis for subsequent decision-making and reducing the deviation of subjective judgment. The present application can determine the auxiliary control strategy of the vehicle by the trust score of the driver, dynamically adjust the vehicle auxiliary control strategy based on the trust score of the driver, thereby realizing dynamic allocation of the control right of the vehicle, and greatly improving the operation experience of the driver. Finally, the present application realizes real-time auxiliary control of the vehicle through the auxiliary control strategy during driving, effectively reduces the operation risk caused by the poor state of the driver, and improves the driving safety.

[0178] Please refer to Figure 7 , Figure 7 FIG. 1 shows a structural schematic diagram of an electronic device provided by an embodiment of the present application.

[0179] As Figure 7 shown, the electronic device 600 can include at least one processor 601, at least one network interface 604, a user interface 603, a memory 605, a touch screen 606, and at least one communication bus 602.

[0180] The communication bus 602 can be used to realize the connection and communication of the above-mentioned components.

[0181] The user interface 603 can include a key, and the optional user interface can further include a standard wired interface, a wireless interface.

[0182] The network interface 604 can optionally include a Bluetooth module, an NFC module, a Wi-Fi module, etc.

[0183] The processor 601 can include one or more processing cores. The processor 601 connects various parts in the entire electronic device 600 by various interfaces and lines, executes various functions of the routing device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 605, and calling data stored in the memory 605. Alternatively, the processor 601 can be implemented in at least one of a hardware form of a DSP, an FPGA, and a PLA. The processor 601 can integrate a combination of one or more of a CPU, a GPU, and a modem. Among them, the CPU mainly processes operating systems, user interfaces, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 601, but can be implemented by a separate chip.

[0184] The memory 605 can include a RAM and can also include a ROM. Alternatively, the memory 605 includes a non-transitory computer readable medium. The memory 605 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 605 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area can store data involved in the above-mentioned various method embodiments, etc. The memory 605 can also be at least one storage device located away from the aforementioned processor 601. As shown in the figure, the memory 605 as a computer storage medium can include an operating system, a network communication module, a user interface module, and a vehicle auxiliary control application program. Figure 7 As shown in the figure, the memory 605 as a computer storage medium can include an operating system, a network communication module, a user interface module, and a vehicle auxiliary control application program.

[0185] Specifically, the processor 601 can be used to call the vehicle auxiliary control application program stored in the memory 605, and specifically perform the following operations: Obtain vehicle driving data and driver physiological data; According to the vehicle driving data and the driver physiological data, the trust degree of the driver is evaluated to obtain a trust score of the driver; wherein the trust score represents the driving state of the driver and the reliability of the driver driving the vehicle in the driving state; According to the trust score, an auxiliary control strategy for assisting the driver to drive the vehicle is determined; wherein the auxiliary control strategy includes a plurality of auxiliary control actions and prompt information corresponding to the actions; According to the auxiliary control strategy, the auxiliary control is performed in the process of the driver driving the vehicle.

[0186] In some possible embodiments, the processor 601 performs a trust evaluation on the driver according to the vehicle driving data and the driver physiological data, and obtains a trust score of the driver, specifically performs: extracting state feature data representing a current driving state of the driver according to the vehicle driving data and the driver physiological data; determining difference information between the driving state of the driver and a normal driving state according to the state feature data of the driver and a preset personal portrait of the driver; the personal portrait includes a normal driving state of the driver driving the vehicle in a historical scenario with a trust score higher than a trust score threshold; calculating the vehicle driving data, the driver physiological data and the difference information to obtain the trust score of the driver.

[0187] In some possible embodiments, the vehicle driving data at least includes in-vehicle voice data from the driver and vehicle operation data of the driver driving the vehicle; The processor 601 extracts state feature data representing a current driving state of the driver according to the vehicle driving data and the driver physiological data, specifically performs: extracting behavior feature data related to the driver according to the vehicle driving data at least including the in-vehicle voice data and the vehicle operation data; extracting state feature data representing a current driving state of the driver according to the behavior feature data and the driver physiological data.

[0188] In some possible embodiments, the trust score is negatively correlated with a difference degree between the driving state of the driver and the normal driving state in the difference information.

[0189] In some possible embodiments, the processor 601 performs an auxiliary control strategy for assisting the driver in driving the vehicle according to the trust score, specifically performs: establishing a safety risk evaluation function and a comfort evaluation function of the vehicle according to the vehicle driving data and the driver physiological data; determining a trust score belonging grade of the trust score according to an interval range to which the trust score belongs; different grades correspond to function calculation targets; taking a plurality of candidate actions corresponding to the trust score belonging grade of the trust score as inputs of the safety risk evaluation function and the comfort evaluation function for calculation, and determining an auxiliary control strategy required for assisting the driver in driving the vehicle when a calculation result meets a function calculation target corresponding to the trust score belonging grade of the trust score.

[0190] In some possible embodiments, the establishment of the safety risk evaluation function is related to a driving environment of the vehicle, and the driving environment of the vehicle at least includes one of the following information: whether there is an unyielding obstacle in the driving environment, a remaining time of collision between the vehicle and the obstacle, and a lane deviation degree of the vehicle.

[0191] In some possible embodiments, the establishment of the comfort evaluation function is related to a driving state of the vehicle, and the driving state of the vehicle at least includes one of the following information: a speed change rate of acceleration or deceleration of the vehicle and a steering amplitude change rate.

[0192] In some possible embodiments, the processor 601 performs calculation on the multiple candidate actions corresponding to the level to which the trust score belongs as input of the safety risk evaluation function and the comfort evaluation function, and when the calculation result meets the function calculation target corresponding to the level to which the trust score belongs, determines an auxiliary control strategy required for assisting the driver to drive the vehicle, and specifically performs: When it is detected that the trust score belongs to the first level, it is determined that the function calculation target corresponding to the first level is that the function value of the comfort evaluation function is maximum. The multiple candidate actions corresponding to the first level are calculated as input of the comfort evaluation function, and when the calculation result meets the function value maximum of the comfort evaluation function, the auxiliary control strategy including at least one action of the multiple candidate actions corresponding to the first level and the prompt information corresponding to the action is determined according to the calculation result.

[0193] In some possible embodiments, the candidate action corresponding to the first level at least includes one of the following actions: deceleration, lane changing and parking.

[0194] In some possible embodiments, the auxiliary control strategy corresponding to the first level specifically includes displaying the prompt information corresponding to the action through a display device of the vehicle.

[0195] In some possible embodiments, the processor 601 performs calculation on the multiple candidate actions corresponding to the level to which the trust score belongs as input of the safety risk evaluation function and the comfort evaluation function, and when the calculation result meets the function calculation target corresponding to the level to which the trust score belongs, determines an auxiliary control strategy required for assisting the driver to drive the vehicle, and specifically performs: When it is detected that the trust score belongs to the second level, it is determined that the function calculation target corresponding to the second level is that the function value of the safety risk evaluation function and the function value of the comfort evaluation function are balanced. The multiple candidate actions corresponding to the second level are calculated as input of the safety risk evaluation function and the comfort evaluation function, and when the calculation result meets the function calculation target corresponding to the second level, the auxiliary control strategy including at least one action of the multiple candidate actions corresponding to the second level and the prompt information corresponding to the action is determined according to the calculation result.

[0196] In some possible embodiments, the candidate action corresponding to the second level at least includes one of the following actions: deceleration and lane changing.

[0197] In some possible embodiments, the second level corresponds to the auxiliary control strategy specifically including outputting prompt information corresponding to the action through a voice device of the vehicle and / or outputting a vibration reminder through a wearable device worn by the driver.

[0198] In some possible embodiments, the processor 601 performs establishment of a safety risk evaluation function and a comfort evaluation function of the vehicle according to the vehicle driving data and the driver physiological data, specifically performs: performing multi-modal data fusion on the vehicle driving data and the driver physiological data to obtain fusion feature data; constructing the safety risk evaluation function according to the fusion feature data; establishing the comfort evaluation function according to the vehicle driving data.

[0199] The present application can effectively cope with the physiological abnormal conditions of the driver by comprehensively analyzing the driving state of the driver by combining the vehicle driving data and the driver physiological data and the like, can avoid the limitation of a single data source, and can convert the reliability of the driver into a quantifiable trust score, thereby providing an objective basis for subsequent decision-making and reducing the deviation of subjective judgment. The present application can determine the auxiliary control strategy of the vehicle according to the trust score of the driver, can realize dynamic adjustment of the auxiliary control strategy of the vehicle based on the trust score of the driver, thereby realizing dynamic allocation of the control right of the vehicle, and can greatly improve the operation experience of the driver. Finally, the present application can realize real-time auxiliary control of the vehicle by the auxiliary control strategy during driving, can effectively reduce the operation risk caused by the poor state of the driver, and can improve the driving safety.

[0200] The present application also provides a computer readable storage medium, which stores instructions, when the instructions are run on a computer or a processor, the computer or the processor executes one or more steps in any of the above method embodiments. The constituent modules of the above electronic device, if realized in the form of a software function unit and sold or used as an independent product, can be stored in the computer readable storage medium.

[0201] In the above embodiments, all or part of the methods can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the methods can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in or transmitted by a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, Digital Subscriber Line, DSL) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available media can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a Digital Versatile Disc, DVD), or a semiconductor medium (such as a Solid State Disk, SSD), etc.

[0202] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by a computer program instructing related hardware, which can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. The storage medium includes ROM, RAM, magnetic or optical disks, and various program code storage media. In the case of no conflict, the technical features in the embodiments and the embodiments can be combined arbitrarily.

[0203] The above-described embodiments are merely preferred embodiments of the present application, and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements of the technical solutions of the present application made by a person of ordinary skill in the art shall fall within the protection scope of the claims of the present application.

Claims

1. A vehicle assist control method characterized by, The method comprises: acquiring vehicle driving data and driver physiological data; conducting trust assessment on the driver according to the vehicle driving data and the driver physiological data, to obtain a trust score of the driver; wherein the trust score represents a driving state of the driver and a reliability degree of the driver driving the vehicle in the driving state; determining an auxiliary control strategy for assisting the driver in driving the vehicle according to the trust score; wherein the auxiliary control strategy comprises a plurality of auxiliary control actions and prompt information corresponding to the actions; conducting auxiliary control according to the auxiliary control strategy in the process of the driver driving the vehicle.

2. The vehicle assist control method according to claim 1, characterized by The method comprises: extracting state feature data representing the driving state currently assumed by the driver according to the vehicle driving data and the driver physiological data; determining difference information between the driving state of the driver and a normal driving state of the driver according to the state feature data of the driver and a preset personal portrait of the driver; wherein the personal portrait comprises a normal driving state of the driver driving the vehicle in a historical scenario with a trust score higher than a trust score threshold; calculating the vehicle driving data, the driver physiological data and the difference information to obtain the trust score of the driver.

3. The vehicle assist control method according to claim 2, characterized by The vehicle driving data at least comprises in-vehicle voice data from the driver and vehicle operation data of the driver driving the vehicle; The method comprises: extracting behavior feature data related to the driver according to the vehicle driving data at least comprising the in-vehicle voice data and the vehicle operation data; extracting state feature data representing the driving state currently assumed by the driver according to the behavior feature data and the driver physiological data.

4. The vehicle assist control method according to claim 2, characterized by The trust score is negatively correlated with the difference degree between the driving state of the driver and the normal driving state in the difference information.

5. The vehicle assist control method according to claim 1, characterized by The method comprises: establishing a safety risk evaluation function and a comfort evaluation function of the vehicle according to the vehicle driving data and the driver physiological data; determining a grade to which the trust score belongs according to the interval range to which the trust score belongs; wherein different grades correspond to function calculation targets; taking a plurality of candidate actions corresponding to the grade to which the trust score belongs as inputs of the safety risk evaluation function and the comfort evaluation function to perform calculation, and determining the auxiliary control strategy required for assisting the driver in driving the vehicle when the calculation result meets the function calculation target corresponding to the grade to which the trust score belongs.

6. The vehicle assist control method according to claim 5, characterized by The safety risk evaluation function is established in relation to a driving environment of the vehicle, and the driving environment of the vehicle at least includes one of the following information: whether there is an unyielding obstacle in the driving environment, a remaining time of collision between the vehicle and the obstacle, and a lane deviation degree of the vehicle.

7. The vehicle assist control method according to claim 5, characterized by The comfort evaluation function is established in relation to a driving state of the vehicle, and the driving state of the vehicle at least includes one of the following information: a speed change rate of acceleration or deceleration of the vehicle, and a steering amplitude change rate.

8. A vehicle assist control device characterized by comprising: The device comprises: a data acquisition module configured to acquire vehicle driving data and driver physiological data; a trust score module configured to evaluate a trust degree of the driver according to the vehicle driving data and the driver physiological data, and obtain a trust score of the driver; wherein the trust score represents a driving state of the driver and a reliable degree of the driver driving the vehicle in the driving state; a strategy making module configured to determine an auxiliary control strategy for assisting the driver in driving the vehicle according to the trust score; wherein the auxiliary control strategy includes a plurality of auxiliary control actions and prompt information corresponding to the actions; an auxiliary control module configured to perform auxiliary control according to the auxiliary control strategy in the process of the driver driving the vehicle.

9. An electronic device, comprising: The computer readable storage medium stores a computer program, and the processor executes the computer program to perform the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the processor executes the computer program to perform the method of any one of claims 1-7.

Images