Automatic driving vehicle safety control method and device based on multi-modal data

Abstract

The invention discloses an automatic driving vehicle safety control method and device based on multi-modal data. The method comprises the steps that first driving information including first position information and first speed information, 3D point cloud data including a driving environment and video stream driving environment information including the driving environment in the driving process of a target driving vehicle are obtained; determining feature information of each first obstacle in at least one first obstacle in the driving environment based on the driving environment information, wherein the feature information comprises second position information and second speed information; the first driving information and the feature information of each first obstacle are utilized to determine the collision time of the target driving vehicle and each first obstacle, and at least one collision time corresponding to at least one first obstacle is obtained; and predicting whether the driver needs to take over the target driving vehicle according to the at least one collision time. And the prediction accuracy of predicting whether the driver needs to take over the vehicle driving in a complex urban road scene is improved.

Description

Technical Field

[0001] The present application belongs to the field of vehicle takeover technology, and in particular, relates to a method and device for safety control of an autonomous driving vehicle based on multimodal data. Background Art

[0002] At present, most vehicles have the function of automatic driving. However, when encountering some emergencies during driving, it may be necessary to predict whether the driver needs to take over the driving of the vehicle, so that when the vehicle encounters emergencies that cannot be solved by the automatic driving of the vehicle, the driver can be reminded in time to take over the driving of the vehicle to solve the emergencies encountered by the vehicle during driving. However, the existing technology on the driver's takeover behavior focuses on highways and simple road scenes, and lacks research on predicting whether the driver needs to take over the driving of the vehicle in complex urban road scenes. Therefore, the prediction accuracy of whether the driver needs to take over the driving of the vehicle in complex urban road scenes is low. Summary of the invention

[0003] The embodiment of the present application provides an implementation scheme different from the prior art to solve the technical problem of low prediction accuracy in predicting whether the driver needs to take over vehicle driving.

[0004] In a first aspect, the present application provides a method for safety control of an autonomous driving vehicle based on multimodal data, comprising: obtaining first driving information and driving environment information of a target vehicle during driving, the first driving information comprising: first position information and first speed information; the driving environment information comprising: 3D point cloud data of the driving environment, and a video stream of the driving environment; determining characteristic information of each first obstacle in the driving environment based on the driving environment information, the characteristic information comprising second position information and second speed information; determining a collision time between the target vehicle and each first obstacle using the first driving information and the characteristic information of each first obstacle, and obtaining at least one collision time corresponding to the at least one first obstacle; and predicting whether a driver needs to take over driving the target vehicle based on the at least one collision time.

[0005] In the second aspect, the present application provides an autonomous driving vehicle safety control device based on multimodal data, including: an acquisition unit, used to acquire first driving information and driving environment information of a target vehicle during driving, the first driving information including: first position information and first speed information; the driving environment information including: 3D point cloud data of the driving environment, and a video stream of the driving environment; a determination unit, used to determine characteristic information of each first obstacle in the driving environment based on the driving environment information, the characteristic information including second position information and second speed information; the determination unit is also used to determine the collision time between the target vehicle and each first obstacle using the first driving information and the characteristic information of each first obstacle, and obtain at least one collision time corresponding to the at least one first obstacle; a prediction unit, used to predict whether the driver needs to take over driving the target vehicle based on the at least one collision time.

[0006] In a third aspect, the present application provides an electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any method in the first aspect or any possible implementation manner of the first aspect by executing the executable instructions.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program implements any method in the first aspect or any possible implementation manner of the first aspect.

[0008] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in the first aspect or any possible implementation manner of the first aspect.

[0009] The present application provides a method for obtaining first driving information and driving environment information of a target driving vehicle during driving, wherein the first driving information includes: first position information and first speed information; the driving environment information includes: 3D point cloud data of the driving environment, and a video stream of the driving environment; based on the driving environment information, characteristic information of each first obstacle in the driving environment is determined, and the characteristic information includes second position information and second speed information; the first driving information and the characteristic information of each first obstacle are used to determine the collision time between the target driving vehicle and each first obstacle, and at least one collision time corresponding to the at least one first obstacle is obtained; and a scheme for predicting whether the driver needs to take over driving the target driving vehicle is provided according to the at least one collision time. At least one first obstacle is determined by the acquired driving environment information, and the collision time between each first obstacle and the target driving vehicle is predicted, thereby obtaining at least one collision time, and predicting whether the driver needs to take over driving the target driving vehicle based on at least one collision time, thereby improving the prediction accuracy of whether the driver needs to take over driving the vehicle in complex urban road scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0011] Figure 1 A flowchart of a method for controlling the safety of an autonomous driving vehicle based on multimodal data provided in one embodiment of the present application;

[0012] Figure 2 A schematic diagram of the structure of a multi-modal data-based autonomous driving vehicle safety control device provided in one embodiment of the present application;

[0013] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0014] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, but cannot be understood as limiting the present application.

[0015] The terms "first" and "second" etc. in the specification, claims and drawings of the embodiments of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0016] First, some terms in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0017] Single variable analysis method: also known as univariate analysis or univariate analysis, is an analysis method in statistics that focuses on describing and analyzing one variable without considering the impact of other variables.

[0018] 3D point cloud data: a data set consisting of a large number of points in three-dimensional space, which usually represent the shape of the external environment or objects. Point cloud data is widely used in various fields, such as surveying and mapping, construction, manufacturing, self-driving cars, cultural heritage protection, etc.

[0019] At present, most vehicles have the function of automatic driving. However, when encountering some emergencies during driving, it may be necessary to predict whether the driver needs to take over the driving of the vehicle, so that when the vehicle encounters emergencies that the automatic driving of the vehicle cannot solve, the driver can be reminded in time to take over the driving of the vehicle to solve the emergencies encountered by the vehicle during driving. However, the existing technology on the driver's takeover behavior is mostly focused on highways and simple road scenes, and there is a lack of research on predicting whether the driver needs to take over the driving of the vehicle in complex urban road scenes. For example, there is a lack of research on predicting whether the driver needs to take over the driving of the vehicle in complex urban road scenes such as signal intersections and illegal driving of non-motor vehicles. Therefore, the prediction accuracy of whether the driver needs to take over the driving of the vehicle in complex urban road scenes is low.

[0020] In addition, when it is predicted that the driver needs to take over the driving of the vehicle, a prompt message needs to be issued to prompt the driver to take over the driving of the vehicle. In the prior art, the prompt message is mostly displayed through a single first sound prompt method or a first visual prompt method. The prompt effect of a single prompt method is not good. For example, when the driver is distracted, it is easy for him to not hear or see the prompt message, resulting in the driver's untimely takeover and accidents. Although there are methods in the prior art for judging the driver's ability to take over the driving of the vehicle based on the driver's physiological data and behavioral data, the driver's physiological data and behavioral data are analyzed using a single variable analysis method to analyze the driver's ability to take over the driving of the vehicle. The single variable analysis method is equivalent to only being able to analyze the driver's takeover ability based on one type of data, which makes the accuracy of the analysis result low.

[0021] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0022] Figure 1 A flowchart of a method for controlling the safety of an autonomous driving vehicle based on multimodal data is provided as an exemplary embodiment of the present application. The execution subject of the method may be a server. The method includes at least the following steps S101-S104:

[0023] S101, obtaining first driving information and driving environment information of a target driving vehicle during driving, wherein the first driving information includes: first position information and first speed information; the driving environment information includes: 3D point cloud data of the driving environment, and a video stream of the driving environment;

[0024] Optionally, the first speed information refers to the driving speed of the target driving vehicle at the position corresponding to the first position information.

[0025] Optionally, the driving environment information is driving environment information around the target driving vehicle when the target driving vehicle is at a position corresponding to the first position information.

[0026] Optionally, 3D point cloud data may be acquired through a laser radar, and the acquired 3D point cloud data may be uploaded to a server.

[0027] Optionally, a video stream may be acquired through a camera, and the acquired video stream may be uploaded to a server.

[0028] Optionally, the 3D point cloud data and video stream can also be collected by the vehicle-mounted environment perception system and uploaded to the server.

[0029] Optionally, there may be multiple laser radars and multiple cameras.

[0030] S102, determining characteristic information of each first obstacle in the at least one first obstacle in the driving environment based on the driving environment information, the characteristic information including second position information and second speed information;

[0031] Optionally, the first obstacle refers to an object that may hinder the travel of the target vehicle, and the object may be a motor vehicle, a pedestrian, a non-motor vehicle and a fixed obstacle, wherein the fixed obstacle may be a roadblock, a stone pier or other obstacles that are not movable.

[0032] Optionally, the second speed information refers to the driving speed or walking speed of the first obstacle at a position corresponding to the second position information, wherein if the first obstacle is a roadblock, the second speed information is 0.

[0033] Optionally, in the aforementioned S102, determining the characteristic information of each first obstacle in the at least one first obstacle in the driving environment based on the driving environment information includes the following steps S1021-S1024:

[0034] S1021. Determine feature information of each second obstacle in at least one second obstacle in the driving environment based on the 3D point cloud data;

[0035] S1022. Determine feature information of each third obstacle in at least one third obstacle in the driving environment based on the video stream;

[0036] S1023: performing fusion processing on feature information of at least one fourth obstacle that is repeated between the at least one second obstacle and the at least one third obstacle to obtain target feature information of the repeated obstacle;

[0037] Optionally, if there is an obstacle with the same position information between the at least one second obstacle and the at least one third obstacle, it is deemed that there is at least one repeated fourth obstacle between the at least one second obstacle and the at least one third obstacle.

[0038] Optionally, the step of fusing the feature information of at least one fourth obstacle that is repeated between the at least one second obstacle and the at least one third obstacle in the aforementioned S1023 to obtain the target feature information of the repeated obstacle includes S10231-S10232:

[0039] S10231. Acquire at least one fourth obstacle having the same position information as the at least one second obstacle and the at least one third obstacle;

[0040] S10232. For each fourth obstacle in the at least one fourth obstacle, based on a preset data fusion formula, fuse the speed information of the fourth obstacle in the at least one second obstacle and the speed information of the fourth obstacle in the at least one third obstacle to obtain target speed information of the fourth obstacle, and use the target speed information and the position information of the fourth obstacle as target feature information of the fourth obstacle, so as to obtain target feature information of at least one fourth obstacle.

[0041] Optionally, the speed information of the fourth obstacle in the at least one second obstacle and the speed information of the fourth obstacle in the at least one third obstacle are fused based on a preset speed data fusion formula to obtain the target speed information of the fourth obstacle, which can be expressed as the following formula:

[0042] S=αS LiDAR +(1-α)S Camera

[0043] Among them, S is the target speed information, S LiDAR S is the speed information of the fourth obstacle in the at least one second obstacle; Camera is the speed information of the fourth obstacle in the at least one third obstacle, and α is a preset weight value.

[0044] S1024. Use the remaining second obstacles in the at least one second obstacle except the at least one fourth obstacle, the remaining third obstacles in the at least one third obstacle except the at least one fourth obstacle, and each obstacle in the at least one fourth obstacle as first obstacles, and further obtain characteristic information of each first obstacle in the at least one first obstacle.

[0045] Optionally, in the aforementioned S102, determining the characteristic information of each first obstacle in the at least one first obstacle in the driving environment based on the driving environment information includes the following steps S2021-S2025:

[0046] S2021. Determine, based on the 3D point cloud data, feature information of each second obstacle in at least one second obstacle in the driving environment and appearance information of each second obstacle;

[0047] S2022. Determine, based on the video stream, feature information of each third obstacle among at least one third obstacle in the driving environment and appearance information of each third obstacle;

[0048] S2023: Determine whether there is at least one repeated fourth obstacle between the at least one second obstacle and the at least one third obstacle based on the appearance information of the at least one second obstacle and the appearance information of the at least one third obstacle;

[0049] Optionally, if there is an obstacle having the same appearance information between the at least one second obstacle and the at least one third obstacle, it is deemed that there is at least one repeated fourth obstacle between the at least one second obstacle and the at least one third obstacle.

[0050] S2024: If yes, perform fusion processing on the feature information of at least one fourth obstacle that is repeated between the at least one second obstacle and the at least one third obstacle to obtain target feature information of the repeated obstacle;

[0051] Optionally, the step of fusing the feature information of at least one fourth obstacle that is repeated between the at least one second obstacle and the at least one third obstacle in the aforementioned S2024 to obtain the target feature information of the repeated obstacle includes S20231-S20232:

[0052] S20231. Acquire at least one fourth obstacle having the same appearance information as the at least one second obstacle and the at least one third obstacle;

[0053] S20232. For each fourth obstacle in the at least one fourth obstacle, fuse the feature information of the fourth obstacle in the at least one second obstacle and the feature information of the fourth obstacle in the at least one third obstacle based on a preset data fusion formula to obtain target feature information of the fourth obstacle, and further obtain target feature information of at least one fourth obstacle.

[0054] Optionally, in the aforementioned S20232, the feature information of the fourth obstacle in the at least one second obstacle and the feature information of the fourth obstacle in the at least one third obstacle are fused based on a preset data fusion formula to obtain the target feature information of the fourth obstacle, including the following steps S2000-S2002:

[0055] S2000, fusing the speed information of the fourth obstacle in the at least one second obstacle and the speed information of the fourth obstacle in the at least one third obstacle based on a preset speed data fusion formula to obtain target speed information of the fourth obstacle;

[0056] Optionally, the speed information of the fourth obstacle in the at least one second obstacle and the speed information of the fourth obstacle in the at least one third obstacle are fused based on a preset speed data fusion formula to obtain the target speed information of the fourth obstacle, which can be expressed as the following formula:

[0057] S=αS LiDAR +(1-α)S Camera

[0058] Among them, S is the target speed information, S LiDAR S is the speed information of the fourth obstacle in the at least one second obstacle; Camera is the speed information of the fourth obstacle in the at least one third obstacle, and α is a preset weight value.

[0059] S2001, fusing the position information of the fourth obstacle in the at least one second obstacle and the position information of the fourth obstacle in the at least one third obstacle based on a preset position data fusion formula to obtain target position information of the fourth obstacle; wherein the data fusion formula includes a speed data fusion formula and a position data fusion formula;

[0060] Optionally, the position information of the fourth obstacle in the at least one second obstacle and the position information of the fourth obstacle in the at least one third obstacle are fused based on a preset data fusion formula to obtain the target position information of the fourth obstacle, which can be expressed as the following formula:

[0061] X=αX LiDAR +(1-α)X Camera

[0062] Among them, X is the target location information, X LiDAR is the position information of the fourth obstacle in the at least one second obstacle; Camera is the position information of the fourth obstacle among the at least one third obstacle, and α is the aforementioned preset weight value.

[0063] S2002, using the target position information and the target speed information as target feature information of the fourth obstacle;

[0064] S2025. Take the remaining second obstacles in the at least one second obstacle except the at least one fourth obstacle, the remaining third obstacles in the at least one third obstacle except the at least one fourth obstacle, and each obstacle in the at least one fourth obstacle as first obstacles, and further obtain characteristic information of each first obstacle in the at least one first obstacle.

[0065] In summary, by fusing the characteristic information of the same obstacle, the position information and speed information of the obstacle are ensured to be more consistent and reliable, thereby improving the accuracy of the subsequent prediction of the collision time based on the obstacle characteristic information.

[0066] S103, determining a collision time between the target vehicle and each of the first obstacles by using the first driving information and the characteristic information of each of the first obstacles, and obtaining at least one collision time corresponding to the at least one first obstacle;

[0067] Optionally, the collision time refers to the collision duration.

[0068] Optionally, in the aforementioned S103, determining the collision time between the target driving vehicle and each first obstacle by using the first driving information and the characteristic information of each first obstacle includes the following steps S1031-S1032:

[0069] S1031, estimating the trajectory information of each first obstacle using a preset prediction model;

[0070] Optionally, the prediction model may be referred to in the prior art, which will not be described in detail here.

[0071] S1032: Determine a collision time between the target driving vehicle and each of the first obstacles based on the trajectory information of each of the first obstacles, the first driving information, and the characteristic information of each of the first obstacles.

[0072] Optionally, the determining of the collision time between the target driving vehicle and each first obstacle based on the trajectory information of each first obstacle, the first driving information and the characteristic information of each first obstacle in S1032 includes the following steps S10321-S10322:

[0073] S10321, determining an estimated distance between the target traveling vehicle and each first obstacle based on the trajectory information of each first obstacle, the first position information, and the second position information of each first obstacle;

[0074] S10322: Determine a collision time between the target moving vehicle and each of the first obstacles based on the estimated distance between the target moving vehicle and each of the first obstacles, the speed information of each of the first obstacles, and the first speed information.

[0075] Optionally, the determining of the collision time between the target moving vehicle and each first obstacle based on the estimated distance between the target moving vehicle and each first obstacle, the speed information of each first obstacle and the first speed information can be expressed as the following formula:

[0076]

[0077] Wherein, TTC is the collision time between the target vehicle and each first obstacle; obj V is the estimated distance between the target vehicle and each first obstacle; ego is the first speed information; V obj is the speed information of each first obstacle.

[0078] S104: Predict whether a driver needs to take over driving the target vehicle according to the at least one collision time.

[0079] Optionally, in the aforementioned S104, the predicting whether the driver needs to take over driving the target vehicle according to the at least one collision time includes the following steps S1041-S1042:

[0080] S1041, obtaining the shortest collision time among the at least one collision time;

[0081] S1042: When the shortest collision time is less than a preset threshold, it is determined that the driver needs to take over driving the target vehicle.

[0082] It should be understood that if the vehicle is located on a complex urban road and the shortest collision time is less than a preset threshold, the vehicle's automatic driving function is very likely to be unable to cope with possible collision situations. Therefore, in order to avoid danger, the driver needs to take over driving the target vehicle.

[0083] Optionally, when the shortest collision time is less than a preset threshold, the method further includes the following steps S01-S04:

[0084] S01. Acquire first state information of the driver, the first state information comprising: first physiological data and first eye data, wherein the first physiological data comprises: heart rate, heart rate variability and skin conductance, and the first eye data comprises: pupil diameter, blinking frequency and fixation duration;

[0085] Optionally, the first physiological data of the driver may be collected by a physiological monitoring device, and the first physiological data may be uploaded to a server. For example, the driver's heart rate, heart rate variability, and skin conductance may be collected by a heart rate belt, and the collected heart rate, heart rate variability, and skin conductance may be uploaded to a server.

[0086] Optionally, the physiological monitoring device may be located near the driver's seat of the target moving vehicle, or may be worn on the driver.

[0087] Optionally, the driver's first eye data may be collected by an eye tracker and uploaded to a server.

[0088] Optionally, the eye tracker can be installed near the steering wheel or instrument panel to collect the driver's first eye data.

[0089] S02, determining the driver's concentration level based on the characteristic information of the target obstacle corresponding to the shortest collision time among the at least one first obstacle, the driving information, and the first state information, wherein the concentration level includes: concentration, distraction, or lack of attention;

[0090] Optionally, the concentration indicates that the driver's attention level is good and the psychological pressure is small; the distraction indicates that the driver's attention level is poor or the psychological pressure is large; the lack of attention indicates that the driver's attention level is poor and the psychological pressure is large.

[0091] Optionally, in the aforementioned S02, determining the concentration level of the driver based on the characteristic information of the target obstacle corresponding to the shortest collision time among the at least one first obstacle, the driving information, and the first state information includes S021-S023:

[0092] S021. Based on the feature information of the target obstacle corresponding to the shortest collision time among the at least one first obstacle, the driving information, and the first state information, an input feature vector is formed;

[0093] S022. Processing the input feature vector using a preset classifier formula to obtain a focus value;

[0094] Optionally, the input feature vector is processed using a preset classifier formula to obtain a focus value, which can be expressed as the following formula:

[0095] f(x)=sign(w T x+b)

[0096] Among them, f(x) is the focus value, x is the input feature vector, and w and b are classifier parameters.

[0097] S023. Determine the concentration level of the driver based on the concentration value.

[0098] If f(x)=1, the driver's concentration level is focused; if f(x)=0, the driver's concentration level is distracted; if f(x)=-1, the driver's concentration level is not focused.

[0099] S03, determining a corresponding first prompting method based on the concentration level;

[0100] Optionally, determining the corresponding first prompt method based on the concentration level includes: determining the corresponding first prompt method based on a preset prompt rule and the concentration level.

[0101] Optionally, the preset prompt rules include: if the driver's concentration level is "focused", a standard prompt method is used; if the driver's concentration level is "distracted", a multimodal prompt method is used; if the driver's concentration level is "unfocused", an emergency prompt method is used.

[0102] Optionally, the standard prompting method may be a first sound prompting method or a first tactile prompting method.

[0103] Optionally, the multimodal prompting method includes a combination of at least two of the following prompting methods: a first sound prompting method, a first tactile prompting method, and a first visual prompting method.

[0104] Optionally, the emergency prompt method is a first sound prompt or a first visual prompt, which reduces the content of the prompt information, extends the prompt duration of the first sound prompt or the first visual prompt, and / or repeats the prompt information through the first sound prompt or the first visual prompt multiple times, retaining some automatic driving functions to execute some basic functions.

[0105] Optionally, the prompt information displayed or broadcasted in an emergency prompting manner only includes the purpose and reason for taking over when prompting the driver to take over driving, such as "There is an obstacle ahead, please slow down."

[0106] In summary, when the driver's concentration level is "not paying attention", it means that the driver's attention level is poor and the psychological pressure is high. Too many prompts conveyed by multiple prompts may cause greater pressure on the driver. Therefore, this solution uses a single prompt method to prompt the driver. At the same time, in order to reduce the driver's pressure and reduce the amount of information that the driver needs to process, the prompt information needs to be simplified, and the driver is concisely prompted to take over the purpose and reason of taking over the driving target vehicle, so that the driver can quickly perform the corresponding operation to achieve the purpose of taking over the driving target vehicle. In order to reduce the driver's pressure and driving operation, in the early stage of the driver taking over the driving, the automatic driving function will not be completely stopped, and some basic functions will still be performed, such as lane keeping, vehicle deceleration and other basic functions, so that the driver can focus on the most important operations, such as braking, adjusting the direction and other operations.

[0107] Optionally, the first visual prompt may be a prompt message displayed on the dashboard indicating that the vehicle needs to be taken over.

[0108] Optionally, the first sound prompt may be a prompt message for generating a takeover vehicle and broadcasted in the target vehicle.

[0109] Optionally, the first tactile prompt may be to prompt the driver to take over driving the target vehicle by vibrating the steering wheel of the target vehicle.

[0110] S04. Prompting the driver to take over driving the target vehicle through the first prompting method.

[0111] Optionally, the method further comprises the following steps S10-S14:

[0112] S10, determining the takeover capability index of the driver based on the preset weight information and the heart rate variability, the pupil diameter and the blinking frequency, wherein the takeover capability index is used to indicate the timeliness of the driver taking over the target vehicle, and the larger the takeover capability index, the more timely the driver takes over the target vehicle;

[0113] Optionally, the preset weight information includes first weight information, second weight information and third weight information; the first weight information refers to the degree of influence of the driver's physiological state represented by heart rate variability on the driver's takeover ability; the second weight information refers to the degree of influence of the driver's physiological state represented by pupil diameter on the driver's takeover ability; the third weight information refers to the degree of influence of the driver's behavioral state represented by blinking frequency on the driver's takeover ability.

[0114] In the aforementioned S10, the driver's takeover ability index is determined based on the preset weight information and the heart rate variability, the pupil diameter and the blinking frequency, including: determining the driver's takeover ability index based on the first weight information, the second weight information, the third weight information, the heart rate variability, the pupil diameter and the blinking frequency.

[0115] Optionally, the determination of the driver's takeover capability index based on the first weight information, the second weight information, the third weight information, the heart rate variability, the pupil diameter and the blinking frequency can be expressed as the following formula:

[0116] TCI=w1·HRV+w2·Pupil Size+w3·Blink Rate

[0117] Among them, TCI is the takeover capacity index, W1 is the first weight information, HRV is the heart rate variability, W2 is the second weight information, Pupil size is the pupil diameter, W3 is the third weight information, and Blink rate is the blinking frequency.

[0118] Exemplarily, the preset weight information is usually obtained through training of a machine learning model to balance the importance of each feature. The pupil diameter can reflect the driver's attention level and psychological load status; the blinking frequency can reflect the driver's fatigue or distraction state. The blinking frequency can indicate the driver's visual attention to a specific area (such as an obstacle). The driver's takeover ability is comprehensively evaluated by collecting the driver's heart rate variability, pupil diameter, and blinking frequency, so that the accuracy of the determined driver's takeover ability is higher, and it can more accurately reflect the timeliness of the driver's takeover of the target vehicle.

[0119] S11, obtaining the road type information where the first location information is located, and the violation information of the illegal obstacle among the at least one first obstacle;

[0120] Optionally, the road type information may be any type of intersection, straight section, left turn, right turn, etc.

[0121] Optionally, the violation information may be information on various violations of traffic rules, such as pedestrians running red lights, non-motor vehicles running red lights, non-motor vehicles driving in the wrong direction, motor vehicles running red lights, and motor vehicles illegally parking.

[0122] S12, obtaining multiple alternative prompting methods;

[0123] Optionally, the multiple alternative prompting methods include at least the following three: a second visual prompt, a second sound prompt, and a second tactile prompt.

[0124] Optionally, the second visual prompt may be prompt information including conflict information displayed on a dashboard, wherein the conflict information includes a predicted collision time and a type of obstacle predicted to collide.

[0125] Optionally, the types of obstacles include motor vehicles, pedestrians, non-motor vehicles and fixed obstacles.

[0126] Optionally, the second sound prompt may generate prompt information including conflict information and broadcast it in the target moving vehicle.

[0127] Optionally, the second tactile prompt may be provided by vibrating the steering wheel of the target vehicle, and setting different vibration frequencies to represent different risk levels.

[0128] S13, determining a second prompting method from the multiple alternative prompting methods based on the takeover capability index, the road type information, and the violation information;

[0129] Optionally, the second prompting method includes at least one prompting method among a plurality of alternative prompting methods.

[0130] Optionally, in the aforementioned S13, based on the takeover capability index, the road type information, and the violation information, determining the second prompting method from the multiple alternative prompting methods includes:

[0131] The takeover capability index, the road type information, the violation information and the multiple alternative prompting methods are processed based on a preset strategy utility function to obtain a second prompting method.

[0132] Specifically, the takeover capability index, the road type information, the violation information, and the multiple alternative prompting methods are processed based on the preset strategy utility function to obtain a second prompting method, which can be expressed as the following formula:

[0133]

[0134] Among them, S opt is the second prompt mode, S is the multiple alternative prompt modes, TCI is the takeover capability index, and Scene is the road type information and violation information. For the relevant content of the strategy utility function, please refer to the prior art and no further details will be given here.

[0135] S14. Prompting the driver to take over driving the target vehicle through the second prompting method.

[0136] Exemplarily, if the shortest collision time is 1 minute, the obstacle corresponding to the shortest collision time is a pedestrian, and the determined second prompt method is the second sound prompt and the second visual prompt, then the driver is prompted to take over driving the target vehicle by playing a message in the car saying "There will be a collision with a pedestrian in 1 minute, please take over driving the vehicle" and displaying a message on the dashboard saying "There will be a collision with a pedestrian in one minute, please take over driving the vehicle" to indicate that the driver is taking over driving the target vehicle.

[0137] Optionally, the method further comprises the following steps S001-S004:

[0138] S001, obtaining the interval between the time when the driver is prompted to take over driving the target vehicle by the second prompting method and the time when the driver takes over driving the target vehicle;

[0139] S002, after determining that the driver takes over driving the target vehicle, acquiring the acceleration of the target vehicle from the takeover moment to the current moment, the second state information of the driver, the second driving information of the target vehicle after the driver takes over driving the target vehicle, the characteristic information of the target obstacle, and the shortest collision time;

[0140] Optionally, the second status information refers to the status information of the driver obtained when the driver takes over the vehicle, and the second status information includes: second physiological data and second eye data, wherein the second physiological data includes: heart rate, heart rate variability and skin conductance, and the second eye data includes: pupil diameter, blinking frequency and gaze duration.

[0141] S003, obtaining historical information of the target vehicle when it was last taken over by a historical driver before the takeover moment, the historical information including: historical interval duration, historical acceleration, historical state information of the historical driver, historical driving information of the target vehicle, characteristic information of historical target obstacles, and historical shortest collision time;

[0142] Optionally, the historical driver may be the same as or different from the current driver.

[0143] Optionally, the historical interval duration refers to the interval duration between the prompting moment of the historical driver to take over driving the target driving vehicle and the taking over moment of the historical driver taking over the target driving vehicle;

[0144] Optionally, the historical acceleration refers to the acceleration at a historical takeover moment when the driver takes over driving the target vehicle and a moment after the takeover moment.

[0145] Optionally, the historical state information of the historical driver refers to the state information of the historical driver obtained at the takeover moment of the historical driver taking over the vehicle; the historical state information includes: historical physiological data and historical eye data, wherein the historical physiological data includes: heart rate, heart rate variability and skin conductance, and the historical eye data includes: pupil diameter, blinking frequency and fixation duration;

[0146] Optionally, the historical driving information of the target driving vehicle refers to historical position information and historical speed information of the target driving vehicle during the driving process.

[0147] Optionally, the historical target obstacle refers to an obstacle that is predicted to collide with the target moving vehicle earliest at a position corresponding to the historical position information of the target moving vehicle.

[0148] The characteristic information of the historical target obstacle includes obstacle position information and obstacle speed information.

[0149] Optionally, the historical shortest collision time refers to the duration of a predicted collision between the target moving vehicle and the historical target obstacle at a position corresponding to the historical position information of the target moving vehicle.

[0150] S004. Input the interval duration, the acceleration, the second state information, the second driving information, the characteristic information of the target obstacle, the shortest collision time, and the historical information into a preset model to obtain the first takeover ability score of the driver.

[0151] Optionally, the preset model may be a deep learning model, for example, an LSTM network.

[0152] Optionally, in the aforementioned S004, the interval duration, the acceleration, the second state information, the second driving information, the characteristic information of the target obstacle, the shortest collision time, and the historical information are input into a preset model to obtain the first takeover ability score of the driver, including the following steps S0041-S0043:

[0153] S0041, the preset model processes the interval duration, the acceleration, the second state information, the second driving information, the characteristic information of the target obstacle and the shortest collision time to obtain a first multimodal data vector;

[0154] S0042. The preset model processes the historical information to obtain a second multimodal data vector;

[0155] S0043. The preset model processes the first multimodal data vector and the second multimodal data vector to obtain a first takeover ability score of the driver.

[0156] Specifically, the preset model in the aforementioned S0043 processes the first multimodal data vector and the second multimodal data vector to obtain the first takeover ability score of the driver, which can be expressed as the following formula:

[0157] y t =σ(W x x t +W □ □ t-1 +b)

[0158] Among them, y t is the first takeover capability score, σ() is the activation function, W x is the weight matrix of the first multimodal data vector, W □ is the weight matrix of the second multimodal data vector, x t is the first multimodal data vector, □ t-1 is the second multimodal data vector, and b is the bias term.

[0159] In summary, by scoring the driver's takeover ability, the driver's reaction and handling capabilities in taking over the driving of the target vehicle can be evaluated, which is convenient for comprehensively evaluating the driver's takeover performance. Afterwards, the relevant data with a high first takeover ability score can be used to train and optimize autonomous driving decisions.

[0160] Optionally, the method further comprises the following steps S0001-S0002:

[0161] S0001. Acquire a plurality of sample information and a sample result corresponding to each sample information, wherein the sample information includes a sample interval duration, a sample acceleration, a sample state information, a sample driving information, characteristic information of a sample target obstacle, a sample shortest collision time, and a sample history information, and the sample result is a second takeover capability score of the sample driver determined based on the sample interval duration, the sample acceleration, the sample state information, the sample driving information, characteristic information of the sample target obstacle, the sample shortest collision time, and the sample history information;

[0162] Optionally, the sample result refers to a numerical value of the sample driver's ability to take over driving the vehicle.

[0163] Optionally, the sample interval duration refers to the interval duration between the prompting moment for prompting the sample driver to take over driving the sample vehicle and the taking over moment for the sample driver to take over driving the sample vehicle;

[0164] Optionally, the sample acceleration refers to the acceleration between the moment when the sample driver takes over driving the sample vehicle and the moment next to the moment of taking over.

[0165] Optionally, the sample state information refers to the state information of the sample driver at the time of taking over the sample vehicle; the sample state information includes: sample physiological data and sample eye data, wherein the sample physiological data includes: heart rate, heart rate variability and skin conductance, and the sample eye data includes: pupil diameter, blinking frequency and fixation duration;

[0166] Optionally, the sample driving information refers to sample position information and sample speed information of a sample vehicle during driving.

[0167] Optionally, the sample target obstacle refers to the sample target obstacle that is predicted to collide with the sample vehicle earliest at the position corresponding to the sample position information of the sample vehicle. The characteristic information of the sample target obstacle includes obstacle position information and obstacle speed information.

[0168] Optionally, the sample shortest collision time refers to the predicted duration of a collision between the sample vehicle and a sample target obstacle at a position corresponding to the sample position information of the sample vehicle.

[0169] Optionally, the sample history information refers to the sample history information of the last time the sample vehicle was taken over by the sample history driver before the takeover moment when the sample driver took over the sample vehicle, and the sample history information includes: sample history interval duration, sample history acceleration, sample history state information of the sample history driver, sample history driving information of the sample vehicle, characteristic information of the sample history target obstacle, and sample history shortest collision time;

[0170] Optionally, the sample historical driver and the sample driver may be the same or different.

[0171] Optionally, the sample history interval duration refers to the interval duration between the prompting moment of the sample history driver to take over driving the sample vehicle and the taking over moment of the sample history driver taking over the sample vehicle;

[0172] Optionally, the sample historical acceleration refers to the acceleration between the takeover moment when the sample historical driver takes over driving the sample vehicle and the next moment after the takeover moment.

[0173] Optionally, the sample history state information of the sample history driver refers to the state information of the sample history driver obtained at the takeover moment when the sample history driver takes over the sample vehicle; the sample history state information includes: sample history physiological data and sample history eye data, wherein the sample history physiological data includes: heart rate, heart rate variability and skin conductance, and the sample history eye data includes: pupil diameter, blinking frequency and fixation duration;

[0174] Optionally, the sample historical driving information of the sample vehicle refers to sample historical position information and sample historical speed information of the sample vehicle during driving.

[0175] Optionally, the sample historical target obstacle refers to an obstacle that is predicted to collide with the sample vehicle at the earliest position corresponding to the sample historical position information of the sample vehicle. The feature information of the sample historical target obstacle includes obstacle position information and obstacle speed information.

[0176] Optionally, the sample history shortest collision time refers to the duration of a predicted collision between the sample vehicle and the sample history target obstacle at a position corresponding to the sample history position information of the sample vehicle.

[0177] S0002. Train the initial model based on the multiple sample information and the sample results corresponding to each sample information to obtain the preset model.

[0178] In summary, the present application provides a method for obtaining the first driving information and driving environment information of a target driving vehicle during driving, wherein the first driving information includes: first position information and first speed information; the driving environment information includes: 3D point cloud data of the driving environment, and a video stream of the driving environment; based on the driving environment information, characteristic information of each first obstacle in the driving environment is determined, and the characteristic information includes second position information and second speed information; the first driving information and the characteristic information of each first obstacle are used to determine the collision time between the target driving vehicle and each first obstacle, and at least one collision time corresponding to the at least one first obstacle is obtained; and a scheme for predicting whether the driver needs to take over driving the target driving vehicle is provided according to the at least one collision time. At least one first obstacle is determined by the acquired driving environment information, and the collision time between each first obstacle and the target driving vehicle is predicted, thereby obtaining at least one collision time, so as to predict whether the driver needs to take over driving the target driving vehicle based on at least one collision time, thereby improving the prediction accuracy of whether the driver needs to take over driving the vehicle in complex urban road scenes.

[0179] Specifically, the 3D point cloud data of the driving environment obtained by using laser radar, camera and vehicle environment perception system, as well as the video stream of the driving environment are analyzed and processed, and multiple values ​​of the same feature information of the same obstacle are fused and processed, thereby improving the feature information of the determined obstacle. By predicting the collision time between each obstacle among multiple obstacles and the target driving vehicle, it is determined whether the driver needs to take over the vehicle based on the predicted multiple collision times, thereby improving the accuracy of predicting whether the driver needs to take over the vehicle. In the case of complex road environments, it is possible to quickly obtain information on multiple obstacles that may collide with the target driving vehicle, and quickly estimate the collision time between each obstacle and the target driving vehicle, so as to accurately determine whether the driver needs to take over the vehicle in a complex road environment.

[0180] In addition, the driver's concentration level is comprehensively evaluated by obtaining the driver's physiological data and eye data, so as to prompt the driver to take over driving through a prompt method corresponding to the driver's concentration level. Prompting the driver to take over driving through a prompt method corresponding to the driver's concentration level is more effective, avoiding inappropriate prompt methods that cause the driver to fail to obtain prompt information in time, resulting in danger. It is also possible to dynamically determine a prompt method suitable for the driver by comprehensively analyzing the driver's physiological data and eye data, road type information and violation information, so that the determined prompt method is more suitable for the driver, thereby making the prompt effect better and improving the takeover efficiency.

[0181] Figure 2A schematic diagram of a structure of a multimodal data-based autonomous driving vehicle safety control device provided as an exemplary embodiment of the present application; wherein the device comprises:

[0182] The acquisition unit 21 is used to acquire first driving information and driving environment information of the target driving vehicle during driving, wherein the first driving information includes: first position information and first speed information; the driving environment information includes: 3D point cloud data of the driving environment, and a video stream of the driving environment;

[0183] a determining unit 22, configured to determine characteristic information of each first obstacle in the at least one first obstacle in the driving environment based on the driving environment information, wherein the characteristic information includes second position information and second speed information;

[0184] The determining unit 22 is further configured to determine a collision time between the target driving vehicle and each of the first obstacles by using the first driving information and the characteristic information of each of the first obstacles, and obtain at least one collision time corresponding to the at least one first obstacle;

[0185] The prediction unit 23 is used to predict whether a driver needs to take over driving the target vehicle according to the at least one collision time.

[0186] Optionally, when the device is used to determine the feature information of each first obstacle in the at least one first obstacle in the driving environment based on the driving environment information, it is specifically used to: determine the feature information of each second obstacle in the at least one second obstacle in the driving environment based on the 3D point cloud data; determine the feature information of each third obstacle in the at least one third obstacle in the driving environment based on the video stream; fuse the feature information of the at least one second obstacle and the at least one fourth obstacle that is repeated in the at least one third obstacle to obtain target feature information of the repeated obstacle; and use the remaining second obstacles in the at least one second obstacle except the at least one fourth obstacle, the remaining third obstacles in the at least one third obstacle except the at least one fourth obstacle, and each obstacle in the at least one fourth obstacle as first obstacles, thereby obtaining the feature information of each first obstacle in the at least one first obstacle.

[0187] Optionally, when the device is used to predict whether the driver needs to take over driving the target vehicle based on the at least one collision time, it is specifically used to: obtain the shortest collision time among the at least one collision time; when the shortest collision time is less than a preset threshold, determine that the driver needs to take over driving the target vehicle.

[0188] Optionally, when the shortest collision time is less than a preset threshold, the device is further used to: obtain first state information of the driver, the first state information including: first physiological data and first eye data, wherein the first physiological data includes: heart rate, heart rate variability and skin conductance, and the first eye data includes: pupil diameter, blinking frequency and gaze duration; determine the driver's concentration level based on characteristic information of the target obstacle corresponding to the shortest collision time among the at least one first obstacle, the driving information, and the first state information, wherein the concentration level includes: concentration, distraction, or lack of attention; determine a corresponding first prompt method based on the concentration level; and prompt the driver to take over driving the target driving vehicle through the first prompt method.

[0189] Optionally, the device is also used to: determine the driver's takeover ability index based on preset weight information and the heart rate variability, the pupil diameter and the blinking frequency, the takeover ability index is used to indicate the timeliness of the driver taking over the target vehicle, and the larger the takeover ability index, the more timely the driver takes over the target vehicle; obtain the road type information where the first position information is located, and the violation information of the illegal obstacle among the at least one first obstacle; obtain multiple alternative prompt methods; based on the takeover ability index, the road type information, and the violation information, determine a second prompt method from the multiple alternative prompt methods; prompt the driver to take over driving the target vehicle through the second prompt method.

[0190] Optionally, the device is also used to: obtain the interval time between the prompting moment of the driver taking over the driving of the target vehicle through the second prompting method and the takeover moment when the driver takes over the driving of the target vehicle; after determining that the driver takes over the driving of the target vehicle, obtain the acceleration of the target vehicle from the takeover moment to the current moment, the second state information of the driver, the second driving information of the target vehicle after the driver takes over the driving of the target vehicle, the characteristic information of the target obstacle, and the shortest collision time; obtain the historical information of the target vehicle when it was last taken over by a historical driver before the takeover moment, and the historical information includes: historical interval time, historical acceleration, historical state information of the historical driver, historical driving information of the target vehicle, characteristic information of the historical target obstacle, and historical shortest collision time; input the interval time, the acceleration, the second state information, the second driving information, the characteristic information of the target obstacle, the shortest collision time, and the historical information into a preset model to obtain the first takeover ability score of the driver.

[0191] Optionally, the device is also used to: obtain multiple sample information and sample results corresponding to each sample information, wherein the sample information includes sample interval duration, sample acceleration, sample status information, sample driving information, characteristic information of the sample target obstacle, the sample shortest collision time, and sample history information, and the sample result is a second takeover ability score of the sample driver determined based on the sample interval duration, the sample acceleration, the sample status information, the sample driving information, the characteristic information of the sample target obstacle, the sample shortest collision time, and the sample history information; train the initial model based on the multiple sample information and the sample results corresponding to each sample information to obtain the preset model.

[0192] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, the device may perform the above method embodiment, and the above and other operations and / or functions of each module in the device are the corresponding processes in each method in the above method embodiment, respectively, and no further description is given here for the sake of brevity.

[0193] The above describes the device of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0194] Figure 3 is a schematic block diagram of an electronic device provided in an embodiment of the present application, and the electronic device may include:

[0195] The memory 301 and the processor 302, the memory 301 is used to store the computer program and transmit the program code to the processor 302. In other words, the processor 302 can call and run the computer program from the memory 301 to implement the method in the embodiment of the present application.

[0196] For example, the processor 302 may be configured to execute the above method embodiments according to instructions in the computer program.

[0197] In some embodiments of the present application, the processor 302 may include but is not limited to:

[0198] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0199] In some embodiments of the present application, the memory 301 includes but is not limited to:

[0200] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0201] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 301 and executed by the processor 302 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0202] like Figure 3 As shown, the electronic device may also include:

[0203] The transceiver 303 may be connected to the processor 302 or the memory 301 .

[0204] The processor 302 may control the transceiver 303 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 303 may include a transmitter and a receiver. The transceiver 303 may further include an antenna, and the number of antennas may be one or more.

[0205] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0206] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0207] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0208] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0209] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0210] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0211] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for controlling the safety of an autonomous driving vehicle based on multimodal data, characterized in that: include: Acquire first driving information and driving environment information of the target driving vehicle during driving, wherein the first driving information includes: first position information and first speed information; the driving environment information includes: 3D point cloud data of the driving environment, and a video stream of the driving environment; Determining characteristic information of each first obstacle in at least one first obstacle in the driving environment based on the driving environment information, the characteristic information including second position information and second speed information; Determine the collision time between the target vehicle and each of the first obstacles by using the first driving information and the characteristic information of each of the first obstacles, and obtain at least one collision time corresponding to the at least one first obstacle; Whether a driver needs to take over driving the target vehicle is predicted based on the at least one collision time.

2. The method according to claim 1, characterized in that: The determining, based on the driving environment information, characteristic information of each first obstacle in the at least one first obstacle in the driving environment includes: Determining feature information of each of at least one second obstacle in the driving environment based on the 3D point cloud data; determining feature information of each of at least one third obstacle in the driving environment based on the video stream; performing fusion processing on feature information of at least one fourth obstacle that is repeated in the at least one second obstacle and the at least one third obstacle to obtain target feature information of the repeated obstacle; The remaining second obstacles in the at least one second obstacle except the at least one fourth obstacle, the remaining third obstacles in the at least one third obstacle except the at least one fourth obstacle, and each obstacle in the at least one fourth obstacle are taken as first obstacles, so as to obtain characteristic information of each first obstacle in the at least one first obstacle.

3. The method according to claim 1, characterized in that The predicting whether the driver needs to take over driving the target vehicle according to the at least one collision time includes: Obtaining the shortest collision time among the at least one collision time; When the shortest collision time is less than a preset threshold, it is determined that the driver needs to take over driving the target vehicle.

4. The method according to claim 3, characterized in that When the shortest collision time is less than a preset threshold, the method further includes: Acquire first state information of the driver, the first state information comprising: first physiological data and first eye data, wherein the first physiological data comprises: heart rate, heart rate variability and skin conductance, and the first eye data comprises: pupil diameter, blinking frequency and fixation duration; Determining the driver's concentration level based on characteristic information of a target obstacle corresponding to the shortest collision time among the at least one first obstacle, the driving information, and the first state information, wherein the concentration level includes: concentration, distraction, or lack of attention; Determining a corresponding first prompting method based on the concentration level; The driver is prompted to take over driving the target vehicle through the first prompting method.

5. The method according to claim 4, characterized in that The method further comprises: Determining the takeover capability index of the driver based on the preset weight information and the heart rate variability, the pupil diameter and the blinking frequency, wherein the takeover capability index is used to indicate the timeliness of the driver taking over the target driving vehicle, and the greater the takeover capability index, the more timely the driver takes over the target driving vehicle; Acquire the road type information where the first location information is located, and the violation information of the illegal obstacle among the at least one first obstacle; Get multiple alternative prompt methods; Determining a second prompting method from the plurality of alternative prompting methods based on the takeover capability index, the road type information, and the violation information; The driver is prompted to take over driving the target vehicle through the second prompting method.

6. The method according to claim 5, characterized in that The method further comprises: Obtaining the interval between the time when the driver is prompted to take over driving the target vehicle by the second prompting method and the time when the driver takes over driving the target vehicle; After determining that the driver takes over driving the target vehicle, acquiring the acceleration of the target vehicle from the takeover moment to the current moment, the second state information of the driver, the second driving information of the target vehicle after the driver takes over driving the target vehicle, the characteristic information of the target obstacle, and the shortest collision time; Acquire historical information of the target vehicle when it was most recently taken over by a historical driver before the takeover moment, the historical information including: historical interval duration, historical acceleration, historical state information of the historical driver, historical driving information of the target vehicle, characteristic information of historical target obstacles, and historical shortest collision time; The interval duration, the acceleration, the second state information, the second driving information, the characteristic information of the target obstacle, the shortest collision time, and the historical information are input into a preset model to obtain the first takeover ability score of the driver.

7. The method according to claim 6, characterized in that The method further comprises: Acquire a plurality of sample information and a sample result corresponding to each sample information, wherein the sample information includes a sample interval duration, a sample acceleration, a sample state information, a sample driving information, characteristic information of a sample target obstacle, a sample shortest collision time, and a sample history information, and the sample result is a second takeover ability score of the sample driver determined based on the sample interval duration, the sample acceleration, the sample state information, the sample driving information, characteristic information of the sample target obstacle, the sample shortest collision time, and the sample history information; The initial model is trained based on the multiple sample information and the sample results corresponding to each sample information to obtain the preset model.

8. An autonomous driving vehicle safety control device based on multimodal data, characterized in that: include: An acquisition unit is used to acquire first driving information and driving environment information of a target driving vehicle during driving, wherein the first driving information includes: first position information and first speed information; the driving environment information includes: 3D point cloud data of the driving environment, and a video stream of the driving environment; a determining unit, configured to determine characteristic information of each first obstacle in the at least one first obstacle in the driving environment based on the driving environment information, wherein the characteristic information includes second position information and second speed information; The determination unit is further configured to determine a collision time between the target driving vehicle and each of the first obstacles by using the first driving information and the characteristic information of each of the first obstacles, and obtain at least one collision time corresponding to the at least one first obstacle; A prediction unit is used to predict whether a driver needs to take over driving the target vehicle based on the at least one collision time.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images