Emergency Assist Takeover Methods, Systems, Devices and Storage Media for Automated Vehicles

Abstract

This invention discloses an emergency assisted takeover method, system, and storage medium for autonomous vehicles, comprising: acquiring facial image information, voice information, physiological parameter information, and human motion information of the driver of a target vehicle in autonomous driving state; inputting the facial image information, voice information, physiological parameter information, and human motion information into a pre-trained takeover state prediction model to obtain the driver's current takeover state; acquiring the hazard level of the target vehicle, and predicting whether the driver can take over the target vehicle in a timely manner based on the current takeover state and the hazard level; if the driver cannot take over the target vehicle in a timely manner, determining a corresponding emergency assisted takeover decision based on the hazard level and the current takeover state, and then executing the emergency assisted takeover decision through the target vehicle. This invention improves the driving safety of autonomous vehicles and can be widely applied in the field of vehicle control technology.

Description

Technical Field

[0001] This invention relates to the field of vehicle control technology, and in particular to an emergency assisted takeover method, system, device, and storage medium for autonomous vehicles. Background Technology

[0002] With the development of intelligent and connected vehicles, vehicle monitoring and control technologies are becoming increasingly intelligent, and autonomous driving technology has become a key focus and research area in the automotive industry. When implementing autonomous driving functions, timely driver intervention is required in dangerous situations; otherwise, collisions and other accidents may occur. The speed at which the driver takes over the vehicle and the system's assisted decision-making and control significantly impact driving safety. Timely intervention places high demands on the driver's attention shift speed and decision-making ability. Therefore, predicting the driver's potential for intervention and ensuring the driving safety of autonomous vehicles when the driver cannot take over in time have become urgent problems to be solved. Summary of the Invention

[0003] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0004] Therefore, one objective of this invention is to provide an emergency assisted takeover method for autonomous vehicles, which improves the driving safety of autonomous vehicles.

[0005] Another objective of this invention is to provide an emergency assisted takeover system for autonomous vehicles.

[0006] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of the present invention include:

[0007] In a first aspect, embodiments of the present invention provide an emergency assisted takeover method for an autonomous vehicle, comprising the following steps:

[0008] Acquire facial image information, voice information, physiological parameter information, and human movement information of the driver of the target vehicle in autonomous driving mode;

[0009] The facial image information, the voice information, the physiological parameter information, and the human motion information are input into a pre-trained takeover state prediction model to obtain the driver's current takeover state.

[0010] The danger level of the target vehicle is obtained, and the driver is predicted to take over the target vehicle in a timely manner based on the current takeover status and the danger level.

[0011] If the driver is unable to take over the target vehicle in a timely manner, an emergency auxiliary takeover decision is determined based on the level of danger and the current takeover status, and then the emergency auxiliary takeover decision is executed through the target vehicle.

[0012] Furthermore, in one embodiment of the present invention, the step of acquiring the facial image information, voice information, physiological parameter information, and human motion information of the driver of the target vehicle in autonomous driving mode specifically includes:

[0013] The driver's facial image information is obtained by a camera device installed in the target vehicle;

[0014] The driver's voice information is acquired by a voice acquisition device installed in the target vehicle;

[0015] The physiological parameters of the driver are obtained through the physiological detection device worn by the driver.

[0016] The driver's body movement information is obtained through the posture sensor worn by the driver.

[0017] Furthermore, in one embodiment of the present invention, the emergency assisted takeover method for autonomous vehicles further includes a step of pre-training the takeover state prediction model, which specifically includes:

[0018] Acquire multiple preset historical sample data, each of which includes tester's facial image sample data, voice sample data, physiological parameter sample data, and human movement sample data;

[0019] Obtain the driver takeover response time corresponding to each of the historical sample data, and determine the corresponding tag information based on the driver takeover response time;

[0020] A training dataset is constructed based on the historical sample data and the corresponding label information;

[0021] The training dataset is input into a pre-built convolutional neural network for training to obtain the trained takeover state prediction model.

[0022] The tag information includes at least one of the following: fast response status, normal response status, and abnormal response status.

[0023] Furthermore, in one embodiment of the present invention, the step of inputting the training dataset into a pre-built convolutional neural network for training to obtain the trained takeover state prediction model specifically includes:

[0024] The training dataset is input into the convolutional neural network, and the takeover state prediction result is output.

[0025] The loss value of the convolutional neural network is determined based on the takeover state prediction result and the label information;

[0026] The model parameters of the convolutional neural network are updated using the backpropagation algorithm based on the loss value, and the process of inputting the training dataset into the convolutional neural network is returned.

[0027] When the loss value reaches a preset first threshold or the number of iterations reaches a preset second threshold, training stops, and the trained takeover state prediction model is obtained.

[0028] Furthermore, in one embodiment of the present invention, the step of obtaining the danger level of the target vehicle and predicting whether the driver can take over the target vehicle in a timely manner based on the current takeover status and the danger level specifically includes:

[0029] The estimated collision time of the target vehicle is determined, and the hazard level of the target vehicle is determined based on the estimated collision time, wherein the hazard level is a general hazard scenario, an emergency hazard scenario, or a no-hazard scenario;

[0030] When the danger scenario level is an emergency danger scenario, if the current takeover status is a rapid response status, it is determined that the driver can take over the target vehicle in a timely manner; if the current takeover status is a normal response status or an abnormal response status, it is determined that the driver cannot take over the target vehicle in a timely manner.

[0031] When the danger scenario level is a general danger scenario, if the current takeover status is a rapid response status or a normal response status, it is determined that the driver can take over the target vehicle in a timely manner; if the current takeover status is an abnormal response status, it is determined that the driver cannot take over the target vehicle in a timely manner.

[0032] When the danger scenario level is no danger scenario, it is determined that the driver can take over the target vehicle in a timely manner.

[0033] Furthermore, in one embodiment of the present invention, the step of determining the estimated collision time of the target vehicle and determining the hazard level of the target vehicle based on the estimated collision time specifically includes:

[0034] The real-time distance between the target vehicle and the vehicle in front is obtained, and the relative speed between the target vehicle and the vehicle in front is determined.

[0035] The estimated collision time of the target vehicle is calculated based on the real-time distance and the relative speed.

[0036] When the estimated collision time is less than or equal to a preset third threshold, the dangerous scenario level is determined to be an emergency dangerous scenario;

[0037] When the estimated collision time is greater than the third threshold and less than or equal to the preset fourth threshold, the dangerous scenario level is determined to be a general dangerous scenario.

[0038] When the estimated collision time is greater than the fourth threshold, the dangerous scenario level is determined to be a non-dangerous scenario.

[0039] Furthermore, in one embodiment of the present invention, the step of determining a corresponding emergency assisted takeover decision based on the hazardous scenario level and the current takeover status, and then executing the emergency assisted takeover decision through the target vehicle, specifically includes:

[0040] Obtain a preset emergency auxiliary takeover decision library, and perform index matching in the emergency auxiliary takeover decision library according to the danger scenario level and the current takeover status to obtain the corresponding emergency auxiliary takeover decision;

[0041] The emergency takeover decision is sent to the autonomous driving system of the target vehicle, so that the autonomous driving system executes the emergency takeover decision.

[0042] Secondly, embodiments of the present invention provide an emergency assisted takeover system for autonomous vehicles, comprising:

[0043] The information acquisition module is used to acquire facial image information, voice information, physiological parameter information, and human movement information of the driver of the target vehicle in autonomous driving mode.

[0044] The takeover state prediction module is used to input the facial image information, the voice information, the physiological parameter information, and the human body action information into a pre-trained takeover state prediction model to obtain the driver's current takeover state.

[0045] The takeover judgment module is used to obtain the danger scenario level of the target vehicle and predict whether the driver can take over the target vehicle in a timely manner based on the current takeover status and the danger scenario level.

[0046] The emergency assisted takeover module is used to determine the corresponding emergency assisted takeover decision based on the danger scenario level and the current takeover status when the driver is unable to take over the target vehicle in a timely manner, and then execute the emergency assisted takeover decision through the target vehicle.

[0047] Thirdly, embodiments of the present invention provide an emergency assisted takeover device for an autonomous vehicle, comprising:

[0048] At least one processor;

[0049] At least one memory for storing at least one program;

[0050] When the at least one program is executed by the at least one processor, the at least one processor implements the above-described emergency assisted takeover method for an autonomous vehicle.

[0051] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the above-described emergency assisted takeover method for an autonomous vehicle.

[0052] The advantages and beneficial effects of the present invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention:

[0053] This invention acquires facial image information, voice information, physiological parameter information, and human motion information of the driver of a target vehicle in autonomous driving mode. This information is then input into a pre-trained takeover state prediction model to obtain the driver's current takeover state. The hazard level of the target vehicle is then obtained. Based on the current takeover state and the hazard level, the model predicts whether the driver can take over the target vehicle in a timely manner. If the driver cannot take over in a timely manner, a corresponding emergency assisted takeover decision is determined based on the hazard level and the current takeover state, and then executed by the target vehicle. This invention predicts the driver's takeover state based on four dimensions of data: facial image, voice, physiological parameters, and human motion. This accurately assesses the driver's current ability to take over in emergencies. Furthermore, by combining this with the current hazard level, it accurately determines whether the driver can take over the autonomous vehicle in a timely manner and matches a corresponding emergency assisted takeover decision for the autonomous vehicle to execute when the driver cannot take over, thus improving the driving safety of autonomous vehicles. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments of the present invention are described below. It should be understood that the drawings described below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1A flowchart illustrating the steps of an emergency assisted takeover method for an autonomous vehicle provided in an embodiment of the present invention;

[0056] Figure 2 A structural block diagram of an emergency assisted takeover system for an autonomous vehicle provided in an embodiment of the present invention;

[0057] Figure 3 This is a structural block diagram of an emergency auxiliary takeover device for an autonomous vehicle provided in an embodiment of the present invention. Detailed Implementation

[0058] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0059] In the description of this invention, "multiple" means two or more. The use of "first" and "second" is for distinguishing technical features only and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or the order of the indicated technical features. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art.

[0060] Reference Figure 1 This invention provides an emergency assisted takeover method for autonomous vehicles, specifically including the following steps:

[0061] S101. Obtain facial image information, voice information, physiological parameter information, and human motion information of the driver of the target vehicle in autonomous driving mode.

[0062] Specifically, embodiments of the present invention predict the driver takeover state based on four dimensions: facial image information, voice information, physiological parameter information, and human motion information. It is understandable that when a driver speaks at a high level (more than 70% of the time) for a certain period of time, it indicates that the driver may be arguing or that their attention is mainly focused on the conversation, and they may not be able to take over the vehicle in time. Therefore, this embodiment of the invention uses voice information as input data to determine the driver's takeover status. When the driver's eyes are not on the road for a period of time, it indicates that the driver is busy with other things, or when the driver closes their eyes for a period of time, it indicates that the driver is fatigued and drowsy, and may not be able to take over the vehicle in time. Therefore, this embodiment of the invention uses facial image information as input data to determine the driver's takeover status. Physiological parameter information is used to monitor whether the driver's body is in an abnormal state. When there are abnormal indicators (such as a heart rate difference of 20% from the normal heart rate, blood oxygen level below 90%, etc.), it indicates that the driver may be unable to take over the vehicle in time due to fatigue. Therefore, this embodiment of the invention uses physiological parameter information as input data to determine the driver's takeover status. In addition, human body movement information can directly reflect the driver's current human posture and the actions being performed. Therefore, this embodiment of the invention uses human body movement information as input data to determine the driver's takeover status.

[0063] As a further optional implementation, the step of acquiring the driver's facial image information, voice information, physiological parameter information, and human motion information of the target vehicle in autonomous driving mode specifically includes:

[0064] S1011. Obtain the driver's facial image information through a camera device installed in the target vehicle;

[0065] S1012. Obtain the driver's voice information through a voice acquisition device installed in the target vehicle;

[0066] S1013. Obtain the driver's physiological parameter information through a physiological detection device worn by the driver;

[0067] S1014. Obtain the driver's human body movement information through the posture sensor worn by the driver.

[0068] Specifically, facial image information can be acquired through a camera device installed in front of the driver's seat; voice information can be acquired through a voice acquisition device installed in the cockpit; physiological parameter information, including one or more of heart rate, heart rate variability, heart rate interval (RRI), electroencephalogram (EEG), pulse wave, pulse rate variability, blood pressure, body temperature, and sweating rate, can be detected and collected through physiological detection devices worn by the driver, such as smart bracelets, smartwatches, and smart headbands; human motion information can be obtained by collecting posture data of various parts of the human body. For the driver, collecting posture data of key parts such as the arms, shoulders, head, and neck can obtain human motion information reflecting their driving behavior. Specifically, posture sensors can be set up at various key parts of the human body to collect posture data of the corresponding parts and integrate them to obtain human motion information.

[0069] S102. Input the facial image information, voice information, physiological parameter information and human motion information into the pre-trained takeover state prediction model to obtain the driver's current takeover state.

[0070] Specifically, the takeover state prediction model of this invention is trained by a convolutional neural network. By inputting the acquired facial image information, voice information, physiological parameter information and human action information into the takeover state prediction model, the corresponding takeover state prediction result, i.e., the driver's current takeover state, can be obtained.

[0071] As an optional implementation, the emergency assisted takeover method for autonomous vehicles further includes a step of pre-training a takeover state prediction model, which specifically includes:

[0072] S201. Obtain multiple preset historical sample data, each of which includes the tester's facial image sample data, voice sample data, physiological parameter sample data, and human movement sample data.

[0073] S202. Obtain the driver takeover response time corresponding to each historical sample data, and determine the corresponding tag information based on the driver takeover response time;

[0074] S203. Construct a training dataset based on historical sample data and corresponding label information;

[0075] S204. Input the training dataset into the pre-built convolutional neural network for training to obtain a trained takeover state prediction model.

[0076] The tag information includes at least one of the following: rapid response status, normal response status, and abnormal response status.

[0077] Specifically, when constructing the training dataset, multiple historical sample data covering various takeover states are acquired. Simultaneously, based on the actual driver takeover response time, label information is determined for each historical sample data point. This label information indicates the driver takeover state corresponding to each historical sample data point, including rapid response, normal response, and abnormal response states. The training dataset is then generated based on the historical sample data and the corresponding label information.

[0078] In some optional embodiments, when the actual driver takeover response time is within 3 seconds, the corresponding tag information can be determined to be a fast response state; when the actual driver takeover response time is more than 3 seconds but less than 5 seconds, the corresponding tag information can be determined to be a normal response state; when the actual driver takeover response time is more than 5 seconds, the corresponding tag information can be determined to be an abnormal response state.

[0079] As an optional implementation, the step of inputting the training dataset into a pre-built convolutional neural network for training to obtain a trained takeover state prediction model specifically includes:

[0080] S2041. Input the training dataset into the convolutional neural network and output the takeover state prediction result.

[0081] S2042. Determine the loss value of the convolutional neural network based on the takeover status prediction results and label information;

[0082] S2043. Update the model parameters of the convolutional neural network using the backpropagation algorithm based on the loss value, and return to the step of inputting the training dataset into the convolutional neural network;

[0083] S2044. When the loss value reaches the preset first threshold or the number of iterations reaches the preset second threshold, training is stopped, and the trained takeover state prediction model is obtained.

[0084] Specifically, after inputting the data from the training dataset into the initialized convolutional neural network model, the model outputs the recognition result, i.e., the takeover state prediction result. The accuracy of the model's prediction can be evaluated based on the takeover state prediction result and the aforementioned label information, thereby updating the model's parameters. For the takeover state prediction model, the accuracy of the model's prediction result can be measured by a loss function. The loss function is defined on a single training data point and is used to measure the prediction error of that training data point. Specifically, the loss value of that training data point is determined by the label of that individual training data point and the model's prediction result for that training data point. In actual training, a training dataset contains many training data points. Therefore, a cost function is generally used to measure the overall error of the training dataset. The cost function is defined on the entire training dataset and is used to calculate the average prediction error of all training data points, providing a better measure of the model's prediction performance. For general machine learning models, the aforementioned cost function, plus a regularization term to measure model complexity, can be used as the training objective function. Based on this objective function, the loss value of the entire training dataset can be calculated. There are many commonly used loss functions, such as 0-1 loss, squared loss, absolute loss, logarithmic loss, and cross-entropy loss, all of which can be used as loss functions for machine learning models. These will not be elaborated upon here. In this embodiment of the invention, any one of these loss functions can be selected to determine the training loss value. Based on the training loss value, the backpropagation algorithm is used to update the model parameters. After several iterations, a well-trained takeover state prediction model can be obtained. The specific number of iterations can be preset, or training can be considered complete when the accuracy requirement on the test set is met.

[0085] S103. Obtain the danger level of the target vehicle, and predict whether the driver can take over the target vehicle in a timely manner based on the current takeover status and danger level.

[0086] As a further optional implementation, the step of obtaining the hazard level of the target vehicle and predicting whether the driver can take over the target vehicle in a timely manner based on the current takeover status and the hazard level specifically includes:

[0087] S1031. Determine the estimated collision time of the target vehicle, and determine the hazard level of the target vehicle based on the estimated collision time. The hazard level is classified as a general hazard, an emergency hazard, or a no-hazard scenario.

[0088] S1032. When the danger scenario level is an emergency danger scenario, if the current takeover status is a rapid response status, it is determined that the driver can take over the target vehicle in a timely manner; if the current takeover status is a normal response status or an abnormal response status, it is determined that the driver cannot take over the target vehicle in a timely manner.

[0089] S1033. When the danger scenario level is a general danger scenario, if the current takeover status is a rapid response status or a normal response status, it is determined that the driver can take over the target vehicle in a timely manner; if the current takeover status is an abnormal response status, it is determined that the driver cannot take over the target vehicle in a timely manner.

[0090] S1034. When the danger level is no danger level, ensure that the driver can take over the target vehicle in a timely manner.

[0091] As a further optional implementation, the step of determining the expected collision time of the target vehicle and determining the hazard level of the target vehicle based on the expected collision time specifically includes:

[0092] S10311. Obtain the real-time distance between the target vehicle and the vehicle in front, and determine the relative speed between the target vehicle and the vehicle in front.

[0093] S10312. Calculate the estimated collision time of the target vehicle based on the real-time distance and relative speed.

[0094] S10313. When the estimated collision time is less than or equal to the preset third threshold, the dangerous scenario level is determined to be an emergency dangerous scenario.

[0095] S10314. When the estimated collision time is greater than the third threshold and less than or equal to the preset fourth threshold, the dangerous scenario level is determined to be a general dangerous scenario.

[0096] S10315. When the expected collision time is greater than the fourth threshold, the dangerous scenario level is determined to be a non-dangerous scenario.

[0097] Specifically, in this embodiment of the invention, the danger level of the target vehicle is determined based on the estimated time to collision (TTC). When the estimated time to collision is less than or equal to the third threshold (e.g., 3 seconds), the corresponding danger level is an emergency danger scenario. When the estimated time to collision is greater than the third threshold and less than or equal to the fourth threshold (e.g., 10 seconds), the corresponding danger level is a general danger scenario. When the estimated time to collision is greater than the fourth threshold, the autonomous driving system can avoid the collision on its own without driver intervention, and the corresponding danger level is a no-danger scenario.

[0098] After determining the level of danger, the driver's ability to take over the target vehicle in a timely manner is determined based on the level of danger and the driver's current takeover status. If the danger level is an emergency danger scenario and the current takeover status is a normal response status or an abnormal response status, it is determined that the driver cannot take over the target vehicle in a timely manner. If the danger level is a general danger scenario and the current takeover status is an abnormal response status, it is determined that the driver cannot take over the target vehicle in a timely manner. In all other cases, it can be determined that the driver can take over the target vehicle in a timely manner.

[0099] S104. When the driver is unable to take over the target vehicle in a timely manner, the corresponding emergency auxiliary takeover decision shall be determined based on the level of danger and the current takeover status, and then the emergency auxiliary takeover decision shall be executed through the target vehicle.

[0100] Specifically, when it is determined that the driver cannot take over the target vehicle in a timely manner, an emergency assisted takeover decision is made and executed by the target vehicle to avoid collisions involving autonomous vehicles.

[0101] As an optional implementation, the step of determining a corresponding emergency auxiliary takeover decision based on the level of danger and the current takeover status, and then executing the emergency auxiliary takeover decision through the target vehicle, specifically includes:

[0102] S1041. Obtain the preset emergency auxiliary takeover decision library, and perform index matching in the emergency auxiliary takeover decision library according to the level of dangerous scenario and the current takeover status to obtain the corresponding emergency auxiliary takeover decision.

[0103] S1042. The emergency assisted takeover decision is sent to the autonomous driving system of the target vehicle, so that the autonomous driving system executes the emergency assisted takeover decision.

[0104] Specifically, this embodiment of the invention pre-configures an emergency assisted takeover decision library. This library contains multiple emergency assisted takeover decisions, each with a corresponding "hazard level - current takeover status" array. Based on the hazard level and current takeover status determined in the preceding steps, a corresponding emergency assisted takeover decision can be matched within the library. The target vehicle's autonomous driving system executes this emergency assisted takeover decision, thereby enabling assisted takeover control of the target vehicle when the driver is unable to take over in a timely manner, improving the driving safety of the autonomous vehicle.

[0105] The method steps of the embodiments of the present invention have been described above. It can be understood that the embodiments of the present invention predict the driver's takeover state based on four dimensions of data: driver's facial image, voice, physiological parameters, and human body movements. This can accurately assess the driver's current ability to take over in response to emergencies. At the same time, combined with the current dangerous scenario, it can accurately determine whether the driver can take over the autonomous vehicle in a timely manner. When the driver cannot take over the autonomous vehicle, it matches the corresponding emergency auxiliary takeover decision for the autonomous vehicle to execute, thereby improving the driving safety of the autonomous vehicle.

[0106] Reference Figure 2 This invention provides an emergency assisted takeover system for autonomous vehicles, comprising:

[0107] The information acquisition module is used to acquire facial image information, voice information, physiological parameter information, and human movement information of the driver of the target vehicle in autonomous driving mode.

[0108] The takeover state prediction module is used to input facial image information, voice information, physiological parameter information and human action information into a pre-trained takeover state prediction model to obtain the driver's current takeover state.

[0109] The takeover judgment module is used to obtain the danger level of the target vehicle and predict whether the driver can take over the target vehicle in a timely manner based on the current takeover status and danger level.

[0110] The emergency assisted takeover module is used to determine the corresponding emergency assisted takeover decision based on the level of danger and the current takeover status when the driver is unable to take over the target vehicle in a timely manner, and then execute the emergency assisted takeover decision through the target vehicle.

[0111] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0112] Reference Figure 3 This invention provides an emergency takeover assist device for autonomous vehicles, comprising:

[0113] At least one processor;

[0114] At least one memory for storing at least one program;

[0115] When the above-mentioned at least one program is executed by the above-mentioned at least one processor, the above-mentioned at least one processor implements the above-mentioned emergency assisted takeover method for an autonomous vehicle.

[0116] The content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0117] This invention also provides a computer-readable storage medium storing a processor-executable program that, when executed by a processor, performs the aforementioned emergency assisted takeover method for an autonomous vehicle.

[0118] This invention provides a computer-readable storage medium that can execute an emergency assisted takeover method for an autonomous vehicle provided in an embodiment of the invention. It can execute any combination of the implementation steps of the method embodiment and has the corresponding functions and beneficial effects of the method.

[0119] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0120] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the aforementioned blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0121] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0122] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0124] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the aforementioned program can be printed, because the aforementioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or, if necessary, processing in other suitable ways, and then stored in computer memory.

[0125] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0126] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0127] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0128] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for emergency assisted takeover of an autonomous vehicle, characterized in that, Includes the following steps: Acquire facial image information, voice information, physiological parameter information, and human movement information of the driver of the target vehicle in autonomous driving mode; The facial image information, the voice information, the physiological parameter information, and the human motion information are input into a pre-trained takeover state prediction model to obtain the driver's current takeover state. The danger level of the target vehicle is obtained, and the driver is predicted to take over the target vehicle in a timely manner based on the current takeover status and the danger level. When the driver is unable to take over the target vehicle in a timely manner, an emergency auxiliary takeover decision is determined based on the level of danger and the current takeover status, and then the emergency auxiliary takeover decision is executed through the target vehicle. The step of obtaining the danger level of the target vehicle and predicting whether the driver can take over the target vehicle in a timely manner based on the current takeover status and the danger level specifically includes: The estimated collision time of the target vehicle is determined, and the hazard level of the target vehicle is determined based on the estimated collision time, wherein the hazard level is a general hazard scenario, an emergency hazard scenario, or a no-hazard scenario; When the danger scenario level is an emergency danger scenario, if the current takeover status is a rapid response status, it is determined that the driver can take over the target vehicle in a timely manner; if the current takeover status is a normal response status or an abnormal response status, it is determined that the driver cannot take over the target vehicle in a timely manner. When the danger scenario level is a general danger scenario, if the current takeover status is a rapid response status or a normal response status, it is determined that the driver can take over the target vehicle in a timely manner; if the current takeover status is an abnormal response status, it is determined that the driver cannot take over the target vehicle in a timely manner. When the danger scenario level is no danger scenario, it is determined that the driver can take over the target vehicle in a timely manner.

2. The emergency assisted takeover method for an autonomous vehicle according to claim 1, characterized in that, The step of acquiring the facial image information, voice information, physiological parameter information, and human movement information of the driver of the target vehicle in autonomous driving mode specifically includes: The driver's facial image information is obtained by a camera device installed in the target vehicle; The driver's voice information is acquired by a voice acquisition device installed in the target vehicle; The physiological parameters of the driver are obtained through the physiological detection device worn by the driver. The driver's body movement information is obtained through the posture sensor worn by the driver.

3. The emergency assisted takeover method for an autonomous vehicle according to claim 1, characterized in that, The emergency assisted takeover method for autonomous vehicles also includes the step of pre-training the takeover state prediction model, which specifically includes: Acquire multiple preset historical sample data, each of which includes tester's facial image sample data, voice sample data, physiological parameter sample data, and human movement sample data; Obtain the driver takeover response time corresponding to each of the historical sample data, and determine the corresponding tag information based on the driver takeover response time; A training dataset is constructed based on the historical sample data and the corresponding label information; The training dataset is input into a pre-built convolutional neural network for training to obtain the trained takeover state prediction model. The tag information includes at least one of the following: fast response status, normal response status, and abnormal response status.

4. The emergency assisted takeover method for an autonomous vehicle according to claim 3, characterized in that, The step of inputting the training dataset into a pre-built convolutional neural network for training to obtain the trained takeover state prediction model specifically includes: The training dataset is input into the convolutional neural network, and the takeover state prediction result is output. The loss value of the convolutional neural network is determined based on the takeover state prediction result and the label information; The model parameters of the convolutional neural network are updated using the backpropagation algorithm based on the loss value, and the process of inputting the training dataset into the convolutional neural network is returned. When the loss value reaches a preset first threshold or the number of iterations reaches a preset second threshold, training stops, and the trained takeover state prediction model is obtained.

5. The emergency assisted takeover method for an autonomous vehicle according to claim 1, characterized in that, The step of determining the estimated collision time of the target vehicle and determining the hazard level of the target vehicle based on the estimated collision time specifically includes: The real-time distance between the target vehicle and the vehicle in front is obtained, and the relative speed between the target vehicle and the vehicle in front is determined. The estimated collision time of the target vehicle is calculated based on the real-time distance and the relative speed. When the estimated collision time is less than or equal to a preset third threshold, the dangerous scenario level is determined to be an emergency dangerous scenario; When the estimated collision time is greater than the third threshold and less than or equal to the preset fourth threshold, the dangerous scenario level is determined to be a general dangerous scenario. When the estimated collision time is greater than the fourth threshold, the dangerous scenario level is determined to be a non-dangerous scenario.

6. A method for emergency assisted takeover of an autonomous vehicle according to any one of claims 1 to 5, characterized in that, The step of determining the corresponding emergency auxiliary takeover decision based on the hazardous scenario level and the current takeover status, and then executing the emergency auxiliary takeover decision through the target vehicle, specifically includes: Obtain a preset emergency auxiliary takeover decision library, and perform index matching in the emergency auxiliary takeover decision library according to the danger scenario level and the current takeover status to obtain the corresponding emergency auxiliary takeover decision; The emergency takeover decision is sent to the autonomous driving system of the target vehicle, so that the autonomous driving system executes the emergency takeover decision.

7. An emergency takeover assistance system for autonomous vehicles, characterized in that, A method for implementing an emergency assisted takeover method for an autonomous vehicle as described in any one of claims 1 to 6, comprising: The information acquisition module is used to acquire facial image information, voice information, physiological parameter information, and human movement information of the driver of the target vehicle in autonomous driving mode. The takeover state prediction module is used to input the facial image information, the voice information, the physiological parameter information, and the human body action information into a pre-trained takeover state prediction model to obtain the driver's current takeover state. The takeover judgment module is used to obtain the danger scenario level of the target vehicle and predict whether the driver can take over the target vehicle in a timely manner based on the current takeover status and the danger scenario level. The emergency assisted takeover module is used to determine the corresponding emergency assisted takeover decision based on the danger scenario level and the current takeover status when the driver is unable to take over the target vehicle in a timely manner, and then execute the emergency assisted takeover decision through the target vehicle.

8. An emergency takeover assist device for an autonomous vehicle, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements an emergency assisted takeover method for an autonomous vehicle as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform an emergency assisted takeover method for an autonomous vehicle as described in any one of claims 1 to 6.

Images