A method and system for autonomous driving takeover based on cognitive-executive two-level decision-making

Abstract

This invention belongs to the field of road vehicle control systems and relates to an autonomous driving takeover method and system based on cognitive-execution two-layer decision-making. The system includes a multimodal large model, a takeover mode judgment module, a warning intensity calculation module, a perception and alert module, a readiness state assessment module, an adaptive weight allocation module, a control signal output module, and a guidance interface generation module. The multimodal large model is used to infer from input data and output structured takeover decision information. The warning intensity calculation module calculates the takeover warning intensity over time. The readiness state assessment module assesses the driver's readiness for takeover. The control signal output module, based on the driver's adaptive control weights, achieves a smooth transfer of human-machine control under safety constraints through a control fusion mechanism. This system not only improves the accuracy and interpretability of takeover decisions but also effectively enhances takeover performance and reduces takeover risks.

Description

Technical Field

[0001] This invention belongs to the field of road vehicle control systems and relates to the takeover of autonomous vehicles. Specifically, it relates to an autonomous driving takeover method and system based on cognitive-execution dual-layer decision-making, which is particularly suitable for human-machine co-driving control switching scenarios in L2-L3 level autonomous vehicles. Background Technology

[0002] In recent years, with the rapid development of autonomous driving technology, its practical application has become very common. However, due to the limitations of existing technical solutions, L2-L3 level autonomous driving systems still need to transfer vehicle control to the driver in certain scenarios that are difficult for other autonomous driving systems to handle, thus involving the issue of driver takeover. Existing takeover decision-making methods are mainly based on preset rules or threshold judgments, which have problems such as poor scenario adaptability, uneven control transfer, lack of interpretability in the decision-making process, and lack of effective takeover guidance, making it difficult to meet the safety takeover requirements in complex driving environments.

[0003] With the rapid development of Multimodal Large Language Models (MLLM), their powerful language understanding and cognitive reasoning capabilities have brought new technological pathways to autonomous driving systems. For example, Chinese patent CN 118810764 A discloses a multi-level driving risk management method, system, device, and medium. At the vehicle end, an initial risk assessment is performed on the simultaneously collected forward-looking video stream, vehicle information data, and DMS video stream to obtain real-time and cumulative risks. Real-time risks are promptly alerted, and the data corresponding to cumulative risks is uploaded to the cloud. A second-level risk assessment is then performed using multiple trained MLLM models to obtain driver status and algorithm confidence information. Alarm interventions are initiated for high-confidence, high-risk data, improving risk warning efficiency. However, multimodal large language models are not currently used in autonomous driving takeover systems. Therefore, how to fully utilize the cognitive reasoning capabilities of multimodal large models and combine them with adaptive execution control strategies to design an accurate, smooth, interpretable, and guided takeover decision-making method has become a key issue that urgently needs to be addressed in the development of current autonomous driving technology. Summary of the Invention

[0004] In view of the above-mentioned technical problems and defects, the purpose of this invention is to provide an autonomous driving takeover method based on a cognitive-execution dual-layer decision-making approach. This method collects environmental perception information around the vehicle, the vehicle's state, the driver's state, and the health status of the intelligent driving system. At the cognitive layer, multimodal deep features are extracted and fused using a cross-attention mechanism. Based on a multimodal large model, risk assessment, takeover decision parameters, safe operating boundaries, and natural language prompts are output. The necessity and urgency of takeover determine whether to trigger takeover and the appropriate takeover mode. At the execution layer, based on the takeover mode determined by the cognitive layer, multi-level warning signals guide the driver to prepare for takeover. Control weights are dynamically adjusted according to the driver's readiness state, achieving a smooth transfer of human-machine control under safety constraints. This cognitive-execution dual-layer architecture not only improves the accuracy and interpretability of takeover decisions but also effectively enhances the takeover performance of L2-L3 level autonomous driving and reduces takeover risks.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] An autonomous driving takeover method based on cognitive-executive two-layer decision-making includes the following steps:

[0007] Step S10. Collect environmental perception information, vehicle motion status information, driver status information, and intelligent driving system health monitoring information;

[0008] Step S20. Extract features from the multi-source information obtained in step S10, and then perform multimodal feature fusion;

[0009] Step S30. Based on the fusion features of step S20, reasoning is performed through a multimodal large model to output structured takeover decision information that includes risk assessment results, takeover decision parameters, safe operation boundaries, and natural language prompts;

[0010] Step S40. Determine whether to trigger takeover and the takeover mode based on the takeover decision parameters output in step S30. When it is determined that driver takeover is required, execute step S50.

[0011] Step S50. Based on the takeover mode determined by the cognitive layer, the execution layer guides the driver to prepare for takeover through multi-level warning signals;

[0012] Step S60. After the warning guidance signal in step S50 is activated, the execution layer, based on the real-time feedback of the driver's readiness status, gradually realizes the smooth transfer of human-machine control through risk-adaptive control weight allocation, control fusion under safety constraints, and real-time visual guidance.

[0013] As a preferred embodiment of the present invention, the environmental perception information includes image data collected by a visual sensor, point cloud data collected by a lidar, and target detection data collected by a millimeter-wave radar; the vehicle motion status information includes vehicle speed, acceleration, and heading angle; the driver status information includes eye movement trajectory, head posture, and steering wheel grip strength; and the intelligent driving system health monitoring information includes sensor operating status, positioning accuracy, and computing power load rate.

[0014] As a preferred embodiment of the present invention, step S20 includes the following steps:

[0015] Step S201. Input the acquired environmental perception information, vehicle motion state information, driver state information, and intelligent driving system health monitoring information into the environmental perception fusion encoder, vehicle dynamics encoder, driver state encoder, and intelligent driving system monitoring information encoder for feature extraction, respectively;

[0016] Step S202. After concatenating the four types of feature vectors, a cross-attention mechanism is used to fuse the extracted multimodal features.

[0017] As a preferred embodiment of the present invention, in step S30, the risk assessment result R includes the severity of the risk and the distribution of the risk source boundaries; the takeover decision parameter D includes the necessity of takeover, the urgency, and the available takeover time; the safe operating boundary B includes the speed safety boundary, the acceleration safety boundary, and the position safety boundary; and the natural language prompt is generated by the natural language generation module based on the risk assessment result R, the takeover decision parameter D, and the safe operating boundary B, and is used to convert the decision information into text prompts that the driver can understand.

[0018] As a preferred embodiment of the present invention, in step S40, the necessity of the control system output in step S30 is considered. and urgency Determine whether to trigger takeover and select the takeover mode:

[0019] ;

[0020] In the formula, Threshold for the necessity of takeover; The threshold for low urgency; This is the high urgency threshold.

[0021] As a preferred embodiment of the present invention, step S50 includes the following steps:

[0022] Step S501. Based on the severity of the risk in step S30 Available takeover time and the takeover mode of step S40 Calculate the takeover warning intensity over time. :

[0023] ;

[0024] In the formula, The takeover trigger moment; For urgency adjustment factor; exp It is an exponential function; The basic warning intensity is determined based on the takeover mode;

[0025] Step S502. Select one or more of the driver perception reminder modes to activate based on the takeover warning intensity calculated in step S501. The driver perception reminder modes include visual reminder mode, auditory reminder mode, and tactile reminder mode.

[0026] As a preferred embodiment of the present invention, step S60 includes the following steps:

[0027] Step S601. Calculate the driver readiness H(t) based on the driver status information and assess the driver's takeover readiness status;

[0028] Step S602. Based on the driver readiness H(t) obtained in step S601, calculate the driver's adaptive control weights. ;

[0029] Step S603. Based on the driver adaptive control weights obtained in step S602 A smooth transfer of human-machine control under security constraints is achieved through a control fusion mechanism:

[0030] Fusion control signals The final control signal after safety constraint processing should meet the safe operating boundary B(t) output in step S30. for:

[0031] ;

[0032] In the formula, The projection operator to the safety boundary ensures that the control signal is always within the safe operating boundary.

[0033] Step S604. During the control fusion process of executing step S603, the takeover decision information output in step S30 is converted into an intuitive guidance interface to assist the driver in completing the takeover operation.

[0034] As a further preferred embodiment of the present invention, the expression for the driver readiness degree H(t) is:

[0035] ;

[0036] In the formula, Alertness level; In a holding position; To increase visual attention; , , These are the weights for alertness, gripping status, and eye contact.

[0037] As a further preferred embodiment of the present invention, the driver's adaptive control weights The expression is:

[0038] ;

[0039] In the formula, Here is the transition rate parameter; t is the current time. At the moment of weight transfer center, the value is [value]. ,in The takeover trigger moment.

[0040] This invention also provides an autonomous driving takeover system based on cognitive-execution dual-layer decision-making. The autonomous driving takeover system includes a data acquisition module, a data processing module, a multimodal large model, a takeover mode judgment module, a warning intensity calculation module, a perception reminder module, a readiness state assessment module, an adaptive weight allocation module, a control signal output module, and a guidance interface generation module. The data acquisition module is connected to an onboard sensor system and is used to collect environmental perception information, vehicle motion state information, driver state information, and intelligent driving system health monitoring information.

[0041] The data processing module is used to extract features from the collected data and perform multimodal feature fusion processing using a cross-attention mechanism.

[0042] The multimodal large model is used to perform reasoning based on the fused features and output structured takeover decision information that includes risk assessment, takeover decision parameters, safe operation boundaries and natural language prompts.

[0043] The takeover mode determination module is used to determine whether to trigger takeover and the takeover mode based on the takeover decision parameters output by the multimodal large model.

[0044] The warning intensity calculation module is used to calculate the time-varying warning intensity based on the risk assessment, takeover decision parameters, and determined takeover mode output by the multimodal large model.

[0045] The perception and alert module is used to selectively activate visual, auditory, and tactile alert modes according to the intensity of the takeover warning, to remind the driver to prepare for takeover.

[0046] The readiness assessment module is used to assess the driver's readiness to take over after the reminder mode is activated.

[0047] The adaptive weight allocation module is used to calculate the driver's adaptive control weights based on the driver's readiness level.

[0048] The control signal output module, based on the driver's adaptive control weights, achieves a smooth transfer of human-machine control under safety constraints through a control fusion mechanism, and outputs the final control signal.

[0049] The guidance interface generation module is used to generate a guidance interface based on takeover decision parameters, safe operation boundaries, and natural language prompts, and to present the drivable area, the location of the risk source, and operation suggestions in real time.

[0050] The advantages and beneficial effects of this invention are:

[0051] (1) This invention proposes a cognitive-execution dual-layer decision-making architecture for autonomous driving takeover. This method divides the takeover process into high-level decision-making in the cognitive layer and low-level control in the execution layer. The cognitive layer outputs risk assessment, takeover decision parameters, safe operation boundaries and natural language prompts based on multimodal large model inference, and determines whether to trigger takeover and the takeover mode according to the necessity and urgency of takeover. The execution layer guides the driver to prepare for takeover based on the takeover mode determined by the cognitive layer through multi-level warning signals, and dynamically adjusts the control weight according to the driver's preparation status. Under safety constraints, the human-machine control is smoothly transferred. The cognitive-execution dual-layer architecture not only improves the accuracy and interpretability of takeover decisions, but also effectively improves the takeover performance of L2-L3 level autonomous driving, reduces takeover risks, breaks through the limitations of traditional single decision-making mode, and provides a new hierarchical decision-making framework for autonomous driving takeover in complex scenarios.

[0052] (2) This invention proposes a takeover cognitive reasoning method based on a multimodal large model. The output structure is specially designed according to the takeover control requirements. The trained model can provide specific parameters such as clear takeover trigger conditions, time constraints, and safety boundaries, so that the driver can understand "why take over" (through R risk assessment and T semantic information) and "how to take over" (through D decision parameters and B safety boundaries). This ensures the completeness and executability of the takeover decision output and solves the technical problem that the existing takeover decision process lacks interpretability, drivers have difficulty understanding the system's intent, and the human-machine trust relationship is seriously affected.

[0053] (3) In this invention, MLLM achieves accurate identification, risk assessment and takeover necessity judgment of complex scenarios through deep semantic understanding, overcomes the shortcomings of traditional rule-based methods that are poorly adaptable to unpreset scenarios and are difficult to deal with complex unpreset scenarios, and significantly improves the accuracy and generalization ability of takeover decisions.

[0054] (4) This invention organically combines the scene understanding capability of MLLM with the real-time control requirements. Through a two-layer architecture, it achieves the collaborative optimization of high-level cognition and low-level control, realizes the smoothness (through weight gradual change) and security (through boundary constraints) of the takeover process, and improves the practicality and reliability of the system.

[0055] (5) This invention proposes an adaptive control weight allocation scheme, which adjusts the human-machine control allocation ratio in real time based on the driver's readiness, solves the problem of vehicle instability caused by sudden changes in control rights in traditional takeover methods, and realizes a smooth and safe transfer of control rights.

[0056] (6) The present invention integrates a natural language generation module and a real-time visual guidance interface, which converts the decision information of the cognitive layer into semantic prompts and intuitive visual guidance that the driver can understand, significantly improving the interpretability of takeover decisions and the efficiency of human-computer interaction, and reducing the cognitive load and operational risks of the driver.

[0057] (7) This invention proposes a multi-mode takeover strategy adaptive selection mechanism, which selects progressive, collaborative and emergency takeover modes according to the necessity and urgency of takeover, and achieves the optimal balance between takeover efficiency and safety under different risk levels.

[0058] (8) The present invention introduces safe operation boundary constraints to ensure that the entire takeover process is always within the safe domain, effectively reducing the takeover risk. Attached Figure Description

[0059] Figure 1 This is an overall flowchart of the autonomous driving takeover method based on cognitive-execution two-layer decision-making provided in the embodiments of the present invention;

[0060] Figure 2 This is a structural diagram of the multimodal large model (MLLM) of the cognitive layer in an embodiment of the present invention;

[0061] Figure 3 This is a flowchart of the takeover control process of the execution layer in an embodiment of the present invention;

[0062] Figure 4 This is a structural block diagram of an autonomous driving takeover system based on a two-layer cognitive-execution decision-making system provided in an embodiment of the present invention. Detailed Implementation

[0063] To enable those skilled in the art to better understand the technical solutions and advantages of the present invention, the present application will be described in detail below with reference to the accompanying drawings, but this is not intended to limit the scope of protection of the present invention.

[0064] Example 1:

[0065] Figure 1A flowchart of an autonomous driving takeover method based on a cognitive-execution two-layer decision-making approach provided by this invention is shown below. Figure 1 As shown, the autonomous driving takeover method includes the following steps:

[0066] Step S10. Collect environmental perception information, vehicle motion status information, driver status information, and intelligent driving system (hereinafter referred to as intelligent driving system) health monitoring information through the vehicle sensor system;

[0067] Specifically, in this embodiment, the environmental perception information includes, but is not limited to, image data collected by visual sensors, point cloud data collected by lidar, and target detection data collected by millimeter-wave radar; the vehicle motion state information includes, but is not limited to, dynamic parameters such as vehicle speed, acceleration, and heading angle; the driver state information includes, but is not limited to, driver behavior state parameters such as eye movement trajectory, head posture, and steering wheel grip strength; and the intelligent driving system health monitoring information includes, but is not limited to, the working status of each sensor, positioning accuracy, and computing load rate.

[0068] Step S20. The cognitive layer extracts features from the multi-source information obtained in step S10 through the corresponding feature encoder and performs multimodal feature fusion processing using a cross-attention mechanism.

[0069] like Figure 2 As shown, in this embodiment, step S20 includes the following steps:

[0070] Step S201. Input the acquired data information into the corresponding feature encoder for feature extraction;

[0071] For environmental perception information, an environmental perception fusion encoder is used to output a unified scene feature vector. :

[0072]

[0073] In the formula, An environmental perception fusion encoder; Visual data; Point cloud data; For target detection data.

[0074] Specifically, in this embodiment, a convolutional neural network is used to extract visual features, a point cloud feature extraction network is used to process point cloud data, a multilayer perceptron is used to process target detection data, and the extracted features are concatenated and then input into a fully connected layer, finally outputting a unified scene feature vector. The expression is:

[0075]

[0076] In the formula, This represents a convolutional neural network used to extract visual features; This represents a point cloud feature extraction network used to extract point cloud features. This represents a multilayer perceptron, used to process target detection data; For feature splicing operations; As weight; For bias; It is the activation function for GELU (Gaussian Error Linear Unit).

[0077] For vehicle motion state information, a vehicle dynamics encoder is used to output motion pattern feature vectors. :

[0078]

[0079] In the formula, For vehicle dynamics encoders; This provides information on the vehicle's motion status.

[0080] Specifically, in this embodiment, the motion state vector at each time step is first obtained, then local temporal features are extracted through one-dimensional convolution, and then temporal dependencies are captured through a long short-term memory network to output a motion pattern feature vector. The expression is:

[0081]

[0082] In the formula, Let be the motion state vector at time t; Represents one-dimensional convolution, used to extract local temporal features; LSTM This is a Long Short-Term Memory network used to capture temporal dependencies; n is the length of the time window.

[0083] For driver state information, a driver state encoder is used to output a driver state feature vector. :

[0084]

[0085] In the formula, For driver status encoder; This is the driver's status information.

[0086] Specifically, in this embodiment, features are first extracted from eye movement, head posture, and grip strength data, then multi-source features are adaptively fused through an attention mechanism, and finally the driver state feature vector is output. The expression is:

[0087]

[0088] In the formula, , , These are data on eye movement, head posture, and grip strength, respectively. , , For the corresponding feature extraction function; Attention An attention mechanism is used to achieve adaptive fusion of multi-source features.

[0089] For the health monitoring information of the intelligent driving system, the intelligent driving system monitoring information feature vector is output through the intelligent driving system monitoring information encoder. :

[0090]

[0091] In the formula, Encoder for monitoring information in intelligent driving systems; This is information monitored by the intelligent driving system.

[0092] Specifically, in this embodiment, the health monitoring information of the intelligent driving system is first normalized, and then output as a feature vector of the intelligent driving system monitoring information after passing through two fully connected layers. The expression is:

[0093]

[0094] In the formula, This is the normalized system state vector; , This is the weight matrix; , It is the bias vector; To modify the activation function of the linear unit.

[0095] It should be noted that the feature extraction process for environmental perception information, vehicle motion state information, driver state information, and intelligent driving system health monitoring information in this embodiment is only for illustrative purposes and is not intended to limit the present invention. Those skilled in the art can refer to the prior art to extract the above feature vectors.

[0096] Step S202. After concatenating the four types of feature vectors, a cross-attention mechanism is used to fuse the extracted multimodal features:

[0097]

[0098] In the formula, The fused feature vector; This is a cross-attention fusion module.

[0099] Step S30. Based on the fusion features of step S20, reasoning is performed through a multimodal large model to output structured takeover decision information that includes risk assessment, takeover decision parameters, safe operation boundaries and natural language prompts;

[0100] Continue as Figure 2 As shown, in this embodiment, step S30 includes the following steps:

[0101] The fused feature vector obtained in step S20 The input is fed into a pre-trained multimodal large model (MLLM), which outputs structured takeover decision information. :

[0102]

[0103] In the formula, This is the result of a risk assessment. For takeover decision parameters; For safe operating boundaries; Provides natural language prompts.

[0104] The risk assessment result R includes the risk level (risk severity) and the distribution of risk source boundaries:

[0105]

[0106] In the formula, To assess the severity of the risk; This represents the boundary location of each risk source.

[0107] Takeover decision parameter D contains key parameters for takeover control:

[0108]

[0109] In the formula, The necessity of taking over; In terms of urgency; This is the available takeover time.

[0110] Safe operating boundary B provides a clear safety assurance boundary for the subsequent takeover process:

[0111]

[0112] In the formula, For speed safety boundaries; For acceleration safety boundaries; For location safety boundaries.

[0113] Natural Language Prompts It is generated by a natural language generation module based on risk assessment results R, takeover decision parameters D, and safe operating boundaries B.

[0114]

[0115] In the formula, The natural language generation module converts decision information into text prompts that drivers can understand.

[0116] In this embodiment, the collected data is used to train a multimodal large model (MLLM). For the training data, an expert system combined with real-world driving scenarios is used to construct the training set. Each driving scenario contains a structured annotation of the {R,D,B,T} quadruple. Specifically, the training dataset covers diverse operating conditions such as normal driving, takeover scenarios, and emergency scenarios, ensuring that the model can accurately output the aforementioned structured information. Regarding the model architecture, this embodiment largely follows the existing MLLM structure, focusing on the specialized design of the input encoding and output structure. Four dedicated encoders are set at the input end: EncoderScene (environmental perception fusion encoder), EncoderMotion (vehicle dynamics encoder), EncoderDriver (driver state encoder), and EncoderSystem (intelligent driving system monitoring information encoder). The feature vectors output by these encoders are fused through a cross-attention mechanism to form a unified fused feature F. At the output end, a structured output head is added to the MLLM, enabling the model to directly output structured decision information TD={R,D,B,T}. The specific training process for the model can refer to existing methods; after training, the trained model can directly output risk assessment results, takeover decision parameters, safe operation boundaries and natural language prompts based on the input data, so that the driver can understand "why take over" and also know "how to take over".

[0117] Step S40. Based on the necessity and urgency of takeover output in step S30, determine whether to trigger takeover and the takeover mode; when step S40 determines that driver takeover is required, execute step S50.

[0118] like Figure 3 As shown, in this embodiment, in step S40: based on the takeover necessity output in step S30... and urgency Determine whether to trigger takeover and select the takeover mode:

[0119]

[0120] In the formula, Threshold for the necessity of takeover; The threshold for low urgency; This is the high urgency threshold.

[0121] Step S50. Based on the takeover mode determined by the cognitive layer, the execution layer guides the driver to prepare for takeover through multi-level warning signals;

[0122] Continue as Figure 3 As shown, in this embodiment, step S50 includes the following steps:

[0123] Step S501. Based on the severity of the risk in step S30 Available takeover time and the takeover mode of step S40 Calculate the takeover warning intensity over time. :

[0124]

[0125] In the formula, The takeover trigger moment; As an urgency adjustment factor, ;exp It is an exponential function; The basic warning intensity is determined according to the takeover mode, and the value is determined as follows:

[0126]

[0127] In the formula, The level of basic early warning is set at 0.3. The basic warning level is set at 0.6, representing a medium level. The level of basic early warning is set to 1.0.

[0128] Step S502. Activate the corresponding driver perception alert mode based on the takeover warning intensity calculated in step S501:

[0129]

[0130] In the formula, Visual reminder mode; For auditory alert mode; Haptic alert mode; Activate the threshold for visual cues; Activate the threshold for auditory cues; Set the activation threshold for tactile cues.

[0131] Step S60. After the warning guidance signal in step S50 is activated, the execution layer, based on the real-time feedback of the driver's readiness status, gradually realizes the smooth transfer of human-machine control through risk-adaptive control weight allocation, control fusion under safety constraints, and real-time visual guidance.

[0132] Continue as Figure 3As shown, in this embodiment, step S60 includes the following steps:

[0133] Step S601. Based on the driver status information obtained in step S10, calculate the driver readiness level and assess the driver's takeover readiness status; define the driver readiness level function. :

[0134]

[0135] In the formula, Alertness level; In a holding position; To increase visual attention; , , These are the weights for alertness, gripping status, and eye contact.

[0136] Specifically, alertness Calculated using eye-tracking and head posture data:

[0137]

[0138] In the formula, The percentage of eyelid closure time at time t is calculated from eye movement trajectory data and has a value range of [0,1]. The angle at which the head deviates from directly forward is obtained through head posture data. The maximum deviation of the head from the threshold is typically set to 45°.

[0139] Holding state Calculated based on steering wheel grip strength data:

[0140]

[0141] In the formula, The value measured by the steering wheel grip force sensor at time t; The standard grip strength during normal driving is generally 15N.

[0142] visual attention Calculated using eye-tracking data:

[0143]

[0144] In the formula, The time during which the line of sight focuses on the road area within the time window; To assess the length of the time window, a 3-second window is typically used.

[0145] Step S602. Based on the driver readiness H(t) obtained in step S601, calculate the driver's adaptive control weights. :

[0146]

[0147] In the formula, The transition rate parameter has a value range of [1, 5]; t is the current time. At the moment of weight transfer center, the value is [value]. ,in The takeover trigger moment.

[0148] Step S603. Based on the driver adaptive control weights obtained in step S602 A smooth transfer of human-machine control under security constraints is achieved through a control fusion mechanism:

[0149]

[0150] In the formula, The fused control signal; For the driver's control input; It serves as the control input for the autonomous driving system.

[0151] In addition, the fused control signals The final control signal after safety constraint processing should satisfy the safety boundary B(t) defined in step S30. for:

[0152]

[0153] In the formula, The projection operator to the safety boundary ensures that the control signal is always within the safe operating boundary.

[0154] Step S604. During the control fusion process of step S603, the takeover decision information output in step S30 is converted into an intuitive guidance interface to assist the driver in completing the takeover operation:

[0155]

[0156] In the formula, This is the function for rendering the user interface. The generated boot screen, boot screen The head-up display or central control screen displays the drivable area, the location of risk sources, and operating suggestions in real time, allowing the driver to intuitively understand the current takeover status and safety boundaries, thus ensuring the safety of the takeover process.

[0157] Example 2:

[0158] Figure 4A structural block diagram of an autonomous driving takeover system based on a cognitive-execution two-layer decision-making mechanism provided by the present invention is shown below. Figure 4 As shown, the autonomous driving takeover system includes a data acquisition module, a data processing module, a multimodal large model, a takeover mode judgment module, a warning intensity calculation module, a perception and reminder module, a readiness status assessment module, an adaptive weight allocation module, a control signal output module, and a guidance interface generation module.

[0159] The data acquisition module is connected to the vehicle sensor system and is used to collect environmental perception information, vehicle motion status information, driver status information, and intelligent driving system health monitoring information.

[0160] The data processing module is used to extract features from the collected data and perform multimodal feature fusion processing using a cross-attention mechanism.

[0161] The multimodal large model is used to perform reasoning based on the fused features and output structured takeover decision information that includes risk assessment, takeover decision parameters, safe operation boundaries and natural language prompts.

[0162] The takeover mode determination module is used to determine whether to trigger takeover and the takeover mode based on the takeover decision parameters output by the multimodal large model.

[0163] The warning intensity calculation module is used to calculate the time-varying warning intensity based on the risk assessment, takeover decision parameters, and determined takeover mode output by the multimodal large model.

[0164] The perception and alert module is used to selectively activate visual, auditory, and tactile alert modes according to the intensity of the takeover warning, to remind the driver to prepare for takeover.

[0165] The readiness assessment module is used to assess the driver's readiness to take over after the reminder mode is activated.

[0166] The adaptive weight allocation module is used to calculate the driver's adaptive control weights based on the driver's readiness level.

[0167] The control signal output module, based on the driver's adaptive control weights, achieves a smooth transfer of human-machine control under safety constraints through a control fusion mechanism, and outputs the final control signal.

[0168] The guidance interface generation module is used to generate a guidance interface based on takeover decision parameters, safe operation boundaries, and natural language prompts. It presents the drivable area, the location of risk sources, and operation suggestions in real time, so that the driver can intuitively understand the current takeover status and safety boundaries, ensuring the safety of the takeover process.

[0169] The present invention also provides an electronic device, comprising: one or more processors and a memory; wherein the memory is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the above-described cognitive-execution two-layer decision-making method for autonomous driving takeover.

[0170] The present invention also provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described method for autonomous driving takeover based on a cognitive-executive two-layer decision-making process.

[0171] Those skilled in the art will understand that all or part of the functions of the various methods / modules in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the above functions can be implemented by executing the program with a computer. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be implemented.

[0172] In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the programs can also be stored in storage media such as servers, other computers, disks, optical discs, flash drives, or portable hard drives. They can be downloaded or copied to the memory of the local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be implemented.

[0173] The above-described specific examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention. Therefore, the scope of protection of this invention should be determined by the scope of the claims.

Claims

1. An autonomous driving takeover method based on a cognitive-execution two-layer decision-making process, characterized in that, The method includes the following steps: Step S10. Collect environmental perception information, vehicle motion status information, driver status information, and intelligent driving system health monitoring information; Step S20. Extract features from the multi-source information obtained in step S10, and then perform multimodal feature fusion; Step S30. Based on the fusion features of step S20, reasoning is performed through a multimodal large model to output structured takeover decision information that includes risk assessment results, takeover decision parameters, safe operation boundaries, and natural language prompts; Step S40. Determine whether to trigger takeover and the takeover mode based on the takeover decision parameters output in step S30. When it is determined that driver takeover is required, execute step S50. Step S50. Based on the takeover mode determined by the cognitive layer, the execution layer guides the driver to prepare for takeover through multi-level warning signals; Step S60. After the warning guidance signal in step S50 is activated, the execution layer, based on real-time feedback of the driver's readiness status, gradually achieves a smooth transfer of human-machine control through risk-adaptive control weight allocation, control fusion under safety constraints, and real-time visual guidance. Specifically, this includes the following steps: Step S601. Based on the driver's status information, calculate the driver's readiness H(t) and assess the driver's readiness to take over; the expression for the driver's readiness H(t) is: H(t)=w1·h alert (t)+w2·h hand (t)+w3·h gaze (t); In the formula, h alert (t) represents alertness level; h hand (t) represents the holding state; h gaze (t) represents the degree of eye attention; w1, w2, and w3 are the weights for alertness, gripping status, and eye attention, respectively. Step S602. Based on the driver readiness H(t) obtained in step S601, calculate the driver's adaptive control weight α(t); the expression for the driver's adaptive control weight α(t) is: In the formula, k is the transition rate parameter; t is the current time; t c The time of the weight transfer center is t. c =t0+0.5·t w Where t0 is the takeover trigger time; Step S603. Based on the driver adaptive control weight α(t) obtained in step S602, a smooth transfer of human-machine control under safety constraints is achieved through a control fusion mechanism: you fusion =α(t)·u human +(1-α(t))·u auto ; In the formula, u fusion The fused control signal; u human For the driver's control input; u auto For the control input of the autonomous driving system; The fused control signal u fusion The final control signal u after safety constraint processing should meet the safe operating boundary output in step S30. safe for: In the formula, Π B The projection operator is used to ensure that the control signal is always within the safe operating boundary. B(t) is the safe operating boundary at time t. Step S604. During the control fusion process of executing step S603, the takeover decision information output in step S30 is converted into an intuitive guidance interface to assist the driver in completing the takeover operation.

2. The autonomous driving takeover method based on cognitive-execution two-layer decision-making according to claim 1, characterized in that, The environmental perception information includes image data collected by visual sensors, point cloud data collected by lidar, and target detection data collected by millimeter-wave radar; the vehicle motion status information includes vehicle speed, acceleration, and heading angle; the driver status information includes eye movement trajectory, head posture, and steering wheel grip strength; and the intelligent driving system health monitoring information includes sensor operating status, positioning accuracy, and computing power load rate.

3. The autonomous driving takeover method based on cognitive-execution two-layer decision-making according to claim 1, characterized in that, Step S20 includes the following steps: Step S201. Input the acquired environmental perception information, vehicle motion state information, driver state information, and intelligent driving system health monitoring information into the environmental perception fusion encoder, vehicle dynamics encoder, driver state encoder, and intelligent driving system monitoring information encoder for feature extraction, respectively. Step S202. After concatenating the four types of feature vectors, a cross-attention mechanism is used to fuse the extracted multimodal features.

4. The autonomous driving takeover method based on cognitive-execution two-layer decision-making according to claim 1, characterized in that, In step S30, the risk assessment result R includes the severity of the risk and the distribution of the risk source boundaries, and the takeover decision parameter D includes the necessity of takeover, the urgency, and the available takeover time. The safe operating boundary B includes the speed safety boundary, acceleration safety boundary, and position safety boundary. Natural language prompts are generated by the natural language generation module based on the risk assessment result R, the takeover decision parameter D, and the safe operating boundary B, and are used to convert the decision information into text prompts that the driver can understand.

5. The autonomous driving takeover method based on cognitive-execution two-layer decision-making according to claim 4, characterized in that, In step S40, based on the takeover necessity n output in step S30... t Based on the urgency level (u), determine whether to trigger takeover and select the takeover mode (Mode): Takeover mode In the formula, n th Threshold for takeover necessity; u th1 The threshold for low urgency; u th2 This is the high urgency threshold.

6. The autonomous driving takeover method based on cognitive-execution two-layer decision-making according to claim 4, characterized in that, Step S50 includes the following steps: Step S501. Based on the risk severity r and available takeover time t from step S30. w And the takeover mode Mode in step S40, calculating the takeover warning intensity I that changes over time. warn (t): In the formula, t0 is the takeover trigger time; t is the current time; β is the urgency adjustment factor; exp(·) is the exponential function; I0 is the basic warning intensity, which is determined according to the takeover mode; Step S502. Select one or more of the driver perception reminder modes to activate based on the takeover warning intensity calculated in step S501. The driver perception reminder modes include visual reminder mode, auditory reminder mode, and tactile reminder mode.

7. An autonomous driving takeover system based on a cognitive-execution two-layer decision-making process, characterized in that, The autonomous driving takeover system includes a data acquisition module, a data processing module, a multimodal large model, a takeover mode judgment module, a warning intensity calculation module, a perception and reminder module, a readiness status assessment module, an adaptive weight allocation module, a control signal output module, and a guidance interface generation module. The data acquisition module is connected to the vehicle sensor system and is used to collect environmental perception information, vehicle motion status information, driver status information, and intelligent driving system health monitoring information. The data processing module is used to extract features from the collected data and perform multimodal feature fusion processing using a cross-attention mechanism. The multimodal large model is used to perform reasoning based on the fused features and output structured takeover decision information that includes risk assessment, takeover decision parameters, safe operation boundaries and natural language prompts. The takeover mode determination module is used to determine whether to trigger takeover and the takeover mode based on the takeover decision parameters output by the multimodal large model. The warning intensity calculation module is used to calculate the time-varying warning intensity based on the risk assessment, takeover decision parameters, and determined takeover mode output by the multimodal large model. The perception and alert module is used to selectively activate visual, auditory, and tactile alert modes according to the intensity of the takeover warning, to remind the driver to prepare for takeover. The readiness assessment module is used to assess the driver's readiness to take over after the reminder mode is activated; the expression for the driver's readiness H(t) is: H(t)=w1·h alert (t)+w2·h hand (t)+w3·h gaze (t); In the formula, h alert (t) represents alertness level; h hand (t) represents the holding state; h gaze (t) represents the degree of eye attention; w1, w2, and w3 are the weights for alertness, gripping status, and eye attention, respectively. The adaptive weight allocation module is used to calculate the driver's adaptive control weights based on the driver's readiness level; the expression for the driver's adaptive control weight α(t) is: In the formula, k is the transition rate parameter; t is the current time; t c The time of the weight transfer center is t. c =t0+0.5·t w Where t0 is the takeover trigger time; The control signal output module, based on the driver's adaptive control weights, achieves a smooth transfer of human-machine control under safety constraints through a control fusion mechanism, and outputs the final control signal. you fusion =α(t)·u human +(1-α(t))·u auto ; In the formula, u fusion The fused control signal; u human For the driver's control input; u auto For the control input of the autonomous driving system; The fused control signal u fusion The final control signal u should meet the safe operating boundaries and be processed by safety constraints. safe for: In the formula, Π B The projection operator is used to ensure that the control signal is always within the safe operating boundary. B(t) is the safe operating boundary at time t. The guidance interface generation module is used to generate a guidance interface based on takeover decision parameters, safe operation boundaries, and natural language prompts, and to present the drivable area, the location of the risk source, and operation suggestions in real time.

Images