Man-machine co-driving control method based on non-intrusive electroencephalogram perception, vehicle, medium and product

Abstract

The application discloses a non-invasive electroencephalogram-based human-machine co-driving control method, a vehicle, a medium and a product. The method comprises the following steps: acquiring non-invasive electroencephalogram signals of a driver, driver state information and vehicle environment information; determining an implicit intention recognition result of the driver based on the non-invasive electroencephalogram signals, the driver state information and the vehicle environment information, wherein the implicit intention recognition result is used to represent a cognitive state or a driving decision tendency of the driver which is not reflected in a driver-side control input; determining a driver control weight and a vehicle intelligent driving system control weight according to the implicit intention recognition result; acquiring the driver-side control input and the vehicle intelligent driving system-side control input; calculating a vehicle control instruction according to the driver control weight, the vehicle intelligent driving system control weight, the driver-side control input and the vehicle intelligent driving system-side control input; and outputting the vehicle control instruction to a vehicle execution system.

Description

Technical Field

[0001] This application relates to the field of vehicle control technology, and in particular to a human-machine co-driving control method, vehicle, medium and product based on non-invasive EEG perception. Background Technology

[0002] With the development of advanced driver assistance systems (ADAS) and intelligent driving technologies, vehicles can identify road targets, traffic participants, and driving risks based on environmental perception devices such as cameras and millimeter-wave radar, and combine this with driver status monitoring results for assisted control. In human-machine co-driving scenarios, vehicle control typically needs to consider both the driver's operational intentions and the control strategies of the onboard intelligent driving system to improve the coordination and safety of the vehicle during driving.

[0003] In related technologies, driver condition monitoring often relies on external devices such as cameras, posture sensors, steering wheel or pedal sensors to identify external behaviors such as closing eyes, nodding, body tilting, steering wheel operation, and pedal operation. While these solutions can identify fatigue, distraction, or abnormal operation that has already manifested, their ability to identify cognitive states, risk assessments, or decision-making tendencies that the driver has not yet expressed through external control input is limited.

[0004] Furthermore, some EEG monitoring solutions are primarily used to determine fatigue levels or identify driving intentions, typically only outputting status prompts or auxiliary judgment results, without further utilizing the real-time driver status reflected in the EEG signals for human-machine co-driving control. Consequently, in scenarios involving complex road conditions, potential risks, or unclear driver operations, issues such as mismatched control strategies, abrupt control switching, or insufficient coordination may arise between the in-vehicle intelligent driving system and the driver.

[0005] Therefore, there is a need for a technical solution that can identify the driver's implicit intentions based on non-invasive EEG perception and use these implicit intentions to determine the control weights for human-machine co-driving and generate vehicle control commands. Summary of the Invention

[0006] This application provides a human-machine co-driving control method, vehicle, medium, and product based on non-invasive EEG perception, to solve the problem in related technologies that it is difficult to use the cognitive state or driving decision tendency of the driver that is not reflected in the control input for human-machine co-driving control, resulting in insufficient coordination between the driver and the vehicle intelligent driving system.

[0007] To achieve the above objectives, some embodiments of this application provide the following aspects:

[0008] In a first aspect, some embodiments of this application provide a human-machine co-driving control method based on non-invasive EEG sensing, the method comprising:

[0009] Acquire non-invasive EEG signals from the driver, driver status information, and vehicle environment information;

[0010] Based on the non-invasive EEG signal, the driver's state information, and the vehicle environment information, the driver's implicit intention recognition result is determined. The implicit intention recognition result is used to characterize the driver's cognitive state or driving decision tendency that is not reflected in the driver's side control input.

[0011] Based on the implicit intent recognition results, the driver control weight and the in-vehicle intelligent driving system control weight are determined;

[0012] Acquire driver-side control inputs and vehicle-side control inputs from the intelligent driving system;

[0013] The vehicle control command is calculated based on the driver control weight, the vehicle intelligent driving system control weight, the driver-side control input, and the vehicle intelligent driving system-side control input.

[0014] The vehicle control commands are output to the vehicle execution system.

[0015] Optionally, based on the non-invasive EEG signals, the driver's state information, and the vehicle environment information, the result of determining the driver's implicit intention is determined, including:

[0016] The non-invasive EEG signal is denoised, and EEG features are extracted from the denoised non-invasive EEG signal.

[0017] The driver's external operating status is determined based on the driver status information;

[0018] Determine the driving risk status around the vehicle based on the vehicle environment information;

[0019] When no control change occurs in the driver's external operating state corresponding to the driving risk state, the driver's implicit intention recognition result is determined based on the EEG characteristics.

[0020] The above scheme allows for the use of EEG characteristics to determine the driver's implicit intentions when there are driving risks around the vehicle but no corresponding changes in the driver's external operating state. This enables the driver to incorporate cognitive states that have not yet been expressed through control inputs such as steering, braking, or acceleration into the basis of control decisions.

[0021] Optionally, determining the driver's implicit intention recognition result based on the EEG characteristics includes:

[0022] When there are no abnormalities in the driver's eye movements, limbs, and steering wheel operation, and the proportion of alpha waves in the brain increases and is accompanied by a decrease in the rate of eye swabbing, the implicit intention recognition result is determined to be implicit attentional distraction.

[0023] When the driver does not exhibit overt fatigue and the overall brainwave energy decreases while the proportion of slow waves increases, the implicit intent recognition result is determined to be implicit fatigue.

[0024] When there are potential risk sources around the vehicle, the driver does not make steering, braking or acceleration operations, and the activity of brain beta waves is increased, the implicit intention recognition result is determined to be a risk prediction intention.

[0025] When the vehicle is in a complex driving scenario, the driver's brain waves fluctuate between alpha and beta waves, accompanied by slight fluctuations in steering wheel grip force and high-frequency eye saccades, the implicit intention recognition result is determined to be a decision hesitation intention.

[0026] The above scheme can establish corresponding judgment rules for implicit attention distraction, implicit fatigue, risk prediction intention and decision hesitation intention, so that implicit intention recognition is no longer limited to generalized driving state classification, but combines EEG characteristics, driver state information and vehicle environment information to form an executable recognition logic.

[0027] Optionally, based on the implicit intent recognition result, the driver control weight and the in-vehicle intelligent driving system control weight are determined, including:

[0028] Determine the intent category and intent confidence level corresponding to the implicit intent recognition result;

[0029] The direction of weight adjustment is determined based on the intent category, and the magnitude of weight adjustment is determined based on the intent confidence level.

[0030] The driver control weight and the vehicle intelligent driving system control weight are updated according to the weight adjustment direction and the weight adjustment range.

[0031] With the above scheme, the driver control weight and the vehicle intelligent driving system control weight are not fixed configurations, but are dynamically updated according to the implicit intent recognition results, so that human-machine co-driving control can adjust the degree of participation of both parties in vehicle control according to the driver's current cognitive state or driving decision tendency.

[0032] Optionally, the driver control weights and the in-vehicle intelligent driving system control weights are updated, including:

[0033] The control weights of the in-vehicle intelligent driving system are limited to a preset system weight range;

[0034] Between adjacent control cycles, the change in the control weight of the in-vehicle intelligent driving system is limited to within a preset change amount;

[0035] When the driver's active operation is detected, the control weight of the in-vehicle intelligent driving system will be reduced back to the weight of the basic system.

[0036] The above approach can avoid sudden changes in the control weight of the in-vehicle intelligent driving system, while reducing the control weight of the in-vehicle intelligent driving system when the driver actively operates it. This approach helps to balance system-assisted control and driver-led operation, reducing the discomfort caused by sudden takeover or control jumps.

[0037] Optionally, vehicle control commands are calculated based on the driver control weights, the in-vehicle intelligent driving system control weights, the driver-side control inputs, and the in-vehicle intelligent driving system-side control inputs, including:

[0038] The driver-side control component is determined based on the driver control weight and the driver-side control input;

[0039] The system-side control components are determined based on the control weights of the in-vehicle intelligent driving system and the control inputs of the in-vehicle intelligent driving system.

[0040] Based on the driver-side control component and the system-side control component, at least one of the steering control command, braking control command, and acceleration control command is calculated to obtain the vehicle control command.

[0041] Through the above scheme, dynamic control weights can be specifically applied to vehicle control quantities such as steering, braking and acceleration, so as to establish a clear control link between the implicit intent recognition result and the vehicle execution control, rather than just serving as status prompt information.

[0042] Optionally, after calculating the vehicle control commands, the method further includes:

[0043] Determine whether there is a conflict between the driver-side control input and the vehicle intelligent driving system-side control input;

[0044] In the event of a conflict, the arbitration priority shall be determined based on the driver's operation type, vehicle environment information, and implicit intent recognition results.

[0045] The vehicle control command is modified according to the arbitration priority, and the modified vehicle control command is output to the vehicle execution system.

[0046] The above scheme allows for arbitration of control commands when the driver's control input and the vehicle's intelligent driving system control input are inconsistent. This can be achieved by combining the driver's operation type, vehicle environment information, and implicit intent recognition results, which helps improve the rationality and continuity of vehicle control commands in conflict scenarios.

[0047] Secondly, some embodiments of this application also provide a vehicle, which includes a non-invasive EEG acquisition device, a driver state acquisition device, a vehicle environment acquisition device, a vehicle control system, and a vehicle execution system.

[0048] The non-invasive EEG acquisition device is used to acquire non-invasive EEG signals from the driver.

[0049] The driver status acquisition device is used to collect driver status information;

[0050] The vehicle environment acquisition device is used to collect vehicle environment information;

[0051] The vehicle control system is used to execute the method described in any of the above-mentioned methods to obtain vehicle control commands;

[0052] The vehicle execution system is used to execute the vehicle control commands.

[0053] Thirdly, some embodiments of this application also provide a computer-readable medium having computer program instructions stored thereon, which can be executed by a processor to implement the steps of the method described above.

[0054] Fourthly, some embodiments of this application also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described above.

[0055] Compared with related technologies, the solution provided in this application obtains the driver's non-invasive EEG signals, driver state information, and vehicle environment information to determine the driver's implicit intention recognition result. Based on the implicit intention recognition result, the driver's control weight and the vehicle intelligent driving system's control weight are determined. Then, the vehicle control command is calculated by combining the driver's control input and the vehicle intelligent driving system's control input. This allows the driver's cognitive state or driving decision tendency not reflected in the control input to participate in vehicle control, which is beneficial to improving the degree of coordination between the driver and the vehicle intelligent driving system during human-machine co-driving and reducing the risks caused by control mutations. Attached Figure Description

[0056] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0057] Figure 1 A flowchart illustrating a human-machine co-driving control method based on non-invasive EEG sensing, provided as an exemplary embodiment of this application;

[0058] Figure 2 This is a schematic diagram of the frame of a vehicle provided for an exemplary embodiment of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] In this embodiment of the disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information comply with relevant laws and regulations and do not violate public order and good morals.

[0061] Figure 1 A schematic diagram of a human-machine co-driving control method based on non-invasive EEG sensing, provided for an exemplary embodiment of this application, the method comprising:

[0062] S101. Obtain non-invasive EEG signals from the driver, driver status information, and vehicle environment information.

[0063] Specifically, non-invasive EEG signals can be the driver's brain EEG signals. Driver state information can include eye movement data, steering wheel grip force data, and driver operation data. Vehicle environment information can include information related to obstacles ahead, merging traffic, road conditions, inclement weather, or other potential risk sources. By acquiring the above information simultaneously, subsequent judgments can not only rely on external operations but also incorporate the real-time state reflected in the driver's brain signals.

[0064] In one embodiment, the non-invasive EEG signal can be first denoised to filter out interference generated in dynamic in-vehicle scenarios such as vehicle bumps and electromagnetic interference. The denoised EEG signal is used to extract EEG features, driver state information is used to extract eye movement features, grip strength change features, or operational state features, and vehicle environment information is used to extract environmental risk features.

[0065] S102. Based on non-invasive EEG signals, driver state information, and vehicle environment information, determine the driver's implicit intention recognition result.

[0066] Specifically, EEG features, driver state features, and environmental risk features can be fused to obtain fused features, and the driver's intention category and its confidence level can be determined based on the fused features. Intention categories can include distraction, fatigue, hesitation, emergency, or normality. The results of implicit intention recognition can be further used to characterize the driver's cognitive state or driving decision-making tendency that is not reflected in the driver's side control input.

[0067] For example, when a driver's external actions show no obvious abnormalities, but EEG signals and eye movement data indicate a declining trend in attention, it can be determined that the driver has latent inattention. When a driver does not exhibit external fatigue-related actions such as nodding or slouching, but EEG signals show fatigue-related changes, it can be determined that the driver is experiencing latent fatigue. When environmental information indicates a potential risk ahead, and the driver does not take steering, braking, or acceleration actions, while EEG signals show risk-prediction-related changes, it can be determined that the driver intends to anticipate risk. When the vehicle is in a complex intersection, sudden cutting in, or emergency avoidance scenario, and the driver's input does not form a stable trend, accompanied by changes in EEG, grip strength, and eye movement, it can be determined that the driver intends to hesitate in decision-making.

[0068] S103. Based on the results of implicit intent recognition, determine the driver control weight and the control weight of the in-vehicle intelligent driving system.

[0069] Specifically, the direction of weight adjustment can be determined based on the intent category in the implicit intent recognition results, and the magnitude of weight adjustment can be determined based on the intent confidence level. If the driver is focused or in a normal state, the driver's control weight can be maintained at a high level, and the control weight of the in-vehicle intelligent driving system can be maintained at a basic assistance level. If the driver is in a state of implicit fatigue, implicit inattention, or decision hesitation, the control weight of the in-vehicle intelligent driving system can be increased, and the driver's control weight can be decreased accordingly.

[0070] Furthermore, the control weights of the in-vehicle intelligent driving system can be limited to a preset range, and the change in weights between adjacent control cycles can be limited to a preset change, allowing the weights to change continuously. When the driver actively performs steering wheel operations, braking operations, or other takeover operations, the control weights of the in-vehicle intelligent driving system can fall back to the weights of the basic system.

[0071] S104. Obtain driver-side control input and vehicle intelligent driving system-side control input.

[0072] Specifically, driver-side control inputs may include steering, braking, or acceleration inputs generated by the driver. In-vehicle intelligent driving system-side control inputs may include steering, braking, or acceleration inputs generated based on lane keeping, obstacle avoidance, following, deceleration, or other safety strategies. Both types of control inputs can be acquired within the same control cycle to calculate the vehicle control commands corresponding to the current control cycle.

[0073] S105. Calculate vehicle control commands based on driver control weights, vehicle intelligent driving system control weights, driver-side control inputs, and vehicle intelligent driving system-side control inputs.

[0074] Specifically, the influence of driver-side control inputs in the current control cycle can be determined based on driver control weights, and the influence of on-board intelligent driving system-side control inputs in the current control cycle can be determined based on on-board intelligent driving system control weights, thereby calculating vehicle control commands. Vehicle control commands may include at least one of steering control commands, braking control commands, and acceleration control commands.

[0075] Furthermore, when conflicts arise between driver-side control input and in-vehicle intelligent driving system-side control input, arbitration priority can be determined based on the driver's operation type, vehicle environmental information, and implicit intent recognition results. For example, when the driver performs an emergency operation, the driver-side control input can be responded to first; when the driver is operating normally but the environment is risky and implicit distraction or hesitation is detected, the driver-side control input can be corrected; when the driver is not operating and an abnormal state is detected, the influence of in-vehicle intelligent driving system-side control input on vehicle control commands can be increased.

[0076] S106. Output vehicle control commands to the vehicle execution system.

[0077] Specifically, vehicle control commands can be used to control steering, braking, or acceleration, so that the final control result simultaneously considers the driver-side control input, the on-board intelligent driving system-side control input, and the driver's state changes that are not directly reflected in the control input.

[0078] Through the above-described scheme, this application does not solely rely on EEG signals for driver status cues. Instead, it combines non-invasive EEG signals, driver status information, and vehicle environmental information to determine the driver's implicit intent. Furthermore, it determines the driver's control weights and the vehicle's intelligent driving system's control weights based on these implicit intent recognition results. Therefore, the calculation process for vehicle control commands considers not only the driver's control inputs generated through the steering wheel, brake pedal, or accelerator pedal, but also the vehicle's intelligent driving system-side control inputs generated by the system based on environmental perception and control strategies. It also considers the driver's cognitive state or driving decision-making tendencies not yet reflected in the driver's control inputs. This allows for dynamic adjustments to the degree of driver and system involvement in vehicle control when the driver is fatigued, distracted, experiencing risk assessment, or hesitant in decision-making. This makes the vehicle control results more closely aligned with the current driver and vehicle environment states, thereby improving the coordination between the driver and the intelligent driving system during human-machine co-driving and reducing problems such as mismatched control intents, abrupt control switching, or delayed auxiliary intervention caused by solely relying on external behavior monitoring or fixed control weights.

[0079] In one embodiment, determining the driver's implicit intention recognition result based on the non-invasive EEG signal, the driver's state information, and the vehicle environment information includes:

[0080] The non-invasive EEG signal is denoised, and EEG features are extracted from the denoised non-invasive EEG signal.

[0081] The driver's external operating status is determined based on the driver status information;

[0082] Determine the driving risk status around the vehicle based on the vehicle environment information;

[0083] When no control change occurs in the driver's external operating state corresponding to the driving risk state, the driver's implicit intention recognition result is determined based on the EEG characteristics.

[0084] Specifically, after acquiring non-invasive EEG signals, the signals can be denoised to filter out interference components caused by road bumps, electromagnetic interference, or changes in contact conditions during vehicle operation. After denoising, EEG features can be extracted from the non-invasive EEG signals, reflecting the driver's brain activity. Compared to directly using the raw EEG signals, this processing method improves the stability of the EEG data upon which subsequent implicit intent recognition is based.

[0085] The driver's state information can be used to determine the driver's external operating status. This external operating status can include the driver's eye movement, steering wheel grip changes, and whether the driver is performing steering, braking, or acceleration operations. By analyzing the driver's external operating status, it can be determined whether the driver has demonstrated a corresponding driving response through external actions or vehicle control inputs.

[0086] Vehicle environmental information can be used to determine the driving risk status around the vehicle. This driving risk status can be determined based on information such as surrounding targets, obstacles ahead, merging traffic, complex road scenarios, or inclement weather. When there are potential risk sources around the vehicle that require control responses from the driver or the onboard intelligent driving system, a corresponding driving risk status can be determined.

[0087] Furthermore, after determining that a driving risk state exists, it can be determined whether the driver's external operating state has undergone control changes corresponding to the driving risk state. For example, when there is an obstacle in front of the vehicle or approaching traffic, it can be determined whether the driver has performed steering, braking, or acceleration operations, or whether the driver's eye movement state and steering wheel grip state show changes related to risk avoidance or decision-making. If the driver's external operating state has not undergone control changes corresponding to the driving risk state, it indicates that the driver's relevant cognitive state or driving decision-making tendency has not been fully expressed through driver-side control input. In this case, the driver's implicit intention recognition result can be further determined based on EEG characteristics.

[0088] In this embodiment, by first denoising the non-invasive EEG signals and extracting EEG features, and then combining the driver's external operating state with the driving risk state around the vehicle for correlation judgment, the implicit intention recognition does not rely solely on EEG signals or solely on external behavior. Especially when there is a driving risk state around the vehicle but the driver's external operating state has not undergone corresponding control changes, using EEG features to determine the implicit intention recognition result can identify state changes or decision tendencies that the driver has not yet expressed through control inputs such as steering, braking, or acceleration. This reduces the recognition lag or missed judgments caused by relying solely on external behavior monitoring, and provides a more reliable basis for subsequently determining the driver's control weight and the control weight of the in-vehicle intelligent driving system.

[0089] Furthermore, in one embodiment, after obtaining the noise-reduced non-invasive EEG signal, driver state information, and vehicle environment information, features for implicit intent recognition can be extracted from the different information respectively.

[0090] Specifically, EEG feature vectors can be extracted from non-invasive EEG signals after noise reduction, eye movement feature vectors can be extracted from eye movement data in driver state information, steering wheel grip force behavior features can be extracted from steering wheel grip force behavior data, and environmental sensor features can be extracted from vehicle environment information.

[0091] Among these features, EEG feature vectors can reflect the driver's brain activity state, eye movement feature vectors can reflect the driver's saccades of the road environment, steering wheel grip behavior features can reflect the driver's hand control stability or operational hesitation, and environmental sensor features can reflect the state of road targets, obstacles, intersecting traffic, or other driving risks around the vehicle. By using features from these different sources together in subsequent identification, misjudgments caused by a single information source can be avoided.

[0092] In one embodiment, the multimodal features can be weighted and fused according to the following formula to obtain the fused features:

[0093]

[0094] in, As a feature of fusion, For EEG feature vectors, For eye-tracking feature vectors, Steering wheel grip behavior characteristics For environmental sensor features, α, β, γ, and δ are the fusion weights corresponding to EEG feature vector, eye movement feature vector, steering wheel grip force behavior feature, and environmental sensor features, respectively.

[0095] In one example It can be a 128-dimensional EEG feature vector. It can be a 32-dimensional eye-tracking feature vector. The system can utilize 64-dimensional environmental sensor features. α, β, γ, and δ can be optimized using a genetic algorithm; for example, α=0.6, β=0.25, γ=0.1, and δ=0.05 can be chosen. With this weighting configuration, EEG features have a relatively high proportion in the fused features, giving the driver's brain state a strong influence on implicit intent recognition. Simultaneously, eye movements, grip strength, and environmental features can also serve as auxiliary information in the judgment.

[0096] In this embodiment, by using the aforementioned multimodal feature fusion method, EEG features, eye movement features, steering wheel grip behavior features, and environmental sensor features can be converted into unified fused features, enabling subsequent intent confidence decoding to simultaneously consider driver brain activity, external state, and vehicle environment. This improves the adaptability of implicit intent recognition to complex dynamic in-vehicle scenarios.

[0097] In one embodiment, determining the driver's implicit intention recognition result based on the EEG characteristics includes:

[0098] When there are no abnormalities in the driver's eye movements, limbs, and steering wheel operation, and the proportion of alpha waves in the brain increases and is accompanied by a decrease in the rate of eye swabbing, the implicit intention recognition result is determined to be implicit attentional distraction.

[0099] When the driver does not exhibit overt fatigue and the overall brainwave energy decreases while the proportion of slow waves increases, the implicit intent recognition result is determined to be implicit fatigue.

[0100] When there are potential risk sources around the vehicle, the driver does not make steering, braking or acceleration operations, and the activity of brain beta waves is increased, the implicit intention recognition result is determined to be a risk prediction intention.

[0101] When the vehicle is in a complex driving scenario, the driver's brain waves fluctuate between alpha and beta waves, accompanied by slight fluctuations in steering wheel grip force and high-frequency eye saccades, the implicit intention recognition result is determined to be a decision hesitation intention.

[0102] Specifically, when determining implicit inattention, one can first assess whether there are external abnormalities in the driver's eye movements, limb movements, and steering wheel operation based on the driver's state information. If the driver does not exhibit external abnormalities such as closed eyes, obvious head tilting, body tilting, or abnormal steering wheel operation, but the proportion of alpha waves in the EEG characteristics is increased, and the eye movement data indicates a decreased scanning rate, then it can be determined that the driver's activity in receiving external road condition information is reduced. In this case, the implicit intent recognition result can be identified as implicit inattention.

[0103] When assessing latent fatigue, driver status information can be used to determine whether the driver exhibits overt fatigue behaviors. Overt fatigue behaviors can include external actions such as yawning, nodding, or leaning to one side. If the driver does not exhibit these overt fatigue behaviors, but EEG characteristics indicate a decrease in overall brainwave energy and an increase in the proportion of slow waves, then it can be determined that the driver's central nervous system activity is reduced. In this case, the latent intent recognition result can be identified as latent fatigue.

[0104] When determining the intent to anticipate risk, the presence of potential risk sources around the vehicle can be assessed based on vehicle environmental information. Potential risk sources may include obstacles ahead, oncoming traffic, or inclement weather. When potential risk sources exist around the vehicle, and the driver has not made any steering, braking, or acceleration actions, if EEG characteristics indicate increased beta wave activity, it can be determined that the driver has entered a risk warning-related cognitive state but has not yet responded through driver-side control input. In this case, the implicit intent recognition result can be identified as the risk anticipation intent.

[0105] When determining the intent to hesitate in decision-making, vehicle environmental information can be used to determine whether the vehicle is in a complex driving scenario. Complex driving scenarios can include complex intersections, sudden lane-cutting, or emergency swerving. If the vehicle is in a complex driving scenario, and EEG characteristics show alternating alpha and beta wave fluctuations, while steering wheel grip data shows slight fluctuations in driver grip strength, and eye movement data shows high-frequency saccades, then it can be determined that the driver is in a state of decision-making indecision. In this case, the implicit intent recognition result can be identified as an intent to hesitate in decision-making.

[0106] In this embodiment, by establishing corresponding judgment conditions for different implicit intentions, EEG features, driver state information, and vehicle environment information can be transformed into executable recognition rules. Compared to simply outputting a general fatigue level or driving state, this embodiment can distinguish between different states such as implicit inattention, implicit fatigue, risk prediction intention, and decision hesitation intention. This allows subsequent weight adjustments to adopt different control strategies for different implicit intentions, thereby improving the adaptability of human-machine co-driving control to the driver's actual state.

[0107] Furthermore, in one embodiment, after obtaining the fused features, the driver's intention category and its confidence level can be determined based on the fused features. Specifically, the fused features can be input into a lightweight Transformer model, which performs classification processing on the fused features to output the confidence levels corresponding to multiple driver intention categories.

[0108] In one embodiment, the confidence level for each driver intent category can be calculated using the following formula:

[0109]

[0110] in, Let be the confidence level of the driver's intention for the i-th type. As a feature of fusion, Here, b is the weight matrix for a lightweight Transformer model, b is the bias vector, and Softmax is used to convert the model output into a probability distribution corresponding to different intent categories. Driver intent categories can include distraction, fatigue, hesitation, emergency, and normal intent.

[0111] Specifically, the lightweight Transformer model can perform correlation analysis on different feature dimensions in the fused features based on the self-attention mechanism. For example, when EEG features, eye-tracking features, and environmental sensor features in the fused features all point to a state of fatigue or distraction, the confidence level of the corresponding intention category can be increased; when vehicle environmental information indicates a potential risk, but the driver's external control input does not produce a corresponding change, the model can output the confidence level of the risk-related or hesitation-related categories based on the fused features.

[0112] In one embodiment, the driver's intent category with the highest confidence level can be used as the current intent category. Alternatively, a preset confidence threshold can be used to determine whether to output the corresponding implicit intent recognition result. For example, when the confidence level of the fatigue category is higher than the preset threshold, it can be determined that the driver is in a fatigue-related state; when the confidence level of the hesitation category is higher than the preset threshold, and the vehicle environment information indicates that the vehicle is in a complex driving scenario, it can be determined that the driver has a decision-making hesitation intent.

[0113] Furthermore, the Transformer model can be lightweighted through model distillation, reducing the number of model parameters from a large scale to a size suitable for automotive deployment. In one example, the number of model parameters can be compressed from over 200M to around 15M, ensuring that the overall latency of the intent decoding process does not exceed 100ms. This enables real-time intent confidence decoding even with limited automotive computing resources.

[0114] In this embodiment, by inputting fused features into a lightweight Transformer model and outputting the confidence level of each intent category, the driver intent recognition result can be expressed in a quantifiable way. Subsequently, control weights can be calculated based on the intent category and intent confidence level, enabling human-machine co-driving control to be continuously adjusted according to the degree of change in the driver's state, rather than based on fixed state switching.

[0115] In one embodiment, determining the driver control weight and the in-vehicle intelligent driving system control weight based on the implicit intent recognition result includes:

[0116] Determine the intent category and intent confidence level corresponding to the implicit intent recognition result;

[0117] The direction of weight adjustment is determined based on the intent category, and the magnitude of weight adjustment is determined based on the intent confidence level.

[0118] The driver control weight and the vehicle intelligent driving system control weight are updated according to the weight adjustment direction and the weight adjustment range.

[0119] Specifically, the implicit intent recognition result can include intent category and intent confidence. Intent category is used to characterize the driver's current state type, such as distracted, fatigued, hesitant, urgent, or normal; intent confidence is used to characterize the credibility of the corresponding intent category. By simultaneously determining intent category and intent confidence, fixed control based on a single state label can be avoided, thus providing a continuous basis for subsequent weight adjustments.

[0120] The direction of weight adjustment can be determined based on the intent category. If the intent category indicates that the driver is focused or in a normal state, the driver's control weight can be maintained higher than that of the in-vehicle intelligent driving system, allowing the driver to maintain dominant control. If the intent category indicates that the driver is fatigued, distracted, or hesitant in decision-making, the control weight of the in-vehicle intelligent driving system can be increased, while the driver's control weight can be correspondingly decreased, allowing the in-vehicle intelligent driving system to provide stronger assisted control. If the intent category indicates that the driver intends to anticipate risks, the control weight of the in-vehicle intelligent driving system can be slightly increased without forcibly replacing driver control, allowing the vehicle to enter an assisted preparation state earlier.

[0121] The weight adjustment range can be determined based on the confidence level of the intent. A higher confidence level indicates a more stable intent category, allowing for a larger weight adjustment range; conversely, a lower confidence level allows for a smaller weight adjustment range to avoid frequent changes in control weights due to short-term fluctuations. This enables the driver's control weights and the in-vehicle intelligent driving system's control weights to dynamically change according to the confidence level of the arbitrary graph.

[0122] Furthermore, within each control cycle, the driver control weights and the vehicle intelligent driving system control weights from the previous control cycle can be updated according to the weight adjustment direction corresponding to the current intent category and the weight adjustment magnitude corresponding to the current intent confidence level, thus obtaining the driver control weights and vehicle intelligent driving system control weights for the current control cycle. This update method ensures continuous weight adjustments, rather than abruptly switching between different driving states.

[0123] In this embodiment, by splitting the implicit intent recognition result into intent category and intent confidence level, and using them to determine the weight adjustment direction and magnitude respectively, the human-machine co-driving control weight can reflect both the driver's current state type and the confidence level of that state. Therefore, the in-vehicle intelligent driving system can maintain basic assistance when the driver's state is stable, and gradually enhance assistance when the driver's state is abnormal or decision-making is unstable. This makes the control weight adjustment smoother and more controllable, and reduces insufficient human-machine collaboration caused by fixed weights or sudden takeovers.

[0124] In one embodiment, updating the driver control weights and the in-vehicle intelligent driving system control weights includes:

[0125] The control weights of the in-vehicle intelligent driving system are limited to a preset system weight range;

[0126] Between adjacent control cycles, the change in the control weight of the in-vehicle intelligent driving system is limited to within a preset change amount;

[0127] When the driver's active operation is detected, the control weight of the in-vehicle intelligent driving system will be reduced back to the weight of the basic system.

[0128] Specifically, when determining the control weight of the in-vehicle intelligent driving system, a preset system weight range can be set to limit the upper and lower limits of the control weight. For example, the control weight of the in-vehicle intelligent driving system can be limited to the range of 30% to 80%, and correspondingly, the driver's control weight can be within the range of 20% to 70%. By setting the above range, it is possible to prevent the in-vehicle intelligent driving system from completely exiting control, and also to prevent the in-vehicle intelligent driving system from completely replacing the driver's control, so that both the human and the machine remain involved in the control process.

[0129] Between adjacent control cycles, the amount of change in the control weights of the in-vehicle intelligent driving system can be limited. For example, the maximum adjustment range of the control weights every 10ms can be limited to no more than 1%. When it is necessary to increase or decrease the control weights of the in-vehicle intelligent driving system based on the results of implicit intent recognition, the weights can be adjusted gradually over multiple control cycles, rather than making a large switch in a single control cycle. This reduces the abruptness of vehicle control caused by sudden changes in weights.

[0130] When active driver intervention is detected, the control weight of the in-vehicle intelligent driving system can be reduced back to the base system weight. Active driver intervention can include steering wheel operation, brake pedal operation, or other actions that indicate active driver participation in vehicle control. For example, when active driver intervention is detected, the control weight of the in-vehicle intelligent driving system can be gradually reduced back to 30% of the base system weight, thereby increasing the driver's control weight and maintaining the driver's dominant role in vehicle control.

[0131] Furthermore, if a driver's active emergency operation is detected, such as a sharp turn of the steering wheel or a hard press on the brake, the control weight of the in-vehicle intelligent driving system can be quickly reduced back to the basic system weight, and subsequent vehicle control commands can respond more to the driver's control input. If the driver does not perform any active operation and the implicit intent recognition results indicate that the driver is fatigued, distracted, or hesitant in decision-making, the control weight of the in-vehicle intelligent driving system can be gradually increased within the preset system weight range.

[0132] In this embodiment, by setting range limits for the control weight of the in-vehicle intelligent driving system, limits for changes in adjacent control cycles, and a fallback mechanism when the driver actively operates, the adjustment of control weight can be made continuous and smooth, avoiding the problem of sudden changes in control authority in traditional takeover modes. At the same time, the fallback of the control weight of the in-vehicle intelligent driving system when the driver actively operates can preserve the driver's control over the vehicle; and the gradual increase of the system control weight when the driver is in an abnormal state and does not actively operate can help provide safety redundancy.

[0133] Furthermore, in one embodiment, after obtaining the driver's intention category and its confidence level, the control weight of the in-vehicle intelligent driving system can be calculated based on the intention confidence level, and the driver's control weight can be determined by the control weight of the in-vehicle intelligent driving system.

[0134] Specifically, the confidence levels of fatigue intent and distraction intent can be considered as the main factors influencing the control weights of the in-vehicle intelligent driving system.

[0135] In one embodiment, the control weights of the in-vehicle intelligent driving system can be determined according to the following formula:

[0136]

[0137] in, For the control weight of the in-vehicle intelligent driving system, For the confidence level of fatigue intention, The confidence level for distraction intent. The driver control weights can satisfy:

[0138]

[0139] in, The driver's control weights.

[0140] Specifically, when both the confidence levels for fatigue intention and distraction intention are low, With a weighting close to that of the basic system, driver control has a higher weighting, and vehicle control is primarily based on driver-side inputs. The in-vehicle intelligent driving system mainly provides basic assistance. When the confidence levels of fatigue intent or distraction intent increase... As the value increases, the influence of the control input on the vehicle intelligent driving system in the calculation of vehicle control commands increases, thereby providing more safety assistance in scenarios where the driver's state is abnormal.

[0141] Furthermore, it is possible to... Apply range constraints. For example, you can... By limiting the driver's control weight to between 30% and 80%, the driver's control weight is positioned between 20% and 70%. This constraint prevents the in-vehicle intelligent driving system from having too low a control weight, resulting in insufficient assistance, or from having too high a control weight, leading to a single entity completely taking over the vehicle.

[0142] Furthermore, restrictions can be imposed between adjacent control cycles. The amount of change. For example, it can be... The maximum adjustment increment every 10ms is limited to 1%. When the confidence level of fatigue intent or distraction intent changes... The control commands can be adjusted gradually over multiple control cycles, rather than changing abruptly. This reduces the likelihood of sudden changes in vehicle control commands.

[0143] In one embodiment, when it is detected that the driver has actively performed steering wheel operation, brake pedal operation, or other active control operation, it can be enabled. The weight of the system is adjusted downwards. For example, when the driver actively operates the system, the control weight of the in-vehicle intelligent driving system can be reduced to around 30%, allowing the driver's control weight to increase accordingly. If the driver does not actively operate the system, and the confidence level of fatigue intent or distraction intent continues to rise, the system can be gradually increased within a preset range. .

[0144] In this embodiment, the control weight calculation method described above maps the driver's intention confidence level to the control weight between the driver and the in-vehicle intelligent driving system, allowing the degree of participation of the in-vehicle intelligent driving system in control to be dynamically adjusted according to changes in the driver's state. Simultaneously, by using basic weights, range constraints, and variation limits, abrupt switching of control authority between the driver and the in-vehicle intelligent driving system can be avoided, thereby improving the smoothness of human-machine co-driving control.

[0145] Furthermore, in one embodiment, after identifying a risk prediction intent or a decision-making hesitation intent, an intent buffer period can be set. The intent buffer period can be 200ms. During the intent buffer period, an in-vehicle audio-visual warning can be output first to alert the driver to potential risks around the vehicle or unstable driving decisions. After the intent buffer period ends, the control weights of the in-vehicle intelligent driving system are gradually adjusted according to the corresponding intent category and intent confidence level. By providing a warning before adjusting the weights, the abruptness caused to the driver by sudden changes in vehicle control weights can be reduced.

[0146] Furthermore, in one embodiment, the change in the control weight of the in-vehicle intelligent driving system can be limited according to the control cycle. For example, when the adjacent control cycles are 10ms, the maximum adjustment range of the control weight of the in-vehicle intelligent driving system can be limited to no more than 1%. When it is necessary to increase the weight from the basic system weight to a higher system weight based on the implicit intent recognition result, the weight adjustment can be completed gradually over multiple control cycles; when the driver's active operation is detected, the control weight of the in-vehicle intelligent driving system can also be reduced back to the basic system weight over multiple control cycles.

[0147] In this embodiment, by setting a 200ms intention buffer period, in-vehicle audio and visual warnings, and a weight adjustment constraint of no more than 1% every 10ms, the in-vehicle intelligent driving system can gradually transition from the basic assistance state to the enhanced assistance state, avoiding the control abrupt changes caused by traditional hard switching takeover and improving the smoothness of human-machine co-driving control.

[0148] In one embodiment, vehicle control commands are calculated based on the driver control weights, the in-vehicle intelligent driving system control weights, the driver-side control inputs, and the in-vehicle intelligent driving system-side control inputs, including:

[0149] The driver-side control component is determined based on the driver control weight and the driver-side control input;

[0150] The system-side control components are determined based on the control weights of the in-vehicle intelligent driving system and the control inputs of the in-vehicle intelligent driving system.

[0151] Based on the driver-side control component and the system-side control component, at least one of the steering control command, braking control command, and acceleration control command is calculated to obtain the vehicle control command.

[0152] Specifically, driver-side control inputs may include at least one of the following: driver steering wheel angle input, brake pedal input, and accelerator pedal input. Vehicle-side control inputs may include at least one of the following: steering input, braking input, and acceleration input generated by the vehicle-side intelligent driving system based on control strategies such as lane keeping, obstacle avoidance, following, deceleration, or cruise control.

[0153] Specifically, driver-side control components can be determined based on driver control weights and driver-side control inputs. These driver-side control components represent the degree of influence of the driver-side control inputs in the current vehicle control command. Similarly, system-side control components can be determined based on in-vehicle intelligent driving system control weights and in-vehicle intelligent driving system-side control inputs. These system-side control components represent the degree of influence of the in-vehicle intelligent driving system-side control inputs in the current vehicle control command.

[0154] Furthermore, steering control commands can be calculated based on driver-side control components and system-side control components. For example, in steering control, the final steering control command can be calculated by combining the driver-side steering component corresponding to the driver's steering wheel angle input, and the system-side steering component generated by the onboard intelligent driving system based on lane keeping or obstacle avoidance strategies. This allows the final steering result to simultaneously reflect both driver operation and the safety corrections made by the onboard intelligent driving system.

[0155] Furthermore, braking control commands can be calculated based on driver-side control components and system-side control components. For example, under normal operating conditions, braking control can primarily respond to the driver's brake pedal input; when latent risks, fatigue, or inattentiveness are detected, the system-side braking component can be increased based on the control weights of the onboard intelligent driving system, so that the braking control command includes auxiliary deceleration.

[0156] Furthermore, acceleration control commands can be calculated based on driver-side control components and system-side control components. For example, during acceleration or cruising, acceleration control commands can be calculated by combining the driver's accelerator pedal input and the system-side acceleration input generated by the onboard intelligent driving system based on road conditions, distance to the target vehicle, or driver status. When the driver is fatigued, distracted, or hesitant in making decisions, excessive acceleration changes can be limited through system-side control components to reduce the risk of aggressive driving.

[0157] In this embodiment, by applying driver control weights and on-board intelligent driving system control weights to the driver-side control input and the on-board intelligent driving system-side control input respectively, the implicit intent recognition result can be transformed into actual vehicle control commands. Compared with simply outputting driver status prompts, this method enables the recognition result to specifically participate in steering, braking, and acceleration control, allowing the vehicle to jointly form control results based on driver status and system control strategies in complex driving scenarios, thereby improving the continuity and coordination of human-machine co-driving control.

[0158] In one embodiment, after calculating the vehicle control commands, the method further includes:

[0159] Determine whether there is a conflict between the driver-side control input and the vehicle intelligent driving system-side control input;

[0160] In the event of a conflict, the arbitration priority shall be determined based on the driver's operation type, vehicle environment information, and implicit intent recognition results.

[0161] The vehicle control command is modified according to the arbitration priority, and the modified vehicle control command is output to the vehicle execution system.

[0162] Specifically, consistency judgment can be performed on the driver-side control input and the vehicle-side intelligent driving system control input to determine whether there is a conflict between them. The conflict can include control direction conflict or control amplitude conflict. A control direction conflict could be that the driver-side control input indicates a left turn while the vehicle-side intelligent driving system control input indicates a right turn, or that the driver-side control input indicates acceleration while the vehicle-side intelligent driving system control input indicates deceleration. A control amplitude conflict could be that the driver-side control input and the vehicle-side intelligent driving system control input correspond to the same control direction, but the difference in their control amplitude exceeds a preset amplitude threshold.

[0163] In the event of a conflict, arbitration priority can be determined based on the driver's operation type, vehicle environmental information, and implicit intent recognition results. The driver's operation type indicates whether the driver's current action is an active emergency action, a normal operation, or no action. Vehicle environmental information indicates whether there are external risks around the vehicle. Implicit intent recognition results indicate whether the driver is fatigued, distracted, or hesitant in risk assessment or decision-making.

[0164] In one example, when the driver's action type is an active emergency maneuver, the driver-side control input can be determined to have a higher arbitration priority. Active emergency maneuvers can include evasive maneuvers such as sharp steering or hard braking. In this case, the vehicle control command can be modified so that the modified vehicle control command follows the driver-side control input more closely, while the control weight of the in-vehicle intelligent driving system can be reduced back to the weight of the basic system.

[0165] In another example, when the driver's operation type is normal, vehicle environmental information indicates the presence of external risks, and implicit intent recognition results indicate that the driver has implicit inattention or decision hesitation, it can be determined that the on-board intelligent driving system's control input is used to assist in correcting the driver's control input. In this case, the main control direction of the driver's control input can be retained, and the control amplitude can be corrected based on the on-board intelligent driving system's control input.

[0166] In another example, when the driver does not perform steering, braking, or acceleration operations, and the implicit intent recognition results indicate that the driver is fatigued, distracted, or hesitant in decision-making, the role of the on-board intelligent driving system's side control input in the modified vehicle control commands can be increased, enabling the vehicle to drive according to a more conservative safety strategy.

[0167] Furthermore, after modifying the vehicle control command according to the arbitration priority, the modified vehicle control command can be output to the vehicle execution system, so that the vehicle execution system can perform steering, braking or acceleration control according to the modified vehicle control command.

[0168] In this embodiment, after calculating the vehicle control command, it further determines whether there is a conflict between the driver-side control input and the on-board intelligent driving system-side control input. If a conflict exists, it determines the arbitration priority by combining the driver's operation type, vehicle environment information, and implicit intent recognition results. This avoids the instability in vehicle control caused by simply superimposing control inputs from both sides. This approach preserves the driver's operational priority during proactive emergency operations and enhances the assistance provided by the on-board intelligent driving system when the driver is in an abnormal state or fails to operate in a timely manner, thereby improving the rationality and safety of vehicle control commands in conflict scenarios.

[0169] In one embodiment, the EEG data can also be processed in a privacy-compliant manner. Specifically, the non-invasive acquisition, feature extraction, and intent decoding of EEG signals can be completed locally on the vehicle, without the vehicle uploading the raw EEG data to the backend. When cloud-based statistics, model updates, or fleet-level strategy optimization are required, only the decoded status labels can be uploaded. These status labels are used to characterize recognition results such as high fatigue and decision hesitation, without including the original EEG waveform data.

[0170] Furthermore, in one embodiment, homomorphic encryption and a federated learning architecture can be used for privacy protection. The vehicle can train or update the intent recognition model locally based on data collected within the vehicle, and use the encrypted model parameters, gradient information, or state labels for joint training. The backend can complete model aggregation or policy updates without obtaining the raw EEG data, and then send the updated model parameters to the vehicle.

[0171] In this embodiment, by enabling EEG signal acquisition, feature extraction, and intent decoding to be completed locally on the vehicle, and by combining homomorphic encryption and federated learning architecture, the risk of leakage of raw EEG data can be reduced, making the EEG data more compliant with privacy requirements such as personal information when used for human-machine co-driving control.

[0172] In one embodiment, redundancy checks can also be performed on non-invasive EEG signals, driver status information, and vehicle environment information.

[0173] Specifically, EEG signals, camera data, and millimeter-wave radar data can be used as three verification signals to jointly determine the driver's state, the risks around the vehicle, and the corresponding control strategies.

[0174] Furthermore, when the sensor corresponding to one of the information channels fails, the signal quality falls below preset quality conditions, or the information output is abnormal, the influence of that information channel on implicit intent recognition and vehicle control command calculation can be automatically reduced, and the system can switch to the remaining two information channels to complete driver state recognition and control decisions. For example, when the quality of the EEG signal is insufficient, eye-tracking data collected by the camera and environmental data collected by millimeter-wave radar can be combined for auxiliary judgment; when the camera is obstructed or affected by lighting, control decisions can continue to be made by combining EEG signals and millimeter-wave radar information.

[0175] In this embodiment, by using triple redundancy verification of EEG, camera and millimeter-wave radar, the continuity of the human-machine co-driving control link can be maintained when any sensor fails or the signal is abnormal, which improves the stability of the system in vehicle dynamic scenarios and helps to meet the functional safety requirements of road vehicles.

[0176] Figure 2 This is a schematic diagram of the architecture of a vehicle provided for an exemplary embodiment of this application. The vehicle includes a non-invasive EEG acquisition device 1, a driver state acquisition device 2, a vehicle environment acquisition device 3, a vehicle control system 4, and a vehicle execution system 5.

[0177] The non-invasive EEG acquisition device is used to acquire non-invasive EEG signals from the driver.

[0178] The driver status acquisition device is used to collect driver status information;

[0179] The vehicle environment acquisition device is used to collect vehicle environment information;

[0180] The vehicle control system is used to execute the method described in any of the above-mentioned methods to obtain vehicle control commands;

[0181] The vehicle execution system is used to execute the vehicle control commands.

[0182] Specifically, the non-invasive EEG acquisition device 1 is used to acquire non-invasive EEG signals from the driver. In one embodiment, the non-invasive EEG acquisition device may include a flexible dry electrode EEG acquisition structure, which can be integrated above the steering wheel. The flexible dry electrode EEG acquisition structure can be configured with 4 or 8 sets of dry electrodes, and the contact impedance when the dry electrodes contact the driver can be controlled to be less than 10kΩ to continuously acquire the driver's brain EEG signals. By using flexible dry electrodes, the driver's EEG signals can be acquired without using invasive acquisition methods, facilitating deployment in in-vehicle scenarios.

[0183] In another embodiment, the vehicle can be used for autonomous driving testing in a closed environment. In this scenario, the number of flexible dry electrodes can be increased to 16 sets to collect richer EEG data on driver decision-making. The collected EEG data can be used to train or optimize the autonomous driving decision-making model, enabling the model to better fit the driver's decision-making logic in the test scenario.

[0184] In an alternative embodiment, the non-invasive EEG acquisition device can also be a head-mounted EEG acquisition device. Head-mounted EEG acquisition devices can obtain EEG signals with a high signal-to-noise ratio, but they need to be worn by the driver and are suitable for vehicle models with high requirements for EEG signal quality or as an optional feature.

[0185] The driver status acquisition device 2 is used to collect driver status information. This device may include an in-vehicle camera and a steering wheel sensor. The in-vehicle camera can be used to collect driver eye movement data, such as scan rate, gaze direction, or eye state. The steering wheel sensor can be used to collect steering wheel grip behavior data, such as grip strength, grip fluctuation, or grip stability. The information collected by the driver status acquisition device can be used to determine the driver's external operating status.

[0186] The vehicle environment acquisition device 3 is used to collect vehicle environment information. This device may include the vehicle's existing environmental perception camera and millimeter-wave radar. The environmental perception camera can be used to acquire image information related to road scenes, obstacles ahead, intersecting traffic flows, or inclement weather. The millimeter-wave radar can be used to acquire the position, distance, or motion status of targets around the vehicle. The vehicle environment information can be used to determine whether there are potential risk sources or driving risk conditions around the vehicle.

[0187] The vehicle control system 4 is used to determine the driver's implicit intention recognition result based on non-invasive EEG signals, driver state information and vehicle environment information, determine the driver control weight and the vehicle intelligent driving system control weight based on the implicit intention recognition result, and calculate the vehicle control command based on the driver control weight, the vehicle intelligent driving system control weight, the driver-side control input and the vehicle intelligent driving system-side control input.

[0188] In one embodiment, the vehicle control system 4 can reuse the vehicle's existing intelligent driving domain controller and deploy algorithms related to EEG signal processing, feature fusion, intent confidence decoding, dynamic weight calculation, command fusion, and conflict arbitration within the intelligent driving domain controller. The additional computing power requirement for these algorithms can be no more than 15 TOPS and is compatible with intelligent driving domain controllers with a capacity of 15 TOPS or higher. Therefore, non-invasive EEG perception and human-machine co-driving control can be achieved on the basis of existing high-level intelligent driving hardware without significantly increasing the overall vehicle computing hardware.

[0189] Specifically, the vehicle control system 4 can perform noise reduction processing on the non-invasive EEG signal and extract EEG features from the noise-reduced non-invasive EEG signal; the vehicle control system can also extract eye movement features and steering wheel grip force behavior features based on driver state information, and extract environmental features based on vehicle environment information; subsequently, the vehicle control system can fuse EEG features, eye movement features, steering wheel grip force behavior features and environmental features to obtain fused features, and determine the driver's intention category and its confidence level based on the fused features.

[0190] Furthermore, the vehicle control system 4 can determine the driver control weight and the in-vehicle intelligent driving system control weight based on the driver's intention category and its confidence level. When the driver is focused or in a normal state, the vehicle control system can maintain a basic assistance state with a higher driver control weight and a lower in-vehicle intelligent driving system control weight; when the driver is fatigued, distracted, in a state of risk assessment or decision hesitation, the vehicle control system can dynamically adjust the in-vehicle intelligent driving system control weight based on the corresponding confidence level.

[0191] Furthermore, the vehicle control system 4 can simultaneously acquire driver-side control inputs and on-board intelligent driving system-side control inputs. Driver-side control inputs may include driver steering wheel angle inputs, brake pedal inputs, or accelerator pedal inputs; on-board intelligent driving system-side control inputs may include steering inputs, braking inputs, or acceleration inputs generated by control strategies such as lane keeping, obstacle avoidance, following, or deceleration. The vehicle control system can calculate vehicle control commands based on the driver control weights, on-board intelligent driving system control weights, and the aforementioned control inputs from both sides.

[0192] The vehicle execution system 5 is used to execute vehicle control commands. The vehicle execution system may include at least one of EPS, ESC, and VCU. EPS can be used to execute steering control commands, ESC can be used to execute braking control commands or vehicle stability control commands, and VCU can be used to execute acceleration control commands or power control commands. After generating vehicle control commands, the vehicle control system can send these commands to the EPS, ESC, or VCU, causing the vehicle to operate according to the human-machine collaborative driving control results.

[0193] In one embodiment, the vehicle's post-execution state can be fed back to the vehicle control system, enabling the system to continue updating the driver's implicit intent recognition results, driver control weights, onboard intelligent driving system control weights, and vehicle control commands in subsequent control cycles. Thus, the vehicle can form a closed-loop control process encompassing "EEG signal acquisition, intent recognition, weight determination, control command calculation, and vehicle execution feedback."

[0194] In this embodiment, by installing a non-invasive EEG acquisition device in the vehicle and reusing the driver state acquisition device, vehicle environment acquisition device, and vehicle control system, the driver's EEG signals, external state information, and vehicle environment information can be used together for human-machine co-driving control. By outputting vehicle control commands to vehicle execution systems such as EPS, ESC, and VCU, the driver state recognition results can specifically participate in steering, braking, and power control, thereby improving the collaborative control capability between the vehicle and the driver in complex driving scenarios.

[0195] In some embodiments, the vehicle control system is also equipped with an intent buffer mechanism to prevent sudden takeover and a multi-sensor fault tolerance mechanism.

[0196] Specifically, when the vehicle control system detects that the driver intends to anticipate risks or hesitate in decision-making, a preset buffer period (e.g., 200ms) can be set. During this buffer period, the vehicle prioritizes issuing a warning through in-vehicle audio-visual equipment, and then gradually adjusts the control weight of the on-board intelligent driving system according to a preset change amount. Furthermore, if any of the non-invasive EEG acquisition device, driver state acquisition device, or vehicle environment acquisition device malfunctions or loses data, the vehicle control system can automatically shield the failed data and switch to the remaining two normal data streams for downgraded decoding of intent confidence and control decision-making, ensuring the continuity of control and system safety during dynamic driving.

[0197] In this embodiment, a computer-readable medium stores a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in any one or more of the above embodiments. This computer-readable medium may be included in the vehicle described in the above embodiments; or it may exist independently and not be installed in the device. The aforementioned computer-readable medium carries one or more computer-readable instructions.

[0198] Memory can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor executes various server functions and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the program instructions / modules corresponding to the methods provided in any one or more of the embodiments described above in this application.

[0199] The memory may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the vehicle, etc. Furthermore, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the vehicle via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0200] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0201] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technologies, read-only optical discs, digital versatile optical discs or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0202] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including local area networks (LANs) or wide area networks (WANs), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0203] In the above embodiments, all or part of the implementation can be achieved through software, hardware, firmware, or any combination thereof. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the above steps or functions. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. In addition, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0204] The computer program product provided in this application includes one or more computer programs / instructions. When executed by a processor, these computer programs / instructions generate, in whole or in part, the processes or functions described in this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0205] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0206] The scope of this application is defined by the appended claims rather than the foregoing description, and is therefore intended to encompass all variations falling within the meaning and scope of equivalents of the claims. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural.

[0207] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily made by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims, and the above embodiments should be regarded as exemplary and non-limiting.

Claims

1. A human-machine co-driving control method based on non-invasive EEG perception, characterized in that, The method includes: Acquire non-invasive EEG signals from the driver, driver status information, and vehicle environment information; Based on the non-invasive EEG signal, the driver's state information, and the vehicle environment information, the driver's implicit intention recognition result is determined. The implicit intention recognition result is used to characterize the driver's cognitive state or driving decision tendency that is not reflected in the driver's side control input. Based on the implicit intent recognition results, the driver control weight and the in-vehicle intelligent driving system control weight are determined; Acquire driver-side control inputs and vehicle-side control inputs from the intelligent driving system; The vehicle control command is calculated based on the driver control weight, the vehicle intelligent driving system control weight, the driver-side control input, and the vehicle intelligent driving system-side control input. The vehicle control commands are output to the vehicle execution system.

2. The method as described in claim 1, characterized in that, Based on the non-invasive EEG signals, the driver's state information, and the vehicle environment information, the driver's implicit intention recognition result is determined, including: The non-invasive EEG signal is denoised, and EEG features are extracted from the denoised non-invasive EEG signal. The driver's external operating status is determined based on the driver status information; Determine the driving risk status around the vehicle based on the vehicle environment information; When no control change occurs in the driver's external operating state corresponding to the driving risk state, the driver's implicit intention recognition result is determined based on the EEG characteristics.

3. The method as described in claim 2, characterized in that, Determining the driver's implicit intent based on the aforementioned EEG characteristics includes: When there are no abnormalities in the driver's eye movements, limbs, and steering wheel operation, and the proportion of alpha waves in the brain increases and is accompanied by a decrease in the rate of eye swabbing, the implicit intention recognition result is determined to be implicit attentional distraction. When the driver does not exhibit overt fatigue and the overall brainwave energy decreases while the proportion of slow waves increases, the implicit intent recognition result is determined to be implicit fatigue. When there are potential risk sources around the vehicle, the driver does not make steering, braking or acceleration operations, and the activity of brain beta waves is increased, the implicit intention recognition result is determined to be a risk prediction intention. When the vehicle is in a complex driving scenario, the driver's brain waves fluctuate between alpha and beta waves, accompanied by slight fluctuations in steering wheel grip force and high-frequency eye saccades, the implicit intention recognition result is determined to be a decision hesitation intention.

4. The method as described in claim 1, characterized in that, Based on the implicit intent recognition results, the driver control weight and the in-vehicle intelligent driving system control weight are determined, including: Determine the intent category and intent confidence level corresponding to the implicit intent recognition result; The direction of weight adjustment is determined based on the intent category, and the magnitude of weight adjustment is determined based on the intent confidence level. The driver control weight and the vehicle intelligent driving system control weight are updated according to the weight adjustment direction and the weight adjustment range.

5. The method as described in claim 4, characterized in that, Update the driver control weights and the control weights of the in-vehicle intelligent driving system, including: The control weights of the in-vehicle intelligent driving system are limited to a preset system weight range; Between adjacent control cycles, the change in the control weight of the in-vehicle intelligent driving system is limited to within a preset change amount; When the driver's active operation is detected, the control weight of the in-vehicle intelligent driving system will be reduced back to the weight of the basic system.

6. The method as described in claim 1, characterized in that, Based on the driver control weight, the in-vehicle intelligent driving system control weight, the driver-side control input, and the in-vehicle intelligent driving system-side control input, vehicle control commands are calculated, including: The driver-side control component is determined based on the driver control weight and the driver-side control input; The system-side control components are determined based on the control weights of the in-vehicle intelligent driving system and the control inputs of the in-vehicle intelligent driving system. Based on the driver-side control component and the system-side control component, at least one of the steering control command, braking control command, and acceleration control command is calculated to obtain the vehicle control command.

7. The method as described in claim 1, characterized in that, After calculating the vehicle control commands, the method further includes: Determine whether there is a conflict between the driver-side control input and the vehicle intelligent driving system-side control input; In the event of a conflict, the arbitration priority shall be determined based on the driver's operation type, vehicle environment information, and implicit intent recognition results. The vehicle control command is modified according to the arbitration priority, and the modified vehicle control command is output to the vehicle execution system.

8. A vehicle, characterized in that, This includes non-invasive EEG acquisition devices, driver status acquisition devices, vehicle environment acquisition devices, vehicle control systems, and vehicle execution systems. The non-invasive EEG acquisition device is used to acquire non-invasive EEG signals from the driver. The driver status acquisition device is used to collect driver status information; The vehicle environment acquisition device is used to collect vehicle environment information; The vehicle control system is used to execute the method of any one of claims 1 to 7 to obtain vehicle control commands; The vehicle execution system is used to execute the vehicle control commands.

9. A computer-readable medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images