Control method and system for active activation of combined auxiliary driving system

Abstract

The invention provides a control method and system for active activation of a combined auxiliary driving system, and belongs to the field of automobile auxiliary driving. The method comprises the steps that driver operation information and visual monitoring information are collected and preprocessed; based on the trained driver state recognition model, whether the driver is in an inconscious state or not is judged; when it is judged that the driver is in the inconscious state, the combined auxiliary driving system is actively activated; a visual image and three-dimensional perception information are collected, deep fusion and feature extraction are completed through a multi-modal fusion neural network model, and environment fusion feature data are output; the environment fusion feature data and map information are spliced and then input into a path planning decision network to complete dynamic path planning, and a driving or parking control instruction is output according to the taking-over state of a driver; and carrying out man-machine interaction reminding by adopting a gradient upgrading alarm strategy. According to the method, the problems that the failure judgment precision is low during driving of a driver, and no reasonable vehicle control scheme exists after the vehicle is out of control are effectively solved.

Description

Technical Field

[0001] This invention belongs to the field of automotive driver assistance technology, and particularly relates to a control method and system for the active activation of a combined driver assistance system. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the continuous improvement of vehicle intelligence, combined driver assistance systems (such as adaptive cruise control, lane keeping assist, traffic jam assist, etc.) have been widely used in modern passenger vehicles, significantly improving driving comfort and safety.

[0004] However, in current vehicle driver assistance systems, all cruise control and other driver assistance functions require active driver activation. Only active safety functions can be triggered by the system itself, and even the safe parking strategy only activates after driver assistance is actively engaged. In normal driving scenarios, drivers are prone to brief or sustained loss of control due to driver inattention. Relying solely on active safety functions for collision avoidance in such situations not only results in a poor driving experience but also significantly increases the risk of secondary collisions, and the vehicle may deviate from its intended trajectory. Furthermore, current technologies for assessing driver inattention often rely on single sensor data or simple logical judgments, such as prolonged absence of steering wheel operation, resulting in low accuracy.

[0005] Therefore, there is no reasonable judgment standard or proactive takeover and vehicle control plan for this type of vehicle loss of control caused by driver inattention, and it is impossible to effectively reduce the risk of accidents without disrupting road driving order. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, the present invention provides a control method and system for active activation of a combined driver assistance system, which solves the problems of low accuracy in driver's judgment during loss of awareness and lack of reasonable vehicle control plan after loss of awareness leads to vehicle loss of control. The system can actively take over the vehicle, reduce the risk of accidents without disrupting the current driving order, and improve driving safety and experience.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a control method for the active activation of a combined driver assistance system; A control method for actively activating a combined driver assistance system includes: Collect and preprocess driver operation information and visual monitoring information; The operation information and visual monitoring information are input into the trained driver state recognition model to determine whether the driver is in a state of unconsciousness. When the driver is determined to be in a state of unconsciousness, the combined driver assistance system is actively activated; visual images and three-dimensional perception information are collected through multiple sensors, and deep fusion and feature extraction are completed through a multimodal fusion neural network model to output environmental fusion feature data containing target behavior trend prediction. After splicing environmental feature data with map information, the data is input into the path planning decision network to complete dynamic path planning without disrupting the driving order, and output driving or parking control commands according to the driver's takeover status. A tiered alarm strategy is adopted for human-machine interaction reminders. Combined with the control commands of the path planning decision network, the system sequentially executes the primary alarm, the escalation alarm, and the minimum risk safe parking operation.

[0008] As a further technical solution, driver operation information and visual monitoring information are collected and preprocessed, including: The system collects driver operation information in real time through pressure sensors on the steering wheel and displacement sensors on the accelerator and brake pedals; it also collects image sequences of the driver's face, eyes, hands, and limb movements through an in-vehicle camera. The acquired image sequences are processed for face detection and cropping, noise reduction, and illumination normalization; the operational data acquired by the sensors are processed for outlier removal, timestamp alignment, and standardized mapping.

[0009] As a further technical solution, the operational information and visual monitoring information are input into a trained driver state recognition model to determine whether the driver is in a state of unconsciousness, including; The preprocessed image frame sequence is input into the CNN spatial feature extraction layer of the driver state recognition model to extract the spatial dimension features of the driver's gaze direction, eye opening and closing, facial orientation, and the contact area between the hand and the steering wheel, and outputs a spatiotemporal feature vector. The spatiotemporal feature vector is input into the LSTM temporal analysis layer. Combined with the temporal attention mechanism, weights are assigned to the abnormal state features of the driver to enhance the extraction of temporal correlation features of abnormal states and output a temporal fusion feature vector. The temporal fusion feature vector is input into the fully connected decision layer, and the softmax classification function is used to output the driver's loss of consciousness state and the corresponding state confidence.

[0010] As a further technical solution, visual images and 3D perception information are acquired through multiple sensors, and deep fusion and feature extraction are completed through a multimodal fusion neural network model. The output is environmental fusion feature data containing target behavior trend prediction, including: Preprocess the acquired visual images and 3D perception information; The preprocessed multi-source data is input into a multimodal fusion neural network model. Visual semantic feature vectors are extracted based on a lightweight backbone network, and radar geometric feature vectors are extracted through a fully connected network. A cross-modal attention fusion layer is used to adjust the weights of bimodal features and output a cross-modal fused feature vector. The cross-modal fusion feature vector is reduced in dimensionality and redundancy is removed by the feature encoding layer, and the core environment fusion feature vector is output. Based on the core environment fusion feature vector, the environment fusion feature data is output by the multi-task prediction output layer.

[0011] As a further technical solution, the environmental fusion feature data includes dynamic and static target fusion information, basic environmental fusion information, and parking feature information; Among them, the dynamic and static target fusion information includes target displacement velocity, acceleration, heading angle, lateral and longitudinal distance, type, size, and target behavior trend prediction results based on time series modeling output; The environmental infrastructure fusion information includes drivable area boundary modeling, road condition complexity classification, and traffic priority determination results; Parking feature information includes the location coordinates of the parking area, the area size, the distance from the vehicle, and the feasibility assessment results.

[0012] As a further technical solution, environmental fusion feature data and map information are stitched together and input into the path planning decision network to complete dynamic path planning without disrupting traffic order. Based on the driver's takeover status, driving or parking control commands are output, including: By combining environmental fusion feature data with map information data, a path planning state tensor is constructed. The state tensor is input into a path planning decision network based on deep reinforcement learning. The key feature representations of the drivable and parking areas are enhanced by the feature extraction layer of the fully connected network. The policy network assigns weights to the features through a constrained attention mechanism, outputs the probability distribution of continuous actions, and generates control commands. A weighted reward function is introduced to evaluate the planning results, and the path is planned according to priority. Based on the driver takeover status, driving control commands or safe parking control commands combined with parking feature information are dynamically output.

[0013] As a further technical solution, a tiered escalation alarm strategy is adopted for human-machine interaction reminders. Combined with control commands from the path planning decision network, primary alarms, escalation alarms, and minimum-risk safe parking operations are executed sequentially, including: First, a primary alarm is triggered, and a takeover reminder is issued to the driver through at least one means, such as sound or light. At the same time, based on the driving control instructions of the route planning decision network, the vehicle is kept moving smoothly along the planned route. If the driver fails to take over, an upgraded alarm will be activated immediately, employing a differentiated reminder method that combines sound, light, and electricity. At the same time, the path will be dynamically optimized based on real-time environmental data, and speed reduction control commands will be output simultaneously. If the driver does not take over, the minimum risk safe parking operation will be activated. Based on the parking control instructions output by the path planning decision network, the vehicle will be controlled to move laterally and longitudinally, select a safe location, and complete a smooth parking. During the parking process, the vehicle's hazard lights will be activated to warn surrounding vehicles. After parking, the alarm will continue until the driver takes over.

[0014] A second aspect of the present invention provides a control system for the active activation of a combined driver assistance system.

[0015] A control system for active activation of a combined driver assistance system, comprising: The data acquisition and preprocessing module is configured to: acquire driver operation information and visual monitoring information and perform preprocessing; The driver state judgment module is configured to input the operation information and visual monitoring information into the trained driver state recognition model to determine whether the driver is in a state of unconsciousness. The environmental fusion feature data acquisition module is configured to: actively activate the combined driver assistance system when it is determined that the driver is in a state of unconsciousness; collect visual images and three-dimensional perception information through multiple sensors, complete deep fusion and feature extraction through a multimodal fusion neural network model, and output environmental fusion feature data containing target behavior trend prediction; The path planning module is configured to: stitch together environmental fusion feature data and map information and input them into the path planning decision network to complete dynamic path planning without disrupting the driving order, and output driving or parking control commands according to the driver takeover status. The interactive reminder module is configured to use a tiered escalation alarm strategy for human-computer interactive reminders, and in conjunction with the control instructions of the path planning decision network, execute the primary alarm, escalation alarm and minimum risk safe parking operation in sequence.

[0016] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a control method for active activation of a combined driver assistance system as described in the first aspect of the present invention.

[0017] A fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a control method for active activation of a combined driver assistance system as described in the first aspect of the present invention.

[0018] The above one or more technical solutions have the following beneficial effects: (1) The present invention adopts a CNN-LSTM hybrid neural network model, which combines CNN spatial feature extraction with an improved LSTM time sequence analysis that introduces an attention mechanism. This solves the technical problems of traditional logical judgment relying on a single threshold, being prone to misjudgment and missed judgment, and being unable to capture continuous changes in the driver's state. It provides a highly reliable trigger basis for the system's active activation, effectively reducing driving interference caused by misactivation and the risk of vehicle loss of control caused by missed activation.

[0019] (2) In the process of environmental perception fusion, this invention achieves deep complementary fusion of visual and radar multi-source heterogeneous information through an improved multimodal fusion neural network model (CM-AFNet). It adopts a cross-modal attention mechanism to dynamically adjust weights according to the environment, thus solving the perception limitations of a single sensor in scenarios such as poor lighting and radar clutter interference. This model can not only identify basic information such as target type, position, and speed, but also output 1-3 second target behavior trend prediction results, providing comprehensive, accurate, and forward-looking environmental data support for path planning.

[0020] (3) The path planning decision network based on deep reinforcement learning in this invention adopts a continuous action space design and a multi-dimensional weighted reward function, which solves the problems of poor planning flexibility and difficulty in adapting to complex dynamic scenarios of traditional fixed logic algorithms. It can optimize driving / parking paths in real time under the principle of safety first, so that the vehicle drives smoothly and does not disrupt the road traffic order. At the same time, it reserves time for the driver to take over, which significantly improves the safety and comfort of human-machine co-driving.

[0021] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1 This is a flowchart of the method in the first embodiment.

[0024] Figure 2 This is a system structure diagram of the second embodiment. Detailed Implementation

[0025] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0027] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0028] The overall approach of this invention addresses the technical challenge of vehicle loss of control due to driver inattention. It employs a CNN-LSTM hybrid neural network model to accurately identify the driver's inattention state, triggering proactive system activation. A multimodal fusion neural network achieves deep complementary fusion of visual and radar information, outputting environmental features that include target behavior predictions. A path planning and decision-making network based on deep reinforcement learning dynamically plans driving or safe parking paths according to the fused features and map information. This is complemented by a graded human-machine interaction strategy that hierarchically executes alarms and vehicle control. All three models have undergone lightweight modifications to adapt to different computing platforms and are continuously optimized using online incremental learning, achieving an upgrade from "passive activation" to "intelligent proactive activation," comprehensively improving driving safety and the human-machine co-driving experience in scenarios of driver inattention.

[0029] Example 1 This embodiment discloses a control method for active activation of a combined driver assistance system. It accurately identifies the driver's unconscious state and triggers active activation through a CNN-LSTM model; it deeply integrates visual and radar information using an improved multimodal fusion network to output environmental features and target behavior predictions; it dynamically plans driving or parking paths based on a deep reinforcement learning-based path planning network; and it enhances driving safety in unconscious scenarios by implementing graded alarm and vehicle control through gradient-based human-machine interaction.

[0030] Specifically, such as Figure 1 As shown, a control method for actively activating a combined driver assistance system includes: Step S1: Collect driver operation information and visual monitoring information and perform preprocessing.

[0031] The information collection process is based on a combined driver assistance hardware system equipped with multiple sensors. The core hardware includes a HOD steering wheel integrated pressure sensor, accelerator and brake pedal integrated displacement sensors, and an in-vehicle high-definition visual monitoring camera.

[0032] First, the driver's steering wheel grip pressure data is collected in real time by a thin-film pressure sensor built into the HOD steering wheel, serving as the raw operational information for determining the steering wheel grip state. Simultaneously, displacement sensors built into the accelerator and brake pedals collect data on accelerator pedal opening and brake pedal travel, respectively, serving as the raw operational information for determining the accelerator and brake operation states. High-definition visual monitoring cameras installed inside the vehicle capture continuous image sequences of the driver's face, eyes, hands, and limb movements in real time. Each image sequence carries a timestamp, maintaining synchronization with the operation information timestamps, providing raw visual data for extracting the driver's state spatial features.

[0033] Secondly, for the collected driver operation information and visual monitoring information, modality-specific preprocessing strategies are adopted to eliminate environmental interference, data redundancy and abnormal deviations, and output standardized and normalized preprocessed data.

[0034] For visual monitoring information, a face detection algorithm is used to detect the driver's facial region in each frame of the image sequence, accurately selecting key areas such as the face, eyes, and hands, and removing irrelevant areas such as the background and interior decorations. The cropped key area images are uniformly scaled to 224×224 pixels to ensure consistent model input size. A Gaussian filtering algorithm is used to denoise the cropped images, eliminating image noise caused by in-vehicle light reflections and camera noise, thus improving image clarity. A histogram equalization algorithm is used to normalize the illumination of the denoised images, eliminating the influence of lighting factors such as nighttime, backlighting, and changes in the brightness of in-vehicle lights on the driver's facial and eye features, ensuring the accuracy of feature extraction under different lighting conditions. For driver operation information, the 3σ principle is used to identify outliers in the grip pressure, throttle opening, and brake stroke data collected by the sensors. Abnormal data points caused by sensor vibration and vehicle bumps are removed. The missing data points after removal are filled in using linear interpolation to ensure data continuity. At the same time, the timestamps of the operation information and the visual monitoring information are aligned at the millisecond level to ensure the matching of driver operation behavior and visual state in the same time dimension and avoid time series deviation. Finally, the preprocessed grip pressure, throttle opening, and brake stroke data are mapped to the [0,1] interval through the min-max standardization algorithm to eliminate the influence of the difference in data units of different sensors on the model analysis and output a standardized operation feature input dataset.

[0035] Step S2 involves inputting the operational information and visual monitoring information into a trained driver state recognition model to determine whether the driver is in a state of unconsciousness. The trained driver state recognition model includes a CNN spatial feature extraction layer, an LSTM temporal feature fusion layer, and a fully connected layer. Specifically: Step S21 involves structurally integrating the two types of preprocessed data to form a standard input set for the driver state recognition model: First, a sequence of standardized image frames of the driver's face, eyes, hands, and limbs after face detection, cropping, denoising, and illumination normalization is continuously input in time stamp order; Second, a dataset of driver operation values ​​after outlier removal, time stamp alignment, and standardized mapping is used to achieve millisecond-level temporal matching with the image frame sequence, ensuring the synchronization of spatial features and operational behavior.

[0036] Step S22: The structured and integrated standardized image frame sequence is input into the CNN spatial feature extraction layer of the driver state recognition model. This layer is based on a modified lightweight MobileNetV3-small architecture and includes three depthwise separable convolutional layers, two pooling layers, and a batch normalization layer. Its core function is to accurately extract the spatial dimension features of the driver's state. The convolutional layers perform layer-by-layer feature convolution on the input image frames, sequentially extracting low-level visual features such as the driver's eye opening, gaze direction, pupil position, facial orientation, hand-steering wheel contact area, and body posture. The pooling layer performs dimensionality reduction on the convolutional features through max pooling, eliminating redundant feature information and retaining the core spatial features. After the batch normalization layer normalizes the feature data, it outputs a single-frame spatiotemporal feature vector. This vector fully represents the spatial state features of the driver in a single frame image and is bound to the corresponding driver operation numerical data to form a fused feature unit.

[0037] Step S23 involves inputting the sequence of single-frame spatiotemporal feature vectors in the continuous time dimension into the LSTM temporal analysis layer of the model. This layer is a three-layer stacked structure with an embedded temporal attention mechanism, which is the core implementation of temporal correlation analysis and anomaly feature enhancement for the driver's state. The LSTM base layer performs temporal modeling on the continuous spatiotemporal feature vector sequence, capturing the changing patterns of the driver's state over time, and mining the temporal correlation features of states such as gaze deviation, eye closure, no contact with the steering wheel, and unintentional accelerator pedal pressing. The temporal attention mechanism assigns weights to the modeled temporal features, assigning a high weight of 0.7-1.0 to the abnormal driving features and a low weight of 0.1-0.3 to the normal driving features, thereby emphasizing the abnormal driving features and simplifying the normal driving features. After weight optimization and temporal fusion, a temporal fusion feature vector is output. This vector simultaneously represents the spatial characteristics of the driver's state, operational behavior, and continuous temporal change patterns, accurately reflecting the dynamic process of the driver from normal driving to a state of loss of consciousness.

[0038] Step S24: The temporal fusion feature vector is input into the fully connected decision layer of the model. This layer contains two fully connected layers and a Softmax classification function, which is the core implementation of binary classification and confidence output for the driver's loss of consciousness state. The fully connected layer performs deep feature mapping on the temporal fusion feature vector, transforming the high-dimensional features into binary decision features for the driver's "normal driving" and "loss of consciousness state". The Softmax classification function is used to calculate the probability of the binary decision features, outputting the probability value of the driver being in different states. The confidence threshold is set to 0.8. When the confidence of the "loss of consciousness state" is ≥0.8, it is preliminarily determined that the driver has a tendency to be in a loss of consciousness state. In step S25, after receiving the judgment result output by the model, the combined assisted driving control module, in conjunction with the driver operation information collected in step S1, verifies each of the preset loss of consciousness judgment conditions. Only when all six conditions are met simultaneously—the model determines that the driver is not in the loop and the duration is greater than 3 seconds, the driver has not pressed the brake, the driver has pressed the accelerator pedal and the accelerator opening is greater than 10%, the driver has not held the steering wheel and the duration is greater than 3 seconds, the vehicle is in a forward state, and the assisted driving system has not been actively activated—is the driver ultimately determined to be in a state of loss of consciousness, and the combined assisted driving system is immediately triggered to actively activate the system, entering the subsequent multimodal environment information collection and fusion processing stage. If any condition is not met, the driver is determined to be in a normal driving state, the model continuously monitors the driver's state in real time, and no trigger command is output.

[0039] Step S3: When it is determined that the driver is in a state of unconsciousness, the combined driver assistance system is actively activated; visual images and three-dimensional perception information are collected through multiple sensors, and deep fusion and feature extraction are completed through a multimodal fusion neural network model to output environmental fusion feature data containing target behavior trend prediction.

[0040] Step S31: The combined driver assistance system uses an onboard multi-sensor kit to simultaneously collect visual image information and three-dimensional perception information of the vehicle's surroundings.

[0041] High-definition cameras for both front-view and surround-view capture visual image information of lane lines, traffic signs, dynamic and static targets (vehicles, pedestrians, non-motorized vehicles, obstacles), and road boundaries in front of, to the sides, and behind the vehicle, covering a 360° perception range around the vehicle. By using a forward-facing millimeter-wave radar with a 10Hz acquisition frequency, data such as the relative distance, speed, heading angle, and radar cross section (RCS) of surrounding targets are acquired, enabling accurate perception of dynamic targets at medium to long ranges. By using an onboard solid-state lidar, three-dimensional point cloud data of the vehicle's surrounding environment is collected, accurately representing spatial geometric information such as target outlines, road terrain, and drivable areas. By using an ultrasonic radar mounted on the vehicle body, the relative distance information of obstacles near the vehicle is acquired, compensating for the accuracy shortcomings of lidar and millimeter-wave radar in near-range perception.

[0042] The acquired visual image information and 3D perception information are preprocessed according to their specific modes to eliminate environmental interference, data deviation and dimensional differences, and unify all perception information under the vehicle coordinate system to achieve spatiotemporal synchronization of multi-source data.

[0043] Step S32: Input the standardized multi-source perception dataset into the multimodal fusion neural network model (CM-AFNet). This model adopts a four-layer structure of dual input channels + cross-modal attention fusion layer + feature encoding layer + multi-task prediction output layer, which sequentially completes the extraction of visual semantic features and radar geometric features, the weighted complementary fusion of dual-modal features, and the output of core environmental fusion features.

[0044] For the visual feature channel, based on the YOLOv8-nano lightweight backbone network, the preprocessed visual image is convolved and feature extracted layer by layer to capture semantic information such as target type, lane line features, and road boundaries, and finally outputs a visual semantic feature vector to fully represent the visual semantic attributes of the environment. For the radar feature channel, the PointNet-tiny network is used to extract geometric features from the downsampled point cloud data of the lidar. Combined with a 3-layer lightweight fully connected network, the standardized numerical data of millimeter-wave radar and ultrasonic radar are encoded and fused to output the radar geometric feature vector, which accurately represents the spatial geometry of the environment and the motion attributes of the target.

[0045] By employing a dual-attention head mutual attention mechanism, the mutual attention weights of visual semantic features and radar geometric features are calculated, and the weight allocation is dynamically adjusted according to the current driving environment of the vehicle. This achieves weighted complementary fusion of dual-modal features, effectively avoiding the perception limitations of a single sensor, and outputting a cross-modal fusion feature vector after fusion.

[0046] Step S33: By combining two lightweight fully connected layers with batch normalization and Dropout layers, the 1024-dimensional cross-modal fusion feature vector is reduced in dimensionality and redundancy is removed. Invalid feature information is eliminated, the core feature expression is strengthened, and the core environment fusion feature vector is finally output as the basic feature data for multi-task prediction.

[0047] Step S34: Based on the core environment fusion feature vector, the multi-task prediction output layer synchronously completes the prediction and output of information in three dimensions: dynamic and static target fusion, environmental basic fusion, and parking features, forming complete environment fusion feature data that includes target behavior trend prediction.

[0048] Among them, the dynamic and static target fusion information includes target displacement velocity, acceleration, heading angle, lateral and longitudinal distance, type, size, and target behavior trend prediction results based on time series modeling output; The environmental infrastructure fusion information includes drivable area boundary modeling, road condition complexity classification, and traffic priority determination results; Parking feature information includes the location coordinates of the parking area, the area size, the distance from the vehicle, and the feasibility assessment results.

[0049] Step S4: After stitching together the environmental fusion feature data and map information, input the data into the path planning decision network to complete dynamic path planning without disrupting the driving order, and output driving or parking control commands according to the driver's takeover status.

[0050] Step S41: Retrieve real-time map information data from the vehicle navigation cruise map module, and perform feature stitching and tensor construction with core environmental fusion feature data to provide standardized input for the path planning decision network. Obtain core geographic information of the vehicle's current driving situation in real time from the map module, including current lane attributes, navigation planned route, distance to the next intersection / ramp / exit, road speed limit, and reversible lane area markers. Digitally encode and vectorize the above geographic information to output map feature vectors, ensuring dimensional matching with the environmental fusion feature data. The core environmental fusion feature data and map feature vectors are sequentially concatenated according to feature dimensions to construct a path planning state tensor. This tensor simultaneously contains real-time environmental perception features of the vehicle's surroundings and road geographic planning features, fully representing the full-dimensional state information required for path planning. The path planning state tensor is normalized to eliminate the dimensional differences between different feature dimensions, ensuring the consistency and validity of the network input data. The processed state tensor is then input into the state input layer of the path planning decision network in real time.

[0051] Step S42: The path planning decision network performs deep feature processing on the input path planning state tensor, strengthening key features through a feature extraction layer and assigning feature weights through a policy network. Four lightweight fully connected layers perform layer-by-layer feature mapping and deep extraction on the path planning state tensor, focusing on strengthening key path planning features such as drivable area boundaries, surrounding dynamic and static target locations, parking area coordinates, navigation path direction, and road speed limits. Invalid and redundant features are eliminated, and the core feature vector of the plan is output, accurately representing the core decision basis of path planning. The core feature vector of the plan is input into the policy network, which introduces a constrained attention mechanism. With "safe driving and no disruption of road order" as the core constraint, different weights are assigned to different features. Safety features such as drivable areas, safe distances, and parking areas are assigned the highest weight of 0.6-0.8. Order features such as navigation routes, road speed limits, and traffic priorities are assigned the second highest weight of 0.2-0.3. Experience features such as driving smoothness are assigned the basic weight of 0.1-0.2. The core guidance of planning decisions is achieved through weight allocation. While the policy network processes the data, the value network evaluates the driving state corresponding to the current core feature vector of the plan and outputs a state value score, providing a reference for the selection of the optimal action in the future and ensuring the rationality and optimality of the path planning.

[0052] Step S43: Based on the feature weight allocation results, the path planning decision network generates continuous lateral and longitudinal control commands for the vehicle, and evaluates and optimizes the planning results in real time through a multi-dimensional weighted reward function to achieve dynamic iteration of path planning. The strategy network, based on the weighted core feature vector of the planning, outputs the probability distribution of continuous actions (vehicle speed, steering angle, braking intensity) in the continuous action space. Combined with the state value score of the value network, it selects the action combination with the optimal state value to generate real-time continuous control commands. All commands meet the vehicle's physical constraints: acceleration / deceleration amplitude ≤ 2m / s², steering angle ≤ 5° / s, avoiding aggressive maneuvers. A multi-dimensional weighted reward function is introduced to quantitatively evaluate the results of each planning decision. The total reward formula is R = 0.5R1 + 0.2R2 + 0.15R3 + 0.15R4, and the rewards and evaluation criteria for each dimension are as follows: R1 is a safety reward (weight 0.5): No collision, maintaining a safe distance, and driving within the drivable area are positive rewards, while collision risk and driving out of the drivable area are negative penalties. R2 is the order reward (weight 0.2): driving along the navigation route, not interfering with surrounding vehicles, and meeting road speed limits are positive rewards, while changing lanes arbitrarily, deviating from the navigation route, and speeding / driving at low speeds are negative penalties; R3 is the experience reward (weight 0.15): Smooth vehicle acceleration and deceleration and small steering angle are positive rewards, while sudden acceleration, sudden braking and large-angle steering are negative penalties; R4 is the takeover reward (weight 0.15): Allowing the driver ≥10s of takeover time and having a clearly planned driving / parking route are positive rewards, while insufficient takeover time and unclear route planning are negative penalties.

[0053] Based on the quantitative evaluation results of the reward function, the network fine-tunes the feature weights and action outputs of the policy network in real time to achieve dynamic optimization of path planning and ensure that the planning results meet the requirements of safety, order, experience, and takeover.

[0054] Step S44: The path planning decision network uses the driver takeover state as the core switching basis, follows the preset path planning priority principle, dynamically completes driving path planning or safe parking path planning, and outputs corresponding driving / parking control commands, which are transmitted in real time to the vehicle braking, power, and steering execution modules. The priority principle and command output are implemented as follows: Prioritize driving scenarios with normal traffic flow, few vehicles, and no frequent changes in road conditions; when navigation is enabled, drive smoothly along the navigation route for a short period to allow the driver sufficient time to regain control; if the driver does not take over continuously, immediately combine the parking feature information output in step S3 to plan the optimal safe parking route and parking location.

[0055] The combined driver assistance system uses a driving state judgment module to monitor the driver's steering wheel grip, pedal operation, and visual state in real time to determine whether the driver has taken over the vehicle. The monitoring results are updated every 100ms and fed back to the path planning decision network in real time.

[0056] If the driver does not trigger a takeover operation but is still within the takeover range, the network completes dynamic planning of the driving path according to the priority principle, outputs driving control commands, including target speed, real-time steering angle, and brake / accelerator opening, controls the vehicle to drive smoothly along the planned path, and continuously reserves ≥10s takeover time for the driver, during which environmental fusion feature data is updated in real time to dynamically optimize the driving path.

[0057] If the driver does not take over the vehicle for more than 10 seconds, the network immediately triggers a safe parking path planning. Combining the location coordinates, area, and traffic feasibility assessment results of the parking area in the parking feature information, the network selects the optimal safe parking location, such as the emergency lane or an open area on the right side of the road, and plans a full-process parking path of "gradual deceleration → slight safe lane change → smooth entry → precise parking". The network outputs safe parking control commands, including stepped deceleration commands, small-angle steering commands, and smooth braking commands, to ensure that the vehicle can complete the parking operation safely, orderly, and smoothly without driver intervention.

[0058] Step S5: A tiered escalation alarm strategy is adopted for human-machine interaction reminders. Combined with the control instructions of the path planning decision network, the primary alarm, escalation alarm and minimum risk safe parking operation are executed in sequence.

[0059] After the combined driver assistance system is actively activated, the driving status judgment module continuously monitors the driver takeover status at a period of 100ms. The combined driver assistance control module links the monitoring results with the real-time control commands of the path planning decision network and executes operations according to the gradient logic of "initial alarm to maintain driving → upgraded alarm to dynamically reduce speed → minimum risk safe stop". The stages are seamlessly connected, and the alarm intensity and vehicle control strategy are gradually upgraded as the driver does not take over. At the same time, the driver's right to take over the vehicle at any time is retained throughout the process. After taking over, the system immediately exits the active control mode and restores the driver's full control of the vehicle.

[0060] After the system triggers active activation, it immediately initiates a primary alarm process, adhering to the core principle of "lightweight reminder + smooth driving maintenance," balancing the effectiveness of the reminder with the smoothness of the ride. Specifically, the process is as follows: The cockpit module provides initial human-machine interaction reminders, using at least one of sound and light methods to issue takeover prompts. The visual reminder is a high-frequency flashing blue warning light on the instrument panel and a white "Please take over the vehicle immediately" text prompt displayed in the center of the central control screen (the font is enlarged and unobstructed). The auditory reminder is a 60dB low-frequency soft prompt tone played by the in-vehicle audio system (played in a loop, without harshness). The reminder content is concise and clear to avoid interfering with the driver's recovery. The combined driver assistance control module receives driving control commands output from the path planning decision network and sends them to the braking, power, and steering modules. It strictly follows the planned path to maintain stable vehicle driving. The control parameters meet the following requirements: acceleration and deceleration amplitude ≤ 2m / s², steering angle ≤ 5° / s, vehicle speed matches the current road speed limit and the speed of surrounding traffic flow, maintains a safe following distance, and avoids sudden operations, providing the driver with sufficient reaction and takeover time. During the initial alarm period, the system continuously monitors the driver's takeover status. If the system detects that the driver has completed a valid takeover (holding the steering wheel, pressing the brake / accelerator pedal and returning to normal status), the alarm will be terminated immediately and the system will exit active control. If no valid takeover operation is detected for 10 seconds, the escalation alarm process will be triggered immediately.

[0061] If the driver fails to take over after the initial alarm period expires, the system seamlessly initiates an upgraded alarm process. Based on the core principle of "enhanced audio-visual alerts + dynamic speed reduction and route optimization," this process strengthens the alert while preparing for a safe stop. Specifically, the implementation involves: The cockpit module has been upgraded to improve the human-machine interaction reminder method, adopting a differentiated reminder system that links vision, hearing, and touch. The visual reminder is upgraded to a full-screen red warning interface on the instrument panel / central control screen with a constantly lit red warning light, and simultaneously displays the dynamic text "Automatic parking is imminent, please take over immediately"; the auditory reminder is upgraded to the in-vehicle audio system playing an 80dB high-frequency rapid alarm sound (1-second interval, highly recognizable); the tactile reminder is a seat vibration module that continuously vibrates at a frequency of 50Hz (vibration reminder is optional on the steering wheel), which enhances the reminder effect from multiple dimensions to ensure that the driver can perceive the emergency situation; The combined assisted driving control module receives optimized control commands from the path planning decision network combined with real-time updated environmental fusion feature data. On the one hand, it dynamically adjusts the parking path, prioritizing the locking of optimal parking positions such as the emergency lane on the right side of the road and open areas, and optimizing the driving trajectory for lane changing and entering. On the other hand, it issues step-by-step deceleration commands to the power module, gradually reducing the vehicle speed with a smooth deceleration increment of 1m / s², while maintaining the vehicle's stable driving in the original lane to avoid the risk of rear-end collisions caused by sudden deceleration, and strictly adhering to road driving order during the deceleration process. During the escalation alarm period, the system continuously monitors the driver takeover status at high frequency. If effective takeover is detected, all alarms are immediately terminated and driver control is restored. If no effective takeover is detected, the minimum risk safe parking operation is triggered immediately after the path planning decision network completes the final optimization of the safe parking path.

[0062] If the driver still fails to take over after the alarm is upgraded, the system will immediately initiate a minimum-risk safe parking operation. Based on the core principles of "safe parking + external warning + continuous reminders," the system will achieve precise control of the vehicle throughout the entire process from driving to parking. Specifically, this will be implemented as follows: The combined driver assistance control module receives the safe parking control command output by the path planning decision network and coordinates with the braking, power, and steering modules to complete integrated operation: the power module cuts off power output and maintains idle speed; the steering module slowly adjusts the direction at a small angle of ≤3° / s, smoothly driving into the preset safe parking area according to the planned path, avoiding surrounding dynamic and static targets during lane changes to ensure safe driving trajectory; the braking module gradually applies braking force according to vehicle speed, adopting a "light first, then gradual" braking strategy to achieve smooth vehicle deceleration until the vehicle is completely parked, without sudden braking or sharp steering, ensuring the safety of the vehicle itself and the surrounding road; At the same time as the parking operation is initiated, the combined driver assistance control module sends a command to the vehicle's electrical module to immediately turn on the vehicle's hazard lights, sending a warning signal to surrounding vehicles and pedestrians, indicating that the vehicle is in an abnormal state and is about to stop, reminding surrounding traffic participants to pay attention and avoid the accident, reducing the risk of scratches, collisions and other accidents; After the vehicle is fully parked, the system maintains the intensity of the upgraded alarm sound, light and electricity reminders, the red warning interface of the cockpit module, the high-frequency alarm sound, the seat vibration are uninterrupted, and the vehicle's hazard lights remain on until the system detects that the driver has completed effective takeover (such as unlocking the vehicle, holding the steering wheel, or pressing the brake pedal) or a third party intervenes, ensuring that the vehicle is in a safe warning state before the driver returns to normal. After the vehicle is parked, the combined driver assistance system maintains low power consumption and continuously monitors the driver's status and the vehicle's surrounding environment. If a risk of collision or scrape is detected in the surrounding area, the warning will be further enhanced through the vehicle's electrical module (such as sounding the horn). At the same time, the vehicle's basic power supply and braking functions are preserved to ensure the vehicle's parking safety.

[0063] This process utilizes a tiered, phased alarm and vehicle control strategy to achieve full-process intelligence of "reminder-control-stopping." It not only provides drivers with ample time and operational space for takeover but also ensures that the vehicle is safely and orderly stopped in a designated area when the driver is continuously unaware, fundamentally preventing road traffic accidents caused by loss of vehicle control while simultaneously ensuring road traffic order and driver safety.

[0064] Example 2 This embodiment discloses a control system for the active activation of a combined driver assistance system; like Figure 2 As shown, a control system for active activation of a combined driver assistance system includes: The data acquisition and preprocessing module 201 is configured to: acquire driver operation information and visual monitoring information and perform preprocessing; The driver state judgment module 202 is configured to input the operation information and visual monitoring information into the trained driver state recognition model to determine whether the driver is in a state of unconsciousness. The environmental fusion feature data acquisition module 203 is configured to: actively activate the combined assisted driving system when it is determined that the driver is in a state of unconsciousness; collect visual images and three-dimensional perception information through multiple sensors, complete deep fusion and feature extraction through a multimodal fusion neural network model, and output environmental fusion feature data containing target behavior trend prediction; The path planning module 204 is configured to: input environmental fusion feature data and map information into the path planning decision network, complete dynamic path planning without disrupting the driving order, and output driving or parking control commands according to the driver takeover status. The interactive reminder module 205 is configured to use a gradient escalation alarm strategy for human-computer interactive reminders, and in conjunction with the control instructions of the path planning decision network, execute the primary alarm, escalation alarm and minimum risk safe parking operation in sequence.

[0065] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0066] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a control method for active activation of a combined driver assistance system as described in Embodiment 1.

[0067] Example 4 The purpose of this embodiment is to provide an electronic device.

[0068] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in a control method for active activation of a combined driver assistance system as described in Embodiment 1.

[0069] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0070] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0071] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A control method for actively activating a combined driver assistance system, characterized in that, include: Collect and preprocess driver operation information and visual monitoring information; The operation information and visual monitoring information are input into the trained driver state recognition model to determine whether the driver is in a state of unconsciousness. When the driver is determined to be in a state of unconsciousness, the combined driver assistance system is actively activated; visual images and three-dimensional perception information are collected through multiple sensors, and deep fusion and feature extraction are completed through a multimodal fusion neural network model to output environmental fusion feature data containing target behavior trend prediction. After splicing environmental feature data with map information, the data is input into the path planning decision network to complete dynamic path planning without disrupting the driving order, and output driving or parking control commands according to the driver's takeover status. A tiered alarm strategy is adopted for human-machine interaction reminders. Combined with the control commands of the path planning decision network, the system sequentially executes the primary alarm, the escalation alarm, and the minimum risk safe parking operation.

2. The control method for active activation of a combined driver assistance system as described in claim 1, characterized in that, Collect and preprocess driver operation information and visual monitoring information, including: The system collects driver operation information in real time through pressure sensors on the steering wheel and displacement sensors on the accelerator and brake pedals; it also collects image sequences of the driver's face, eyes, hands, and limb movements through an in-vehicle camera. The acquired image sequences are processed for face detection and cropping, noise reduction, and illumination normalization; the operational data acquired by the sensors are processed for outlier removal, timestamp alignment, and standardized mapping.

3. The control method for active activation of a combined driver assistance system as described in claim 1, characterized in that, The operational information and visual monitoring information are input into a trained driver state recognition model to determine whether the driver is in a state of unconsciousness, including: The preprocessed image frame sequence is input into the CNN spatial feature extraction layer of the driver state recognition model to extract the spatial dimension features of the driver's gaze direction, eye opening and closing, facial orientation, and the contact area between the hand and the steering wheel, and outputs a spatiotemporal feature vector. The spatiotemporal feature vector is input into the LSTM temporal analysis layer. Combined with the temporal attention mechanism, weights are assigned to the abnormal state features of the driver to enhance the extraction of temporal correlation features of abnormal states and output a temporal fusion feature vector. The temporal fusion feature vector is input into the fully connected decision layer, and the softmax classification function is used to output the driver's loss of consciousness state and the corresponding state confidence.

4. The control method for active activation of a combined driver assistance system as described in claim 1, characterized in that, Visual images and 3D perception information are acquired through multiple sensors, and deep fusion and feature extraction are performed using a multimodal fusion neural network model. The output is environmental fusion feature data containing target behavior trend predictions, including: Preprocess the acquired visual images and 3D perception information; The preprocessed multi-source data is input into a multimodal fusion neural network model. Visual semantic feature vectors are extracted based on a lightweight backbone network, and radar geometric feature vectors are extracted through a fully connected network. A cross-modal attention fusion layer is used to adjust the weights of bimodal features and output a cross-modal fused feature vector. The cross-modal fusion feature vector is reduced in dimensionality and redundancy is removed by the feature encoding layer, and the core environment fusion feature vector is output. Based on the core environment fusion feature vector, the environment fusion feature data is output by the multi-task prediction output layer.

5. The control method for active activation of a combined driver assistance system as described in claim 4, characterized in that, The environmental fusion feature data includes dynamic and static target fusion information, basic environmental fusion information, and parking feature information; Among them, the dynamic and static target fusion information includes target displacement velocity, acceleration, heading angle, lateral and longitudinal distance, type, size, and target behavior trend prediction results based on time series modeling output; The environmental infrastructure fusion information includes drivable area boundary modeling, road condition complexity classification, and traffic priority determination results; Parking feature information includes the location coordinates of the parking area, the area size, the distance from the vehicle, and the feasibility assessment results.

6. The control method for active activation of a combined driver assistance system as described in claim 1, characterized in that, Environmental feature data and map information are stitched together and input into the path planning decision network to complete dynamic path planning without disrupting traffic order. Based on the driver's takeover status, driving or parking control commands are output, including: By combining environmental fusion feature data with map information data, a path planning state tensor is constructed. The state tensor is input into a path planning decision network based on deep reinforcement learning. The key feature representations of the drivable and parking areas are enhanced by the feature extraction layer of the fully connected network. The policy network assigns weights to the features through a constrained attention mechanism, outputs the probability distribution of continuous actions, and generates control commands. A weighted reward function is introduced to evaluate the planning results, and the path is planned according to priority. Based on the driver takeover status, driving control commands or safe parking control commands combined with parking feature information are dynamically output.

7. The control method for active activation of a combined driver assistance system as described in claim 1, characterized in that, A tiered escalation alarm strategy is adopted for human-machine interaction alerts. Combined with control commands from the path planning decision network, the system sequentially executes primary alarms, escalation alarms, and minimum-risk safe parking operations, including: First, a primary alarm is triggered, and a takeover reminder is issued to the driver through at least one means, such as sound or light. At the same time, based on the driving control instructions of the route planning decision network, the vehicle is kept moving smoothly along the planned route. If the driver fails to take over, an upgraded alarm will be activated immediately, employing a differentiated reminder method that combines sound, light, and electricity. At the same time, the path will be dynamically optimized based on real-time environmental data, and speed reduction control commands will be output simultaneously. If the driver does not take over, the minimum risk safe parking operation will be activated. Based on the parking control instructions output by the path planning decision network, the vehicle will be controlled to move laterally and longitudinally, select a safe location, and complete a smooth parking. During the parking process, the vehicle's hazard lights will be activated to warn surrounding vehicles. After parking, the alarm will continue until the driver takes over.

8. A control system for active activation of a combined driver assistance system, characterized in that, include: The data acquisition and preprocessing module is configured to: acquire driver operation information and visual monitoring information and perform preprocessing; The driver state judgment module is configured to input the operation information and visual monitoring information into the trained driver state recognition model to determine whether the driver is in a state of unconsciousness. The environmental fusion feature data acquisition module is configured to: actively activate the combined driver assistance system when it is determined that the driver is in a state of unconsciousness; collect visual images and three-dimensional perception information through multiple sensors, complete deep fusion and feature extraction through a multimodal fusion neural network model, and output environmental fusion feature data containing target behavior trend prediction; The path planning module is configured to: stitch together environmental fusion feature data and map information and input them into the path planning decision network to complete dynamic path planning without disrupting the driving order, and output driving or parking control commands according to the driver takeover status. The interactive reminder module is configured to use a tiered escalation alarm strategy for human-computer interactive reminders, and in conjunction with the control instructions of the path planning decision network, execute the primary alarm, escalation alarm and minimum risk safe parking operation in sequence.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by the processor, the program implements the steps in the control method for active activation of a combined driver assistance system as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the control method for active activation of a combined driver assistance system as described in any one of claims 1-7.

Images