Ramp driving-in system and method for assisting driver in high-speed abnormal driving state

Abstract

The invention discloses a ramp entry system and method for assisting a driver in a high-speed abnormal driving state, and belongs to the technical field of auxiliary driving, the ramp entry system comprises a sensing module, a control module and an execution module, the sensing module comprises a driver monitoring system DMS, a road environment sensing module and a vehicle-mounted information sensing module, the sensing module is used for collecting driver states, traffic environment information and vehicle operation information, the control module comprises a central processing unit and is used for processing the information obtained by the sensing module and generating a control instruction, and the execution module comprises an acousto-optic early warning module, a steering system and an automatic driving system and is used for executing the control instruction; according to the ramp driving-in system and method for assisting the driver in the high-speed abnormal driving state, the driver can be effectively prevented from missing the ramp on the premise that the personal safety of the driver is highly guaranteed, and the passing efficiency of vehicles on the high-speed road is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of assisted driving, and in particular to a ramp entry system and method for assisting a driver in an abnormal driving state on a highway. Background Art

[0002] Most of the existing technical methods for assisting a driver to enter a highway ramp rely on navigation announcements, which ignore the possibility that the driver may not receive relevant signals and take corresponding actions in an abnormal driving state such as distraction or fatigue. To address this problem, Chinese invention patent CN114013453B proposes a ramp entry method and device for an autonomous driving vehicle. The prerequisite for this method is that the autonomous driving system takes control of the vehicle before entering the ramp, without considering the scenario where the driver controls the vehicle before entering the ramp, which has certain limitations for vehicles equipped with L2-level autonomous driving.

[0003] The phenomenon that drivers miss the ramp intersection on the highway is still common at present. There are also a small number of drivers who perform a series of dangerous behaviors after missing the ramp intersection, such as immediate stopping and reversing. Such behaviors are extremely dangerous on a road with a relatively high speed, which will not only cause large-area blockage of vehicles and local road paralysis, but also seriously threaten the personal safety of the driver.

[0004] First, the existing technologies on actual vehicles related to ramps currently provide real-time navigation announcements based on high-precision maps and positioning information, which are mainly used to remind the driver that there is a ramp ahead, without judging the driving state of the driver and monitoring the subsequent action responses of the driver. The monitoring and decision-making technologies after the reminder need to be further improved. Second, for the existing technical methods, they are aimed at the situation where the autonomous driving system takes control of the vehicle when approaching the ramp. The autonomous driving system makes corresponding decisions independently by combining map information and road environment information; while for another situation, that is, when approaching the ramp, the subsequent auxiliary methods for the driver to control the vehicle are not fully considered. At this time, the existing technical methods are completely ineffective. Third, the existing methods show that when the vehicle encounters special situations, the system will request the driver to take over the vehicle, and the system returns the driving right to the driver. The problem faced in this process is that the driver suddenly receives a takeover request in an abnormal driving state. After a reaction adjustment time, the ramp may have been missed, which will lead to the driver reducing trust in the system and affecting the driving experience of the driver. Summary of the Invention

[0005] The purpose of the present invention is to provide a ramp entry system and method for assisting a driver in an abnormal driving state on a highway to solve the problems mentioned in the above background art.

[0006] To achieve the above object, the present invention provides a ramp entry system for assisting a driver in abnormal driving states on a highway, including:

[0007] A sensing module, configured to collect driver status, traffic environment information, and vehicle operation information, including:

[0008] A driver monitoring system DMS, including an in-vehicle camera, for monitoring the driver's driving status in real time;

[0009] A road environment sensing module, including an in-vehicle navigation system, an in-vehicle map, an out-of-vehicle camera, and a lidar, for sensing traffic environment information;

[0010] An in-vehicle information sensing module, including a vehicle speed sensor, a steering wheel angle sensor, a turn signal sensor, and an accelerator and brake pedal opening sensor, for collecting vehicle operation information;

[0011] A control module: including a central processing unit, for processing the information obtained by the sensing module and generating control instructions;

[0012] An execution module, for executing the control instructions, including:

[0013] An acoustic and optical warning module, including a central control screen, an instrument panel, and a HUD, for sending warning signals to the driver;

[0014] A steering system, for controlling the vehicle steering operation;

[0015] An autonomous driving system, for taking over the vehicle when it is not possible to ensure a safe lane change.

[0016] Preferably, the driver monitoring system DMS obtains images of the driver's head and upper body through an in-vehicle camera. After being processed by an image processing module, the driving status of the driver is analyzed and judged through a driver monitoring algorithm, and the analysis result is output.

[0017] A method for assisting a driver to enter a ramp in abnormal driving states on a highway, including the following steps:

[0018] S1. Judge the driving status and attention of the driver through the driver monitoring system DMS. If the driver is in an abnormal driving state, enter S2;

[0019] S2. Sense the traffic environment through the road environment sensing module. If there is a ramp entry ahead on the road and it conforms to the driving route of the driver's destination, a voice broadcast is made;

[0020] S3. Monitor the driver's reaction through the driver monitoring system DMS to judge whether the driver has an intention to change lanes. If the driver does not respond, continuous warning is carried out through the acoustic and optical warning module. If the driver still does not respond, the warning stops. Otherwise, execute step S4;

[0021] S4. After detecting that the driver has the intention to change lanes, relevant information is obtained through the steering wheel angle sensor, vehicle speed sensor and turn signal sensor and transmitted to the central processor to determine whether the driver has executed the lane change operation;

[0022] S5. By combining the in-vehicle navigation with the current vehicle speed, the opening degrees of the accelerator pedal and the brake pedal, and the relevant information obtained by the external vehicle camera and lidar, it is judged whether it is safe to change lanes in the current environment. If it is safe, the ramp entry system continues to monitor. If it is not safe, the ramp entry system forcibly prevents the lane change and gives an early warning through the sound and light warning module, and executes step S6;

[0023] S6. The autonomous driving system requests to access and execute corresponding operations.

[0024] Preferably, the specific steps of S1 are as follows:

[0025] S11. After the driver gets on the vehicle, the ramp entry system prompts the driver through voice to face the in-vehicle camera in a standard posture to obtain the parameter characteristics of the driver's face in the natural state;

[0026] S12. Calculate the parameter characteristics of the driver's face between facing the camera and in the normal driving state according to the angle conversion formula, and set a threshold as the judgment standard;

[0027] S13. Judge whether the driver is in an abnormal driving state, and the abnormal driving state includes distracted driving and drowsy driving.

[0028] Preferably, the judgment basis for drowsy driving in S13 is the eye aspect ratio, mouth aspect ratio and head pose parameters, and the judgment basis for distracted driving is the number of times the driver gazes at the distracted area, gaze percentage, long gaze duration and long gaze percentage.

[0029] Preferably, the specific operation of S2 is: the ramp entry system obtains the optimal route of the driver's destination and whether there is a ramp entrance information in front of the road through the in-vehicle map and in-vehicle navigation. If there is a ramp entrance and it conforms to the driving route of the driver's destination, a voice broadcast is made.

[0030] Preferably, the specific steps of S3 are as follows:

[0031] S31. Through the driver gaze area estimation module of the driver monitoring system DMS, monitor whether the driver observes the right rearview mirror. If the driver is not observed to be looking at the right rearview mirror, the system defaults that the driver does not choose to enter the ramp. If the driver is observed to be looking at the right rearview mirror, make the next judgment;

[0032] S32. Determine whether the frequency of the driver frequently observing the left and right rearview mirrors within a specified time exceeds the initial threshold. If the frequency exceeds the threshold, it is defaulted that the driver has received the warning signal and made a response, and the operation process ends. If the frequency does not exceed the threshold, trigger further warning measures of the ramp entry system;

[0033] S33. The ramp entry system controls the in-vehicle navigation to continuously broadcast, and the central control screen, instrument panel, and HUD give a warning with flashing lights for 5 seconds. If the driver still does not make any response, it is defaulted that the driver does not choose to enter the ramp. Otherwise, proceed to the next step.

[0034] Preferably, the specific steps of S5 are as follows:

[0035] S51. Obtain relevant information through the in-vehicle navigation in combination with the current vehicle speed, the opening degrees of the accelerator pedal and the brake pedal, as well as the external vehicle camera and lidar, and continuously monitor the vehicle speed, acceleration and deceleration of the own vehicle, and whether there is a vehicle behind the own vehicle and the distance from the own vehicle;

[0036] S52. Transmit the above information to the central processing unit, and the central processing unit judges whether it is safe to change lanes in the current environment by comparing the safe lane change time and distance;

[0037] S53. If it is safe, the ramp entry system continuously monitors the distance between the vehicle behind and the own vehicle, and the central processing unit controls the in-vehicle audio to broadcast, and dynamically displays the distance from the vehicle behind through the instrument panel or HUD;

[0038] S54. If it is not safe, the ramp entry system issues an instruction to the steering system to forcibly prevent the vehicle from changing lanes, and gives an audible and visual warning through the in-vehicle audio, instrument panel, central control screen or HUD, and enters S6.

[0039] Therefore, the present invention adopts the above-mentioned ramp entry system and method for assisting the driver in abnormal driving states on highways, and has the following beneficial effects:

[0040] (1) On the one hand, it can effectively avoid the dangerous behaviors of the driver stopping and reversing after missing the ramp entrance. On the other hand, after the voice broadcasts that there is a ramp entrance ahead, it effectively monitors whether the driver has received the signal;

[0041] (2) It makes up for the blank of the method for the driver to master the vehicle control right and assist in ramp entry under the condition of L2-level automatic driving, and can effectively improve the traffic efficiency on highways and fully guarantee the road traffic and the personal safety of the driver.

[0042] The technical solutions of the present invention will be further described in detail below with reference to the drawings and embodiments. Description of the Drawings

[0043] Figure 1This is the structural composition diagram of a ramp entry system for assisting drivers in abnormal driving states on highways according to the present invention;

[0044] Figure 2 This is the flowchart of a method for assisting drivers in entering ramps in abnormal driving states on highways according to the present invention. Specific embodiments

[0045] The following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0046] As Figure 1 shown, the present invention provides a ramp entry system for assisting drivers in abnormal driving states on highways, including:

[0047] A sensing module for collecting driver status, traffic environment information, and vehicle operation information, including:

[0048] A driver monitoring system DMS, including algorithms such as in-vehicle cameras, driver driving state judgment, human body posture estimation, and gaze area recognition, for real-time monitoring of the driver's driving state;

[0049] A road environment sensing module, including in-vehicle navigation, in-vehicle maps, out-of-vehicle cameras, and lidar, for sensing traffic environment information;

[0050] An in-vehicle information sensing module, including a vehicle speed sensor, a steering wheel angle sensor, a turn signal sensor, and an accelerator and brake pedal opening sensor, for collecting vehicle operation information;

[0051] A control module: including a central processing unit for processing the information obtained by the sensing module and generating control instructions;

[0052] An execution module for executing control instructions, including:

[0053] An acoustic and optical warning module, including a central control screen, an instrument panel, and a HUD, for sending warning signals to the driver;

[0054] A steering system for controlling vehicle steering operations;

[0055] An autonomous driving system for taking over the vehicle when safe lane change cannot be guaranteed.

[0056] The Driver Monitoring System (DMS) obtains images of the driver's head and upper body through an in-vehicle camera installed in a suitable position. After being processed by the image processing module, it serves as the input for the model algorithm. The model performs operations such as feature extraction, and analyzes and judges the driver's driving state through the driver monitoring algorithm, and outputs the analysis result.

[0057] As Figure 2 shown, a method for assisting a driver to enter a ramp in an abnormal driving state at high speed includes the following steps:

[0058] S1. Judge the driver's driving state and attention through the Driver Monitoring System (DMS). If the driver is in an abnormal driving state, enter S2.

[0059] S11. After the driver gets in the car, the ramp entry system uses voice prompts to ask the driver to face the in-vehicle camera in a standard posture, and obtains the parameter features of the driver's face in a natural state.

[0060] S12. Calculate the parameter features of the driver's face when facing the camera and in the normal driving state according to the angle conversion formula, and set a certain range above and below this parameter as the threshold judgment standard.

[0061] S13. Judge whether the driver is in an abnormal driving state by comparing whether there are significant differences in the human posture, head posture, and fixation area between the current multi-frame images and the images of the driver in normal driving. The abnormal driving state includes distracted driving and drowsy driving.

[0062] The judgment basis for drowsy driving is the aspect ratio of the driver's eyes, the aspect ratio of the mouth, and the head posture parameters calculated by processing images through computer vision technology, so as to obtain whether the driver shows behaviors such as squinting, closing eyes, yawning, nodding, or shaking the head. If the number of occurrences of any of the above behaviors exceeds the set threshold, it is considered that the driver is in a drowsy driving state; distracted driving is mainly judged by the change of the driver's attention. It is stipulated that other areas except the area in front of the road and the areas of the left and right rearview mirrors are distraction areas. The judgment basis is the number of times the driver looks at the distraction area, the fixation percentage, the long fixation duration, and the long fixation percentage within a certain period of time.

[0063] S2. Sense the traffic environment through the road environment perception module and transmit the information to the driver. If there is a ramp entry in front of the road and it conforms to the driver's destination driving route, a voice broadcast is made.

[0064] The ramp entry system obtains the optimal route of the driver's destination and whether there is a ramp entrance information in front of the road through the in-vehicle map and in-vehicle navigation. If there is a ramp entrance and it conforms to the driver's destination driving route, a voice broadcast is made and the next step is entered.

[0065] S3. Monitor the driver's reaction through the Driver Monitoring System (DMS) to determine whether the driver has the intention to change lanes. If the driver shows no reaction, continuously give warnings through the acoustic and optical warning module. If the driver still shows no reaction, stop the warning. Otherwise, execute step S4.

[0066] S31. After giving a voice warning, monitor through the driver's gaze area estimation module of the Driver Monitoring System (DMS) whether the driver observes the right rearview mirror. If the driver is not observed to be looking at the right rearview mirror, the system defaults that the driver does not choose to enter the ramp. If the driver is observed to be looking at the right rearview mirror, make the next judgment.

[0067] S32. Determine whether the frequency of the driver frequently observing the left and right rearview mirrors within a specified time exceeds the initial threshold, which can be obtained through experience. If the frequency exceeds the threshold, it is defaulted that the driver has received the warning signal and made a reaction, and the operation process ends. If the frequency does not exceed the threshold, trigger further warning measures of the ramp entry system.

[0068] S33. The ramp entry system controls the in-vehicle navigation to continuously broadcast, and there are light flashing warnings on the central control screen, instrument panel, and HUD for 5 seconds. If the driver still shows no reaction, it is defaulted that the driver does not choose to enter the ramp. Otherwise, proceed to the next step.

[0069] S4. After detecting that the driver has the intention to change lanes, obtain information such as the steering wheel angle of the vehicle through the steering wheel angle sensor, the driving speed of the vehicle through the vehicle speed sensor, and whether the right turn signal is on through the turn signal sensor, and transmit this information to the central processing unit. The central processing unit determines whether the driver has executed the lane change operation through the built-in algorithm.

[0070] S5. Combine the in-vehicle navigation with the current vehicle speed, the opening degrees of the accelerator pedal and the brake pedal, and the relevant information obtained by the external camera and lidar to judge whether it is safe to change lanes in the current environment. If it is safe, the ramp entry system continuously monitors. If it is not safe, the ramp entry system forcibly prevents the lane change and gives a warning through the acoustic and optical warning module, and execute step S6;

[0071] S51. Combine the in-vehicle navigation with the current vehicle speed, the opening degrees of the accelerator pedal and the brake pedal, and the external camera and lidar to obtain relevant information, and continuously monitor the vehicle speed, acceleration and deceleration of the own vehicle, and whether there is a vehicle behind the own vehicle and the distance from the own vehicle.

[0072] S52. Transmit the above information to the central processing unit, and the central processing unit judges whether it is safe to change lanes in the current environment by comparing the safe lane change time and distance.

[0073] S53. If it is safe, after the ramp entry system monitors the distance between the following vehicle and the own vehicle in real time, the central processor controls the in-vehicle audio to broadcast, and dynamically displays the distance from the following vehicle through the instrument panel or HUD;

[0074] S54. If it is not safe, the ramp entry system issues an instruction to the steering system to forcibly prevent the vehicle from changing lanes, and gives an audible and visual warning through the in-vehicle audio, instrument panel, center console screen or HUD, and enters S6.

[0075] S6. If the auxiliary ramp entry system determines that the current environment cannot ensure a safe lane change, the automatic driving system requests to access, and then the automatic driving system performs corresponding operations on the premise of ensuring safety.

[0076] Therefore, the present invention adopts the above-mentioned auxiliary ramp entry system and method for the driver in an abnormal driving state on the highway, which can effectively prevent the driver from missing the ramp on the premise of highly ensuring the personal safety of the driver, and greatly improve the traffic efficiency of vehicles on the highway.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that they can still modify or equivalently replace the technical solutions of the present invention, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. An on-ramp entry system for assisting a driver in a high-speed abnormal driving state, characterized in that Including: A perception module, configured to collect driver status, traffic environment information, and vehicle operation information, including: A driver monitoring system DMS, including an in-vehicle camera, configured to monitor the driving status of the driver in real time; A road environment perception module, including an in-vehicle navigation system, an in-vehicle map, an out-vehicle camera, and a lidar, configured to perceive traffic environment information; An on-vehicle information perception module, including a vehicle speed sensor, a steering wheel angle sensor, a turn signal sensor, and an accelerator and brake pedal opening sensor, configured to collect vehicle operation information; A control module: including a central processing unit, configured to process the information obtained by the perception module and generate control instructions; An execution module, configured to execute the control instructions, including: An acoustic and optical warning module, including a central control screen, an instrument panel, and a HUD, configured to send warning signals to the driver; A steering system, configured to control the steering operation of the vehicle; An autonomous driving system, configured to take over the vehicle when it is impossible to ensure a safe lane change.

2. The ramp entry system for assisting a driver in an abnormal driving state at high speed according to claim 1, wherein: The driver monitoring system DMS obtains images of the driver's head and upper body through the in-vehicle camera. After being processed by the image processing module, the driving status of the driver is analyzed and judged through the driver monitoring algorithm, and the analysis result is output.

3. A ramp entry method for assisting a driver in a high-speed abnormal driving state, characterized in that, Applied to a ramp entry system for assisting a driver in an abnormal driving state at high speed as described in any one of claims 1-2, including the following steps: S1. Judge the driving status and attention of the driver through the driver monitoring system DMS. If the driver is in an abnormal driving state, enter S2; S2. Perceive the traffic environment through the road environment perception module. If there is a ramp entry ahead on the road and it conforms to the driving route of the driver's destination, a voice broadcast is performed; S3. Monitor the driver's reaction through the driver monitoring system DMS to judge whether the driver has an intention to change lanes. If the driver does not respond, continuous warning is performed through the acoustic and optical warning module. If the driver still does not respond, the warning stops. Otherwise, execute step S4; S4. After detecting that the driver has an intention to change lanes, obtain relevant information through the steering wheel angle sensor, vehicle speed sensor, and turn signal sensor and transmit it to the central processing unit to judge whether the driver has executed the lane change operation; S5. Combine the current vehicle speed, the opening conditions of the accelerator pedal and the brake pedal, and the relevant information obtained by the out-vehicle camera and the lidar through the in-vehicle navigation system to judge whether the current environment lane change is safe. If it is safe, the ramp entry system continuously monitors. If it is not safe, the ramp entry system forcibly prevents the lane change and issues a warning through the acoustic and optical warning module, and execute step S6; S6. The autonomous driving system requests to access and execute corresponding operations.

4. The ramp entry method for assisting a driver in an abnormal driving state on a highway according to claim 3, characterized in that, The specific steps of S1 are as follows: S11. After the driver gets in the vehicle, the ramp entry system prompts the driver through voice to face the in-vehicle camera in a standard posture to obtain the parameter characteristics of the driver's face in a natural state; S12. Calculate the face feature parameters of the driver when facing the camera and in the normal driving state according to the angle conversion formula, and set a threshold as the judgment standard; S13. Judge whether the driver is in an abnormal driving state. The abnormal driving state includes distracted driving and fatigued driving.

5. The ramp entry method for assisting a driver in an abnormal driving state at high speed according to claim 4, characterized in that: The judgment basis for fatigue driving in S13 is the aspect ratio of the eyes, the aspect ratio of the mouth, and the head pose parameters, and the judgment basis for distracted driving is the number of times the driver gazes at the distracted area, the fixation percentage, the long fixation duration, and the long fixation percentage.

6. The ramp entry method for assisting a driver in an abnormal driving state on a highway according to claim 3, wherein The specific operation of S2 is as follows: The ramp entry system obtains the optimal route to the driver's destination and whether there is a ramp entry information ahead on the road through the in-vehicle map and in-vehicle navigation. If there is a ramp entry and it conforms to the driver's destination driving route, a voice prompt will be given.

7. The ramp entry method for assisting a driver in an abnormal driving state at high speed according to claim 3, characterized in that The specific steps of S3 are as follows: S31. Monitor whether the driver observes the right rearview mirror through the driver gaze area estimation module of the driver monitoring system DMS. If the driver is not observed to be looking at the right rearview mirror, the system defaults that the driver does not choose to enter the ramp. If the driver is observed to be looking at the right rearview mirror, proceed to the next judgment. S32. Judge whether the frequency of the driver frequently observing the left and right rearview mirrors within a specified time exceeds the initial threshold. If the frequency exceeds the threshold, it is defaulted that the driver has received the warning signal and made a response, and the operation process ends. If the frequency does not exceed the threshold, trigger further warning measures of the ramp entry system. S33. The ramp entry system controls the in-vehicle navigation to continuously broadcast, and the central control screen, instrument panel, and HUD emit a flashing light warning for 5 seconds. If the driver still does not make any response, it is defaulted that the driver does not choose to enter the ramp. Otherwise, proceed to the next step.

8. The ramp entry method for assisting a driver in an abnormal driving state at high speed according to claim 3, wherein The specific steps of S5 are as follows: S51. Obtain relevant information through the in-vehicle navigation in combination with the current vehicle speed, the opening degrees of the accelerator pedal and the brake pedal, as well as the external vehicle camera and lidar, and continuously monitor the vehicle speed, acceleration and deceleration of the host vehicle, and whether there is a vehicle behind the host vehicle and the distance from the host vehicle. S52. Transmit the above information to the central processing unit, and the central processing unit judges whether it is safe to change lanes in the current environment by comparing the safe lane change time and distance. S53. If it is safe, the ramp entry system continuously monitors the distance between the vehicle behind and the host vehicle, and the central processing unit controls the in-vehicle audio to broadcast, and dynamically displays the distance from the vehicle behind through the instrument panel or HUD. S54. If it is not safe, the ramp entry system issues an instruction to the steering system to forcibly prevent the vehicle from changing lanes, and gives an audible and visual warning through the in-vehicle audio, instrument panel, central control screen or HUD, and enters S6.

Images