Multi-algorithm cooperative intelligent driving safety early warning system

Abstract

The application belongs to the field of intelligent driving safety warning, and relates to data analysis technology, and is used for solving the problem that the prior art cannot consider the timeliness of warning and the consistency of results when multiple algorithms are used for warning, and specifically relates to an intelligent driving safety warning system based on multiple algorithms, which comprises a safety warning platform, and the safety warning platform is communicatively connected with a driving monitoring module, a driving monitoring module, a road monitoring module, a multi-level warning module and a database; the scheme of the application can effectively solve the problem of how to accurately evaluate road safety when the monitoring results of multiple sensors are inconsistent; under complex and variable environmental conditions, the reliability of different monitoring methods will be affected by different key parameters; by quantifying these influences and selecting the monitoring result with the highest confidence under the current environment as the final judgment basis, false positives or false negatives caused by the failure of a single or partial sensor under certain conditions can be avoided; the robustness and accuracy of the intelligent driving safety warning system under special scenarios are significantly improved.

Description

Technical Field

[0001] This invention belongs to the field of intelligent driving safety early warning and involves data analysis technology, specifically an intelligent driving safety early warning system with multi-algorithm collaboration. Background Technology

[0002] Traditional single-algorithm early warning systems (such as those relying solely on rules or a single model) are prone to failure in complex, dynamic, and long-tailed open road scenarios. The core idea of ​​a multi-algorithm collaborative intelligent driving safety early warning system is to integrate the advantages of multiple perception, prediction, and decision-making algorithms, and through information fusion and collaborative decision-making, achieve more comprehensive, robust, and timely identification and early warning of potential risks, thereby providing valuable "safety redundancy" for vehicle control systems or drivers.

[0003] The invention patent with publication number CN111452723B discloses an intelligent vehicle and its warning control method, device, system, and storage medium. This control method can effectively identify gaps in the median strip and the corresponding first target position, target obstacles, and the corresponding second target position. When the first target position and the second target position are close, it can be determined that the gap in the median strip is caused by the obstruction of the target obstacle, thereby generating a vehicle warning signal in a timely manner and effectively improving the safety of intelligent driving. However, this control method only uses a single data structure and analysis method for obstacle identification and risk analysis, resulting in low accuracy of risk analysis results. Existing intelligent driving vehicles are generally equipped with several types of obstacle identification devices and analysis algorithms, but the fusion analysis of multi-source heterogeneous data to generate monitoring results will affect the computational efficiency and often lead to untimely warnings. Furthermore, parallel monitoring by multiple algorithms will result in output redundancy and disorder when the results are inconsistent.

[0004] To address the aforementioned technical problems, this application proposes a solution. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-algorithm collaborative intelligent driving safety warning system to solve the problem that existing technologies cannot simultaneously ensure the timeliness of warnings and the consistency of results when multiple algorithms provide warnings.

[0006] The technical problem to be solved by this invention is: how to provide a multi-algorithm collaborative intelligent driving safety warning system that can take into account both the timeliness of warnings and the consistency of results when multiple algorithms provide warnings.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] A multi-algorithm collaborative intelligent driving safety early warning system includes a safety early warning platform, which is communicatively connected to a driving monitoring module, a vehicle monitoring module, a road monitoring module, a multi-level early warning module, and a database.

[0009] The driving monitoring module is used to monitor and analyze the driver's driving status: it acquires the driver's facial information in real time through the cockpit camera, extracts the driver's eye and facial data in the most recent L1 seconds, and assesses the driver's fatigue status through the eye and facial data.

[0010] The driving monitoring module is used to monitor and analyze the vehicle's driving status: acquiring emergency data and deviation data during vehicle driving, and evaluating the vehicle's driving status based on the driving data and deviation data;

[0011] The road monitoring module is used to monitor and analyze the road safety of vehicles: it uses optical monitoring, laser monitoring and millimeter wave monitoring to monitor road obstacles.

[0012] The multi-level early warning module is used to perform multi-level early warning analysis on the road safety of vehicles: it compares and analyzes the road safety monitoring results of optical monitoring, laser monitoring, and millimeter wave monitoring; if they are consistent and all meet the requirements, the road safety is deemed qualified; if they are consistent and all do not meet the requirements, the road safety is deemed unqualified, a road early warning signal is generated and sent to the controller, and the controller issues a safety warning upon receiving the road early warning signal; otherwise, the two consistent monitoring results are marked as the immediate output results, and confidence analysis is performed simultaneously.

[0013] Furthermore, the eye data is the proportion of time the eyes close within the most recent L1 second, calculated using the PERCLOS algorithm, while the facial data is the sum of the driver's yawning and nodding frequencies within the most recent L1 second.

[0014] Furthermore, the specific process for assessing driver fatigue includes: obtaining eye and facial thresholds from a database, comparing the eye and facial data with the eye and facial thresholds respectively; if the eye data is less than the eye threshold and the facial data is less than the facial threshold, then the driver's fatigue status is determined to meet the requirements; otherwise, the driver's fatigue status is determined to not meet the requirements, a fatigue warning signal is generated and sent to the controller, and the controller issues a safety warning upon receiving the fatigue warning signal.

[0015] Furthermore, the process of acquiring driving data and deviation data includes: acquiring the distance to the vehicle in front and the driving speed of the current vehicle in real time, marking the ratio of the distance to the vehicle in front to the driving speed of the current vehicle as emergency data, and acquiring the maximum duration of continuous deviation of the vehicle from the lane line within the most recent L1 seconds and marking it as deviation data.

[0016] Furthermore, the specific process of evaluating the vehicle's driving status includes: obtaining the emergency threshold and deviation threshold from the database, comparing the emergency data and deviation data with the emergency threshold and deviation threshold respectively; if the emergency data is less than the emergency threshold and the deviation data is less than the deviation threshold, then the vehicle's driving status is determined to meet the requirements; otherwise, the vehicle's driving status is determined to not meet the requirements, a driving warning signal is generated and sent to the controller, and the controller issues a safety warning after receiving the driving warning signal.

[0017] Furthermore, the specific process of optical monitoring includes: recording the road where the vehicle is traveling using an onboard camera; decomposing the acquired video recording into continuous video frame images; performing noise reduction, HDR, and color enhancement on the video frame images; then using a deep learning model to extract pixel regions in the video frame images that match obstacle characteristics, including pedestrians, vehicles, cyclists, and traffic signs; the deep learning models include YOLO and CNN; associating the same target in consecutive frames using a multi-target tracking algorithm to form an obstacle trajectory; the multi-target tracking algorithms include SORT and DeepSORT; obtaining the radial relative velocity between the vehicle and the obstacle using the obstacle trajectory and marking it as an optical warning coefficient; and using the optical warning coefficient to determine the road safety monitoring results corresponding to the optical monitoring.

[0018] Furthermore, the specific process for determining the road safety monitoring results corresponding to optical monitoring includes: obtaining the optical warning threshold from the database, and comparing the optical warning coefficients of all obstacles with the optical warning threshold in sequence; if all optical warning coefficients are less than the optical warning threshold, the road safety monitoring results corresponding to optical monitoring are determined to meet the requirements; otherwise, the road safety monitoring results corresponding to optical monitoring are determined to not meet the requirements.

[0019] Furthermore, the specific process of laser monitoring includes: emitting a laser beam through a vehicle-mounted laser emitter and receiving its reflected signal; calculating the distance between the vehicle and the obstacle using the time of flight; marking the ratio of distance to vehicle speed as the laser warning coefficient; obtaining the laser warning threshold from the database; and comparing the laser warning coefficient with the laser warning threshold: if the laser warning coefficient is less than the laser warning threshold, the road safety monitoring result corresponding to the laser monitoring is deemed to meet the requirements; otherwise, the road safety monitoring result corresponding to the laser monitoring is deemed not to meet the requirements.

[0020] Furthermore, the specific process of millimeter-wave monitoring includes: transmitting radio waves, detecting the radial relative velocity between the vehicle and the obstacle by analyzing the frequency changes of the reflected waves and marking it as a millimeter-wave warning coefficient, obtaining the millimeter-wave warning threshold from the database, and comparing the millimeter-wave warning coefficient with the millimeter-wave warning threshold: if the millimeter-wave warning coefficient is less than the millimeter-wave warning threshold, it is determined that the road safety monitoring result corresponding to the millimeter-wave monitoring meets the requirements; otherwise, it is determined that the road safety monitoring result corresponding to the millimeter-wave monitoring does not meet the requirements.

[0021] Furthermore, the specific process of confidence analysis includes: retrieving the values ​​of key parameters and corresponding failure thresholds for optical monitoring, laser monitoring, and millimeter-wave monitoring. The key parameters for optical monitoring, laser monitoring, and millimeter-wave monitoring are light intensity, rainfall, and road curvature, respectively. The ratio of the value of the key parameter to the corresponding failure threshold is marked as the confidence value of the monitoring method. The road safety monitoring result corresponding to the monitoring method with the smallest confidence value is marked as the updated output result. The control update of the road safety warning is performed through the updated output result.

[0022] The present invention has the following beneficial effects:

[0023] 1. Judgment is made based on the comparison of real-time data with these thresholds, which effectively avoids the ambiguity of subjective judgment and makes the identification of fatigue state more accurate and reliable; when the driver is judged to be in a fatigue state, the system can generate a fatigue warning signal in time and send it to the controller; key parameters during vehicle driving are acquired and processed in real time and dynamically, and highly indicative emergency data and deviation data are constructed, thus providing a solid data foundation for the driving monitoring module to accurately assess the vehicle driving status;

[0024] 2. The optical monitoring process of this application is refined and intelligent. The vehicle-mounted camera, combined with advanced image processing technology, ensures the acquisition of high-quality visual data under various complex lighting and environmental conditions. The efficient application of deep learning models enables the system to accurately and quickly identify various obstacles on the road, avoiding the limitations of traditional visual algorithms in complex scenarios. The introduction of multi-target tracking algorithms not only continuously tracks the movement of obstacles, but more importantly, it can form accurate obstacle trajectories, providing crucial dynamic information for subsequent risk assessment.

[0025] 3. The solution proposed in this application can effectively address the problem of accurately assessing road safety when the monitoring results from multiple sensors are inconsistent. Under complex and variable environmental conditions, the reliability of different monitoring methods is affected by different key parameters. This solution quantifies these effects and selects the monitoring result with the highest confidence level under the current environment as the final judgment basis, avoiding misjudgments or omissions caused by the failure of a single or partial sensor under specific conditions. This significantly improves the robustness and accuracy of the intelligent driving safety warning system under adverse weather, insufficient lighting, or special road geometry conditions, thereby providing drivers with more reliable and timely safety warnings and effectively reducing the potential risk of traffic accidents. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a system block diagram of Embodiment 1 of the present invention;

[0028] Figure 2 This is a flowchart of the method in Embodiment 2 of the present invention. Detailed Implementation

[0029] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] In traditional multi-algorithm collaborative intelligent driving safety warning systems, the fusion and analysis process of multi-source heterogeneous data leads to a decrease in the accuracy of risk analysis. At the same time, the parallel monitoring of multiple algorithms produces output redundancy and disorder when the results are inconsistent. The computational complexity of data fusion significantly affects the timeliness of warning response, while the conflicting state of monitoring results further weakens the reliability of system decision-making. As a result, the timeliness and accuracy of safety warnings cannot meet the needs of dynamic open road scenarios.

[0031] For example, in low-visibility rainy or foggy weather conditions, when a vehicle is driving on a highway curve, the optical monitoring system suffers from image quality degradation due to rainwater adhering to the camera, the laser monitoring system experiences distance measurement deviations due to rain interference, and the millimeter-wave monitoring system misjudges radial relative speed due to changes in road curvature. At this time, the road safety monitoring results output by each monitoring module are contradictory, and the system cannot quickly determine road safety. Furthermore, the multi-level warning modules trigger redundant processing procedures during the comparison and analysis process due to inconsistent results, resulting in a delay in the generation of warning signals. Specifically, the effective warning is not output in the cockpit in a timely manner, and the vehicle control system cannot obtain sufficient safety redundancy time to intervene.

[0032] If the above problems are not addressed, the decision-making reliability of the early warning system will be continuously compromised, and it may be unable to effectively identify potential risks in complex traffic scenarios, thereby increasing the likelihood of accidents. At the same time, erratic system output will reduce drivers' trust in the early warning mechanism, thus affecting the overall application effectiveness of intelligent driving technology.

[0033] For ease of understanding, the following explains some key terms in this embodiment:

[0034] Safety Early Warning Platform: This platform is the core of the entire intelligent driving safety early warning system. It is responsible for integrating, coordinating, and processing data from different monitoring modules, thereby generating and managing safety early warning information. Its main function is to serve as an information aggregation and decision-making center, ensuring the coordinated execution of various safety monitoring tasks.

[0035] Database: This database stores various parameters, thresholds, historical data, and model information required for system operation. For example, eye-related thresholds and facial thresholds required for driver fatigue assessment, emergency thresholds and deviation thresholds required for vehicle driving status assessment, and optical warning thresholds, laser warning thresholds, and millimeter-wave warning thresholds required for road safety monitoring are all stored in this database for querying and use by various modules.

[0036] Example 1: As Figure 1 As shown, the intelligent driving safety early warning system with multi-algorithm collaboration includes a safety early warning platform, which is communicatively connected to a driving monitoring module, a vehicle monitoring module, a road monitoring module, a multi-level early warning module, and a database.

[0037] The driving monitoring module is used to monitor and analyze the driver's driving status: it acquires the driver's facial information in real time through the cockpit camera, extracts the driver's eye and facial data within the most recent L1 second. The eye data is the proportion of time the eyes are closed within the most recent L1 second calculated using the PERCLOS algorithm, and the facial data is the sum of the driver's yawning and nodding frequencies within the most recent L1 second. It obtains eye and facial thresholds from the database and compares the eye and facial data with the eye and facial thresholds respectively: if the eye data is less than the eye threshold and the facial data is less than the facial threshold, the driver's fatigue status is determined to meet the requirements; otherwise, the driver's fatigue status is determined to not meet the requirements, a fatigue warning signal is generated and sent to the controller, and the controller issues a safety warning upon receiving the fatigue warning signal.

[0038] The PERCLOS algorithm is a widely used objective indicator for fatigue detection. Its core lies in quantifying the degree and duration of eye closure for drivers. By calculating the proportion of eye closure within a specific time period (e.g., the most recent L1 second), the driver's level of drowsiness can be directly reflected. Implementation methods include: using images captured by an in-vehicle camera, identifying eye regions through image processing technology, and monitoring the opening and closing of the eyelids in real time to calculate the proportion of closure time; or, using a dedicated eye-tracking device to directly acquire eyelid position data and calculate the PERCLOS value based on this data. Yawning and nodding are typical behavioral manifestations of driver fatigue. Including the frequency of these two behaviors as part of facial data can supplement eye-related data in the assessment of fatigue status from a behavioral perspective. The implementation methods may include: analyzing the video stream captured by the cockpit camera using computer vision technology, identifying the driver's facial feature points, judging yawning behavior based on the degree and duration of mouth opening, judging nodding behavior based on head posture changes and movement trajectories, and then counting the number of occurrences in the most recent L1 seconds; or, combining deep learning models to train the model to recognize yawning and nodding patterns, thereby more accurately counting their frequency.

[0039] The driving monitoring module is used to monitor and analyze the vehicle's driving status: it acquires the distance to the vehicle in front and the speed of the current vehicle in real time, marks the ratio of the distance to the current vehicle as emergency data, acquires the maximum duration of continuous lane departure within the most recent L1 seconds and marks it as deviation data, retrieves emergency thresholds and deviation thresholds from the database, and compares the emergency data and deviation data with the emergency thresholds and deviation thresholds respectively: if the emergency data is less than the emergency threshold and the deviation data is less than the deviation threshold, the vehicle's driving status is determined to meet the requirements; otherwise, the vehicle's driving status is determined to not meet the requirements, a driving warning signal is generated and sent to the controller, and the controller issues a safety warning upon receiving the driving warning signal.

[0040] The ratio of the distance to the vehicle ahead to the vehicle's speed is labeled as emergency data, which quantifies the relative safety margin between the two vehicles. This ratio visually represents the time required for the vehicle to reach the position of the vehicle ahead at the current speed, i.e., the "time interval" or "time gap." The smaller the ratio, the shorter the safe distance between the vehicle and the vehicle ahead, the higher the potential collision risk, and the more urgent the driving situation. In practical calculations, the real-time measured distance to the vehicle ahead (e.g., in meters) can be directly divided by the vehicle's speed (e.g., in meters per second) to obtain emergency data in seconds. Furthermore, to improve data stability, the raw distance and speed data can be appropriately filtered or smoothed before calculation to eliminate the effects of instantaneous noise or fluctuations.

[0041] The system acquires and marks the maximum duration of continuous lane departures within the most recent L1 second as deviation data. This deviation data measures the severity of the vehicle's deviation from the predetermined lane line during driving. Lane departures are often signs of driver inattention, fatigue, or loss of vehicle control. The longer the continuous lane departure, the more severe the departure and the greater the potential danger. In practice, this can be achieved using an onboard camera combined with image recognition technology to detect the lane line position and the vehicle's lateral position relative to the lane line in real time. When the vehicle's tires cross the lane line or the vehicle body exceeds the lane boundary, the system starts timing and records the duration of this deviation. Within a preset L1-second time window, the system records the duration of all continuous lane departures and selects the maximum value as the deviation data. Another approach is to utilize a high-precision positioning system (such as RTK-GPS) combined with pre-installed high-precision map data to compare the vehicle's actual driving trajectory with the lane centerline in real time. Once the vehicle's lateral position exceeds a preset safe area, the system starts timing the deviation duration and takes the maximum value within L1 seconds.

[0042] The road monitoring module is used to monitor and analyze the road safety of vehicles: it uses optical monitoring, laser monitoring and millimeter wave monitoring to monitor road obstacles.

[0043] The specific process of optical monitoring includes: recording the road surface where the vehicle is traveling using an onboard camera; decomposing the acquired video recording into consecutive video frames; performing noise reduction, HDR, and color enhancement on the video frames; then using a deep learning model to extract pixel regions in the video frames that match obstacle characteristics, including pedestrians, vehicles, cyclists, and traffic signs; using deep learning models such as YOLO and CNN; associating the same target in consecutive frames using a multi-target tracking algorithm to form an obstacle trajectory; using multi-target tracking algorithms such as SORT and DeepSORT; obtaining the radial relative velocity between the vehicle and obstacles from the obstacle trajectory and marking it as an optical warning coefficient; obtaining the optical warning threshold from a database; and comparing the optical warning coefficients of all obstacles sequentially with the optical warning threshold; if all optical warning coefficients are less than the optical warning threshold, the road safety monitoring result corresponding to the optical monitoring is deemed to meet the requirements; otherwise, the road safety monitoring result corresponding to the optical monitoring is deemed not to meet the requirements.

[0044] Among them, an in-vehicle camera is an image or video acquisition device installed on a vehicle, whose function is to capture real-time information about the road environment in front of or around the vehicle. It can be a monocular camera, a binocular camera, or a multi-camera array, providing visual input and forming the basis for intelligent driving systems to perceive the external environment. Decomposing the acquired video recording into consecutive video frames transforms the dynamic video stream into discrete image data that can be processed by computer vision algorithms, allowing for independent analysis and processing of each frame. Denoising the video frames involves using algorithms to eliminate unwanted random noise in the image, such as graininess caused by low lighting conditions or sensor defects, to improve image clarity and the accuracy of subsequent processing. Common denoising methods include mean filtering, Gaussian filtering, median filtering, or more complex deep learning-based denoising networks. HDR (High Dynamic Range) processing aims to expand the brightness range of an image, enabling it to simultaneously present details in extremely bright and extremely dark areas. This is crucial for ensuring the visibility of all important areas (such as roads and obstacles) in complex lighting conditions such as strong light, backlight, or tunnel entrances and exits. This can be achieved by fusing multiple images with different exposures, or through a local tone mapping algorithm on a single image. Color enhancement refers to adjusting the color saturation, contrast, or brightness of an image to make the image content easier to identify and analyze. For example, the contrast of specific colors (such as the red and yellow of traffic signs) can be enhanced to make them stand out more against a complex background.

[0045] Deep learning models are used to extract pixel regions in video frame images that match obstacle features. Deep learning models are machine learning methods based on neural networks, capable of learning and recognizing complex patterns from large amounts of data. Here, it is used to analyze processed video frame images, automatically identifying and locating pixel regions with obstacle features. Obstacles include pedestrians, vehicles, cyclists, and traffic signs—common target types in road environments that pose potential threats to driving safety and require system identification and attention. Deep learning models include YOLO and CNN. YOLO (You Only Look Once) is a real-time object detection algorithm characterized by its ability to simultaneously predict the bounding boxes and categories of all objects in an image within a single neural network, boasting extremely high detection speed and suitability for real-time intelligent driving scenarios. CNN (Convolutional Neural Network) is a neural network structure in the field of deep learning specifically designed for processing grid-like data such as images and videos. CNNs, through structures such as convolutional layers and pooling layers, can automatically learn features in images and are widely used in tasks such as image classification, object detection, and semantic segmentation.

[0046] As a specific implementation, the vehicle-mounted camera can be a high-resolution wide-angle camera, such as a CMOS image sensor with a resolution of 1920x1080 pixels and a frame rate of 30 frames per second. The acquired video recordings are processed by the vehicle-mounted computing unit, decomposing them into 30 independent video frames per second. In the image preprocessing stage, a denoising algorithm based on non-local means can be used to reduce image noise; HDR processing can employ an exposure fusion algorithm to fuse images acquired under different exposure parameters to obtain a wider dynamic range; color enhancement can be achieved by adjusting the image's gamma value and saturation. In the obstacle extraction stage, the deep learning model can specifically adopt the YOLOv7 model based on the ResNet-50 backbone network. This model is pre-trained on a mixed dataset containing a large number of road scene images (such as the COCO dataset and the BDD100K dataset) to identify various obstacles such as pedestrians, vehicles, cyclists, and traffic signs. Multi-target tracking algorithms can employ DeepSORT, where a Kalman filter predicts the target's position in the next frame, while a deep feature extraction network (such as a ReID network) is used for re-identification after the target is occluded, ensuring tracking continuity. The obstacle trajectory generated by DeepSORT allows calculation of the rate of change of distance between the vehicle and the center of gravity of each obstacle, i.e., the radial relative velocity. For example, if the obstacle's positions in two consecutive frames are (x1, y1) and (x2, y2), and the vehicle's positions are (cx1, cy1) and (cx2, cy2), the radial relative velocity can be calculated using geometric relationships and time intervals. This radial relative velocity is used as an optical warning coefficient and compared with a preset optical warning threshold. For instance, if the absolute value of the radial relative velocity exceeds a certain threshold (indicating rapid approach or departure), it is deemed unsafe.

[0047] The specific process of laser monitoring includes: emitting a laser beam through a vehicle-mounted laser emitter and receiving its reflected signal; calculating the distance between the vehicle and the obstacle using the time of flight; marking the ratio of distance to vehicle speed as the laser warning coefficient; obtaining the laser warning threshold from a database; and comparing the laser warning coefficient with the laser warning threshold: if the laser warning coefficient is less than the laser warning threshold, the road safety monitoring result corresponding to the laser monitoring is deemed to meet the requirements; otherwise, the road safety monitoring result corresponding to the laser monitoring is deemed not to meet the requirements.

[0048] The vehicle-mounted laser emitter emits a laser beam towards the area in front of or around the vehicle. It can employ a LiDAR (Light Detection and Ranging) sensor based on solid-state or semiconductor lasers, or use scanning mirrors or phased array technology to scan the laser beam. The reflected signal refers to the signal reflected back from an obstacle after the laser beam emitted by the vehicle-mounted laser emitter encounters it; this signal carries information such as the obstacle's position and distance. The time of flight (TOF) is the time it takes for the laser beam to travel from the vehicle-mounted laser emitter, through its propagation, reflection from an obstacle, and finally to its reception by the receiver. By accurately measuring this time, the distance the laser beam travels can be calculated. The distance between the vehicle and the obstacle is calculated based on the TF and represents the spatial interval between the vehicle and obstacles in front or to the side. It is typically calculated by multiplying the speed of light by half the TF. The vehicle speed refers to the vehicle's current real-time speed, which can be obtained through onboard sensors such as wheel speed sensors, inertial measurement units (IMUs), or global positioning systems (GPS). The laser warning coefficient is the ratio of the distance between the vehicle and the obstacle to the vehicle's speed. It quantifies the time required for the vehicle to reach the obstacle at the current speed, providing a direct reflection of potential collision risks. The database stores various parameters and thresholds required for system operation, such as the laser warning threshold mentioned in this embodiment. This can be a local storage unit, such as an onboard memory, or a cloud server connected to the vehicle. The laser warning threshold is a preset standard value used to determine whether the laser monitoring results meet safety requirements. This threshold can be set based on vehicle type, driving environment, safety regulations, or empirical data. Comparison refers to numerically comparing the calculated laser warning coefficient with the preset laser warning threshold. Determining whether the road safety monitoring results corresponding to laser monitoring meet or not requirements is based on the comparison results. If the laser warning coefficient is less than the laser warning threshold, it indicates that the relative approach speed between the vehicle and the obstacle is too fast or the distance is too close, posing a safety risk; therefore, it is determined that the requirements are not met. Conversely, if the coefficient is greater than or equal to the threshold, it is determined that the requirements are met.

[0049] This application's solution actively emits a laser beam using a vehicle-mounted laser transmitter and receives signals reflected back from road obstacles. Based on the laser beam's time of flight, the system can accurately calculate the real-time distance between the vehicle and the obstacle. Simultaneously, combining this with the vehicle's current speed, the ratio of this distance to the vehicle's speed is calculated as a laser warning coefficient. This laser warning coefficient directly reflects the potential time window for a collision between the vehicle and the obstacle at the current speed; the smaller the value, the greater the potential danger. Subsequently, the system retrieves a preset laser warning threshold from the database and compares the calculated laser warning coefficient with this threshold. If the laser warning coefficient is less than the laser warning threshold, it indicates that the safe distance or safe time between the vehicle and the obstacle is insufficient, posing a collision risk, thus determining that the road safety monitoring result corresponding to laser monitoring does not meet the requirements; otherwise, it is determined that the requirements are met. Through this series of clear steps and quantitative indicators, this solution provides a specific and quantifiable safety assessment mechanism for laser monitoring in road monitoring modules, enabling laser monitoring to output clear and reliable road safety monitoring results. This aligns with the overall concept of using laser monitoring for road obstacle safety monitoring in the road monitoring module, providing a solid data foundation for subsequent comparative analysis by the multi-level early warning module, thereby improving the accuracy and reliability of the entire intelligent driving safety early warning system's judgment on road safety.

[0050] The specific process of millimeter-wave monitoring includes: transmitting radio waves, analyzing the frequency changes of the reflected waves to detect the radial relative velocity between the vehicle and the obstacle and marking it as a millimeter-wave warning coefficient, obtaining the millimeter-wave warning threshold from the database, and comparing the millimeter-wave warning coefficient with the millimeter-wave warning threshold: if the millimeter-wave warning coefficient is less than the millimeter-wave warning threshold, the road safety monitoring result corresponding to the millimeter-wave monitoring is determined to meet the requirements; otherwise, the road safety monitoring result corresponding to the millimeter-wave monitoring is determined to not meet the requirements.

[0051] Millimeter-wave monitoring refers to the process of detecting and assessing obstacles on a vehicle's path using millimeter-wave radar technology. This process typically involves the transmission and reception of millimeter-wave signals, signal processing, and safety assessments based on the processing results. Its purpose is to provide an effective obstacle detection method that can operate even in adverse weather conditions (such as rain, snow, and fog), overcoming the limitations of optical and laser monitoring. The fundamental principle of millimeter-wave radar is the transmission of radio waves. Millimeter-wave radar transmits radio waves within a specific frequency range to the surrounding environment via its built-in transmitter. These radio waves typically operate in the millimeter-wave band, such as 24 GHz, 77 GHz, or 79 GHz, which offer good penetration and high resolution. When the radio waves emitted by the millimeter-wave radar encounter an obstacle, some of the waves are reflected back. If the obstacle is moving relative to the radar, the frequency of the reflected wave changes; this phenomenon is called the Doppler effect. By analyzing the frequency difference (Doppler shift) between the reflected and transmitted waves, the radial relative velocity between the vehicle and the obstacle can be accurately calculated. For example, digital signal processing techniques such as Fast Fourier Transform (FFT) can be used to analyze frequency changes.

[0052] Radial relative velocity is a key parameter for measuring the potential collision risk between a vehicle and an obstacle. Labeling it as a millimeter-wave warning coefficient is for standardization or as a direct input for subsequent safety assessments. This coefficient can be a raw velocity value or a transformed or weighted value to unify assessment standards. The millimeter-wave warning threshold is a reference value used to determine whether the millimeter-wave monitoring results meet safety requirements. This threshold can be pre-stored in a database and dynamically adjusted or selected based on factors such as vehicle type, speed, and road conditions. The database can be a local storage unit or a cloud server. Comparing the millimeter-wave warning coefficient with the millimeter-wave warning threshold is a core step in safety judgment. By comparing the real-time calculated millimeter-wave warning coefficient with the preset millimeter-wave warning threshold, it can be determined whether the current relative motion between the vehicle and the obstacle is within a safe range.

[0053] Based on the comparison results, the system will make a clear judgment. If the millimeter-wave warning coefficient is less than the millimeter-wave warning threshold, it usually means that the relative speed between the vehicle and the obstacle is at a safe level, or that the obstacle is far enough away and the risk is low, thus the system is judged to meet the requirements. Conversely, if the millimeter-wave warning coefficient is greater than or equal to the millimeter-wave warning threshold, it indicates that there is a potential danger, and the system is judged to not meet the requirements.

[0054] The multi-level early warning module is used to perform multi-level early warning analysis on the road safety of vehicles: it compares and analyzes the road safety monitoring results of optical monitoring, laser monitoring, and millimeter-wave monitoring; if they are consistent and all meet the requirements, the road safety is deemed qualified; if they are consistent but neither meets the requirements, the road safety is deemed unqualified, a road early warning signal is generated and sent to the controller, which issues a safety warning upon receiving the signal; otherwise, the two consistent monitoring results are marked as the immediate output result, and confidence analysis is performed: the values ​​of key parameters and corresponding failure thresholds of optical monitoring, laser monitoring, and millimeter-wave monitoring are retrieved. The key parameters for optical monitoring, laser monitoring, and millimeter-wave monitoring are light intensity, rainfall, and road curvature, respectively. Rainfall and road curvature can be obtained through phased rainfall data and analysis of road sections of a preset length; the failure threshold is the threshold value of the key parameter for the corresponding monitoring method in a failure scenario; the ratio of the key parameter value to the corresponding failure threshold is marked as the confidence value of the monitoring method, and the road safety monitoring result corresponding to the monitoring method with the smallest confidence value is marked as the updated output result. The control update of the road safety early warning is performed through the updated output result.

[0055] Specifically, the "key parameters" refer to environmental or road conditions that affect the performance and reliability of optical monitoring, laser monitoring, and millimeter-wave monitoring systems. For example, light intensity is a crucial factor affecting the image quality and target recognition capabilities of optical monitoring (such as vehicle-mounted cameras); rainfall significantly impacts visibility in optical monitoring, signal attenuation and scattering in laser monitoring, and signal attenuation in millimeter-wave monitoring; road curvature can affect the effective detection range and angle of laser and millimeter-wave radar, especially at sharp bends. The "numerical values" refer to the quantitative data of these key parameters acquired in real-time or near real-time, which can be obtained through vehicle-mounted environmental sensors, meteorological data interfaces, or image analysis. The "failure threshold" refers to the threshold at which the performance of the corresponding monitoring method will significantly decrease or even fail under specific key parameter conditions. These thresholds are usually pre-set based on extensive experimental data, simulations, or expert experience. For example, different lighting, rainfall, and curvature conditions are simulated in a laboratory or closed environment to test sensor performance and thus determine its failure boundary. The "rainfall" can be obtained through vehicle-mounted rain sensors, weather station data interfaces, or visual image-based raindrop detection algorithms, while the "road curvature" can be obtained through the vehicle's inertial measurement unit (IMU), GPS data combined with high-precision maps, or visual road line detection algorithms. The "confidence value" is a quantitative indicator reflecting the reliability of the monitoring method under the current environment. It is calculated as the ratio of the key parameter value to the corresponding failure threshold. The smaller the ratio, the farther the current environment is from the failure threshold, and the more reliable the monitoring method; the larger the ratio, the closer it is to or beyond the failure threshold, and the lower the reliability.

[0056] The following example will provide a more detailed explanation of the above technical solution:

[0057] Suppose user A is driving a vehicle equipped with this intelligent driving safety warning system on a complex urban road. During the journey, the safety warning platform continuously receives data from various monitoring modules.

[0058] First, the driver monitoring module acquires user A's facial information in real time via the cockpit camera. This module continuously extracts user A's eye and facial data from the most recent L1 second. For example, the module might make a preliminary assessment of user A's fatigue level by analyzing the degree of eye closure and the frequency of yawning. If the module detects that user A's eye closure time is too long or the yawning frequency is abnormal, it will preliminarily determine that the driver may be at risk of fatigue.

[0059] Meanwhile, the driving monitoring module continuously monitors the vehicle's driving status. This module acquires the distance between the vehicle and the vehicle in front, as well as the vehicle's speed in real time, and calculates emergency data. For example, if the vehicle in front suddenly decelerates, causing the distance between the two vehicles to decrease rapidly, the module will identify a potential emergency. Furthermore, the module also acquires the duration of continuous lane departures within the most recent L1 seconds and marks it as lane departure data. For example, if the vehicle deviates from its lane multiple times or for an extended period within a short time, the module will identify a risky driving situation.

[0060] Regarding the road environment, the road monitoring module employs three parallel methods—optical monitoring, laser monitoring, and millimeter-wave monitoring—to monitor road obstacles for safety. For example, optical monitoring uses onboard cameras to identify obstacles such as pedestrians and vehicles on the road and assess their potential risks; laser monitoring uses lidar to accurately measure the distance and relative velocity of obstacles; and millimeter-wave monitoring uses millimeter-wave radar to detect the radial relative velocity of obstacles. Each monitoring method independently generates its corresponding road safety monitoring results.

[0061] Subsequently, the multi-level early warning module received these three types of road safety monitoring results and conducted comparative analysis.

[0062] Scenario 1: If the results of optical monitoring, laser monitoring and millimeter wave monitoring all consistently determine that the road safety is qualified, for example, if all sensors confirm that there are no obstacles on the road ahead or that the distance to obstacles is safe, then the multi-level warning module determines that the road safety is qualified and the system continues to operate normally.

[0063] Scenario 2: If all three monitoring results consistently indicate that road safety does not meet requirements—for example, if all sensors detect an emergency obstacle ahead—the multi-level warning module determines that road safety does not meet requirements and generates a road warning signal. This road warning signal is sent to the controller, which, upon receiving the signal, immediately triggers an audible and visual alarm and may assist the vehicle in slowing down or braking to avoid a collision.

[0064] Scenario 3: If the three monitoring results are not entirely consistent—for example, optical and laser monitoring both indicate an obstacle ahead at a dangerous distance, while millimeter-wave monitoring indicates safety—the multi-level early warning module will mark the consistent results from optical and laser monitoring as the immediate output, and simultaneously perform confidence analysis on the optical, laser, and millimeter-wave monitoring. The confidence analysis retrieves key parameters and failure thresholds for each of the three monitoring methods and calculates confidence values. If the confidence analysis indicates that the millimeter-wave monitoring has a low confidence value under the current environment (e.g., heavy rain causing a decrease in millimeter-wave radar performance), the system will prioritize the results from optical and laser monitoring for early warning and update the early warning control strategy.

[0065] Through the aforementioned collaborative work, the system can comprehensively and in real time monitor the driver's status, vehicle driving status, and road environment, and perform intelligent comparison and confidence analysis between different monitoring results. This provides accurate and timely multi-level safety warnings in complex and ever-changing driving scenarios, effectively solving the problems of traditional single-algorithm warning systems being prone to failure, having low accuracy in risk analysis, being untimely in warnings, and having redundant and disordered outputs when the results of parallel monitoring by multiple algorithms are inconsistent.

[0066] Example 2: Figure 2 As shown, the intelligent driving safety warning method based on multi-algorithm collaboration includes the following steps:

[0067] Step 1: Monitor and analyze the driver's driving status: The driver's facial information is acquired in real time through the camera in the cockpit, and the driver's fatigue status is assessed based on the facial information.

[0068] Step 2: Monitor and analyze vehicle driving status: Acquire emergency data and deviation data of the vehicle in real time, and evaluate the vehicle driving status based on the emergency data and deviation data;

[0069] Step 3: Monitor and analyze the road safety of the vehicle: Use optical monitoring, laser monitoring and millimeter wave monitoring to monitor road obstacles.

[0070] Step 4: Conduct multi-level early warning analysis on road safety for vehicles: Compare and analyze the road safety monitoring results from optical monitoring, laser monitoring, and millimeter wave monitoring.

[0071] The intelligent driving safety warning system, which uses multiple algorithms in collaboration, acquires the driver's facial information in real time through the camera in the cockpit and assesses the driver's fatigue status based on the facial information.

[0072] Step 2: Monitor and analyze vehicle driving status: Acquire emergency data and deviation data of the vehicle in real time, and evaluate the vehicle driving status based on the emergency data and deviation data;

[0073] Step 3: Monitor and analyze the road safety of the vehicle: Use optical monitoring, laser monitoring and millimeter wave monitoring to monitor road obstacles.

[0074] Step 4: Conduct multi-level early warning analysis on road safety for vehicles: Compare and analyze the road safety monitoring results from optical monitoring, laser monitoring, and millimeter wave monitoring.

[0075] The above description is merely an example and illustration of the structure of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the structure of the invention or exceed the scope defined in the claims, all of which should fall within the protection scope of the present invention.

[0076] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0077] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multi-algorithm collaborative intelligent driving safety warning system, characterized in that, It includes a safety early warning platform, which is communicatively connected to a driving monitoring module, a vehicle monitoring module, a road monitoring module, a multi-level early warning module, and a database; The driving monitoring module is used to monitor and analyze the driver's driving status: it acquires the driver's facial information in real time through the cockpit camera, extracts the driver's eye shape data and facial data within the most recent L1 seconds, and assesses the driver's fatigue status through the eye shape data and facial data. The driving monitoring module is used to monitor and analyze the vehicle's driving status: acquiring emergency data and deviation data during vehicle driving, and evaluating the vehicle's driving status based on the driving data and deviation data; The road monitoring module is used to monitor and analyze the road safety of vehicles: it uses optical monitoring, laser monitoring and millimeter wave monitoring to monitor road obstacles. The multi-level early warning module is used to perform multi-level early warning analysis on the road safety of vehicles: it compares and analyzes the road safety monitoring results of optical monitoring, laser monitoring and millimeter wave monitoring: if they are consistent and all meet the requirements, the road safety is deemed qualified; if they are consistent and all do not meet the requirements, the road safety is deemed unqualified, a road early warning signal is generated and sent to the controller, and the controller issues a safety warning after receiving the road early warning signal. Otherwise, two consistent monitoring results will be marked as instant output results, and confidence analysis will be performed simultaneously. The specific process of confidence analysis includes: retrieving the values ​​of key parameters and corresponding failure thresholds for optical monitoring, laser monitoring, and millimeter-wave monitoring. The key parameters for optical monitoring, laser monitoring, and millimeter-wave monitoring are light intensity, rainfall, and road curvature, respectively. The ratio of the value of the key parameter to the corresponding failure threshold is marked as the confidence value of the monitoring method. The road safety monitoring result corresponding to the monitoring method with the smallest confidence value is marked as the updated output result. The control update of the road safety warning is performed by updating the output result.

2. The intelligent driving safety early warning system with multi-algorithm collaboration according to claim 1, characterized in that, The eye data is the proportion of time the eyes are closed in the most recent L1 second, calculated using the PERCLOS algorithm. The facial data is the sum of the driver's yawning frequency and nodding frequency in the most recent L1 second.

3. The intelligent driving safety early warning system with multi-algorithm collaboration according to claim 2, characterized in that, The specific process for assessing driver fatigue includes: obtaining eye and facial thresholds from a database, comparing the eye and facial data with the eye and facial thresholds respectively; if the eye data is less than the eye threshold and the facial data is less than the facial threshold, the driver's fatigue is determined to meet the requirements; otherwise, the driver's fatigue is determined to not meet the requirements, a fatigue warning signal is generated and sent to the controller, and the controller issues a safety warning upon receiving the fatigue warning signal.

4. The intelligent driving safety early warning system with multi-algorithm collaboration according to claim 3, characterized in that, The process of acquiring driving data and deviation data includes: acquiring the distance to the vehicle in front and the driving speed of the current vehicle in real time, marking the ratio of the distance to the vehicle in front to the driving speed of the current vehicle as emergency data, and acquiring the maximum duration of continuous deviation of the vehicle from the lane line within the most recent L1 seconds and marking it as deviation data.

5. The intelligent driving safety early warning system with multi-algorithm collaboration according to claim 4, characterized in that, The specific process of evaluating the vehicle's driving status includes: obtaining the emergency threshold and deviation threshold from the database, comparing the emergency data and deviation data with the emergency threshold and deviation threshold respectively; if the emergency data is less than the emergency threshold and the deviation data is less than the deviation threshold, the vehicle's driving status is determined to meet the requirements; otherwise, the vehicle's driving status is determined to not meet the requirements, a driving warning signal is generated and sent to the controller, and the controller issues a safety warning after receiving the driving warning signal.

6. The intelligent driving safety early warning system with multi-algorithm collaboration according to claim 5, characterized in that, The specific process of optical monitoring includes: recording the road where the vehicle is traveling using an onboard camera; decomposing the acquired video recording into continuous video frame images; performing noise reduction, HDR, and color enhancement on the video frame images; then using a deep learning model to extract pixel regions in the video frame images that match obstacle characteristics, including pedestrians, vehicles, cyclists, and traffic signs; the deep learning models include YOLO and CNN; associating the same target in consecutive frames using a multi-target tracking algorithm to form an obstacle trajectory; the multi-target tracking algorithms include SORT and DeepSORT; obtaining the radial relative velocity between the vehicle and the obstacle using the obstacle trajectory and marking it as an optical warning coefficient; and using the optical warning coefficient to determine the road safety monitoring results corresponding to the optical monitoring.

7. The intelligent driving safety early warning system with multi-algorithm collaboration according to claim 6, characterized in that, The specific process for determining the road safety monitoring results corresponding to optical monitoring includes: obtaining the optical warning threshold from the database, and comparing the optical warning coefficients of all obstacles with the optical warning threshold in sequence; if all optical warning coefficients are less than the optical warning threshold, the road safety monitoring results corresponding to optical monitoring are determined to meet the requirements; otherwise, the road safety monitoring results corresponding to optical monitoring are determined to not meet the requirements.

8. The intelligent driving safety early warning system with multi-algorithm collaboration according to claim 7, characterized in that, The specific process of laser monitoring includes: emitting a laser beam through a vehicle-mounted laser emitter and receiving its reflected signal; calculating the distance between the vehicle and the obstacle using the time of flight; marking the ratio of distance to vehicle speed as the laser warning coefficient; obtaining the laser warning threshold from a database; and comparing the laser warning coefficient with the laser warning threshold: if the laser warning coefficient is less than the laser warning threshold, the road safety monitoring result corresponding to the laser monitoring is deemed to meet the requirements; otherwise, the road safety monitoring result corresponding to the laser monitoring is deemed not to meet the requirements.

9. The intelligent driving safety early warning system with multi-algorithm collaboration according to claim 8, characterized in that, The specific process of millimeter-wave monitoring includes: transmitting radio waves, analyzing the frequency changes of the reflected waves to detect the radial relative velocity between the vehicle and the obstacle and marking it as a millimeter-wave warning coefficient, obtaining the millimeter-wave warning threshold from the database, and comparing the millimeter-wave warning coefficient with the millimeter-wave warning threshold: if the millimeter-wave warning coefficient is less than the millimeter-wave warning threshold, the road safety monitoring result corresponding to the millimeter-wave monitoring is determined to meet the requirements; otherwise, the road safety monitoring result corresponding to the millimeter-wave monitoring is determined to not meet the requirements.

Images