Vehicle blind area intelligent monitoring early warning and active risk prevention and control system and method

Abstract

The invention discloses a vehicle blind area intelligent monitoring early warning and active risk prevention and control system, which comprises a sensing module, a central AI processing unit, an early warning module and an execution control module, and is characterized in that the sensing module is used for collecting vehicle blind area information in a full-dimension manner; the central AI processing unit is used for processing the information acquired by the sensing module, performing risk assessment according to the processed blind area information and the working state of the vehicle, sending an early warning signal to the early warning module, and sending a control signal to the execution control module; the early warning module is used for sending a risk type and grade signal which can be perceived by a driver according to an early warning signal output by the central AI processing unit; and the execution control module is used for outputting control and driving signals according to the control signal sent by the central AI processing unit, so that a corresponding vehicle power device acts, and active risk prevention is realized. The system effectively prevents and controls blind area collision accidents, adapts to various vehicle types such as cars, SUVs, light trucks and heavy trucks, and does not need to redesign a hardware architecture.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of vehicle active safety technology, in particular to a vehicle blind area intelligent monitoring and early warning and active risk prevention system and method. BACKGROUND

[0002] At present, with the increase of road traffic flow, the proportion of traffic accidents caused by vehicle blind area is increasing year by year. According to the data of the Ministry of Transport, the right turning blind area accident of truck accounts for more than 35% of the total truck traffic accidents, and the pedestrian collision accident caused by the A-pillar blind area and the blind area under the rear bumper of car accounts for 28% of car accidents. The existing vehicle blind area solution has obvious defects:

[0003] 1. Limited coverage: traditional blind area monitoring system (BSD) relies on single side rearview mirror radar, which only covers the narrow area behind the vehicle, and cannot cover the key blind areas such as the right front wheel under the truck, the inner side of the A-pillar, and the rear bumper of the car;

[0004] 2. Low recognition accuracy: single camera solution is easily affected by backlight, rain and fog, and cannot accurately distinguish pedestrians, non-motor vehicles and fixed obstacles (such as kerbs and manholes), with high false alarm / omission rate;

[0005] 3. Early warning and control are disconnected: most systems only alarm through instrument panel icons or simple voice, without combining vehicle dynamic parameters (such as speed and steering state) to evaluate risk level, and lack of active brake intervention in high-risk scenarios, resulting in delayed driver response;

[0006] 4. Poor adaptability to different vehicle types: the blind area morphology of truck and car is quite different (the blind area of truck is 3-5 times that of car), and most existing systems are designed for single vehicle type, with weak universality.

[0007] Therefore, an integrated system that can realize blind area full coverage, high-precision identification, graded warning and active control is needed to solve the blind area safety pain points of different vehicle types. SUMMARY

[0008] The present application provides a vehicle blind area intelligent monitoring and early warning and active risk prevention system and method to solve the technical problems in the prior art.

[0009] The technical solution adopted by the present application to solve the technical problems in the prior art is:

[0010] A vehicle blind area intelligent monitoring and early warning and active risk prevention system, which comprises a perception module, a central AI processing unit, a warning module and an execution control module.

[0011] The perception module is used for full-dimensional acquisition of vehicle blind area information; the perception module includes a camera, a radar, and an environmental sensor; the camera is used for acquiring images of the vehicle blind area; the radar is used for perceiving the distance between the target in the vehicle blind area and the vehicle; and the environmental sensor is used for acquiring parameters related to driving safety in the vehicle driving environment;

[0012] The central AI processing unit is used for processing the information collected by the perception module, performing risk assessment according to the processed blind area information and the working state of the vehicle, and sending a warning signal to the warning module and a control signal to the execution control module; a deep learning model for processing multi-modal data is arranged therein, which performs data fusion and target prediction processing on multi-modal data, and outputs risk type and level signals; and further outputs corresponding warning signals and control signals corresponding to the risk type and level signals;

[0013] The warning module is used for sending risk type and level signals perceived by the driver according to the warning signals output by the central AI processing unit; the warning module includes a visual warning unit, a voice warning unit, and a vibration warning module, which are used for making the driver perceive the risk type and level signals through vision, hearing, and touch;

[0014] The execution control module is used for outputting control and driving signals to make the corresponding vehicle power device act to actively prevent risks according to the control signals sent by the central AI processing unit.

[0015] Further, the central AI processing unit includes a CPU, a GPU, and a FPGA; the CPU is used for system control and risk assessment algorithm running, and outputs warning signals and control signals; the GPU is used for multi-modal data fusion and target detection; and the FPGA is used for filtering, encoding, and data synchronization preprocessing of the signals collected by the perception module; the data processed by the FPGA is sent to the CPU and the GPU; the data processed by the GPU is sent to the CPU, and the time and space synchronization deviation of the data processed by the FPGA and the data of the CPU and the GPU is ≤50μs.

[0016] Further, the deep learning model includes a target detection model, which is constructed based on a YOLOv8 model, an attention mechanism is added to the backbone layer of YOLOv8 to enhance the extraction of target key features; and a structure combined with a feature pyramid network and a path aggregation network is adopted in the neck of the target detection model to improve the multi-scale target detection performance through bidirectional feature fusion.

[0017] Further, the CPU is connected with the driver monitoring system and / or engine control unit on the vehicle body; the CPU reads the driver state information from the driver monitoring system in real time; the vehicle dynamic parameters are read from the engine control unit in real time, and the control signal for gradually active prevention and control of danger is output based on the risk level, road conditions and vehicle state.

[0018] Further, the execution control module is linked with the ESP system, braking system and power system of the vehicle;

[0019] The execution control module adopts a dual-MCU redundant architecture, and the dual MCUs synchronously receive the control signal of the central AI processing unit, and the control signal received by the two MCUs is consistent before the strategy is executed;

[0020] The execution control module is provided with a braking control interface, a power control interface and a state feedback interface:

[0021] The braking control interface accesses the vehicle ESP system through the CAN FD bus and outputs a target deceleration instruction, so that the brake force distribution is realized by the ESP control ABS actuator;

[0022] The power control interface accesses the engine ECU and the gearbox TCU, and outputs the engine stop oil supply instruction to the ECU and the instruction for automatically reducing the gearbox to a low gear to the TCU when the risk is high;

[0023] The state feedback interface inputs the fuel pressure, wheel speed and deceleration data of the vehicle braking system in real time as the feedback data of the MCU.

[0024] Further, the environmental sensors include a light intensity sensor, a rainfall sensor and a road friction coefficient sensor, the light intensity sensor is used to collect the environmental light intensity to link the camera to adjust the exposure parameter; the rainfall sensor is used to detect the rainfall level through the optical principle to link the camera lens heating sheet and the rain and fog removal algorithm; and the road friction coefficient sensor collects the road friction coefficient in real time.

[0025] Further, the visual warning unit includes an instrument panel LCD screen, a blind area indicator light and a HUD head-up display, the instrument panel LCD screen displays the risk level dynamic icons of the left and right blind area positions on the left and right sides of the screen, the icon color is yellow, orange and red in turn as the risk level increases, and the size increases as the traffic conflict time decreases; the blind area indicator light is installed on the inner side of the left and right rearview mirrors, the color of the indicator light flashes is yellow, orange and red in turn as the risk level increases, and the frequency of the indicator light flashes increases as the risk level increases; the HUD head-up display is projected when the risk level is high, and the HUD head-up display is provided with a red emergency brake prompt box, superimposed with the target distance vehicle body information, and the display brightness is automatically adjusted according to the ambient light.

[0026] Further, the voice early warning unit comprises a vehicle-mounted audio device; the vehicle-mounted audio device voice broadcasts the risk and processing prompt of the left and right blind areas; the vibration early warning module comprises a steering wheel vibration module and a seat vibration module, the steering wheel vibration module comprises left and right steering wheel vibration motors corresponding to the risks of the left and right blind areas, and the vibration frequency and intensity of the left and right steering wheel vibration motors are increased with the increase of the risk level; the seat vibration module is installed on the left and right sides of the seat backrest, and the seat module corresponding to the blind area side vibrates when the risk is high, and the vibration mode is intermittent cyclic vibration between a short pulse and a long pulse.

[0027] The application further provides a vehicle blind area intelligent monitoring and early warning and active risk prevention and control method using the vehicle blind area intelligent monitoring and early warning and active risk prevention and control system.

[0028] The camera collects images of the vehicle blind area; the radar senses the distance between the target in the vehicle blind area and the vehicle; and the environmental sensor collects parameters related to driving safety in the vehicle driving environment.

[0029] The central AI processing unit corrects the positional deviation of the camera caused by the perspective relationship based on the laser radar point cloud positioning.

[0030] The same target is given a weight by the multi-sensor identification result, and the target confidence after fusion is greater than or equal to 0.8 to determine that it is an effective target, so as to avoid single sensor misjudgment.

[0031] The target image data collected by the camera, the distance between the target and the vehicle body collected by the radar, and the vehicle state data from the engine control unit are fused by a deep learning model: the target category is identified; the target in the blind area is identified in real time and classified as: pedestrian, non-motor vehicle, dynamic obstacle and fixed obstacle.

[0032] A multi-dimensional risk assessment model is constructed based on the vehicle coordinate system, vehicle dynamic parameters and target state, the basic collision time is combined with the target danger coefficient to calculate the comprehensive collision time, and the comprehensive collision time is taken as a risk level judgment index to determine the risk level; the comprehensive collision time is calculated according to the following formula:

[0033] ;

[0034] ;

[0035] ;

[0036] In the above formula:

[0037] TTC is the comprehensive collision time;

[0038] K is the risk coefficient given to different targets; wherein, the value of K for pedestrians > the value of K for non-motor vehicles > the value of K for dynamic obstacles > the value of K for fixed obstacles;

[0039] is the time of straight-on collision;

[0040] is the time of turning side collision;

[0041] v is the current vehicle speed;

[0042] θ is the vehicle steering angle;

[0043] μ is the road friction coefficient; wherein, the value of μ for wet and slippery roads < the value of μ for dry roads;

[0044] M is the vehicle weight coefficient; wherein, the value of M for larger vehicles > the value of M for smaller vehicles;

[0045] X is the longitudinal distance between the target and the vehicle;

[0046] Y is the lateral distance between the target and the vehicle, and Y takes a positive value when the steering direction is consistent with the target position, and vice versa;

[0047] Let μ be the road friction coefficient and M be the vehicle weight coefficient; after the calculation of the comprehensive collision time is completed, the TTC warning threshold is corrected in combination with the road friction coefficient μ and the vehicle weight coefficient M, and the correction formula is as follows:

[0048] The TTC warning threshold for wet and slippery roads = the TTC warning threshold for dry roads × (0.8 / μ);

[0049] The TTC warning threshold for a fully loaded truck = the TTC warning threshold for an empty truck × M;

[0050] Based on the comprehensive collision time, three risk levels of low, medium and high are set, the low risk level condition includes: TTC > low risk TTC warning threshold, the medium risk level condition includes: high risk TTC warning threshold < TTC ≤ low risk TTC warning threshold, and the high risk level condition includes: TTC ≤ high risk TTC warning threshold;

[0051] The following treatments are performed corresponding to different risk levels:

[0052] When the risk level is low risk or medium risk, the central AI processing unit outputs corresponding warning signals to the warning module according to the risk type and level signal, without interfering with the vehicle driving;

[0053] When the risk level is high risk, the central AI processing unit outputs a corresponding pre-warning signal in addition to the risk type and level signal, and outputs a control signal to make the execution control module perform a hierarchical active intervention, adopts a phased and gradual intervention strategy according to the vehicle speed, road conditions and load difference, realizes priority speed reduction and then braking, and ensures the stability of the vehicle;

[0054] The central AI processing unit outputs signals to the execution control module according to different vehicle driving scenes, so that the vehicle realizes the following actions:

[0055] When the road surface is wet / icy, based on the road surface friction coefficient μ, the maximum deceleration of the vehicle is limited to μ×0.8, and the brake prediction distance is extended, and the intervention is triggered 0.5s in advance;

[0056] When the truck is in a heavy load scene, according to the load data, the corresponding TTC warning threshold is increased by 20%, and the brake force gradual time is extended to 1.5s;

[0057] In the reversing scene, when the vehicle is in R gear, the maximum deceleration of the vehicle is adjusted to 0.3g, and the engine is turned off, and only the brake control is used to prevent rear collision caused by sudden braking when reversing.

[0058] Further, according to the vehicle speed, road conditions and load difference, the following phased and gradual intervention strategy is adopted:

[0059] First stage: power speed reduction, trigger condition: TTC≤1.5s, no brake;

[0060] The central AI processing unit outputs an engine fuel cut command, and the execution control module outputs a signal to cut off the fuel supply, and outputs a gearbox downshift command, and the execution control module outputs a signal to control the deceleration to 0.2-0.3g, and turns on the double flash light to remind the rear vehicle to avoid;

[0061] Second stage: start partial braking, trigger condition: TTC≤1.2s, and the power speed reduction effect is insufficient;

[0062] The central AI processing unit outputs a target deceleration 0.4-0.6g command to make the ESP distribute brake force according to the wheel speed, so that the front wheel brake force accounts for 60%, and the rear wheel brake force accounts for 40%, and the ABS anti-lock function is activated; if it is a heavy load truck scene, the brake force is reduced by 20% to prevent rear wheel slip caused by heavy load;

[0063] Third stage: emergency braking, trigger condition: TTC≤1.0s, partial braking still cannot avoid collision;

[0064] Central AI processing unit, output maximum deceleration 0.8g instruction in dry road conditions, output maximum deceleration 0.5g instruction in wet road conditions, make ESP linkage EBD adjust front and rear wheel brake force ratio; At the same time, output seat belt pretensioning instruction, make the seat belt tighten 2-3cm, reduce the driver / passenger body forward distance during collision;

[0065] Brake rear control: after emergency brake stop, keep double flash light on for 30s, until the driver manually release, at the same time, lock the throttle, avoid mispressing the throttle to cause the vehicle to start suddenly, only allow to put into P or N.

[0066] Further, the central AI processing unit and the perception module, the early warning module and the execution control module realize data interaction through Ethernet and CAN FD bus double link;

[0067] Ethernet transmission adopts SOME / IP protocol and TLS 1.3 encryption, key length 256 bits, to prevent perception data from being tampered with;

[0068] CAN FD bus adopts CANoe encryption, uses AES-256 algorithm, and execution control instruction is attached with CRC check, check bit 16 bits, to avoid instruction transmission error;

[0069] OTA upgrade security: firmware upgrade adopts double partition storage, main partition plus backup partition, automatically fallback to backup partition when upgrade fails, to avoid system paralysis;

[0070] Upgrade package adopts RSA-2048 signature and AES-128 encryption, only accepts officially authorized upgrade package, to prevent malicious firmware attack.

[0071] Further, the perception module is self-diagnosed according to the following method steps, and the corresponding diagnosis results are processed as follows:

[0072] The central AI processing unit detects the health status of each sensor in the perception module every 100ms, including whether the camera is blocked, whether the collected image is blurred, whether the radar signal strength is normal, whether there is target output, and whether the laser radar point cloud quantity meets the standard;

[0073] If it is determined that a sensor fails, the name of the faulty sensor is displayed through the instrument panel, and the system automatically switches to the degraded mode at the same time;

[0074] Degraded working mode includes:

[0075] When a single sensor in each type of sensor fails, the system enters a first degraded mode of operation, retaining other available sensors, maintaining 90% early warning function;

[0076] When half of each type of sensor fails, the system enters a second degraded mode of operation, retaining only the monitoring of the core blind area, and shutting down the non-core blind area warning;

[0077] When all sensors fail, the system enters a third degraded mode of operation, causing the instrument panel to display a red fault light and a language prompt, shutting down the active control risk function, and retaining only the blind area warning icon.

[0078] The present application has the advantages and positive effects of:

[0079] 1. Blind area full coverage: through the combination of multiple cameras + millimeter wave radar, the system is adapted to different blind area forms of sedans and trucks, realizes no dead angle monitoring, and solves the problem of incomplete coverage of traditional BSD systems;

[0080] 2. High-precision identification: combined with deep learning and environment adaptive algorithm, the target identification accuracy in complex environment is ≥98%, and the false alarm rate is <1%, far exceeding existing systems (false alarm rate is generally 5%-10%);

[0081] 3. Scientific graded warning: risk grading based on TTC, avoiding one-size-fits-all warning to interfere with the driver, balancing safety and driving experience;

[0082] 4. Active control in time: active speed reduction / braking in high-risk scenarios, making up for the driver's 0.5-1s reaction delay, and reducing blind area collision accidents by more than 85% according to simulation tests;

[0083] 5. Strong universality: by adjusting the number of cameras, installation position and AI model parameters, the system can be adapted to various vehicle models such as sedans, SUVs, light trucks and heavy trucks, without the need to redesign the hardware architecture. BRIEF DESCRIPTION OF DRAWINGS

[0084] Fig. 1 is a vehicle blind area intelligent monitoring and early warning and active risk prevention system structure diagram of the present application.

[0085] Fig. 2 is a distribution diagram of a sensing assembly composed of a camera and a radar deployed on a passenger car of the present application.

[0086] Fig. 3 is a distribution diagram of a sensing assembly composed of a camera and a radar deployed on a truck of the present application.

[0087] In the figure: 1, sensing assembly. DETAILED DESCRIPTION

[0088] The present application will be described in detail below with reference to the accompanying drawings and embodiments, and it should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not intended to limit the present application.

[0089] In the description of the present application, the terms up, down, front, back, left, right, vertical, horizontal, top, bottom, etc. indicate the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present application and are not required to be constructed and operated in a specific orientation, therefore cannot be understood as a limitation of the present application. The terms connected and connected used in the present application should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected; it can be directly connected, or it can be indirectly connected through intermediate components; it can be electrically connected or signal transmission; for those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0090] The Chinese interpretation of the following English words, phrases and abbreviations is as follows:

[0091] CPU: Central Processing Unit (CPU).

[0092] GPU: Graphics Processing Unit (GPU).

[0093] FPGA: Field Programmable Gate Array, a hardware reconfigurable integrated circuit, its core feature is "programmability" - unlike fixed hardware architecture chips such as CPU, GPU, FPGA contains a large number of configurable logic units (such as lookup table LUT, flip-flop FF), programmable interconnection resources and special function modules (such as multiplier, RAM), which can be defined by hardware description language (such as Verilog, VHDL) to define circuit logic, to realize customized data processing flow. In the vehicle blind area monitoring system described, FPGA plays a core role in "perception data preprocessing and real-time synchronization", and is the key hardware to ensure low latency and high reliability of the system.

[0094] AI: Artificial Intelligence (AI).

[0095] YOLOv8: You Only Look Once Version 8, a real-time target detection algorithm released in 2023.

[0096] The workflow of YOLOv8 in the document system cooperates with other modules as follows:

[0097] 1. Input: Blind area images collected by the perception module (pre-processed by FPGA, remove rain and fog, correct exposure).

[0098] 2. Inference: The YOLOv8 model completes forward propagation through the GPU, outputting the "position coordinates + category + confidence" of the target.

[0099] 3. Screening: The central AI processing unit screens the model output, retaining only valid targets with "confidence ≥ 0.8" (to avoid single sensor misjudgment).

[0100] 4. Subsequent processing: The information of valid targets is transmitted to the risk assessment module, combined with the distance data collected by the radar and the vehicle dynamic parameters (speed, steering angle), to calculate the comprehensive TTC (time to collision), finally triggering the hierarchical warning or active control.

[0101] Backbone: Backbone network, is the core component of deep learning model responsible for feature extraction, gradually extracts information from "low-level features" (such as edges, textures) to "high-level features" (such as target contours, key components) from raw input data (such as blind area images collected by the camera), providing high-quality feature support for subsequent target detection, classification and other tasks.

[0102] MCU: Microcontroller Unit (MCU), is a compact embedded chip that integrates Central Processing Unit (CPU), memory (RAM / ROM), peripheral interfaces (such as CAN, UART) and digital / analog modules, essentially a "miniature computer system". Its core features are "high integration, low power consumption, strong real-time performance", which can quickly execute specific control tasks under limited resources. In the vehicle blind area intelligent monitoring system described, MCU is the core hardware of the control module, directly responsible for converting the risk decisions of the central AI processing unit into specific actions of the vehicle power and braking systems, and is the key execution link of the "warning - control" closed loop.

[0103] LCD: Liquid Crystal Display (LCD).

[0104] HUD: Head-Up Display (HUD).

[0105] CAN FD: CAN with Flexible Data-Rate, CAN flexible data rate, is an upgraded vehicle-mounted high-speed communication protocol based on traditional CAN bus (Controller Area Network). The core breakthrough is "flexible data rate and larger data packet capacity" - compatible with low-rate control scenarios of traditional CAN bus, and can meet the demand of "high bandwidth and low delay" data transmission of vehicle-mounted system by improving transmission rate (up to 8 Mbps) and data field length (maximum 64 bytes). In the vehicle blind area intelligent monitoring system, CAN FD is the core bus of data interaction between key modules, undertakes the dual tasks of "control command transmission" and "state feedback", and is the communication basis for realizing the "early warning - control" closed loop.

[0106] ESP: Electronic Stability Program (ESP) is the core system of vehicle active safety. Its essence is "the multi-dimensional posture control center of integrated braking, power and steering systems" - by monitoring vehicle dynamic parameters (wheel speed, steering angle, deceleration, etc.) and driver's operation intention (steering, braking), it can determine whether the vehicle has the risk of "skidding and losing control", and then actively adjust the brake force of the braking system, the output of the power system or the steering auxiliary force to ensure the vehicle to travel stably according to the driver's intention. In the vehicle blind area intelligent monitoring system, ESP is not an independent functional module, but a key carrier for the execution control module to realize "active risk prevention", which, through deep linkage with ABS (anti-lock braking system), EBD (electronic brake force distribution), engine ECU (electronic control unit), etc., converts the "risk control instructions" of the central AI processing unit into precise vehicle braking / power actions, and is the core execution link of the "early warning - control" closed loop.

[0107] ABS: Anti-lock Braking System (ABS).

[0108] ECU: Electronic Control Unit is the core control center of each subsystem of the vehicle. Its essence is an embedded control unit that integrates microprocessors (MCU), memory (RAM / ROM), input / output interfaces (I / O), and communication modules. It can achieve precise control of specific vehicle systems (such as engines, bodies, and brakes) by collecting sensor data and executing pre-set control logic. In the vehicle blind area intelligent monitoring system, the ECU is not a single module, but rather the engine ECU, which serves as the key link object of the execution control module and undertakes the task of "power system intervention" - converting the "risk control instructions" of the central AI processing unit into engine fuel cutoff and power regulation actions. It is the core power control carrier for "active speed reduction / prevention and control" in high-risk scenarios.

[0109] TCU: Transmission Control Unit.

[0110] TTC: Time to Collision.

[0111] EBD: Electronic Brakeforce Distribution is an active safety auxiliary function of the vehicle braking system. Its essence is an intelligent control logic that dynamically adjusts the front and rear wheel brake force ratio - by monitoring vehicle wheel speed, load, road friction coefficient, and other parameters in real time, it automatically distributes the brake pressure of the front and rear wheels to avoid single wheel (especially the rear wheel) from locking due to excessive brake force, while ensuring the shortest braking distance and stable vehicle attitude. In the vehicle blind area intelligent monitoring system, EBD is not an independent hardware, but rather a core sub-function of the ESP (Electronic Stability Program) system, which works in conjunction with ABS (Anti-lock Braking System) to play a key role in "precise brake force distribution" in active braking intervention in high-risk scenarios. It is one of the core technologies for achieving "gradual and high-safety active prevention and control".

[0112] SOME / IP: Scalable service-Oriented Middleware over IP is a communication protocol designed specifically for vehicle Ethernet, with the core feature of "service-oriented communication" - encapsulating functions in the vehicle system (such as blind area target detection, risk assessment) as standardized "services", and modules interacting through "request-response", "publish-subscribe" and other modes, rather than traditional bus "point-to-point fixed transmission". In the described intelligent monitoring system for vehicle blind areas, SOME / IP is the core data transmission protocol between the central AI processing unit and the perception module (especially high-bandwidth devices), responsible for "high-reliability transmission of multi-modal perception data (such as laser radar point cloud, high-definition camera video)", and is the key communication support for "full-dimensional perception of blind areas and low-latency data processing".

[0113] TLS 1.3: Transport Layer Security 1.3.

[0114] CANoe: CAN open environment is a vehicle CAN / CAN FD bus full-life cycle development and testing tool kit developed by Vector Company, not a single hardware or software, but a comprehensive solution integrating "software function modules, hardware interface devices, protocol stacks". Its core value lies in providing "design-development-testing-diagnosis-maintenance" full-process support for vehicle bus systems, especially focusing on the safety, reliability verification and data encryption of bus communication. In the described intelligent monitoring system for vehicle blind areas, the core application scenario of CANoe is to ensure the secure transmission and communication reliability verification of CAN FD bus control commands, directly serving the system ISO26262 ASIL D functional safety level requirements, and is a technical support tool for "safe command interaction" between the execution control module and key nodes such as vehicle ESP, ECU and TCU.

[0115] AES-256: Advanced Encryption Standard 256-bit.

[0116] CRC: Cyclic Redundancy Check.

[0117] OTA: Over-the-Air Technology is a technology that enables remote upgrading and updating of device firmware, software, or parameters through wireless networks such as 4G / 5G, Wi-Fi, without the need for physical connection hardware (such as through a data cable) to complete system iteration. In the intelligent monitoring system for vehicle blind areas described, the core value of OTA is to ensure the continuous optimization and fault repair of system functions - it can remotely update the deep learning model of the central AI processing unit (such as the YOLOv8 target detection algorithm) and the intervention strategy of the execution control module (such as the TTC warning threshold for different vehicle models), while ensuring the safety and stability of the upgrade process, and is one of the key technologies to achieve "reliable operation of the system throughout its life cycle".

[0118] RSA-2048: RSA encryption algorithm 2048-bit (Rivest-Shamir-Adleman 2048-bit).

[0119] AES-128: Advanced Encryption Standard 128-bit (Advanced Encryption Standard 128-bit).

[0120] See Figs. 1 to 3 , an intelligent monitoring and early warning and active risk prevention system for vehicle blind areas, which includes a perception module, a central AI processing unit, a warning module, and an execution control module.

[0121] The perception module is used to collect information in all dimensions of the vehicle blind area; the perception module includes a camera, a radar, and an environmental sensor; the camera is used to collect images of the vehicle blind area; the radar is used to sense the distance between obstacles and vehicles in the vehicle blind area; the environmental sensor is used to collect parameters related to driving safety in the vehicle driving environment.

[0122] The central AI processing unit is used to process the information collected by the perception module, assess risks based on the processed blind area information and vehicle operating status, send warning signals to the warning module, and send control signals to the execution control module; it has a deep learning model for processing multi-modal data, which performs data fusion and obstacle prediction processing on multi-modal data, and outputs risk type and level signals; and further outputs corresponding warning signals and control signals corresponding to the risk type and level signals.

[0123] The early warning module is configured to output a risk type and level signal perceived by the driver according to the early warning signal output by the central AI processing unit; the early warning module includes a visual early warning unit, a voice early warning unit, and a vibration early warning module; the visual early warning unit is configured to enable the driver to visually perceive the risk type and level signal; the voice early warning unit is configured to enable the driver to audibly perceive the risk type and level signal; and the vibration early warning module is configured to enable the driver to tactilely perceive a higher level risk signal.

[0124] The execution control module is configured to output a control and driving signal according to the control signal output by the central AI processing unit, so as to enable the corresponding vehicle power device to act and actively prevent risks.

[0125] The number, parameters, and installation posture of the cameras are set according to the blind area and position characteristics of the vehicle type, and the overlap degree of the field of view coverage is 15-20%; and radars with different detection distances are combined and arranged.

[0126] The radar can be an electromagnetic wave radar or a laser radar; the electromagnetic wave radar can be a millimeter wave radar with a shorter detection distance or a millimeter wave radar with a longer distance.

[0127] The cameras and radars can be combined together to form a perception component 1, which is arranged at various positions of the passenger car and the truck, including the left and right rearview mirrors, the left and right upright columns, and the rear bumper. Fig. 2 、 Fig. 3 The cameras and radars can be combined together to form a perception component 1, which is arranged at various positions of the passenger car and the truck, including the left and right rearview mirrors, the left and right upright columns, and the rear bumper.

[0128] Preferably, the central AI processing unit can include a CPU, a GPU, an FPGA, and a memory; the CPU is configured to perform system control and risk assessment algorithm operation, and outputs the early warning signal and the control signal; the GPU is configured to perform multi-modal data fusion and target detection; the FPGA is configured to perform filtering, encoding, and data synchronization preprocessing on the signals collected by the perception module; the data processed by the FPGA is sent to the CPU and the GPU; the data processed by the GPU is sent to the CPU. The memory is configured to store the data before and after the processing of the CPU, the GPU, and the FPGA. The time and space synchronization deviation of the data processed by the FPGA and the data of the CPU and the GPU is less than or equal to 50 μs.

[0129] Preferably, the deep learning model can include a target detection model, which is constructed based on a YOLOv8 model; an attention mechanism is added to the backbone layer of the YOLOv8 model to enhance the extraction of key features of the target; and a structure combining a feature pyramid network and a path aggregation network is used in the neck of the target detection model to improve the multi-scale target detection performance through bidirectional feature fusion.

[0130] Preferably, the CPU can be connected with the driver monitoring system and / or engine control unit on the vehicle body; the CPU can read the driver state information in real time from the driver monitoring system; the vehicle dynamic parameters can be read in real time from the engine control unit, and the control signal for gradually active prevention and control of danger can be output based on the risk level, road conditions, and vehicle state.

[0131] Preferably, the execution control module can be linked with the ESP system, braking system, and power system of the vehicle.

[0132] The execution control module can adopt a dual-MCU redundant architecture, and the dual MCUs synchronously receive the control signal of the central AI processing unit, and the control signal received by the two MCUs is consistent before the strategy is executed.

[0133] The execution control module can be provided with a braking control interface, a power control interface, and a state feedback interface.

[0134] The braking control interface can access the vehicle ESP system through the CAN FD bus, output a target deceleration instruction, and realize brake force distribution by the ESP control ABS actuator.

[0135] The power control interface can access the engine ECU and the gearbox TCU, and output an engine stop fuel supply instruction to the ECU and an automatic gearbox drop to low gear instruction to the TCU when the risk is high.

[0136] The state feedback interface can input the fuel pressure, wheel speed, and deceleration data of the vehicle braking system in real time as feedback data of the MCU.

[0137] Preferably, the environmental sensors include a light intensity sensor, a rainfall sensor, and a road friction coefficient sensor, the light intensity sensor is used to collect the environmental light intensity to link the camera to adjust the exposure parameters; the rainfall sensor is used to detect the rainfall level through the optical principle to link the camera lens heating sheet and the rain and fog removal algorithm; the road friction coefficient sensor collects the road friction coefficient in real time.

[0138] Preferably, the visual warning unit can include an instrument panel LCD screen, a blind area indicator light, and a HUD head-up display, the instrument panel LCD screen can display the risk level dynamic icons of the left and right blind area positions on the left and right sides of the screen, the icon color can be yellow, orange, and red in turn as the risk level increases, and the size increases as the traffic conflict time decreases; the blind area indicator light can be installed on the inner side of the left and right rearview mirrors, the color of the indicator light when flashing can be yellow, orange, and red in turn as the risk level increases, and the frequency of the indicator light when flashing increases as the risk level increases; the HUD head-up display can be projected when the risk level is high, and it can be provided with a red emergency brake prompt box, superimposed with target distance vehicle body information, and the display brightness is automatically adjusted according to the ambient light.

[0139] Preferably, the voice warning unit can include a vehicle audio device; the vehicle audio device voice broadcasts the risk of left and right blind areas and processing prompts; the vibration warning module can include a steering wheel vibration module and a seat vibration module, the steering wheel vibration module includes left and right steering wheel vibration motors corresponding to the risk of left and right blind areas, and the vibration frequency and intensity of the left and right steering wheel vibration motors are increased with the increase of the risk level; the seat vibration module is installed on the left and right sides of the seat backrest, and when the risk is high, the seat module corresponding to the blind area side vibrates, and the vibration mode is intermittent cycle vibration between short pulses and long pulses.

[0140] The application also provides a vehicle blind area intelligent monitoring and early warning and active risk prevention method using the vehicle blind area intelligent monitoring and early warning and active risk prevention system.

[0141] The camera collects images of the vehicle blind area; the radar senses the distance between the target in the vehicle blind area and the vehicle; and the environmental sensor collects parameters related to driving safety in the vehicle driving environment.

[0142] The central AI processing unit corrects the positional deviation of the camera caused by the perspective relationship based on the laser radar point cloud positioning. It gives weight to the multi-sensor identification results of the same target, and determines that the target is effective when the target confidence after fusion is greater than or equal to 0.8, so as to avoid misjudgment of a single sensor. It performs data fusion on the target image data collected by the camera, the distance between the target and the vehicle body collected by the radar, and the vehicle state data from the engine control unit through a deep learning model: identifies the target category; identifies the target in the blind area in real time and classifies it as: pedestrian, non-motor vehicle, dynamic obstacle and fixed obstacle. It constructs a multi-dimensional risk assessment model based on the vehicle coordinate system, vehicle dynamic parameters and target state, combines the basic collision time with the target danger coefficient to calculate the comprehensive collision time, and uses the comprehensive collision time as the risk level judgment index to determine the risk level; the comprehensive collision time is calculated according to the following formula:

[0143]

[0144]

[0145]

[0146] In the above formula:

[0147] TTC is the comprehensive collision time; K is the danger coefficient given to different targets; wherein, the K value of the pedestrian > the K value of the non-motor vehicle > the K value of the dynamic obstacle > the K value of the fixed obstacle; is the straight-line head-on collision time; ​​​is the steering side collision time; v is the current vehicle speed; θ is the vehicle steering angle; μ is the road friction coefficient; wherein the μ value of the wet and slippery road < the μ value of the dry road; M is the vehicle weight coefficient; wherein the M value of the heavier vehicle > the M value of the lighter vehicle; X is the longitudinal distance between the target and the vehicle; Y is the lateral distance between the target and the vehicle, and Y takes a positive value when the steering direction is consistent with the target position, and vice versa.

[0148] Let μ be the road friction coefficient; M be the vehicle weight coefficient; after the comprehensive collision time calculation is completed, the TTC warning threshold is corrected in combination with the road friction coefficient μ and the vehicle weight coefficient M, and the correction formula is as follows:

[0149] The TTC warning threshold of the wet and slippery road = the TTC warning threshold of the dry road × (0.8 / μ);

[0150] The TTC warning threshold of the fully loaded truck = the TTC warning threshold of the empty truck × M;

[0151] Based on the comprehensive collision time, three risk levels of low, medium and high are set, the low risk level condition includes: TTC > low risk TTC warning threshold, the medium risk level condition includes: high risk TTC warning threshold < TTC ≤ low risk TTC warning threshold, and the high risk level condition includes: TTC ≤ high risk TTC warning threshold.

[0152] The following treatments are performed corresponding to different risk levels:

[0153] When the risk level is low risk or medium risk, the central AI processing unit outputs corresponding warning signals to the warning module according to the risk type and level signals, without interfering with the vehicle driving. When the risk level is high risk, the central AI processing unit outputs control signals to the execution control module to perform hierarchical active intervention, adopts a phased and gradual intervention strategy according to the vehicle speed, road conditions, and load difference, realizes priority deceleration and then braking, and ensures the vehicle stability.

[0154] The central AI processing unit outputs signals to the execution control module according to different vehicle driving scenes, so that the vehicle realizes the following actions:

[0155] When the road surface is wet and slippery / icy, based on the road friction coefficient μ, the maximum deceleration of the vehicle is limited to μ × 0.8, and the brake prediction distance is extended, triggering the intervention 0.5s in advance. When the truck is overloaded, according to the load data, the corresponding TTC warning threshold is increased by 20%, and the brake force gradual change time is extended to 1.5s.

[0156] In reverse driving scenario, when the vehicle is in R gear, the maximum deceleration is adjusted to 0.3g, and the engine is turned off to prevent rear collision caused by sudden braking.

[0157] Preferably, according to the vehicle speed, road conditions, and load differences, the following phased gradual intervention strategy can be adopted:

[0158] First stage: power reduction, trigger condition: TTC≤1.5s, no brake pedal depression.

[0159] Central AI processing unit outputs engine fuel cut command, and the execution control module outputs signal to cut off fuel supply. The transmission downshift command is output, and the execution control module outputs signal to control the deceleration to 0.2-0.3g, and the double flash light is turned on to remind the rear vehicle to avoid.

[0160] Second stage: start partial braking, trigger condition: TTC≤1.2s, and power reduction is not enough.

[0161] Central AI processing unit outputs target deceleration 0.4-0.6g command, ESP distributes brake force according to wheel speed, front wheel brake force ratio is 60%, rear wheel brake force ratio is 40%, and ABS anti-lock function is activated; if it is a heavy truck scene, the brake force is reduced by 20% to prevent rear wheel slip caused by heavy load.

[0162] Third stage: emergency braking, trigger condition: TTC≤1.0s, partial braking still cannot avoid collision.

[0163] Central AI processing unit outputs maximum deceleration 0.8g command in dry road conditions and maximum deceleration 0.5g command in wet road conditions, ESP links EBD to adjust front and rear wheel brake force ratio; at the same time, the seat belt pretensioning command is output, and the seat belt is tightened by 2-3cm to reduce the driver / passenger body forward distance during collision.

[0164] Brake control: after emergency braking stops, keep double flash light on for 30s until the driver manually releases, and lock the throttle to prevent sudden start caused by mispressing the throttle, only allow P or N gear.

[0165] Preferably, the central AI processing unit can realize data interaction with the perception module, warning module and execution control module through Ethernet and CAN FD bus double link.

[0166] The Ethernet transmission adopts the SOME / IP protocol and TLS 1.3 encryption, the key length is 256 bits, and the perception data is prevented from being tampered with. The CAN FD bus adopts CANoe encryption, adopts the AES-256 algorithm, and performs control instructions with CRC check, the check bit is 16 bits, to avoid transmission errors of instructions. OTA upgrade security: the firmware upgrade adopts double-partition storage, the main partition and the backup partition, automatically reverts to the backup partition when the upgrade fails, to avoid system paralysis. The upgrade package adopts RSA-2048 signature and AES-128 encryption, only accepts the upgrade package authorized by the official, to prevent malicious firmware attacks.

[0167] Preferably, the perception module can be self-diagnosed according to the following method steps, and the corresponding diagnosis results can be degraded as follows:

[0168] The central AI processing unit detects the health status of each sensor in the perception module every 100 ms, and the health status detection items include: whether the camera is blocked, whether the collected image is blurred; whether the radar signal strength is normal, whether there is no target output, whether the laser radar point cloud quantity meets the standard; if it is determined that a sensor fails, the name of the faulty sensor is displayed through the instrument panel, and the system automatically switches to the degraded mode.

[0169] The degraded working mode includes:

[0170] When a single sensor in each type of sensor fails, the system enters a first degraded working mode, retains other available sensors, and maintains 90% of the early warning function. When half of the sensors in each type of sensor fail, the system enters a second degraded working mode, only retains the monitoring of the core blind area, and closes the non-core blind area warning. When all sensors fail, the system enters a third degraded working mode, displays a red fault light and a language prompt on the instrument panel, closes the active risk prevention function, and only retains the blind area warning icon.

[0171] The structure, working process and working principle of the application will be further described below with the preferred embodiments of the application:

[0172] A vehicle blind area intelligent monitoring and early warning and active risk prevention system, the system includes a perception module, a central AI processing unit, an early warning module and an execution control module.

[0173] The perception module is used for full-dimensional acquisition of vehicle blind area information; the perception module includes a camera, a radar and an environmental sensor; the camera is used for acquiring images of the vehicle blind area; the radar is used for sensing the distance between the vehicle and the obstacles in the vehicle blind area; and the environmental sensor is used for collecting parameters related to driving safety in the vehicle driving environment.

[0174] The central AI processing unit is used for processing the information collected by the perception module, risk assessment according to the processed blind area information and vehicle working state, and sending warning signals to the warning module and control signals to the execution control module; it is internally provided with a deep learning model for processing multi-modal data, which performs data fusion and obstacle prediction processing on multi-modal data, and outputs risk type and level signals; and further outputs corresponding warning signals and control signals corresponding to the risk type and level signals.

[0175] The warning module is used to send risk type and level signals perceived by the driver according to the warning signals output by the central AI processing unit; the warning module includes a visual warning unit, a voice warning unit and a vibration warning module, which are used to make the driver perceive the risk type and level signals through vision, hearing and touch.

[0176] The execution control module is used to output control and drive signals according to the control signals sent by the central AI processing unit, so that the corresponding vehicle power device acts to actively prevent risks.

[0177] The disclosed vehicle blind area intelligent monitoring and early warning and active risk prevention system is based on the closed-loop design concept of full-scene perception, high-precision calculation, hierarchical warning and safety control, integrates perception modules, central AI processing units, warning modules, execution control modules and auxiliary collaboration modules, and realizes low-delay and high-reliable data interaction through Ethernet and CAN FD bus double link, which is suitable for multiple vehicle types such as sedans, SUVs, light trucks and heavy trucks. The specific technical solutions are as follows:

[0178] I. Perception module: multi-source fusion blind area information full-dimensional acquisition

[0179] The perception module adopts a multi-source heterogeneous fusion scheme of camera, millimeter wave radar, laser radar (optional) and environmental sensor, designs different layout and data processing strategies according to the spatial form difference and environmental interference characteristics of different vehicle blind areas, and realizes blind area information acquisition without dead angle, missed judgment and anti-interference.

[0180] 1. Core perception components: customized layout according to vehicle type

[0181] (1) High-definition intelligent camera assembly

[0182] The number, parameters and installation posture of the camera are designed according to the blind area area and position characteristics of the vehicle, so as to ensure that the coverage overlap of each blind area is ≥15% (to avoid monitoring failure caused by single-point shielding).

[0183] Taking a sedan / compact SUV (blind area area about 8-12m²) as an example, a total of 5 industrial-grade high-definition cameras are arranged, and the core parameters and installation positions are as follows:

[0184] Table 1: Car / Compact SUV high-definition camera layout position and parameter table

[0185]

[0186] For example, heavy truck / commercial vehicle (blind area area about 30-50m²), a total of 6 high anti-interference camera, for truck long wheelbase, high body, heavy load characteristics optimization, the core parameters and installation position as follows:

[0187] Table 2: Heavy truck / commercial vehicle high anti-interference camera layout position and parameter table

[0188]

[0189] (2) millimeter wave radar auxiliary unit:

[0190] Adopt 24GHz short range radar + 77GHz medium range radar combination, solve the problem of camera recognition failure in rain, fog, snow, strong light and other extreme environments, realize all-weather uninterrupted perception.

[0191] 24GHz short range radar (mainly adapt to car / truck near blind area):

[0192] Model: Continental ARS408-21, detection distance 0.5-30m, horizontal detection angle ±60°, vertical detection angle ±10°, ranging accuracy ±0.1m, speed range 0-100km / h.

[0193] Layout position: 1 way in the middle of car rear bumper (covering the lower rear near blind area), 1 way above the right front wheel of truck (covering the right front wheel 1-5m near blind area).

[0194] Core function: through FMCW (frequency modulation continuous wave) modulation technology, real-time output of target distance, speed, angle information, make up for the loss of target caused by camera image fogging in rain and fog.

[0195] 77GHz medium range radar (mainly adapt to truck far blind area):

[0196] Model: Bosch LRR6, detection distance 5-150m, horizontal detection angle ±25°, ranging accuracy ±0.5m, speed range 0-200km / h.

[0197] Layout position: 1 way on the left and right of the truck box tail (covering the side rear 5-50m far blind area).

[0198] Core function: To solve the problem of late target detection in the far blind area caused by the long wheelbase of trucks, the system can identify dynamic targets at a long distance (such as fast-approaching non-motor vehicles) in advance, providing more reaction time for early warning.

[0199] Laser radar supplementary unit (optional, suitable for high-end vehicles / complex scenarios).

[0200] Model: Hesai AT128, detection distance 0.3-200m, point cloud density 1.536 million points / s, horizontal field of view 120°, vertical field of view -25° to +15°.

[0201] Layout position: 1 line in the middle of the front bumper of a car, 1 line on the top of the cab of a truck.

[0202] Core value: In heavy rain, dense fog (visibility <50m), or strong light backlight (such as direct sunlight at noon), the system can accurately identify the target outline (such as pedestrian limbs, obstacle edges) through laser point cloud, forming a triple-redundancy perception with cameras and radars, and avoiding missed detection caused by failure of a single sensor.

[0203] (3) Environment-adaptive sensors:

[0204] Ambient light sensor (model: ams TSL2591): installed on the inner side of the front windshield, it can collect the ambient light intensity (0.01-100000 lux) in real time, and adjust the exposure parameters of the camera (such as reducing the exposure in strong light, and enabling night vision mode in weak light).

[0205] Rain sensor (model: Bosch RLS13): integrated in the front windshield, it detects the rain level (0-5 levels) through optical principles, and triggers the camera lens heating plate (automatically turned on above moderate rain) and rain and fog removal algorithm (enhance image contrast in heavy rain).

[0206] Road friction coefficient sensor (integrated in the ESP system): it can collect the road adhesion coefficient (0.1-1.0, dry road ≈0.8, wet and slippery road ≈0.4) in real time, and provide the basis for brake force adjustment for the execution control module.

[0207] II. The central AI processing unit constructs a multi-dimensional fusion risk intelligent evaluation system in the following way:

[0208] The central AI processing unit is based on a heterogeneous computing platform (CPU+GPU+FPGA), integrating image preprocessing acceleration module, multi-source fusion target detection module, scene matching and risk quantification module, realizing end-to-end low-latency processing from target recognition to risk classification (total delay ≤100ms), and supporting OTA remote iterative optimization algorithm.

[0209] 1. Hardware computing platform:

[0210] Adopting a heterogeneous chip set conforming to the ISO 26262 ASIL D functional safety level:

[0211] Main CPU: Infineon AURIX TC497 (dual-core 32-bit, 300MHz), responsible for system control, risk decision-making, and execution of control instruction output. GPU: NVIDIA Jetson AGX Orin (200TOPS of computing power), responsible for deep learning target detection and image fusion.

[0212] FPGA: Xilinx XC7Z045 (280K logic units), responsible for perception data preprocessing acceleration (such as filtering, encoding) and sensor data synchronization.

[0213] Perception data preprocessing and synchronization mechanism method is as follows:

[0214] Data preprocessing: The raw video frames (2K / 4K) output by each camera are first preprocessed in real time by the built-in FPGA chip, including Gaussian filtering (3x3 convolution kernel, removing salt and pepper noise), dark channel prior rain and fog removal (window size 15x15, restoring foggy image details), multi-exposure fusion (3 exposure levels combined, dynamic range increased to 140dB), preprocessing delay ≤10ms.

[0215] Space-time synchronization: Multi-sensor data synchronization is achieved through hardware timestamping + software calibration — each sensor is connected to the vehicle's 1PPS (1 pulse per second) clock signal, generating timestamps accurate to the microsecond level; when the central AI processing unit receives data, it aligns the timestamps (deviation ≤50μs) to avoid target position calculation deviations caused by sensor sampling delays (e.g., target distance error <0.1m due to asynchronous camera and radar data).

[0216] Data compression and transmission: Preprocessed video data is compressed using H.265 encoding (compression ratio 50:1, 2K video bitrate reduced to 4Mbps), transmitted to the central AI processing unit via Ethernet (1000BASE-T1); radar data is transmitted using CAN FD bus (transmission rate 8Mbps), ensuring single-frame data transmission delay ≤30ms.

[0217] Memory and storage: 8GB LPDDR5 memory (supports high-speed data caching), 64GB eMMC flash memory (stores AI models, log data, and OTA firmware).

[0218] 2. Core algorithm modules:

[0219] (1) Multi-source fusion target detection and classification module:

[0220] Based on the improved YOLOv8 deep learning model + multi-sensor data fusion, accurate target recognition is achieved, solving the problems of small target missed judgment and multi-target overlapping misjudgment:

[0221] YOLOv8 model optimization:

[0222] Sample library construction: Collect 120,000 blind area scene samples (including 12 kinds of environments such as day and night / rain and fog / snow, and 8 kinds of targets such as pedestrians / non-motor vehicles / fixed obstacles), real-time recognition of targets in blind area and classification as: pedestrians (including children, the elderly), non-motor vehicles (bicycles, electric vehicles), dynamic obstacles (moving cartons, stones), fixed obstacles (curbs, guardrails), recognition accuracy ≥98%, single frame processing time ≤20ms; Among them, small target samples (such as children, pets, small obstacles) account for 35%, ensuring the model's ability to recognize low / small targets.

[0223] Network structure improvement: Add attention mechanism (CBAM) to YOLOv8 backbone layer to enhance the extraction of key features of targets (such as pedestrian heads and non-motor vehicle wheels); Use multi-scale feature fusion (FPN+PAN) in neck layer to support simultaneous recognition of small targets (such as 5cm×5cm stones) within 10m and large targets (such as trucks) within 50m.

[0224] Model quantization and acceleration: Quantize the model to INT8 precision using TensorRT tools, and the inference speed is increased to 25fps (2K image single frame processing time ≤40ms), recognition accuracy: pedestrians ≥99.2%, non-motor vehicles ≥98.8%, fixed obstacles ≥98.5%, dynamic obstacles ≥98.0%, false positive rate <0.8%.

[0225] Multi-sensor data fusion algorithm:

[0226] Use weighted probability fusion strategy to fuse camera (target category, outline), radar (distance, speed), and lidar (outline, material) data:

[0227] Category fusion: Based on the camera recognition result (high accuracy in category judgment), radar data is used to assist in filtering non-target interference (such as roadside tree shadows, and radar without speed signal is determined as non-dynamic target).

[0228] Position fusion: Based on laser radar point cloud positioning (distance accuracy ±0.05m), correct the position deviation caused by the perspective of the camera (such as the distance calculation error of the target in the distance from 0.5m to 0.1m).

[0229] Confidence fusion: Assign weights to multi-sensor identification results of the same target (camera category confidence weight 0.6, radar speed confidence weight 0.3, laser radar profile confidence weight 0.1), after fusion, target confidence ≥0.8 is determined as valid target, to avoid single sensor misjudgment.

[0230] 3. Construct the scene matching and risk quantification module as follows:

[0231] Based on vehicle coordinate system + dynamic parameters + target behavior, construct a multi-dimensional risk assessment model to achieve accurate classification and early warning:

[0232] Vehicle coordinate system and scene matching:

[0233] The central AI processing unit pre-stores the electronic drawings of the body size of each vehicle model (such as sedan length 4.6m x width 1.8m, truck length 12m x width 2.5m), establishes a vehicle coordinate system with the center point of the cab as the origin, the vehicle longitudinal direction as the X axis, and the lateral direction as the Y axis; When receiving perception data, convert the pixel coordinates of the target in the image into actual coordinates in the vehicle coordinate system (X: longitudinal distance, Y: lateral distance) through camera intrinsic parameters (focal length, principal point) + extrinsic parameters (installation angle, height), matching accuracy ≤0.1m (such as a pedestrian in the right front wheel blind area identified by the camera, the actual position error after conversion is <0.05m).

[0234] Dynamic parameter acquisition and fusion:

[0235] Read vehicle dynamic parameters (vehicle working state parameters) in real time through CAN FD bus (update frequency 100Hz), including: driving parameters: vehicle speed v (0-200km / h, accuracy ±0.1km / h), steering angle θ (-45° to +45°, accuracy ±0.5°), throttle opening (0-100%), brake state (0 = no brake, 1 = light brake, 2 = heavy brake).

[0236] Vehicle state: ESP working state (0 = normal, 1 = active anti-lock), gear (P / R / N / D), load (truck through air suspension sensor, accuracy ±50kg).

[0237] Driver state: Obtain driver head orientation (0 = forward, 1 = left blind area, 2 = right blind area), blink frequency (judge whether tired) through DMS (Driver Monitoring System), and adjust the warning strategy (e.g. if the driver has looked at the right blind area, the risk warning intensity in the right blind area is reduced by 50%).

[0238] 4. Collision risk quantification and classification:

[0239] Introduce improved TTC (Time to Collision) + target danger coefficient to build a risk assessment model, to avoid misjudgment caused by traditional TTC relying only on distance / speed:

[0240] 1) Basic TTC calculation:

[0241] Straight driving scenario: (X is the longitudinal distance between the target and the vehicle, v is the current speed).

[0242] Steering scenario: (θ is the steering angle, Y is the lateral distance of the target, when the steering direction is consistent with the target position, Y takes a positive value, otherwise takes a negative value).

[0243] 2) Target danger coefficient correction:

[0244] Assign a danger coefficient K to different targets (pedestrian K=1.2, non-motor vehicle K=1.0, dynamic obstacle K=0.8, fixed obstacle K=0.6), and correct the actual (Such as under the same TTC, the actual risk level of pedestrians is higher).

[0245] 3) Road condition and load correction:

[0246] Combine the road friction coefficient μ (wet road μ=0.4, dry road μ=0.8) and the truck load M (full load M=1.0, empty load M=0.7), and correct the TTC warning threshold as follows:

[0247] TTC threshold of wet road = TTC threshold of dry road × (0.8 / μ).

[0248] TTC warning threshold of full load truck = TTC warning threshold of empty load truck × M.

[0249] Based on the corrected collision time, the risk level is divided as follows:

[0250] Low risk level: TTC > 3s, target far away from vehicle or moving direction has no intersection with vehicle. Medium risk level: 1s < TTC < 3s, target close to vehicle, potential collision possible. High risk level: TTC < 1s, target close fast, collision will happen in short time.

[0251] See the following table for details:

[0252] Table 3: Risk level and modified TTC range comparison table

[0253]

[0254] 5. Warning module:

[0255] The warning module is constructed into a multi-modal collaborative hierarchical warning system in the following way:

[0256] The warning module realizes personalized and non-interfering warning based on driver attention state + risk level + scene environment, adopts visual + auditory + tactile + HUD four-modal coordination, and avoids the problems of insufficient warning or excessive interference caused by traditional single warning.

[0257] The warning components of the warning module and the hardware configuration are as follows:

[0258] 1) Visual warning unit:

[0259] Instrument panel LCD screen (resolution 1920x720): display dynamic icons at the screen corresponding blind area position (left blind area -> left side, right blind area -> right side), icon color changes with risk level (yellow -> orange -> red), size increases with decreasing TTC (TTC = 3s, icon size 2cmx2cm, TTC = 1s, increased to 4cmx4cm).

[0260] HUD head-up display (model: Continental HUD100, projection distance 2.5m): high risk, project a red emergency brake prompt box (size 15cmx8cm), superimpose target distance information (such as right blind area pedestrian, distance 3m), brightness automatically adjusts with ambient light (0-100000lux adaptive).

[0261] Blind area indicator light (installed inside the left and right rearview mirrors): low risk -> yellow always on, medium risk -> orange flashing (frequency 2Hz), high risk -> red flashing (frequency 5Hz), light intensity > 500cd / m² (ensure visible in daylight).

[0262] 2) Auditory warning unit:

[0263] Vehicle audio (supporting partition sound): left blind area warning → only left speaker sound, right blind area warning → only right speaker sound, realizing accurate direction prompt.

[0264] Voice broadcast content (customized according to risk level):

[0265] Low risk: attention: [blind area location] has [target type] (such as attention: right front blind area has pedestrians), volume 60dB (10dB lower than normal music), single broadcast. Medium risk: alert: [blind area location] [target type] is close, please slow down, volume 70dB, broadcast 1 time every 2s. High risk: danger! [blind area location] is about to collide, brake immediately!, volume 85dB (sharp warning sound + voice), continuous broadcast until the risk is removed.

[0266] Audio format optimization: use 2000-4000Hz frequency band (most sensitive range of human ear), avoid low-frequency noise interference to ensure broadcast clarity.

[0267] 3) Tactile warning unit:

[0268] Steering wheel vibration motor (1 left and 1 right, model: Nidec VB-20): left blind area warning → left motor vibration, right blind area warning → right motor vibration, vibration frequency and intensity increase with risk level: medium risk: frequency 5Hz, intensity 3N (slight vibration, does not affect steering operation). High risk: frequency 10Hz, intensity 8N (obvious vibration, forced to attract attention).

[0269] Seat vibration module (installed on the left and right sides of the seat backrest, model: Bosch VS-30): seat module on the corresponding blind area side vibrates when high risk (such as right blind area high risk → seat right side vibration), vibration mode is short pulse + long pulse (0.5s vibration + 0.5s stop, cycle 3 times), avoid continuous vibration causing driver discomfort.

[0270] 4) Multi-modal coordination mechanism of warning module:

[0271] The triggering of each warning mode has the following time difference control: first, the visual warning is implemented by the warning module (0ms triggering), then the auditory warning is implemented by the warning module after a delay of 100ms (100ms delay, giving the driver a first look icon reaction time), and the tactile warning is implemented by the warning module after a further delay of 100-300ms (medium risk 300ms delay, high risk 100ms delay), avoiding sensory overload caused by simultaneous triggering of multiple modes.

[0272] Driver state linkage: adjust warning intensity in combination with DMS system data:

[0273] If the driver's head is oriented towards the blind spot (e.g. when the right blind spot warning, the driver is looking at the right rearview mirror): medium risk warning intensity is reduced by 50% (e.g. vibration frequency from 5Hz to 2.5Hz), low risk warning only retains visual cues. If the driver is in a fatigue state (blink frequency <5 times / minute): all risk level warning intensity is increased by 30% (e.g. volume from 60dB to 78dB).

[0274] Environmental adaptive adjustment:

[0275] High-speed driving (v≥100km / h): medium risk TTC warning threshold is relaxed from 3s to 4s, early warning. Urban congestion (v≤30km / h): low-risk warning only retains visual cues to avoid frequent voice / vibration interference with driving.

[0276] 6. Execution control module: The execution control module builds a safe and redundant active intervention system:

[0277] The execution control module is deeply linked with the vehicle ESP (electronic stability program), braking system, and power system, and based on risk level + road condition + vehicle state, it realizes gradual and high-safety active control, avoiding vehicle out-of-control or cargo overturn caused by traditional emergency braking, while meeting functional safety (ISO 26262 ASIL D) and expected functional safety (SOTIF) requirements.

[0278] 1) Execution control hardware architecture:

[0279] Core control unit: dual-MCU redundant design - the master MCU (Infineon AURIX TC497) and the slave MCU (Renesas RH850) synchronously receive risk instructions from the central AI processing unit, and through the two-of-two voting mechanism (only when the two MCUs are consistent, the instruction is executed), to avoid false triggering or non-triggering caused by single MCU failure.

[0280] Braking control interface: access the vehicle ESP system (e.g. Bosch ESP 9.3) through the CAN FD bus, output target deceleration instruction (0-1.0g), and realize precise brake force distribution by ESP control ABS actuator.

[0281] Power control interface: access the engine ECU (e.g. Continental EDC17) and transmission TCU, output fuel cut instruction (engine stops fuel supply) and downshift instruction (automatic transmission drops to low gear, uses engine braking to slow down) in high-risk situations.

[0282] State feedback interface: Real-time acquisition of brake system pressure (0-150 bar), wheel speed (0-200 km / h), deceleration (-1.0-1.0g), forming a control-feedback closed loop to ensure that the execution action meets the expectations.

[0283] 2) Scene-based execution of control strategy:

[0284] (1) Low risk / medium risk: only warning, no intervention:

[0285] In low-risk (TTC> 3s) and medium-risk (1s

[0286] (2) High risk: hierarchical active intervention:

[0287] According to the vehicle speed, road conditions, and load differences, a gradual intervention strategy is adopted to prioritize speed reduction and then braking to ensure vehicle stability:

[0288] Phase 1: Power speed reduction (trigger condition: TTC≤1.5s, no brake pedal pressed):

[0289] Output engine fuel cut command (cut off fuel supply) and transmission downshift command (such as D to 2), use engine braking to slow down, deceleration control in 0.2-0.3g (avoid passenger forward inclination), while turning on the double flash (frequency 100 times / minute), reminding the rear vehicle to avoid.

[0290] Phase 2: Partial braking (trigger condition: TTC≤1.2s, insufficient effect of power speed reduction):

[0291] Output target deceleration 0.4-0.6g command to ESP system, ESP distributes brake force according to wheel speed signal (front wheel brake force ratio 60%, rear wheel ratio 40%), while activating ABS anti-lock function (to avoid wheel lock); if it is a truck heavy load scene, the brake force is reduced by 20% (such as 0.4g to 0.32g), to prevent rear wheel slip caused by heavy load.

[0292] Phase 3: Emergency braking (trigger condition: TTC≤1.0s, partial braking still cannot avoid collision:

[0293] Output Maximum deceleration 0.8g instruction (dry road) or 0.5g instruction (wet and slippery road), ESP linkage EBD (electronic brake force distribution) adjusts the front and rear wheel brake force ratio (wet and slippery road front wheel ratio 70%, reduces rear wheel lock); At the same time, trigger the seat belt pretensioning (if the vehicle is equipped), tighten the seat belt by 2-3 cm, reduce the driver / passenger body forward distance during collision.

[0294] Brake rear control: after emergency brake stop, keep double flash open for 30s (until the driver manually releases), at the same time lock the throttle (avoid mispressing the throttle to cause the vehicle to start suddenly), only allow to shift into P or N.

[0295] (3) Special scene adaptation strategy:

[0296] Wet / snowy road: based on the road friction coefficient μ collected by ESP, limit the maximum deceleration to μ×0.8 (such as μ=0.4, maximum deceleration 0.32g), at the same time, extend the brake distance prediction (such as dry road brake distance 10m, wet road adjusted to 20m), trigger intervention 0.5s in advance.

[0297] Heavy load truck scene: according to the load data collected by the air suspension sensor, widen the TTC warning threshold by 20% (such as empty load TTC=1s, full load adjusted to 1.2s), brake force gradual change time extended to 1.5s (empty load 0.8s), avoid the vehicle nodding or cargo overturning caused by heavy load.

[0298] Rear scene: when the vehicle is in R, the rear blind area high-risk intervention strategy is adjusted to maximum deceleration 0.3g, at the same time, the engine is turned off (avoid power interruption when reversing), only through brake control, prevent rear collision caused by reversing emergency brake.

[0299] 7、Auxiliary coordination and system safety module:

[0300] 1) Driver monitoring coordination (DMS linkage):

[0301] Access DMS system (collect driver's face image through in-vehicle camera), judge whether the driver is paying attention to the blind area in real time: if the driver's head turns to the blind area (such as right blind area warning, head right turning angle >30°) and the line of sight focuses on the blind area (eye gaze direction consistent with the blind area), the medium risk warning intensity is reduced by 50% (such as vibration intensity from 3N to 1.5N), avoid the interference of strong warning when the driver has already noticed. If the driver does not pay attention to the blind area (head facing forward, line of sight not looking at the warning direction) and is in a fatigue state (blinking frequency <5 times / minute), the high-risk warning is triggered 0.2s in advance, at the same time, the seat vibration intensity is increased by 50%.

[0302] 2) System self-diagnosis and degradation mechanism:

[0303] Sensor self-diagnosis: The central AI processing unit performs health status detection on each sensor every 100ms - including camera (whether blocked, blurred image), radar (whether signal strength is normal, whether there is no target output), laser radar (whether the number of point clouds meets the standard); if a sensor fails (such as a camera blocked by mud), immediately display [sensor name] failure (such as right front camera failure) on the dashboard, and the system automatically switches to a degraded mode (such as using only radar data for basic warning).

[0304] 3) Function degradation strategy:

[0305] Primary degradation (single camera / radar failure): retain other available sensors, maintain 90% warning function (such as right front camera failure, use right front radar data to judge target). Secondary degradation (half sensor failure): only retain monitoring of core blind areas (such as truck right front, car A pillar), turn off non-core blind area warning. Tertiary degradation (all sensor failure): trigger system failure alarm (dashboard red failure light + voice prompt), turn off active control function, only retain blind area warning icon (prompt driver to manually observe).

[0306] 4) Data security and functional safety:

[0307] Data transmission security:

[0308] Ethernet transmission uses SOME / IP protocol + TLS 1.3 encryption (key length 256 bits) to prevent perception data from being tampered with. CAN FD bus uses CANoe encryption (AES-256 algorithm) and executes control instructions with CRC check (check bit 16 bits) to avoid transmission errors.

[0309] OTA upgrade security:

[0310] Firmware upgrade uses dual partition storage (primary partition + backup partition), automatically reverts to backup partition in case of upgrade failure, avoiding system paralysis. Upgrade package uses RSA-2048 signature + AES-128 encryption, only accepts officially authorized upgrade packages to prevent malicious firmware attacks. Functional safety certification: the system as a whole meets the requirements of ISO 26262 ASIL D level - key components (such as central AI processing unit, execution control MCU) use hardware redundancy, software uses fault injection testing (simulate 1000+ fault scenarios) to ensure that single point failure will not cause safety risks.

[0311] 8. Data storage and traceability:

[0312] Local storage: built-in 64GB eMMC flash memory, automatically stores 10s of key data before and after high-risk warnings - including perception module raw data (camera video clips, radar target information), central AI risk calculation results, execution control instructions, and vehicle state data, data storage format meets GB / T 32960 (New Energy Vehicle Data Recording) requirements, supports export through USB interface (password verification required).

[0313] Data encryption: stored data is encrypted using AES-256, only authorized personnel (such as traffic police, vehicle manufacturer technical personnel) can decrypt it using special tools, preventing data leakage or tampering, for accident traceability and system optimization.

[0314] The following is an example of a heavy truck (vehicle length 12m, width 2.5m, height 3.8m) to illustrate the implementation process of this system:

[0315] (I) Perception module installation:

[0316] Camera layout:

[0317] (1) Right side below the cab (1.2m from the ground): install 1 wide-angle camera (model: IMX490, resolution 1080P, wide-angle 150°), covering a 1.5m x 3m blind area in front of the right wheel.

[0318] (2) Left and right sides of the front of the cargo box (2.0m from the ground): each install 1 camera (same model), covering a 2m x 5m blind area in front of the left and right sides.

[0319] (3) Left and right sides of the middle of the cargo box (2.0m from the ground): each install 1 camera (same model), covering a 2m x 8m blind area in the rear of the left and right sides.

[0320] (4) Rear of the cargo box (1.5m from the ground): install 1 camera (same model), covering a 1.5m x 3m blind area in the rear.

[0321] Millimeter wave radar layout: install 1 24GHz radar (model: ARS408) above the right front wheel (1.5m from the ground), with a detection range of 0.5-50m and an angle of ±60°, assisting in identifying rain and fog targets in the right front wheel blind area.

[0322] (II) Central AI processing unit debugging:

[0323] Load the YOLOv8 truck blind spot dedicated model: This model was trained with 50,000 frames of truck blind spot video samples (including 20,000 frames in rainy weather, 15,000 frames at night, and 15,000 frames of pedestrians / non-motorized vehicles), and the recognition accuracy of pedestrians in the right front wheel blind spot of trucks reached 99.2%.

[0324] Vehicle dimensions input: Enter the truck dimensions (length 12m, width 2.5m, wheelbase 2.0m) into the electronic map and establish a vehicle coordinate system (origin is the center point of the cab, X-axis is the longitudinal direction of the vehicle, and Y-axis is the lateral direction).

[0325] TTC calculation parameter calibration: Through real vehicle testing, the TTC calculation error at different vehicle speeds is calibrated to ensure that the TTC error is ≤0.1s (e.g., at a vehicle speed of 30km / h, the deviation between the measured TTC and the theoretical value is <0.05s).

[0326] (III) Early Warning and Execution Control Test:

[0327] Low-risk scenario (TTC=4s): A pedestrian appears in the right front wheel blind spot (12m away from the vehicle, speed 10km / h). A yellow pedestrian icon appears on the right side of the instrument panel, and the car audio system announces: "Caution: Pedestrian in right front wheel blind spot." There is no vibration warning. Medium-risk scenario (TTC=2s): An electric vehicle appears in the right front blind spot (16.7m away from the vehicle, speed 30km / h). The instrument panel icon flashes orange, and an announcement is made every 2 seconds: "Caution: Electric vehicle in right front blind spot." The steering wheel vibrates (5Hz). High-risk scenario (TTC=0.8s): A moving cardboard box appears in the right rear wheel blind spot (4m away from the vehicle, speed 18km / h). The instrument panel icon is solid red, and an emergency announcement is made: "Danger! Brake immediately!" The seat vibrates, and automatic braking (braking force 0.8g) is triggered. Hazard lights activate, and the vehicle stops within 1.2s. The hazard lights remain on for 30 seconds after braking.

[0328] (iv) Passenger car adaptation adjustment:

[0329] For compact sedans (4.6m long, 1.8m wide), only the following adjustments are needed: Number of cameras: Reduced to 5 (1 on each side of the A-pillar, 1 below the right rearview mirror, and 1 on each side of the rear bumper). AI model: Replaced with the YOLOv8 model specifically designed for blind spots in sedans (optimized for pedestrians and obstacles below and behind the A-pillar). Automatic braking parameters: When the vehicle speed is ≥30km / h, the speed will first be reduced to 20km / h before braking to avoid sudden braking that could cause passenger discomfort.

[0330] The present invention provides a method for intelligent monitoring, early warning, and proactive risk prevention of vehicle blind spots. The method steps are shown in the table below:

[0331] Table 4: Central AI processing unit working order table

[0332]

[0333] The following are the detailed steps:

[0334] I. Overall logic of the process:

[0335] Perception of multi-source data input → Data reception and spatio-temporal synchronization → FPGA image preprocessing acceleration → GPU multi-source fusion target detection → CPU scene matching and risk quantification → Output risk level (to the warning module / execution control module) → Real-time self-diagnosis and OTA iteration (Core hardware support: Heterogeneous computing platform "CPU + GPU + FPGA", memory LPDDR5 cache data, eMMC flash storage AI model / log / firmware).

[0336] II. Detailed process steps (including document key parameters and rules):

[0337] Step 1: Multi-source data reception and spatio-temporal synchronization (delay ≤ 30ms):

[0338] 1.1 Data source access (classified by transmission protocol):

[0339] Ethernet (1000BASE-T1) access: pre-processed camera video data (H.265 compression, 2K bitrate 4Mbps), laser radar point cloud data (Hesai AT128: 0.3-200m detection, 1.536 million points / second), environmental sensor data (illumination: ams TSL2591 0.01-100000lux; rainfall: Bosch RLS13 0-5 levels; road surface friction coefficient: ESP integrated 0.1-1.0).

[0340] CAN FD bus (8Mbps) access: millimeter wave radar data (24GHz short range: Continental ARS408-21, 0.5-30m detection; 77GHz medium range: Bosch LRR6, 5-150m detection), vehicle dynamic parameters (update frequency 100Hz).

[0341] Driving parameters: vehicle speed v (0-200km / h, ±0.1km / h), steering angle θ (-45°~+45°, ±0.5°), throttle opening, brake state (0 = no brake / 1 = light brake / 2 = heavy brake).

[0342] Vehicle status: ESP working status, gear (P / R / N / D), truck load (air suspension collection ±50kg); Driver status (DMS output): head orientation (0 = forward / 1 = left blind area / 2 = right blind area), blink frequency (judgment of fatigue).

[0343] 1.2 Spatiotemporal synchronization mechanism:

[0344] Hardware basis: Each sensor accesses the vehicle 1PPS (1 pulse per second) clock signal to generate microsecond-level timestamps;

[0345] Synchronization calibration: The central AI processing unit aligns multi-source data through timestamps, with a synchronization deviation ≤50μs, ensuring that the target position calculation error is <0.1m (e.g., avoiding distance deviation caused by asynchronous camera and radar data).

[0346] Step 2: FPGA image preprocessing acceleration (delay ≤10ms).

[0347] (Based on Xilinx XC7Z045 FPGA, responsible for real-time data processing, reducing GPU / CPU load).

[0348] 2.1 Basic image processing:

[0349] Gaussian filter denoising: Use a 3×3 convolution kernel to filter salt and pepper noise in the original camera video frame (2K / 4K), preserving target edge details.

[0350] Dark channel prior rain and fog removal: Extract the image dark channel through a 15×15 sliding window, estimate the atmospheric light value and transmittance, and restore the details of the rain and fog image (e.g., pedestrian outline in heavy fog).

[0351] Multi-exposure fusion: Collect 3-exposure (low / medium / high) image frames, and through pixel-level fusion, improve the dynamic range to 140dB, solving the problem of overexposure / underexposure of targets in backlight (midday direct sunlight) or weak light (night).

[0352] 2.2 Environment adaptive adjustment:

[0353] Linkage ambient light sensor: Trigger camera night vision mode when illumination <10lux (minimum illumination 0.0005lux adapted for truck), and reduce exposure when illumination >10000lux.

[0354] Linkage rain sensor: When the rainfall is ≥3 levels (moderate rain), trigger the camera lens heating plate (-30℃~85℃ working) and enhance the rain and fog removal algorithm strength.

[0355] Oil stain / occlusion processing: For rear bumper camera, eliminate exhaust pipe oil stain occlusion through oil stain filtering algorithm, if the occlusion area > 30%, trigger sensor failure warning (follow-up self-diagnosis link).

[0356] Step 3: GPU multi-source fusion target detection (single frame processing ≤40ms, computing power support: NVIDIA Jetson AGX Orin 200TOPS).

[0357] 3.1 Improved YOLOv8 model target recognition:

[0358] Model training basis: Based on 120,000 + blind area scene samples (including 12 types of environments such as day and night / rain and fog / snow, 8 types of targets such as pedestrians / non-motor vehicles / fixed obstacles / dynamic obstacles), among which small target samples (children, pets, 5cm×5cm stones) account for 35%.

[0359] Network structure optimization:

[0360] Backbone layer: Add CBAM (channel attention mechanism) to enhance the extraction of key features of targets (pedestrian head, non-motor vehicle wheel).

[0361] Neck layer: Use FPN+PAN multi-scale feature fusion to support the simultaneous recognition of small targets (such as kerbs) within 10m and large targets (such as trucks) within 50m.

[0362] Model quantization and acceleration: Quantize the model to INT8 precision through the TensorRT tool, and the inference speed is increased to 25fps, and the recognition accuracy is:

[0363] Pedestrian ≥99.2%, non-motor vehicle ≥98.8%, fixed obstacle ≥98.5%, dynamic obstacle ≥98.0%, false positive rate <0.8%.

[0364] 3.2 Multi-sensor data weighted fusion:

[0365] Solve the single sensor failure problem, use weighted probability fusion strategy, weight is distributed according to sensor accuracy, please refer to the following table:

[0366] Table 5: Weighted fusion weight and rule table of multi-sensor data

[0367]

[0368] After fusion, when the target confidence is ≥0.8, it is marked as an effective target, and when it is <0.8, it is determined as an interference item (such as birds, leaves), to avoid false positives.

[0369] Step 4: CPU scenario matching and risk quantification (main CPU: Kontron AURIX TC497, decision delay ≤ 50ms).

[0370] 4.1 Vehicle coordinate system and target position matching:

[0371] Coordinate system establishment: Pre-store electronic drawings of the body size of each vehicle model (e.g., sedan 4.6m x 1.8m, truck 12m x 2.5m), with the center point of the cab as the origin, the longitudinal direction of the vehicle as the X-axis, and the lateral direction as the Y-axis to establish the vehicle coordinate system.

[0372] Coordinate conversion: Convert the pixel coordinates of the target in the image into actual coordinates in the vehicle coordinate system (X = longitudinal distance, Y = lateral distance) through camera intrinsic parameters (focal length, principal point) + extrinsic parameters (installation angle, height), with a matching accuracy of ≤0.1m (e.g., actual position error of a right front wheel blind area pedestrian <0.05m).

[0373] 4.2 Dynamic parameters and driver state fusion:

[0374] Vehicle dynamic parameter integration: Update parameters such as vehicle speed, steering angle, and load at a frequency of 100Hz, and adjust the risk threshold when the truck is fully loaded (M = 1.0). Driver state linkage: Determine whether the driver is paying attention to the blind area through DMS data (e.g., when the right blind area warning is triggered, the driver's head turns right by >30° and the line of sight is focused), which is used for risk level correction in the future (paying attention reduces the warning intensity).

[0375] 4.3 Collision risk quantification and level division:

[0376] Based on the improved TTC + multi-dimensional correction, avoid the defects of traditional TTC relying only on distance / vehicle speed.

[0377] Basic TTC calculation: Straight-line driving scenario: Distance / v (current vehicle speed).

[0378] Steering scenario: (θ = steering angle, Y is positive if it is in the same direction as steering, otherwise negative).

[0379] Target danger coefficient correction: Assign a target danger coefficient K: pedestrian K = 1.2, non-motor vehicle K = 1.0, dynamic obstacle K = 0.8, fixed obstacle K = 0.6. Comprehensive TTC = basic TTC / K (e.g., under the same TTC = 2s, the actual TTC of a pedestrian is 1.67s, with higher risk).

[0380] Road condition and load correction: road friction coefficient correction: wet and slippery road (μ = 0.4) TTC threshold = dry road (μ = 0.8) threshold x (0.8 / μ);

[0381] Truck load correction: full load (M = 1.0) TTC threshold = empty load (M = 0.7) threshold x M.

[0382] According to the corresponding risk level of the corresponding TTC and target state table, output the corresponding risk level:

[0383] Table 6: Risk level corresponding TTC and target state table

[0384]

[0385] Step 5: Auxiliary process (system safety and iteration):

[0386] 5.1 Real-time self-diagnosis (once every 100ms):

[0387] Sensor health detection: detect camera (whether blocked / blurred), radar (signal strength / no target output), lidar (whether the number of point clouds meets the standard). Degradation strategy trigger: first level degradation (single sensor failure): keep other sensors, maintain 90% warning function (e.g. right front wheel camera failure → use right front wheel radar data). Second level degradation (half sensor failure): only monitor core blind area (truck right front wheel, car A pillar), turn off non-core blind area warning. Third level degradation (all sensor failure): trigger system failure alarm (instrument panel red light + voice), turn off active control, only keep blind area warning icon.

[0388] 5.2 OTA remote iteration:

[0389] Firmware / model storage: use dual partition storage (main partition + backup partition), upgrade package is signed with RSA-2048 and encrypted with AES-128. Iteration logic: receive official authorized upgrade package → backup current firmware / model → upgrade main partition → verify successful upgrade → enable new version; if upgrade fails, automatically roll back to backup partition to avoid system paralysis.

[0390] The above-mentioned perception module, central AI processing unit, early warning module, execution control module, camera, radar and environment sensor, CPU, GPU, FPGA, target detection model, YOLOv8 model, driver monitoring system, engine control unit, ESP system of the vehicle, braking system and power system, MCU, ESP, ABS, engine ECU, gearbox TCU and other modules, components, systems, models can adopt the modules, components, systems, models suitable for the prior art, or adopt the modules, components, systems, models in the prior art, and adopt conventional technical means to construct.

[0391] The following table is a high-risk scenario execution control timing table:

[0392] Table 7: High-risk scenario execution control timing table

[0393]

[0394] The above examples are only used to illustrate the technical ideas and characteristics of the present application, and cannot be limited to the patent scope of the present application only by the present examples, i.e. any equivalent changes based on the present application still fall within the patent scope of the present application.

Claims

1. A vehicle blind spot intelligent monitoring, early warning, and active risk prevention system, characterized in that, The system includes a perception module, a central AI processing unit, an early warning module, and an execution control module; The perception module is used to collect vehicle blind spot information in all dimensions; the perception module includes a camera, radar and environmental sensors; the camera is used to collect images of the vehicle's blind spots; the radar is used to sense the distance between the vehicle and the target in the vehicle's blind spots; the environmental sensors are used to collect parameters related to driving safety in the vehicle's driving environment; The central AI processing unit processes the information collected by the perception module, performs risk assessment based on the processed blind spot information and vehicle operating status, issues warning signals to the warning module, and issues control signals to the execution control module. It has a built-in deep learning model for processing multimodal data, performs data fusion and target prediction processing on multimodal data, and outputs risk type and level signals. It further outputs corresponding warning signals and control signals according to the risk type and level signals. The warning module is used to issue risk type and level signals to the driver based on the warning signals output by the central AI processing unit; the warning module includes a visual warning unit, a voice warning unit and a vibration warning module, which are used to enable the driver to perceive the risk type and level signals through vision, hearing and touch. The execution control module is used to output control and drive signals based on the control signals issued by the central AI processing unit, so as to make the corresponding vehicle power unit act and realize proactive risk prevention.

2. The intelligent vehicle blind spot monitoring, early warning, and proactive risk prevention system according to claim 1, characterized in that, The central AI processing unit includes a CPU, a GPU, and an FPGA. The CPU is used for system control and risk assessment algorithm operation, and it outputs early warning signals and control signals. The GPU is used for multimodal data fusion and target detection. The FPGA is used for filtering, encoding, and data synchronization preprocessing of signals collected by the perception module. The data processed by the FPGA is sent to the CPU and GPU. The data processed by the GPU is sent to the CPU. The spatiotemporal synchronization deviation between the data processed by the FPGA and the data of the CPU and GPU is ≤50μs.

3. The intelligent vehicle blind spot monitoring, early warning, and proactive risk prevention system according to claim 2, characterized in that, Deep learning models include object detection models, which are built on the YOLOv8 model. The object detection model adds an attention mechanism to the backbone layer of YOLOv8 to enhance the extraction of key features of the target. The neck of the target detection model employs a structure combining a feature pyramid network and a path aggregation network, which improves multi-scale target detection performance through bidirectional feature fusion.

4. The intelligent vehicle blind spot monitoring, early warning, and proactive risk prevention system according to claim 2, characterized in that, The CPU connects to the driver monitoring system and / or engine control unit on the vehicle body; the CPU reads driver status information from the driver monitoring system in real time; and reads vehicle dynamic parameters from the engine control unit in real time, and outputs gradual active hazard prevention control signals based on risk level, road conditions, and vehicle status.

5. The intelligent vehicle blind spot monitoring, early warning, and active risk prevention system according to claim 1, characterized in that, The execution control module is linked with the vehicle's ESP system, braking system, and power system; The execution control module adopts a dual-MCU redundant architecture. The two MCUs synchronously receive control signals from the central AI processing unit, and the strategy is only executed when the control signals received by the two MCUs are consistent. The execution control module is equipped with a braking control interface, a power control interface, and a status feedback interface. The braking control interface is connected to the vehicle's ESP system via the CAN FD bus, outputting the target deceleration command, which in turn controls the ABS actuator to distribute braking force. The power control interface connects to the engine ECU and the transmission TCU. In case of high risk, it outputs a command to stop the engine from supplying fuel to the ECU and a command to downshift the automatic transmission to the TCU. The status feedback interface receives real-time data on fuel pressure, wheel speed, and deceleration from the vehicle's braking system, which is then used as feedback data for the MCU.

6. The intelligent vehicle blind spot monitoring, early warning, and active risk prevention system according to claim 1, characterized in that, Environmental sensors include light intensity sensors, rain sensors, and road friction coefficient sensors. The light intensity sensor is used to collect ambient light intensity to coordinate with the camera to adjust exposure parameters. The rain sensor is used to detect rainfall levels through optical principles, which in turn triggers the camera lens heating element and rain / fog removal algorithm. The road surface friction coefficient sensor collects the road surface friction coefficient in real time.

7. The intelligent vehicle blind spot monitoring, early warning, and active risk prevention system according to claim 1, characterized in that, The visual warning unit includes an instrument panel LCD screen, blind spot indicator lights, and a head-up display (HUD). The instrument panel LCD screen displays dynamic icons representing the risk level of the left and right blind spots on the left and right sides of the screen, with the icon colors changing from yellow to orange to red as the risk level increases, and the size increasing as the traffic conflict time decreases. The blind spot indicator lights are installed inside the left and right rearview mirrors, and the flashing colors change from yellow to orange to red as the risk level increases, with the flashing frequency also increasing. The HUD projects a red emergency braking warning frame when the risk level is high, overlaying the target distance information to the vehicle body, and the display brightness automatically adjusts according to the ambient light.

8. The intelligent vehicle blind spot monitoring, early warning, and active risk prevention system according to claim 1, characterized in that, The voice warning unit includes an in-vehicle audio system; the in-vehicle audio system broadcasts voice warnings about the risks and handling prompts in the left and right blind spots; The vibration warning module includes a steering wheel vibration module and a seat vibration module. The steering wheel vibration module includes left and right steering wheel vibration motors corresponding to the left and right blind spot risks. The vibration frequency and intensity of the left and right steering wheel vibration motors increase with the risk level. The seat vibration module is installed on the left and right sides of the seat back. When the risk assessment is high, the seat module on the blind spot side vibrates. The vibration mode is intermittent cyclic vibration of shorter pulses and longer pulses.

9. A method for intelligent monitoring, early warning, and active risk prevention of vehicle blind spots using the intelligent vehicle blind spot monitoring, early warning, and active risk prevention system described in claim 1, characterized in that, This method includes the following steps: Cameras capture images of the vehicle's blind spots; radar senses the distance between targets and the vehicle in the blind spots; environmental sensors collect parameters related to driving safety in the vehicle's driving environment. The central AI processing unit uses LiDAR point cloud positioning as a reference to correct the positional deviation of the camera caused by perspective. It assigns weights to the multi-sensor identification results of the same target, and determines the target as a valid target when the confidence score of the fused target is ≥0.8, thus avoiding misjudgment by a single sensor; It uses a deep learning model to fuse target image data captured by cameras, target-vehicle distance data captured by radar, and vehicle status data from the engine control unit to identify target categories. Real-time identification and classification of targets within blind spots: pedestrians, non-motorized vehicles, dynamic obstacles, and fixed obstacles; It constructs a multi-dimensional risk assessment model based on the vehicle coordinate system, vehicle dynamic parameters, and target state. It combines the basic collision time with the target hazard coefficient to calculate the comprehensive collision time, and uses the comprehensive collision time as a risk level assessment indicator to determine the risk level. The comprehensive collision time is calculated using the following formula: ； ； ； In the above formula: TTC stands for Time Between Collisions; K represents the risk coefficient assigned to different targets; among them, pedestrian K value > non-motorized vehicle K value > dynamic obstacle K value > fixed obstacle; For a straight-line head-on collision; For the time of the side impact during steering; v represents the current vehicle speed; θ is the vehicle steering angle; μ is the road surface friction coefficient; where the μ value of wet roads is less than that of dry roads. M is the vehicle weight coefficient; where the M value of a heavier vehicle is greater than that of a lighter vehicle. X represents the longitudinal distance between the target and the vehicle; Y represents the lateral distance between the target and the vehicle. Y takes a positive value when the steering direction is consistent with the target position, and a negative value otherwise. Let μ be the road surface friction coefficient and M be the vehicle weight coefficient. After the comprehensive collision time calculation is completed, the TTC warning threshold is corrected by combining the road surface friction coefficient μ and the vehicle weight coefficient M. The correction formula is as follows: The TTC warning threshold for wet and slippery roads = the TTC warning threshold for dry roads × (0.8 / μ); The TTC warning threshold for a fully loaded truck = the TTC warning threshold for an empty truck × M; Based on the comprehensive collision time, three risk levels are set: low, medium, and high. The conditions for low risk level include: TTC > low risk TTC warning threshold; the conditions for medium risk level include: high risk TTC warning threshold < TTC ≤ low risk TTC warning threshold; and the conditions for high risk level include: TTC ≤ high risk TTC warning threshold. The following actions will be taken in accordance with different risk levels: When the risk level is low or medium, the central AI processing unit outputs the corresponding warning signal to the warning module based on the risk type and level, without interfering with the vehicle's operation. When the risk level is high, the central AI processing unit outputs corresponding warning signals in addition to the risk type and level signals, and also outputs control signals to enable the execution control module to actively intervene in stages. Based on the differences in vehicle speed, road conditions and load, a phased and gradual intervention strategy is adopted to achieve priority deceleration and then braking to ensure vehicle stability. The central AI processing unit outputs signals to the execution control module according to different vehicle driving scenarios, enabling the vehicle to perform the following actions: When the road surface is wet / slippery / icy, based on the road friction coefficient μ, the maximum deceleration of the vehicle is limited to μ×0.8, and the braking prediction distance is extended, triggering intervention 0.5s in advance; When the truck is heavily loaded, the corresponding TTC warning threshold will be increased by 20% based on the load data, and the braking force change time will be extended to 1.5s. In a reversing scenario, when the vehicle is in reverse gear, the maximum deceleration of the vehicle is adjusted to 0.3g, and the engine fuel is cut off. Control is achieved solely through the brakes to prevent rear-end collisions caused by sudden braking while reversing.

10. The intelligent monitoring, early warning, and proactive risk prevention method for vehicle blind spots according to claim 9, characterized in that, Based on differences in vehicle speed, road conditions, and load, the following phased, gradual intervention strategy is adopted: Phase 1: Power deceleration, trigger condition: TTC≤1.5s, no brake applied; The central AI processing unit outputs an engine fuel cut-off command, which is then signaled by the execution control module to cut off the fuel supply. It also outputs a transmission downshift command, which is signaled by the execution control module to control the deceleration at 0.2-0.3g. At the same time, it activates the hazard lights to warn vehicles behind to avoid the vehicle. Second stage: Initiate partial braking, trigger condition: TTC≤1.2s, and the power deceleration effect is insufficient; The central AI processing unit outputs a command with a target deceleration of 0.4-0.6g, which causes the ESP to distribute braking force according to the wheel speed, with the front wheels receiving 60% of the braking force and the rear wheels receiving 40%, while activating the ABS anti-lock braking function; in the case of a heavy-load truck, the braking force is reduced by 20% to prevent rear wheel slippage caused by heavy load; Phase 3: Emergency braking, triggered by TTC ≤ 1.0s, where partial braking still cannot prevent a collision; The central AI processing unit outputs a maximum deceleration command of 0.8g under dry road conditions and a maximum deceleration command of 0.5g under wet and slippery road conditions, enabling ESP to work with EBD to adjust the braking force ratio of the front and rear wheels; at the same time, it outputs a seat belt pretensioning command to tighten the seat belt by 2-3cm, reducing the forward lean distance of the driver / passenger's body during a collision. Post-brake control: After emergency braking to a stop, keep the hazard lights on for 30 seconds until the driver manually releases them. At the same time, lock the accelerator to prevent accidental acceleration that could cause the vehicle to start suddenly. Only P or N gear is allowed.

11. The intelligent monitoring, early warning, and proactive risk prevention method for vehicle blind spots according to claim 9, characterized in that, The central AI processing unit interacts with the sensing module, early warning module, and execution control module via dual links of Ethernet and CAN FD bus. Ethernet transmission uses the SOME / IP protocol and TLS 1.3 encryption with a key length of 256 bits to prevent the perceived data from being tampered with. The CAN FD bus uses CANoe encryption and the AES-256 algorithm. Control commands are executed with CRC checksums, with 16 check bits, to prevent command transmission errors. OTA upgrade security: Firmware upgrades use dual-partition storage, with a primary partition and a backup partition. In case of upgrade failure, it will automatically roll back to the backup partition to avoid system crashes. The upgrade package uses RSA-2048 signature and AES-128 encryption, and only officially authorized upgrade packages are accepted to prevent malicious firmware attacks.

12. The intelligent monitoring, early warning, and proactive risk prevention method for vehicle blind spots according to claim 9, characterized in that, Perform self-diagnosis on the sensing module using the following steps, and then perform the following degrading processing based on the diagnostic results: The central AI processing unit performs health status checks on each sensor in the perception module every 100ms. The health status checks include: whether the camera is blocked or whether the captured image is blurry; whether the radar signal strength is normal or whether there is no target output; and whether the number of LiDAR point clouds meets the standard. If a sensor is determined to be faulty, the name of the faulty sensor will be displayed on the instrument panel, and the system will automatically switch to degraded mode. Degradation work modes include: When a single sensor in each type of sensor fails, the system enters a first-level degraded operating mode, retaining other available sensors and maintaining 90% of the early warning function; When half of the sensors in each type fail, the system enters a secondary degraded working mode, retaining only the monitoring of the core blind area and turning off the warning of non-core blind areas. When all sensors fail, the system enters a level 3 degraded operating mode, which displays a red fault light and voice prompts on the instrument panel, disables the active control risk function, and retains only the blind spot warning icon.

Images