Intelligent vehicle-mounted blind area early warning device based on multi-modal sensor fusion

Abstract

The invention relates to the technical field of intelligent early warning, in particular to an intelligent vehicle-mounted blind area early warning device based on multi-modal sensor fusion, which comprises a sensing array module, a driver sensing module, an environment recognition module, an early warning decision module and a linkage execution module, the sensing array module is used for being connected with a vehicle-mounted system to obtain vehicle-mounted data related to a driver and a driving environment. The driver sensing module is used for analyzing driving habit information of the driver according to the vehicle-mounted data. The environment identification module is used for analyzing environment identification information of the vehicle according to the vehicle-mounted data. The early warning decision module is used for generating a grading threshold value of blind area early warning monitoring according to the driving habit information and the environment identification information; a personalized early warning and monitoring model is established by using a deep learning algorithm, and dynamic risk levels of a driver in a driving process are divided. According to the method, the personalized driving habit model is generated through deep learning, invalid interference is reduced, and the blind area early warning efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent warning technology, and in particular to an intelligent vehicle-mounted blind spot warning device based on multimodal sensor fusion. Background Art

[0002] Modern vehicles are typically equipped with multiple types of sensors to monitor the vehicle's surroundings. Each sensor has its own specific advantages and limitations. Since a single type of sensor cannot provide accurate information in all situations, fusing data from different types of sensors has become an effective solution.

[0003] Existing technologies include the C-Scape system proposed by Weifu Technology, which integrates a combination of lidar and 4D imaging radar, has dynamic deflection scanning capabilities, can effectively cover the vehicle's A-pillar blind spot, and accurately monitor low obstacles, improving the vehicle's perception and safety in complex traffic environments.

[0004] While these products can monitor blind spots, they lack a post-monitoring feedback warning mechanism and may not fully consider the current driving scenario, leading to false or missed alerts and negative feedback, impacting driver trust and user experience. They also fail to learn and analyze drivers' operating habits, making it impossible to optimize warning strategies based on individual differences. Therefore, it is necessary to propose an intelligent in-vehicle blind spot warning device based on multimodal sensor fusion. Summary of the Invention

[0005] To solve the above problems, the present invention provides an intelligent vehicle-mounted blind spot warning device based on multimodal sensor fusion. By establishing a dynamic coupling model of driving habits and risks, integrating the driver's biological characteristics and operating behaviors, and using deep learning to generate a personalized driving habit model, it can reduce invalid interference and improve the efficiency of blind spot warning.

[0006] In order to achieve the above objectives, the technical solution of the present invention is as follows: an intelligent vehicle-mounted blind spot warning device based on multimodal sensor fusion, including a sensor array module, a driver perception module, an environment recognition module, a warning decision module and a linkage execution module.

[0007] The sensor array module is used to connect with the vehicle system to obtain vehicle data related to the driver and the driving environment.

[0008] The driver perception module is used to analyze the driver's driving habit information based on vehicle data.

[0009] The environment recognition module is used to analyze the vehicle's environment recognition information based on the vehicle's onboard data.

[0010] The early warning decision module is used to generate grading thresholds for blind spot early warning monitoring based on driving habit information and environmental recognition information; use deep learning algorithms to establish a personalized early warning monitoring model to divide the driver's dynamic risk level during driving; and combine the high-precision map data data of the on-board system to identify the scene and link the vehicle dynamics model to predict the collision time of blind spot monitoring.

[0011] The linkage execution module is used to integrate the beamforming speaker, light director and tactile feedback mechanism to forecast the warning feedback information of the classification threshold according to the dynamic risk level warning line and collision time.

[0012] The technical principles of the above scheme are as follows:

[0013] The sensor array module collects and analyzes information from the various sensors. Based on the driver's concentration and steering wheel grip distribution, the system combines steering information to determine driving intent and analyzes driving habits based on individual needs, enhancing blind spot warning capabilities. It also identifies scenario patterns based on environmental characteristics, triggering severe weather warnings and compensating the monitoring weights of each sensor for inclement weather conditions, improving data collection and analysis capabilities. Based on this information analysis, it determines blind spot monitoring and warning thresholds, provides corresponding warning strategies, and implements various forecasting functions in a coordinated manner, forming an intelligent warning model during driving, thereby improving the reliability of blind spot warning monitoring.

[0014] The above scheme has the following beneficial effects:

[0015] 1. This invention builds a robust perception closed loop through dynamic modeling of environmental characteristics and adaptive weighting of multiple sensors. It analyzes environmental data in real time and dynamically adjusts sensor dominant weights. It also incorporates multimodal data generation technology. When cameras fail due to extreme conditions, an algorithm synthesizes radar point cloud data into a virtual visual image in real time, filling critical information gaps. This environment-driven collaborative perception mechanism ensures stable target detection in all weather conditions and scenarios, thereby reducing the risk of scene blindness.

[0016] 2. This invention integrates driver biometrics and operational behavior by establishing a dynamic coupling model of driving habits and risks. Through concentration tracking and steering wheel grip force distribution analysis, the concentration decay curve is quantified in real time. Deep learning is used to generate a personalized driving habit model based on historical steering behavior. The driver's intention is simultaneously monitored during decision-making. If their line of sight covers the risk area, only gentle tactile feedback is triggered. When distraction and unawareness of risk are detected, audio and visual intervention is initiated to reduce ineffective interference and improve the efficiency of blind spot warnings.

[0017] 3. This invention upgrades discrete warning outputs into coherent evasive guidance through hierarchical execution and spatiotemporal coordination. A multi-level response is divided based on collision time, giving the driver time to maneuver. Hardware clocks align acoustic, optical, and tactile signals, ensuring that optical positioning, acoustic guidance, and tactile guidance form a unified action-inducing chain across time and space. This upgrade of discrete warning outputs into evasive guidance with precise and controllable spatiotemporal precision shifts closed-loop protection logic from passive risk notification to active behavior correction, achieving a dynamic, adaptive model where perception is protection.

[0018] Furthermore, the driver perception module includes the following units:

[0019] The driving information acquisition unit is used to connect to the vehicle-mounted system, which is connected to the in-vehicle camera and the piezoelectric film array arranged on the steering wheel; based on the in-vehicle camera, it obtains the driver's eye movement direction, blinking frequency and head posture concentration information in real time; monitors the steering wheel grip position, pressure intensity and grip duration distribution information through the piezoelectric film array; and obtains vehicle steering information such as turn signal activation status, gear switching signal and accelerator and brake pedal depth through the in-vehicle system.

[0020] The multi-source data fusion unit is used to align concentration information, grip force distribution information and vehicle steering information by timestamp, and fuse them to generate a driving behavior intention prediction model.

[0021] The driving habit modeling unit is used to collect statistics on long-term vehicle steering information to form the driver's lane change habits and concentration attenuation curve; based on the pre-recorded driving behavior intention prediction model, a risk association model associated with driving habits is established.

[0022] Beneficial Effects: By capturing driver biometrics and vehicle operating signals in real time, a multi-source data fusion unit generates a behavioral intention prediction model after spatiotemporal alignment. The driving habit modeling unit then analyzes long-term operating patterns and establishes a risk association model. The core of this approach is to couple physiological status, operating habits, and real-time behavior to create a personalized early warning benchmark. This system accurately identifies distraction thresholds through a quantified concentration decay curve, predicts operating intentions based on lane change habits, and dynamically adapts to the driver's state, significantly improving the efficiency and reliability of human-machine collaboration.

[0023] Furthermore, the environment recognition module includes the following units:

[0024] The environmental data integration unit is used to connect to the on-board system to obtain vehicle positioning, driving direction and speed information in real time; and based on the networking of the on-board system, it obtains high-precision map data and extracts driving scenario information such as the coordinates of tunnel and bridge entrances and exits, the hierarchical structure of overpasses and the location of traffic signs; based on the networking of the on-board system, it obtains precipitation intensity and then uses the on-board system's infrared sensor to identify road waterlogging and icing conditions.

[0025] The environmental feature extraction unit is used to match tunnel coordinates through vehicle positioning, analyze lighting using images from the vehicle's external camera system, and use the vehicle's high-precision map data and lidar to detect overlapping multi-layer road structures; trigger weather warnings based on precipitation intensity; and construct environmental feature vectors.

[0026] The dynamic weight allocation unit is used to assign acquisition weights to different sensors based on different driving scenarios and environmental recognition information; and calculate the reliability of each sensor based on a Bayesian network; and compensate for the collected information through a generative adversarial network to fill information gaps.

[0027] The scene semantic modeling unit is used to integrate high-precision map data with real-time sensor data, generate an environmental network model, and output key semantic label warnings.

[0028] The cross-validation and output unit is used to verify the tunnel boundaries in the HD map data through the lidar point cloud and filter out temporary structural changes caused by construction. It also compares the millimeter-wave radar target trajectory of the vehicle system with the preset lane centerline in the HD map data to detect the risk of illegal lane changes by vehicles.

[0029] Beneficial effects: By fusing positioning, map, and meteorological data, the system can analyze complex scene structures such as tunnels and overpasses in real time. The environmental feature extraction unit quantifies the relationship between sudden changes in illumination and road topology, constructing a dynamic environmental feature vector. The dynamic weight allocation unit dynamically optimizes sensor weights based on a Bayesian network and compensates for data loss under extreme conditions through a generative adversarial network. The scene semantic modeling unit fuses static maps with real-time sensor data to generate semantic risk labels for areas such as slippery areas in tunnels and conflict points at overpasses. The cross-validation unit filters out construction interference and positioning drift through multi-source data cross-verification. By transforming discrete environmental information into a scene-adaptive perception strategy, leveraging the semantic prior knowledge of high-precision maps, and dynamically correcting the environmental model through real-time sensing, the system maintains stable and reliable perception capabilities even in failure scenarios such as heavy rain, tunnels, and multi-layer overpasses, providing a robust environmental cognitive foundation for early warning decisions.

[0030] Furthermore, the early warning decision module includes the following units:

[0031] A personalized risk assessment unit is used to predict the driving behavior intention model and risk association model; a deep learning algorithm is used to analyze the individualized risk index.

[0032] The dynamic grading decision unit is used to combine the grading threshold and the personalized risk index to generate a dynamic risk level warning line; when identifying special scenarios, it activates the vehicle system's lidar to scan the movement trajectory of targets in blind spots in advance; and calculates the vehicle's expected safety distance in real time to trigger a graded warning mechanism.

[0033] The collision time prediction unit is used to predict the future trajectory of the blind spot target based on the motion trajectory of the blind spot target and the vehicle status; and output the collision time prediction and the feasible collision avoidance area.

[0034] The feedback optimization unit is used to record the driver's response delay and collision avoidance success rate data for each warning, and dynamically adjust the weight to optimize the output results of the personalized risk index warning.

[0035] Beneficial Effects: A personalized risk assessment unit constructs driver operating patterns, dynamically quantifying individual risk indices such as distraction frequency and operating habits. A dynamic hierarchical decision unit generates graded warning thresholds tailored to the driver's characteristics, and uses scenario semantics to pre-activate blind spot scanning. A collision time prediction unit integrates the target trajectory and ego vehicle state to calculate the collision avoidance time window and feasible region. A feedback optimization unit uses actual response data to close-loop adjust model parameters. Deep learning transforms driver habits into dynamic protection thresholds, mitigating the universality limitations of traditional fixed warning standards. Reinforcement learning mechanisms enable continuous system evolution and optimization, resulting in an active protection system that integrates individual adaptation, scenario prediction, and self-evolution.

[0036] Furthermore, the linkage execution module includes the following units:

[0037] The acoustic feedback unit is used to provide multi-directional forecasts of the graded warning mechanism through beamforming speakers; and to generate specific voice instructions based on the type of graded warning mechanism.

[0038] The optical feedback unit is used to generate different optical feedback strategies for the graded warning mechanism; and the graded warning mechanism is projected onto different areas using a light director.

[0039] The tactile feedback unit is used to provide different feedback strengths for the graded warning mechanism through the tactile feedback mechanism, and prompt the driver with risk warning signals for blind spot monitoring.

[0040] Beneficial Effects: Directional voice commands delivered by the acoustic feedback unit reduce directional misjudgment, while the optical feedback unit uses graded, regionalized projection to enhance spatial perception. The tactile feedback unit uses graded force to match the urgency of the risk, achieving multimodal collaborative guidance. By translating abstract warnings into behavioral guidance, with acoustic commands clarifying the direction to avoid the risk, optical projection identifying the spatial location of the risk, and tactile feedback driving physical responses based on urgency, drivers can quickly understand the nature of the risk and avoidance path in complex scenarios, improving the operability of warning information and the efficiency of risk avoidance, and enhancing the safety protection logic of human-machine interaction.

[0041] Furthermore, the tactile feedback mechanism includes a mounting bracket symmetrically embedded within the driver's seat and an airbag symmetrically sleeved onto the outer wall of the steering wheel. The inner walls of the mounting brackets are slidably fitted with a vibration plate. The bottoms of the vibration plates are fixedly connected to a plurality of tension springs, the bottom ends of which are fixedly connected to the inner bottom wall of the mounting brackets. The bottoms of the vibration plates are fixedly connected to sleeves, the inner walls of which are slidably fitted with a push rod. The mounting brackets are internally provided with a drive assembly for reciprocating the push rod and an inflation assembly for inflating the airbags.

[0042] Beneficial Effects: The combined design of a seat vibration plate and steering wheel airbag provides graded physical guidance. The vibration plate, buffered by tension springs, transmits high-frequency microvibrations to indicate common risks, while the airbag inflation generates a strong thrust to warn of emergencies. By synergizing spatially directional guidance with graded force, seat vibrations are dampened by tension springs to ensure comfortable microvibration, while steering wheel airbag inflation clearly indicates the location of risk. This dual-channel feedback reduces the confusion of a single tactile sense while enabling strong intervention in emergencies through mechanical structure, enhancing the driver's intuitive judgment of the direction and level of risk while reducing the possibility of accidental touch interference.

[0043] The drive assembly further includes a controller and a dual output member fixedly connected to the inner bottom wall of the fixed base. The controller is used to control the rotation of the dual output member. The output shaft of the dual output member is coaxially fixedly connected to a first rotating wheel. A first connecting rod is eccentrically hinged to the side of the first rotating wheel away from the dual output member. The top end of the first connecting rod is hinged to the bottom of the push rod. The push rod is internally provided with a limit assembly for limiting the range of movement of the push rod.

[0044] Beneficial Effects: By coaxially linking the eccentric first rotor with dual output components, rotational motion is converted into controllable linear reciprocating motion, synergizing mechanical transmission reliability with motion precision. The eccentric articulation design leverages the principle of leverage to amplify micro-rotational displacement, enabling the physical transmission of high-frequency vibrations. This mechanical linkage solution remains stable in high-temperature and electromagnetic interference environments. Furthermore, by adjusting the eccentric connection to accommodate varying warning intensity requirements, it provides a highly robust dynamic foundation for tactile feedback.

[0045] Furthermore, the limit assembly includes an induction spring, and the controller is used to control the elastic expansion and contraction of the induction spring; a fixed rod is fixedly connected to the bottom of the vibration plate, the bottom end of the fixed rod extends to the inside of the push rod and is fixedly connected to the induction spring, and the top of the induction spring is fixedly connected to the push rod.

[0046] Beneficial Effect: Dynamically adjust the push rod's travel through the elastic expansion and contraction of the sensing spring. Through the dual optimization of adaptive cushioning and precise stroke control, the sensing spring, adjusted by the controller, adapts to varying vibration intensities in real time, buffering mechanical impact to protect the structure while also rigidly constraining displacement through elastic deformation. This design helps improve the durability of the mechanism in high-frequency vibration scenarios while ensuring that the tactile feedback force precisely matches the warning level, providing clear physical guidance to the driver.

[0047] Furthermore, the inflation component includes a piston cylinder fixedly connected to the inner bottom wall of the fixed seat, and the inner wall of the piston cylinder is slidably fitted with a piston plate; a piston rod is fixedly connected to the piston plate, and a delivery pipe is connected to the side of the piston cylinder away from the piston rod; the end of the delivery pipe away from the piston cylinder is connected to the interior of the airbag; a transmission component for driving the piston rod to move back and forth is provided on the dual output parts.

[0048] Beneficial Effects: The piston-cylinder structure enables directional gas delivery, with the transmission assembly driving the reciprocating motion of the piston rod. Through an efficient and reliable pneumatic cycle, the piston plate's sealed sliding motion ensures zero gas leakage. Leveraging the kinetic energy of the existing dual output components, the airbag can be rapidly inflated and deflated without the need for an additional air pump. This mechanical pneumatic solution operates stably in vibration, high- and low-temperature environments, and precisely adjusts the inflation volume through stroke control, providing physical thrust for emergency tactile feedback.

[0049] Furthermore, the transmission assembly includes a second wheel coaxially fixedly connected to the end of the dual output member away from the first wheel, a second connecting rod is eccentrically hinged on the side of the second wheel away from the dual output member, and the end of the second connecting rod away from the second wheel is hinged to the piston rod.

[0050] Beneficial Effects: The eccentric articulation of the coaxial second rotor achieves efficient kinetic energy conversion: The second rotor is linked to the dual output member, and the eccentrically articulated second connecting rod converts the rotational motion into linear reciprocating motion of the piston rod. The coaxial design reduces multi-stage transmission energy loss and ensures synchronous operation of the inflation and vibration mechanisms. The eccentric articulation structure matches the piston stroke with the warning intensity. This purely mechanical transmission solution is stable in high temperature and vibration environments, providing power support for airbag inflation and deflation.

[0051] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a structural block diagram of the intelligent vehicle-mounted blind spot warning device based on multimodal sensor fusion of the present invention.

[0053] Figure 2 for Figure 1 Figure 1. Structural block diagram of the driver perception module connections.

[0054] Figure 3 for Figure 1 Structural block diagram of the connection of the environment recognition module.

[0055] Figure 4 for Figure 1 Structural block diagram of the connection between the early warning decision modules.

[0056] Figure 5for Figure 1 Structural diagram of the connection between the linkage execution modules.

[0057] Figure 6 for Figure 5 Cross-sectional view of the tactile feedback mechanism.

[0058] Figure 7 for Figure 6 Axonometric drawing of the installation of the drive assembly.

[0059] Figure 8 for Figure 6 Axonometric drawing of the installation of the transmission assembly.

[0060] Figure 9 for Figure 6 Cross-sectional view of the inflatable component.

[0061] The figure marks in the drawings of the specification include: 1. fixed seat; 2. vibration plate; 3. tension spring; 4. sleeve; 5. push rod; 6. dual output member; 7. first rotating wheel; 8. first connecting rod; 9. induction spring; 10. fixed rod; 11. piston cylinder; 12. piston plate; 13. piston rod; 14. second rotating wheel; 15. second connecting rod. DETAILED DESCRIPTION

[0062] The following is further described in detail through specific implementation methods:

[0063] Example 1:

[0064] As attached Figure 1 As shown: An intelligent vehicle-mounted blind spot warning device based on multimodal sensor fusion includes a sensor array module for collecting information, a driver perception module for obtaining driving habit information, an environment recognition module for obtaining environmental parameters, a warning decision module for providing warnings, and a linkage execution module for generating feedback.

[0065] The following is a detailed introduction to the functions of each module:

[0066] The sensor array module is used to connect with the vehicle's onboard system to obtain onboard data related to the driver and the driving environment. The onboard system mainly connects to the millimeter-wave radar, lidar, external camera, internal camera (wide-angle camera with 120° FOV), ultrasonic probe (40kHz), infrared sensor (8-14μm band), inertial sensor, and piezoelectric film array on the steering wheel; the collected information is transmitted to the driver perception module and environmental recognition module. In this embodiment, the millimeter-wave radar is used to detect moving targets (operating in the temperature range of -30℃ to 85℃) with a maximum detection range of 80m; the infrared sensor is used to detect temperature and life in rain, fog, or darkness (triggering with a temperature difference of ≥0.5℃); and the ultrasonic probe is used to detect static obstacles at close range.

[0067] The driver perception module is used to connect with the vehicle system to analyze the driver's concentration information (driver's eye movement direction, blinking frequency and head posture) and steering wheel grip distribution information (grip position, pressure intensity and grip duration), and collect vehicle steering information in real time (turn signal activation status, gear shift signal and accelerator and brake pedal depth); generate the driver's driving habit information based on concentration information, grip distribution information and vehicle steering information, and transmit the driving habit information to the early warning decision module.

[0068] Specific, combined Figure 2 As shown, the driver perception module includes a driving information collection unit, a multi-source data fusion unit and a driving habit modeling unit.

[0069] The driving information collection unit is used to connect to the vehicle system and use the in-vehicle camera (located on the steering column of the steering wheel or the central rearview mirror) to obtain real-time concentration information such as the driver's eye movement direction, blinking frequency and head posture (for example, when holding the steering wheel with one hand and the pressure is less than 5N, it is marked as "distracted state"); monitor the steering wheel holding position, pressure intensity and grip force distribution information of the holding time through the piezoelectric film array (capacitive sensing technology); and obtain the turn signal activation status, gear switching signal and vehicle steering information such as the depth of the accelerator and brake pedal by reading the CAN bus data (such as the turn signal is on to the left, combined with the left blind spot risk upgrade alarm).

[0070] The multi-source data fusion unit is used to align the concentration information, grip force distribution information, and vehicle steering information by timestamp, and fuse them to generate a driving behavior intention prediction model. In this embodiment, the visual data (eye focus), tactile data (grip force distribution), and vehicle signals (turn signals) are aligned by timestamp to reduce the timing deviation of each sensor. By fusing the three inputs, a driving behavior label is generated. If the turn signal is detected to be activated and the eyes are looking at the rearview mirror, it is marked as "active lane change intention"; if the turn signal is detected to be inactive and the eyes are frequently scanning the blind spot, it is marked as "potential lane change demand."

[0071] The driving habit modeling unit uses statistical data from long-term vehicle steering to generate a driver's lane-changing habits (e.g., conservative or aggressive) and concentration decay curves. A risk association model associated with driving habits is established based on a pre-recorded driving behavior intention prediction model. In this embodiment, the risk association model is built using an LSTM network. This model lowers the concentration warning threshold (TTC threshold by 30%) for aggressive drivers and extends the turn signal and action response time window for conservative drivers.

[0072] The environmental recognition module is used to connect with the vehicle system, collect positioning data, high-precision map data and rainfall sensor data, and identify driving scenarios (such as tunnels, bridges or urban roads) and environmental information of severe weather in real time; build an environmental feature assessment model based on environmental information, and use the environmental feature assessment model to assign monitoring weights to the sensor array module (such as the radar weight in heavy rain weather is adjusted to 0.8, the visual weight is adjusted to 0.1, and the infrared weight is adjusted to 0.7), output the environmental recognition results, and transmit the environmental recognition results to the early warning decision module.

[0073] Specific, combined Figure 3 As shown, the environment recognition module includes an environment data integration unit, an environment feature extraction unit, a weight dynamic allocation unit, a scene semantic modeling unit and a cross-validation and output unit.

[0074] The environmental data integration unit connects to the vehicle's onboard systems to obtain real-time information about the vehicle's location, driving direction, and speed. It also uses the onboard system's network to obtain high-precision map data (such as OpenDRIVE format) and extract driving scenario information such as the coordinates of tunnel and bridge entrances and exits, the hierarchical structure of overpasses, and the locations of traffic signs. It also dynamically matches the vehicle's current location with key map nodes (such as areas with weak GPS signals within tunnels). Precipitation intensity is obtained in real time through the onboard system's network, and the onboard network system's infrared sensors are used to identify road surface waterlogging and icing.

[0075] The environmental feature extraction unit is used to match tunnel coordinates through vehicle positioning, analyze lighting (such as strong light at the tunnel entrance and dimness inside) in combination with the on-board system's external camera images, and use the on-board system's high-precision map data and lidar point cloud to detect overlapping multi-layer road structures; trigger weather warnings based on precipitation intensity (>5mm / h); and construct environmental feature vectors (such as positioning credibility, map semantic weight, or meteorological influence coefficient).

[0076] The dynamic weight allocation unit assigns weights to different sensors based on different driving scenarios and environmental recognition information. It also calculates the reliability of each sensor using a Bayesian network. For example, when rainfall exceeds 10 mm / h, the camera's reliability decreases by 50%, and the millimeter-wave radar's weight is automatically increased. A generative adversarial network (GAN) compensates for this information gap and fills in gaps. For example, if a camera fails, a GAN is used to generate a virtual image with contours from the radar point cloud.

[0077] The scene semantic modeling unit is used to integrate high-precision map data (such as tunnel length) with real-time sensor data (such as the location of water accumulation in the tunnel), generate an environmental network model, and output key semantic label warnings (tunnel slippery areas, overpass merging conflict points, and low visibility warning areas).

[0078] The cross-validation and output unit verifies tunnel boundaries in HD map data using LiDAR point clouds, filtering out temporary structural changes caused by construction. It also compares the vehicle's millimeter-wave radar target trajectory with the lane centerlines preset in the HD map data to detect the risk of illegal lane changes. This embodiment aggregates environmental recognition results and outputs them in structured data packets, transmitting them in real time to the decision-making system to trigger sensor mode switching (e.g., disabling the camera's HDR mode in a tunnel).

[0079] The warning decision module is used to generate graded thresholds for blind spot warning monitoring based on driving habit information (1.2s for aggressive type or 2.5s for conservative type) and environmental identification information; it uses a deep learning algorithm to establish a personalized warning monitoring model to divide the driver's dynamic risk level during driving; and it combines the high-precision map data data of the on-board system to identify the scene and link the vehicle dynamics model to predict the collision time TTC for blind spot monitoring.

[0080] Specific, combined Figure 4 As shown, the warning decision module includes a personalized risk assessment unit, a dynamic hierarchical decision unit, a collision time prediction unit and a feedback optimization unit.

[0081] The personalized risk assessment unit is used to analyze the individualized risk index based on a driving behavior intention model (based on visual and grip data) and a risk association model (such as steering habits and historical braking data). This embodiment uses an LSTM network to construct a personalized risk field model and dynamically calculate three types of warning thresholds. For conservative drivers, the warning response window is extended (e.g., triggering with a TTC of less than 2.5 seconds); for aggressive drivers, the warning window is compressed (triggering with a TTC of less than 1.0 seconds), and the tactile feedback intensity is enhanced.

[0082] The dynamic grading decision unit combines grading thresholds with a personalized risk index to generate dynamic risk level warning lines. In this embodiment, the dynamic risk level warning lines are divided into three levels: Level 1 has a TTC range of 2.0-3.0 seconds, Level 2 has a TTC range of 1.0-2.0 seconds, and Level 3 has a TTC range of less than 1.0 seconds. When identifying special scenarios (such as interchange merge areas), the lidar is activated 200 meters in advance to scan the movement trajectory of targets in blind spots. The expected safe distance between vehicles is calculated in real time, triggering the graded warning mechanism.

[0083] The collision time prediction unit predicts the future trajectory of targets in the blind spot based on their trajectory (millimeter-wave radar point cloud) and the vehicle's state (steering wheel angle and acceleration). It integrates high-precision map road topology (such as curve curvature) to correct prediction errors and outputs a collision time prediction and a feasible collision avoidance area. When TTC exceeds 1.0 seconds, a secondary warning is triggered, with audio and visual guidance of the avoidance path. When TTC is ≤ 0.8 seconds, the Electronic Stability Program (ESP) is activated to implement steering torque assistance.

[0084] The feedback optimization unit records the driver's response delay and collision avoidance success rate for each warning. Using a reinforcement learning (PPO) algorithm, it dynamically adjusts the weights to optimize the output of personalized risk index warnings. For example, after a successful collision avoidance, the threshold for similar scenarios is lowered by 5%, while after three false alarms, the threshold is raised by 10%.

[0085] The linkage execution module is used to integrate beamforming speakers, light directors and tactile feedback mechanisms. It predicts the warning feedback information of the graded threshold according to the dynamic risk level warning line and collision time, and uses beamforming speakers (sound source positioning error ≤ 5°), light directors and tactile feedback mechanisms (steering wheel and seat tactile feedback linkage) to predict the warning feedback information in a graded manner.

[0086] Specific, combined Figure 5 As shown, the linkage execution module includes an acoustic feedback unit, an optical feedback unit and a tactile feedback unit.

[0087] The acoustic feedback unit uses a beamforming speaker to implement a multi-directional warning mechanism (e.g., a warning signal for a left rear risk is emitted from the left side of the driver's seat). In this embodiment, a 75-80dB directional sound field is used, and digital beamforming technology is used to confine the sound propagation within a 30° cone angle. Specific voice commands (e.g., "Pedestrian approaching from the right rear, please slow down") are generated based on the type of warning feedback.

[0088] The optical feedback unit is used to generate different optical feedback strategies for a graded warning mechanism based on the warning feedback information; the graded warning mechanism is projected onto different areas. In this embodiment, this is projected via a light guide. For example, at level one, the A-pillar LED light strip flashes slowly, at level two, a red light curtain is projected onto the windshield, and at level three, red and blue strobe lights (3Hz) and a ground-based laser strip are used.

[0089] The tactile feedback unit is used to provide different levels of feedback intensity based on the graded warning mechanism, prompting the driver with a blind spot monitoring risk warning signal. In this embodiment, the tactile feedback mechanism vibrates the steering wheel and seat on one side when TTC is ≤ 3 seconds, such as when the left blind spot is detected. When TTC is ≤ 1 second, high-frequency vibration is applied to both sides.

[0090] Example 2:

[0091] As attached Figure 6As shown, the difference from the above embodiment is that the tactile feedback mechanism includes a fixed base 1 symmetrically embedded and installed inside the driver's seat and an airbag symmetrically sleeved on the outer wall of the steering wheel. The inner wall of the fixed base 1 is slidably fitted with a vibration plate 2; the bottom of the vibration plate 2 is screw-fixed with a plurality of tension springs 3, and the bottom ends of the tension springs 3 are screw-fixed with the inner bottom wall of the fixed base 1; the bottom of the vibration plate 2 is integrally formed with a sleeve 4, and the inner wall of the sleeve 4 is slidably fitted with a push rod 5. The interior of the fixed base 1 is equipped with a drive assembly for driving the push rod 5 to move back and forth, and an inflation assembly for inflating the airbag.

[0092] Combine Figure 7 As shown, the drive assembly includes a controller and a dual output member 6 screw-fastened to the inner bottom wall of the fixing base 1. In this embodiment, the dual output member 6 is a double-headed motor, and the controller is used to control the rotation of the dual output member 6. The output shaft of the dual output member 6 is coaxially connected to a first rotating wheel 7 via a coupling. A first connecting rod 8 is eccentrically hinged to the side of the first rotating wheel 7 away from the dual output member 6. The top end of the first connecting rod 8 is hinged to the bottom of the push rod 5. The push rod 5 is internally provided with a limit assembly to limit the range of movement of the push rod 5.

[0093] The limiting assembly includes an induction spring 9, and the controller is used to control the elastic expansion and contraction amount of the induction spring 9; the bottom screw of the vibration plate 2 has a fixing rod 10, the bottom end of the fixing rod 10 extends to the inside of the pushing rod 5 and is fixedly connected with the induction spring 9 by the screw, and the top end of the induction spring 9 is fixedly connected with the pushing rod 5 by the screw.

[0094] Specifically, the controller controls the rotation of the dual output member 6, which coaxially drives the first rotating wheel 7 to rotate synchronously. The eccentric hinge point of the first rotating wheel 7 generates circular motion as it rotates, driving the first connecting rod 8 to swing back and forth. The first connecting rod 8 converts the circular motion into linear motion, driving the push rod 5 to slide up and down along the inner wall of the sleeve 4.

[0095] When the push rod 5 rises, the fixed rod 10 moves upward with the vibration plate 2, stretching the induction spring 9. The controller adjusts the spring's expansion and contraction in real time (for example, compressing the spring to 80% deformation during strong vibrations increases the travel of the push rod 5). When the push rod 5 descends, the induction spring 9 releases its elastic potential energy, assisting the push rod 5 in quickly returning to its original position. The reciprocating motion of the push rod 5 strikes the vibration plate 2, buffering the high-frequency microvibrations transmitted by the tension spring 3. The vibration plate 2 transfers this mechanical energy to the seat surface, creating a graded tactile cue.

[0096] Combine Figure 9 As shown, the inflation assembly includes a piston cylinder 11 fixedly connected to the inner bottom wall of the fixing seat 1 with screws, and the inner wall of the piston cylinder 11 is slidably fitted with a piston plate 12; a piston rod 13 is fixedly bonded to the piston plate 12, and a delivery pipe is connected to the side of the piston cylinder 11 away from the piston rod 13; the end of the delivery pipe away from the piston cylinder 11 is connected to the interior of the airbag; a transmission assembly for driving the piston rod 13 to move back and forth is provided on the dual output member 6.

[0097] Combine Figure 8 As shown, the transmission assembly includes a second rotor 14 coaxially fixedly connected to the end of the dual output member 6 away from the first rotor 7 through a coupling, and a second connecting rod 15 is eccentrically hinged on the side of the second rotor 14 away from the dual output member 6, and the end of the second connecting rod 15 away from the second rotor 14 is hinged to the piston rod 13.

[0098] Specifically, the controller receives a warning signal (such as the collision time TTC value) and controls the rotation speed of the dual output member 6. For example, during a low-frequency warning, the rotation speed is set to 200 rpm to generate 1-2 Hz inflation; during a high-frequency emergency, the rotation speed is set to 800 rpm to generate 4-5 Hz inflation. The dual output member 6 drives the second wheel 14 to rotate at the same speed, and the eccentric point on the second wheel 14 pushes the second connecting rod 15 to swing back and forth. The second connecting rod 15 converts the swing into a linear reciprocating motion of the piston rod 13, squeezing the piston plate 12 to slide outside the cylinder, and the compressed gas is input into the airbag through the delivery pipe; the piston plate 12 slides in the opposite direction, and a negative pressure is formed in the cylinder to draw the gas into the inside of the piston cylinder 11; the steering wheel tactile dynamic feedback is realized in conjunction with the seat vibration. In this embodiment, a portion of the gas is preferentially filled inside the airbag.

[0099] The specific implementation process is as follows:

[0100] First, the controller receives the warning signal (such as TTC value) and controls the rotation of the double-headed motor (the speed is adjusted according to the risk level: 200rpm corresponds to low-frequency warning, and 800rpm corresponds to high-frequency emergency); the double-headed motor drives the first wheel 7 to rotate coaxially through the coupling. The eccentric hinge point of the first wheel 7 pushes the first connecting rod 8 to swing back and forth, and the first connecting rod 8 converts the swing into vertical linear motion of the push rod 5. The push rod 5 rises and pushes the vibration plate 2 upward, stretching the induction spring 9. The push rod 5 descends and the induction spring 9 releases potential energy, assisting the push rod 5 to quickly reset. The push rod 5 hits the vibration plate 2, and the high-frequency micro-vibration is transmitted to the seat surface through the tension spring 3 for buffering.

[0101] The dual-headed motor synchronously drives the second rotor 14 to rotate. The eccentric hinge point of the second rotor 14 pushes the second connecting rod 15 to swing, converting it into linear reciprocating motion of the piston rod 13. The piston pushes the compressed gas, which is fed into the airbag through the delivery tube. The piston plate 12 slides in the opposite direction, creating negative pressure, which draws the gas into the piston cylinder 11.

[0102] This embodiment uses seat vibration (tension spring 3 buffering) and steering wheel airbag (pneumatic pulse) to ensure that the physical feedback direction corresponds to the risk direction (such as left risk triggers left-side feedback); by adjusting the speed to correspond to the inflation frequency and vibration frequency, the three-level mapping of feedback strength is adapted to match the TTC risk level, thereby providing a blind spot warning effect.

[0103] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will readily appreciate that other variations or modifications based on the above descriptions are possible. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. An intelligent vehicle-mounted blind spot warning device based on multimodal sensor fusion, comprising a sensor array module, a driver perception module, and an environment recognition module. The sensor array module is used to connect with the vehicle system to obtain vehicle-mounted data related to the driver and the driving environment. The driver perception module is used to analyze the driver's driving habits based on the vehicle-mounted data. The environment recognition module is used to analyze the vehicle's environment recognition information based on the vehicle-mounted data. The device is characterized by: Also includes the following modules: The warning decision module generates thresholds for blind spot warning monitoring based on driving habits and environmental information. It uses deep learning algorithms to establish a personalized warning monitoring model, classifying the driver's dynamic risk level warning lines during driving. It also uses high-precision map data from the vehicle system to identify scenarios and link with the vehicle dynamics model to predict collision times for blind spot monitoring. The linkage execution module is used to integrate the beamforming speaker, light director and tactile feedback mechanism to forecast the warning feedback information of the classification threshold according to the dynamic risk level warning line and collision time.

2. The intelligent vehicle-mounted blind spot warning device based on multimodal sensor fusion according to claim 1 is characterized in that: The driver perception module includes the following units: A driving information collection unit, configured to connect to an onboard system connected to an in-vehicle camera and a piezoelectric film array disposed on the steering wheel; Then, based on the in-car camera, the driver's eye movement direction, blinking frequency and head posture concentration information are obtained in real time; The piezoelectric film array is used to monitor the grip position, pressure intensity and grip duration of the steering wheel; and obtaining vehicle steering information such as turn signal activation status, gear shift signals, and accelerator and brake pedal depth through the vehicle system; A multi-source data fusion unit is used to align concentration information, grip force distribution information, and vehicle steering information by timestamp and fuse them to generate a driving behavior intention prediction model; Driving habit modeling unit, used to collect statistics on long-term vehicle steering information to form a driver's lane change habits and concentration decay curve; A risk association model associated with driving habits is established based on the pre-recorded driving behavior intention prediction model.

3. The intelligent vehicle-mounted blind spot warning device based on multimodal sensor fusion according to claim 2 is characterized in that: The environment recognition module includes the following units: The environmental data integration unit is used to connect to the vehicle system to obtain real-time vehicle positioning, driving direction, and speed information. It also uses the vehicle system network to obtain high-precision map data and extract driving scenario information such as the coordinates of tunnel and bridge entrances and exits, the hierarchical structure of overpasses, and the location of traffic signs. It also uses the vehicle system network to obtain real-time precipitation intensity and then uses the vehicle system's infrared sensor to identify road surface waterlogging and ice formations. The environmental feature extraction unit is used to match tunnel coordinates through vehicle positioning, analyze lighting using images from the vehicle's external camera system, and use the vehicle's high-precision map data and lidar to detect overlapping multi-layer road structures; trigger weather warnings based on precipitation intensity; and construct environmental feature vectors. A dynamic weight allocation unit assigns acquisition weights to different sensors based on different driving scenarios and environmental recognition information. It also calculates the reliability of each sensor based on a Bayesian network and uses a generative adversarial network to compensate for the acquired information and fill in information gaps. The scene semantic modeling unit is used to integrate high-precision map data with real-time sensor data, generate an environmental network model, and output key semantic label warnings; The cross-validation and output unit is used to verify the tunnel boundaries in the HD map data through the lidar point cloud and filter out temporary structural changes caused by construction. It also compares the millimeter-wave radar target trajectory of the vehicle system with the preset lane centerline in the HD map data to detect the risk of illegal lane changes by vehicles.

4. The intelligent vehicle-mounted blind spot warning device based on multimodal sensor fusion according to claim 3 is characterized in that: The early warning decision module includes the following units: Personalized risk assessment unit, used to predict the driving behavior intention model and risk association model; using deep learning algorithm to analyze the individualized risk index; Dynamic grading decision-making unit, used to combine grading thresholds with personalized risk indexes to generate dynamic risk level warning lines; When identifying special scenarios, the vehicle system's LiDAR system is activated in advance to scan the target's movement trajectory in the blind spot; the vehicle's expected safety distance is calculated in real time, triggering a graded warning mechanism; The collision time prediction unit is used to predict the future trajectory of the target in the blind spot based on its motion trajectory and the vehicle's state; Output collision time prediction and collision avoidance feasible area; The feedback optimization unit is used to record the driver's response delay and collision avoidance success rate data for each warning, and dynamically adjust the weight to optimize the output results of the personalized risk index warning.

5. The intelligent vehicle-mounted blind spot warning device based on multimodal sensor fusion according to claim 4 is characterized in that: The linkage execution module includes the following units: An acoustic feedback unit, configured to provide multi-directional forecasts of the graded warning mechanism through a beamforming speaker; and to generate specific voice commands based on the type of graded warning mechanism; An optical feedback unit is used to generate different optical feedback strategies for a graded warning mechanism; the graded warning mechanism is projected onto different areas using a light director; The tactile feedback unit is used to provide different feedback strengths for the graded warning mechanism through the tactile feedback mechanism, and prompt the driver with risk warning signals for blind spot monitoring.

6. The intelligent vehicle-mounted blind spot warning device based on multimodal sensor fusion according to claim 5 is characterized in that: The tactile feedback mechanism comprises a fixing seat (1) symmetrically embedded and installed inside the driver's seat and an airbag symmetrically sleeved on the outer wall of the steering wheel; the inner side wall of the fixing seat (1) is slidably matched with a vibration plate (2); the bottom of the vibration plate (2) is fixedly connected with a plurality of tension springs (3), and the bottom ends of the tension springs (3) are fixedly connected to the inner bottom wall of the fixing seat (1); the bottom of the vibration plate (2) is fixedly connected with a sleeve (4), and the inner wall of the sleeve (4) is slidably matched with a push rod (5); A driving component for driving the push rod (5) to move back and forth and an inflation component for inflating the airbag are provided inside the fixing seat (1).

7. The intelligent vehicle-mounted blind spot warning device based on multimodal sensor fusion according to claim 6, characterized in that: The driving assembly comprises a controller and a dual output member (6) fixedly connected to the inner bottom wall of the fixing seat (1), the controller being used to control the dual output member (6) to rotate; the output shaft of the dual output member (6) is coaxially fixedly connected to a first rotating wheel (7), a first connecting rod (8) is eccentrically hinged on a side of the first rotating wheel (7) away from the dual output member (6); the top end of the first connecting rod (8) is hinged to the bottom end of the push rod (5); A limiting assembly for limiting the movement range of the pushing rod (5) is provided inside the pushing rod (5).

8. The intelligent vehicle-mounted blind spot warning device based on multimodal sensor fusion according to claim 7 is characterized in that: The limiting component includes an induction spring (9), and the controller is used to control the elastic expansion and contraction amount of the induction spring (9); a fixing rod (10) is fixedly connected to the bottom of the vibration plate (2), the bottom end of the fixing rod (10) extends to the inside of the push rod (5) and is fixedly connected to the induction spring (9), and the top end of the induction spring (9) is fixedly connected to the push rod (5).

9. The intelligent vehicle-mounted blind spot warning device based on multimodal sensor fusion according to claim 8, characterized in that: The inflation assembly comprises a piston cylinder (11) fixedly connected to the inner bottom wall of the fixing seat (1); the inner wall of the piston cylinder (11) is slidably fitted with a piston plate (12); a piston rod (13) is fixedly connected to the piston plate (12); a side of the piston cylinder (11) away from the piston rod (13) is connected to a delivery pipe; an end of the delivery pipe away from the piston cylinder (11) is connected to the interior of the airbag; The dual output member (6) is provided with a transmission assembly for driving the piston rod (13) to move back and forth.

10. The intelligent vehicle-mounted blind spot warning device based on multimodal sensor fusion according to claim 9, characterized in that: The transmission assembly comprises a second rotating wheel (14) coaxially fixedly connected to an end of the dual output member (6) away from the first rotating wheel (7); a second connecting rod (15) is eccentrically hinged to a side of the second rotating wheel (14) away from the dual output member (6); and an end of the second connecting rod (15) away from the second rotating wheel (14) is hinged to the piston rod (13).

Images