Digital vehicle lamp vehicle-road interaction system based on multi-mode perception fusion

Abstract

The invention discloses a digital vehicle lamp vehicle-road interaction system based on multi-modal sensing fusion, and relates to the technical field of intelligent driving and vehicle control, a roadside collaborative sensing unit is used for acquiring and processing multi-modal data of a roadside sensor, and generating a local dynamic map containing the position, speed and motion trail of a traffic participant; the vehicle-mounted multi-mode sensing unit is used for acquiring multi-mode data of a sensor of a vehicle and outputting vehicle-mounted sensing data; and the conflict detection and arbitration triggering unit is used for fusing the local dynamic map and the vehicle-mounted sensing data so as to identify traffic participants and predict passing intentions of the traffic participants. According to the invention, by introducing an intention uncertainty evaluation and dynamic negotiation mechanism, the problems of static and rigid interaction modes in the prior art are effectively solved. A traditional scheme cannot distinguish subtle intention differences of traffic participants and often causes invalid projection or communication loss, but the system can actively recognize conflict scenes with fuzzy intentions and trigger intelligent arbitration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving and vehicle control, and particularly relates to a digital vehicle lamp vehicle-road interaction system based on multi-modal perception fusion. BACKGROUND

[0002] With the development of intelligent and networked vehicle technology, a technical solution has been proposed to perceive the environment using vehicle-mounted or roadside sensors and project information to the outside world through vehicle lamps for interaction. A typical implementation idea of this kind of prior art can be summarized as a perception-response mode: after detecting specific traffic participants (such as pedestrians, non-motor vehicles) or events (such as congestion ahead) through sensors such as cameras and radars, the system triggers a pre-set, fixed interaction instruction to control the intelligent vehicle lamp to project corresponding symbols (such as warning icons, guide arrows) to the ground for warning or guidance.

[0003] However, this kind of technical route based on simple rule triggering has a fundamental limitation: its interaction is static and passive, lacking the ability to understand and negotiate the dynamic and ambiguous intentions of traffic participants. This leads to misjudgment of intentions in complex real traffic scenarios, especially at intersections without signal control, pedestrian and vehicle mixed areas, etc., causing ineffective interaction and even safety risks.

[0004] Specifically, the defects of the prior art are fully exposed in the following typical scenarios: when a vehicle equipped with intelligent interaction vehicle lamps approaches a zebra crossing without a signal light, the existing system may only detect that there is a pedestrian on the roadside through visual recognition, and mechanically trigger the light language of giving way to the zebra crossing for projection. However, the real intention of the pedestrian can be diverse - it may be preparing to cross the street, or it may just be standing and waiting or watching. The system cannot distinguish these subtle differences in intention. If the pedestrian has no intention to cross the street, the vehicle's courtesy projection will cause confusion, interfere with the judgment of other traffic participants, and reduce traffic efficiency; if the system chooses not to project due to ambiguous intentions, it may miss the key communication opportunity when the pedestrian really needs to cross the street, and create a safety hazard. The core problem is that the existing solution simplifies complex traffic interaction into a binary judgment of the presence or absence of a target, and completely fails to handle the ubiquitous intermediate state of uncertain target intention. SUMMARY

[0005] The purpose of the present application is to provide a digital vehicle lamp vehicle-road interaction system based on multi-modal perception fusion to solve the problems in the background art.

[0006] In order to achieve the above purpose, the present application provides the following technical solution: a digital vehicle lamp vehicle-road interaction system based on multi-modal perception fusion, comprising: The roadside cooperative sensing unit is used to acquire and process multimodal data from roadside sensors to generate a local dynamic map containing the location, speed, and trajectory of traffic participants. The vehicle-mounted multimodal perception unit is used to acquire multimodal data from the vehicle's own sensors and output vehicle-mounted perception data. The conflict detection and arbitration triggering unit is communicatively connected to the roadside cooperative perception unit and the vehicle-mounted multimodal perception unit. It is used to fuse the local dynamic map and the vehicle-mounted perception data to identify traffic participants and predict their travel intentions. When a conflict is detected between the predicted paths of at least two traffic participants and their intention certainty is lower than a preset threshold, an arbitration triggering command is generated. An uncertainty arbitration decision unit, connected to the conflict detection and arbitration triggering unit, is used to initiate an uncertainty arbitration algorithm based on a dynamic weight model and generate an interaction scheme after receiving the arbitration triggering instruction. The digital vehicle lighting interaction execution unit is connected to the uncertainty arbitration decision unit and is used to receive the interaction scheme and control the pixelated vehicle lights to convert the interaction scheme into a dynamic light projection pattern and project it onto the external environment.

[0007] In a preferred embodiment, the uncertainty arbitration decision unit includes: An uncertainty arbitration module is used to calculate and generate at least two alternative tentative passage sequences based on the dynamic weight model after receiving the arbitration trigger instruction, wherein the dynamic weight model integrates the weight coefficients by weighted summation. The interactive feedback learning module is used to update the dynamic weight model and optimize the interaction scheme based on the real-time behavioral feedback of external traffic participants after the digital vehicle light interaction execution unit projects a light projection pattern corresponding to the tentative passage sequence.

[0008] In a preferred embodiment, the uncertainty arbitration decision unit includes: An uncertainty arbitration module is used to calculate and generate at least two alternative tentative passage sequences based on the dynamic weight model after receiving the arbitration trigger instruction, wherein the dynamic weight model integrates the weight coefficients by weighted summation. The interactive feedback learning module is used to update the dynamic weight model and optimize the interaction scheme based on the real-time behavioral feedback of external traffic participants after the digital vehicle light interaction execution unit projects a light projection pattern corresponding to the tentative passage sequence.

[0009] In a preferred embodiment, the dynamic weight model W The following weighting coefficients are combined using the formula: W =α· + β· + γ· ； in, α , β , γ This is an adjustable coefficient. α + β + γ =1, the initial value can be adaptively set according to the traffic scenario: Static rule weights based on traffic regulations ; Safety risk weights based on real-time collision risk ; And collaborative feedback weights based on historical interaction behavior .

[0010] In a preferred embodiment, the interactive feedback learning module is specifically used for: The roadside cooperative sensing unit or the vehicle-mounted multimodal sensing unit monitors the response behavior of external traffic participants to the exploratory passage sequence. If the response behavior involves deceleration, stopping, or turning to avoid, it is determined as compliance or courtesy, and the corresponding traffic participant's collaborative feedback weight in the dynamic weight model is increased. If the response behavior is characterized by acceleration, not changing course, or intrusion into the conflict zone, it is determined as contention or ignoring, the security risk weight is increased, and the uncertainty arbitration module is triggered to recalculate the passage sequence. The monitoring and updating process is carried out within a preset interaction time window. The length of the interaction time window is dynamically set according to the distance of the conflict point and the average vehicle speed, and ranges from 2 to 10 seconds.

[0011] In a preferred embodiment, the digital vehicle lighting interaction execution unit is configured to execute the interaction scheme in stages: First, the pixelated headlights are controlled to project tentative light patterns corresponding to each of the tentative passage sequences in a flashing or pulsating manner, either sequentially or in sections. Subsequently, based on the final interaction scheme optimized and generated by the interactive feedback learning module, the pixelated headlights are controlled to project a deterministic guiding light pattern in a continuously lit manner.

[0012] In a preferred embodiment, the tentative light signal pattern is a flashing arrow or question mark used to solicit opinions; the definitive guiding light signal pattern is a constantly lit crosswalk, zebra crossing, or stop symbol used to provide clear instructions.

[0013] In a preferred embodiment, the system further includes: The cockpit information coordination unit is connected to the conflict detection and arbitration triggering unit and the uncertainty arbitration decision-making unit, and is used to receive the arbitration triggering command, interaction scheme and decision basis; The key traffic participants targeted by the interaction scheme and their assigned right-of-way information are rendered in real time as augmented reality labels with highlighted borders and intent icons, and precisely overlaid on the actual image location of the traffic participant, and displayed on the vehicle's augmented reality head-up display.

[0014] In a preferred embodiment, the cockpit information coordination unit is further configured to dispatch in-vehicle prompt channels according to the urgency of the interaction scheme; When the time difference between the expected arrival times of the conflicting parties at the conflict point is less than 2 seconds, it is determined to be of a high urgency level, and the red flashing warning mark on the augmented reality head-up display, the seat vibration tactile prompt, and the auditory voice warning are triggered simultaneously. When the time difference is greater than or equal to 2 seconds, it is determined to be of low urgency, and only the augmented reality annotation is triggered.

[0015] In a preferred embodiment, the intent certainty C is calculated using the following formula: C= · · · ； in, , , As a weighting factor, For trajectory prediction confidence, The degree of abnormality in motion state change. The degree of deviation from the standard traffic behavior model, + + =1, the initial value can be adaptively set according to the traffic scenario; And / or, The security risk weight The calculation incorporates the estimated time difference between the vehicle's arrival at the point of conflict and that of the other party involved. Relative velocity ΔV and current environmental visibility V or road surface adhesion coefficient μ ,in With 1 / , Δ VPositively correlated with V , μ It shows a negative correlation.

[0016] In a preferred embodiment, the uncertainty arbitration module and the interactive feedback learning module work together to implement a progressive interactive process: The tentative passage sequence is generated and projected based on the initial dynamic weight model; within a preset time window, the passage sequence is iteratively optimized based on the updated weight coefficients. When the results of the passing sequence generated by three consecutive iterations are consistent, or when the change in the weight coefficient is less than the convergence threshold of 5%, the deterministic guiding light language is output.

[0017] The technical effects and advantages provided by the present invention in the above technical solution are as follows: This invention effectively overcomes the static and rigid interaction methods of existing technologies by introducing an intent uncertainty assessment and dynamic negotiation mechanism. Traditional solutions cannot distinguish subtle differences in the intentions of traffic participants, often leading to ineffective projections or communication gaps. In contrast, this system can proactively identify conflict scenarios with ambiguous intentions and trigger intelligent arbitration.

[0018] This invention, based on a dynamic weighting model and real-time feedback learning, enables a progressive and negotiated interactive process. By first projecting tentative light signals to solicit opinions, and then optimizing decisions and projecting deterministic guidance based on behavioral feedback, the system significantly improves the accuracy, safety, and traffic efficiency of vehicle-to-infrastructure (V2I) interaction, making digital vehicle lights a true medium for collaborative decision-making and intent communication. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0020] Figure 1 This is a system block diagram of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Example 1, please refer to Figure 1As shown in this embodiment, a digital vehicle-lighting and vehicle-road interaction system based on multimodal perception fusion includes: The roadside cooperative sensing unit is used to acquire and process multimodal data from roadside sensors to generate a local dynamic map containing the location, speed, and trajectory of traffic participants. The vehicle-mounted multimodal perception unit is used to acquire multimodal data from the vehicle's own sensors and output vehicle-mounted perception data. The conflict detection and arbitration triggering unit is used to fuse the local dynamic map and the vehicle perception data to identify traffic participants and predict their travel intentions; when a conflict is detected between the predicted paths of at least two traffic participants and their intention certainty is lower than a preset threshold, an arbitration triggering command is generated. An uncertainty arbitration decision unit is used to initiate an uncertainty arbitration algorithm based on a dynamic weight model and generate an interaction scheme after receiving the arbitration trigger instruction. The digital vehicle lighting interaction execution unit is used to receive the interaction scheme and control the pixelated vehicle lights to convert the interaction scheme into a dynamic light projection pattern and project it onto the external environment.

[0023] As described above, this invention effectively overcomes the static and rigid interaction methods of existing technologies by introducing an intent uncertainty assessment and dynamic negotiation mechanism. Traditional solutions cannot distinguish subtle differences in the intentions of traffic participants, often leading to ineffective projections or communication gaps. This system, however, can proactively identify conflict scenarios with ambiguous intentions and trigger intelligent arbitration. Based on a dynamic weight model and real-time feedback learning, the system achieves a progressive and negotiated interaction process. By first projecting tentative light signals to solicit opinions, and then optimizing decisions and projecting deterministic guidance based on behavioral feedback, the system significantly improves the accuracy, safety, and traffic efficiency of vehicle-road interaction, making digital vehicle lights a true medium for collaborative decision-making and intent communication.

[0024] In one possible implementation, the roadside cooperative sensing unit is centered on an edge computing device (such as the NVIDIA Jetson AGX Orin platform), equipped with a heterogeneous sensor array, and performs data processing through a proprietary algorithm stack.

[0025] Specifically, the sensor array and data acquisition include: Visual perception layer: Deploys high-definition cameras with global shutter resolution of 8 megapixels or higher, equipped with HDR functionality, to address challenges such as backlighting at night and alternating light and dark conditions at tunnel entrances. Employs a multi-view stereo layout to achieve depth perception of the intersection area.

[0026] Radar perception layer: Equipped with 4D millimeter-wave radar, providing point cloud data including range, azimuth, velocity, and altitude information, with strong rain and fog penetration capabilities for stable target tracking. Simultaneously, a solid-state lidar (LiDAR) layer is deployed, providing ultra-high-resolution 3D point clouds for precise contour recognition and real-time comparison with static high-precision maps.

[0027] Environmental sensing layer: integrates meteorological sensors to acquire real-time information on road surface temperature, humidity, and water accumulation. μ ) and visibility ( V The data provides input for the risk model.

[0028] Timing and Positioning Layer: Built-in GNSS / IMU integrated navigation module and PTP (Precise Time Protocol) synchronization clock, giving all data a unified microsecond-level timestamp and UTM coordinates.

[0029] The data processing and fusion pipeline specifically includes: Preprocessing: Distortion removal and calibration of camera images; clustering and filtering of radar point clouds; ground segmentation and dynamic object extraction of LiDAR point clouds.

[0030] Target perception: A multi-task learning neural network (such as the Transformer-based BEVFormer model) is used to fuse the front-view camera image, the surrounding view image, and the LiDAR point cloud in the bird's-eye view (BEV) space at the feature level, and output a unified output including detection boxes, semantic categories, accurate 3D positions (Px, Py, Pz), velocity vectors (Vx, Vy) and heading angles of various traffic participants such as vehicles, pedestrians, and non-motorized vehicles.

[0031] Target tracking and trajectory prediction: Based on the perception results of consecutive frames, a deep learning-based multi-target tracking algorithm (such as ByteTrack) is used to assign a unique ID to each target and fit its smooth motion trajectory (T). The trajectory prediction module, based on social LSTM or graph neural network (GNN), comprehensively considers the interaction relationships between targets and predicts multiple possible motion trajectories and their probabilities within the next 3-5 seconds. .

[0032] Map publishing: The final generated local dynamic map is published in the form of ROS2 messages or Apache Kafka streaming data. The message structure includes timestamps, IDs of all tracked targets, real-time status (position, velocity, heading), predicted trajectory set, and environmental parameters.

[0033] In one possible implementation, the on-board unit focuses on fine-grained perception of the vehicle's near-field environment, complementing the roadside unit.

[0034] The vehicle sensor configuration specifically includes: Forward perception: Employing a combination of high-resolution telephoto and wide-angle cameras, forward-facing long-range LiDAR, and forward-facing millimeter-wave radar, it is responsible for accurate ranging and identification of the main field of view area.

[0035] Surround view perception: Surround view fisheye cameras and corner radars are deployed around the vehicle to achieve 360-degree close-range coverage without blind spots, which is specifically used to detect blind spot risks such as pedestrians suddenly appearing out of nowhere and nearby vehicles changing lanes.

[0036] Positioning and Status: Relying on the on-board high-precision positioning unit (combining GNSS, IMU and wheel speed gauge) to provide centimeter-level self-positioning, and to obtain vehicle dynamic parameters such as self-speed, turn signal status and yaw rate in real time through the CAN bus.

[0037] Data processing and feature extraction, specifically including: The data processing flow is similar to that of the roadside unit, but the model is deployed in a lightweight manner to adapt to automotive-grade computing platforms (such as NVIDIA Orin or Huawei MDC).

[0038] Key feature output: In addition to providing near-field target detection and tracking results similar to those of roadside units, this unit places particular emphasis on capturing the micro-level behavioral intentions of traffic participants, such as: Pedestrians: head direction, gait, whether they are looking at their phones.

[0039] Vehicle: wheel turning angle, whether it crosses the line, and consistency between turn signals and actual actions.

[0040] Data association: Through spatiotemporal alignment and feature matching, the targets detected by the vehicle are associated and fused with the targets in the local dynamic map published by the roadside unit to form a global target list for vehicle-road integration.

[0041] In one possible implementation, the conflict detection and arbitration triggering unit receives local dynamic maps from the roadside unit and vehicle-mounted perception data from the vehicle-mounted unit, and performs a second fusion to construct a globally consistent traffic situation. Its core function is to identify potential conflicts and assess intent certainty, specifically by executing the following steps: Step S31: Intent Prediction and Conflict Detection. Based on the fused trajectory data, predict the travel paths of all parties (such as the vehicle itself (HV), the target vehicle (RV), and the pedestrian (P)) within the next 3-5 seconds. If the predicted paths intersect in both the spatial and temporal domains (i.e., conflict points), they are marked as potential conflicts.

[0042] Step S32: Intent Determinism Calculation. For each traffic participant involved in the conflict, calculate their intent determination C using the following formula: C= · · · ； in: , , These are weighting factors, representing preferences for trajectory prediction reliability, motion anomaly sensitivity, and behavioral norm compliance, respectively. The sum of these three is 1 (i.e., ...). + + =1).

[0043] The initial values ​​can be set according to the adaptive principle of the traffic scenario: Under structural driving conditions (such as highway cruising), settings can be set. =0.7, =0.2, =0.1, indicating greater confidence in the prediction model; In complex mixed traffic conditions (such as school intersections), settings can be configured. =0.4, =0.3, =0.3, which increases sensitivity to abnormal and deviant behavior.

[0044] The confidence level for trajectory prediction (0~1) is determined by the output probability of the prediction model; The degree of abnormality in motion state is obtained by calculating the deviation of the absolute values ​​of the rate of change of acceleration and steering angle from the historical normal baseline. The C-value represents the degree of behavioral deviation, obtained by comparing the current movement pattern (such as whether it crosses the line or approaches the roadside) with the standard traffic behavior model. The lower the C-value, the more ambiguous the intention.

[0045] Step S33: Arbitration Trigger Judgment. When a conflict is detected, and the intent certainty C of at least two parties is lower than a preset threshold (e.g., set to 0.6), the current situation is determined to be a high-uncertainty conflict scenario, and an arbitration trigger instruction is generated. This instruction includes information such as the conflicting parties' IDs, predicted paths, intent certainty C values, and the coordinates of the conflict points.

[0046] Additionally, the unit also calculates the security risk weights. Some required parameters, such as the estimated time difference between the vehicle's arrival at the point of conflict and that of the other party. Relative velocity ΔV wait.

[0047] In one possible implementation, the uncertainty arbitration decision-making unit is activated upon receiving an arbitration trigger instruction. It contains an uncertainty arbitration module and an interactive feedback learning module, which work together to execute a progressive interactive process.

[0048] Step S41: Generate tentative solutions. The uncertainty arbitration module calculates and generates at least two alternative passage sequences based on the dynamic weight model.

[0049] The dynamic weight model W The expression is: W = α· + β· + γ· ； in, α , β , γ These are adjustable coefficients, representing the degree of preference for traffic rules, real-time safety risks, and collaborative feedback from traffic participants, respectively. The sum of the three is 1 (i.e., α + β + γ =1). Its initial value can be adaptively set according to the road type and scene, as shown in Table 1 below:

[0050] Table 1 in, Static rule weights assign different basic priorities to each party based on traffic regulations (such as the right-of-way rule and the rule that turning vehicles yield to straight-going vehicles).

[0051] Safety risk weights, the calculation of which takes into account time differences. Relative velocity ΔV And the ambient visibility reported by the roadside unit V or road surface adhesion coefficient μ Its value is usually related to 1 / , ΔV Positively correlated with V , μ It shows a negative correlation.

[0052] For example, a formula can be used. = *(1 / )+ * ΔV - * V Quantification , , (This is the adjustment coefficient).

[0053] Collaboration feedback weight, with an initial value of 0, will be updated during the interaction.

[0054] Based on the current model weights, the module scores different passage sequences (such as HV first, RV waiting, RV first, HV waiting) and selects the two highest-scoring sequences as trial passage sequences to be output to the vehicle light execution unit.

[0055] Step S42: Perform trial and feedback learning. After the digital headlight execution unit projects a trial light signal, the interactive feedback learning module monitors the external response through the sensing unit.

[0056] If a party's behavior involves slowing down, stopping, or swerving to avoid a collision, it is judged as compliant or courteous, and its ranking in the model is increased accordingly. Weight (e.g., increase by 0.1).

[0057] If a party's behavior involves accelerating, changing course, or intruding into the conflict zone, it will be judged as either contesting or ignoring the action, and the penalty will be increased accordingly. The weights are assigned, and the uncertainty arbitration module is immediately triggered to recalculate the passage sequence based on the new weights.

[0058] This monitoring and updating process takes place within a dynamically set interactive time window, the length of which is... L (Unit: seconds) Based on the distance from the conflict point D (meters) and average speed of each party (meters per second) setting, the calculation formula can be: L =max(2,min(D / (2*) ), 10), that is, the range is usually controlled within 2 to 10 seconds.

[0059] Step S43: Iterative Convergence and Final Decision. Within a preset time window, the system continuously iteratively updates weights and optimizes the passage sequence based on feedback. The process stops and the finalized interaction scheme is output when any of the following convergence conditions are met: Condition 1: The optimal travel sequence generated in three consecutive iterations is consistent.

[0060] Condition 2: Key weight coefficients in the dynamic weight model ( α , β , γ The change in the value is less than the convergence threshold of 5%.

[0061] Afterward, the system will project a deterministic guiding light message based on the final scheme.

[0062] In one possible implementation, the digital vehicle lighting interaction execution unit controls the vehicle's pixelated headlights or projection lights. Under the guidance of the arbitration decision unit, light signals are projected in stages. The core of this unit is pixel-level precise control and reliable environmental adaptability, specifically implemented in the following aspects: The hardware carrier specifically includes: It adopts intelligent digital headlights based on DMD (Digital Micromirror Device) or a million-level Micro-LED pixel matrix. The headlights have the ability to independently control the brightness and color of each pixel, with a projection resolution of no less than 1 million pixels and a brightness of more than 1 million candela, ensuring that the pattern remains clearly visible even under strong daylight.

[0063] The integrated adaptive optics system can adjust the projection focus and image distortion in real time according to vehicle attitude (pitch), load changes and road surface curvature, ensuring that the projected pattern is always clear and undistorted on the target road surface.

[0064] The light language control engine specifically includes: Pattern Library and Rendering: A built-in standardized light symbol library includes various symbols such as arrows, zebra crossings, stop signs, exclamation marks, and question marks. After receiving interactive scheme instructions, the rendering engine can generate corresponding dynamic lighting effects in real time.

[0065] Layered projection strategies, specifically including: Probing Phase: Control the vehicle lights to flash in a low duty cycle (e.g., 30%) with pulsed flashing (frequency 2Hz), sequentially projecting light signals onto the ground corresponding to each probing sequence. For example, first project a flashing green straight arrow in front of the vehicle to indicate its intention to proceed first, then project another option (e.g., a flashing orange question mark) in front of oncoming vehicles to solicit their opinion. By projecting different options at different spatial locations, a clear visual distinction and a sense of inquiry are achieved.

[0066] Decision-making phase: Based on the final decision, control the vehicle lights to project clear guidance patterns in a continuous, 100% duty cycle. These patterns are typically large light strips or complete symbols covering the width of one lane, with clear instructions. For example, after determining that the vehicle must yield, project a continuously illuminated red stop line in front of the vehicle, while simultaneously projecting a set of continuously illuminated green zebra crossings in front of the pedestrians.

[0067] Environmental Adaptation: The controller automatically adjusts the projection brightness and contrast based on data from the ambient light sensor. On wet or slippery surfaces, it can automatically increase the outline lighting of the pattern or use light of a specific wavelength to reduce glare and water reflection interference.

[0068] In one possible implementation, the cockpit information coordination unit synchronizes the system's external interactions with internal information, enhancing the driver's trust and situational awareness. Its core is to achieve accurate augmented reality information mapping and intelligent multi-channel prompt scheduling, specifically including: AR-HUD augmented reality systems and information mapping specifically include: An augmented reality head-up display (AR-HUD) with a virtual image distance greater than 10 meters and a field of view greater than 10° x 4° is used.

[0069] Spatial Registration and Fusion: The unit receives the precise 3D positions of key targets (such as conflicting pedestrians and vehicles) described in the vehicle coordinate system from the perception fusion unit. Through calibration and coordinate system transformation, combined with IMU data to compensate for vehicle dynamics, spatial registration is achieved, thereby stably and accurately anchoring the virtual annotations to the physical location of the real target.

[0070] Based on this, the system can render an augmented reality (AR) marker with a highlighted border (such as yellow) and an intent icon (such as a pedestrian icon) at the real image location of the key traffic participants being monitored (such as pedestrians being yielded to), and display it on the AR-HUD, achieving millimeter-level alignment between the virtual image and the real scene.

[0071] A multimodal human-computer interaction (HMI) scheduler, specifically including: This module is responsible for intelligently scheduling different notification channels based on the urgency of the conflict. Its core is the information priority arbitration and hierarchical notification strategy.

[0072] Information Priority Arbitration: This module integrates the level of the arbitration trigger instruction, the estimated time difference (TTC) value of the point of conflict, and the results of driver status monitoring (such as eyes off the road) to dynamically decide the channel, intensity, and content of the information presented.

[0073] Tiered notification strategy: Based on the urgency of the TTC, implement tiered notifications to ensure effective information delivery without excessive interference. Standard prompt (TTC≥2 seconds): Determined as low urgency, only triggers and maintains visual annotations on the AR-HUD (such as blue highlighted boxes).

[0074] Level 1 Warning (1 second ≤ TTC < 2 seconds): The AR-HUD label turns yellow and flashes slowly, accompanied by a short warning sound.

[0075] Level 2 Emergency Warning (TTC < 1 second): Determined to be of high urgency, the following are triggered simultaneously: the marker on the AR-HUD turns red and flashes at high frequency, the driver's seat emits high-intensity gradient vibration (the vibration gradually increases in intensity), and a clear directional voice command is played (such as "brake!").

[0076] Interaction consistency: The system ensures that the targets marked in the AR-HUD, the patterns projected by the digital headlights, and the content of the voice prompts are completely consistent in semantics and direction, providing the driver with a unified and contradictory interactive cognition.

[0077] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A digital vehicle-lighting and vehicle-road interaction system based on multimodal perception fusion, characterized in that, include: The roadside cooperative sensing unit is used to acquire and process multimodal data from roadside sensors to generate a local dynamic map containing the location, speed, and trajectory of traffic participants. The vehicle-mounted multimodal perception unit is used to acquire multimodal data from the vehicle's own sensors and output vehicle-mounted perception data. A conflict detection and arbitration triggering unit is used to fuse the local dynamic map and the vehicle perception data to identify traffic participants and predict their travel intentions. An arbitration trigger instruction is generated when a conflict is detected between the predicted paths of at least two traffic participants and the certainty of their intent is below a preset threshold. An uncertainty arbitration decision unit is used to initiate an uncertainty arbitration algorithm based on a dynamic weight model and generate an interaction scheme after receiving the arbitration trigger instruction. The digital vehicle lighting interaction execution unit is used to receive the interaction scheme and control the pixelated vehicle lights to convert the interaction scheme into a dynamic light projection pattern and project it onto the external environment.

2. The digital vehicle-lighting and vehicle-road interaction system based on multimodal perception fusion according to claim 1, characterized in that: The uncertainty arbitration decision-making unit includes: An uncertainty arbitration module is used to calculate and generate at least two alternative tentative passage sequences based on the dynamic weight model after receiving the arbitration trigger instruction, wherein the dynamic weight model integrates the weight coefficients by weighted summation. The interactive feedback learning module is used to update the dynamic weight model and optimize the interaction scheme based on the real-time behavioral feedback of external traffic participants after the digital vehicle light interaction execution unit projects a light projection pattern corresponding to the tentative passage sequence.

3. A digital vehicle-lighting and vehicle-road interaction system based on multimodal perception fusion according to claim 2, characterized in that: The dynamic weight model W The following weighting coefficients are combined using the formula: W = α· + β· + γ· ； in α , β , γ This is an adjustable coefficient. α + β + γ =1, the initial value can be adaptively set according to the traffic scenario: Static rule weights based on traffic regulations ; Safety risk weights based on real-time collision risk ; And collaborative feedback weights based on historical interaction behavior .

4. A digital vehicle-lighting and vehicle-road interaction system based on multimodal perception fusion according to claim 3, characterized in that: The interactive feedback learning module is specifically used for: The roadside cooperative sensing unit or the vehicle-mounted multimodal sensing unit monitors the response behavior of external traffic participants to the exploratory passage sequence. If the response behavior involves deceleration, stopping, or turning to avoid, it is determined as compliance or courtesy, and the corresponding traffic participant's collaborative feedback weight in the dynamic weight model is increased. If the response behavior is characterized by acceleration, not changing course, or intrusion into the conflict zone, it is determined as contention or ignoring, the security risk weight is increased, and the uncertainty arbitration module is triggered to recalculate the passage sequence. The monitoring and updating process is carried out within a preset interaction time window, the length of which is dynamically set based on the distance between the conflict points and the average vehicle speed.

5. A digital vehicle-lighting and vehicle-road interaction system based on multimodal perception fusion according to claim 2, characterized in that: The digital vehicle lighting interaction execution unit is configured to execute the interaction scheme in stages: First, the pixelated headlights are controlled to project tentative light patterns corresponding to each of the tentative passage sequences in a flashing or pulsating manner, either sequentially or in sections. Subsequently, based on the final interaction scheme optimized and generated by the interactive feedback learning module, the pixelated headlights are controlled to project a deterministic guiding light pattern in a continuously lit manner.

6. A digital vehicle-lighting and vehicle-road interaction system based on multimodal perception fusion according to claim 5, characterized in that: The tentative light signals are flashing arrows or question marks used to solicit opinions; the definitive guiding light signals are constantly lit traffic strips, zebra crossings, or stop symbols used to provide clear instructions.

7. A digital vehicle-lighting and vehicle-road interaction system based on multimodal perception fusion according to claim 1, characterized in that: The system also includes: The cockpit information coordination unit is used to receive the arbitration trigger command, interaction scheme, and decision basis; The key traffic participants targeted by the interaction scheme and their assigned right-of-way information are rendered in real time as augmented reality labels with highlighted borders and intent icons, and precisely overlaid on the actual image location of the traffic participant, and displayed on the vehicle's augmented reality head-up display.

8. A digital vehicle-lighting and vehicle-road interaction system based on multimodal perception fusion according to claim 7, characterized in that: The cockpit information coordination unit is also used to schedule in-vehicle prompt channels according to the urgency of the interaction scheme; When the time difference between the expected arrival time of the conflicting parties at the conflict point is less than 2 seconds, it is determined to be of a high degree of urgency, and the red flashing warning mark on the augmented reality head-up display, the seat vibration tactile prompt, and the auditory voice warning are triggered simultaneously. When the time difference is greater than or equal to 2 seconds, it is determined to be of low urgency, and only the augmented reality annotation is triggered.

9. A digital vehicle-lighting and vehicle-road interaction system based on multimodal perception fusion according to claim 3, characterized in that: The intent certainty C is calculated using the following formula: C= · · · ； in , , As a weighting factor, For trajectory prediction confidence, The degree of abnormality in motion state change. The degree of deviation from the standard traffic behavior model, + + =1, the initial value can be adaptively set according to the traffic scenario; And / or, The security risk weight The calculation incorporates the estimated time difference between the vehicle's arrival at the point of conflict and that of the other party involved. Relative velocity ΔV and current environmental visibility V or road surface adhesion coefficient μ ,in With 1 / , ΔV Positively correlated with V , μ It shows a negative correlation.

10. A digital vehicle-lighting and vehicle-road interaction system based on multimodal perception fusion according to claim 2, characterized in that: The uncertainty arbitration module and the interactive feedback learning module work together to achieve a progressive interactive process: The tentative passage sequence is generated and projected based on the initial dynamic weight model; within a preset time window, the passage sequence is iteratively optimized based on the updated weight coefficients. When the results of the passing sequence generated by three consecutive iterations are consistent, or when the change in the weight coefficient is less than the convergence threshold of 5%, the deterministic guiding light language is output.

Images