An intelligent environment perception and video risk early warning system and method for a vehicle-mounted display screen

Abstract

The present application belongs to the field of intelligent transportation and vehicle-mounted electronic technology, and specifically relates to an intelligent environment perception and video risk early warning system and method for a vehicle-mounted display screen, comprising an environment perception module, a space-time reference generation module, a multi-modal data fusion module, a risk decision engine module, a video frame synchronization processing module and a display rendering module. Through a unified space-time reference framework and motion extrapolation correction mechanism, position lag or jump of risk warning signs in high-speed driving scenarios is avoided; multi-modal cross-scale feature fusion and a double-path risk decision architecture are utilized to maintain the semantic integrity of risk identification in complex urban traffic environments; and precise binding of structured metadata and video frames is achieved, ensuring that visual elements and real-world targets are strictly aligned in geometric projection.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation and vehicle electronics technology, specifically an intelligent environmental perception and video risk warning system and method for vehicle-mounted displays. Background Technology

[0002] With the rapid development of intelligent connected vehicle technology, in-vehicle displays have evolved from traditional information display terminals into core components of intelligent cockpits, integrating environmental perception, human-machine interaction, and proactive safety warnings. Against the backdrop of the increasing prevalence of advanced driver assistance systems and autonomous driving functions, in-vehicle displays not only need to present vehicle operating status and navigation information in real time, but have also been entrusted with the important responsibility of intelligently identifying, assessing risks, and providing visual warnings of complex dynamic environments inside and outside the vehicle. This evolutionary trend has significantly improved driving safety and interactive experience, while also placing unprecedented technical demands on the environmental perception capabilities, video processing timeliness, and risk decision-making reliability of in-vehicle display systems.

[0003] Current mainstream technical solutions typically employ a multi-sensor fusion architecture, combining a front-facing camera, millimeter-wave radar, and in-vehicle vision sensors to collect road scene and driver status data. This data is then used to make preliminary judgments on potential risks (such as lane departure, sudden braking by the vehicle in front, and driver distraction) through preset rules or lightweight machine learning models, and the warning information is then overlaid on the in-vehicle display screen in graphical or textual form. Such solutions can indeed provide basic risk warning functions under specific conditions, and their design logic stems from the effective integration of single-modal perception and static threshold judgment, demonstrating a certain degree of engineering rationality and feasibility in the early stages of driver assistance. However, with the exponential growth of complex urban traffic scenarios and users' higher expectations for system response accuracy and reliability, the inherent limitations of this architecture are becoming increasingly apparent.

[0004] At its core, existing technologies suffer from a deep-seated and irreconcilable technical contradiction: a structural mismatch between the dynamic adaptability of environmental perception and the spatiotemporal consistency of video risk warnings. Specifically, traditional systems often decouple the perception module from the display and warning module—the perception end outputs discrete risk events based on independent sampling frequencies and processing delays, while the display end updates the image according to a fixed refresh rate. The two lack a strict alignment mechanism in terms of time and spatial coordinates. Under this constraint, when a vehicle is traveling at high speed or encounters a sudden traffic event (such as a pedestrian crossing or a non-motorized vehicle suddenly cutting in), although the perception system may detect the risk target, the lack of a precise spatiotemporal mapping relationship with the video frame stream causes the superimposed position of the warning sign on the display screen to lag, shift, or even flicker and jump. Furthermore, if multi-source heterogeneous sensor data (such as infrared, visible light, and radar point clouds) is introduced, the differences in sampling timing and coordinate systems between the various modal data make it easier to cause geometric distortion of the fusion result on the display plane, severely weakening the driver's trust in the warning information and their reaction efficiency. It is worth noting that this problem cannot be solved simply by increasing the sensor refresh rate or optimizing the communication bandwidth. The root cause lies in the fact that the existing system architecture lacks a unified spatiotemporal reference framework that runs through the entire chain from perception, fusion, decision-making to visualization, which makes the logic of risk information generation and its presentation logic in the human-machine interface disconnected.

[0005] Therefore, the present invention provides an intelligent environmental perception and video risk warning system and method for vehicle-mounted displays. Summary of the Invention

[0006] To achieve the aforementioned objectives, this invention provides an intelligent environmental perception and video risk warning system and method for vehicle-mounted displays. This system constructs a unified spatiotemporal reference framework, rigorously synchronizing, dynamically calibrating, and co-evolving multi-source heterogeneous environmental perception data with the video stream from the vehicle-mounted display on both the time axis and spatial coordinates. This ensures that the risk warning information possesses geometric accuracy, semantic consistency, and temporal reliability during visualization.

[0007] The system described in this invention includes an environmental perception module, a spatiotemporal reference generation module, a multimodal data fusion module, a risk decision engine module, a video frame synchronization processing module, and a display rendering module; the modules are interconnected through an onboard high-speed communication bus and are globally scheduled and managed by a central control unit.

[0008] The environmental perception module includes a front-facing visible light camera, an infrared thermal imaging sensor, a millimeter-wave radar array, and an in-vehicle driver vision monitoring unit. The front-facing visible light camera and infrared thermal imaging sensor are co-located in the central area inside the vehicle's windshield, with their optical centers coinciding and their fields of view aligned, forming a dual-spectrum imaging subsystem. The millimeter-wave radar array is distributed inside the vehicle's front bumper, with its transmitting and receiving antennas arranged in a MIMO topology to generate high-resolution point cloud output. The in-vehicle driver vision monitoring unit is mounted above the instrument panel and includes a near-infrared light source and a global shutter image sensor, used to collect information on the driver's facial posture, eye movement trajectory, and head orientation. All sensors are equipped with a hardware-level timestamp trigger interface, enabling frame-level sampling alignment under external synchronization signal drive.

[0009] The spatiotemporal reference generation module is deployed in the secure execution environment of the vehicle domain controller. Its core consists of a high-precision clock source and a vehicle motion state calculation unit. The high-precision clock source uses a temperature-compensated crystal oscillator combined with a GNSS time synchronization correction mechanism to provide a nanosecond-level time reference signal. The vehicle motion state calculation unit acquires wheel speed signals, steering angle signals, yaw rate signals, and longitudinal acceleration signals in real time through the CAN bus, and calculates the vehicle's six-degree-of-freedom pose information in the world coordinate system by fusing the output of the inertial measurement unit based on the extended Kalman filter algorithm. This pose information and the time reference signal together constitute a unified spatiotemporal reference, which serves as the reference origin for the spatiotemporal mapping of all subsequent sensing data.

[0010] The multimodal data fusion module receives the raw sensor data stream from the environmental perception module and performs spatiotemporal alignment and coordinate transformation based on a unified spatiotemporal reference. Specifically, visible light image frames and infrared image frames first undergo pixel-level geometric correction via a dual-spectral registration unit to eliminate parallax caused by lens distortion and installation deviation. Millimeter-wave radar point cloud data is then converted to the image plane coordinate system through joint radar and vision calibration parameters to form a dense semantic map with depth information. The driver's state feature vector is extracted by the visual monitoring unit and then supplemented with a gaze direction vector in the vehicle's internal coordinate system. After completing their respective preprocessing, all modal data are fed into a multi-scale feature pyramid network for cross-modal semantic association modeling, outputting a probability distribution tensor containing target category, confidence level, 3D position, motion vector, and behavioral intent.

[0011] The risk decision engine module performs multi-level risk assessment based on multimodal fusion results. Internally, it employs a dual-path processing mechanism: a static rule base and a dynamic learning model. The static rule base defines the mapping relationship between traffic participant types and their minimum safe distance thresholds, and dynamically adjusts the collision time window based on the current vehicle speed. The dynamic learning model is a lightweight convolutional and attention hybrid neural network, whose input is a continuous multi-frame fusion tensor sequence, and whose output is a risk level label and emergency response suggestion for each potential threat target. The results from the two paths are fused through a weighted voting mechanism to generate a final list of risk events. Each event record includes a unique target identifier, risk level, estimated collision time, suggested avoidance direction, and a visual prompt style encoding.

[0012] The video frame synchronization processing module is responsible for accurately matching the risk event list with the current video frame to be rendered on the vehicle display screen. This module has a built-in frame buffer queue and a latency compensation calculator. The frame buffer queue stores the most recent few frames of original video data and their corresponding timestamps. The latency compensation calculator calculates the offset of the actual occurrence time of the risk event relative to the capture time of the current video frame based on the end-to-end transmission delay of the perception link, the fusion calculation time, and the decision reasoning cycle. Based on this offset, the system performs motion extrapolation correction on the spatial position of the risk target to ensure that its projected position on the video screen is visually consistent with its real physical position. The corrected risk event data is encapsulated into structured metadata and appended to the end of the corresponding video frame to form an enhanced video frame.

[0013] The display rendering module runs in a dedicated rendering pipeline of the vehicle's graphics processor. It receives enhanced video frames and parses the structured metadata packets within them. Based on the visual prompt style encoding in the metadata packets, it calls a preset graphic template library to generate semi-transparent warning icons, dynamic bounding boxes, or trajectory prediction arrows. All graphic elements are drawn using vertex shaders in screen space coordinates, and their Z-depth values ​​are forcibly set to be slightly higher than the background video layer but lower than the human-computer interaction control layer to ensure reasonable visual priority. In addition, the rendering module also integrates a color adaptive adjustment unit, which can automatically adjust the brightness and contrast of the warning graphics based on feedback from the ambient light sensor to avoid decreased visibility under strong light or glare at night.

[0014] Preferably, the method of the present invention includes the following steps: Step 1: Activate the environmental perception module, which simultaneously triggers the front-facing visible light camera, infrared thermal imaging sensor, millimeter-wave radar array, and in-vehicle driver visual monitoring unit to perform frame-level sampling, and adds a hardware timestamp to each frame of data. Step 2: The spatiotemporal reference generation module continuously outputs a unified spatiotemporal reference signal, including a global timestamp and vehicle six-degree-of-freedom pose information; Step 3: The multimodal data fusion module performs spatiotemporal alignment, coordinate transformation, and cross-modal semantic fusion on the raw data from each sensor based on a unified spatiotemporal reference, generating a fused tensor sequence. Step four: The risk decision engine module performs static rule matching and dynamic model reasoning based on the fused tensor sequence, and outputs a list of risk events; Step 5: The video frame synchronization processing module performs motion extrapolation correction on the spatial location of the risk event based on the sensing link delay parameters, and encapsulates the corrected data into a structured meta data packet and attaches it to the corresponding video frame. Step six: The display rendering module parses the metadata in the enhanced video frames, calls the graphics template library to generate visual warning elements, and completes the final image composition and output.

[0015] Preferably, the multi-scale feature pyramid network in the multimodal data fusion module adopts an encoder and decoder architecture. The encoder consists of a dual-branch convolutional layer with shared weights, which processes visible light and infrared inputs respectively. The decoder introduces a cross-attention mechanism to enhance the visual features guided by radar point clouds at different scale levels. During the network training phase, a multimodal driving scene dataset with spatiotemporal labels is used. The loss function includes a classification cross-entropy term, a position regression L1 term, and a motion consistency constraint term. All learnable parameters are optimized through end-to-end backpropagation.

[0016] Preferably, the dynamic learning model in the risk decision engine module is deployed in a compressed manner using a knowledge distillation strategy; the teacher model is a large-scale Transformer architecture, and its attention weights and intermediate feature responses are extracted after training in the cloud; the student model is a lightweight MobileNetV3 backbone network, and transfer learning is completed by minimizing the KL divergence of the output distribution of the teacher and student at key spatiotemporal nodes; the distilled student model is embedded in an in-vehicle embedded AI accelerator and supports INT8 quantized inference.

[0017] Preferably, the delay compensation calculator in the video frame synchronization processing module uses a sliding window statistical mechanism to dynamically estimate the average delay of the sensing link; the window length covers several recent complete sensing cycles, and the mean and variance of the delay samples within the window are updated each time a new event arrives; if the current delay exceeds the historical statistical confidence interval, an abnormal alarm is triggered and the system switches to a conservative extrapolation mode, using only the assumption of uniform linear motion for position correction.

[0018] Preferably, the color adaptive adjustment unit in the display rendering module is connected to the ambient light sensor and the OLED display driving circuit; the light sensor outputs an analog voltage signal which is converted into a digital illuminance value by an ADC; this value is used as a lookup table index and mapped to a preset gamma curve offset parameter; the driving circuit dynamically adjusts the luminous intensity of the RGB sub-pixels accordingly, so that the warning graphic maintains a constant subjective brightness under different lighting conditions.

[0019] The beneficial effects of this invention are as follows: This invention, through the aforementioned system architecture and methodology, achieves strict synchronization and dynamic calibration of environmental perception data and in-vehicle display video streams in the spatiotemporal dimensions. Thus, by using a unified spatiotemporal reference framework and motion extrapolation correction mechanism, it avoids positional lag or jumps in risk warning signs during high-speed driving scenarios; it utilizes multimodal cross-scale feature fusion and a dual-path risk decision architecture to maintain the semantic integrity of risk identification in complex urban traffic environments; it achieves precise binding of structured metadata and video frames, ensuring strict alignment of visual elements with real-world targets on geometric projection; it ensures consistency between the depth priority of the graphics rendering layer and human-computer interaction logic, preventing key warning information from being obscured by interface controls; and through an adaptive ambient lighting adjustment mechanism, it guarantees the recognizability of warning graphics across the entire lighting range; simultaneously, through knowledge distillation and quantization deployment strategies, it enables complex AI models to run efficiently on resource-constrained in-vehicle platforms, meeting functional safety level requirements. Attached Figure Description

[0020] The invention will now be further described with reference to the accompanying drawings.

[0021] Figure 1 This is a schematic diagram of the overall structure of an intelligent environmental perception and video risk warning system for an in-vehicle display screen according to the present invention; Figure 2 This is a flowchart of a method for intelligent environmental perception and video risk warning for vehicle-mounted displays according to the present invention. Detailed Implementation

[0022] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0023] like Figure 1 and Figure 2 As shown in the embodiment of the present invention, an intelligent environmental perception and video risk warning system and method for vehicle-mounted displays comprises an environmental perception module, a spatiotemporal reference generation module, a multimodal data fusion module, a risk decision engine module, a video frame synchronization processing module, and a display rendering module. Each module is interconnected through a vehicle-mounted high-speed communication bus and is globally scheduled and managed by a central control unit.

[0024] In some implementations, the environmental perception module includes a front-facing visible light camera, an infrared thermal imaging sensor, a millimeter-wave radar array, and an in-vehicle driver vision monitoring unit. The front-facing visible light camera and the infrared thermal imaging sensor are both located in the central area inside the windshield of the vehicle, with their optical centers coinciding and their field of view aligned, forming a dual-spectrum imaging subsystem. As mentioned earlier, the visible light camera of this dual-spectrum imaging subsystem uses a 1 / 2.8-inch CMOS image sensor with a resolution of 1920×1080, a frame rate of 30fps, a lens focal length of 4.2mm, and a horizontal field of view of 78°; the infrared thermal imaging sensor uses an uncooled microbolometer focal plane array with a resolution of 640×512, a frame rate of 25fps, an operating wavelength of 8-14μm, and a horizontal field of view of 76°. As mentioned above, both are fixed to the same mounting base by rigid brackets to ensure that the relative pose deviation is less than 0.1mm under vehicle vibration conditions. The millimeter-wave radar array is distributed inside the front bumper of the vehicle, including 3 transmitting antennas and 4 receiving antennas, forming a MIMO topology. It operates at a frequency of 77GHz, a bandwidth of 4GHz, a range resolution of 0.075m, a speed resolution of 0.1m / s, and a maximum detection range of 200m. The driver's visual monitoring unit is installed above the instrument panel, including an 850nm near-infrared LED light source array and a global shutter CMOS image sensor with a resolution of 1280×720 and a frame rate of 60fps. It is used to collect information on the driver's facial posture, eye movement trajectory, and head orientation. Furthermore, all sensors are equipped with hardware-level timestamp trigger interfaces, enabling frame-level sampling alignment to be completed under the drive of external synchronization signals, with time synchronization accuracy better than ±500ns.

[0025] In some implementations, the spatiotemporal reference generation module is deployed in the secure execution environment of the vehicle domain controller. Its core consists of a high-precision clock source and a vehicle motion state calculation unit. The high-precision clock source uses a temperature-compensated crystal oscillator (TCXO) combined with a GNSS timing correction mechanism. The frequency stability of the TCXO is ±0.5ppm. The GNSS module supports joint positioning of GPS, GLONASS, Galileo and BeiDou systems, with a timing accuracy better than ±100ns. This clock source provides PPS (Pulse Per Second) signals and a 10MHz reference clock as the time reference for the entire system. As mentioned above, the vehicle motion state calculation unit acquires wheel speed signals, steering angle signals, yaw rate signals, and longitudinal acceleration signals in real time via the CAN bus at a 10ms cycle, and integrates the output of a six-axis inertial measurement unit (IMU). The IMU includes a three-axis accelerometer (range ±8g, zero bias stability <0.1mg) and a three-axis gyroscope (range ±2000° / s, zero bias stability <0.01° / s). Next, based on the extended Kalman filter (EKF) algorithm, the vehicle's six-degree-of-freedom pose information in the WGS-84 world coordinate system is calculated, including position (x, y, z), heading angle ψ, pitch angle θ, and roll angle φ. This pose information, together with the time reference signal, constitutes a unified spatiotemporal reference, which serves as the reference origin for the spatiotemporal mapping of all subsequent sensing data.

[0026] In some implementations, the multimodal data fusion module receives the raw sensor data stream from the environmental perception module and performs spatiotemporal alignment and coordinate transformation on it according to a unified spatiotemporal reference. It is understandable that the visible light image frame and the infrared image frame first complete pixel-level geometric correction through the dual-spectral registration unit. This correction process is based on the pre-calibrated intrinsic and extrinsic parameter matrices. The intrinsic parameters include focal length, principal point offset and radial / tangential distortion coefficients. The extrinsic parameters are the rotation matrix R∈SO(3) and translation vector t∈R³ between the two sensors. The registration algorithm adopts an affine transformation model based on feature point matching. The matching point pairs are obtained through SIFT feature extraction and FLANN nearest neighbor search. Finally, RANSAC is used to eliminate mismatches. The reprojection error is controlled within 0.5 pixels. The millimeter-wave radar point cloud data is converted to the image plane coordinate system through radar and vision joint calibration parameters. The calibration process uses a checkerboard and metal reflector composite calibration material. Synchronous data is collected in a static scene. The extrinsic parameters are solved by the objective function of minimizing the distance from the point to the image edge. After calibration, the root mean square error of the point cloud projection is less than 2 pixels. The transformed radar point cloud forms a dense semantic map with depth information, where each pixel is associated with a depth value d(x,y). The driver state feature vector, extracted by the visual monitoring unit, is supplemented with a gaze direction vector g∈R³ in the vehicle's internal coordinate system. This vector is calculated using a 3D face keypoint fitting and eye center localization algorithm, achieving an accuracy better than ±2°. It should be noted that after each modal data has completed its own preprocessing, it is fed into a multi-scale feature pyramid network for cross-modal semantic association modeling.

[0027] In some implementations, the multi-scale feature pyramid network employs an encoder-decoder architecture. The encoder consists of dual-branch convolutional layers with shared weights, processing visible light and infrared inputs respectively. Each branch contains five convolutional blocks, each consisting of a 3×3 convolution, batch normalization, and a ReLU activation function, with channel numbers of 64, 128, 256, 512, and 512 respectively. The decoder introduces a cross-attention mechanism to achieve radar point cloud-guided visual feature enhancement at different scale levels. Specifically, in the l-th layer decoding stage, the radar depth map dl is mapped to key Kl and value Vl through a 1×1 convolution, and the visual feature Fl is used to generate query Ql through linear projection. The attention weights are calculated as follows: Where dk is the dimension of the key vector, which is 64 here. The network training phase uses a multimodal driving scene dataset with spatiotemporal labels, containing 100,000 synchronously acquired video-radar-driver state sequences, covering various scenarios such as urban, highway, rural, and nighttime driving. The loss function is defined as: in, For the classification cross-entropy term, For the L1 term of location regression, As a motion consistency constraint term, λ1=1.0, λ2=2.0, λ3=0.5, all learnable parameters are optimized through end-to-end backpropagation, and the final output is a probability distribution tensor containing the target category, confidence, 3D position, motion vector and behavioral intent.

[0028] In some implementations, the risk decision engine module performs multi-level risk assessment based on multimodal fusion results. Internally, it employs a dual-path processing mechanism combining a static rule base and a dynamic learning model. The static rule base defines the mapping relationship between traffic participant types and their minimum safe distance thresholds; for example, pedestrians correspond to 1.5m, bicycles to 2.0m, and motor vehicles to 3.0m. It also dynamically adjusts the collision time window (TTC) threshold based on the current vehicle speed v. The calculation formula is as follows: Where dmin is the minimum safe distance, and ε is a small constant (0.1 m / s) to prevent division by zero. If the target's TTC is less than this threshold, it is marked as a potential threat. The dynamic learning model is a lightweight hybrid convolutional and attention neural network. Its input is a fusion tensor sequence of 5 consecutive frames (time step of 100 ms), and its output is a risk level label (low, medium, high) and emergency response suggestions (such as deceleration, left avoidance, right avoidance) for each potential threat target. In some implementations, the dynamic learning model employs a knowledge distillation strategy for compressed deployment. The teacher model is a large Transformer architecture with a 12-layer encoder, 768 hidden dimensions, and 12 attention heads, trained on 500,000 samples in the cloud; the student model is a lightweight MobileNetV3 backbone network with a bottleneck layer expansion factor of 4 and 96 output channels. During the distillation process, minimize the KL divergence of the output distributions of teachers and students at key spatiotemporal nodes: Where zt and zs are the logits outputs of the teacher and student models, respectively, and τ is the temperature parameter with a value of 3. The distilled student model is embedded in an in-vehicle embedded AI accelerator (such as NVIDIA OrinNX), supports INT8 quantization inference, and has an inference latency of less than 15ms.

[0029] The results of the two paths are merged through a weighted voting mechanism to generate a final list of risk events. Each event record includes a unique identifier for the target (UUID), risk level (0-2 integer code), estimated collision time (unit: seconds), suggested avoidance direction (-1 for left, +1 for right, 0 for no direction), and visual prompt style code (e.g., 0x01 for red flashing box, 0x02 for yellow trajectory arrow).

[0030] The video frame synchronization processing module is responsible for accurately matching the risk event list with the current video frame to be rendered on the vehicle display. This module has a built-in frame buffer queue and a latency compensation calculator. The frame buffer queue adopts a circular buffer structure to store the most recent 10 frames of raw video data and their corresponding timestamps. Each frame contains a YUV422 format image and a 128-byte metadata header. The latency compensation calculator calculates the offset Δt of the actual occurrence time of the risk event relative to the current video frame capture time based on the end-to-end transmission latency of the perception link, the fusion calculation time, and the decision reasoning cycle. The perception link latency includes sensor acquisition latency (about 5ms), CAN / FlexRay transmission latency (about 2ms), fusion calculation latency (about 20ms), and decision reasoning latency (about 15ms), with a total latency of about 42ms. In some implementations, the delay compensation calculator uses a sliding window statistical mechanism to dynamically estimate the average delay of the sensing link. The window length covers the most recent 20 complete sensing cycles (i.e., 2 seconds), and the mean μ and variance σ² of the delay samples within the window are updated each time a new event arrives. Continuing from the above, if the current delay exceeds the historical statistical confidence interval [μ-2σ,μ+2σ], an anomaly alarm will be triggered and the system will switch to conservative extrapolation mode, using only the assumption of uniform linear motion for position correction. Continuing, in normal mode, the system uses a constant acceleration model to extrapolate and correct the spatial position of the risk target: Among them, p0, v0, and a0 are the position, velocity, and acceleration vectors of the target at the moment the event occurs, respectively. They are all output by the multimodal fusion module. The corrected risk event data is encapsulated into structured meta data packets and appended to the end of the corresponding video frame to form an enhanced video frame. The meta data packets adopt the TLV (Type-Length-Value) encoding format and the total length does not exceed 256 bytes.

[0031] In some implementations, the display rendering module runs in a dedicated rendering pipeline of the automotive graphics processor (such as Qualcomm Adreno 640). It receives enhanced video frames and parses the structured metadata packets within them. Based on the visual prompt style encoding in the metadata packets, it calls a preset graphics template library to generate semi-transparent warning icons, dynamic bounding boxes, or trajectory prediction arrows. All graphic elements are drawn using vertex shaders in screen space coordinates, and their Z depth value is forcibly set to 0.95 (1.0 for the background video layer and 0.9 for the human-computer interaction control layer) to ensure reasonable visual priority. In some implementations, the color adaptive adjustment unit in the display rendering module is connected to the ambient light sensor and the OLED display driving circuit. The light sensor is a silicon photodiode with a spectral response range of 400-1100nm. Its output analog voltage signal is converted into a digital illuminance value E (unit: lux) by a 12-bit ADC, ranging from 0-100,000 lux. This value serves as a lookup table index, mapped to a preset gamma curve offset parameter γoffset. The driving circuit dynamically adjusts the luminous intensity of the RGB sub-pixels accordingly, ensuring the warning graphic maintains a constant subjective brightness under different lighting conditions. The gamma correction formula is: Where γ is the default gamma value (taken as 2.2), and γoffset(E) is obtained from experimental calibration. It is +0.3 when E < 100 lux and -0.2 when E > 10,000 lux. The intermediate value is linearly interpolated.

[0032] In summary, the aforementioned system architecture and methodology achieve strict synchronization and dynamic calibration of environmental perception data and in-vehicle display video streams across the spatiotemporal dimensions. This ensures that, through a unified spatiotemporal reference framework and motion extrapolation correction mechanism, positional lag or abrupt changes in risk warning signs are avoided in high-speed driving scenarios; multimodal cross-scale feature fusion and a dual-path risk decision architecture maintain the semantic integrity of risk identification in complex urban traffic environments; precise binding of structured metadata to video frames ensures strict alignment of visual elements with real-world targets on geometric projection; consistency between the depth priority of the graphics rendering layer and human-computer interaction logic is ensured, preventing critical warning information from being obscured by interface controls; and the ambient lighting adaptive adjustment mechanism guarantees the recognizability of warning graphics across the entire illumination range. Furthermore, knowledge distillation and quantization deployment strategies enable complex AI models to operate efficiently on resource-constrained in-vehicle platforms, meeting functional safety requirements.

[0033] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A smart environmental perception and video risk warning system and method for vehicle-mounted displays, characterized in that, include: The system includes an environmental perception module, a spatiotemporal reference generation module, a multimodal data fusion module, a risk decision engine module, a video frame synchronization processing module, and a display rendering module. The environmental perception module includes a front-facing visible light camera, an infrared thermal imaging sensor, a millimeter-wave radar array, and an in-vehicle driver visual monitoring unit. Each sensor is equipped with a hardware-level timestamp trigger interface to complete frame-level sampling alignment under the drive of an external synchronization signal. The spatiotemporal reference generation module is deployed in the execution environment of the vehicle domain controller and is used to output a unified spatiotemporal reference signal, which includes a global timestamp and the vehicle's six-degree-of-freedom pose information in the world coordinate system. The multimodal data fusion module receives the raw sensor data stream from the environmental perception module and performs spatiotemporal alignment, coordinate transformation and cross-modal semantic fusion on each modal data according to the unified spatiotemporal reference signal to generate a probability distribution tensor containing target category, confidence level, three-dimensional position, motion vector and behavioral intent. The risk decision engine module performs static rule matching and dynamic model reasoning based on the probability distribution tensor, and outputs a list of risk events containing a unique target identifier, risk level, estimated collision time, suggested avoidance direction, and visual prompt style encoding. The video frame synchronization processing module has a built-in frame buffer queue and a delay compensation calculator, which is used to calculate the offset of the actual occurrence time of the risk event relative to the current video frame capture time based on the end-to-end delay of the sensing link, and to perform motion extrapolation correction on the spatial position of the risk target based on the offset. The corrected risk event data is encapsulated into a structured meta data packet and attached to the corresponding video frame to form an enhanced video frame. The display rendering module runs in the dedicated rendering pipeline of the vehicle graphics processor. It is used to parse the structured metadata in the enhanced video frame, call the preset graphics template library to generate visual warning elements, and adaptively adjust their brightness and contrast according to the ambient lighting conditions to complete the image synthesis output.

2. The intelligent environmental perception and video risk warning system for vehicle-mounted displays according to claim 1, characterized in that, The front-facing visible light camera and infrared thermal imaging sensor are co-located in the central area inside the vehicle's windshield, with their optical centers coinciding and their field of view aligned, forming a dual-spectrum imaging subsystem; the millimeter-wave radar array is distributed inside the vehicle's front bumper, with its transmitting and receiving antennas arranged in a MIMO topology; the in-vehicle driver vision monitoring unit is installed above the dashboard to collect information on the driver's facial posture, eye movement trajectory, and head orientation.

3. The intelligent environmental perception and video risk warning system for vehicle-mounted displays according to claim 1, characterized in that, The cross-modal semantic fusion in the multimodal data fusion module is achieved through a multi-scale feature pyramid network. The network adopts an encoder and decoder architecture. The encoder part includes a dual-branch convolutional layer that processes visible light and infrared inputs respectively. The decoder introduces a cross-attention mechanism to guide visual feature enhancement using radar point clouds at different scale levels.

4. The intelligent environmental perception and video risk warning system for vehicle-mounted displays according to claim 1, characterized in that, The risk decision engine module adopts a dual-path processing mechanism: the static rule base sets the minimum safe distance threshold according to the type of traffic participant, and dynamically adjusts the collision time window in combination with the current vehicle speed; The dynamic learning model is a lightweight hybrid convolutional and attention neural network. The input is a continuous multi-frame fused tensor sequence, and the output is a risk level label and emergency response suggestions. The results from the two paths are merged through a weighted voting mechanism to generate the final list of risk events.

5. The intelligent environmental perception and video risk warning system for vehicle-mounted displays according to claim 4, characterized in that, The dynamic learning model is deployed in a compressed manner through a knowledge distillation strategy: the teacher model is a large-scale Transformer architecture, and the student model is a MobileNetV3 backbone network. Transfer learning is achieved by minimizing the KL divergence of the output distribution of the two at key spatiotemporal nodes. The distilled student model is embedded in an in-vehicle embedded AI accelerator and supports INT8 quantization inference.

6. The intelligent environmental perception and video risk warning system for vehicle-mounted displays according to claim 1, characterized in that, The delay compensation calculator in the video frame synchronization processing module uses a sliding window statistical mechanism to dynamically estimate the average delay of the sensing link. When the delay exceeds the historical statistical confidence interval, it switches to a conservative extrapolation mode and uses only the assumption of uniform linear motion for position correction. In normal mode, a constant acceleration model is used to perform motion extrapolation correction on the position of the risk target.

7. The intelligent environmental perception and video risk warning system for vehicle-mounted displays according to claim 1, characterized in that, The visual warning elements in the display rendering module include semi-transparent warning icons, dynamic bounding boxes, or trajectory prediction arrows. Their Z-depth values ​​are forcibly set to be slightly higher than the background video layer but lower than the human-computer interaction control layer to ensure reasonable visual priority.

8. The intelligent environmental perception and video risk warning system for vehicle-mounted displays according to claim 1, characterized in that, The display rendering module integrates a color adaptive adjustment unit, which is connected to the ambient light sensor and the OLED display driving circuit. It dynamically adjusts the luminous intensity of RGB sub-pixels according to the digital illuminance value, so that the warning graphic maintains a constant subjective brightness under different lighting conditions.

9. A method for intelligent environmental perception and video risk warning for vehicle-mounted displays, applicable to the intelligent environmental perception and video risk warning system for vehicle-mounted displays as described in any one of claims 1-8, characterized in that, Includes the following steps: Start the environmental perception module to synchronously trigger each sensor to perform frame-level sampling and attach a hardware timestamp; The spatiotemporal reference generation module continuously outputs a unified spatiotemporal reference signal; The multimodal data fusion module performs spatiotemporal alignment, coordinate transformation, and cross-modal semantic fusion on the original sensor data based on the unified spatiotemporal reference to generate a fused tensor sequence; The risk decision engine module performs static rule matching and dynamic model reasoning based on the fused tensor sequence, and outputs a list of risk events; The video frame synchronization processing module performs motion extrapolation correction on the spatial location of risk events based on the perceived link delay parameters, and encapsulates the corrected data into structured metadata packets and appends them to the corresponding video frames. The display rendering module parses the metadata in the enhanced video frames, calls the graphics template library to generate visual warning elements, and completes the final image composition and output.

10. A method for intelligent environmental perception and video risk warning for vehicle-mounted displays according to claim 9, characterized in that, In the multimodal data fusion step, the visible light image and the infrared image are subjected to pixel-level geometric correction through a pre-calibrated intrinsic and extrinsic parameter matrix; the millimeter-wave radar point cloud is transformed to the image plane coordinate system through radar and vision joint calibration parameters; and the driver state feature vector is supplemented with the gaze direction vector in the vehicle's internal coordinate system.

Images