Mountainous anti-resistance vehicle road coordination terminal multi-modal perception and low latency communication method

Abstract

The application discloses a kind of mountainous resistance vehicle road coordination terminal's multimodal perception and low delay communication method, comprising: the multi-source perception information of target area and environmental state information are collected, and current road environment working condition is determined according to environmental state information;According to the fusion participation degree of each perception data source of road environment working condition, dynamically adjust, generate fusion perception result;Based on fusion perception result, execute edge computing processing, generate traffic road warning information, and send traffic road warning information to vehicle terminal, roadside facility or traffic management platform through target communication link.The application can improve the perception accuracy and communication reliability under the harsh environment of mountainous area, so as to enhance the safety warning capability of vehicle road coordination.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation and vehicle-road cooperation, specifically to a multimodal perception and low-latency communication method for a mountain-resistant vehicle-road cooperative terminal. Background Technology

[0002] Vehicle-road cooperative technology can significantly improve road traffic safety and efficiency through real-time information exchange between roadside terminals and vehicle-mounted terminals. However, existing vehicle-road cooperative terminals and methods are mainly designed for plains or conventional climates, and have significant limitations in adaptability when facing the complex working conditions unique to certain mountainous areas in Southwest China.

[0003] Currently, in terms of environmental perception, existing technologies struggle to simultaneously address winter road icing (low adhesion coefficient), dense fog obstruction in high humidity environments, and frequent slopes and strong vibrations on mountain roads. Single sensors (such as purely visual or purely millimeter-wave radar) are prone to misjudgments under icing conditions, especially in dense fog, where performance degrades significantly. Regarding communication transmission, the complex terrain of mountainous areas leads to severe signal obstruction, and single communication modules (such as those relying solely on C-V2X or 4G / 5G) are prone to disconnection in tunnels or deep valleys, failing to meet the practical requirements of low latency and high reliability for security applications.

[0004] Therefore, to solve the above problems, a multimodal perception and low-latency communication method for a mountain-resistant vehicle-road cooperative terminal is needed to improve the perception accuracy and communication reliability in harsh mountain environments, thereby enhancing the vehicle-road cooperative safety early warning capability. Summary of the Invention

[0005] In view of this, the purpose of this invention is to overcome the deficiencies in the prior art and provide a multimodal perception and low-latency communication method for a mountain-resistant vehicle-road cooperative terminal, which can improve the perception accuracy and communication reliability in harsh mountain environments, thereby enhancing the vehicle-road cooperative safety early warning capability.

[0006] The multimodal sensing and low-latency communication method for the mountain-resistant vehicle-road cooperative terminal of the present invention includes:

[0007] Collect multi-source sensing information and environmental status information of the target area, and determine the current road environment conditions based on the environmental status information;

[0008] The degree of fusion participation of each sensing data source is dynamically adjusted according to the road environment conditions to generate fused sensing results;

[0009] Edge computing processing is performed based on the fusion perception results to generate traffic and road warning information, which is then sent to vehicle terminals, roadside facilities, or traffic management platforms via target communication links.

[0010] Furthermore, the sensing unit is used to collect multi-source sensing information of the target area; the sensing unit includes a visible light camera, an infrared thermal imaging sensor, and a millimeter-wave radar.

[0011] Furthermore, the environmental status information includes ambient temperature, ambient humidity, light intensity, visibility, precipitation status, road surface temperature, and wind speed;

[0012] The current road environmental conditions are determined based on environmental status information, specifically including:

[0013] The risk of road icing is determined based on ambient temperature, ambient humidity, and road surface temperature: when the ambient humidity is greater than the first preset threshold and the road surface temperature is lower than the second preset threshold, the current road environment is determined to be a freezing condition.

[0014] The road is determined to be in a low visibility state based on visibility and light intensity: when the visibility is less than the third preset threshold, the current road environment is determined to be a low visibility state.

[0015] When both freezing and low visibility conditions are met simultaneously, the current road environment is determined to be a complex and severe condition.

[0016] Furthermore, the degree of fusion participation of each sensing data source is dynamically adjusted according to the road environment conditions to generate fused sensing results, specifically including:

[0017] When visibility is less than n meters or road surface icing is detected, reduce the weight of the visible light camera, activate the infrared thermal imaging sensor, and increase the weight of the millimeter-wave radar.

[0018] Preprocessing is performed on infrared thermal imaging data and millimeter-wave radar data respectively to achieve time synchronization, spatial registration and noise suppression;

[0019] A convolutional neural network is used to extract thermal feature maps from infrared thermal imaging data. These thermal feature maps are used to characterize the temperature distribution features of the target area.

[0020] A point cloud feature extraction network is used to extract spatial location features from millimeter-wave radar point clouds. These spatial location features are used to characterize the target's distance, orientation, and motion state.

[0021] The thermal feature map and spatial location features are mapped to a unified feature space to construct a cross-modal fusion feature set; the cross-modal fusion feature set is input into the channel attention module to calculate the attention weight coefficients corresponding to each feature channel;

[0022] The thermal feature map and spatial location features are weighted and enhanced based on the attention weight coefficient, and the feature response of regions with significant temperature changes, significant distance changes and significant motion changes is automatically enhanced.

[0023] The enhanced multimodal features are concatenated along their feature dimensions to generate a unified fusion feature map. This unified fusion feature map is then input into the target detection network, which outputs the target category, target location coordinates, target motion velocity, and target confidence information.

[0024] Based on the thermal characteristics of the road surface area, radar echo intensity, and target movement characteristics, the road freezing status is identified and the corresponding freezing risk level is output.

[0025] Furthermore, the target communication link is configured as follows:

[0026] In open road sections, C-V2X communication links are used for broadcasting, with latency controlled within tms; when vehicles enter tunnels or signal blind spots, causing the quality of the C-V2X communication link to fall below a preset threshold, the system switches to 4G / 5G cellular networks; when the C-V2X communication link is restored, the system switches back again.

[0027] Furthermore, the following method is used to determine whether the quality of the C-V2X communication link is lower than a preset threshold:

[0028] Calculate the Link Quality Index (LQI); if the C-V2X LQI is lower than a preset threshold, then the C-V2X communication link quality is lower than the preset threshold; the Link Quality Index (LQI) is calculated according to the following formula:

[0029] ;

[0030] in, , as well as All are set coefficients. The packet loss rate for C-V2X; For C-V2X latency; This represents the maximum permissible latency for C-V2X. This represents the signal receiving power.

[0031] Furthermore, by using an attitude sensor to monitor the attitude changes of the sensing unit caused by vibration in real time, online self-calibration is triggered, specifically including:

[0032] Real-time acquisition of pitch angle, roll angle and vibration acceleration data of the sensing unit;

[0033] The current attitude parameters are compared with the preset reference attitude parameters to calculate the attitude offset. When the pitch angle change or roll angle change exceeds the preset angle threshold, it is determined that the vibration has caused the installation state of the sensing unit to shift, and the visual-inertial navigation joint calibration algorithm is triggered.

[0034] The joint visual-inertial navigation calibration algorithm includes:

[0035] An initial extrinsic parameter matrix between the camera and the millimeter-wave radar is constructed based on attitude data, and a spatial transformation relationship is established under a unified coordinate system.

[0036] Acquire visible light or infrared images and corresponding millimeter-wave radar point cloud data; extract stationary rigid feature points from the visible light or infrared images and generate image feature descriptors;

[0037] Extract the corresponding static target point cloud features from millimeter-wave radar point cloud data and establish the corresponding matching relationship between image feature points and point cloud feature points;

[0038] A reprojection error function is constructed based on the matching results, and the reprojection error function is iteratively optimized to minimize the error between the image observation position and the point cloud projection position.

[0039] The camera's intrinsic parameters, the millimeter-wave radar's extrinsic parameters, and the relative pose relationship between them are updated based on the optimization results. The updated calibration parameters are written into the terminal configuration storage area, and the spatial mapping relationship of the multi-source sensing data is re-established using the updated calibration parameters, thereby completing the real-time online calibration of the sensing unit.

[0040] Furthermore, the traffic and road early warning information includes blind spot vehicle approach warning information, road freezing warning information, slope passing warning information, road rockfall warning information, abnormal vehicle parking warning information, and road congestion warning information.

[0041] The beneficial effects of this invention are as follows: This invention discloses a multimodal perception and low-latency communication method for a mountain-resistant vehicle-road cooperative terminal. It utilizes environmental state information to identify freezing conditions, low visibility conditions, and combined adverse conditions, and dynamically adjusts the fusion participation of visible light cameras, infrared thermal imaging sensors, and millimeter-wave radar. A feature-level fusion algorithm based on an attention mechanism generates high-precision fusion perception results, improving target detection and road condition recognition capabilities in harsh environments such as dense fog and freezing. Real-time evaluation of the C-V2X communication link is performed using a link quality index, enabling adaptive switching of the communication link and improving the continuity and reliability of information transmission in signal-obstructed scenarios such as mountain tunnels and deep valleys. Attitude sensors monitor the attitude displacement of the sensing unit caused by vibration and trigger joint visual-inertial navigation calibration, achieving online correction of the extrinsic parameters of the camera and millimeter-wave radar, thereby ensuring the spatial alignment accuracy of multi-source sensing data under long-term operating conditions. Attached Figure Description

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0043] Figure 1 This is a schematic diagram illustrating the principle of the multimodal sensing and low-latency communication method of the present invention;

[0044] Figure 2 This is a communication switching timing diagram of the present invention. Detailed Implementation

[0045] The present invention will be further described below with reference to the accompanying drawings, as shown in the figures:

[0046] This embodiment discloses a multimodal perception and low-latency communication method for a mountain-resistant vehicle-road cooperative terminal, including the following steps:

[0047] Collect multi-source sensing information and environmental status information of the target area, and determine the current road environment conditions based on the environmental status information;

[0048] The degree of fusion participation of each sensing data source is dynamically adjusted according to the road environment conditions to generate fused sensing results;

[0049] Edge computing processing is performed based on the fusion perception results to generate traffic and road warning information, which is then sent to vehicle terminals, roadside facilities, or traffic management platforms via target communication links.

[0050] In this embodiment, a sensing unit is used to collect multi-source sensing information of the target area to achieve comprehensive perception of complex road scenes. The sensing unit includes a visible light camera, an infrared thermal imaging sensor, and a millimeter-wave radar. The visible light camera is used to collect visible light image information of the road scene to obtain visual features such as lane lines, traffic signs, vehicle shapes, and pedestrian outlines. The infrared thermal imaging sensor is used to collect infrared thermal radiation information of the target area to obtain the temperature distribution characteristics of the target, thereby enhancing the target recognition capability at night, in dense fog, or in low light conditions, and assisting in the identification of abnormal conditions such as road surface freezing. The millimeter-wave radar is used to transmit and receive millimeter-wave signals to obtain the target's distance, speed, and orientation information, thereby achieving stable target detection and motion state perception under rain, snow, fog, haze, and obstruction conditions. Through the coordinated operation of the three sensing devices, multi-dimensional information about the road environment is acquired, providing a data foundation for subsequent multi-modal fusion perception and vehicle-road cooperative decision-making.

[0051] In this embodiment, the environmental status information includes ambient temperature, ambient humidity, light intensity, visibility, precipitation status, road surface temperature, and wind speed.

[0052] The current road environmental conditions are determined based on environmental status information, specifically including:

[0053] The risk of road icing is determined based on ambient temperature, ambient humidity, and road surface temperature: when the ambient humidity is greater than 85% and the road surface temperature is lower than 2℃, the current road environment is determined to be a freezing condition.

[0054] Determine whether a road is in a low visibility condition based on visibility and light intensity: when visibility is less than 100 meters, the current road environment is determined to be a low visibility condition.

[0055] When both freezing and low visibility conditions are met simultaneously, the current road environment is determined to be a complex and severe condition.

[0056] In this embodiment, the degree of fusion participation of each sensing data source is dynamically adjusted according to the road environment conditions to generate fused sensing results, specifically including:

[0057] When visibility is below 100 meters or road surface icing is detected, the weight of the visible light camera is reduced, the infrared thermal imaging sensor is activated, and the weight of the millimeter-wave radar is increased. Reducing the weight of the visible light camera means decreasing the participation of visible light image features in the multimodal fusion process; increasing the weight of the millimeter-wave radar means increasing the contribution ratio of millimeter-wave radar features in the fusion result calculation; activating the infrared thermal imaging sensor means initiating the infrared thermal imaging data acquisition and processing process, and introducing infrared thermal features into the fusion model to participate in target detection and road condition recognition. By dynamically adjusting the fusion participation of each sensing data source, high target detection accuracy and environmental perception capabilities can be maintained even in harsh environments such as low visibility and icing.

[0058] An attention-based feature-level fusion algorithm is used to fuse infrared images and radar point clouds, including:

[0059] Preprocessing is performed on infrared thermal imaging data and millimeter-wave radar data respectively to achieve time synchronization, spatial registration and noise suppression;

[0060] A convolutional neural network is used to extract thermal feature maps from infrared thermal imaging data. These thermal feature maps are used to characterize the temperature distribution features of the target area.

[0061] A point cloud feature extraction network is used to extract spatial location features from millimeter-wave radar point clouds. These spatial location features are used to characterize the target's distance, orientation, and motion state.

[0062] The thermal feature map and spatial location features are mapped to a unified feature space to construct a cross-modal fusion feature set; the cross-modal fusion feature set is input into the channel attention module to calculate the attention weight coefficients corresponding to each feature channel;

[0063] The thermal feature map and spatial location features are weighted and enhanced based on the attention weight coefficient, and the feature response of regions with significant temperature changes, significant distance changes and significant motion changes is automatically enhanced.

[0064] The enhanced multimodal features are concatenated along their feature dimensions to generate a unified fusion feature map. This unified fusion feature map is then input into the target detection network, which outputs the target category, target location coordinates, target motion velocity, and target confidence information.

[0065] Based on the thermal characteristics of the road surface area, radar echo intensity, and target movement characteristics, the road freezing status is identified and the corresponding freezing risk level is output.

[0066] In this embodiment, as Figure 2 As shown, the target communication link is set up according to the following method:

[0067] In open road sections, the low-latency C-V2X communication link (PC5 interface) is prioritized for broadcasting, with latency controlled within 10ms. When a vehicle enters a tunnel or a signal blind spot, causing the C-V2X communication link quality to fall below a preset threshold, the system switches to 4G / 5G cellular networks (Uu interface) to maintain the transmission of basic safety messages by leveraging the wide coverage of cellular networks. Once the C-V2X communication link is restored, the system switches back to 4G / 5G cellular networks. This method ensures the reliability of vehicle-to-infrastructure communication through a dual-mode hot backup mechanism.

[0068] Determine whether the quality of the C-V2X communication link is below a preset threshold using the following method:

[0069] Calculate the Link Quality Index (LQI); if the C-V2X LQI is lower than a preset threshold (e.g., 60), then the C-V2X communication link quality is lower than the preset threshold; the Link Quality Index (LQI) is calculated according to the following formula:

[0070] ;

[0071] Where A=0.4, B=0.3, C=0.3, The packet loss rate for C-V2X; For C-V2X latency; This represents the maximum permissible latency for C-V2X. For signal receiving power, The unit is dBm, and the normal range is... ~0dBm.

[0072] In this embodiment, it further includes: real-time monitoring of the attitude change of the sensing unit caused by vibration using an attitude sensor to trigger online self-calibration, specifically including:

[0073] Real-time acquisition of pitch angle, roll angle and vibration acceleration data of the sensing unit;

[0074] The current attitude parameters are compared with the preset reference attitude parameters to calculate the attitude offset; when the pitch angle change or roll angle change exceeds the preset angle threshold (e.g., ...), the attitude offset is calculated. When vibration causes a shift in the installation state of the sensing unit, the visual-inertial navigation joint calibration algorithm is triggered.

[0075] The joint visual-inertial navigation calibration algorithm includes:

[0076] An initial extrinsic parameter matrix between the camera and the millimeter-wave radar is constructed based on attitude data, and a spatial transformation relationship is established under a unified coordinate system.

[0077] Acquire visible light or infrared images and corresponding millimeter-wave radar point cloud data; extract stationary rigid feature points corresponding to road guardrails, lane lines, traffic sign poles, lighting poles, or building edges from the visible light or infrared images, and generate image feature descriptors using the Scale Invariant Feature Transform (SIFT) algorithm or the Oriented Fast Rotation Binary Descriptor (ORB) algorithm.

[0078] Extract the corresponding static target point cloud features from millimeter-wave radar point cloud data and establish the corresponding matching relationship between image feature points and point cloud feature points;

[0079] Based on the matching results, a reprojection error function is constructed, and the Levenberg-Marquardt optimization algorithm is used to iteratively optimize the reprojection error function to minimize the error between the image observation position and the point cloud projection position.

[0080] The camera's intrinsic parameters, the millimeter-wave radar's extrinsic parameters, and the relative pose relationship between them are updated based on the optimization results. The updated calibration parameters are written into the terminal configuration storage area, and the spatial mapping relationship of the multi-source sensing data is re-established using the updated calibration parameters, thereby completing the real-time online calibration of the sensing unit.

[0081] In this embodiment, a dual-mode communication unit is configured to select the target communication link; the dual-mode communication unit includes a C-V2XPC5 interface module and a 4G / 5G Uu interface module; the computing unit performs edge computing processing based on the fusion perception results; the computing unit can be implemented using existing edge computing processors or embedded computing platforms, such as ARM-based embedded processors, x86-based low-power industrial computers, automotive-grade SoC chips, or edge computing modules based on FPGA, DSP, and GPU acceleration, etc.

[0082] The computing unit integrates a CPU, an image processing unit (ISP), a neural network acceleration unit (NPU), and a communication interface module. This supports parallel processing of multi-source sensing data and deep learning inference operations. It can also run lightweight object detection models, feature fusion networks, and rule-based reasoning algorithms, enabling real-time analysis of fused sensing results, traffic risk identification, and early warning information generation. For blind spot scenarios on mountain slopes, a virtual sensor list (including forward blind spot vehicle warnings and road surface freezing coefficients) is generated. Data packets are compressed using the MQTT protocol, and standardized messages are broadcast to surrounding vehicles via the target communication link, with the end-to-end latency controlled within 50ms.

[0083] Furthermore, the sensing unit, dual-mode communication unit, and computing unit can be integrated into a single protective housing. The housing is filled with a high thermal conductivity shock-absorbing gel, which not only secures the components and absorbs high-frequency vibrations but also effectively prevents the condensation of high-humidity mist on the circuit board.

[0084] In this embodiment, the traffic and road early warning information includes blind spot vehicle approach warning information, road icing warning information, slope passing warning information, road rockfall warning information, abnormal vehicle parking warning information, and road congestion warning information. This configuration expands the warning content from a single event to a multi-category collaborative warning covering typical high-risk scenarios in mountainous areas. This enables differentiated identification and classification of different road risk types, improving coverage of sudden traffic risks and hidden dangers in complex terrain. By combining target communication links for real-time broadcasting to vehicle terminals, roadside facilities, and traffic management platforms, the dissemination range and timeliness of warning information are improved, further enhancing traffic safety assurance capabilities in complex mountainous road environments.

[0085] This invention solves the problems of poor visibility and inaccurate identification under conditions of high humidity, fog, and winter freezing by adaptively fusing infrared and radar. It overcomes communication interruptions caused by terrain obstruction in mountainous areas through C-V2X and 4G / 5G dual-mode intelligent redundancy switching, ensuring continuous low-latency connections. It addresses structural loosening and accuracy degradation caused by strong slope vibrations by utilizing seismic-resistant potting and online self-calibration technologies. Finally, it reduces the reliance on high-end hardware by implementing an attention-based feature-level fusion algorithm and a joint visual-inertial navigation calibration algorithm.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multimodal sensing and low-latency communication method for a mountain-resistant vehicle-road cooperative terminal, characterized in that: include: Collect multi-source sensing information and environmental status information of the target area, and determine the current road environment conditions based on the environmental status information; The degree of fusion participation of each sensing data source is dynamically adjusted according to the road environment conditions to generate fused sensing results; Edge computing processing is performed based on the fusion perception results to generate traffic and road warning information, which is then sent to vehicle terminals, roadside facilities, or traffic management platforms via target communication links.

2. The multimodal sensing and low-latency communication method for the mountain-resistant vehicle-road cooperative terminal according to claim 1, characterized in that: The system utilizes a sensing unit to collect multi-source sensing information about the target area; the sensing unit includes a visible light camera, an infrared thermal imaging sensor, and a millimeter-wave radar.

3. The multimodal sensing and low-latency communication method for the mountain-resistant vehicle-road cooperative terminal according to claim 1, characterized in that: The environmental status information includes ambient temperature, ambient humidity, light intensity, visibility, precipitation status, road surface temperature, and wind speed; The current road environmental conditions are determined based on environmental status information, specifically including: The risk of road icing is determined based on ambient temperature, ambient humidity, and road surface temperature: when the ambient humidity is greater than the first preset threshold and the road surface temperature is lower than the second preset threshold, the current road environment is determined to be a freezing condition. The road is determined to be in a low visibility state based on visibility and light intensity: when the visibility is less than the third preset threshold, the current road environment is determined to be a low visibility state. When both freezing and low visibility conditions are met simultaneously, the current road environment is determined to be a complex and severe condition.

4. The multimodal sensing and low-latency communication method for the mountain-resistant vehicle-road cooperative terminal according to claim 2, characterized in that: The degree of fusion participation of each sensing data source is dynamically adjusted based on road environmental conditions to generate fused sensing results, specifically including: When visibility is less than n meters or road surface icing is detected, reduce the weight of the visible light camera, activate the infrared thermal imaging sensor, and increase the weight of the millimeter-wave radar. Preprocessing is performed on infrared thermal imaging data and millimeter-wave radar data respectively to achieve time synchronization, spatial registration and noise suppression; A convolutional neural network is used to extract thermal feature maps from infrared thermal imaging data. These thermal feature maps are used to characterize the temperature distribution features of the target area. A point cloud feature extraction network is used to extract spatial location features from millimeter-wave radar point clouds. These spatial location features are used to characterize the target's distance, orientation, and motion state. The thermal feature map and spatial location features are mapped to a unified feature space to construct a cross-modal fusion feature set; the cross-modal fusion feature set is input into the channel attention module to calculate the attention weight coefficients corresponding to each feature channel; The thermal feature map and spatial location features are weighted and enhanced based on the attention weight coefficient, and the feature response of regions with significant temperature changes, significant distance changes and significant motion changes is automatically enhanced. The enhanced multimodal features are concatenated along their feature dimensions to generate a unified fusion feature map. This unified fusion feature map is then input into the target detection network, which outputs the target category, target location coordinates, target motion velocity, and target confidence information. Based on the thermal characteristics of the road surface area, radar echo intensity, and target movement characteristics, the road freezing status is identified and the corresponding freezing risk level is output.

5. The multimodal sensing and low-latency communication method for a mountain-resistant vehicle-road cooperative terminal according to claim 1, characterized in that: Configure the target communication link using the following method: In open road sections, C-V2X communication links are used for broadcasting, with latency controlled within tms; when vehicles enter tunnels or signal blind spots, causing the quality of the C-V2X communication link to fall below a preset threshold, the system switches to 4G / 5G cellular networks; when the C-V2X communication link is restored, the system switches back again.

6. The multimodal sensing and low-latency communication method for the mountain-resistant vehicle-road cooperative terminal according to claim 5, characterized in that: Determine whether the quality of the C-V2X communication link is below a preset threshold using the following method: Calculate the Link Quality Index (LQI); if the C-V2X LQI is lower than a preset threshold, then the C-V2X communication link quality is lower than the preset threshold; the Link Quality Index (LQI) is calculated according to the following formula: ； in, , as well as All are set coefficients. The packet loss rate for C-V2X; For C-V2X latency; This represents the maximum permissible latency for C-V2X. This represents the signal reception power.

7. The multimodal sensing and low-latency communication method for the mountain-resistant vehicle-road cooperative terminal according to claim 2, characterized in that: Real-time monitoring of the sensor unit's attitude change caused by vibration using an attitude sensor triggers online self-calibration, specifically including: Real-time acquisition of pitch angle, roll angle and vibration acceleration data of the sensing unit; The current attitude parameters are compared with the preset reference attitude parameters to calculate the attitude offset. When the pitch angle change or roll angle change exceeds the preset angle threshold, it is determined that the vibration has caused the installation state of the sensing unit to shift, and the visual-inertial navigation joint calibration algorithm is triggered. The joint visual-inertial navigation calibration algorithm includes: An initial extrinsic parameter matrix between the camera and the millimeter-wave radar is constructed based on attitude data, and a spatial transformation relationship is established under a unified coordinate system. Acquire visible light or infrared images and corresponding millimeter-wave radar point cloud data; extract stationary rigid feature points from the visible light or infrared images and generate image feature descriptors; Extract the corresponding static target point cloud features from millimeter-wave radar point cloud data and establish the corresponding matching relationship between image feature points and point cloud feature points; A reprojection error function is constructed based on the matching results, and the reprojection error function is iteratively optimized to minimize the error between the image observation position and the point cloud projection position. The camera's intrinsic parameters, the millimeter-wave radar's extrinsic parameters, and the relative pose relationship between them are updated based on the optimization results. The updated calibration parameters are written into the terminal configuration storage area, and the spatial mapping relationship of the multi-source sensing data is re-established using the updated calibration parameters, thereby completing the real-time online calibration of the sensing unit.

8. The multimodal sensing and low-latency communication method for a mountain-resistant vehicle-road cooperative terminal according to claim 1, characterized in that: The traffic and road early warning information includes blind spot vehicle approach warning information, road freezing warning information, slope passing warning information, road rockfall warning information, abnormal vehicle parking warning information, and road congestion warning information.

Images