Highway vehicle-road cooperation data management method and system based on multi-modal perception

Abstract

The application provides a highway vehicle-road cooperation data management method and system based on multi-modal perception. The method comprises: obtaining a multi-modal vehicle-road cooperation data set and preprocessing to obtain a multi-modal vehicle-road cooperation standard data set, then data extraction to obtain a vehicle-road cooperation effective feature data set, dynamic fusion to generate a vehicle-road cooperation three-dimensional target feature graph, analysis combined with a constructed twin model to obtain road network state change prediction data within a preset time period, generation of lane-level decision data combined with the vehicle-road cooperation three-dimensional target feature graph, and pushing to the corresponding vehicle end, while performing blockchain storage and data access verification on the vehicle-road cooperation effective feature data set; the application covers all scenarios of data collection through adaptive multi-modal perception scheduling, feature-level dynamic fusion, digital twin mapping and blockchain storage technology, improves data fusion effectiveness, and thus realizes effective management of vehicle-road cooperation data.

Description

Technical Field

[0001] This application relates to the field of highway vehicle-road cooperative and data governance technology, and more specifically, to a highway vehicle-road cooperative data governance method and system based on multimodal perception. Background Technology

[0002] As a core support for intelligent transportation systems, vehicle-road cooperative technology faces significant limitations with traditional data acquisition techniques. Single sensors are greatly affected by the environment and have blind spots, such as camera recognition rates of less than 50% in heavy rain, and the attenuation of LiDAR signals in tunnels, failing to cover long straight roads, sharp curves, tunnels, and service areas. Data fusion processing adopts a post-fusion mode, which processes data from each device separately before stitching them together, resulting in high fusion latency. Data sharing relies on static encryption and lacks a full-process traceability mechanism, making it susceptible to data tampering or privacy leaks. Therefore, there is an urgent need for a highway vehicle-road cooperative data governance method that can achieve full-scene perception, dynamic fusion, and secure control.

[0003] Effective technical solutions are urgently needed to address the above problems. Summary of the Invention

[0004] The purpose of this application is to provide a method and system for highway vehicle-road cooperative data governance based on multimodal perception. It can cover the entire data collection scenario through adaptive multimodal perception scheduling, feature-level dynamic fusion, digital twin mapping and blockchain storage technology, improve the effectiveness of data fusion, and thus achieve effective governance of vehicle-road cooperative data.

[0005] Firstly, this application provides a method for highway vehicle-road cooperative data governance based on multimodal perception, including the following steps:

[0006] Obtain a multimodal vehicle-road cooperative dataset and perform data preprocessing to obtain a standard multimodal vehicle-road cooperative dataset;

[0007] Data is extracted from the multimodal vehicle-road cooperative standard dataset to obtain an effective feature dataset for vehicle-road cooperation;

[0008] Dynamic fusion processing is performed on the effective feature dataset of vehicle-road cooperation to generate a three-dimensional target feature map of vehicle-road cooperation.

[0009] Historical vehicle-road cooperative 3D target feature maps and static road network data are acquired to construct a twin model, thereby obtaining a vehicle-road cooperative twin model. The vehicle-road cooperative 3D target feature maps are then analyzed and processed to obtain road network state change prediction data within a preset time period.

[0010] The vehicle-road cooperative three-dimensional target feature map is analyzed and processed in conjunction with the road network state change prediction data to generate lane-level decision data, which is then pushed to the corresponding vehicle terminal.

[0011] Optionally, in the highway vehicle-road cooperative data governance method based on multimodal perception described in this application, the step of acquiring a multimodal vehicle-road cooperative dataset and performing data preprocessing to obtain a multimodal vehicle-road cooperative standard dataset includes:

[0012] Acquire a multimodal vehicle-road cooperative dataset, including meteorological record data, roadside perception record data, and vehicle perception record data. Among them, the vehicle perception record data includes real-time vehicle speed, braking status feature data, steering angle, and vehicle positioning data.

[0013] Based on the meteorological record data, the roadside sensing record data is filtered and processed to obtain valid roadside sensing record data;

[0014] Data is extracted from the valid data recorded by the roadside perception to obtain three-dimensional road structure data, vehicle identity data, and vehicle motion trajectory data.

[0015] The road 3D structure data, vehicle identity data, vehicle motion trajectory data, real-time vehicle speed, braking status feature data, steering angle, and vehicle positioning data are preprocessed by standardizing the format, spatiotemporal calibration, and data quality inspection to obtain a multimodal vehicle-road cooperative standard dataset.

[0016] Optionally, the highway vehicle-road cooperative data governance method based on multimodal perception described in this application further includes:

[0017] The road 3D structure data, vehicle identity data, vehicle motion trajectory data, real-time vehicle speed, braking status feature data, steering angle, and vehicle positioning data are input into a preset data quality target detection model for analysis and processing to obtain the corresponding data accuracy.

[0018] The data accuracy rate is compared with a preset data accuracy threshold.

[0019] If the data accuracy is less than a preset data accuracy threshold, it is marked as abnormal data and removed.

[0020] If the data accuracy is greater than or equal to a preset data accuracy threshold, then vehicle positioning error analysis is performed to obtain the vehicle positioning error.

[0021] The vehicle positioning error is compared with a preset vehicle positioning error threshold.

[0022] If the vehicle positioning error is greater than the preset vehicle positioning error threshold, the sensing device self-test is triggered.

[0023] If the vehicle positioning error is less than or equal to a preset vehicle positioning error threshold, then data integrity and timeliness analysis are performed to obtain a qualified integrity status and a qualified timeliness status.

[0024] Perform a bitwise AND operation on the integrity and timeliness compliance status. If the result is compliant, the multimodal vehicle-road cooperative standard dataset is determined to be valid. If the result is non-compliant, the multimodal vehicle-road cooperative standard dataset is determined to be invalid, and a warning response is output.

[0025] Optionally, in the highway vehicle-road cooperative data governance method based on multimodal perception described in this application, the step of extracting data from the multimodal vehicle-road cooperative standard dataset to obtain an effective vehicle-road cooperative feature dataset includes:

[0026] Data is extracted from the multimodal vehicle-road cooperative standard dataset to obtain a vehicle-road cooperative effective feature dataset, including vehicle effective feature data, traffic environment effective feature data, and effective feature association data;

[0027] The effective vehicle feature data includes effective vehicle identity feature data, effective vehicle status feature data, effective vehicle location feature data, and effective vehicle behavior feature data;

[0028] The effective traffic environment feature data includes effective meteorological environment feature data and effective road condition feature data.

[0029] Optionally, in the highway vehicle-road cooperative data governance method based on multimodal perception described in this application, the step of dynamically fusing the effective feature dataset of vehicle-road cooperation to generate a three-dimensional target feature map of vehicle-road cooperation includes:

[0030] The effective feature dataset of vehicle-road cooperation is input into a preset vehicle-road cooperation scenario evaluation model for analysis and processing to obtain vehicle-road cooperation scenario type feature data, including overtaking scenario type feature data, congestion type feature data, tunnel type feature data, accident type feature data, rain and fog type feature data, or service area entrance and exit type feature data.

[0031] Based on the vehicle-road cooperative scenario type feature data, query the preset data fusion weight value list to obtain the LiDAR data weight value, millimeter-wave radar data weight value, and camera data weight value;

[0032] The effective feature dataset of vehicle-road cooperation is normalized and then weighted and fused with the weight values ​​of the lidar data, millimeter-wave radar data, and camera data to generate a three-dimensional target feature map of vehicle-road cooperation.

[0033] Optionally, in the highway vehicle-road cooperative data governance method based on multimodal perception described in this application, the step of acquiring historical vehicle-road cooperative three-dimensional target feature maps and static road network data to construct a twin model, obtaining a vehicle-road cooperative twin model, and combining it with the vehicle-road cooperative three-dimensional target feature maps for analysis and processing to obtain road network state change prediction data within a preset time period includes:

[0034] Acquire historical vehicle-road cooperative 3D target feature maps and static road network data, whereby the static road network data includes road geometric parameter data and road static feature data;

[0035] A digital twin model is constructed based on the road geometric parameter data and road static feature data to generate an initial twin model for vehicle-road cooperation.

[0036] The historical vehicle-road cooperative 3D target feature map is input into the vehicle-road cooperative initial twin model for initialization to obtain the vehicle-road cooperative twin model;

[0037] The vehicle-road cooperative three-dimensional target feature map is input into the vehicle-road cooperative twin model for analysis and processing to obtain road network status change prediction data within a preset time period, including vehicle dynamic data, road surface status data, meteorological dynamic data, and equipment operation dynamic data.

[0038] Optionally, in the highway vehicle-road cooperative data governance method based on multimodal perception described in this application, the step of analyzing and processing the vehicle-road cooperative three-dimensional target feature map in conjunction with the road network state change prediction data to generate lane-level decision data and pushing it to the corresponding vehicle terminal includes:

[0039] Based on the vehicle-road cooperative three-dimensional target feature map, the speed limit and congestion critical density of the preset road segment are extracted. Based on the road network state change prediction data, the average vehicle speed and traffic flow density of the preset road segment are extracted. Based on the speed limit, congestion critical density, average vehicle speed and traffic flow density, the traffic efficiency score corresponding to each lane is obtained.

[0040] The effective meteorological environment characteristic data and the effective road condition characteristic data are input into a preset traffic environment type assessment model for analysis and processing to obtain traffic environment type characteristic data.

[0041] Based on the traffic environment type characteristic data, query the preset traffic efficiency correction coefficient list to obtain the traffic efficiency correction coefficient;

[0042] The traffic efficiency score is corrected according to the traffic efficiency correction coefficient to obtain the traffic efficiency optimization value, and then sorted in descending order. The lane with the highest traffic efficiency optimization value is pushed to the ordinary vehicle end.

[0043] The vehicle-road cooperative three-dimensional target feature map and the road network state change prediction data are input into a preset multimodal over-limit analysis model for analysis and processing to obtain weight over-limit judgment results and size over-limit judgment results;

[0044] If the weight exceeds the limit and the size exceeds the limit, a Level 1 warning will be issued to the freight vehicle.

[0045] If the weight exceeds the limit or the size exceeds the limit, a level 2 warning will be issued to the freight vehicle.

[0046] Optionally, the highway vehicle-road cooperative data governance method based on multimodal perception described in this application further includes:

[0047] Based on the vehicle-road cooperative effective feature dataset at a preset time point, a corresponding data fingerprint is generated, including a data hash value and evidence storage metadata;

[0048] The data hash value and evidence metadata are uploaded to a distributed storage node with a preset data volume, and the data is verified using a preset Byzantine fault-tolerant algorithm.

[0049] If the data verification is successful, the sensitive fields of the vehicle-road cooperative effective feature dataset are obtained, and the sensitivity level is determined according to the sensitive fields. According to the sensitivity level, preset noise is injected through a preset differential privacy algorithm and written into a preset block. The hash value of the vehicle-road cooperative effective feature dataset at the time point before the preset time point is also associated.

[0050] If data verification fails, a data alert will be triggered.

[0051] The system performs identity verification, permission matching, and data sensitivity checks based on the data access instructions.

[0052] If the verification passes, an access token is generated, and access data is obtained through the data node based on the access token;

[0053] If the verification fails, an abnormal access log will be generated and an alert response will be output.

[0054] Secondly, this application provides a highway vehicle-road cooperative data governance system based on multimodal perception. The system includes a memory and a processor. The memory includes a program for a highway vehicle-road cooperative data governance method based on multimodal perception. When the program for the highway vehicle-road cooperative data governance method based on multimodal perception is executed by the processor, it implements the following steps:

[0055] Obtain a multimodal vehicle-road cooperative dataset and perform data preprocessing to obtain a standard multimodal vehicle-road cooperative dataset;

[0056] Data is extracted from the multimodal vehicle-road cooperative standard dataset to obtain an effective feature dataset for vehicle-road cooperation;

[0057] Dynamic fusion processing is performed on the effective feature dataset of vehicle-road cooperation to generate a three-dimensional target feature map of vehicle-road cooperation.

[0058] Historical vehicle-road cooperative 3D target feature maps and static road network data are acquired to construct a twin model, thereby obtaining a vehicle-road cooperative twin model. The vehicle-road cooperative 3D target feature maps are then analyzed and processed to obtain road network state change prediction data within a preset time period.

[0059] The vehicle-road cooperative three-dimensional target feature map is analyzed and processed in conjunction with the road network state change prediction data to generate lane-level decision data, which is then pushed to the corresponding vehicle terminal.

[0060] Optionally, in the multimodal perception-based highway vehicle-road cooperative data governance system described in this application, the step of acquiring a multimodal vehicle-road cooperative dataset and performing data preprocessing to obtain a multimodal vehicle-road cooperative standard dataset includes:

[0061] Acquire a multimodal vehicle-road cooperative dataset, including meteorological record data, roadside perception record data, and vehicle perception record data. Among them, the vehicle perception record data includes real-time vehicle speed, braking status feature data, steering angle, and vehicle positioning data.

[0062] Based on the meteorological record data, the roadside sensing record data is filtered and processed to obtain valid roadside sensing record data;

[0063] Data is extracted from the valid data recorded by the roadside perception to obtain three-dimensional road structure data, vehicle identity data, and vehicle motion trajectory data.

[0064] The road 3D structure data, vehicle identity data, vehicle motion trajectory data, real-time vehicle speed, braking status feature data, steering angle, and vehicle positioning data are preprocessed by standardizing the format, spatiotemporal calibration, and data quality inspection to obtain a multimodal vehicle-road cooperative standard dataset.

[0065] As can be seen from the above, the highway vehicle-road cooperative data governance method and system based on multimodal perception provided in this application, through adaptive multimodal perception scheduling, feature-level dynamic fusion, digital twin mapping and blockchain storage technology, covers the entire data collection scenario, improves the effectiveness of data fusion, and thus realizes effective governance of vehicle-road cooperative data.

[0066] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings. Attached Figure Description

[0067] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 A flowchart illustrating a highway vehicle-road cooperative data governance method based on multimodal perception, provided in an embodiment of this application;

[0069] Figure 2 A flowchart illustrating the process of obtaining a multimodal vehicle-road cooperative standard dataset for a highway vehicle-road cooperative data governance method based on multimodal perception, as provided in this application embodiment.

[0070] Figure 3 A flowchart illustrating the generation of a three-dimensional target feature map for a highway vehicle-road cooperative data governance method based on multimodal perception, as provided in this application embodiment;

[0071] Figure 4 A flowchart illustrating the process of obtaining predicted road network state changes within a preset time period using a multimodal perception-based highway vehicle-road cooperative data governance method provided in this application embodiment;

[0072] Figure 5 A high-level flowchart of the highway vehicle-road cooperative data governance method based on multimodal perception provided in the embodiments of this application. Detailed Implementation

[0073] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0074] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0075] Please refer to Figure 1 , Figure 1 This is a flowchart of a highway vehicle-road cooperative data governance method based on multimodal perception, according to some embodiments of this application. This highway vehicle-road cooperative data governance method based on multimodal perception is used in terminal devices, such as computers and mobile terminals. The highway vehicle-road cooperative data governance method based on multimodal perception includes the following steps:

[0076] S11. Obtain the multimodal vehicle-road cooperative dataset and perform data preprocessing to obtain the multimodal vehicle-road cooperative standard dataset.

[0077] S12. Extract data from the multimodal vehicle-road cooperative standard dataset to obtain a vehicle-road cooperative effective feature dataset;

[0078] S13. Dynamically fuse the vehicle-road cooperative effective feature dataset to generate a vehicle-road cooperative three-dimensional target feature map.

[0079] S14. Obtain historical vehicle-road cooperative three-dimensional target feature map and static road network data to construct a twin model, obtain a vehicle-road cooperative twin model, and combine it with the vehicle-road cooperative three-dimensional target feature map for analysis and processing to obtain road network status change prediction data within a preset time period.

[0080] S15. Analyze and process the three-dimensional target feature map of vehicle-road cooperation combined with the road network state change prediction data to generate lane-level decision data, and push it to the corresponding vehicle terminal.

[0081] Further explanation is needed. In order to achieve comprehensive collection and accurate governance of vehicle-road cooperative data on highways, firstly, a multimodal vehicle-road cooperative dataset is collected based on adaptive acquisition, and then filtered and preprocessed for data extraction. Secondly, dynamic fusion processing is performed based on the data collection scenario to generate a three-dimensional target feature map of vehicle-road cooperative data. Finally, simulation processing is performed through a constructed digital twin model to generate lane-level decision data, which is then pushed to different types of vehicles. At the same time, a dual security mechanism of blockchain and differential privacy is adopted to achieve full-scene perception, dynamic fusion, and data security.

[0082] Please refer to Figure 2 , Figure 2This is a flowchart illustrating the process of obtaining a multimodal vehicle-road cooperative standard dataset in some embodiments of the highway vehicle-road cooperative data governance method based on multimodal perception. According to embodiments of the present invention, obtaining the multimodal vehicle-road cooperative dataset and performing data preprocessing to obtain the multimodal vehicle-road cooperative standard dataset includes:

[0083] S21. Obtain a multimodal vehicle-road cooperative dataset, including meteorological record data, roadside perception record data, and vehicle perception record data. Among them, the vehicle perception record data includes real-time vehicle speed, braking status feature data, steering angle, and vehicle positioning data.

[0084] S22. Based on the meteorological record data, the roadside sensing record data is filtered and processed to obtain valid roadside sensing record data;

[0085] S23. Extract data from the valid data recorded by the roadside perception to obtain road three-dimensional structure data, vehicle identity data and vehicle motion trajectory data;

[0086] S24. Perform format unification, spatiotemporal calibration, and data quality inspection preprocessing on the road three-dimensional structure data, vehicle identity data, vehicle motion trajectory data, real-time vehicle speed, braking status feature data, steering angle, and vehicle positioning data to obtain a multimodal vehicle-road cooperative standard dataset.

[0087] Further explanation is needed. To achieve adaptive multimodal data acquisition and improve data applicability, firstly, full-scene perception data, including meteorological records, roadside perception records, and vehicle-mounted perception records, is collected using LiDAR, millimeter-wave radar, and high-definition cameras. Then, the roadside perception records are filtered based on the meteorological records to obtain valid roadside perception records. For example, the goal of rain and fog weather data acquisition is to resist rain and fog interference. Raindrops and fog scatter light, reducing the contrast of ordinary camera images. Rainwater adheres to the lens surface, forming a water film that obstructs part of the field of view and creates perception blind spots. Data collected by infrared cameras and millimeter-wave radar is filtered, and then the filtered data is extracted and preprocessed to further improve the usability of the data.

[0088] According to an embodiment of the present invention, it further includes:

[0089] The road 3D structure data, vehicle identity data, vehicle motion trajectory data, real-time vehicle speed, braking status feature data, steering angle, and vehicle positioning data are input into a preset data quality target detection model for analysis and processing to obtain the corresponding data accuracy.

[0090] The data accuracy rate is compared with a preset data accuracy threshold.

[0091] If the data accuracy is less than a preset data accuracy threshold, it is marked as abnormal data and removed.

[0092] If the data accuracy is greater than or equal to a preset data accuracy threshold, then vehicle positioning error analysis is performed to obtain the vehicle positioning error.

[0093] The vehicle positioning error is compared with a preset vehicle positioning error threshold.

[0094] If the vehicle positioning error is greater than the preset vehicle positioning error threshold, the sensing device self-test is triggered.

[0095] If the vehicle positioning error is less than or equal to a preset vehicle positioning error threshold, then data integrity and timeliness analysis are performed to obtain a qualified integrity status and a qualified timeliness status.

[0096] Perform a bitwise AND operation on the integrity and timeliness compliance status. If the result is compliant, the multimodal vehicle-road cooperative standard dataset is determined to be valid. If the result is non-compliant, the multimodal vehicle-road cooperative standard dataset is determined to be invalid, and a warning response is output.

[0097] Further explanation is needed regarding the data preprocessing process. First, the accuracy of the data is evaluated using a pre-trained model, and data accuracy is determined by threshold comparison. Data below the threshold is discarded. Then, the vehicle-mounted BeiDou module outputs WGS84 coordinates in real time, which are converted to a local coordinate system consistent with the roadside equipment using a coordinate transformation algorithm (such as seven-parameter transformation) to evaluate vehicle positioning errors, i.e., spatial calibration. Finally, the roadside equipment accesses the NTP service via Ethernet, and the vehicle-mounted equipment accesses it via a 5G network, sending a time synchronization request every 100ms. The receiving end dynamically adjusts its local time based on network latency to ensure that all data timestamp deviations are correct, i.e., the timeliness is deemed acceptable. Simultaneously, data integrity is verified. Only when both timeliness and integrity meet the requirements is the multimodal vehicle-road cooperative standard dataset deemed valid. The preset data quality target detection model is trained by acquiring a large amount of historical sample road 3D structure data, vehicle identity data, vehicle motion trajectory data, real-time vehicle speed, braking status feature data, steering angle and vehicle positioning data, and the corresponding data accuracy.

[0098] According to an embodiment of the present invention, data extraction is performed on the multimodal vehicle-road cooperative standard dataset to obtain a vehicle-road cooperative effective feature dataset, including:

[0099] Data is extracted from the multimodal vehicle-road cooperative standard dataset to obtain a vehicle-road cooperative effective feature dataset, including vehicle effective feature data, traffic environment effective feature data, and effective feature association data;

[0100] The effective vehicle feature data includes effective vehicle identity feature data, effective vehicle status feature data, effective vehicle location feature data, and effective vehicle behavior feature data;

[0101] The effective traffic environment feature data includes effective meteorological environment feature data and effective road condition feature data.

[0102] It needs to be further explained that high-value data that can directly support subsequent fusion computing and intelligent decision-making are extracted from the multimodal raw data. The effective feature dataset for vehicle-road cooperation is extracted from the raw data of lidar, millimeter-wave radar and cameras, and is used to accurately identify vehicle identity, status and behavior. Examples of extraction are shown in Table 1.

[0103] Table 1

[0104] Feature subtype Extraction source Specific data form Extraction logic Effective feature data of vehicle identity High-definition camera (infrared mode) License plate number (such as "Zhe A12345"), vehicle type (sedan / truck / bus / emergency vehicle), body color (black / white) Through the "license plate recognition module" and "vehicle type classification module" of the lightweight CNN model (such as the streamlined version of MobileNetV3), locate the license plate area and recognize characters from the camera image, and judge the vehicle type by combining the body contour Effective feature data of vehicle status Millimeter-wave radar Real-time speed (110.5 km / h), acceleration (0.8 m / s², positive for acceleration and negative for deceleration), driving direction (heading angle 85°) The millimeter-wave radar calculates the radial speed of the vehicle through the Doppler effect, obtains the acceleration by combining the difference of consecutive frame data, and judges the driving direction of the vehicle through the antenna array Effective feature data of vehicle position Lidar + millimeter-wave radar Absolute position (latitude and longitude: 30.123456°N, 120.654321°E; altitude: 100.5 m), relative position (distance from the vehicle in front: 150.3 m, lane number: the first lane on the left) The lidar obtains the absolute position by point cloud matching with the preset coordinate system of the road, the millimeter-wave radar measures the distance from adjacent vehicles, and determines the lane number by combining the recognition result of the camera lane line Effective feature data of vehicle behavior Camera + millimeter-wave radar Turn signal status (left turn / right turn / no turn), brake status (brake light on / off), lane change intention (such as continuous deviation to the left for 2 seconds) The camera recognizes the vehicle light status (brightness change of the brake light / turn signal), and the millimeter-wave radar monitors the lateral displacement of the vehicle (such as a lateral displacement > 1.5 meters within 5 seconds is judged as a lane change intention)

[0105] Effective traffic environment feature data is extracted from raw data from meteorological sensors, cameras, and lidar. It is used to determine whether the current traffic environment is safe. Effective meteorological environment feature data includes weather type (rainy, foggy, or sunny), visibility (150m), road surface condition (dry, waterlogged, or icy), and precipitation. Effective road condition feature data includes lane line integrity feature data (lane line integrity includes complete, damaged, or missing), road obstacles (e.g., construction cones 1 km ahead, coordinates: 30.124567°N, 120.655432°E), and tunnel entrance / exit signs. Effective feature correlation data involves structured compression of vehicle and traffic environment effective feature data to preserve the correlation between data (e.g., the correlation between a vehicle's speed and the corresponding meteorological environment) and avoid data fragmentation.

[0106] Please refer to Figure 3 , Figure 3 This is a flowchart illustrating the generation of a vehicle-road cooperative three-dimensional target feature map in a highway vehicle-road cooperative data governance method based on multimodal perception, as described in some embodiments of this application. According to an embodiment of the present invention, the step of dynamically fusing the vehicle-road cooperative effective feature dataset to generate a vehicle-road cooperative three-dimensional target feature map includes:

[0107] S31. Input the effective feature dataset of vehicle-road cooperation into the preset vehicle-road cooperation scenario evaluation model for analysis and processing to obtain vehicle-road cooperation scenario type feature data, including overtaking scenario type feature data, congestion type feature data, tunnel type feature data, accident type feature data, rain and fog type feature data, or service area entrance and exit type feature data.

[0108] S32. Query the preset data fusion weight value list according to the vehicle-road cooperative scenario type feature data to obtain the lidar data weight value, millimeter-wave radar data weight value and camera data weight value;

[0109] S33. Normalize the effective feature dataset of vehicle-road cooperation, and perform weighted fusion processing by combining the weight values ​​of the lidar data, millimeter-wave radar data, and camera data to generate a three-dimensional target feature map of vehicle-road cooperation.

[0110] Further explanation is needed. To achieve dynamic fusion processing across different scenarios, firstly, the extracted effective feature dataset of vehicle-road cooperation is analyzed and processed through a preset vehicle-road cooperation scenario evaluation model to obtain vehicle-road cooperation scenario type feature data. Then, a preset data fusion weight value list is queried to obtain the corresponding dynamic weight values. Finally, after data normalization, weighted fusion is performed to generate a 3D target feature map of vehicle-road cooperation. The preset vehicle-road cooperation scenario evaluation model is obtained by training with a large number of historical samples of effective feature datasets of vehicle-road cooperation and corresponding vehicle-road cooperation scenario type feature data. The preset data fusion weight value list is analyzed and pre-constructed through a large number of historical samples and can be dynamically adjusted according to specific applications.

[0111] Please refer to Figure 4 , Figure 4 This is a flowchart illustrating the process of obtaining predicted road network state change data within a preset time period using a multimodal perception-based highway vehicle-road cooperative data governance method, as described in some embodiments of this application. According to embodiments of the present invention, the step of acquiring historical vehicle-road cooperative three-dimensional target feature maps and static road network data to construct a twin model, obtaining a vehicle-road cooperative twin model, and then analyzing and processing the vehicle-road cooperative three-dimensional target feature maps to obtain predicted road network state change data within a preset time period includes:

[0112] S41. Obtain historical vehicle-road cooperative three-dimensional target feature maps and static road network data, wherein the static road network data includes road geometric parameter data and road static feature data;

[0113] S42. Construct a digital twin model based on the road geometric parameter data and road static feature data to generate an initial twin model for vehicle-road cooperation;

[0114] S43. Input the historical vehicle-road cooperative three-dimensional target feature map into the vehicle-road cooperative initial twin model for initialization to obtain the vehicle-road cooperative twin model;

[0115] S44. Input the vehicle-road cooperative three-dimensional target feature map into the vehicle-road cooperative twin model for analysis and processing to obtain road network status change prediction data within a preset time period, including vehicle dynamic data, road surface status data, meteorological dynamic data and equipment operation dynamic data.

[0116] Further explanation is needed. In order to achieve intelligent decision-making through real-time scene simulation, firstly, static road network modeling and model initialization are performed. Then, based on real-time fusion processing, a three-dimensional target feature map of vehicle-road cooperation is generated, which includes four types of dynamic data: vehicle dynamic data, road surface condition data, meteorological dynamic data, and equipment operation dynamic data. Among them, vehicle dynamic data is such as truck ID: truck_007, location: 30.123456°N, 120.654321°E, lane 3, speed 90km / h; road surface condition data is such as icing 500 meters from the tunnel exit, coverage 30%; meteorological dynamic data is such as visibility 1500 meters, precipitation 0.2mm / min; and equipment operation dynamic data is such as RSU-008 lidar is normal, and camera infrared mode is on.

[0117] According to an embodiment of the present invention, the step of analyzing and processing the vehicle-road cooperative three-dimensional target feature map in conjunction with the road network state change prediction data to generate lane-level decision data and pushing it to the corresponding vehicle terminal includes:

[0118] Based on the vehicle-road cooperative three-dimensional target feature map, the speed limit and congestion critical density of the preset road segment are extracted. Based on the road network state change prediction data, the average vehicle speed and traffic flow density of the preset road segment are extracted. Based on the speed limit, congestion critical density, average vehicle speed and traffic flow density, the traffic efficiency score corresponding to each lane is obtained.

[0119] The effective meteorological environment characteristic data and the effective road condition characteristic data are input into a preset traffic environment type assessment model for analysis and processing to obtain traffic environment type characteristic data.

[0120] Based on the traffic environment type characteristic data, query the preset traffic efficiency correction coefficient list to obtain the traffic efficiency correction coefficient;

[0121] The traffic efficiency score is corrected according to the traffic efficiency correction coefficient to obtain the traffic efficiency optimization value, and then sorted in descending order. The lane with the highest traffic efficiency optimization value is pushed to the ordinary vehicle end.

[0122] The vehicle-road cooperative three-dimensional target feature map and the road network state change prediction data are input into a preset multimodal over-limit analysis model for analysis and processing to obtain weight over-limit judgment results and size over-limit judgment results;

[0123] If the weight exceeds the limit and the size exceeds the limit, a Level 1 warning will be issued to the freight vehicle.

[0124] If the weight exceeds the limit or the size exceeds the limit, a level 2 warning will be issued to the freight vehicle.

[0125] Further explanation is needed regarding the accurate output of lane-level intelligent decisions. First, the speed limit and congestion critical density of the preset road segment are extracted based on the vehicle-road cooperative three-dimensional target feature map. Then, the average vehicle speed and traffic flow density of the preset road segment are extracted based on road network state change prediction data. The traffic efficiency score for each lane is calculated as (average vehicle speed / speed limit) * 0.7 - (traffic flow density / congestion critical density) * 0.3. A traffic efficiency correction coefficient is then queried from real-time assessed traffic environment type feature data and multiplied by the traffic efficiency score for rate correction, used to evaluate the traffic efficiency of each lane and improve assessment accuracy. Next, the lane with the highest traffic efficiency is pushed to ordinary vehicles. Simultaneously, intelligent assessment is performed for weight and size over-limit warnings that are prone to occur with trucks. A Level 1 warning is output when both weight and size exceed limits; a Level 2 warning is output when either weight or size exceeds limits, with Level 1 warnings being higher than Level 2 warnings. The warning system includes traffic environment types such as rainy / foggy weather (traffic efficiency correction factor of 0.7), road icing (traffic efficiency correction factor of 0.6), road flooding (traffic efficiency correction factor of 0.6), or temporary construction (traffic efficiency correction factor of 0). Traffic environment type characteristic data are represented by unique identifiers. The preset traffic environment type assessment model is obtained by training with a large amount of historical samples of effective meteorological environment characteristic data, effective road condition characteristic data, and corresponding traffic environment type characteristic data. The preset multimodal over-limit analysis model is obtained by training with a large amount of historical samples of vehicle-road cooperative three-dimensional target feature maps, road network condition change prediction data, and corresponding weight over-limit judgment results and size over-limit judgment results. The preset traffic efficiency correction factor list is constructed by those skilled in the art based on the analysis of a large amount of historical samples and can be modified according to specific applications.

[0126] According to an embodiment of the present invention, it further includes:

[0127] Based on the vehicle-road cooperative effective feature dataset at a preset time point, a corresponding data fingerprint is generated, including a data hash value and evidence storage metadata;

[0128] The data hash value and evidence metadata are uploaded to a distributed storage node with a preset data volume, and the data is verified using a preset Byzantine fault-tolerant algorithm.

[0129] If the data verification is successful, the sensitive fields of the vehicle-road cooperative effective feature dataset are obtained, and the sensitivity level is determined according to the sensitive fields. According to the sensitivity level, preset noise is injected through a preset differential privacy algorithm and written into a preset block. The hash value of the vehicle-road cooperative effective feature dataset at the time point before the preset time point is also associated.

[0130] If data verification fails, a data alert will be triggered.

[0131] The system performs identity verification, permission matching, and data sensitivity checks based on the data access instructions.

[0132] If the verification passes, an access token is generated, and access data is obtained through the data node based on the access token;

[0133] If the verification fails, an abnormal access log will be generated and an alert response will be output.

[0134] Further explanation is needed regarding data security protection. To address the core pain points of difficulty in tracing data tampering and a one-size-fits-all approach to privacy protection, a source hashing and evidence storage method is used during the data collection phase to generate data fingerprints. Simultaneously, sensitive fields and their corresponding sensitivity levels are obtained from the effective feature dataset of vehicle-road cooperative systems. For example, precise vehicle trajectories correspond to a high sensitivity level, requiring global differential privacy and Gaussian noise injection; vehicle weight corresponds to a medium sensitivity level, requiring local differential privacy and Laplace noise injection; and average road segment speed corresponds to a low sensitivity level, requiring no noise injection, only outlier removal. A differential privacy algorithm is employed to protect vehicle data without affecting its statistical characteristics. Sensitive information about personnel is stored on a distributed blockchain and then linked to the hash value of the vehicle-road cooperative effective feature dataset from a previous time point. If any node tamperes with a block of data, the hash verification of all subsequent blocks will fail, ensuring that the data is immutable. During the data access phase, identity legitimacy, permission matching, and data sensitivity are verified. Identity legitimacy verifies whether the digital certificate signature is valid and within its validity period. Permission matching checks whether the access request is within the scope of the subject's permissions. Data sensitivity requires additional verification of a special authorization code (temporarily issued by the operating unit) if accessing sensitive data (such as emergency vehicle trajectories).

[0135] Please refer to Figure 5 , Figure 5 This is a high-level flowchart of some embodiments of this application, used to implement a highway vehicle-road cooperative data governance method based on multimodal perception.

[0136] This invention also discloses a highway vehicle-road cooperative data governance system based on multimodal perception, comprising a memory and a processor. The memory includes a program for a highway vehicle-road cooperative data governance method based on multimodal perception. When the processor executes the program for the highway vehicle-road cooperative data governance method based on multimodal perception, it performs the following steps:

[0137] Obtain a multimodal vehicle-road cooperative dataset and perform data preprocessing to obtain a standard multimodal vehicle-road cooperative dataset;

[0138] Data is extracted from the multimodal vehicle-road cooperative standard dataset to obtain an effective feature dataset for vehicle-road cooperation;

[0139] Dynamic fusion processing is performed on the effective feature dataset of vehicle-road cooperation to generate a three-dimensional target feature map of vehicle-road cooperation.

[0140] Historical vehicle-road cooperative 3D target feature maps and static road network data are acquired to construct a twin model, thereby obtaining a vehicle-road cooperative twin model. The vehicle-road cooperative 3D target feature maps are then analyzed and processed to obtain road network state change prediction data within a preset time period.

[0141] The vehicle-road cooperative three-dimensional target feature map is analyzed and processed in conjunction with the road network state change prediction data to generate lane-level decision data, which is then pushed to the corresponding vehicle terminal.

[0142] Further explanation is needed. In order to achieve comprehensive collection and accurate governance of vehicle-road cooperative data on highways, firstly, a multimodal vehicle-road cooperative dataset is collected based on adaptive acquisition, and then filtered and preprocessed for data extraction. Secondly, dynamic fusion processing is performed based on the data collection scenario to generate a three-dimensional target feature map of vehicle-road cooperative data. Finally, simulation processing is performed through a constructed digital twin model to generate lane-level decision data, which is then pushed to different types of vehicles. At the same time, a dual security mechanism of blockchain and differential privacy is adopted to achieve full-scene perception, dynamic fusion, and data security.

[0143] According to an embodiment of the present invention, the step of acquiring a multimodal vehicle-road cooperative dataset and performing data preprocessing to obtain a multimodal vehicle-road cooperative standard dataset includes:

[0144] Acquire a multimodal vehicle-road cooperative dataset, including meteorological record data, roadside perception record data, and vehicle perception record data. Among them, the vehicle perception record data includes real-time vehicle speed, braking status feature data, steering angle, and vehicle positioning data.

[0145] Based on the meteorological record data, the roadside sensing record data is filtered and processed to obtain valid roadside sensing record data;

[0146] Data is extracted from the valid data recorded by the roadside perception to obtain three-dimensional road structure data, vehicle identity data, and vehicle motion trajectory data.

[0147] The road 3D structure data, vehicle identity data, vehicle motion trajectory data, real-time vehicle speed, braking status feature data, steering angle, and vehicle positioning data are preprocessed by standardizing the format, spatiotemporal calibration, and data quality inspection to obtain a multimodal vehicle-road cooperative standard dataset.

[0148] Further explanation is needed. To achieve adaptive multimodal data acquisition and improve data applicability, firstly, full-scene perception data, including meteorological records, roadside perception records, and vehicle-mounted perception records, is collected using LiDAR, millimeter-wave radar, and high-definition cameras. Then, the roadside perception records are filtered based on the meteorological records to obtain valid roadside perception records. For example, the goal of rain and fog weather data acquisition is to resist rain and fog interference. Raindrops and fog scatter light, reducing the contrast of ordinary camera images. Rainwater adheres to the lens surface, forming a water film that obstructs part of the field of view and creates perception blind spots. Data collected by infrared cameras and millimeter-wave radar is filtered, and then the filtered data is extracted and preprocessed to further improve the usability of the data.

[0149] According to an embodiment of the present invention, it further includes:

[0150] The road 3D structure data, vehicle identity data, vehicle motion trajectory data, real-time vehicle speed, braking status feature data, steering angle, and vehicle positioning data are input into a preset data quality target detection model for analysis and processing to obtain the corresponding data accuracy.

[0151] The data accuracy rate is compared with a preset data accuracy threshold.

[0152] If the data accuracy is less than a preset data accuracy threshold, it is marked as abnormal data and removed.

[0153] If the data accuracy is greater than or equal to a preset data accuracy threshold, then vehicle positioning error analysis is performed to obtain the vehicle positioning error.

[0154] The vehicle positioning error is compared with a preset vehicle positioning error threshold.

[0155] If the vehicle positioning error is greater than the preset vehicle positioning error threshold, the sensing device self-test is triggered.

[0156] If the vehicle positioning error is less than or equal to a preset vehicle positioning error threshold, then data integrity and timeliness analysis are performed to obtain a qualified integrity status and a qualified timeliness status.

[0157] Perform a bitwise AND operation on the integrity and timeliness compliance status. If the result is compliant, the multimodal vehicle-road cooperative standard dataset is determined to be valid. If the result is non-compliant, the multimodal vehicle-road cooperative standard dataset is determined to be invalid, and a warning response is output.

[0158] Further explanation is needed regarding the data preprocessing process. First, the accuracy of the data is evaluated using a pre-trained model, and data accuracy is determined by threshold comparison. Data below the threshold is discarded. Then, the vehicle-mounted BeiDou module outputs WGS84 coordinates in real time, which are converted to a local coordinate system consistent with the roadside equipment using a coordinate transformation algorithm (such as seven-parameter transformation) to evaluate vehicle positioning errors, i.e., spatial calibration. Finally, the roadside equipment accesses the NTP service via Ethernet, and the vehicle-mounted equipment accesses it via a 5G network, sending a time synchronization request every 100ms. The receiving end dynamically adjusts its local time based on network latency to ensure that all data timestamp deviations are correct, i.e., the timeliness is deemed acceptable. Simultaneously, data integrity is verified. Only when both timeliness and integrity meet the requirements is the multimodal vehicle-road cooperative standard dataset deemed valid. The preset data quality target detection model is trained by acquiring a large amount of historical sample road 3D structure data, vehicle identity data, vehicle motion trajectory data, real-time vehicle speed, braking status feature data, steering angle and vehicle positioning data, and the corresponding data accuracy.

[0159] According to an embodiment of the present invention, data extraction is performed on the multimodal vehicle-road cooperative standard dataset to obtain a vehicle-road cooperative effective feature dataset, including:

[0160] Data is extracted from the multimodal vehicle-road cooperative standard dataset to obtain a vehicle-road cooperative effective feature dataset, including vehicle effective feature data, traffic environment effective feature data, and effective feature association data;

[0161] The effective vehicle feature data includes effective vehicle identity feature data, effective vehicle status feature data, effective vehicle location feature data, and effective vehicle behavior feature data;

[0162] The effective traffic environment feature data includes effective meteorological environment feature data and effective road condition feature data.

[0163] It needs to be further explained that high-value data that can directly support subsequent fusion computing and intelligent decision-making are extracted from the multimodal raw data. The effective feature dataset for vehicle-road cooperation is extracted from the raw data of lidar, millimeter-wave radar and cameras, and is used to accurately identify vehicle identity, status and behavior. Examples of extraction are shown in Table 1.

[0164] Effective traffic environment feature data is extracted from raw data from meteorological sensors, cameras, and lidar. It is used to determine whether the current traffic environment is safe. Effective meteorological environment feature data includes weather type (rainy, foggy, or sunny), visibility (150m), road surface condition (dry, waterlogged, or icy), and precipitation. Effective road condition feature data includes lane line integrity feature data (lane line integrity includes complete, damaged, or missing), road obstacles (e.g., construction cones 1 km ahead, coordinates: 30.124567°N, 120.655432°E), and tunnel entrance / exit signs. Effective feature correlation data involves structured compression of vehicle and traffic environment effective feature data to preserve the correlation between data (e.g., the correlation between a vehicle's speed and the corresponding meteorological environment) and avoid data fragmentation.

[0165] According to an embodiment of the present invention, the step of dynamically fusing the vehicle-road cooperative effective feature dataset to generate a vehicle-road cooperative three-dimensional target feature map includes:

[0166] The effective feature dataset of vehicle-road cooperation is input into a preset vehicle-road cooperation scenario evaluation model for analysis and processing to obtain vehicle-road cooperation scenario type feature data, including overtaking scenario type feature data, congestion type feature data, tunnel type feature data, accident type feature data, rain and fog type feature data, or service area entrance and exit type feature data.

[0167] Based on the vehicle-road cooperative scenario type feature data, query the preset data fusion weight value list to obtain the LiDAR data weight value, millimeter-wave radar data weight value, and camera data weight value;

[0168] The effective feature dataset of vehicle-road cooperation is normalized and then weighted and fused with the weight values ​​of the lidar data, millimeter-wave radar data, and camera data to generate a three-dimensional target feature map of vehicle-road cooperation.

[0169] Further explanation is needed. To achieve dynamic fusion processing across different scenarios, firstly, the extracted effective feature dataset of vehicle-road cooperation is analyzed and processed through a preset vehicle-road cooperation scenario evaluation model to obtain vehicle-road cooperation scenario type feature data. Then, a preset data fusion weight value list is queried to obtain the corresponding dynamic weight values. Finally, after data normalization, weighted fusion is performed to generate a 3D target feature map of vehicle-road cooperation. The preset vehicle-road cooperation scenario evaluation model is obtained by training with a large number of historical samples of effective feature datasets of vehicle-road cooperation and corresponding vehicle-road cooperation scenario type feature data. The preset data fusion weight value list is analyzed and pre-constructed through a large number of historical samples and can be dynamically adjusted according to specific applications.

[0170] According to an embodiment of the present invention, the step of acquiring historical vehicle-road cooperative three-dimensional target feature maps and static road network data to construct a twin model, obtaining a vehicle-road cooperative twin model, and combining the vehicle-road cooperative three-dimensional target feature maps for analysis and processing to obtain road network state change prediction data within a preset time period includes:

[0171] Acquire historical vehicle-road cooperative 3D target feature maps and static road network data, whereby the static road network data includes road geometric parameter data and road static feature data;

[0172] A digital twin model is constructed based on the road geometric parameter data and road static feature data to generate an initial twin model for vehicle-road cooperation.

[0173] The historical vehicle-road cooperative three-dimensional target feature map is input into the vehicle-road cooperative initial twin model for initialization to obtain the vehicle-road cooperative twin model;

[0174] The vehicle-road cooperative three-dimensional target feature map is input into the vehicle-road cooperative twin model for analysis and processing to obtain road network status change prediction data within a preset time period, including vehicle dynamic data, road surface status data, meteorological dynamic data, and equipment operation dynamic data.

[0175] Further explanation is needed. In order to achieve intelligent decision-making through real-time scene simulation, firstly, static road network modeling and model initialization are performed. Then, based on real-time fusion processing, a three-dimensional target feature map of vehicle-road cooperation is generated, which includes four types of dynamic data: vehicle dynamic data, road surface condition data, meteorological dynamic data, and equipment operation dynamic data. Among them, vehicle dynamic data is such as truck ID: truck_007, location: 30.123456°N, 120.654321°E, lane 3, speed 90km / h; road surface condition data is such as icing 500 meters from the tunnel exit, coverage 30%; meteorological dynamic data is such as visibility 1500 meters, precipitation 0.2mm / min; and equipment operation dynamic data is such as RSU-008 lidar is normal, and camera infrared mode is on.

[0176] According to an embodiment of the present invention, the step of analyzing and processing the vehicle-road cooperative three-dimensional target feature map in conjunction with the road network state change prediction data to generate lane-level decision data and pushing it to the corresponding vehicle terminal includes:

[0177] Based on the vehicle-road cooperative three-dimensional target feature map, the speed limit and congestion critical density of the preset road segment are extracted. Based on the road network state change prediction data, the average vehicle speed and traffic flow density of the preset road segment are extracted. Based on the speed limit, congestion critical density, average vehicle speed and traffic flow density, the traffic efficiency score corresponding to each lane is obtained.

[0178] The effective meteorological environment characteristic data and the effective road condition characteristic data are input into a preset traffic environment type assessment model for analysis and processing to obtain traffic environment type characteristic data.

[0179] Based on the traffic environment type characteristic data, query the preset traffic efficiency correction coefficient list to obtain the traffic efficiency correction coefficient;

[0180] The traffic efficiency score is corrected according to the traffic efficiency correction coefficient to obtain the traffic efficiency optimization value, and then sorted in descending order. The lane with the highest traffic efficiency optimization value is pushed to the ordinary vehicle end.

[0181] The vehicle-road cooperative three-dimensional target feature map and the road network state change prediction data are input into a preset multimodal over-limit analysis model for analysis and processing to obtain weight over-limit judgment results and size over-limit judgment results;

[0182] If the weight exceeds the limit and the size exceeds the limit, a Level 1 warning will be issued to the freight vehicle.

[0183] If the weight exceeds the limit or the size exceeds the limit, a level 2 warning will be issued to the freight vehicle.

[0184] Further explanation is needed regarding the accurate output of lane-level intelligent decisions. First, the speed limit and congestion critical density of the preset road segment are extracted based on the vehicle-road cooperative three-dimensional target feature map. Then, the average vehicle speed and traffic flow density of the preset road segment are extracted based on road network state change prediction data. The traffic efficiency score for each lane is calculated as (average vehicle speed / speed limit) * 0.7 - (traffic flow density / congestion critical density) * 0.3. A traffic efficiency correction coefficient is then queried from real-time assessed traffic environment type feature data and multiplied by the traffic efficiency score for rate correction, used to evaluate the traffic efficiency of each lane and improve assessment accuracy. Next, the lane with the highest traffic efficiency is pushed to ordinary vehicles. Simultaneously, intelligent assessment is performed for weight and size over-limit warnings that are prone to occur with trucks. A Level 1 warning is output when both weight and size exceed limits; a Level 2 warning is output when either weight or size exceeds limits, with Level 1 warnings being higher than Level 2 warnings. The warning system includes traffic environment types such as rainy / foggy weather (traffic efficiency correction factor of 0.7), road icing (traffic efficiency correction factor of 0.6), road flooding (traffic efficiency correction factor of 0.6), or temporary construction (traffic efficiency correction factor of 0). Traffic environment type characteristic data are represented by unique identifiers. The preset traffic environment type assessment model is obtained by training with a large amount of historical samples of effective meteorological environment characteristic data, effective road condition characteristic data, and corresponding traffic environment type characteristic data. The preset multimodal over-limit analysis model is obtained by training with a large amount of historical samples of vehicle-road cooperative three-dimensional target feature maps, road network condition change prediction data, and corresponding weight over-limit judgment results and size over-limit judgment results. The preset traffic efficiency correction factor list is constructed by those skilled in the art based on the analysis of a large amount of historical samples and can be modified according to specific applications.

[0185] According to an embodiment of the present invention, it further includes:

[0186] Based on the vehicle-road cooperative effective feature dataset at a preset time point, a corresponding data fingerprint is generated, including a data hash value and evidence storage metadata;

[0187] The data hash value and evidence metadata are uploaded to a distributed storage node with a preset data volume, and the data is verified using a preset Byzantine fault-tolerant algorithm.

[0188] If the data verification is successful, the sensitive fields of the vehicle-road cooperative effective feature dataset are obtained, and the sensitivity level is determined according to the sensitive fields. According to the sensitivity level, preset noise is injected through a preset differential privacy algorithm and written into a preset block. The hash value of the vehicle-road cooperative effective feature dataset at the time point before the preset time point is also associated.

[0189] If data verification fails, a data alert will be triggered.

[0190] The system performs identity verification, permission matching, and data sensitivity checks based on the data access instructions.

[0191] If the verification passes, an access token is generated, and access data is obtained through the data node based on the access token;

[0192] If the verification fails, an abnormal access log will be generated and an alert response will be output.

[0193] Further explanation is needed regarding data security protection. To address the core pain points of difficulty in tracing data tampering and a one-size-fits-all approach to privacy protection, a source hashing and evidence storage method is used during the data collection phase to generate data fingerprints. Simultaneously, sensitive fields and their corresponding sensitivity levels are obtained from the effective feature dataset of vehicle-road cooperative systems. For example, precise vehicle trajectories correspond to a high sensitivity level, requiring global differential privacy and Gaussian noise injection; vehicle weight corresponds to a medium sensitivity level, requiring local differential privacy and Laplace noise injection; and average road segment speed corresponds to a low sensitivity level, requiring no noise injection, only outlier removal. A differential privacy algorithm is employed to protect vehicle data without affecting its statistical characteristics. Sensitive information about personnel is stored on a distributed blockchain and then linked to the hash value of the vehicle-road cooperative effective feature dataset from a previous time point. If any node tamperes with a block of data, the hash verification of all subsequent blocks will fail, ensuring that the data is immutable. During the data access phase, identity legitimacy, permission matching, and data sensitivity are verified. Identity legitimacy verifies whether the digital certificate signature is valid and within its validity period. Permission matching checks whether the access request is within the scope of the subject's permissions. Data sensitivity requires additional verification of a special authorization code (temporarily issued by the operating unit) if accessing sensitive data (such as emergency vehicle trajectories).

[0194] A third aspect of the present invention provides a readable storage medium storing a program for a highway vehicle-road cooperative data governance method based on multimodal perception. When the program is executed by a processor, it implements the steps of the highway vehicle-road cooperative data governance method based on multimodal perception as described in any of the preceding claims.

[0195] The present invention discloses a method and system for highway vehicle-road cooperative data governance based on multimodal perception. Through adaptive multimodal perception scheduling, feature-level dynamic fusion, digital twin mapping and blockchain storage technology, it covers the entire data collection scenario, improves the effectiveness of data fusion, and thus achieves effective governance of vehicle-road cooperative data.

[0196] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0197] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0198] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0199] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0200] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A method for highway vehicle-road cooperative data governance based on multimodal perception, characterized in that, Includes the following steps: Obtain a multimodal vehicle-road cooperative dataset and perform data preprocessing to obtain a standard multimodal vehicle-road cooperative dataset; Data is extracted from the multimodal vehicle-road cooperative standard dataset to obtain an effective feature dataset for vehicle-road cooperation; Dynamic fusion processing is performed on the effective feature dataset of vehicle-road cooperation to generate a three-dimensional target feature map of vehicle-road cooperation. Historical vehicle-road cooperative 3D target feature maps and static road network data are acquired to construct a twin model, thereby obtaining a vehicle-road cooperative twin model. The vehicle-road cooperative 3D target feature maps are then analyzed and processed to obtain road network state change prediction data within a preset time period. The vehicle-road cooperative three-dimensional target feature map is analyzed and processed in conjunction with the road network state change prediction data to generate lane-level decision data, which is then pushed to the corresponding vehicle terminal. The step of analyzing and processing the vehicle-road cooperative three-dimensional target feature map in conjunction with the road network state change prediction data to generate lane-level decision data and pushing it to the corresponding vehicle terminal includes: Based on the vehicle-road cooperative three-dimensional target feature map, the speed limit and congestion critical density of the preset road segment are extracted. Based on the road network state change prediction data, the average vehicle speed and traffic flow density of the preset road segment are extracted. Based on the speed limit, congestion critical density, average vehicle speed and traffic flow density, the traffic efficiency score corresponding to each lane is obtained. The effective meteorological environmental characteristic data and the effective road condition characteristic data are input into the preset traffic environment type assessment model for analysis and processing to obtain traffic environment type characteristic data. Based on the traffic environment type characteristic data, query the preset traffic efficiency correction coefficient list to obtain the traffic efficiency correction coefficient; The traffic efficiency score is corrected according to the traffic efficiency correction coefficient to obtain the traffic efficiency optimization value, and then sorted in descending order. The lane with the highest traffic efficiency optimization value is pushed to the ordinary vehicle end. The vehicle-road cooperative three-dimensional target feature map and the road network state change prediction data are input into a preset multimodal over-limit analysis model for analysis and processing to obtain weight over-limit judgment results and size over-limit judgment results; If the weight exceeds the limit and the size exceeds the limit, a Level 1 warning will be issued to the freight vehicle. If the weight exceeds the limit or the size exceeds the limit, a level 2 warning is output to the freight vehicle. Also includes: Based on the vehicle-road cooperative effective feature dataset at a preset time point, a corresponding data fingerprint is generated, including a data hash value and evidence storage metadata; The data hash value and evidence metadata are uploaded to a distributed storage node with a preset data volume, and the data is verified using a preset Byzantine fault-tolerant algorithm. If the data verification is successful, the sensitive fields of the vehicle-road cooperative effective feature dataset are obtained, and the sensitivity level is determined according to the sensitive fields. According to the sensitivity level, preset noise is injected through a preset differential privacy algorithm and written into a preset block. The hash value of the vehicle-road cooperative effective feature dataset at the time point before the preset time point is also associated. If data verification fails, a data alert will be triggered. The system performs identity verification, permission matching, and data sensitivity checks based on the data access instructions. If the verification passes, an access token is generated, and access data is obtained through the data node based on the access token; If the verification fails, an abnormal access log will be generated and an alert response will be output.

2. The highway vehicle-road cooperative data governance method based on multimodal perception according to claim 1, characterized in that, The process of acquiring a multimodal vehicle-road cooperative dataset and performing data preprocessing to obtain a standard multimodal vehicle-road cooperative dataset includes: Acquire a multimodal vehicle-road cooperative dataset, including meteorological record data, roadside perception record data, and vehicle perception record data. Among them, the vehicle perception record data includes real-time vehicle speed, braking status feature data, steering angle, and vehicle positioning data. Based on the meteorological record data, the roadside sensing record data is filtered and processed to obtain valid roadside sensing record data; Data is extracted based on the effective recorded data from the roadside perception to obtain three-dimensional road structure data, vehicle identity data, and vehicle motion trajectory data. The road 3D structure data, vehicle identity data, vehicle motion trajectory data, real-time vehicle speed, braking status feature data, steering angle, and vehicle positioning data are preprocessed by standardizing the format, spatiotemporal calibration, and data quality inspection to obtain a multimodal vehicle-road cooperative standard dataset.

3. The highway vehicle-road cooperative data governance method based on multimodal perception according to claim 2, characterized in that, Also includes: The road 3D structure data, vehicle identity data, vehicle motion trajectory data, real-time vehicle speed, braking status feature data, steering angle, and vehicle positioning data are input into a preset data quality target detection model for analysis and processing to obtain the corresponding data accuracy. The data accuracy rate is compared with a preset data accuracy threshold. If the data accuracy is less than a preset data accuracy threshold, it is marked as abnormal data and removed. If the data accuracy is greater than or equal to a preset data accuracy threshold, then vehicle positioning error analysis is performed to obtain the vehicle positioning error. The vehicle positioning error is compared with a preset vehicle positioning error threshold. If the vehicle positioning error is greater than the preset vehicle positioning error threshold, the sensing device self-test is triggered. If the vehicle positioning error is less than or equal to a preset vehicle positioning error threshold, then data integrity and timeliness analysis are performed to obtain a qualified integrity status and a qualified timeliness status. Perform a bitwise AND operation on the integrity and timeliness compliance status. If the result is compliant, the multimodal vehicle-road cooperative standard dataset is determined to be valid. If the result is non-compliant, the multimodal vehicle-road cooperative standard dataset is determined to be invalid, and a warning response is output.

4. The highway vehicle-road cooperative data governance method based on multimodal perception according to claim 1, characterized in that, The step of extracting data from the multimodal vehicle-road cooperative standard dataset to obtain an effective feature dataset for vehicle-road cooperation includes: Data is extracted from the multimodal vehicle-road cooperative standard dataset to obtain a vehicle-road cooperative effective feature dataset, including vehicle effective feature data, traffic environment effective feature data, and effective feature association data; The effective vehicle feature data includes effective vehicle identity feature data, effective vehicle status feature data, effective vehicle location feature data, and effective vehicle behavior feature data; The effective traffic environment feature data includes effective meteorological environment feature data and effective road condition feature data.

5. The highway vehicle-road cooperative data governance method based on multimodal perception according to claim 1, characterized in that, The step of dynamically fusing the vehicle-road cooperative effective feature dataset to generate a vehicle-road cooperative three-dimensional target feature map includes: The effective feature dataset of vehicle-road cooperation is input into a preset vehicle-road cooperation scenario evaluation model for analysis and processing to obtain vehicle-road cooperation scenario type feature data, including overtaking scenario type feature data, congestion type feature data, tunnel type feature data, accident type feature data, rain and fog type feature data, or service area entrance and exit type feature data. Based on the vehicle-road cooperative scenario type feature data, query the preset data fusion weight value list to obtain the LiDAR data weight value, millimeter-wave radar data weight value, and camera data weight value; The effective feature dataset of vehicle-road cooperation is normalized and then weighted and fused with the weight values ​​of the lidar data, millimeter-wave radar data, and camera data to generate a three-dimensional target feature map of vehicle-road cooperation.

6. The highway vehicle-road cooperative data governance method based on multimodal perception according to claim 1, characterized in that, The process involves acquiring historical vehicle-road cooperative 3D target feature maps and static road network data to construct a twin model, obtaining a vehicle-road cooperative twin model, and then analyzing and processing the vehicle-road cooperative 3D target feature maps to obtain road network state change prediction data within a preset time period, including: Acquire historical vehicle-road cooperative 3D target feature maps and static road network data, whereby the static road network data includes road geometric parameter data and road static feature data; A digital twin model is constructed based on the road geometric parameter data and road static feature data to generate an initial twin model for vehicle-road cooperation. The historical vehicle-road cooperative three-dimensional target feature map is input into the vehicle-road cooperative initial twin model for initialization to obtain the vehicle-road cooperative twin model; The vehicle-road cooperative three-dimensional target feature map is input into the vehicle-road cooperative twin model for analysis and processing to obtain road network status change prediction data within a preset time period, including vehicle dynamic data, road surface status data, meteorological dynamic data, and equipment operation dynamic data.

7. A highway vehicle-road cooperative data governance system based on multimodal perception, characterized in that, The system includes a memory and a processor. The memory contains a program for a highway vehicle-road cooperative data governance method based on multimodal perception. When the program for the highway vehicle-road cooperative data governance method based on multimodal perception is executed by the processor, it performs the following steps: Obtain a multimodal vehicle-road cooperative dataset and perform data preprocessing to obtain a standard multimodal vehicle-road cooperative dataset; Data is extracted from the multimodal vehicle-road cooperative standard dataset to obtain an effective feature dataset for vehicle-road cooperation; Dynamic fusion processing is performed on the effective feature dataset of vehicle-road cooperation to generate a three-dimensional target feature map of vehicle-road cooperation. Historical vehicle-road cooperative 3D target feature maps and static road network data are acquired to construct a twin model, thereby obtaining a vehicle-road cooperative twin model. The vehicle-road cooperative 3D target feature maps are then analyzed and processed to obtain road network state change prediction data within a preset time period. The vehicle-road cooperative three-dimensional target feature map is analyzed and processed in conjunction with the road network state change prediction data to generate lane-level decision data, which is then pushed to the corresponding vehicle terminal. The step of analyzing and processing the vehicle-road cooperative three-dimensional target feature map in conjunction with the road network state change prediction data to generate lane-level decision data and pushing it to the corresponding vehicle terminal includes: Based on the vehicle-road cooperative three-dimensional target feature map, the speed limit and congestion critical density of the preset road segment are extracted. Based on the road network state change prediction data, the average vehicle speed and traffic flow density of the preset road segment are extracted. Based on the speed limit, congestion critical density, average vehicle speed and traffic flow density, the traffic efficiency score corresponding to each lane is obtained. The effective meteorological environmental characteristic data and the effective road condition characteristic data are input into the preset traffic environment type assessment model for analysis and processing to obtain traffic environment type characteristic data. Based on the traffic environment type characteristic data, query the preset traffic efficiency correction coefficient list to obtain the traffic efficiency correction coefficient; The traffic efficiency score is corrected according to the traffic efficiency correction coefficient to obtain the traffic efficiency optimization value, and then sorted in descending order. The lane with the highest traffic efficiency optimization value is pushed to the ordinary vehicle end. The vehicle-road cooperative three-dimensional target feature map and the road network state change prediction data are input into a preset multimodal over-limit analysis model for analysis and processing to obtain weight over-limit judgment results and size over-limit judgment results; If the weight exceeds the limit and the size exceeds the limit, a Level 1 warning will be issued to the freight vehicle. If the weight exceeds the limit or the size exceeds the limit, a level 2 warning is output to the freight vehicle. Also includes: Based on the vehicle-road cooperative effective feature dataset at a preset time point, a corresponding data fingerprint is generated, including a data hash value and evidence storage metadata; The data hash value and evidence metadata are uploaded to a distributed storage node with a preset data volume, and the data is verified using a preset Byzantine fault-tolerant algorithm. If the data verification is successful, the sensitive fields of the vehicle-road cooperative effective feature dataset are obtained, and the sensitivity level is determined according to the sensitive fields. According to the sensitivity level, preset noise is injected through a preset differential privacy algorithm and written into a preset block. The hash value of the vehicle-road cooperative effective feature dataset at the time point before the preset time point is also associated. If data verification fails, a data alert will be triggered. The system performs identity verification, permission matching, and data sensitivity checks based on the data access instructions. If the verification passes, an access token is generated, and access data is obtained through the data node based on the access token; If the verification fails, an abnormal access log will be generated and an alert response will be output.

8. The highway vehicle-road cooperative data governance system based on multimodal perception according to claim 7, characterized in that, The process of acquiring a multimodal vehicle-road cooperative dataset and performing data preprocessing to obtain a standard multimodal vehicle-road cooperative dataset includes: Acquire a multimodal vehicle-road cooperative dataset, including meteorological record data, roadside perception record data, and vehicle perception record data. Among them, the vehicle perception record data includes real-time vehicle speed, braking status feature data, steering angle, and vehicle positioning data. Based on the meteorological record data, the roadside sensing record data is filtered and processed to obtain valid roadside sensing record data; Data is extracted based on the effective recorded data from the roadside perception to obtain three-dimensional road structure data, vehicle identity data, and vehicle motion trajectory data. The road 3D structure data, vehicle identity data, vehicle motion trajectory data, real-time vehicle speed, braking status feature data, steering angle, and vehicle positioning data are preprocessed by standardizing the format, spatiotemporal calibration, and data quality inspection to obtain a multimodal vehicle-road cooperative standard dataset.

Images