A multimodal three-dimensional data processing method and system based on road characteristics

Abstract

The present application provides a multimodal three-dimensional data processing method and system based on road features, which relates to the field of electronic digital data processing. The method includes: obtaining the first type of three-dimensional data of the road section within the monitoring range through the road information collection device installed at the traffic node; establishing a communication connection with the car entering the road section within the preset range, obtaining the road information uploaded by the intelligent driving assistance system, and obtaining the second type of three-dimensional data; fusing the first type of three-dimensional data and the second type of three-dimensional data to obtain a multimodal three-dimensional model of the traffic node within the preset range; integrating the multimodal three-dimensional model of each traffic node to obtain a multimodal three-dimensional model of the entire road section; dynamically displaying the road conditions based on the multimodal three-dimensional model of the entire road section. The implementation of this method overcomes the limitations of a single data source through spatiotemporal alignment and multimodal data fusion, constructs a dynamic three-dimensional model of the entire road section, and realizes real-time road condition display of the entire road section.

Description

Technical Field

[0001] The present application relates to the field of electronic digital data processing, and in particular to a multi-modal three-dimensional data processing method and system based on road features. Background Art

[0002] In modern urban traffic management, the development of intelligent road condition monitoring systems is crucial. These systems effectively improve road use efficiency, reduce traffic congestion, and improve vehicle and pedestrian safety by monitoring road conditions in real time, such as traffic flow, accident occurrence, and road surface conditions. With the advancement of intelligent transportation system technology, the demand for data collection and processing is also growing.

[0003] Existing intelligent road monitoring systems mainly rely on fixed cameras and sensor networks to collect road information. These devices are installed at specific traffic nodes, such as intersections and important traffic hubs, and monitor road usage through video and sensor data. The data is transmitted to the central processing system via wired or wireless means for subsequent data analysis, such as traffic flow calculation and accident detection.

[0004] However, existing technologies mainly rely on fixed-position equipment to collect data, which limits the monitoring scope. Even if important nodes can be covered, it is difficult to achieve continuous monitoring of the entire road network. For example, it is difficult to obtain real-time traffic conditions in areas without camera coverage, which affects the effectiveness and accuracy of the overall monitoring system. Summary of the invention

[0005] The present application provides a multimodal three-dimensional data processing method and system based on road features, which is used to address the limitations of monitoring range and data sources caused by fixed-point traffic data collection in the existing technology, making it difficult to fully reflect real-time changes in road conditions.

[0006] In a first aspect, the present application provides a multimodal three-dimensional data processing method based on road features, which is applied to an intelligent road condition monitoring system. The method comprises:

[0007] Acquire the first type of three-dimensional data of the road section within the monitoring range through a road information collection device installed at a traffic node, wherein the road information collection device includes a camera and a sensor;

[0008] Establishing a communication connection with a car that enters a road section within the preset range, acquiring road information uploaded by an intelligent driving assistance system, and obtaining second-type three-dimensional data, wherein the preset range is larger than the monitoring range;

[0009] The first type of three-dimensional data and the second type of three-dimensional data are merged to obtain a multimodal three-dimensional model of the traffic node within a preset range;

[0010] Integrate the multimodal 3D models of each traffic node to obtain a multimodal 3D model of the entire road section;

[0011] The road conditions are dynamically displayed based on the multimodal three-dimensional model of the entire road section.

[0012] Through the above embodiments, the intelligent road condition monitoring system continuously collects high-precision real-time data around traffic nodes through fixed cameras and sensors, and supplements the status information of road sections between nodes through the vehicle-mounted system with a wider coverage. Through spatiotemporal alignment and multimodal data fusion, the limitations of a single data source are overcome, a dynamic three-dimensional model of the entire road section is constructed, and real-time road condition display of the entire road section is realized.

[0013] In some embodiments, the step of fusing the first type of three-dimensional data and the second type of three-dimensional data to obtain a multimodal three-dimensional model of the traffic node within a preset range specifically includes:

[0014] Performing temporal and spatial alignment on the preprocessed first-category three-dimensional data and the second-category three-dimensional data, so that the first-category three-dimensional data and the second-category three-dimensional data are consistent in terms of time stamp and spatial position;

[0015] The aligned first-category three-dimensional data and the second-category three-dimensional data are fused using a preset data fusion technology to obtain a multimodal three-dimensional model of the traffic node within a preset range.

[0016] Through the above embodiments, the intelligent road condition monitoring system aligns time and space to ensure that the data collected by different sensors are consistent in the time and space dimensions. On this basis, data fusion technologies such as weight fusion and decision fusion are used to optimize the combination of the aligned first and second three-dimensional data, overcome the defects of single data, give play to the complementary advantages of multi-source data, and finally obtain a three-dimensional model of the local road section with more comprehensive information and higher accuracy.

[0017] In some embodiments, the step of fusing the aligned first type of three-dimensional data and the second type of three-dimensional data using a preset data fusion technology to obtain a multimodal three-dimensional model of the traffic node within a preset range specifically includes:

[0018] Using the first type of three-dimensional data to construct a basic framework of the multimodal three-dimensional model;

[0019] The second type of three-dimensional data is used to fill the basic framework to obtain a multimodal three-dimensional model of the traffic node within a preset range.

[0020] Through the above embodiments, the intelligent road condition monitoring system constructs a three-dimensional model framework with the first type of data collected by fixed nodes, and then enriches the model details with the second type of data collected by the vehicle-mounted system, thereby further optimizing the data fusion effect and improving the accuracy and coverage of the multimodal three-dimensional model.

[0021] In some embodiments, the step of establishing a communication connection with a car that enters a road section within the preset range, acquiring road information uploaded by an intelligent driving assistance system, and obtaining the second type of three-dimensional data specifically includes:

[0022] If the road section within the preset range is a high-speed road section or an elevated road section, after establishing a communication connection with a car entering the road section within the preset range, the speed data of the car is obtained first;

[0023] Determine whether the vehicle speed data is within a normal vehicle speed range;

[0024] If so, only the road information uploaded by the intelligent driving assistance system of one vehicle is obtained within the preset time interval;

[0025] If not, obtain the road information uploaded by the intelligent driving assistance system of all cars;

[0026] The second type of three-dimensional data is generated according to the road information.

[0027] Through the above embodiments, the intelligent road condition monitoring system introduces vehicle speed data in the data collection stage, and dynamically adjusts the frequency of uploading vehicle information based on it, taking into account data quality while reducing communication pressure. When the vehicle speed is normal, the data upload frequency of non-critical vehicles is reduced; when the system detects abnormal vehicle speed, accidents or serious congestion may occur on the relevant road sections, and at this time, comprehensive vehicle data is collected to provide rich decision-making basis for abnormal road condition analysis.

[0028] In some embodiments, after the step of acquiring the road information uploaded by the intelligent driving assistance systems of all vehicles, the following step is further included:

[0029] After determining that there are other adjacent vehicles ahead according to the road information uploaded by the current vehicle, detecting whether the road information uploaded by the other adjacent vehicles is received;

[0030] If the road information uploaded by the other adjacent vehicles is not received, the real-time position information of the other adjacent vehicles is determined based on the road information uploaded by the current vehicle, and the real-time position information is added to the second type of three-dimensional data.

[0031] Through the above embodiments, when the intelligent road condition monitoring system fails to obtain data of a certain vehicle, it determines the location of the vehicle based on information obtained from nearby vehicles and adds it to its own collected data, thereby restoring the road scene to the greatest extent and ensuring the integrity of the road monitoring information.

[0032] In some embodiments, after the step of integrating the multimodal three-dimensional model of each traffic node to obtain the multimodal three-dimensional model of the entire road section, the method further includes:

[0033] Determining a congested road section from a multimodal three-dimensional model of the entire road section;

[0034] The cause of congestion of the congested road section is determined based on the road data recorded by the multimodal three-dimensional model.

[0035] Through the above embodiments, the intelligent road condition monitoring system identifies potential congested areas where traffic flow density is abnormally increased and vehicle speed is significantly reduced through spatiotemporal analysis of vehicle density and speed in a three-dimensional road model. Then, the system analyzes road data before and after the congested section, such as road conditions, accident information, changes in the number of lanes, etc., to infer the causes of the congestion and provide a reference for management decisions.

[0036] In some embodiments, after the step of dynamically displaying the road condition based on the multimodal three-dimensional model of the entire road section, the method further includes:

[0037] Count the traffic volume at each traffic node within a preset time period;

[0038] Based on the traffic flow, the congested traffic nodes within a preset time in the future are predicted.

[0039] Through the above embodiments, the intelligent road condition monitoring system collects statistics on the traffic flow data of each intersection within a certain period of time to determine the traffic flow distribution characteristics of the node; on this basis, it uses traffic flow prediction models such as time series, machine learning and other methods, combined with the road network structure and the spatiotemporal correlation of traffic flow to predict the traffic flow change trend of each node in the short term, thereby judging the sections where congestion may occur, providing a basis for measures such as traffic induction and signal light timing.

[0040] In a second aspect, the present application provides an intelligent road condition monitoring system, the intelligent road condition monitoring system comprising: one or more processors and a memory;

[0041] The memory is coupled to the one or more processors, and the memory is used to store computer program code, which includes computer instructions. The one or more processors call the computer instructions so that the intelligent road condition monitoring system can implement a multimodal three-dimensional data processing method based on road features provided in the above embodiment, which will not be repeated here.

[0042] In a third aspect, the present application provides a computer-readable storage medium comprising instructions. When the instructions are executed on an intelligent road condition monitoring system, the intelligent road condition monitoring system can implement a multimodal three-dimensional data processing method based on road features provided in the above-mentioned embodiment, which will not be described in detail here.

[0043] In a fourth aspect, the present application provides a computer program product. When the computer program product runs on an intelligent road condition monitoring system, the intelligent road condition monitoring system can implement a multimodal three-dimensional data processing method based on road features provided in the above embodiment, which will not be repeated here.

[0044] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0045] 1. The intelligent road condition monitoring system integrates multimodal data from fixed traffic facilities and vehicle-mounted systems. Through spatiotemporal alignment and multi-level data fusion, it overcomes the limitations of a single data source and achieves dynamic reconstruction and real-time monitoring of the entire road section.

[0046] 2. For low-accident road conditions such as highways and elevated roads, the intelligent road condition monitoring system uses vehicle speed information to dynamically adjust the data upload strategy, achieving a balance between communication load and data quality. At the same time, when the data of a certain vehicle cannot be obtained, the location of the vehicle is determined based on the information obtained from nearby vehicles and added to its own collected data, thereby restoring the road scene to the greatest extent and ensuring the integrity of the road monitoring information.

[0047] 3. The intelligent road condition monitoring system fully exploits the data value of the multimodal three-dimensional road model and realizes road condition analysis in multiple dimensions such as congestion identification, cause analysis and traffic prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flowchart of a multimodal three-dimensional data processing method based on road features in an embodiment of the present application;

[0049] Figure 2 is another flowchart of a multimodal three-dimensional data processing method based on road features in an embodiment of the present application;

[0050] Figure 3 is a schematic diagram of an exemplary scenario in which the intelligent road condition monitoring system acquires the first type of three-dimensional data and the second type of three-dimensional data in an embodiment of the present application;

[0051] Figure 4 It is a schematic diagram of the structure of a physical device of the intelligent road condition monitoring system in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more listed items.

[0053] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0054] For ease of understanding, the following is a flow chart of the method provided by this embodiment. Figure 1 The figure is a flow chart of a multimodal three-dimensional data processing method based on road features in an embodiment of the present application.

[0055] S101. Acquire first-category three-dimensional data of a road section within a monitoring range through a road information collection device installed at a traffic node.

[0056] Specifically, relevant technical personnel deploy road information collection equipment at key traffic nodes, such as important intersections, toll booths, and congested sections. Among them, road information collection equipment includes but is not limited to high-definition cameras, laser radars, ultrasonic sensors, etc., which can obtain road scene data within a certain range around the node from multiple angles and with high precision.

[0057] For example, at an intersection, cameras can capture information such as the location, speed, and model of a vehicle, and identify the status of traffic lights; LiDAR can accurately measure the distance of objects and map road structures such as the road surface, curbs, and isolation belts; ultrasonic waves focus on detecting close-range obstacles, such as pedestrians and non-motor vehicles. The intelligent road condition monitoring system performs spatiotemporal registration of the data from these heterogeneous sensors and fuses them to generate a three-dimensional road model around the node, which is the first type of three-dimensional data.

[0058] S102: Establish a communication connection with a car that enters a road section within a preset range, obtain road information uploaded by an intelligent driving assistance system, and obtain the second type of three-dimensional data.

[0059] Specifically, when a car is detected to enter the preset range of a certain node, the intelligent road condition monitoring system sends a data request to the car. If the car is equipped with an intelligent driving assistance system, the system responds to the request and uploads the road information collected by the on-board sensor to the intelligent road condition monitoring system. Among them, the road information collected by the intelligent driving assistance system includes but is not limited to video clips taken by the driving recorder, accident-prone areas marked on the electronic map, and road slope determined by inertial navigation. After receiving these heterogeneous data, the intelligent road condition monitoring system performs spatiotemporal alignment and coordinate transformation, and finally obtains the three-dimensional trajectory of the road from the perspective of the car, that is, the second type of three-dimensional data.

[0060] Furthermore, as more and more vehicles join the collaborative perception network, the second type of three-dimensional data continuously supplements and extends the coverage of the first type of three-dimensional data. In addition, while a vehicle uploads its own collected road information, it can also obtain the road conditions shared by other vehicles, thereby realizing information interaction between vehicles and roads, making driving decisions more intelligent. It is worth noting that not all vehicle data is equally important. The intelligent road condition monitoring system can dynamically adjust the priority and frequency of data requests based on factors such as vehicle type and driving section to avoid wasting communication resources.

[0061] It should be noted that the preset range in the embodiment of the present application is the communication range corresponding to each traffic node. Within the communication range, the intelligent road condition monitoring system can send a data request instruction to the car, thereby receiving the road information data sent by the car. In addition, the two preset ranges corresponding to adjacent traffic nodes partially intersect or are completely mutually exclusive, so that the communication range covers the entire road section.

[0062] S103: Fusing the first type of three-dimensional data and the second type of three-dimensional data to obtain a multimodal three-dimensional model of the traffic node within a preset range.

[0063] Specifically, the intelligent road condition monitoring system aligns the heterogeneous data collected by different sensors in terms of spatial coordinate system, timestamp, data format, etc. For example, using GPS and IMU data, the local point cloud collected by the vehicle-mounted sensor is projected into the global map coordinate system; the timestamps of the image and radar are synchronized, and the data that lags behind the current moment is removed; heterogeneous data such as images, point clouds, and vector maps are converted into a unified intermediate expression, etc.

[0064] Next, the intelligent road condition monitoring system integrates the multi-dimensional semantic information provided by the first and second three-dimensional data, such as lane lines, traffic signs, road conditions, static and dynamic obstacles, etc., and weights and integrates the aligned data. Optionally, the point cloud map obtained from the roadside is used as the skeleton, and the semantic information collected by the vehicle is rendered on this basis. At the same time, for the errors and conflicts between the first and second three-dimensional data, semantic labels with high confidence are given priority according to the preset confidence evaluation criteria. After multiple rounds of iterative optimization, a local road three-dimensional model that integrates the advantages of multiple sensors is finally obtained.

[0065] S104: Integrate the multimodal three-dimensional model of each traffic node to obtain a multimodal three-dimensional model of the entire road section.

[0066] The intelligent road condition monitoring system groups the local road 3D models of adjacent traffic nodes according to the road network topology, and uses algorithms such as region growing and ICP to achieve coarse alignment within the group to eliminate the scale differences of different models; then, based on the traffic nodes with more overlap, the 3D models of a group of nodes are spliced ​​into sub-segments; then, using the sub-segments as units, the above coarse alignment and fine splicing processes are repeated until a 3D panoramic view of the complete road is formed.

[0067] S105. Dynamically display the road conditions based on the multimodal three-dimensional model of the entire road section.

[0068] The intelligent road condition monitoring system combines the static road structure model with real-time traffic flow data to generate a dynamic panoramic view of road traffic. Managers can freely switch perspectives in the 3D model scene to overlook the traffic conditions of the entire road, or focus on a certain intersection or road to view the movement details of microscopic traffic. For key areas such as accident-prone sections, the system can also automatically switch to the corresponding perspective and issue warnings.

[0069] In the above embodiment, the intelligent road condition monitoring system continuously collects high-precision real-time data around traffic nodes through fixed cameras and sensors, and supplements the status information of the road sections between nodes through the vehicle-mounted system with a wider coverage. Through spatiotemporal alignment and multimodal data fusion, the limitations of a single data source are overcome, a dynamic three-dimensional model of the entire road section is constructed, and real-time road condition display of the entire road section is realized.

[0070] The following is a more detailed description of the process of the method provided by this embodiment. Figure 2 , which is another flow chart of a multimodal three-dimensional data processing method based on road features in an embodiment of the present application.

[0071] S201. Acquire first-category three-dimensional data of a road section within a preset range through a road information collection device installed at a traffic node.

[0072] This step is the same as step S101 and will not be repeated here.

[0073] S202: Whether the road section within the preset range is a high-speed road section or an elevated road section.

[0074] The intelligent road condition monitoring system determines the road grade of the data collection section by comparing the map information. If it is confirmed that the section belongs to a highway or an elevated road, the system will start a targeted data acquisition strategy and enter step S203, while collecting traffic flow information of these sections at a lower frequency. In contrast, considering that the traffic conditions on ordinary roads are more complicated and more prone to safety hazards, the collection frequency and data volume can be appropriately increased, that is, entering step S205 to obtain the road information uploaded by the intelligent driving assistance system of all cars.

[0075] S203: After establishing a communication connection with a car that has entered a road section within a preset range, obtaining the speed data of the car first.

[0076] Modern cars are generally equipped with on-board communication terminals and intelligent driving assistance systems, which can sense the vehicle's own status and surrounding road conditions, and exchange information with roadside facilities through wireless networks. When a vehicle enters the preset range of a certain traffic node, the intelligent road condition monitoring system will actively establish a data link with it through the roadside unit corresponding to the node.

[0077] Furthermore, the intelligent traffic monitoring system obtains the data collected by the vehicle's GPS and speed sensor through the data link, and then determines the vehicle's instantaneous speed. The intelligent traffic monitoring system will continue to track the data and warn of potential accident risks if abnormal speed changes are found.

[0078] S204: Whether the vehicle speed data is within the normal vehicle speed range.

[0079] Based on historical data and traffic rules, the intelligent road condition monitoring system presets the normal speed range for various types of roads at different times. For example, the speed on expressways during the day is usually 80-120km / h, and that on urban expressways is 60-80km / h.

[0080] If the intelligent road condition monitoring system finds that the speed of a certain vehicle deviates significantly from the normal range, such as the speed of the vehicle on the expressway drops to below 20km / h, it is determined that an accident or severe congestion may occur on the road ahead. At this time, it is necessary to increase the frequency of data collection to fully grasp the road conditions ahead, that is, enter step S205 to obtain the road information uploaded by the intelligent driving assistance system of all vehicles to provide a reference for emergency response. On the contrary, if it is detected that the speed of the vehicle within the preset range is always maintained in the normal speed range, it is determined that the vehicle is driving normally, and enter step S209 to obtain only the road information uploaded by the intelligent driving assistance system of one vehicle within the preset time interval, thereby reducing the frequency of obtaining road information data.

[0081] S205: Obtain road information uploaded by the intelligent driving assistance systems of all vehicles.

[0082] When a vehicle on a road section is detected to have an abnormal speed, the intelligent road condition monitoring system will contact all vehicles on the road section and ask them to upload more comprehensive road data. Automobile intelligent driving assistance systems such as adaptive cruise control and automatic emergency braking are equipped with environmental perception modules such as on-board cameras and lidar to detect the road conditions around the vehicle. The intelligent road condition monitoring system obtains detailed second-category road condition data from these on-board sensors, including vehicle spacing, road obstacles, weather conditions, etc., to help determine the specific conditions of abnormal road sections.

[0083] In the above embodiment, the intelligent road condition monitoring system introduces vehicle speed data in the data collection stage, and dynamically adjusts the frequency of uploading vehicle information based on it, taking into account data quality while reducing communication pressure. When the vehicle speed is normal, the data upload frequency of non-critical vehicles is reduced; when the system detects abnormal vehicle speed, accidents or serious congestion may occur on the relevant road section. At this time, comprehensive vehicle data is collected to provide rich decision-making basis for abnormal road condition analysis.

[0084] S206: Whether road information uploaded by other neighboring vehicles is received.

[0085] Some vehicles may not be able to establish communication with the intelligent road condition monitoring system or upload complete data in time due to temporary failures, unstable network, etc. Some vehicles are not even equipped with intelligent driving assistance systems. Therefore, in order to avoid the lack of information affecting the integrity of road condition analysis, the intelligent road condition monitoring system will supplement it with road information uploaded by other nearby vehicles.

[0086] Specifically, the intelligent road condition monitoring system searches for the neighboring vehicles of each vehicle and determines whether the neighboring vehicle has uploaded the road information. If not, the location information of the neighboring vehicle is supplemented according to the road information uploaded by the current vehicle, that is, the process goes to step S207. On the contrary, if it is detected that the neighboring vehicle has uploaded the road information, no additional supplement is made to its location information.

[0087] S207. Determine the real-time location information of other neighboring vehicles based on the road information uploaded by the current vehicle.

[0088] Specifically, the intelligent road condition monitoring system selects vehicles whose field of view coverage is close to the vehicles that have not uploaded data from the vehicles that have successfully uploaded data as references. By analyzing the environmental perception information of these vehicles, such as recognizing the license plate of the preceding vehicle and measuring the distance, the system can infer the relative position of the target vehicle.

[0089] In the above embodiment, when the intelligent road condition monitoring system fails to obtain data of a certain vehicle, it determines the location of the vehicle based on information obtained from nearby vehicles and adds it to its own collected data, thereby restoring the road scene to the greatest extent and ensuring the integrity of the road monitoring information.

[0090] S208: Generate second type of three-dimensional data based on the road information.

[0091] The intelligent road condition monitoring system cleans and standardizes road information data from different sources and formats. Then, it uses data association analysis, semantic understanding and other technologies to fuse multi-source heterogeneous information. For example, it aligns the vehicle video frame with the lidar point cloud to generate a color point cloud to increase the richness of environmental information; it splices the local perception of multiple vehicles into a global map to expand the perception range.

[0092] Through intelligent algorithm processing, the system aggregates the massive amount of scattered vehicle data into a structured and semantic three-dimensional road model. The three-dimensional road model contains high-fidelity restoration of multiple dimensions such as road surface, traffic participants, and weather conditions. This second type of data complements the first type of data and participates in the next step of road condition analysis.

[0093] S209: Obtain road information uploaded by the intelligent driving assistance system of only one vehicle within a preset time interval.

[0094] After step S204, if it is detected that the vehicle speed within the preset range is always maintained within the normal speed range, it is determined that the vehicle is driving normally, and only the road information uploaded by the intelligent driving assistance system of one vehicle is obtained within the preset time interval, thereby reducing the frequency of obtaining road information data.

[0095] S210 , aligning the preprocessed first-category and second-category three-dimensional data in time and space, so that the first-category and second-category three-dimensional data are consistent in time stamp and spatial position.

[0096] After obtaining the first type of data from fixed node cameras and sensors and the second type of data from the vehicle system, the intelligent road condition monitoring system first pre-processes these heterogeneous raw data (such as time synchronization calibration, coordinate system conversion, feature extraction, data cleaning, etc.) to unify the data format, eliminate noise and outliers, and improve data quality. Then, align the three-dimensional data from different sources and formats in the time and space dimensions. Specifically, the intelligent road condition monitoring system unifies the data timestamps of different sensors to the same benchmark, then extracts the timestamp information from the data packet, and uses network time synchronization protocols such as NTP, PTP and other technologies to calculate the time deviation of each data source and perform compensation calibration, so that the data of different devices are completely synchronized on the time axis.

[0097] Next, the system establishes a unified global coordinate frame and estimates the pose transformation matrix of each sensor relative to the global frame to achieve sensor calibration. For example, the camera extrinsic parameters can be solved by using known control points or feature points through computer vision methods such as hand-eye calibration; point cloud registration techniques such as ICP and NDT can be used to calculate the rigid body transformation between the lidar point cloud and the global map. After obtaining the extrinsic parameters of each sensor, the system uniformly transforms the locally collected three-dimensional data into the global coordinate system, thereby achieving spatiotemporal alignment.

[0098] In the above embodiment, the intelligent road condition monitoring system aligns time and space to ensure that the data collected by different sensors are consistent in the time and space dimensions. On this basis, data fusion technologies such as weight fusion and decision fusion are used to optimize the combination of the aligned first and second three-dimensional data, overcome the defects of single data, give play to the complementary advantages of multi-source data, and finally obtain a three-dimensional model of the local road section with more comprehensive information and higher accuracy.

[0099] S211. Use the first type of three-dimensional data to construct a basic framework of the multimodal three-dimensional model, and use the second type of three-dimensional data to fill the basic framework to obtain a multimodal three-dimensional model of the traffic node within a preset range.

[0100] After completing the alignment of spatiotemporal data, the intelligent road condition monitoring system will use these multi-source data to build a multimodal three-dimensional model that reflects the current road conditions. Specifically, the intelligent road condition monitoring system gives priority to the first type of data with fixed nodes, such as lidar point clouds, video images, etc., and constructs the geometric structure of the scene through three-dimensional reconstruction technology (such as structured light, multi-view stereo vision, SFM, SLAM, etc.). By imaging the target scene from different angles and matching the features of different perspectives, the camera motion and scene depth are calculated to restore the three-dimensional shape. In addition, since the lidar directly obtains dense point clouds, it is more used to provide accurate scale information. The point cloud is downsampled by setting the voxel size, and the surface mesh is generated using methods such as greedy triangulation and Poisson reconstruction. The final result is a framework model that reflects the geometry of key road sections.

[0101] Next, the system registers the second type of data collected by the on-board sensors, such as mobile lidar point clouds and lane line recognition results, with the framework model. The scale and position information provided by the framework can improve the registration accuracy of mobile data. Specifically, the system estimates the instantaneous motion of dynamic targets such as vehicles and pedestrians in the scene by tracking the features of multiple consecutive frames of images, and performs motion compensation on the registered point cloud accordingly.

[0102] In the above embodiment, the intelligent road condition monitoring system constructs a three-dimensional model framework with the first type of data collected by fixed nodes, and then enriches the model details with the second type of data collected by the vehicle-mounted system, thereby further optimizing the data fusion effect and improving the accuracy and coverage of the multimodal three-dimensional model.

[0103] S212: Determine the congested road section and the corresponding congestion cause from the multimodal three-dimensional model of the entire road section.

[0104] After the intelligent road condition monitoring system obtains a detailed three-dimensional model of a local intersection, it needs to further seamlessly stitch the models of each node to finally obtain a large-scale model that reflects the real-time status of the entire road. Using the environmental information in the model, the system can analyze potential sections of traffic congestion from the spatiotemporal dimensions of traffic density and speed.

[0105] Specifically, the system divides the three-dimensional road model into grids to form a spatiotemporal grid map composed of multiple cells. Based on the number of vehicles in each cell, the local traffic flow density distribution map can be estimated. By aggregating and analyzing the changing trend of vehicle density in cells on the time axis, abnormal areas where vehicles gradually gather and the stagnation time exceeds the threshold are identified, and they are preliminarily determined as sections of road that may be congested.

[0106] In the above embodiment, the intelligent road condition monitoring system identifies potential congested areas where traffic flow density is abnormally increased and vehicle speed is significantly reduced through spatiotemporal analysis of vehicle density and speed in the three-dimensional road model. Then, the intelligent road condition monitoring system analyzes road data before and after the congested section, such as road surface conditions, accident information, changes in the number of lanes, etc., to infer the causes of the congestion and provide a reference for management decisions.

[0107] S213: Count the traffic volume at each traffic node within a preset time period, and predict the congested traffic nodes within a preset time period in the future.

[0108] Specifically, the system extracts the vehicle traffic data of each intersection or road section within a certain historical time range from the multimodal 3D model of the entire road section, and statistically analyzes the changes in traffic volume at different time periods. Generally, the sampling interval is 15 minutes, and the traffic flow conditions of each time slice in a day are fully recorded.

[0109] Next, the traffic flow prediction model is trained based on the traffic flow time series data. After the training is completed, the model uses the current traffic flow status as input to predict the traffic flow change trend of each time slice in the next hour. If the predicted traffic flow at some nodes is significantly higher than the critical threshold, the system will determine that the node may experience traffic congestion during the corresponding period.

[0110] In the above embodiment, the intelligent road condition monitoring system collects traffic flow data of each intersection within a certain period of time to determine the traffic flow distribution characteristics of the node; on this basis, it uses traffic flow prediction models such as time series, machine learning and other methods, combined with the road network structure and the spatiotemporal correlation of traffic flow to predict the traffic flow change trend of each node in the short term, thereby judging the sections of the road where congestion may occur, and providing a basis for measures such as traffic induction and signal light timing.

[0111] The following is an exemplary schematic diagram of a scenario in which the intelligent road condition monitoring system in the embodiment of the present application obtains the first type of three-dimensional data and the second type of three-dimensional data. Figure 3 shown.

[0112] Among them, serial number 5 is a road information collection device installed at a certain traffic node; serial number 1 is the monitoring range of the road information collection device; serial number 2 is the communication range of the road information collection device, that is, the preset range; serial number 3 is the road section within the preset range; serial number 4 is the road section within the preset range corresponding to the road information collection device installed at the traffic node adjacent to serial number 5; A, B, and C are all cars in motion.

[0113] In an embodiment of the present application, the intelligent road condition monitoring system obtains the first type of three-dimensional data within the range of sequence number 1 through sequence number 5, and obtains the second type of three-dimensional data collected by its intelligent driving assistance system by establishing communication with the car entering the range of sequence number 2.

[0114] It should be noted that Figure 3 Only the collection process of the first and second 3D data in a certain direction of the traffic node is shown. The collection process of the first and second 3D data in the opposite direction is basically the same as that in this direction, and will not be repeated here.

[0115] The intelligent road condition monitoring system of the embodiment of the present invention is applied to electronic equipment. Figure 4 A schematic diagram of the architecture of an electronic device suitable for implementing an embodiment of the present invention is shown.

[0116] It should be noted that Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0117] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructions (computer programs), or by controlling related hardware through instructions (computer programs), and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. The electronic device of this embodiment includes a storage medium and a processor, wherein a plurality of instructions are stored in the storage medium, and the instructions can be loaded by the processor to execute any step of the method provided in the embodiment of the present invention.

[0118] Specifically, the storage medium and the processor are electrically connected directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more signal lines. The storage medium stores computer execution instructions for implementing the data access control method, including at least one software function module that can be stored in the storage medium in the form of software or firmware. The processor executes various functional applications and data processing by running the software program and module stored in the storage medium. The storage medium can be, but is not limited to, random access storage medium (Random Access Memory, referred to as: RAM), read-only storage medium (Read Only Memory, referred to as: ROM), programmable read-only storage medium (Programmable Read-Only Memory, referred to as: PROM), erasable read-only storage medium (Erasable Programmable Read-Only Memory, referred to as: EPROM), electrically erasable read-only storage medium (Electric Erasable Programmable Read-Only Memory, referred to as: EEPROM), etc. Among them, the storage medium is used to store programs, and the processor executes the program after receiving the execution instruction.

[0119] Furthermore, the software programs and modules in the above-mentioned storage medium may also include an operating system, which may include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide an operating environment for other software components. The processor may be an integrated circuit chip having signal processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc., which may implement or execute the various methods, steps, and logic flow diagrams disclosed in this embodiment. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0120] Since the instructions stored in the storage medium can execute the steps in any method provided in the embodiments of the present invention, the beneficial effects of any method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0121] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multimodal three-dimensional data processing method based on road features, applied to an intelligent road condition monitoring system, characterized in that: The method comprises: Acquire the first type of three-dimensional data of the road section within the monitoring range through a road information collection device installed at a traffic node, wherein the road information collection device includes a camera and a sensor; Establishing a communication connection with the car that enters the road section within the preset range, obtaining the road information uploaded by the intelligent driving assistance system, and obtaining the second type of three-dimensional data, wherein the preset range is larger than the monitoring range, and the two preset ranges corresponding to the adjacent traffic nodes partially intersect or are completely mutually exclusive, so that the communication range covers the entire road section, and the car can obtain the road conditions shared by other vehicles while uploading the road information collected by itself; If the road section within the preset range is a highway section or an elevated road section, after establishing a communication connection with the car entering the road section within the preset range, the speed data of the car is obtained first; it is determined whether the speed data is within the normal speed range; if so, only the road information uploaded by the intelligent driving assistance system of one car is obtained within a preset time interval; if not, the road information uploaded by the intelligent driving assistance systems of all cars is obtained; and the second type of three-dimensional data is generated according to the road information; After determining that there are other adjacent vehicles ahead according to the road information uploaded by the current vehicle, detecting whether the road information uploaded by the other adjacent vehicles is received; if the road information uploaded by the other adjacent vehicles is not received, determining the real-time position information of the other adjacent vehicles according to the road information uploaded by the current vehicle, and the real-time position information is added to the second type of three-dimensional data; The first type of three-dimensional data and the second type of three-dimensional data are merged to obtain a multimodal three-dimensional model of the traffic node within a preset range; Integrate the multimodal three-dimensional model of each of the traffic nodes to obtain a multimodal three-dimensional model of the entire road section; The road condition is dynamically displayed based on the multimodal three-dimensional model of the entire road section.

2. The method according to claim 1, characterized in that: The step of fusing the first type of three-dimensional data and the second type of three-dimensional data to obtain a multimodal three-dimensional model of the traffic node within a preset range specifically includes: Performing temporal and spatial alignment on the preprocessed first-category three-dimensional data and the second-category three-dimensional data, so that the first-category three-dimensional data and the second-category three-dimensional data are consistent in terms of time stamp and spatial position; The aligned first-category three-dimensional data and the second-category three-dimensional data are fused using a preset data fusion technology to obtain a multimodal three-dimensional model of the traffic node within a preset range.

3. The method according to claim 2, characterized in that The step of fusing the aligned first-category three-dimensional data and second-category three-dimensional data using a preset data fusion technology to obtain a multimodal three-dimensional model of the traffic node within a preset range specifically includes: Using the first type of three-dimensional data to construct a basic framework of the multimodal three-dimensional model; The basic framework is filled with the second type of three-dimensional data to obtain a multimodal three-dimensional model of the traffic node within a preset range.

4. The method according to claim 1, characterized in that: After the step of integrating the multimodal three-dimensional model of each of the traffic nodes to obtain the multimodal three-dimensional model of the entire road section, the method further includes: Determining a congested road section from the multimodal three-dimensional model of the entire road section; The cause of congestion of the congested road section is determined based on the road data recorded by the multimodal three-dimensional model.

5. The method according to claim 1, characterized in that After the step of dynamically displaying the road condition based on the multimodal three-dimensional model of the entire road section, the method further includes: Count the traffic volume at each traffic node within a preset time period; The traffic congestion nodes within a preset time in the future are predicted based on the traffic flow.

6. An intelligent road condition monitoring system, characterized in that: The intelligent road condition monitoring system includes: one or more processors and memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the intelligent road condition monitoring system to execute the method as described in any one of claims 1-5.

7. A computer-readable storage medium comprising instructions, characterized in that: When the instruction is executed on the intelligent road condition monitoring system, the intelligent road condition monitoring system executes the method as claimed in any one of claims 1 to 5.

8. A computer program product, characterized in that When the computer program product runs on an intelligent road condition monitoring system, the intelligent road condition monitoring system is enabled to execute the method according to any one of claims 1 to 5.

Images