A digital twin method and device for confluence area traffic roads

By using lane line semantic segmentation detection network and cross-modal multi-level feature fusion detection network in confluent area traffic scenarios, combined with depth estimation methods, a digital twin model of traffic roads is solved in real time, and the low degree of automation and construction in the existing technology is solved. The problem of low efficiency achieves efficient real-time monitoring of traffic scenarios in complex confluence areas.

CN118411517BActive Publication Date: 2024-11-15RES INST OF HIGHWAY MINIST OF TRANSPORT
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410457090.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-16
Publication Date
2024-11-15
Estimated Expiration
2044-04-16

AI Technical Summary

Technical Problem

The existing traffic road twin model construction process is low in degree, requires a lot of manpower and material resources, and the construction efficiency is low, so it is impossible to monitor traffic scenes in complex confluence areas in real time.

Method used

Using the radar point cloud and image based on lane line, the lane line semantic segmentation detection network and cross-modal multi-level feature fusion detection network are used to obtain the semantic segmentation results and detection results of lane line and traffic targets, and combined with the depth estimation method, three-dimensional Acquisition of pixel information and position information, perform segmentation fitting and model construction, and build a digital twin model of traffic scenes in the confluent area in real time.

Benefits of technology

The construction efficiency and real-time nature of the digital twin model of traffic scenes in the confluent area is improved, ensuring the real-time simulation of lane lines and traffic target data, adapting to complex traffic scenarios, and real-time supervision of traffic status.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118411517B_ABST
    Figure CN118411517B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of digital twin transportation, and specifically, to a digital twin method and device for confluence area traffic roads. Based on the radar point cloud and image of the lane line, the semantic segmentation result of the lane line is obtained through the lane line semantic segmentation detection network; based on the semantic segmentation result, the instance segmentation result of the lane line is obtained; based on the instance segmentation result, the three-dimensional pixel information of the lane line is obtained through the depth estimation method; based on the three-dimensional pixel information, the segmented fitting is performed to obtain the fitting coordinate result of the lane line; based on the radar point cloud and image of the traffic target, the detection result of the traffic target is obtained through the cross-modal multi-level feature fusion detection network; based on the detection result, the location information of the traffic target is obtained through the depth estimation method; the fitting coordinate result of the lane line is combined with the location information of the traffic target to obtain the digital twin model of the confluence area traffic scene, so as to realize the real-time construction of the digital twin model of the complex confluence area traffic scene.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of digital twin transportation, and in particular to a digital twin method and device for merging area traffic roads. Background Art

[0002] With the development of science and technology, the intelligent vehicle-road cooperative system came into being. The intelligent vehicle-road cooperative system is an intelligent transportation system composed of multiple components such as vehicle-mounted terminals, roadside equipment, and traffic management centers, which implements information interaction between vehicles and roads in all directions. The roadside equipment in the intelligent vehicle-road cooperative system collects and analyzes traffic data, can monitor the traffic status of the road in real time, provide real-time road condition information and driving assistance services, reduce traffic congestion, and improve road safety performance. However, in order to create a smart road and realize real-time monitoring of road traffic status, in addition to the real-time digitization of the people and vehicles participating in the traffic, it is also necessary to digitize the physical structure of the road, that is, the road itself, and build a twin scene of the traffic road.

[0003] At present, the construction of traffic road twin scenes mainly collects road data through information collection methods such as drone mapping, oblique photography, high-resolution remote sensing, satellite images, etc., and manages the data using BIM+GIS technology, and then uses three-dimensional modeling technology to complete the construction of road twin scenes in model rendering software, which is mainly used for offline simulation testing of autonomous driving vehicle perception, planning and control. Although the traffic road twin model constructed by it has a high degree of refinement and restoration, the model construction process requires manual screening and processing of data, with a low degree of automation, requiring a large amount of manpower and material resources, and the transformation process from data to scenes takes a long time, with much manual intervention, and the scene construction efficiency is low. In addition, with the rapid growth of the number of civilian cars in my country, congestion and traffic accidents often occur in the confluence areas of key traffic areas, which not only affects people's travel convenience, but also poses a great threat to travel safety. Therefore, how to complete the real-time construction of traffic road twin models for complex confluence areas based on the original data collected by roadside multi-sensor equipment has become a problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of the present invention is to provide a digital twin method and device for confluence area traffic roads, so as to solve the problem of how to complete the real-time construction of a twin model for complex confluence area traffic roads based on the original data collected by multi-sensor equipment on the roadside.

[0005] An embodiment of the present invention provides a digital twin method for a traffic road in a confluence area, the method comprising the following steps: based on a radar point cloud and an image of a lane line, obtaining a semantic segmentation result of the lane line through a lane line semantic segmentation detection network, the lane line semantic segmentation detection network comprising a dual-channel network feature extraction module and a multi-level feature fusion module; based on the semantic segmentation result, performing instance segmentation through a clustering algorithm to obtain an instance segmentation result of the lane line; based on the instance segmentation result, obtaining three-dimensional pixel information of the lane line through a depth estimation method; performing segmented fitting based on the three-dimensional pixel information to obtain a fitting coordinate result of the lane line; based on the radar point cloud and the image of the traffic target, obtaining a detection result of the traffic target through a cross-modal multi-level feature fusion detection network; based on the detection result, obtaining the position information of the traffic target through a depth estimation method, the position information of the traffic target including the relative position coordinates and relative azimuth angle of the traffic target relative to an intelligent road test device; combining the fitting coordinate result of the lane line with the position information of the traffic target to obtain a digital twin model of the confluence area traffic scene.

[0006] Optionally, the dual-channel network feature extraction module includes a first channel and a second channel; the step of obtaining the semantic segmentation result of the lane line through a lane line semantic segmentation detection network based on the radar point cloud and image of the lane line includes: extracting low-detail features from the radar point cloud and image of the lane line through the variable attention module of the first channel; extracting high-level semantic features from the radar point cloud and image of the lane line through the deformable convolution module of the second channel; and fusing the low-detail features with the high-level semantic features through the multi-level feature fusion module to obtain a semantic segmentation result.

[0007] Optionally, the step of obtaining the detection result of the traffic target through a cross-modal multi-level feature fusion detection network based on the radar point cloud and image of the traffic target includes: performing cylinder column encoding on the radar point cloud of the traffic target, filtering out the radar point cloud of the traffic target in the area that does not intersect with the image of the traffic target; extracting volume column features from the remaining radar point cloud of the traffic target through the PointNet backbone network to obtain feature maps of the radar point cloud of the traffic target at different scales; performing perspective transformation on the image of the traffic target to obtain an image of the traffic target at a specific perspective; performing multi-level feature extraction from the image of the traffic target at a specific perspective through the ResNet backbone network to obtain high-level semantic information and low-level features; adopting a multi-scale strategy to combine high-level semantic information with low-level features to obtain a feature map of the image of the traffic target; and fusing the feature map of the radar point cloud of the traffic target and the feature map of the image of the traffic target through a variable attention module to obtain the detection result of the traffic target.

[0008] Optionally, the instance segmentation result includes two-dimensional data and three-dimensional data; the step of obtaining the three-dimensional pixel information of the lane line by a depth estimation method based on the instance segmentation result includes: projecting the three-dimensional data onto the plane of the two-dimensional data to obtain a depth reference value corresponding to the first part of the pixels; based on the depth reference value, using a bilateral filtering method to estimate the missing depth value of the second part of the pixels, so as to obtain the three-dimensional pixel information of the lane line; wherein the first part of the pixels and the second part of the pixels are complementary to each other and constitute the complete pixel points of the lane line.

[0009] Optionally, the bilateral filtering method includes: searching for four reference pixels that are closest to the pixel with missing depth value and have depth reference values ​​in the neighborhood of the pixel with missing depth value; the depth reference values ​​of the four reference pixels are converted into the depth value of the pixel with missing depth value by using a depth value estimation formula; the depth value estimation formula is:

[0010]

[0011]

[0012]

[0013] Among them, r * is the depth value of the pixel whose depth value is missing, W is the normalized weight coefficient, is the Euclidean distance weight, is the depth value difference weight, x is the pixel point where the depth value is missing, and x i are the four reference pixels, ‖xx i ‖ is the Euclidean norm, r i is the reference depth value of the four reference pixels, r is the minimum value of the reference depth values ​​of the four reference pixels, and i=1, 2, 3, 4.

[0014] Optionally, the step of performing segmented fitting based on the three-dimensional pixel information to obtain the fitting coordinate result of the lane line includes: establishing a cubic polynomial fitting model by the least squares method based on the three-dimensional pixel information; obtaining segmentation points by performing row scanning on the cubic polynomial fitting model; and performing straight line model fitting in a segmented manner based on the two-dimensional pixel coordinates of the segmentation points to obtain the fitting coordinate result of the lane line.

[0015] Optionally, the step of obtaining segmentation points by row scanning the model includes: performing row-by-row pixel scanning on the fitting curve of the cubic polynomial fitting model through a row coordinate axis; when the lane line passed by the row coordinate axis changes from one to two, obtaining the row coordinates of the segmentation point at its critical point; substituting the row coordinates into the fitting curve to obtain the two-dimensional pixel coordinates of the segmentation point; wherein the row coordinate axis is approximately perpendicular to the direction of the lane line.

[0016] Optionally, the step of combining the fitting coordinate results of the lane lines with the position information of the traffic targets to obtain a digital twin model of the confluence area traffic scene includes: writing a road network file according to the fitting coordinate results of the lane lines, and establishing a road model through a traffic road scene editor; establishing a traffic target model through simulation with a simulator according to the position information of the traffic targets; and inputting the traffic target model into the road model to obtain a digital twin model of the confluence area traffic scene.

[0017] Optionally, the method further includes: obtaining the radar point cloud and the image by accurately calibrating the parameters of the intelligent road test equipment; wherein the radar point cloud is three-dimensional data, the image is two-dimensional data, and the intelligent road test equipment includes a laser radar and a visual sensor.

[0018] Compared with the prior art, the digital twin method for merging area traffic roads provided by the present invention has the following beneficial effects:

[0019] The digital twin method for traffic roads in a converging area provided by an embodiment of the present invention obtains semantic segmentation results of lane lines through a lane line semantic segmentation detection network based on radar point clouds and images of lane lines, wherein the lane line semantic segmentation detection network includes a dual-channel network feature extraction module and a multi-level feature fusion module to avoid the problem of missed detection of crossing lane lines in the converging area; based on the above semantic segmentation results, instance segmentation is performed through a clustering algorithm to obtain instance segmentation results of lane lines; based on the above instance segmentation results, three-dimensional pixel information of lane lines is obtained through a depth estimation method; based on the above three-dimensional pixel information, piecewise fitting is performed to obtain the fitting coordinate results of lane lines, and the fitting of lane lines of the converging area traffic scene is realized through piecewise fitting, which is used for the later digital twin of the converging area traffic scene. The digital twin model of the confluence area traffic scene is constructed to provide a data basis, ensuring that the digital twin model of the confluence area traffic scene can adapt to the complex confluence area traffic scene; based on the radar point cloud and image of the traffic target, the detection result of the traffic target is obtained through a cross-modal multi-level feature fusion detection network, so as to realize the accurate detection of the traffic target; based on the above detection results, the position information of the traffic target is obtained through a depth estimation method, and the position information of the above traffic target includes the relative position coordinates and relative azimuth angle of the traffic target relative to the intelligent road test equipment; the fitting coordinate results of the above lane line are combined with the position information of the above traffic target to obtain the digital twin model of the confluence area traffic scene, and the real-time digital simulation of the lane line data and the traffic target data is ensured by the above method, so as to realize the real-time construction of the digital twin model of the confluence area traffic scene.

[0020] An embodiment of the present invention provides a digital twin device for a traffic road in a merging area, the device comprising:

[0021] A first initial module is used to obtain a semantic segmentation result of the lane line through a lane line semantic segmentation detection network based on a radar point cloud and an image of the lane line, wherein the lane line semantic segmentation detection network includes a dual-channel network feature extraction module and a multi-level feature fusion module;

[0022] A second initial module is used to perform instance segmentation based on the semantic segmentation result by a clustering algorithm to obtain an instance segmentation result of the lane line;

[0023] A third initial module, configured to obtain three-dimensional pixel information of the lane line by a depth estimation method based on the instance segmentation result;

[0024] A fourth initial module, configured to perform segmented fitting based on the three-dimensional pixel information to obtain a fitting coordinate result of the lane line;

[0025] A first real-time module is used to obtain a detection result of the traffic target through a cross-modal multi-level feature fusion detection network based on a radar point cloud and an image of the traffic target;

[0026] A second real-time module is used to obtain the position information of the traffic target by a depth estimation method based on the detection result, wherein the position information of the traffic target includes the relative position coordinates and relative azimuth angle of the traffic target relative to the intelligent road test device;

[0027] The third real-time module is used to combine the fitting coordinate results of the lane line with the position information of the traffic target to obtain a digital twin model of the confluence area traffic scene.

[0028] The digital twin device for confluence area traffic roads provided by the embodiment of the present invention has the beneficial effects of: by setting lane line fitting as the initial module, repeated fitting of lane lines can be avoided, thereby improving the efficiency of model construction without losing the real-time performance of lane line fitting. In addition, when the lane line changes, the lane line can be fitted again through the initial module; by setting traffic target positioning and confluence area traffic scene digital twin models as real-time modules, the real-time performance of the confluence area traffic scene digital twin models can be guaranteed, thereby achieving real-time supervision of traffic status. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0030] Figure 1 A schematic flow chart of a digital twin method for a merging area traffic road provided in an embodiment of the present invention;

[0031] Figure 2 A schematic flow chart of a real-time digital twin method for traffic roads in complex confluence areas provided by an embodiment of the present invention;

[0032] Figure 3 A schematic effect diagram of lane line detection for a traffic road in a complex merging area provided by an embodiment of the present invention;

[0033] Figure 4 A schematic rendering of a real-time digital twin of a traffic road in a complex confluence area provided by an embodiment of the present invention;

[0034] Figure 5 A schematic structural diagram of a real-time digital twin device for traffic roads in complex confluence areas provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0036] Early digital twin research focused on traditional urban planning, architecture and civil engineering, and industrial manufacturing. In the field of smart city design and transportation, building three-dimensional models provides a variety of applications such as visualization, spatial analysis, and traffic simulation. Most of the existing traffic road twin technologies are based on graphics rendering technology to model road scenes. Although the model is highly refined, it has problems such as complex data sources, many manual processing operations, low automation, and a long conversion cycle from data to models.

[0037] Traffic congestion and accidents often occur in confluence areas of complex traffic scenarios. It is necessary to build a simplified twin model of the traffic roads in the confluence area to monitor and predict the traffic status in real time to alleviate traffic pressure. However, in the construction of the current traffic road twin model, there are problems such as the need for manual data screening, low degree of automation, and long model construction cycle. In addition, considering that lane lines in traffic roads are important traffic signs, sensing them can provide sufficient road information for the construction of traffic road twin models. Taking all the above factors into consideration, the embodiment of the present invention starts with the original data collected by the multi-sensor equipment on the road side, and based on the lane line detection information and traffic target provided by the embodiment of the present invention, the real-time construction technology of the traffic road twin model in the confluence area improves the construction efficiency of the traffic road twin model, provides underlying technology for real-time supervision and prediction of traffic status and construction of simulated road environment, and is of great significance to promoting the development of intelligent transportation systems.

[0038] The embodiment of the present invention provides a digital twin method for a traffic road in a merging area, see Figure 1 A schematic flow chart of a digital twin method for a merging area traffic road is shown, the method comprising the following steps:

[0039] S110, based on the radar point cloud and image of the lane line, a lane line semantic segmentation detection network is used to obtain a lane line semantic segmentation result.

[0040] Among them, the above-mentioned lane line semantic segmentation detection network includes a dual-channel network feature extraction module and a multi-level feature fusion module.

[0041] Optionally, the dual-channel network feature extraction module includes a first channel and a second channel; the step S110 includes: extracting low-detail features from the radar point cloud and image of the lane line through the variable attention module of the first channel; extracting high-level semantic features from the radar point cloud and image of the lane line through the deformable convolution module of the second channel; and fusing the low-detail features with the high-level semantic features through the multi-level feature fusion module to obtain a semantic segmentation result.

[0042] S112: Based on the above semantic segmentation result, instance segmentation is performed using a clustering algorithm to obtain an instance segmentation result of the lane line.

[0043] Exemplarily, the above clustering algorithm may adopt the DBSCAN clustering algorithm.

[0044] S114, based on the above instance segmentation result, obtain the three-dimensional pixel information of the lane line through a depth estimation method.

[0045] S116, performing segmented fitting based on the above three-dimensional pixel information to obtain the fitting coordinate results of the lane line.

[0046] Optionally, the above step S116 includes: establishing a cubic polynomial fitting model through the least squares method based on the above three-dimensional pixel information; obtaining segmentation points by performing row scanning on the above cubic polynomial fitting model; and fitting a straight line model in a segmented manner based on the two-dimensional pixel coordinates of the above segmentation points to obtain the fitting coordinate result of the lane line.

[0047] It should be noted that the above lane line fitting coordinate results are three-dimensional data.

[0048] Optionally, the step of obtaining segmentation points by row scanning the model includes: performing row-by-row pixel scanning on the fitting curve of the cubic polynomial fitting model through a row coordinate axis; when the lane line passed by the row coordinate axis changes from one to two, obtaining the row coordinates of the segmentation points at the critical point; substituting the row coordinates into the fitting curve to obtain the two-dimensional pixel coordinates of the segmentation points; wherein the row coordinate axis is approximately perpendicular to the direction of the lane line.

[0049] S118, based on the radar point cloud and image of the traffic target, the detection result of the traffic target is obtained through a cross-modal multi-level feature fusion detection network.

[0050] Optionally, the above-mentioned step S118 includes: performing cylindrical column encoding on the radar point cloud of the above-mentioned traffic target, filtering out the radar point cloud of the above-mentioned traffic target in the area that does not intersect with the image of the above-mentioned traffic target; extracting column features from the remaining radar point cloud of the above-mentioned traffic target through the PointNet backbone network to obtain feature maps of radar point clouds of traffic targets of different scales; performing perspective transformation on the image of the above-mentioned traffic target to obtain an image of the traffic target at a specific perspective; performing multi-level feature extraction from the image of the traffic target at the specific perspective through the ResNet backbone network to obtain high-level semantic information and low-level features; adopting a multi-scale strategy to combine high-level semantic information with low-level features to obtain a feature map of the image of the traffic target; and fusing the feature map of the radar point cloud of the above-mentioned traffic target and the feature map of the image of the above-mentioned traffic target through a variable attention module to obtain a detection result of the traffic target.

[0051] It should be noted that the above-mentioned traffic targets include targets such as vehicles and pedestrians, and the above-mentioned specific perspective may be a BEV (Bird's-eye-view) perspective.

[0052] S120: Based on the above detection results, obtain the location information of the traffic target through a depth estimation method.

[0053] The location information of the above-mentioned traffic target includes the relative location coordinates and relative azimuth angle of the traffic target relative to the intelligent road test equipment.

[0054] S122, combining the fitting coordinate results of the lane lines with the position information of the traffic targets to obtain a digital twin model of the confluence area traffic scene.

[0055] Optionally, the above step S122 includes: writing a road network file according to the fitting coordinate results of the above lane lines, and establishing a road model through a traffic road scene editor; establishing a traffic target model through simulation by a simulator according to the location information of the above traffic targets; and inputting the above traffic target model into the above road model to obtain a digital twin model of the confluence area traffic scene.

[0056] It should be noted that the depth estimation method used in the above step S114 is the same as that used in the above step S120, and those skilled in the art can obtain the specific method of use in the above step S120 based on the specific method of use in the above step S114.

[0057] Specifically, the above-mentioned instance segmentation result includes three-dimensional data and two-dimensional data. Exemplarily, the above-mentioned step S114 includes: projecting the above-mentioned three-dimensional data onto the plane of the above-mentioned two-dimensional data to obtain the depth reference value corresponding to the first part of the pixel points; based on the above-mentioned depth reference value, using the bilateral filtering method to estimate the depth value of the missing depth value of the second part of the pixel points, and obtaining the three-dimensional pixel information of the lane line; wherein the above-mentioned first part of the pixel points and the above-mentioned second part of the pixel points are complementary to each other and constitute the pixel points of the complete lane line.

[0058] Optionally, the bilateral filtering method includes: searching for four reference pixels that are closest to the pixel with missing depth value and have depth reference values ​​in the neighborhood of the pixel with missing depth value; the depth reference values ​​of the four reference pixels are converted into the depth value of the pixel with missing depth value by a depth value estimation formula; the depth value estimation formula is:

[0059]

[0060]

[0061]

[0062] Among them, r * is the depth value of the pixel whose depth value is missing, W is the normalized weight coefficient, is the Euclidean distance weight, is the depth value difference weight, x is the pixel point where the above depth value is missing, and x i are the four reference pixels mentioned above, ‖xx i ‖ is the Euclidean norm, r i is the reference depth value of the above four reference pixels, r is the minimum value of the reference depth values ​​of the above four reference pixels, and i=1,2,3,4.

[0063] It should be noted that based on the above-mentioned pixel points with missing depth values ​​and the above-mentioned four reference pixel points, a two-dimensional Euclidean geometric space is constructed, and then the corresponding Euclidean norm is defined; specifically, the above-mentioned Euclidean geometric space is a vector space, in which the length of the directed line segment of each vector is the Euclidean norm of the vector, that is, the above-mentioned ‖xx i ‖.

[0064] Optionally, the method further includes: obtaining the radar point cloud and the image by accurately calibrating the parameters of the intelligent road test equipment; wherein the radar point cloud is three-dimensional data, the image is two-dimensional data, and the intelligent road test equipment includes a laser radar and a visual sensor.

[0065] It should be noted that the radar point cloud and image of the lane line and the radar point cloud and image of the traffic target are the same data, which are detected and acquired by the above-mentioned intelligent road test equipment.

[0066] The digital twin method for converging area traffic roads provided by the embodiment of the present invention obtains the semantic segmentation results of lane lines through a lane line semantic segmentation detection network based on the radar point cloud and image of the lane lines, wherein the lane line semantic segmentation detection network includes a dual-channel network feature extraction module and a multi-level feature fusion module to avoid the problem of missed detection of crossing lane lines in the converging area; based on the above semantic segmentation results, instance segmentation is performed through a clustering algorithm to obtain instance segmentation results of lane lines; based on the above instance segmentation results, three-dimensional pixel information of lane lines is obtained through a depth estimation method; based on the above three-dimensional pixel information, piecewise fitting is performed to obtain the fitting coordinate results of lane lines, and the fitting of lane lines of the converging area traffic scene is realized through piecewise fitting, which is for the later digital converging area traffic scene. The construction of the twin model provides a data basis to ensure that the digital twin model of the confluence area traffic scene can adapt to the complex confluence area traffic scene; based on the radar point cloud and image of the traffic target, the detection results of the traffic target are obtained through a cross-modal multi-level feature fusion detection network to achieve accurate detection of the traffic target; based on the above detection results, the position information of the traffic target is obtained through a depth estimation method, and the position information of the above traffic target includes the relative position coordinates and relative azimuth angle of the traffic target relative to the intelligent road test equipment; the fitting coordinate results of the above lane lines are combined with the position information of the above traffic targets to obtain the digital twin model of the confluence area traffic scene. The above method is used to ensure the real-time digital simulation of the lane lines and traffic targets, and realize the real-time construction of the digital twin model of the confluence area traffic scene.

[0067] See also Figure 2 A schematic flow chart of a real-time digital twin method for complex confluence area traffic roads is shown. An embodiment of the present invention provides a real-time digital twin method for complex confluence area traffic roads. The radar point cloud and image data collected by the intelligent road test equipment sensor are used to perform accurate parameter calibration. The real-time lane line semantic segmentation detection network combined with the multi-level feature cross-fusion module performs real-time semantic segmentation on the lane lines in the confluence area; the cross-modal multi-level feature module detection network performs real-time and accurate detection and positioning of vehicles and pedestrians in the confluence area; finally, the lane line and target detection results are combined to build a real-time digital twin model for the confluence area, providing underlying technology for real-time supervision and prediction of traffic status and construction of simulated road environment, which is of great significance to promoting the development of intelligent transportation systems. The specific steps are as follows:

[0068] Step 1: In complex traffic scenarios (convergence areas), there are intersecting roads in the converging areas, which leads to complex shapes and topological structures of lane lines. To address the problem of missed detection of intersecting lane lines in converging areas, a real-time lane line semantic segmentation detection network combining a variable attention mechanism and a multi-level feature cross-fusion module is used.

[0069] Sub-step 1: Construct a dual-channel network feature extraction module, where the first channel is a module containing a variable attention mechanism to extract low-detail information features, such as the shape of the lane lines and edge shape features in the confluence area. The other channel adopts deformable convolution to adjust the sampling points of the convolution kernel at different positions and adaptively adjust the position of the convolution kernel to reduce the model's sensitivity to background noise, thereby extracting the deep semantic features of the confluence area lane lines.

[0070] Sub-step 2: Design a multi-level feature fusion module to fuse cross-level multimodal features, and fuse the low-level low-detail features and high-level semantic features extracted in sub-step 1. Through the variable attention module, the high-level semantic features are subjected to a variable attention mechanism and then fused with the low-detail features through convolution and the above-mentioned sampling operations; the low-detail features are subjected to a variable attention mechanism and then convolution and maximum pooling operations with a step size of 2, and then multiplicatively fused with the high-level semantic features, thereby achieving effective fusion of feature maps of different scales extracted by the dual-channel network.

[0071] Specifically, the construction method of the above variable attention module is as follows:

[0072] In the lane segmentation task, due to the unique slender structure of lane lines, long-distance dependency is established in the spatial attention module. A hybrid attention module with hole convolution is introduced, and three consecutive 3×3 hole convolution kernels are used in the spatial attention module instead of a single large convolution kernel, effectively increasing the receptive field of the convolution calculation.

[0073] Step 2: In the real-time digital twin of complex traffic scenes (convergence areas), in addition to the effective detection of lane lines, it is also necessary to accurately detect and estimate the relative positions of targets such as vehicles and pedestrians, and use radar and visual multimodal fusion for target detection.

[0074] Sub-step 1: Considering the need for online calibration of cameras and lidars in roadside multi-sensor equipment, the RANSAC algorithm is used to obtain the 3D-2D correspondence between the control points in the lidar coordinate system and the camera coordinate system. The ICP algorithm is used to solve the initial external parameters and obtain the initial external parameters (R, t), where R is the coordinate system rotation vector between the camera and lidar, and t is the translation vector between the two, which is used for target perception and positioning in step 2.

[0075] Sub-step 2: Perform cylindrical encoding on the radar point cloud, filter out the point cloud in the area that does not intersect with the image, encode the disordered point cloud, use the PointNet backbone network to extract the cylindrical features, and output feature maps of different scales.

[0076] Sub-step 3: Since the road test equipment is deployed at a certain angle to the parallel ground, the presence of occlusion is not conducive to target detection. Therefore, the image data is first transformed into the BEV (Bird's-eye-view) perspective through perspective transformation, and then the image data under the BEV perspective is extracted. The ResNet backbone network is selected as the feature extractor, and a multi-scale strategy is adopted to combine high-level semantic information with low-level features. The output feature map size is

[0077] Sub-step 4: Perform feature fusion on the features extracted in sub-steps 2 and 3, use a variable attention module to concatenate and fuse the two feature maps, and finally output the target detection result.

[0078] Step 3: The lane line semantic segmentation map obtained by semantic segmentation needs to be further instance segmented to extract the information of each lane line. The DBSCAN clustering algorithm is used to distinguish the instances of the lane line semantic segmentation results.

[0079] Step 4: When the LiDAR point cloud data is projected onto the image plane, there are still a large number of lane line pixels that do not have depth information. Therefore, it is necessary to perform depth estimation on the lane line pixels and convert the two-dimensional lane line pixels into three-dimensional data.

[0080] Sub-step 1: First, project the LiDAR point cloud onto the image plane to obtain the depth reference point. The transformation relationship between the camera coordinate system and the pixel coordinate system is as follows (1-1):

[0081]

[0082] Among them, K is the camera internal parameter, z c is the depth value of the pixel (u,v).

[0083] Sub-step 2: After obtaining the accurate depth reference value of the pixel point through LiDAR point cloud projection, use the bilateral filtering method to estimate the depth of the lane line pixel point with missing depth value; for the lane line pixel point x with missing depth value, search for the four reference points x with the closest pixel distance and depth reference value in its neighborhood i , where i = 1, 2, 3, 4, the depth value of the pixel point x is estimated as follows (1-2), and finally the three-dimensional coordinates (x) of the lane line pixel point in the pixel coordinate system are obtained. t ,y t ,z t) to realize the conversion of lane line two-dimensional data to three-dimensional data.

[0084]

[0085] in, r * is the depth estimation value of pixel x, W is the normalized weight coefficient, is the Euclidean distance weight, is the depth value difference weight, r i are four reference pixels x i The reference depth value of , r is the reference depth of pixel x, which is set to the minimum depth of the four reference points.

[0086] Step 5: Based on the vehicle and pedestrian detection results, the depth estimation method of the radar point cloud data in step 4 is used to locate the vehicle and pedestrian targets and output the relative position of the target. The output result is expressed as (x t ,y t ,z t ,θ), where (x t ,y t ,z t ) represents the relative position coordinates of the traffic target relative to the intelligent road test equipment, and θ represents the relative azimuth angle of the traffic target relative to the intelligent road test equipment, which is used to provide data support for the digital twin of the confluence area target in the later stage.

[0087] Step 6: Based on the lane line instance segmentation result of step 3, the three-dimensional pixel information is obtained through step 4, and the three-dimensional lane line data is segmented and fitted. Combined with the relative position of the target, a digital twin model of the real-time confluence area road is constructed; due to the special shape structure of the confluence area intersection lane lines, the lane lines are intersecting and cannot be simply fitted using a geometric model. The lane lines before and after the intersection are fitted in a segmented manner using a straight line model.

[0088] Sub-step 1: According to the instance segmentation result of the lane line, the cross lane lines are screened out from each instance lane line, and a cubic polynomial fitting model is established for the pixel points of each instance lane line using the least squares method.

[0089] Sub-step 2: Since the number of crossing lane lines changes from one to two before and after the intersection, the row coordinates of the crossing lane lines are scanned pixel by pixel using a row scanning method. When the lane line crossed by the scan line changes from one to two, the critical point is the row coordinate u1 of the node, which can be substituted into the fitting curve to obtain the two-dimensional pixel coordinates (u1, v1) of the node.

[0090] Sub-step three, performing straight line model fitting in a segmented manner according to the two-dimensional pixel coordinates of the above nodes to obtain the fitting coordinate results of the lane line.

[0091] Step 7: Use the extracted lane line 3D data to build a more sophisticated traffic scene rendering model. Based on the fitting coordinate results of the lane line in step 6, compile a road network file containing information such as road layout, lane width, and lane connection method. Use the traffic road scene editor to build a road model and add some scene details such as guide lines, curbs, trees, etc. to enrich the visualization of the traffic scene. Finally, combine the target positions of pedestrians and vehicles collected in step 5 through the simulator to simulate and input them into the road for rendering.

[0092] An embodiment of the present invention provides a real-time digital twin method for complex confluence area traffic roads, including: to meet the requirements of accurate detection of traffic targets in the confluence area, firstly, the radar vision sensor deployed on the road section is accurately calibrated; to address the problem of missed detection of intersecting lane lines in the confluence area, a real-time lane line semantic segmentation detection network with a multi-level feature cross-fusion module is proposed; to address the problem of accurate detection of vehicle and pedestrian targets in the confluence area, a cross-modal multi-level feature fusion detection network is proposed; based on the vehicle and pedestrian target detection results, the depth estimation of point cloud data is used to locate the vehicle and pedestrian targets, and the relative position of the target is output; based on the semantic segmentation results of the lane line, the three-dimensional lane line data is segmentedly fitted, and a real-time digital twin model of the confluence area road is constructed in combination with the relative position of the target.

[0093] Compared with the prior art, the method provided by the embodiment of the present invention has the following beneficial effects:

[0094] (1) The model is lightweight and easy to deploy. The embodiment of the present invention adopts a segmentation network that combines deformable convolution with a multi-level feature fusion module and a hybrid attention mechanism module. It has high detection accuracy for lane lines and traffic targets in the confluence area. In traffic target detection, 2D convolution is used after volume column encoding of the radar point cloud, which reduces the amount of model calculation and thus realizes the lightweight of the model. The detection frame rate meets the requirements of real-time digital twins.

[0095] (2) Real-time digital twin: In the current construction of traffic road twin models, there are problems such as the need for manual data screening, low degree of automation, and long model construction cycle, which cannot meet the requirements of real-time digital twin. The embodiment of the present invention combines the intelligent road test equipment to accurately perceive the target and accurately detect the lane lines and intersections in the confluence area, and inputs the collected relative positions of traffic targets into the road model through the simulator for rendering, constructing a simplified twin model of the traffic road in the confluence area, thereby monitoring the traffic status in real time and providing personalized solutions for traffic control.

[0096] The method of the embodiment of the present invention is further described below with reference to the accompanying drawings:

[0097] See also Figure 3 A schematic effect diagram of lane line detection for a traffic road in a complex merging area is shown in FIG. Figure 3 The left picture in the middle is the original picture of the confluence area scene, and the right picture is a schematic rendering of the lane line semantic segmentation result. Specifically, a radar vision fusion module is deployed on a certain expressway in Jiangbei, Nanjing to collect point cloud and image data in the confluence area scene; the road surface segmentation is performed using the method steps described in step 1, such as Figure 3 The middle right figure visualizes the effect of lane line segmentation and intersection fitting in the confluence area. The collected radar visual data is fused and the road participant information perceived by the roadside equipment is imported into the traffic road model in real time through the methods described in steps one to five.

[0098] See also Figure 4 A schematic rendering of a real-time digital twin of a complex confluence area traffic road is shown in FIG. Figure 4 The first sub-image from left to right is the original image of the scene of the traffic target in the confluence area, the second sub-image is the first rendering of the scene of the traffic target in the confluence area, and the third sub-image is the second rendering of the scene of the traffic target in the confluence area. The digital twin method based on lane line detection information and precise positioning of traffic targets provided in the embodiment of the present invention fully meets the real-time construction of the twin model of traffic scenes in complex confluence areas, and plays a role in improving the efficiency of building the twin model of traffic roads.

[0099] The embodiment of the present invention provides a digital twin device for a traffic road in a merging area, see Figure 5 The structure diagram of a real-time digital twin device for complex confluence area traffic roads is shown in FIG. 1 , and the device includes:

[0100] The first initial module 510 is used to obtain a semantic segmentation result of the lane line through a lane line semantic segmentation detection network based on the radar point cloud and image of the lane line.

[0101] The lane line semantic segmentation detection network mentioned above includes a dual-channel network feature extraction module and a multi-level feature fusion module.

[0102] The second initial module 512 is used to perform instance segmentation through a clustering algorithm based on the above semantic segmentation result to obtain an instance segmentation result of the lane line.

[0103] The third initial module 514 is used to obtain the three-dimensional pixel information of the lane line through a depth estimation method based on the above instance segmentation result.

[0104] The fourth initial module 516 is used to perform segmented fitting based on the above three-dimensional pixel information to obtain the fitting coordinate results of the lane line.

[0105] The first real-time module 518 is used to obtain the detection result of the traffic target through a cross-modal multi-level feature fusion detection network based on the radar point cloud and image of the traffic target.

[0106] The second real-time module 520 is used to obtain the location information of the traffic target through a depth estimation method based on the above detection results.

[0107] The location information of the above-mentioned traffic target includes the relative location coordinates and relative azimuth angle of the traffic target relative to the intelligent road test equipment.

[0108] The third real-time module 522 is used to combine the fitting coordinate results of the above lane lines with the position information of the above traffic targets to obtain a digital twin model of the confluence area traffic scene.

[0109] The digital twin device for confluence area traffic roads provided by the embodiment of the present invention has the beneficial effects of: by setting lane line fitting as the initial module, repeated fitting of lane lines can be avoided, thereby improving the efficiency of model construction without losing the real-time performance of lane line fitting. In addition, when the lane line changes, the lane line can be fitted again through the initial module; by setting traffic target positioning and confluence area traffic scene digital twin models as real-time modules, the real-time performance of the confluence area traffic scene digital twin models can be guaranteed, thereby achieving real-time supervision of traffic status.

[0110] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the control device through a computer, and the program can be stored in a computer-readable storage medium. When the program is executed, it may include the processes of the above-mentioned method embodiments, wherein the storage medium may be a memory, a disk, an optical disk, etc.

[0111] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0112] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0113] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A digital twin method for traffic roads in confluence areas, It is characterized in that The method comprises the following steps: Based on the radar point cloud and image of the lane line, a semantic segmentation result of the lane line is obtained through a lane line semantic segmentation detection network, wherein the lane line semantic segmentation detection network includes a dual-channel network feature extraction module and a multi-level feature fusion module; Based on the semantic segmentation result, performing instance segmentation by a clustering algorithm to obtain an instance segmentation result of the lane line; Based on the instance segmentation result, obtaining three-dimensional pixel information of the lane line by a depth estimation method; Performing segmented fitting based on the three-dimensional pixel information to obtain a fitting coordinate result of the lane line; Based on the radar point cloud and image of the traffic target, a detection result of the traffic target is obtained through a cross-modal multi-level feature fusion detection network; Based on the detection result, obtaining the position information of the traffic target by a depth estimation method, wherein the position information of the traffic target includes the relative position coordinates and relative azimuth angle of the traffic target relative to the intelligent road test device; The fitting coordinate results of the lane lines are combined with the location information of the traffic targets to obtain a digital twin model of the confluence area traffic scene.

2. The method according to claim 1, It is characterized in that The dual-channel network feature extraction module includes a first channel and a second channel; The step of obtaining the semantic segmentation result of the lane line through a lane line semantic segmentation detection network based on the radar point cloud and the image of the lane line includes: Extracting low-detail features from the radar point cloud and image of the lane line through the variable attention module of the first channel; Extracting high-level semantic features from the radar point cloud and image of the lane line through the deformable convolution module of the second channel; The low-detail features and the high-level semantic features are fused through the multi-level feature fusion module to obtain a semantic segmentation result.

3. The method according to claim 1, It is characterized in that The step of obtaining the detection result of the traffic target through a cross-modal multi-level feature fusion detection network based on the radar point cloud and image of the traffic target includes: Performing cylinder encoding on the radar point cloud of the traffic target, filtering out the radar point cloud of the traffic target in the area that does not intersect with the image of the traffic target; Extracting body column features from the remaining radar point cloud of the traffic target through the PointNet backbone network to obtain feature maps of the radar point cloud of the traffic target at different scales; Performing a perspective transformation on the image of the traffic target to obtain an image of the traffic target at a specific perspective; Perform multi-level feature extraction from the image of the traffic target at a specific viewing angle through a ResNet backbone network to obtain high-level semantic information and low-level features; A multi-scale strategy is used to combine high-level semantic information with low-level features to obtain a feature map of the image of the traffic target; The feature map of the radar point cloud of the traffic target and the feature map of the image of the traffic target are fused through a variable attention module to obtain the detection result of the traffic target.

4. The method according to claim 1, It is characterized in that The instance segmentation result includes two-dimensional data and three-dimensional data; The step of obtaining the three-dimensional pixel information of the lane line by a depth estimation method based on the instance segmentation result comprises: Projecting the three-dimensional data onto the plane of the two-dimensional data to obtain depth reference values ​​corresponding to the first part of pixel points; Based on the depth reference value, a bilateral filtering method is used to estimate the depth value of the missing pixels in the second part to obtain the three-dimensional pixel information of the lane line; The first part of pixel points and the second part of pixel points are in a complementary relationship, and constitute the complete pixel points of the lane line.

5. The method according to claim 4, It is characterized in that The bilateral filtering method comprises: Searching, within a neighborhood of a pixel point where the depth value is missing, for four reference pixels that are closest to the pixel point where the depth value is missing and have depth reference values; The depth reference values ​​of the four reference pixels are converted into the depth values ​​of the pixels whose depth values ​​are missing by using a depth value estimation formula; The depth value estimation formula is: Among them, r * is the depth value of the pixel whose depth value is missing, W is the normalized weight coefficient, is the Euclidean distance weight, is the depth value difference weight, x is the pixel point where the depth value is missing, and x i are the four reference pixels, ‖xx i ‖ is the Euclidean norm, r i is the reference depth value of the four reference pixels, r is the minimum value of the reference depth values ​​of the four reference pixels, and i=1, 2, 3, 4.

6. The method according to claim 1, It is characterized in that The step of performing segmented fitting based on the three-dimensional pixel information to obtain the fitting coordinate result of the lane line comprises: Establishing a cubic polynomial fitting model by least square method based on the three-dimensional pixel information; Obtaining segmentation points by performing row scanning on the cubic polynomial fitting model; Based on the two-dimensional pixel coordinates of the segmentation points, a straight line model is fitted in a segmented manner to obtain a fitting coordinate result of the lane line.

7. The method according to claim 6, It is characterized in that The step of obtaining segmentation points by row scanning the model includes: Scanning the fitting curve of the cubic polynomial fitting model row by row through the row coordinate axis; When the lane line passed by the row coordinate axis changes from one to two, the row coordinates of the segmentation point are obtained at the critical point; Substituting the row coordinates into the fitting curve to obtain the two-dimensional pixel coordinates of the segmentation point; The row coordinate axis is approximately perpendicular to the direction of the lane line.

8. The method according to claim 1, It is characterized in that The step of combining the fitting coordinate result of the lane line with the position information of the traffic target to obtain a digital twin model of the confluence area traffic scene includes: According to the fitting coordinate results of the lane lines, a road network file is compiled, and a road model is established through a traffic road scene editor; According to the location information of the traffic target, a traffic target model is established by simulating through a simulator; The traffic target model is input into the road model to obtain a digital twin model of the confluence area traffic scene.

9. The method according to claim 1, It is characterized in that The method further comprises: The radar point cloud and the image are obtained by accurately calibrating the parameters of the intelligent road test equipment; The radar point cloud is three-dimensional data, the image is two-dimensional data, and the intelligent road test equipment includes a laser radar and a visual sensor.

10. A digital twin device for traffic roads in confluence areas, It is characterized in that The device comprises: A first initial module is used to obtain a semantic segmentation result of the lane line through a lane line semantic segmentation detection network based on a radar point cloud and an image of the lane line, wherein the lane line semantic segmentation detection network includes a dual-channel network feature extraction module and a multi-level feature fusion module; A second initial module is used to perform instance segmentation based on the semantic segmentation result by a clustering algorithm to obtain an instance segmentation result of the lane line; A third initial module, configured to obtain three-dimensional pixel information of the lane line by a depth estimation method based on the instance segmentation result; A fourth initial module, configured to perform segmented fitting based on the three-dimensional pixel information to obtain a fitting coordinate result of the lane line; A first real-time module is used to obtain a detection result of the traffic target through a cross-modal multi-level feature fusion detection network based on a radar point cloud and an image of the traffic target; A second real-time module is used to obtain the position information of the traffic target by a depth estimation method based on the detection result, wherein the position information of the traffic target includes the relative position coordinates and relative azimuth angle of the traffic target relative to the intelligent road test device; The third real-time module is used to combine the fitting coordinate results of the lane line with the position information of the traffic target to obtain a digital twin model of the confluence area traffic scene.

Citation Information

Patent Citations

  • Full-dose full-sample real-time traffic data-based multi-parameter fusion method and system

    CN110807924A

  • Scene flow digital twinning method and system based on dynamic trajectory flow

    CN114970321A

  • Intelligent network connection automobile multi-mode fusion detection method based on vehicle-road cooperation

    CN116912786A

  • Digital twinborn scene intelligent generation method based on multi-modal visual identification

    CN117456136A

  • Method and system for augmenting lidar data

    US20240017747A1