Method and apparatus for extracting road information based on multimodal data

Abstract

This invention proposes a method and apparatus for road information extraction based on multimodal data. The method integrates data from three modalities: remote sensing images, points of interest (POIs), and street view images, overcoming the limitations of single-scale characterization in road information extraction and obtaining richer, multi-scale road information. It distinguishes roads from background areas through connectivity attention network segmentation branches and explores the connectivity relationships between adjacent pixels, resulting in road segmentation with better connectivity. By optimizing the conversion of road raster images into vectorized road centerlines, a road centerline closer to the ground truth map is obtained. Combining POIs on both sides of the road centerline refines the semantic attributes of road segments in the social activity space. Street view images are used to quickly detect lane lines and determine their number. This invention has the advantages of being multimodal, multi-scale, and information-rich, significantly improving the continuity, accuracy, and completeness of road information extraction.

Description

Technical Field

[0001] This invention relates to the field of road image processing, and more specifically, to a method and apparatus for extracting road information based on multimodal data. Background Technology

[0002] Road networks are a crucial component of transportation infrastructure, impacting people's daily travel and socio-economic life. Roads connect regions and cities globally, while lane markings indicate vehicle routes and travel areas locally. Roads and lane markings are descriptive concepts of the same geographical feature at two different scales. Studying the interconnectedness of the same feature at different spatial scales to extract more complete road information is of great significance for the development of intelligent mobility and digital transportation.

[0003] The rapid development of modern computer vision has provided reliable theoretical and technical support for addressing complex road information extraction problems. Deep learning, with its advantages of high accuracy, automatic feature extraction, and robustness, has wide applications in image recognition, providing strong support for extracting road network information with fine geometric topology, rich semantic information, and high update speed. Deep learning is essentially a feature learning method. Under complex road perception conditions, deep learning methods can effectively and adaptively extract relevant road features, exhibiting strong resistance to interference from occlusion, lighting, weather, etc., thus improving the performance of road information extraction.

[0004] However, road information extraction using both traditional and deep learning methods mostly employs a single data scale for characterizing road information, and is largely a single task mode. It is limited to improving the speed, accuracy, and automation of each extraction algorithm, without combining road extraction and lane detection across multiple scales to achieve a more comprehensive extraction of road information. Nevertheless, road information extraction tasks at different scales are highly correlated and complementary.

[0005] Therefore, overcoming the limitations of single-scale data characterization and obtaining multi-scale, richer road geographic and semantic information is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] The main technical problem that this invention aims to solve is how to overcome the limitations of single data scale characterization and obtain multi-scale, richer road geographic information and semantic information.

[0007] The technical solution adopted by this invention to solve its technical problem is as follows: Road centerlines are extracted based on remote sensing imagery and connectivity networks; the name and importance of roads are obtained based on POI and spatial analysis techniques; and the number of lane lines is obtained based on street view images and ultra-fast structure-aware networks, thereby obtaining multi-scale and richer road geographic and semantic information.

[0008] According to a first aspect of the present invention, a method for extracting road information based on multimodal data is characterized by comprising the following steps:

[0009] S1. Acquire remote sensing images and perform preprocessing, including image bit conversion, label image generation, image label matching, and image cropping and augmentation;

[0010] S2. In the preprocessed remote sensing images, the image label pairs are randomly classified and expanded into training and test sets; the first model parameters of the connectivity attention network are trained using the training set, and the road regions in the test set are inferred using the trained first model parameters to obtain the road region segmentation results.

[0011] S3. Using the road area segmentation results and the optimized raster-to-vector algorithm, obtain the vector representation of the road centerline;

[0012] S4. Using the road centerline vector representation and the collected POI data, restrict the POIs to a preset spatial range near the road centerline, and perform general classification and reclassification of the POIs; establish a topology using the road centerline vector representation, and perform topology checks and improvements.

[0013] S5. Calculate the weights of different types of POIs on road importance using the POI general classification and reclassification results and the optimized entropy weight method. Then, sum the importance values ​​of different types of POIs using the weights and the optimized SAVEE model to obtain the importance values ​​of different road segments in the social space. Capture the road name POIs from the POI general classification and reclassification results onto the road centerline vector, adding a road name attribute to the road centerline. Add the importance values ​​of different road segments as attribute information to the road centerline vector, adding a road importance attribute to the road centerline.

[0014] S6. Using the importance values ​​of different road segments, collect points on the road segments at different levels and densities to collect street view images of the road segments before and after the driving direction.

[0015] S7. Crop the street view image and adjust its aspect ratio to obtain street view images of various sizes and aspect ratios;

[0016] S8. Train the ultrafast structure-aware network using the classic TuSimpel dataset and the CULane dataset respectively, and obtain the corresponding second model parameters respectively;

[0017] S9. Using the second model parameters, lane line detection is performed on each street scene image to obtain the number and spatial distribution of lane lines. By comparing the consistency of the number of lane lines before and after driving, the lane line number attribute is added to the road centerline.

[0018] S10. Based on the vector representation of the road centerline and the road name attribute, road importance attribute, and number of road lane lines added to the road centerline, the road information extraction result is obtained.

[0019] According to another aspect of the present invention, a road information extraction device based on multimodal data is provided, comprising the following modules:

[0020] The remote sensing image acquisition and preprocessing module is used to acquire remote sensing images and perform preprocessing, including image bit conversion, label image generation, image label matching, and image cropping and augmentation.

[0021] The connectivity attention network training module is used to randomly classify and expand image label pairs into training and test sets in preprocessed remote sensing images; the first model parameters of the connectivity attention network are trained using the training set; and the road region segmentation results are obtained by using the trained first model parameters to infer road regions in the test set.

[0022] The road centerline vector conversion module is used to obtain the road centerline vector representation using the road area segmentation results and the optimized raster-to-vector algorithm;

[0023] The POI generalization and reclassification module is used to use the road centerline vector representation, combined with the collected POI data, to restrict POIs to a preset spatial range near the road centerline, and to perform generalization and reclassification of POIs.

[0024] The topology creation and inspection module is used to create a topology using the road centerline vector representation, and to perform topology inspection and improvement.

[0025] The module for adding road names and importance attributes is used to calculate the weights of different types of POIs on road importance using the results of POI generalization and reclassification and the optimized entropy weight method. It then uses these weights and the improved SAVEE model to superimpose the importance values ​​of different types of POIs to obtain the importance values ​​of different road segments in the social space. The module also captures the road name POIs from the POI generalization and reclassification results onto the road centerline vector, adding a road name attribute to the road centerline. Finally, it adds the importance values ​​of different road segments as attribute information to the road centerline vector, adding a road importance attribute to the road centerline.

[0026] The street view image acquisition module is used to collect street view images of the road segment in front of and behind the driving direction by taking into account the importance values ​​of different road segments.

[0027] The street view image processing module is used to crop and adjust the aspect ratio of street view images to obtain street view images of various sizes and aspect ratios.

[0028] The ultrafast structure-aware network training module is used to train the ultrafast structure-aware network using the classic TuSimpel dataset and the CULane dataset respectively, and obtain the corresponding second model parameters respectively.

[0029] The lane line quantity attribute addition module is used to detect lane lines in each street scene image using the second model parameters, test the number and spatial distribution of lane lines, and add the lane line quantity attribute to the road centerline by comparing the consistency of lane line quantity before and after driving.

[0030] The integrated road information extraction module is used to obtain road information extraction results based on the road centerline vector representation and the road name attribute, road importance attribute, and road lane line quantity attribute added to the road centerline.

[0031] The technical solution provided by this invention has the following beneficial effects:

[0032] 1. This invention applies deep learning connectivity networks and ultrafast structure perception networks to the research of road information extraction, enabling batch and automated road information extraction;

[0033] 2. Based on the extraction of road spatial information, this invention adds semantic information about road names and the importance of roads in social space to road vector data by incorporating POI data through spatial analysis and multimodal data;

[0034] 3. This invention can obtain richer information about roads at different scales. It obtains a larger spatial range of road centerline vector representation through remote sensing images and adds more detailed attribute information such as the number of lane lines to the road center through street view images. Attached Figure Description

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0036] Figure 1 This is a flowchart illustrating the technical process of a road information extraction method based on multimodal data according to the present invention.

[0037] Figure 2 This is a flowchart illustrating a spatial analysis example in a road information extraction method based on multimodal data according to the present invention.

[0038] Figure 3 This is a network architecture diagram of the connectivity attention network in a road information extraction method based on multimodal data according to the present invention;

[0039] Figure 4 This is a network architecture diagram of the ultrafast structure perception network in the road information extraction method based on multimodal data of the present invention;

[0040] Figure 5 This is a rendering of the road information extraction method based on multimodal data according to the present invention;

[0041] Figure 6 This is a flowchart illustrating a training example of a connectivity attention network in a road information extraction method based on multimodal data according to the present invention.

[0042] Figure 7 This is a flowchart illustrating a training example of an ultrafast structured perception network in a road information extraction method based on multimodal data, as described in this invention.

[0043] Figure 8 This is a schematic diagram of the structure of a road information extraction device based on multimodal data according to the present invention;

[0044] Figure 9 This is a schematic diagram of the structure of an electronic device according to the present invention. Detailed Implementation

[0045] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0046] Multimodal geospatial big data provides unprecedented means of social perception for understanding the distribution patterns, interactions, and dynamic evolution of geographical phenomena. Rich multimodal data, including remote sensing imagery, street view images, and Point of Interest (POI) data, describe and characterize roads from different perspectives. Leveraging the complementary advantages of multimodal data enhances the perception of roads across multiple dimensions, enables joint research on road information at different scales, increases the understanding of road semantic information, and improves the continuity, accuracy, and completeness of road information extraction.

[0047] Remote sensing imagery can acquire road network information over a large area. The number and location of different types of Points of Interest (POIs) on both sides of the road reflect people's perception of the road, while street view imagery can intuitively reflect the characteristics and condition of the roads they live on. Therefore, this invention organically integrates data from these three modalities, which can overcome the limitations of single-scale data characterization and obtain multi-scale, richer road geographic and semantic information.

[0048] refer to Figure 1 and Figure 2 This invention provides a method for extracting road information based on multimodal data, which mainly includes the following steps:

[0049] S1. Acquire remote sensing images and perform preprocessing, including image bit conversion, label image generation, image label matching, and image cropping and augmentation;

[0050] S2. In the preprocessed remote sensing images, the image label pairs are randomly classified and expanded into training and test sets; the first model parameters of the connectivity attention network are trained using the training set, and the road regions in the test set are inferred using the trained first model parameters to obtain the road region segmentation results.

[0051] S3. Using the road area segmentation results and the optimized raster-to-vector algorithm, obtain the vector representation of the road centerline;

[0052] S4. Using the road centerline vector representation and the collected POI data, restrict the POIs to a preset spatial range near the road centerline, and perform general classification and reclassification of the POIs; establish a topology using the road centerline vector representation, and perform topology checks and improvements.

[0053] S5. Calculate the weights of different types of POIs on road importance using the POI general classification and reclassification results and the optimized entropy weight method. Then, sum the importance values ​​of different types of POIs using the weights and the optimized SAVEE model to obtain the importance values ​​of different road segments in the social space. Capture the road name POIs from the POI general classification and reclassification results onto the road centerline vector, adding a road name attribute to the road centerline. Add the importance values ​​of different road segments as attribute information to the road centerline vector, adding a road importance attribute to the road centerline.

[0054] S6. Using the importance values ​​of different road segments, collect points on the road segments at different levels and densities to collect street view images of the road segments before and after the driving direction.

[0055] S7. Crop the street view image and adjust its aspect ratio to obtain street view images of various sizes and aspect ratios;

[0056] S8. Train the ultrafast structure-aware network using the classic TuSimpel dataset and the CULane dataset respectively, and obtain the corresponding second model parameters respectively;

[0057] S9. Using the second model parameters, lane line detection is performed on each street scene image to obtain the number and spatial distribution of lane lines. By comparing the consistency of the number of lane lines before and after driving, the lane line number attribute is added to the road centerline.

[0058] S10. Based on the vector representation of the road centerline and the road name attribute, road importance attribute, and number of road lane lines added to the road centerline, the road information extraction result is obtained.

[0059] Based on, but not limited to, the above methods, step S1 specifically includes the following steps:

[0060] Coordinate projection is performed on the labeled image, and a 3m buffer is generated based on the linear label to produce a total road width of 6m; geographic coordinate information is added to the generated image label according to the geographic coordinate information of the corresponding ground truth label.

[0061] Based on, but not limited to, the above methods, in step S2, the training of the weights of the Connectivity Attention Network (CoANet) model mainly involves simultaneously learning the road image segmentation and the pairwise dependencies of pixels, specifically including the following steps:

[0062] S21. In terms of the encoder for feature extraction, the last two convolutional blocks of ResNet-101 are replaced with dilated convolutions with dilation rates of r=2 and r=4, respectively, to improve the receptive field of feature points while controlling the resolution of feature maps. In the decoder, a dilated spatial pyramid module (ASPP) is added to capture multi-scale features and adapt to the narrow and complex spatial structure of roads. At the same time, the receptive field of feature points is increased to improve the connectivity of roads.

[0063] S22. The decoder module includes four striped convolutional modules used to upsample feature maps of different scales to an appropriate size, while ensuring that each position of the output feature map can be compared with the input feature map. Figure 4 The input tensor is connected to multiple locations in each direction to better capture linear features in the road feature map. After the input tensor is processed by a 1*1 convolution, it is processed in parallel by four strip convolutions in different directions to extract road features. Each strip convolution has 9 parameters, which are the same as the 3*3 convolution kernel. The extracted results are concatenated and then upsampled. The output of the strip convolution module is obtained after adjusting the number of feature layers through a 1*1 convolution.

[0064] S23. After the road features are processed through four strip convolutions, they enter the segmentation branch and the connectivity branch respectively. The segmentation branch yields the pixel-level loss, while the connectivity branch yields the loss between the ground reality connectivity cube and the predicted connectivity cube. The segmentation branch loss function is L. seg The specific mathematical expression is:

[0065] L seg =L BCE +α(1-L Dice )

[0066] Where α is a constant, L BCE Let L be the binary cross-entropy. Dice The dice coefficients are defined as follows:

[0067]

[0068]

[0069] Where N is the number of elements in the H×W slice, H and W are the height and width of the feature map after four strip convolutions, respectively, and y i Let i be the ground truth value at position i. The predicted value for position i obtained from the branch prediction;

[0070] S24. The connectivity branch primarily explores the road connectivity relationships between adjacent pixels through the connectivity attention module to achieve better road segmentation results. First, the input tensor is passed through a 3x3 convolution, followed by a 3x3 dilated convolution with an dilation rate r = d to increase the receptive field of the feature map and learn the relationships between adjacent pixels in a larger spatial range. After obtaining a feature tensor, one branch feeds the tensor into a squeezed excitation block to obtain the tensor's attention on the channels. The other branch multiplies the feature vector by the channel attention within the (0,1) range of the corresponding channel, thereby using the channel attention to recalibrate the predicted connectivity of adjacent pixels, resulting in a predicted connectivity cube O, where each channel corresponds to the connectivity of adjacent pixels. The connectivity attention network has two modules with distances of 1 and 3, corresponding to the same distance between a given pixel and its adjacent pixels as the dilated convolution interval within the connectivity module; the connectivity branch loss function L... con The mathematical expression is:

[0071] L con =L d1 +βL d3

[0072]

[0073] Among them, L con It is connectivity loss, L d1 With L d3 These are the loss functions corresponding to two connectivity modules at different scales, where β is a constant for the loss coefficient, and L... d1 With L d3 The formulas have the same structure, L d3 With L d1 In contrast, the distance of the 3x3 dilated convolution in the connectivity module is set to 3, and the spacing between a given pixel and its neighboring pixels in the corresponding connectivity cube is 3; in L d1 In the formula, C0 is the number of adjacent pixels of a given pixel, and N is the number of elements in the H×W slice; Let be the truth value of the connectivity or disconnectivity between a given pixel at position i and its neighboring pixels at position c. Predicted connectivity for connected components;

[0074] S25. Based on the segmentation branch loss function obtained in step S23 and the connectivity branch loss function obtained in step S24, the total loss function L for training the connectivity attention network is obtained. CoANet The specific formula is as follows:

[0075] L CoANet =L seg +λL con

[0076] Where λ is a constant, L CoANet That is the total loss, L con It is connectivity loss, L seg It is the loss from splitting the branches;

[0077] When the total loss L CoANet When the network converges, stop training and use the connectivity attention network under the current convergence condition as the segmentation network for the road region.

[0078] Furthermore, the model parameters obtained from training are used to infer the road regions of the test dataset and obtain the road region segmentation results.

[0079] It should be noted that in step S2, the inferred road region segmentation result is a binary image. However, in step S3, to obtain the vector representation of the road centerline, the inferred binary image needs to be converted into an inferred road graph G', that is, the road probability graph is converted into a graph, and then the vector representation of the road centerline is generated based on the edges and nodes of the graph. Therefore, step S3 specifically includes:

[0080] S31. Image Preprocessing: The road segmentation map obtained through the connectivity attention network is mapped to the original remote sensing image. Geographic information from the original remote sensing image is added to the predicted road segmentation map, converting it into a TIFF image with geographic information. Next, the generated road TIFF image is preprocessed to remove road areas smaller than 300 pixels, i.e., to remove noise such as small objects and pinholes.

[0081] S32. Refine the road raster image: Determine the set of road pixels that need to be refined, repeatedly remove road pixels that are not road skeletons until the road raster image only contains road skeleton pixels, then remove unnecessary and excessively short road skeleton branches and merge broken road skeleton lines to complete the basic refinement of the road.

[0082] S33. Optimization of Road Skeleton Extraction: Addressing the issue of selecting road skeleton lines from road pixel sets at image edges during skeletonization; Five nearest edge pixels (5 pixels each) were copied from each of the four image boundaries to fill the image borders, and skeletonization was performed on the expanded image. However, the expanded image is more prone to generating road centerline skeletons, which might otherwise appear in another image. Therefore, after generating road skeleton lines on the expanded image (5 pixels each), a 7-pixel reduction was performed. Then, two nearest pixels (2 pixels each) were copied from the reduced image boundary to expand the image boundary, making the error distribution more random. Through these operations, the image was restored to its original size, simultaneously resolving the problem of road skeleton line extraction at image boundaries.

[0083] S34. Vectorize the road grid skeleton: Extract the nodes and edges of the road skeleton line using the skl object of the sknw library, take the intersections and breaks on the road skeleton line as nodes, connect adjacent nodes to establish edges, and organize the points and edges in the form of image pixel coordinates. Traverse all nodes and edges in the skeleton line and output the image pixel coordinates as the corresponding WKT (Well-known Text format) file.

[0084] S35. Optimize nodes and edges based on the specific geometry of the road: For nodes, if a node is located on an edge with a length of less than 10 pixels, delete it; if a node is less than 20 pixels away from other nodes, connect it to other nodes; if a node is almost on the same straight line as other nodes and the distance between them is less than 200 pixels, connect it to other nodes; For edges, since the geometry of roads in the real world is often straight, while the edges generated by raster image conversion are uneven, for each edge, approximate each edge as a straight line segment using OpenCV's approximation algorithm.

[0085] S36. Converting the screen coordinates of vector roads to geographic coordinates: Merge the WKT format file with the geographic information in the corresponding road prediction TIFF image to complete the conversion of pixel coordinates to geographic coordinates and generate a road prediction coordinate sequence in the WGS84 coordinate system.

[0086] S37. Visualization of the inferred road centerline: Convert the WKT format file of the inferred road in the geographic coordinate system into a shp file to complete the visualization of the inferred road centerline vector.

[0087] Based on, but not limited to, the above methods, in step S4, POIs are generally classified into density-type POIs, location-type POIs, and traffic place name POIs. Then, according to the national standard for urban land use classification, the POIs are further classified to a more detailed degree. Based on the differences in the impact of the geographical entities represented by the POIs on roads and the classification standards of POIs by Gaode Map, the reclassification of POIs is completed. Then, based on the vector representation of the road centerline, the reclassified POIs within 30 meters on both sides of the road are screened.

[0088] The importance of a road in a density-based Point of Interest (POI) is primarily measured by the facility point density index, and the specific calculation formula is as follows:

[0089]

[0090] Among them, DR i NP represents the density of facility points within a certain spatial neighborhood of road i. i L represents the total number of facility points within a certain spatial neighborhood of road i. iN* represents the length of road i in meters, and N* represents a positive integer.

[0091] The road importance of a location-based Point of Interest (POI) is primarily measured by the location facility ratio index, and the specific calculation formula is as follows:

[0092]

[0093] Among them, LPR i LP represents the ratio of locational facilities within a certain spatial neighborhood of road i. i LP represents the number of a certain type of POI facility points within a certain spatial neighborhood of road i, LP represents the total number of such facility points within a certain spatial neighborhood of the entire road network in this area, N represents a natural number, and N* represents a positive integer.

[0094] Based on, but not limited to, the above methods, in step S4, topology checking and topology improvement specifically involve: first, establishing a topology for the extracted road network, adding topology judgment rules that prohibit overlap and pseudo-nodes, and performing topology checking and correction of the road network. This mainly involves deleting overlapping parts of road segments and connecting broken road segments caused by image cropping. Due to the issue of selecting road segments at image edges in road recognition, some roads in adjacent images could be connected, but were not connected due to physical cropping and edge detection. For such cases, all roads were extended by 5m and captured, connecting them to the nearest other road segments; for some road segments with a distance exceeding 5m, some clearly identical roads were manually connected; some isolated short segments in the road network were also present, and small segments shorter than 5m were trimmed. Then, the topology checking and correction within the above topology rules were performed again to obtain the final spatial distribution and topology connectivity of the road network.

[0095] Based on, but not limited to, the above methods, in step S5, the optimized entropy weight method and the optimized SAVEE model, the basic process of the entropy weight method is as follows:

[0096] S51. Data Preprocessing: Based on the evaluation indicators contained in the road importance evaluation model, extract m evaluation indicators for n roads. The overall evaluation factor is an n*m matrix M, where x ij It is the value of the j-th indicator for the i-th road, and the specific formula is:

[0097]

[0098] S52. Data Standardization: Standardize the data for each indicator, ensuring they are all on the same order of magnitude, before performing calculations within the same model, where x... ij This is the value of the j-th indicator for the i-th road, and the standardized value s of each indicator. ijThe calculation method is shown in the following formula:

[0099]

[0100] S53. Data Weighting: The weighting of each standardized data point within this indicator is calculated, p. ij S represents the proportion of index j of road i among all roads. ij This represents the standardized value of the j-th index for the i-th road, where n represents the number of roads. The calculation formula is:

[0101]

[0102] S54. Data Entropy Value: The information entropy of each indicator is calculated. If e j The information entropy of the j-th indicator is calculated using the following formula:

[0103]

[0104] Where n represents the number of roads, k = 1 / ln(n), and satisfies e j ≥0, p ij This represents the proportion of index j of road i among all roads. For any given p... ij If the value is 0, simply assign 0 and do not perform lnp. ij Calculation;

[0105] S55. Data Difference Coefficient: The corresponding index difference coefficient d is calculated based on the calculated information entropy value. j , where e j The information entropy of the j-th indicator is calculated using the following formula:

[0106] d j =1-e j

[0107] S56. Indicator Weights: Calculate the weight w of each indicator using the difference coefficient between individual indicators and the overall indicator. j , where d j Here, is the difference coefficient of the j-th indicator, and m is the number of evaluation indicators. The calculation formula is:

[0108]

[0109] S57. Entropy Method Optimization: Combining the actual situation of road importance values ​​with the principle of the entropy method, the entropy weight method is adaptively optimized, mainly by optimizing the calculation of data weights. The calculation formula is as follows:

[0110]

[0111] Where, p ij s represents the proportion of index j of road i among all roads. ij N represents the standardized result of the j-th index of the i-th road. ij Let N be a binary variable, representing whether the j-th index in the i-th road is a non-zero term. If it is a non-zero term, then N is a non-zero term. ij The value is 1 if the road is not in the road, and 0 otherwise, where n represents the number of roads.

[0112] The weights of different types of POIs on road importance were obtained by the optimized entropy weight method. The importance value of the POI factors needs to be superimposed by the SAVEE model to obtain the comprehensive importance value of different road segments.

[0113] The basic process of the SAVEE model is as follows:

[0114] S58, SAVEE Standardization: This standardizes the factors influencing road importance, transforming data of different factors and orders of magnitude into values ​​within the range [-1, 1]. Since the selected influencing factors are all positive and the independent variables are positively correlated with their importance value, the standardization calculation formula is as follows:

[0115]

[0116] Where V is the standardized factor value, X is the independent variable, A is the boundary value of the independent variable X, i.e. X≤|A|; 0≤V≤1, V∝X indicates that the independent variable and the value are positively correlated, i.e. the larger the value of the independent variable, the larger the value of the corresponding factor.

[0117] S59. Optimization of Standardized Calculation: After optimization using density-based or location-based calculations, many data points are much smaller than 1. The addition of 1 to the e-exponential part greatly reduces the presence of non-zero values ​​in the indicator, and the data for positive factors are all greater than 0. Therefore, the optimized standardized equation calculation formula is as follows:

[0118]

[0119] S510, Factor Overlay: Step S59 yields the standardized values ​​of different factors for each road. The standardized values ​​of all factors involved in the evaluation are then pairwise overlaid to obtain the comprehensive evaluation result of the importance of each road. Factor overlay refers to pairwise overlaying of the importance value of each road's individual factors. While ensuring that each factor participates in the calculation, the comprehensive value of each road's importance is calculated, with all values ​​being positive. Where V... A V is the standardized value of factor A. B This is the standardized value of factor B. The formula for calculating factor superposition is:

[0120] V AB =V A +V B -V A V B .

[0121] Furthermore, the traffic place names (POIs) within a certain spatial range near the predicted road centerline obtained in step S37 are connected, so that the predicted road centerline has a road name attribute; according to the road importance level attribute obtained in step S510, the predicted road centerline has a road importance attribute.

[0122] Based on, but not limited to, the above methods, in step S6, the street view images of different densities with varying importance levels for different roads are collected, arranged in descending order of importance from level one to level five. After collecting sampling points at multiple levels according to different importance levels, they are converted into the WGS84 geographic coordinate system to obtain the WGS84 geographic coordinates of all sampling points. Because Baidu Street View uses the double-encrypted Baidu Mercator projection bd09mc, the official Baidu Maps conversion function wgs2bd09mc is used to convert the WGS84 geographic coordinate system to bd09mc, and then the next step calls the Baidu Maps API to collect multi-level street view images.

[0123] Based on, but not limited to, the ultrafast structure-aware network in step S8 uses the entire image as the receptive field for feature extraction, utilizing global features to select the correct lane position on each predetermined row, rather than segmenting each pixel of the lane based on a local receptive field. A lane is represented as a series of horizontal positions on a predetermined row, i.e., row anchors. To represent the position, the first step is to mesh the image. On each row anchor, the position is divided into many units. Additionally, an extra background mesh is introduced outside the image background to represent the case where there is no lane in that row. Thus, lane detection can be described as selecting certain units on predefined row anchors. Compared to segmentation based on local receptive fields, visual cues and information from a wider range of different positions can be learned and utilized. The training of the ultrafast structure-aware network is mainly constrained and optimized by classification loss and lane structure loss. The ultrafast structure-aware network transforms lane line position detection into the process of selecting specific mesh units on different predefined row anchors, i.e., classifying the units on the row anchors.

[0124] Assume the maximum number of lanes is C, the number of anchors is h, the number of grid cells is w, and X is a global image feature, f ij Let P be a classifier used to select the lane position on the i-th lane and the j-th row anchor. Then the lane prediction P i，j，： It can be written as:

[0125] P i,j,:=f ij (X), sti∈[1,C],j∈[1,h]

[0126] Where P i，j，： It is a (w+1)-dimensional vector, representing the probability of selecting (w+1) grid cells for the i-th lane and the j-th row anchor point; assuming T i,j,: It is the encoding of the correct position, and the classification loss function L cls The mathematical formula is:

[0127]

[0128] Among them, L CE It is cross-entropy loss, which uses an extra dimension to represent the absence of lanes, so the position selection of each row anchor is composed of (w+1)-dimensional classification instead of w-dimensional classification;

[0129] Utilizing prior knowledge of lanes, two lane structure loss functions are proposed to model the positional relationships of lane points, enabling the network to learn lane structure information. The first structure loss function is constructed based on lane continuity features. Lane points in adjacent row anchors are close to each other. The position of a lane is represented by a classification vector. Therefore, continuity is achieved by constraining the distribution of the classification vector in adjacent rows. The mathematical expression of the lane continuity-based loss function is:

[0130] The first structural loss function is constructed based on lane continuity features; lane points in adjacent row anchors are close to each other, and the position of a lane is represented by a classification vector. Therefore, the continuity attribute is achieved by constraining the distribution of the classification vector in adjacent rows.

[0131] The structural loss function L based on lane continuity sim The mathematical expression is:

[0132]

[0133] Where ||.||1 represents the L1 specification, P i,j,: P is a prediction of the probability distribution of the position of the lane i and the lane j anchor line. i,j+1,: It is a prediction of the probability distribution of the position of the i-th lane and the (j+1)-th anchor lane line;

[0134] The second structural loss function is based on lane shape features. Most lanes are straight, even curved lanes at bends appear mostly straight due to perspective effects. Therefore, a second-order difference equation is used to constrain lane shape. For straight roads, the second-order difference equation is zero. Lane shape is represented by position constraints by calculating the lane's position at each anchor. The position is obtained from classification predictions by finding the maximum probability of lane lines at each anchor. The probability of different positions (Prob) can be obtained using a differentiable approximation function, softmax. i,j,: The mathematical formula is:

[0135] Prob i,j,: =softmax(P i,j,1:w )

[0136] Where the lane index is i, the row anchor index is j, and P i,j,1:w It is a w-dimensional vector, Prob i,j,: Represents the probability of each position;

[0137] The predicted expected value is used as an approximation of the location, Loc. i,j It can be written as:

[0138]

[0139] Among them, Prob i,j,k It is the probability of the i-th lane and the j-th anchor at the k-th position. For a discrete random variable k, its expected value can be obtained by multiplying each possible value by its probability and then summing them.

[0140] Second-order difference refers to the result obtained by performing two difference operations on a sequence. Therefore, the second-order difference constraint L for road straightness... shp It can be written as:

[0141]

[0142] Among them, Loc i,j It is the position of the i-th lane in the j-th row anchor, Loc i,j+1 It is the position of the i-th lane in the (j+1)-th row anchor, Loc i,j+2 It is the position of the i-th lane in the (j+2)-th row anchor;

[0143] Overall structural loss function L str The mathematical formula is:

[0144] L str =L sim +λL shp

[0145] Where λ is the loss coefficient, L sim It is a structural loss function based on lane continuity, L shp It is a structural loss function based on road straightness;

[0146] The lane structure loss function focuses the loss on prior knowledge and shape features of the road, utilizing both global and local image features through an auxiliary feature aggregation method. It also employs a multi-scale feature-assisted segmentation task to simulate local features. The loss for auxiliary segmentation is the same as the classification loss, using cross-entropy L. CE To measure; overall loss function L total for:

[0147] L total =L cls +αL str +βL seg

[0148] Among them, L cls It is the classification loss function, L str It is the overall structural loss function, L seg This is the segmentation loss function, where α and β are loss coefficients. The auxiliary segmentation task is used only during the training phase and removed during the testing phase. When the total loss function L... total When the network converges, stop training and use the ultrafast structure perception network under the current convergence condition as the network model for fast lane line detection.

[0149] Based on, but not limited to, the above methods, in step S8, the ultra-fast structure-aware network has a model architecture in which the main branch extracts image features through the ResNet network structure and completes the detection of lane lines. During the training phase, an auxiliary segmentation branch is added. The auxiliary segmentation branch upsamples the features of the second residual block and the third residual block of the main branch by 2 times using bilinear interpolation, and the features of the fourth residual block by 4 times using bilinear interpolation. Then, they are concatenated in the channel dimension, and then a convolution operation is performed to convert the number of output channels into the number of lane line categories plus 1, so as to determine the position of each lane line pixel and non-lane line pixels, thus completing the auxiliary segmentation of lane lines.

[0150] Furthermore, the ultrafast structure-aware network was trained using the TuSimpel dataset and the CULane dataset, respectively. When the total loss function converged, the trained network was used to quickly detect lane lines in the acquired street scene images.

[0151] Furthermore, better lane detection results are achieved by continuously adjusting the size and aspect ratio of the test street view images. The model parameters trained using the CULane dataset demonstrate higher accuracy and robustness in lane detection. Simultaneously, based on the aspect ratio of the CULane dataset images, the size and aspect ratio of the street view images are adjusted to perform lane detection on the acquired street view images.

[0152] Finally, based on the lane distribution and quantity of the street scenes before and after the same geographical location, the lane quantity attribute of the road segment is obtained by comparing the number of lanes in the forward and backward driving directions according to step S9. Based on the road importance level attribute, road name attribute, and road lane quantity attribute, the road centerline is inferred to have road name attribute, road importance attribute, and lane quantity attribute, thus completing the extraction of richer geographical and semantic information about the road at different scales.

[0153] refer to Figure 3 The network architecture diagram of the connectivity attention network includes the following:

[0154] In the encoder for feature extraction, the last two convolutional blocks of ResNet-101 are replaced with dilated convolutions with dilation rates of r=2 and r=4, respectively. In the decoder, a dilated spatial pyramid module is added to capture multi-scale features. The decoder module mainly consists of four strip convolution modules. The input tensor is passed through a 1*1 convolution and then through four strip convolutions in different directions in parallel to extract road features. Each strip convolution has 9 parameters, the same as the 3*3 convolution kernel. The extracted results are concatenated and upsampled. After adjusting the number of feature layers through a 1*1 convolution, the output of the strip convolution module can be obtained.

[0155] After passing through four strip convolutions, the road features are then fed into the segmentation branch and the connectivity branch, respectively. The segmentation branch yields the pixel-level loss, while the connectivity branch yields the loss between the ground reality connectivity cube and the predicted connectivity cube. The connectivity branch mainly explores the road connectivity relationships between adjacent pixels through the connectivity attention module to achieve road segmentation results with better continuity.

[0156] When inferring connectivity between adjacent pixels, the input tensor is first passed through a 3x3 convolution, followed by a 3x3 dilated convolution with an expansion rate r = d to obtain a feature tensor. One branch feeds the tensor into a squeezed excitation block to obtain channel attention. The other branch multiplies the feature vector by the channel attention within the (0,1) range of the corresponding channel. The connectivity attention network has two modules with distances of 1 and 3, corresponding to the same distance between a given pixel and its adjacent pixels as the dilated convolution interval within the connectivity module.

[0157] Obtain the total loss for training the connectivity network. When the total loss converges, stop network training and use the connectivity attention network under the current convergent condition as the segmentation network for road regions. Otherwise, retrain the network.

[0158] refer to Figure 4 The network architecture diagram of the ultrafast structure-aware network includes the following:

[0159] The main branch extracts image features using a ResNet network structure. First, the image is input into 64 7x7 convolutional layers to obtain 64-dimensional image features. After max pooling, these features are input into the first residual block. The second residual block is then input, where the image size is halved and the feature dimension is doubled. The third and fourth residual blocks follow the same pattern, with the image size halved and the feature dimension doubled sequentially. After average pooling of the feature vector from the fourth residual block, it is passed through a Flatten layer, which flattens the multidimensional tensor output from the convolutional layers into a one-dimensional vector, and then passes it to a fully connected layer. After passing through the fully connected layer, the feature vector is reshaped, transforming it into the number of row anchors * (number of cells per row + 1) * maximum number of lanes. This allows for the classification of the grid cells on each row anchor, completing the lane line detection in the main branch.

[0160] The main branch is constrained and optimized by classification loss and lane structure loss. The ultrafast structure perception network transforms the detection of lane line positions into the process of selecting specific grid cells on different predefined anchors, i.e., classifying the cells on the anchors.

[0161] Leveraging prior knowledge of lanes, two lane structure loss functions are proposed to model the positional relationships of lane points. The first structure loss function is constructed based on lane continuity features; the second structure loss function is based on lane shape features. The lane shape is represented by positional constraints by calculating the lane's position at each anchor. The position is obtained from classification predictions by finding the maximum probability of lane lines within the anchors, and the probability of different positions is obtained using a differentiable approximation function, softmax. The expected value of the prediction is used as an approximation of the position; a second-order difference constraint for road straightness is used as the second structure loss function; thus, the overall structure loss function is obtained.

[0162] An auxiliary segmentation branch is added during the training phase to enable the model to learn local features of the image. This auxiliary segmentation branch primarily upsamples the features of the second and third residual blocks from the main branch by a factor of 2 using bilinear interpolation, and the features of the fourth residual block by a factor of 4 using bilinear interpolation. These features are then concatenated along the channel dimension, followed by a convolution operation to convert the output channel count to the number of lane line categories plus 1, thus determining the position of each lane line pixel and non-lane line pixels, completing the auxiliary lane line segmentation. The loss for auxiliary segmentation is the same as the classification loss, calculated using cross-entropy (LCE). Therefore, the total loss function is obtained. When the total loss function converges, network training stops, and the ultra-fast structure-aware network at this converged state is used as the network model for fast lane line detection.

[0163] The ultrafast structure-aware network was trained using the TuSimpel and CULane datasets, respectively. When the total loss function converged, the trained network was used to quickly detect lane lines in the acquired street scene images.

[0164] refer to Figure 5 This is a diagram illustrating the implementation effect of road information extraction. Figure 5 (a), (b), and (c) are the renderings of the results obtained by adding road information to remote sensing images of Shanghai roads in the Spacenet3 dataset, including road centerlines, road name attributes, road importance level attributes, and lane number attributes. Specifically, this includes: using remote sensing images as the data source, obtaining vector road centerlines using a connectivity attention network and an optimized raster-to-vector algorithm; using POIs as the data source, adding road name and road importance level attributes to the road centerlines; and using street view images as the data source, adding lane number attributes to the road centerlines.

[0165] refer to Figure 6 A flowchart illustrating a training example of a connectivity attention network in a road information extraction method based on multimodal data, comprising the following steps:

[0166] S201. Data from Shanghai in SpaceNet3 was used as the experimental dataset. A total of 1198 images of Shanghai were generated, with a resolution of 30 cm / pixel and a size of 1300*1300. First, the bit depth of the original images was converted to 8 bits. The GeoJSON format ground truth labels underwent coordinate transformation and were projected onto the 51-degree zone of the UTM coordinate system where Shanghai is located. Then, based on the linear GeoJSON, a 3m buffer was generated using Geopandas, resulting in a total road width of 6m (10 pixels). The coordinate system and projection information were assigned to the rasterized ground truth labels, generating ground truth image labels with geographic information. After cleaning the remote sensing images, a total of 1028 pairs of remote sensing image and ground truth image labels were obtained. The 1028 pairs of image labels were randomly divided into 800 pairs of training data and 228 pairs of test data. Then, the dataset was expanded by creating a 650×650 cropping size. The expanded training data consisted of 800*4=3200 pairs, and the test data consisted of 228*4=912 pairs.

[0167] S202. Input the training data into the connectivity attention network to begin training. For the feature extraction encoder, replace the last two convolutional blocks of ResNet-101 with dilated convolutions with dilation rates of r=2 and r=4, respectively.

[0168] S203. In the decoder, a dilated spatial pyramid module is added to capture multi-scale features. The decoder module mainly includes four strip convolution modules. After the input tensor is passed through a 1*1 convolution, it is passed in parallel through four strip convolutions in different directions to extract road features. Each strip convolution has 9 parameters, the same as the 3*3 convolution kernel. After concatenating the extracted results, upsampling is performed. After adjusting the number of feature layers through 1*1 convolution, the output of the strip convolution module can be obtained.

[0169] S204. After the road features are processed through four strip convolutions, they enter the segmentation branch and the connectivity branch respectively. The segmentation branch yields the pixel-level loss, while the connectivity branch yields the loss between the ground reality connectivity cube and the predicted connectivity cube.

[0170] S205. The connectivity branch primarily explores the road connectivity relationships between adjacent pixels through the connectivity attention module to achieve better road segmentation results with greater continuity. When inferring the connectivity relationships between adjacent pixels, the input tensor is first passed through a 3x3 convolution, followed by a 3x3 dilated convolution with an dilation rate r = d to obtain a feature tensor. One branch then feeds the tensor into a squeezed excitation block to obtain the tensor's attention along the channel. The other branch multiplies this feature vector by the channel attention within the (0,1) range of the corresponding channel. The connectivity attention network has two modules with distances of 1 and 3, corresponding to the same distance between a given pixel and its adjacent pixels as the dilated convolution interval within the connectivity module.

[0171] S206. Based on the loss of the segmentation branch obtained in step S204 and the loss of the connectivity branch obtained in step S205, the total loss for training the connectivity network is obtained. When the total loss value converges, network training is stopped, and the connectivity attention network under the current convergence condition is used as the segmentation network for the road region. Otherwise, return to step S202 and retrain the network.

[0172] refer to Figure 7 A flowchart illustrating the training example of an ultrafast structure-aware network in a road information extraction method based on multimodal data, comprising the following steps:

[0173] S801. Arrange the importance levels from Level 1 to Level 5, decreasing in importance. Roads of Level 5 importance are sampled at 50-meter intervals, Level 4 at 30-meter intervals, Level 3 at 10-meter intervals, Level 2 at 5-meter intervals, and Level 1 at 1-meter intervals. After collecting sampling points at multiple levels according to different importance levels, convert them to the WGS84 geographic coordinate system to obtain the WGS84 geographic coordinates of all sampled points. Because Baidu Street View uses the double-encrypted Baidu Mercator projection bd09mc, the official Baidu Maps conversion function wgs2bd09mc is used to convert the WGS84 geographic coordinate system to bd09mc. Then, the next step is to call the Baidu Maps API to complete the multi-level street view image collection.

[0174] The S802 ultrafast structure-aware network model architecture uses a ResNet network structure in its main branch to extract image features. First, the image is input into 64 7x7 convolutional layers to obtain 64-dimensional image features. After max pooling, these features are input into the first residual block. The second residual block is then input, where the image size is halved and the feature dimension is doubled. The third and fourth residual blocks follow the same pattern, with the image size halved and the feature dimension doubled sequentially. After average pooling of the feature vector from the fourth residual block, it is passed through a Flatten layer, which flattens the multidimensional tensor output from the convolutional layers into a one-dimensional vector, and then passes it to a fully connected layer. After passing through the fully connected layer, the feature vector is reshaped, transforming it into the number of row anchors * (number of cells per row + 1) * maximum number of lanes. This allows for the classification of the grid on each row anchor, completing the lane line detection in the main branch.

[0175] The main branch is constrained and optimized by classification loss and lane structure loss. The ultrafast structure-aware network transforms the detection of lane line positions into the process of selecting specific grid cells on different predefined anchors, i.e., classifying the cells on the anchors, thus yielding an expression for the classification loss function.

[0176] Leveraging prior knowledge of lanes, two lane structure loss functions are proposed to model the positional relationships of lane points. The first structure loss function is constructed based on lane continuity features; the second structure loss function is based on lane shape features. The lane shape is represented by positional constraints by calculating the lane's position at each anchor. The position is obtained from classification predictions by finding the maximum probability of lane lines within the anchors, and the probability of different positions is obtained using a differentiable approximation function, softmax. The expected value of the prediction is used as an approximation of the position; a second-order difference constraint for road straightness is used as the second structure loss function; thus, the overall structure loss function is obtained.

[0177] S803. During the training phase, an auxiliary segmentation branch is added to enable the model to learn local features of the image. The auxiliary segmentation branch primarily upsamples the features of the second and third residual blocks from the main branch by a factor of 2 using bilinear interpolation, and the features of the fourth residual block by a factor of 4 using bilinear interpolation. These features are then concatenated along the channel dimension, followed by a convolution operation to convert the output channel count into the number of lane line categories plus 1, thus determining the position of each lane line pixel and non-lane line pixels, completing the auxiliary lane line segmentation. The loss for auxiliary segmentation is the same as the classification loss, both measured using cross-entropy (LCE). Therefore, the total loss function is obtained. When the total loss function converges, network training stops, and the ultra-fast structure-aware network at this converged state is used as the network model for fast lane line detection.

[0178] S804. The ultrafast structure-aware network was trained using the TuSimpel dataset and the CULane dataset respectively. When the total loss function converged, the trained network was used to quickly detect lane lines in the collected street scene images.

[0179] S805. Improved lane detection results were achieved by continuously adjusting the size and aspect ratio of the test street view images. Using model parameters trained on the CULane dataset, the detection accuracy and robustness of lane lines were improved. Furthermore, based on the aspect ratio of the CULane dataset images (1640 / 590 = 2.78X), the street view image size was adjusted to 800x288 with an aspect ratio of 2.78X, and lane line detection was performed on the acquired street view images.

[0180] The following describes a road information extraction device based on multimodal data provided by the present invention. The road information extraction device described below can be referred to in correspondence with the road information extraction method described above.

[0181] refer to Figure 8 A road information extraction device based on multimodal data includes the following modules:

[0182] The remote sensing image acquisition and preprocessing module 001 is used to acquire remote sensing images and perform preprocessing, including image bit conversion, label image generation, image label matching, and image cropping and augmentation.

[0183] The connectivity attention network training module 002 is used to randomly classify and expand image label pairs into training and test sets in preprocessed remote sensing images; train the first model parameters of the connectivity attention network using the training set; and use the trained first model parameters to infer road regions in the test set to obtain road region segmentation results.

[0184] The road centerline vector conversion module 003 is used to obtain the road centerline vector representation by utilizing the road area segmentation results and the optimized raster-to-vector algorithm;

[0185] The POI generalization and reclassification module 004 is used to use the road centerline vector representation, combined with the collected POI data, to restrict POIs to a preset spatial range near the road centerline, and to perform generalization and reclassification of POIs.

[0186] The topology creation and inspection module 005 is used to create a topology using the road centerline vector representation, and to perform topology inspection and improvement.

[0187] The road name and importance attribute addition module 006 is used to calculate the weights of different types of POIs on road importance using the POI general classification and reclassification results and the optimized entropy weight method. It then uses these weights and the improved SAVEE model to superimpose the importance values ​​of different types of POIs to obtain the importance values ​​of different road segments in the social space. The module also captures the road name POIs from the POI general classification and reclassification results onto the road centerline vector, adding a road name attribute to the road centerline. Finally, it adds the importance values ​​of different road segments as attribute information to the road centerline vector, adding a road importance attribute to the road centerline.

[0188] The street view image acquisition module 007 is used to collect street view images of the road segment before and after the driving direction by using the importance values ​​of different road segments and taking points on the road segment in a hierarchical and multi-density manner.

[0189] The street view image processing module 008 is used to crop and adjust the aspect ratio of street view images to obtain street view images of various sizes and aspect ratios.

[0190] The ultrafast structure-aware network training module 009 is used to train the ultrafast structure-aware network using the classic TuSimpel dataset and the CULane dataset respectively, and obtain the corresponding second model parameters respectively.

[0191] The lane line quantity attribute addition module 010 is used to detect lane lines in each street scene image using the second model parameters, test the number and spatial distribution of lane lines, and add the lane line quantity attribute to the road centerline by comparing the consistency of lane line quantity before and after driving.

[0192] The integrated road information extraction module 011 is used to obtain road information extraction results based on the road centerline vector representation and the road name attribute, road importance attribute, and road lane line quantity attribute added to the road centerline.

[0193] refer to Figure 9The illustration shows a schematic diagram of the physical structure of an electronic device, which may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logical instructions in the memory 630 to execute the steps of the road information extraction method described above, specifically including: acquiring remote sensing images and preprocessing them, including image bit conversion, label image generation, image label pairing, and image cropping and expansion; in the preprocessed remote sensing images, randomly classifying and expanding the image label pairs into training and test sets; using the training set to train the first model parameters of the connectivity attention network, and using the trained first model parameters to infer road regions in the test set, obtaining road region segmentation results; using the road region segmentation results and the optimized raster-to-vector algorithm to obtain the road centerline vector representation; using the road centerline vector representation, combined with the collected POI data, restricting the POIs to a preset spatial range near the road centerline, and performing general classification and reclassification of the POIs; using the road centerline vector representation to establish a topology, and performing topology checking and topology improvement; using the POI general classification and reclassification results and the optimized entropy weight method to calculate the weights of different types of POIs on road importance, and using the weights and the optimized SAVEE model to superimpose the importance values ​​of different types of POIs. The process involves obtaining importance values ​​for different road segments in the social space; capturing road name POIs from POI generalization and reclassification results onto the road centerline vector, adding a road name attribute to the road centerline; adding the importance values ​​of different road segments as attribute information to the road centerline vector, adding a road importance attribute to the road centerline; using the importance values ​​of different road segments, sampling points at different levels and densities on the road segments to collect street view images before and after the driving direction of the road segments; cropping and adjusting the aspect ratio of the street view images to obtain street view images of various sizes and aspect ratios; training an ultrafast structure perception network using the classic TuSimpel dataset and the CULane dataset, respectively, and obtaining the corresponding second model parameters; using the second model parameters, detecting lane lines in each street view image, testing the number and spatial distribution of lane lines, and adding a lane line quantity attribute to the road centerline by comparing the consistency of the number of lane lines before and after driving; and obtaining the road information extraction result based on the road centerline vector representation and the road name attribute, road importance attribute, and road lane line quantity attribute added to the road centerline.

[0194] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0195] In another aspect, embodiments of the present invention also provide a storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned road information extraction method, specifically including: acquiring remote sensing images and performing preprocessing, including image bit conversion, label image generation, image label pairing, and image cropping and expansion; in the preprocessed remote sensing images, randomly classifying and expanding image label pairs into training and test sets; using the training set to train the first model parameters of a connectivity attention network, and using the trained first model parameters to infer road regions in the test set, obtaining road region segmentation results; using the road region segmentation results and an optimized raster-to-vector algorithm to obtain a road centerline vector representation; using the road centerline vector representation, combined with the collected POI data, restricting POIs to a preset spatial range near the road centerline, and performing general classification and reclassification of POIs; establishing a topology using the road centerline vector representation, and performing topology checking and topology improvement; using the POI general classification and reclassification results and the optimized entropy weight method to calculate the weights of different types of POIs on road importance, and using the weights and the optimized SAVEE model to classify different types of POIs... The importance values ​​of POIs are superimposed to obtain the importance values ​​of different road segments in the social space; the road name POIs from the general classification and reclassification results are captured onto the road centerline vector, adding a road name attribute to the road centerline; the importance values ​​of different road segments are added as attribute information to the road centerline vector, adding a road importance attribute to the road centerline; using the importance values ​​of different road segments, points are collected on the road segments at different levels and densities to acquire street view images before and after the driving direction of the road segment; the street view images are cropped and their aspect ratios adjusted to obtain images of various sizes and formats. Street view images with aspect ratios were used. An ultrafast structure perception network was trained using the classic TuSimpel dataset and the CULane dataset, respectively, and the corresponding second model parameters were obtained. Lane detection was performed on each street view image using the second model parameters. The number and spatial distribution of lane lines were tested. By comparing the consistency of the number of lane lines before and after driving, a lane line quantity attribute was added to the road centerline. Based on the vector representation of the road centerline and the road name attribute, road importance attribute, and lane line quantity attribute added to the road centerline, the road information extraction results were obtained.

[0196] This invention provides and implements a method and apparatus for road information extraction based on multimodal data. The method integrates data from three modalities: remote sensing imagery, points of interest (POIs), and street view images, overcoming the limitations of single-scale road information extraction and obtaining richer, multi-scale road information. This invention distinguishes roads from background areas through connectivity attention network segmentation branches and explores the connection relationships between adjacent pixels through connectivity branches, resulting in road segmentation results with better connectivity. By optimizing the vectorization process of converting road raster images into road centerlines, a road centerline closer to the road ground truth map is obtained. Then, the semantic attributes of the road segment in the social activity space are improved by combining POIs on both sides of the road centerline. Finally, urban street view images are used to quickly detect lane lines and determine the number of lane lines. This invention breaks the limitations of single data source and multi-task road information extraction at the same scale, combining the inference results of the road centerline with the detection results of the number of lane lines and the semantic attributes of the road in the social activity space. The vector data of the road centerline has attribute information of road name, road importance, and the number of lane lines, achieving a more complete extraction of road information. Compared with existing road information extraction methods, this invention has the advantages of being multimodal, multi-scale, and having richer information content, which can significantly improve the continuity, accuracy, and completeness of road information extraction.

[0197] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0198] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the unit claims listing several devices, several of these devices may be embodied by the same hardware item. The use of the terms first, second, and third, etc., does not indicate any order and can be interpreted as identifiers.

[0199] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for extracting road information based on multimodal data, characterized in that, Includes the following steps: S1. Acquire remote sensing images and perform preprocessing, including image bit conversion, label image generation, image label matching, and image cropping and augmentation; S2. In the preprocessed remote sensing images, the image label pairs are randomly classified and expanded into training and test sets; The first model parameters of the connectivity attention network are trained using the training set, and the road region segmentation results are obtained by inferring road regions in the test set using the trained first model parameters. S3. Using the road area segmentation results and the optimized raster-to-vector algorithm, obtain the vector representation of the road centerline; S4. Using the road centerline vector representation and the collected POI data, restrict the POIs to a preset spatial range near the road centerline, and perform general classification and reclassification of the POIs; establish a topology using the road centerline vector representation, and perform topology checks and improvements. S5. Calculate the weights of different types of POIs on road importance using the POI general classification and reclassification results and the optimized entropy weight method. Then, sum the importance values ​​of different types of POIs using the weights and the optimized SAVEE model to obtain the importance values ​​of different road segments in the social space. Capture the road name POIs from the POI general classification and reclassification results onto the road centerline vector, adding a road name attribute to the road centerline. Add the importance values ​​of different road segments as attribute information to the road centerline vector, adding a road importance attribute to the road centerline. S6. Using the importance values ​​of different road segments, collect points on the road segments at different levels and densities to collect street view images of the road segments before and after the driving direction. S7. Crop the street view image and adjust its aspect ratio to obtain street view images of various sizes and aspect ratios; S8. Train the ultrafast structure-aware network using the classic TuSimpel dataset and the CULane dataset respectively, and obtain the corresponding second model parameters respectively; S9. Using the second model parameters, lane line detection is performed on each street scene image to obtain the number and spatial distribution of lane lines. By comparing the consistency of the number of lane lines before and after driving, the lane line number attribute is added to the road centerline. S10. Based on the vector representation of the road centerline and the road name attribute, road importance attribute, and number of road lane lines added to the road centerline, the road information extraction result is obtained.

2. The road information extraction method according to claim 1, characterized in that, Step S1 includes: performing coordinate projection on the label image, generating a 3m buffer based on the linear label, and producing a total road width of 6m; adding geographic coordinate information to the generated image label according to the geographic coordinate information of the corresponding ground truth label.

3. The road information extraction method according to claim 1, characterized in that, In step S2, the connectivity attention network is trained, simultaneously learning road image segmentation and pairwise dependencies of pixels, specifically including: S21. In terms of the encoder for feature extraction, the last two convolutional blocks of ResNet-101 are replaced with dilated convolutions with dilation rates of r=2 and r=4, respectively; in the decoder, a dilated spatial pyramid module is added to capture multi-scale features; at the same time, the receptive field of feature points is increased. S22. The decoder module includes four strip convolution modules, which are used to upsample feature maps of different scales to an appropriate size. At the same time, each position of the output feature map can be associated with multiple positions in the four directions of the input feature map. The input tensor is processed by a 1*1 convolution and then by four strip convolutions in different directions in parallel to extract road features. Each strip convolution has 9 parameters, the same as the 3*3 convolution kernel. After the extraction results are concatenated, they are upsampled. The output of the strip convolution module is obtained after adjusting the number of feature layers through 1*1 convolution. S23. After the road features are processed through four strip convolutions, they enter the segmentation branch and the connectivity branch respectively. The segmentation branch yields the pixel-level loss, while the connectivity branch yields the loss between the ground reality connectivity cube and the predicted connectivity cube. The segmentation branch loss function L... seg The mathematical expression is: L seg =L BCE +α(1-L Dice ) Where α is a constant, L BCE Let L be the binary cross-entropy. Dice The dice coefficients are defined as follows: Where N is the number of elements in the H×W slice, H and W are the height and width of the feature map after four strip convolutions, respectively, and y i Let i be the ground truth value at position i. The predicted value for position i obtained from the branch prediction; S24. The connectivity branch explores the road connectivity relationships between adjacent pixels through the connectivity attention module. First, it uses a binary reality mask of the road to generate ground truth values ​​for the connectivity cube of adjacent pixels. When inferring the road connectivity relationships between adjacent pixels, the input tensor is first passed through a 3x3 convolution, followed by a 3x3 dilated convolution with an dilation rate r = d to increase the receptive field of the feature map and learn the relationships between adjacent pixels in a larger spatial range. After obtaining a feature tensor, one branch feeds the feature tensor into a squeezed excitation block to obtain the attention of the feature tensor on the channel; the other branch multiplies the feature tensor by the channel attention within the (0,1) range of the corresponding channel, using the channel attention to recalibrate the predicted connectivity of adjacent pixels, resulting in the predicted connectivity cube O, where each channel corresponds to the connectivity of adjacent pixels. The connectivity attention network has two modules with distances of 1 and 3, corresponding to the same distance between a given pixel and its adjacent pixels as the dilated convolution interval within the connectivity module. The connectivity branch loss function L... con The mathematical expression is: L con =L d1 +βL d3 Among them, L con It is connectivity loss, L d1 With L d3 These are the loss functions corresponding to two connectivity modules at different scales, where β is a constant for the loss coefficient, and L... d1 With L d3 The formulas have the same structure, L d3 With L d1 In contrast, the distance of the 3x3 dilated convolution in the connectivity module is set to 3, and the spacing between a given pixel and its neighboring pixels in the corresponding connectivity cube is 3; in L d1 In the formula, C0 is the number of neighboring pixels of a given pixel, N is the number of elements in the H×W slice, and H and W are the height and width of the feature map after four strip convolutions, respectively. Let be the truth value of the connectivity or disconnectivity between a given pixel at position i and its neighboring pixels at position c, where i is the position of the pixel and c is the position of the neighboring pixel. Predicted connectivity for connected components; S25. Based on the segmentation branch loss function obtained in step S23 and the connectivity branch loss function obtained in step S24, the total loss function L for training the connectivity attention network is obtained. CoANet The specific formula is as follows: THE CoANet =L seg +λL con Where λ is the loss coefficient constant, L CoANet That is the total loss, L con It is connectivity loss, L seg It is the loss from splitting the branches; When the total loss L CoANet When the network converges, stop training and use the connectivity attention network under the current convergence condition as the segmentation network for the road region.

4. The road information extraction method according to claim 1, characterized in that, In step S2, the predicted road region segmentation result is a binary image. The predicted binary image is converted into a predicted road graph G', that is, the road probability graph is converted into a graph. Then, a vector representation of the road centerline is generated based on the edges and nodes of the graph. Step S3 specifically includes: S31. The road segmentation map obtained through the connectivity attention network is matched with the original remote sensing image. The geographic information of the original remote sensing image is added to the predicted road segmentation map, which is then converted into a TIFF image with geographic information. The generated road TIFF image is then preprocessed to remove road areas smaller than 300 pixels, i.e., noise removal. S32. Determine the set of road pixels that need to be refined, repeatedly remove road pixels that are not road skeletons until only road skeleton pixels remain in the road raster image, then remove unnecessary and excessively short road skeleton branches and merge broken road skeleton lines to complete the basic refinement of the road. S33. Five nearest edge pixels of the image boundary were copied to each of the four boundaries of the image to fill the image border. Skeletonization was performed on the new image with the expanded border. After generating the road skeleton line on the new image with the expanded border of 5 pixels, it was reduced by 7 pixels. Two nearest pixels of the reduced image boundary were copied to expand the image boundary to make the error more randomly distributed. Through these operations, the image was restored to its original size. S34. Extract the nodes and edges of the road skeleton line using the sknw library's skel object, take the intersections and breaks on the road skeleton line as nodes, connect adjacent nodes to establish edges. The points and edges are organized in the form of image pixel coordinates. Traverse all nodes and edges in the skeleton line and output the image pixel coordinates as the corresponding WKT format file. S35. For a node, if the node is located on an edge with a length of less than 10 pixels, delete it; if the distance between a node and other nodes is less than 20 pixels, connect it to other nodes; if the node is almost on the same straight line as other nodes and the distance between them is less than 200 pixels, connect the node to other nodes; for each edge, approximate each edge as a straight line segment using OpenCV's approximation algorithm. S36. Merge the WKT format file with the geographic information in the corresponding road prediction TIFF image, complete the conversion of pixel coordinates to geographic coordinates, and generate a road prediction coordinate sequence in the WGS84 coordinate system. S37. Convert the WKT format file of the inferred road in the geographic coordinate system into a shp file to complete the visualization of the centerline vector of the inferred road.

5. The road information extraction method according to claim 1, characterized in that, In step S4, POIs are broadly classified into density-type POIs, location-type POIs, and traffic place name POIs. Then, based on the national standard for urban land use classification, the POIs are further classified to a more detailed degree. According to the differences in the impact of the geographical entities represented by the POIs on roads and the classification standards of POIs by Gaode Map, the POIs are reclassified. Finally, based on the vector representation of the road centerline, the reclassified POIs within 30 meters on both sides of the road are screened. The importance of a road in a density-based Point of Interest (POI) is primarily measured by the facility point density index, and the specific calculation formula is as follows: Among them, DR i NP represents the density of facility points within a certain spatial neighborhood of road i. i L represents the total number of facility points within a certain spatial neighborhood of road i. i N* represents the length of road i in meters, and N* represents a positive integer. The road importance of a location-based Point of Interest (POI) is primarily measured by the location facility ratio index, and the specific calculation formula is as follows: Among them, LPR i LP represents the ratio of locational facilities within a certain spatial neighborhood of road i. i LP represents the number of a certain type of POI facility points within a certain spatial neighborhood of road i, LP represents the total number of such facility points within a certain spatial neighborhood of the entire road network in this area, N represents a natural number, and N* represents a positive integer.

6. The road information extraction method according to claim 1, characterized in that, In step S4, the topology check and topology improvement include: A topology was established based on the extracted road centerline vector representations. Topology judgment rules were added to prevent overlap and pseudo-nodes. Topology checks and corrections were performed on the road network, mainly removing overlapping parts of road segments and connecting broken road segments caused by image cropping. All roads were extended by 5m and captured, connecting them to the nearest other road segments. For some road segments with a distance of more than 5m, some obviously identical roads were manually connected. There were also some isolated short segments in the road network. Small segments with a length of less than 5m were trimmed. The topology checks and corrections within the above topology rules were performed again to obtain the final spatial distribution and topology connection of the road network.

7. The road information extraction method according to claim 1, characterized in that, In the optimized entropy weight method and the optimized SAVEE model described in step S5, the basic process of the entropy weight method is as follows: S51. Based on the evaluation indicators contained in the road importance evaluation model, extract m evaluation indicators for n roads. The overall evaluation factor is an n*m matrix M, where x ij It is the value of the j-th indicator for the i-th road; the specific formula is: S52. Standardize the data for each indicator, ensuring they are all on the same order of magnitude, and then perform calculations within the same model, where x... ij This is the value of the j-th indicator for the i-th road, and the standardized value s of each indicator. ij The calculation method is shown in the following formula: S53. Calculate the proportion of each standardized data within this indicator, p. ij S represents the proportion of index j of road i among all roads. ij This represents the standardized value of the j-th index for the i-th road, where n represents the number of roads. The calculation formula is: S54. Calculate the information entropy of each indicator. If e j The information entropy of the j-th indicator is calculated using the following formula: Where n represents the number of roads, k = 1 / ln(n), and satisfies e j ≥0, p ij This represents the proportion of index j of road i among all roads. For any given p... ij If the value is 0, simply assign 0 and do not perform lnp. ij Calculation; S55. Calculate the corresponding index difference coefficient d based on the calculated information entropy value. j , where e j The information entropy of the j-th indicator is calculated using the following formula: d j =1-e j S56. Calculate the weight w of each indicator using the difference coefficient between individual indicators and the overall indicator. j , where d j Here, is the difference coefficient of the j-th indicator, and m is the number of evaluation indicators. The calculation formula is: S57. Combining the actual situation of road importance values ​​and the principle of the entropy method, adaptive optimization of the entropy weight method is carried out, including: optimizing the calculation of data weights, the calculation formula is as follows: Where, p ij s represents the proportion of index j of road i among all roads. ij N represents the standardized result of the j-th index of the i-th road. ij Let N be a binary variable, representing whether the j-th index in the i-th road is a non-zero term. If it is a non-zero term, then N is a non-zero term. ij The value is 1 if the road is not in the road, and 0 otherwise, where n represents the number of roads. The weights of different types of POIs on road importance were obtained by optimizing the entropy weight method. The importance value of POI factors was superimposed by the SAVEE model to obtain the comprehensive importance value of different road segments. The basic process of the SAVEE model is as follows: S58. Standardize the road importance influencing factors, transforming the numerical values ​​of different factors at different orders of magnitude into values ​​within the range [-1, 1]. The standardization calculation formula is as follows: Where V is the factor value after standardization, X is the independent variable, A is the boundary value of the independent variable X, i.e. X≤|A|, 0≤V≤1, V∝X indicates that the independent variable is positively correlated with the value, that is, the larger the value of the independent variable, the larger the value of the corresponding factor. S59. The optimized standardized equation calculation formula is as follows: S510. Step S59 yields the standardized values ​​of different factors for each road. The standardized values ​​of all factors involved in the evaluation are then summed pairwise to obtain the comprehensive evaluation result of the importance of each road, where V... A V is the standardized value of factor A. B This is the standardized value of factor B. The formula for calculating factor superposition is: V AB =V A +V B -V A V B 。 8. The road information extraction method according to claim 1, characterized in that, The ultrafast structure perception network in step S8 uses the entire image as the receptive field for feature extraction and utilizes global features to select the correct position of the lane in each predetermined row. Lanes are represented as a series of horizontal positions on a predetermined row, i.e., row anchors. To represent the positions, the first step is to mesh the image. On each row anchor, the position is divided into many units. In addition, an extra background mesh is introduced outside the image background to represent the case where there is no lane in that row. The training of the ultrafast structure-aware network is constrained and optimized by a classification loss function and a lane structure loss function. The ultrafast structure-aware network transforms the detection of lane line positions into the process of selecting specific mesh units on different predefined row anchors, i.e., classifying the units on the row anchors. Assume the maximum number of lanes is C, the number of anchors is h, the number of grid cells is w, and X is a global image feature, f ij Let P be a classifier used to select the lane position on the i-th lane and the j-th row anchor. Then the lane prediction P i，j，： It can be written as: P i,j,: =f ij (X),s.t.i∈[1,C],j∈[1,h] Among them, P i，j，： It is a (w+1)-dimensional vector, representing the probability of selecting (w+1) grid cells for the i-th lane and the j-th row anchor point; assuming T i,j,: It is the encoding of the correct position, and the classification loss function L cls The mathematical formula is: Among them, L CE It is cross-entropy loss, which uses an extra dimension to represent the absence of lanes, so the position selection of each row anchor is composed of (w+1)-dimensional classification instead of w-dimensional classification; By utilizing prior knowledge of the lanes, two lane structure loss functions are used to model the positional relationship of lane points; The first structural loss function is constructed based on lane continuity features; lane points in adjacent row anchors are close to each other, and the lane position is represented by a classification vector. Continuity is achieved by constraining the distribution of the classification vector in adjacent rows. The structural loss function L is based on lane continuity. sim The mathematical expression is: Where ||.||1 represents the L1 specification, P i,j,: P is a prediction of the probability distribution of the position of the lane i and the lane j anchor line. i,j+1,: It is a prediction of the probability distribution of the position of the i-th lane and the (j+1)-th anchor lane line; The second structural loss function is based on lane shape features, using a second-order difference equation to constrain the lane shape. For straight roads, the second-order difference equation is zero. The lane shape is represented by calculating the lane's position at each anchor point and using position constraints. The position is obtained from the classification prediction by finding the lane line with the maximum probability among the anchor points, and the probability of different positions (Prob) is obtained using the differentiable approximation function softmax. i,j,: The mathematical formula is: Prob i,j,: =softmax(P i,j,1:w ) Where the lane index is i, the row anchor index is j, and P i,j,1:w It is a w-dimensional vector, Prob i,j,: Represents the probability of each position; The predicted expected value is used as an approximation of the location, Loc. i,j It can be written as: Among them, Prob i,j,k It is the probability of the i-th lane and the j-th anchor at the k-th position. For a discrete random variable k, its expected value can be obtained by multiplying each possible value by its probability and then summing them. Second-order difference constraint L for road straightness shp It can be written as: Among them, Loc i,j It is the position of the i-th lane in the j-th row anchor, Loc i,j+1 It is the position of the i-th lane in the (j+1)-th row anchor, Loc i,j+2 It is the position of the i-th lane in the (j+2)-th row anchor; Overall structural loss function L str The mathematical formula is: THE str =L sim +λL shp Where λ is the loss coefficient, L sim It is a structural loss function based on lane continuity, L shp It is a structural loss function based on road straightness; The lane structure loss function focuses the loss on prior knowledge and shape features of the road, utilizing both global and local image features through an auxiliary feature aggregation method. It also employs a multi-scale feature-assisted segmentation task to simulate local features. The loss for auxiliary segmentation is the same as the classification loss, using cross-entropy L. CE To measure; overall loss function L total for: L total =L cls +αL str +βL seg Among them, L cls It is the classification loss function, L str It is the overall structural loss function, L seg It is the segmentation loss function, where α and β are loss coefficients. The auxiliary segmentation task is only used during the training phase and is removed during the testing phase. When the total loss function L total When the network converges, stop training and use the ultrafast structure perception network under the current convergence condition as the network model for fast lane line detection.

9. The road information extraction method according to claim 1, characterized in that, The ultrafast structure-aware network described in step S8 has a model architecture where the main branch extracts image features using a ResNet network structure, and the main branch detects lane lines. During the training phase, an auxiliary segmentation branch is added. This auxiliary segmentation branch upsamples the features of the second and third residual blocks from the main branch by a factor of 2 using bilinear interpolation, and the features of the fourth residual block by a factor of 4 using bilinear interpolation. These features are then concatenated along the channel dimension, and a convolution operation is performed to convert the number of output channels into the number of lane line categories plus 1, thereby determining the position of each lane line pixel and non-lane line pixels, thus completing the auxiliary segmentation of lane lines.

10. A road information extraction device based on multimodal data, characterized in that, Includes the following modules: The remote sensing image acquisition and preprocessing module is used to acquire remote sensing images and perform preprocessing, including image bit conversion, label image generation, image label matching, and image cropping and augmentation. The connectivity attention network training module is used to randomly classify and expand image label pairs into training and test sets in preprocessed remote sensing images. The first model parameters of the connectivity attention network are trained using the training set, and the road region segmentation results are obtained by inferring road regions in the test set using the trained first model parameters. The road centerline vector conversion module is used to obtain the road centerline vector representation using the road area segmentation results and the optimized raster-to-vector algorithm; The POI generalization and reclassification module is used to use the road centerline vector representation, combined with the collected POI data, to restrict POIs to a preset spatial range near the road centerline, and to perform generalization and reclassification of POIs. The topology creation and inspection module is used to create a topology using the road centerline vector representation, and to perform topology inspection and improvement. The module for adding road names and importance attributes is used to calculate the weights of different types of POIs on road importance using the results of POI generalization and reclassification and the optimized entropy weight method. It then uses these weights and the improved SAVEE model to superimpose the importance values ​​of different types of POIs to obtain the importance values ​​of different road segments in the social space. The module also captures the road name POIs from the POI generalization and reclassification results onto the road centerline vector, adding a road name attribute to the road centerline. Finally, it adds the importance values ​​of different road segments as attribute information to the road centerline vector, adding a road importance attribute to the road centerline. The street view image acquisition module is used to collect street view images of the road segment in front of and behind the driving direction by taking into account the importance values ​​of different road segments. The street view image processing module is used to crop and adjust the aspect ratio of street view images to obtain street view images of various sizes and aspect ratios. The ultrafast structure-aware network training module is used to train the ultrafast structure-aware network using the classic TuSimpel dataset and the CULane dataset respectively, and obtain the corresponding second model parameters respectively. The lane line quantity attribute addition module is used to detect lane lines in each street scene image using the second model parameters, test the number and spatial distribution of lane lines, and add the lane line quantity attribute to the road centerline by comparing the consistency of lane line quantity before and after driving. The integrated road information extraction module is used to obtain road information extraction results based on the road centerline vector representation and the road name attribute, road importance attribute, and road lane line quantity attribute added to the road centerline.

Images