Multimodal fusion-based road topology inference method and system

Abstract

The application provides a multi-modal fusion road topology reasoning method and system, converts the extracted lane mask into a semantic seed point in a historical point cloud, and sets a pseudo label based on the semantic seed point. A lane detection model is obtained through model training based on the historical image and the historical point cloud with the pseudo label. The lane detection model is used to obtain lane line information from real-time images and real-time point clouds, the lane line information is mapped to an aerial view feature map, the topology relationship between the lane lines is calculated based on the lane line information in the aerial view feature map, a topology reasoning matrix is obtained, and a corresponding road topology reasoning result is obtained based on a graph neural network and according to the topology reasoning matrix. According to the scheme, the dependence on manual labeling is reduced, the data preparation cost is reduced, the lane line representation method based on unified mapping in the aerial view space and the effective fusion of lane topology reasoning are introduced, and a high-precision lane topology reasoning result with good topology consistency can be obtained.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving perception technology, and more specifically, to a multimodal fusion road topology reasoning method and system. Background Technology

[0002] In recent years, with the development of intelligent connected vehicles and the rapid advancement of autonomous driving technology, the requirements for road semantic understanding and structured mapping in complex traffic environments have been increasing. As an important component of the perception layer of autonomous driving systems, accurate understanding of road topology not only directly affects the performance of path planning, decision-making, and control, but also relates to the safety and interpretability of the overall system.

[0003] However, traditional road topology mapping methods largely rely on prior structures based on manually generated high-definition maps (HD maps) and rule-based templates, combined with offline matching and mapping using data from vehicle cameras or radar. While these methods offer high accuracy, they suffer from two main drawbacks: first, the reliance on manually generated HD maps results in high system construction and maintenance costs and makes it difficult to adapt to dynamic road environments; second, the over-reliance on rule-based prior models leads to a lack of flexibility and scenario generalization capabilities, making topology misjudgments particularly prone to occur in complex lane structures (such as merging, branching, and U-turns).

[0004] With the rise of crowdsourced map data and the development of data-driven methods, a large amount of vehicle-collected images and point cloud data has been used to train lane detection models, making it possible to automatically infer accurate lane line positions from perception results. However, current data-driven methods generally rely on large amounts of labeled data, which faces challenges such as high labeling costs and insufficient supervision performance in real-world applications. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal fusion road topology reasoning method and system to obtain lane topology reasoning results with high accuracy and good topology consistency, and to reduce data preparation costs.

[0006] In a first aspect, the present invention provides a multimodal fusion road topology reasoning method, the method comprising: Historical images and historical point clouds are collected from the crowdsourced dataset. Lane masks in the historical images are extracted using a multimodal model, and the lane masks are converted into semantic seed points in the historical point cloud. Pseudo-labels are set based on the semantic seed points. A lane detection model is obtained by training a model based on historical images with pseudo-labels and historical point clouds. The lane detection model is used to obtain lane line information from real-time images and real-time point clouds, and the lane line information is mapped onto the bird's-eye view feature map; Based on the lane line information in the bird's-eye view feature map, the topological relationship between lane lines is calculated to obtain a topological inference matrix. Based on the graph neural network and the topological inference matrix, the corresponding road topological inference result is obtained.

[0007] In an optional implementation, the step of extracting the lane mask from the historical image using a multimodal model includes: The historical image is segmented using a multimodal model to generate a background semantic mask. And obtain the description text of each of the background semantic masks; Calculate the similarity between the descriptive text and the predefined category text, and filter the background semantic mask based on the obtained similarity value to determine the lane mask.

[0008] In an optional implementation, the step of converting the lane mask into semantic seed points in the historical point cloud includes: Obtain the maximum and minimum values ​​of the lane mask in the pixel coordinate system; Perform boundary shrinkage processing on the lane mask based on the maximum and minimum values; The lane mask after boundary shrinkage is converted from the historical image to the historical point cloud, and the region corresponding to the lane mask is determined as a semantic seed point.

[0009] In an optional implementation, the step of setting pseudo-labels based on the semantic seed points includes: Clustering operations are performed on the semantic seed points in the historical point cloud to determine multiple clusters; Fit bounding boxes to each of the clusters, and obtain pseudo-labels based on the bounding boxes.

[0010] In an optional implementation, the step of training a lane detection model based on historical images and historical point clouds with pseudo-labels includes: For each pseudo-label in the historical image, calculate the distribution constraint score and meta-shape constraint score of the pseudo-label respectively; After normalizing the distribution constraint score and the meta-shape constraint score, the distribution shape score is obtained by accumulating them according to the set weight factors. The pseudo-labels are filtered based on the distribution shape score, and the lane detection model is trained based on the historical images and historical point clouds corresponding to the filtered pseudo-labels.

[0011] In an optional implementation, the real-time image includes images from multiple perspectives; The step of mapping the lane line information to the bird's-eye view feature map includes: Extract image features from each viewpoint in the real-time image, and convert each pixel in each viewpoint image into a three-dimensional point in the ground coordinate system; The image features of each viewpoint are projected onto the bird's-eye view space according to the three-dimensional coordinates of the three-dimensional points in the ground coordinate system to obtain the initial bird's-eye view feature map. Obtain the two-dimensional coordinates and lane category vectors of each lane line in the bird's-eye view space from the lane line information, and write the two-dimensional coordinates and lane category vectors of each lane line into the initial bird's-eye view feature map to obtain the final bird's-eye view feature map.

[0012] In an optional implementation, the step of calculating the topological relationships between lane lines based on the lane line information in the bird's-eye view feature map to obtain the topological inference matrix includes: Lane queries are obtained by using a lane deformable decoder and based on lane line information in the final bird's-eye view feature map, and multiple directed lane lines are generated based on the lane queries. The lane geometry topology matrix is ​​calculated based on the multiple directed lane lines, and the lane similarity topology matrix is ​​calculated based on the lane query. By combining the lane geometry topology matrix and the lane similarity topology matrix, a topology inference matrix is ​​obtained.

[0013] In an optional implementation, the step of calculating the lane geometric topology matrix based on the multiple directed lane lines and calculating the lane similarity topology matrix based on the lane query includes: Calculate the lane geometric distance between any two of the multiple directed lane lines to obtain the lane geometric distance matrix; Based on the lane geometry distance matrix and the learned mapping function, the lane geometry topology matrix is ​​obtained; The lane query is encoded using two multilayer perceptrons to obtain encoding results. The inner product of the two encoding results is then calculated to obtain the lane similarity. The lane similarity is mapped onto the lane topology to obtain the lane similarity topology matrix.

[0014] In an optional implementation, the step of obtaining the corresponding road topology inference result based on the graph neural network and the topology inference matrix includes: The topology inference matrix and the obtained lane query are input into the graph neural network to aggregate the information of adjacent lane lines corresponding to the lane query, so as to obtain the enhanced lane query. The enhanced lane query is processed using the lane head function to obtain the corresponding road topology inference results.

[0015] Secondly, the present invention provides a multimodal fusion road topology reasoning system, the system comprising: The setting module is used to collect historical images and historical point clouds from the crowdsourced dataset, extract lane masks from the historical images using a multimodal model, convert the lane masks into semantic seed points in the historical point cloud, and set pseudo-labels based on the semantic seed points. The training module is used to train the lane detection model based on historical images and historical point clouds with pseudo-labels. The mapping module is used to obtain lane line information from real-time images and real-time point clouds using the lane detection model, and to map the lane line information onto the bird's-eye view feature map. The inference module is used to calculate the topological relationship between lane lines based on the lane line information in the bird's-eye view feature map, obtain the topological inference matrix, and obtain the corresponding road topological inference result based on the graph neural network and the topological inference matrix.

[0016] This invention provides a multimodal fusion road topology inference method and system. It collects historical images and point clouds from a crowdsourced dataset, extracts lane masks from the historical images using a multimodal model, converts the lane masks into semantic seed points in the historical point cloud, and sets pseudo-labels based on these seed points. A lane detection model is then trained using the pseudo-labeled historical images and point clouds. The lane detection model obtains lane line information from real-time images and point clouds, maps this information to a bird's-eye view feature map, calculates the topological relationships between lane lines based on the lane line information in the bird's-eye view feature map, obtains a topology inference matrix, and finally, uses a graph neural network and the topology inference matrix to obtain the corresponding road topology inference result.

[0017] In this scheme, the semantic mask extraction and pseudo-label generation mechanism guided by the multimodal model reduces the reliance on manual annotation, significantly reducing data preparation costs while ensuring detection quality. Furthermore, the introduction of a lane line representation method based on bird's-eye view spatial unified mapping and the effective fusion of lane topology reasoning can effectively capture the spatial structure between lane lines, resulting in high-precision lane topology reasoning results with good topological consistency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1A flowchart of a multimodal fusion road topology reasoning method provided in an embodiment of the present invention; Figure 2 for Figure 1 A flowchart of the sub-steps included in S11; Figure 3 for Figure 1 A flowchart of the sub-steps included in S12; Figure 4 for Figure 1 A flowchart of the sub-steps contained in S13; Figure 5 for Figure 1 A flowchart of the sub-steps included in S14; Figure 6 for Figure 1 A flowchart of the sub-steps included in S15; Figure 7 for Figure 1 A flowchart of the sub-steps included in S16; Figure 8 for Figure 1 A flowchart of the sub-steps included in S17; Figure 9 A functional block diagram of a road topology reasoning system for multimodal fusion is provided for embodiments of the present invention; Figure 10 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0021] Please see Figure 1 This is a flowchart illustrating a multimodal fusion road topology reasoning method provided in an embodiment of the present invention. This method can be executed by a multimodal fusion road topology reasoning system, which can be implemented in software and / or hardware and can be configured in an electronic device, such as a computer, server, or, for example, a server in a backend control platform. The detailed steps of this multimodal fusion road topology reasoning method are described below.

[0022] S11, Collect historical images and historical point clouds from the crowdsourced dataset, and use a multimodal model to extract lane masks from the historical images.

[0023] S12, convert the lane mask into semantic seed points in the historical point cloud.

[0024] S13, set pseudo-labels based on the semantic seed points.

[0025] S14, a lane detection model is obtained by training the model based on historical images with pseudo-labels and historical point clouds.

[0026] S15, use the lane detection model to obtain lane line information from real-time images and real-time point clouds, and map the lane line information onto the bird's-eye view feature map.

[0027] S16, Calculate the topological relationship between lane lines based on the lane line information in the bird's-eye view feature map to obtain the topological inference matrix.

[0028] S17. Based on the graph neural network and the topology inference matrix, the corresponding road topology inference result is obtained.

[0029] The multimodal fusion road topology reasoning method provided in this embodiment reduces reliance on manual annotation through semantic mask extraction and pseudo-label generation mechanisms guided by a multimodal model, significantly lowering data preparation costs while ensuring detection quality. Furthermore, the introduction of a lane line representation method based on a unified bird's-eye view spatial mapping and the effective fusion of lane topology reasoning effectively captures the spatial structure between lane lines, yielding high-precision lane topology reasoning results with good topological consistency.

[0030] The specific implementation methods of each of the above steps are explained in detail below.

[0031] In this embodiment, the historical images and historical point clouds in the crowdsourced dataset can be uploaded by a large number of users and contain rich road-related information. Each historical image is an image captured by an in-vehicle surround-view camera, and the historical point cloud is a point cloud map detected by an in-vehicle radar device.

[0032] Please see Figure 2 In this embodiment, the step of extracting lane masks from historical images using a multimodal model can be implemented in the following way: S111, the historical image is segmented using a multimodal model to generate a background semantic mask, and the descriptive text of each background semantic mask is obtained.

[0033] S112, calculate the similarity between the description text and the predefined category text, and filter the background semantic mask according to the obtained similarity value to determine the lane mask.

[0034] In this implementation, one of the historical images is used. and its historical point cloud For example, among which and This represents the height and width of the image. Multimodal models can include image segmentation models, such as the FastSAM model, which is used to segment historical images. Perform segmentation to generate a category-independent background semantic mask. ,in This represents the spatial dimension of each background semantic mask. Indicates the number of background semantic masks.

[0035] Next, historical images and background semantic mask Input an image segmentation model, such as the SemanticSAM model, and obtain descriptive text for each background semantic mask. , This represents the semantic description of each background semantic mask. The semantic description can be understood as a natural language phrase generated using the SemanticSAM model for the background semantic masks in historical images, such as lanes, pedestrians, etc.

[0036] Then calculate the description text. and predefined category text The similarity between them can be calculated using cosine similarity. This involves predefined categories of text. , These are category descriptors, mainly including the categories that need to be identified, such as various types of lanes. Next, based on the obtained similarity values, the background semantic mask, which is not of interest, is filtered out to obtain the lane mask. Specifically, through calculation The similarity value between the lanes is compared with a preset threshold. The background semantic mask corresponding to the similarity value greater than the preset threshold is determined. The determined background semantic mask is the lane mask.

[0037] Based on the determined lane mask, the lane mask is converted into semantic seed points in the historical point cloud. For details, please refer to [link to relevant documentation]. Figure 3 This step can be achieved in the following way: S121, obtain the maximum and minimum values ​​of the lane mask in the pixel coordinate system.

[0038] S122, perform boundary shrinkage processing on the lane mask based on the maximum and minimum values.

[0039] S123, the lane mask after boundary shrinkage processing is converted from the historical image to the historical point cloud, and the region corresponding to the lane mask is determined as a semantic seed point.

[0040] In this embodiment, for each lane mask, its pixel coordinate system is calculated. Find the maximum and minimum values ​​in the range, and denote them as follows: Then, based on the maximum and minimum values, the boundaries of the lane mask are limited by shrinking its boundary range according to the following formula: ] ] Where γ is the shrinkage factor. Furthermore, the lane mask after boundary shrinkage processing is denoted as... .

[0041] In addition, the shrunken lane mask is generated using the camera's intrinsic and extrinsic parameter matrices. Historical images from 2D image format Historical point clouds transferred to 3D radar point cloud format In the middle, make radar point cloud map Accurate cross-modal semantic cues. Lane masks with boundary shrinking and transformation. The determined region is defined as a semantic seed point to obtain a historical point cloud containing the semantic seed point.

[0042] The intrinsic parameter matrix contains information such as the camera's focal length and optical center, while the extrinsic parameter matrix describes the relationship between the camera and the world coordinate system.

[0043] After obtaining the historical point cloud containing semantic seed points, pseudo-labels are set based on these semantic seed points. For details, please refer to [link to relevant documentation]. Figure 4 This step can be achieved in the following ways: S131, perform clustering operation on the semantic seed points in the historical point cloud to determine multiple clusters.

[0044] S132, Fit bounding boxes for each of the clusters, and obtain pseudo-labels based on the bounding boxes.

[0045] In this embodiment, the historical point cloud of the lidar is represented as , defined by The semantic seed points covered are ,in . use Let represent the k-th instance in the point cloud framework. Design a function to dynamically update the cluster radius r:

[0046] in, These are hyperparameters set based on experience; δ is an adjustment factor used to prevent r from being too small. This represents the number of seed points in the current instance, where t represents the t-th semantic seed point in the k-th instance. By using this formula, the cluster radius r is dynamically updated during the clustering process. Next, the DBSCAN clustering method is used to analyze the historical point cloud. In the middle, the dynamically updated cluster radius is centered on the current semantic seed point. Density clustering is performed to obtain multiple clusters. Next, bounding boxes are fitted to each cluster. and the bounding box Add to the result set. Iterate through all instances in the historical point cloud using this method to obtain the historical image. The corresponding set of pseudo-tags .

[0047] In this embodiment, a lane detection model can be obtained by training a model using historical images and historical point clouds with pseudo-labels. For details, please refer to [link to relevant documentation]. Figure 5 This step can be achieved in the following way: S141, For each pseudo-label in the historical image, calculate the distribution constraint score and meta-shape constraint score of the pseudo-label respectively.

[0048] S142, after normalizing the distribution constraint score and the meta-shape constraint score, the distribution shape score is obtained by accumulating them according to the set weight factors.

[0049] S143, the pseudo-labels are filtered based on the distribution shape score, and the lane detection model is obtained by training the model based on the historical images and historical point clouds corresponding to the filtered pseudo-labels.

[0050] To quantify the credibility of different pseudo-labels, this embodiment sets a distribution shape score, which is obtained by adding the distribution constraint score and the meta-shape constraint score according to weights. The calculation methods of the distribution constraint score and the meta-shape constraint score are described below.

[0051] For distribution-constrained score calculation, firstly, a pseudo-label is calculated. random variables , Indicates semantic seed points within the bounding box The distance to the boundary of the bounding box. Also, due to a high-quality pseudo-label... In, its internal semantic seed point The distance to the boundary roughly follows a Gaussian distribution. Therefore, by calculating each pseudo-label Corresponding random variable Compared with standard Gaussian distribution The similarity between them can be used to obtain the distribution constraint score, and the specific formula is as follows:

[0052] in, Represents the logarithmic function. It's a fake tag. The semantic seed point set within, This indicates the number of semantic seed points.

[0053] For the meta-shape constraint fraction, define Indicates category The shape of the meta-instance, where These are the standardized length, width, and height. This shape prior and pseudo-label are then used. By combining these methods, the meta-shape constraint fraction is obtained through processing. The specific formula is as follows:

[0054] in, Represents the standardized KL divergence function. Indicates pseudo-tags The standardized shape.

[0055] Finally, by combining the normalized distribution constraint score and the meta-shape constraint score, the final distribution shape score is obtained. :

[0056] in, and It is a weighting factor. and After normalization , .

[0057] Based on this, the distribution shape score of each pseudo-label will be obtained. Instead of using the confidence score of each candidate pseudo-label in the traditional nonmaximum suppression method, the resulting pseudo-label set is... By using traditional nonmaximum suppression methods, high-quality pseudo-labels are selected and low-quality pseudo-labels are suppressed, resulting in a high-quality pseudo-label set. .

[0058] Finally, the pseudo-tag set As a historical image and historical point clouds The historical images and point clouds selected from the crowdsourced dataset were annotated in the same way. Then, deep learning was trained using the training set to obtain the trained lane detection model.

[0059] In practical applications, a pre-trained lane detection model is used to process the acquired real-time images and real-time point clouds to obtain lane line information. The real-time images include images from multiple perspectives, obtained through onboard sensors. The lane line information is then mapped onto a bird's-eye view feature map. For details, please refer to [link to relevant documentation]. Figure 6 This step is achieved in the following way: S151, extract the image features of each viewpoint in the real-time image, and convert each pixel in the image of each viewpoint into a three-dimensional point in the ground coordinate system.

[0060] S152, based on the three-dimensional coordinates of the three-dimensional points of the images from each perspective in the ground coordinate system, the image features of each perspective are projected onto the bird's-eye view space to obtain the initial bird's-eye view feature map.

[0061] S153, obtain the two-dimensional coordinates and lane category vectors of each lane line in the bird's-eye view space, write the two-dimensional coordinates and lane category vectors of each lane line into the initial bird's-eye view feature map, and obtain the final bird's-eye view feature map.

[0062] In this embodiment, real-time surround view images are obtained based on vehicle-mounted sensors. and real-time point cloud ,in This represents the number of different viewpoints in a panoramic image, typically six. A CNN model can be used to extract multi-scale image features from each viewpoint.

[0063] Here, Backbone represents the main network structure in a CNN model.

[0064] Next, using the camera's extrinsic and intrinsic parameters, the pixel points corresponding to each pixel in the image from each viewpoint are projected onto the ground coordinate system:

[0065] in, These are the pixel coordinates on the image plane. These are the three-dimensional coordinates corresponding to this pixel. This is the intrinsic parameter matrix of the camera. This is the extrinsic parameter matrix of the camera.

[0066] Then, the image features from all perspectives are combined with the 3D coordinates and fused and reprojected into a unified bird's-eye view space (BEV space) using a view transformation method to obtain the initial bird's-eye view feature map. .

[0067] Processing real-time images using lane detection models and real-time point cloud The three-dimensional position of each lane line is obtained. .in, Indicates the three-dimensional position of the point. Indicates the category label of the lane lines, Indicates the first The number of three-dimensional points contained in each lane line. This indicates the number of lane lines identified.

[0068] Then, two-dimensional sampling is performed on the lane lines to obtain two-dimensional coordinates. and each point of the lane line The mapping formula for BEV raster is as follows:

[0069]

[0070] in, , Represents the minimum boundary value of the BEV region; , This indicates the actual size of each grid cell in the BEV image.

[0071] Detected lane line categories Features such as lane line 2D coordinates are encoded into vectors and written into the initial bird's-eye view feature map to obtain an enhanced BEV feature map that integrates image and point cloud information and includes lane line position information. Where C represents the channel dimension of the BEV feature, which serves as the final bird's-eye view feature map.

[0072] Based on this, the topological relationships between lane lines are calculated using lane line information from the bird's-eye view feature map, resulting in a topological inference matrix. For details, please refer to [link to relevant documentation]. Figure 7 This step can be achieved in the following ways: S161, a lane query is obtained by using a lane deformable encoder and based on the lane line information in the final bird's-eye view feature map, and multiple directed lane lines are generated based on the lane query.

[0073] S162, calculate the lane geometric topology matrix based on the multiple directed lane lines, and calculate the lane similarity topology matrix based on the lane query.

[0074] S163, Combining the lane geometry topology matrix and the lane similarity topology matrix, a topology inference matrix is ​​obtained.

[0075] In this embodiment, a set of learnable initial query vectors is defined. ,in C is the upper limit of the number of lanes, and C is the channel dimension of the query. This will enhance the bird's-eye view feature map. and initial query vector The input lane deformable decoder, through a deformable attention mechanism, utilizes a bird's-eye view feature map. Extract lane semantics from important locations and output lane query. .

[0076] The initial query vector can be understood as an attention pointer, used to guide the lane deformable decoder from the bird's-eye view feature map. The corresponding lane information is extracted, which serves as the initial parameters. Lane querying includes data from the bird's-eye view feature map. The lane information extracted from different lanes can be used for subsequent lane line generation.

[0077] Based on this, lane head function is used for lane lookup. And generate multiple directed lane lines from the lane line information in the bird's-eye view feature map. .

[0078] Then, the lane geometry topology matrix is ​​calculated based on multiple directed lane lines, and the lane similarity topology matrix is ​​calculated based on the lane query: Calculate the lane geometric distance between any two directed lane lines from the plurality of directed lane lines to obtain a lane geometric distance matrix; based on the lane geometric distance matrix and the learned mapping function, obtain a lane geometric topology matrix; use two multilayer perceptrons to encode the lane query to obtain encoding results, perform inner product calculation on the two encoding results to obtain lane similarity; map the lane similarity onto the lane topology to obtain a lane similarity topology matrix.

[0079] In this embodiment, the connectivity between two different directed lane lines is evaluated by calculating the distance between the start and end points of the lanes. The specific formula is as follows:

[0080]

[0081]

[0082] Where D is the lane geometric distance matrix, and These represent the preceding and following directional lane lines, respectively. Indicates directional lane lines and Geometric distance between them Indicates directional lane lines The last point, Indicates directional lane lines The first point.

[0083] Through learnable mapping functions The lane geometry distance is mapped to the lane topology (lane topology refers to the connections and adjacencies between lanes used for subsequent lane line generation), and the specific formula is as follows:

[0084] in, , It is the lane geometric distance matrix standard deviation and These are learnable parameters, or adjustable parameters, which can be obtained through machine learning.

[0085] Finally, through the function and lane geometry distance matrix The lane geometry topology matrix can be obtained as follows:

[0086] For the lane similarity topology method, two independent multilayer perceptrons are first used to query lanes. Encoding is performed, and then the inner product of the encoding results from the two multilayer perceptrons is applied to obtain lane similarity. The sigmoid function is then used to map the lane similarity onto the lane topology, resulting in the lane similarity topology matrix. The specific formula is as follows:

[0087]

[0088]

[0089] in, ∈ , express Similarity, , This indicates the number of lane queries. This represents the inner product operation of matrices. This represents the matrix transpose operation.

[0090] After obtaining the lane geometry topology matrix and the lane similarity topology matrix, these two lane topology inference results are combined and weighted using learnable coefficients to obtain the final topology inference matrix, as shown in the following formula:

[0091] in, and These are learnable parameters.

[0092] Based on the above, the corresponding road topology inference results are obtained using a graph neural network and the topology inference matrix. For details, please refer to [link to relevant documentation]. Figure 8 This step can be achieved in the following ways: S171, the topology inference matrix and the obtained lane query are input into the graph neural network to aggregate the information of adjacent lane lines corresponding to the lane query, so as to obtain the enhanced lane query.

[0093] S172, the enhanced lane query is processed using the lane head function to obtain the corresponding road topology inference result.

[0094] In this embodiment, lane query and the final topological inference matrix The input is fed into a graph neural network for multi-layer graph convolution processing, aggregating information from adjacent lane lines, and finally outputting an enhanced lane query that improves the accuracy of lane topology reasoning. Finally, enhanced lane queries are performed using the lane head function. The process is performed to obtain the final road topology reasoning result.

[0095] The so-called enhancement refers to lane queries that incorporate the topological inference matrix. Enhanced lane queries include not only information about each lane itself, but also information about its topological lanes, such as its connected lanes. The resulting road topological inference result is the final lane map, which includes information such as the actual spatial shape of lane lines, lane connectivity, and lane type.

[0096] The multimodal fusion road topology reasoning method provided in this embodiment, on the one hand, reduces the reliance on manual annotation through a semantic mask extraction and pseudo-label generation mechanism guided by a large-scale multimodal model, significantly reducing data preparation costs while ensuring detection quality. On the other hand, by introducing a lane line representation method based on BEV space unified mapping and effectively fusing multiple lane topology reasoning methods, it effectively captures the spatial structural relationships between lane segments, outputting lane topology reasoning results with high accuracy, good topological consistency, and semantic interpretability.

[0097] In summary, this solution effectively utilizes information from onboard multimodal sensors (such as cameras and radar) in crowdsourced datasets, fusing image and point cloud multimodal information. Based on unsupervised pseudo-label generation and multimodal semantic alignment mechanisms, it constructs a highly robust lane detection model. Furthermore, it effectively leverages BEV features from bird's-eye view images, integrating geometric relationships and semantic similarity into a lane topology reasoning method. Simultaneously, it utilizes graph neural networks to achieve topology reasoning of lane structures, exhibiting strong interpretability and engineering deployability. This provides a new approach for efficient, safe, and low-cost road mapping and perception systems.

[0098] Based on the same inventive concept, please refer to Figure 9This invention also provides a functional module diagram of a multimodal fusion road topology reasoning system. This embodiment can divide the multimodal fusion road topology reasoning system into functional modules based on the above method embodiments. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0099] For example, when dividing functional modules according to their respective functions, Figure 9 The multimodal fusion road topology reasoning system shown is only a schematic diagram. This system may include a setup module, a training module, a mapping module, and a reasoning module. The functions of each module are described in detail below.

[0100] The setting module is used to collect historical images and historical point clouds from the crowdsourced dataset, extract lane masks from the historical images using a multimodal model, convert the lane masks into semantic seed points in the historical point cloud, and set pseudo-labels based on the semantic seed points. The training module is used to train a lane detection model based on historical images and historical point clouds with pseudo-labels. The mapping module is used to obtain lane line information from real-time images and real-time point clouds using the lane detection model, and to map the lane line information onto the bird's-eye view feature map; The inference module is used to calculate the topological relationship between lane lines based on the lane line information in the bird's-eye view feature map, obtain the topological inference matrix, and obtain the corresponding road topological inference result based on the graph neural network and the topological inference matrix.

[0101] The multimodal fusion road topology reasoning system provided in this embodiment can be used to execute the multimodal fusion road topology reasoning method under any of the above embodiments. For details not covered in this embodiment, please refer to the corresponding descriptions in the above embodiments. This embodiment will not elaborate further here.

[0102] Please see Figure 10This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The electronic device can be a computer device, server, or similar component in an autonomous driving control platform. The electronic device includes a memory, a processor, and a communication module. The memory, processor, and communication module are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0103] The memory is used to store computer programs or data. Memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc.

[0104] The processor is used to read / write data or programs stored in the memory and execute the multimodal fusion road topology reasoning method provided in any embodiment of the present invention.

[0105] The communication module is used to establish communication connections between electronic devices and other communication terminals via a network, and to send and receive data via the network.

[0106] It should be understood that, Figure 10 The structure shown is only a schematic diagram of an electronic device; the electronic device may also include components that are larger than those shown. Figure 10 The more or fewer components shown, or having the same Figure 10 The different configurations shown.

[0107] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing machine-executable instructions, which, when executed, implement the multimodal fusion road topology reasoning method provided in the above embodiments.

[0108] Specifically, the computer-readable storage medium can be a general-purpose storage medium, such as a removable disk or hard disk. When the computer program on the computer-readable storage medium is run, it can execute the aforementioned multimodal fusion road topology reasoning method. The processes involved in the execution of the executable instructions on the computer-readable storage medium can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0109] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0110] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0111] Furthermore, in the various embodiments of the present invention, the functional modules can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0112] It should be noted that if the functionality is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal fusion road topology reasoning method, characterized in that, The method includes: Historical images and historical point clouds are collected from the crowdsourced dataset. Lane masks in the historical images are extracted using a multimodal model, and the lane masks are converted into semantic seed points in the historical point cloud. Pseudo-labels are set based on the semantic seed points. A lane detection model is obtained by training a model based on historical images with pseudo-labels and historical point clouds. The lane detection model is used to obtain lane line information from real-time images and real-time point clouds, and the lane line information is mapped onto the bird's-eye view feature map; Based on the lane line information in the bird's-eye view feature map, the topological relationship between lane lines is calculated to obtain the topological inference matrix. Based on the graph neural network and the topological inference matrix, the corresponding road topological inference result is obtained. The step of setting pseudo-labels based on the semantic seed points includes: Clustering operations are performed on the semantic seed points in the historical point cloud to determine multiple clusters; Fit bounding boxes to each of the clusters, and obtain pseudo-labels based on the bounding boxes.

2. The multimodal fusion road topology reasoning method according to claim 1, characterized in that, The step of extracting lane masks from the historical images using a multimodal model includes: The historical image is segmented using a multimodal model to generate a background semantic mask, and descriptive text for each background semantic mask is obtained. Calculate the similarity between the descriptive text and the predefined category text, and filter the background semantic mask based on the obtained similarity value to determine the lane mask.

3. The multimodal fusion road topology reasoning method according to claim 1, characterized in that, The step of converting the lane mask into semantic seed points in the historical point cloud includes: Obtain the maximum and minimum values ​​of the lane mask in the pixel coordinate system; Perform boundary shrinkage processing on the lane mask based on the maximum and minimum values; The lane mask after boundary shrinkage is converted from the historical image to the historical point cloud, and the region corresponding to the lane mask is determined as a semantic seed point.

4. The multimodal fusion road topology reasoning method according to claim 1, characterized in that, The step of training a lane detection model based on historical images and historical point clouds with pseudo-labels includes: For each pseudo-label in the historical image, calculate the distribution constraint score and meta-shape constraint score of the pseudo-label respectively; After normalizing the distribution constraint score and the meta-shape constraint score, the distribution shape score is obtained by accumulating them according to the set weight factors. The pseudo-labels are filtered based on the distribution shape score, and the lane detection model is trained based on the historical images and historical point clouds corresponding to the filtered pseudo-labels.

5. The multimodal fusion road topology reasoning method according to claim 1, characterized in that, The real-time images include images from multiple perspectives; The step of mapping the lane line information to the bird's-eye view feature map includes: Extract image features from each viewpoint in the real-time image, and convert each pixel in each viewpoint image into a three-dimensional point in the ground coordinate system; The image features of each viewpoint are projected onto the bird's-eye view space according to the three-dimensional coordinates of the three-dimensional points in the ground coordinate system to obtain the initial bird's-eye view feature map. Extract the two-dimensional coordinates and lane category vectors of each lane line in the bird's-eye view space from the lane line information, and write the two-dimensional coordinates and lane category vectors of each lane line into the initial bird's-eye view feature map to obtain the final bird's-eye view feature map.

6. The multimodal fusion road topology reasoning method according to claim 5, characterized in that, The step of calculating the topological relationships between lane lines based on the lane line information in the bird's-eye view feature map to obtain the topological inference matrix includes: Lane queries are obtained by using a lane deformable decoder and based on lane line information in the final bird's-eye view feature map, and multiple directed lane lines are generated based on the lane queries. The lane geometry topology matrix is ​​calculated based on the multiple directed lane lines, and the lane similarity topology matrix is ​​calculated based on the lane query. By combining the lane geometry topology matrix and the lane similarity topology matrix, a topology inference matrix is ​​obtained.

7. The multimodal fusion road topology reasoning method according to claim 6, characterized in that, The steps of calculating the lane geometric topology matrix based on the multiple directed lane lines and calculating the lane similarity topology matrix based on the lane query include: Calculate the lane geometric distance between any two of the multiple directed lane lines to obtain the lane geometric distance matrix; Based on the lane geometry distance matrix and the learned mapping function, the lane geometry topology matrix is ​​obtained; The lane query is encoded using two multilayer perceptrons to obtain encoding results. The inner product of the two encoding results is then calculated to obtain the lane similarity. The lane similarity is mapped onto the lane topology to obtain the lane similarity topology matrix.

8. The multimodal fusion road topology reasoning method according to claim 1, characterized in that, The step of obtaining the corresponding road topology inference result based on the graph neural network and the topology inference matrix includes: The topology inference matrix and the obtained lane query are input into the graph neural network to aggregate the information of adjacent lane lines corresponding to the lane query, so as to obtain the enhanced lane query. The enhanced lane query is processed using the lane head function to obtain the corresponding road topology inference results.

9. A multimodal fusion road topology reasoning system, characterized in that, The system is used to implement the road topology reasoning method for multimodal fusion according to any one of claims 1-8, the system comprising: The setting module is used to collect historical images and historical point clouds from the crowdsourced dataset, extract lane masks from the historical images using a multimodal model, convert the lane masks into semantic seed points in the historical point cloud, and set pseudo-labels based on the semantic seed points. The training module is used to train the lane detection model based on historical images and historical point clouds with pseudo-labels. The mapping module is used to obtain lane line information from real-time images and real-time point clouds using the lane detection model, and to map the lane line information onto the bird's-eye view feature map. The inference module is used to calculate the topological relationship between lane lines based on the lane line information in the bird's-eye view feature map, obtain the topological inference matrix, and obtain the corresponding road topological inference result based on the graph neural network and the topological inference matrix.

Images