A road topology prediction method, electronic device and driving device

Abstract

The application relates to the technical field of image processing, and particularly provides a road topology structure prediction method, an electronic device and a driving device, and aims to solve the problem that the existing road topology structure prediction cannot be effectively performed on complex and various road scenes. To this end, the road topology structure prediction method comprises the following steps: acquiring at least one frame of data containing a road, and the at least one frame of data is marked with a basic element detection result of the road; performing semantic feature extraction on the at least one frame of data to obtain semantic features; inputting the semantic features into a network model, and obtaining the topology structure of the road according to the output of the network model. The method can be applied to more complex and various road scenes by using the network model, and can automatically and effectively perform road topology structure prediction.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically providing a road topology prediction method, electronic equipment, and driving equipment. Background Technology

[0002] For autonomous vehicles, acquiring the surrounding road topology in advance is beneficial for subsequent trajectory planning. However, in traditional autonomous driving solutions, obtaining the road topology typically involves obtaining lane detection results from upstream sources and combining them with rule-based and experience-based methods for lane association to obtain the lane topology. But these rules and experiences only support a portion of conventional road scenarios, such as highways, regular ramps, and regular intersections. They cannot provide accurate lane association information for more complex and diverse road scenarios, meaning they cannot effectively predict the road topology. Summary of the Invention

[0003] This application aims to solve the aforementioned technical problem, namely, to address the issue that existing methods cannot effectively predict road topology for complex and diverse road scenarios.

[0004] In a first aspect, embodiments of this application provide a road topology prediction method, comprising:

[0005] Acquire at least one frame of data containing a road, wherein the at least one frame of data is labeled with the detection results of the basic elements of the road;

[0006] Semantic features are extracted from at least one frame of data to obtain semantic features;

[0007] The semantic features are input into the network model, and the topology of the road is obtained based on the output of the network model.

[0008] In some embodiments, acquiring at least one frame of data containing the road includes:

[0009] Acquire multi-frame data containing roads, the multi-frame data including at least one historical frame data arranged in time sequence and the current frame data.

[0010] In some embodiments, before extracting semantic features from the at least one frame of data, the method further includes at least one preprocessing operation among specified frame extraction, resampling, coordinate transformation, and constructing higher-order features from the multiple frames of data.

[0011] In some embodiments, each frame of data in the at least one frame includes multiple basic elements of a road, and each basic element is represented by multiple sampling points to indicate the detection result of the basic element. Semantic feature extraction is performed on the at least one frame of data to obtain semantic features including:

[0012] Semantic features are extracted from the sampling points corresponding to each basic element in the at least one frame of data to obtain a point-level semantic feature set corresponding to the basic element.

[0013] In some embodiments, the network model includes a transformer model, which includes an encoding layer, a fusion layer, a first self-attention layer, a decoding layer, and an output layer;

[0014] The point-level semantic feature set corresponding to each basic element is aggregated using the coding layer to obtain the line-level semantic feature corresponding to the basic element.

[0015] The fusion layer is used to fuse the line-level semantic features and point-level semantic feature sets corresponding to the basic elements to obtain the target point-level semantic feature set.

[0016] The first self-attention layer is used to fuse the semantic feature datasets at the target point level corresponding to different basic elements to obtain fused features.

[0017] The fused features and road query features are decoded using a decoding layer to obtain the decoded features of the roads;

[0018] The decoded features are input into the output layer to obtain the topology of the road, which includes at least the centerline location information of the road.

[0019] In some embodiments, the road query features include predefined point-level query features and predefined line-level query features, and the decoding layer includes a residual connection layer, a second self-attention layer, and a cross-attention layer;

[0020] The predefined point-level query features and the predefined line-level query features are fused using the residual connection layer to obtain fused query features;

[0021] The fused query features are input into the second self-attention layer to obtain the output features of the self-attention layer;

[0022] The self-attention layer output features and the fused features are input into the cross-attention layer to obtain the road's decoded features.

[0023] In some embodiments, after obtaining the road topology from the output of the network model, the method further includes performing at least one post-processing operation on the road topology, such as filtering, suppressing, and completing.

[0024] In a second aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the road topology prediction method described in any of the preceding claims.

[0025] In a third aspect, embodiments of this application provide an electronic device comprising:

[0026] At least one processor;

[0027] And, a memory communicatively connected to the at least one processor;

[0028] The memory stores a computer program that, when executed by the at least one processor, implements the road topology prediction method described above.

[0029] In a fourth aspect, embodiments of this application provide a driving device, which includes a driving device body and electronic devices as described above.

[0030] By employing the above technical solution, this application can acquire at least one frame of data containing roads, with the at least one frame of data labeled with the detection results of basic road elements; extract semantic features from the at least one frame of data to obtain semantic features; input the semantic features into a network model, and obtain the road topology based on the output of the network model. This method, by using a network model, can be applied to more complex and diverse road scenarios, automatically and effectively predicting road topology. Attached Figure Description

[0031] The preferred embodiments of this application are described below with reference to the accompanying drawings, in which:

[0032] Figure 1 This is a schematic diagram of the main steps of a road topology prediction method provided in an embodiment of this application;

[0033] Figure 2 This is a schematic diagram of the tansformer model architecture provided in the embodiments of this application;

[0034] Figure 3 This is a schematic diagram of the road topology prediction method provided in a preferred embodiment of this application;

[0035] Figures 4a-4e This is a schematic diagram of the road topology prediction results for a lane merging scenario provided in a specific example of this application;

[0036] Figures 5a-5e This is a schematic diagram of the road topology prediction results for a specific example of a scenario with missing lane lines provided in this application;

[0037] Figures 6a-6c This is a schematic diagram of the road topology prediction results for a lane bifurcation scenario provided in a specific example of this application;

[0038] Figures 7a-7c This is a schematic diagram of the road topology prediction results for a high-curvature curve scenario provided in a specific example of this application;

[0039] Figure 8 This is a schematic diagram of the electronic device structure provided in the embodiments of this application. Detailed Implementation

[0040] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.

[0041] See Figure 1 As shown, Figure 1 This is a schematic flowchart illustrating the main steps of a road topology prediction method provided in an embodiment of this application, which may include:

[0042] Step S11: Obtain at least one frame of data containing the road, and the at least one frame of data is marked with the detection results of the basic elements of the road;

[0043] Step S12: Extract semantic features from at least one frame of data to obtain semantic features;

[0044] Step S13: Input the semantic features into the network model and obtain the road topology based on the output of the network model.

[0045] In some embodiments, step S11 may specifically involve obtaining a frame of data obtained by detecting a frame of perception data containing roads using a basic element detection model, wherein the basic element detection results of the roads are marked in the frame of data.

[0046] In other embodiments, step S11 may specifically involve acquiring multi-frame data containing roads, the multi-frame data including at least one historical frame and the current frame arranged in a time sequence. In some embodiments, the multi-frame data can be obtained through the following steps: collecting perception data containing roads at different times to obtain multi-frame perception data; and using a basic element detection model to detect the multi-frame perception data separately to obtain multi-frame data, with each frame labeled with the basic element detection result of the road.

[0047] In some embodiments, the basic element detection results may include basic element type and / or location information.

[0048] As an example, the basic elements of a road may include at least one of lane lines, curbs, and stop lines.

[0049] In some embodiments, each frame of data may include multiple basic elements of a road, each basic element being represented by multiple sampling points, and the basic element detection result may be represented by the location information of multiple sampling points.

[0050] In some embodiments, semantic features may include a point-level semantic feature set. Step S12 may specifically involve extracting semantic features from the sampling points corresponding to each basic element in at least one frame of data to obtain a point-level semantic feature set corresponding to the basic element. Specifically, semantic features may be extracted separately for multiple sampling points in each basic element to obtain point-level semantic features corresponding to each sampling point, and the point-level semantic feature set is obtained from the point-level semantic features corresponding to each sampling point. As an example, the semantic features corresponding to each sampling point may include location information and / or the index of the sampling point in its corresponding basic element, wherein the temporal information of each sampling point can be obtained based on the index of the sampling point in its corresponding basic element.

[0051] In some embodiments, step S12 may employ an MLP (Multi-Layer Perceptron) to extract semantic features from at least one frame of data. In other embodiments, other conventional networks may be used for semantic feature extraction, such as a Convolutional Neural Network (CNN) to extract semantic features from at least one frame of data.

[0052] In some embodiments, the network model can be a transformer model. Step S13 can specifically involve inputting the semantic features corresponding to at least one frame of data into the transformer model. The semantic features corresponding to each frame of data can include a set of point-level semantic features corresponding to each basic element in the corresponding data. The topology of the road is obtained based on the output of the transformer model. It should be noted that the network model can also use other networks that follow the basic self-attention mechanism. As an example, the network model can be RetNet (Retentive Networks).

[0053] In some embodiments, see Figure 2 As shown, Figure 2 This is a schematic diagram of the tansformer model architecture provided in the embodiments of this application, which may include an encoding layer, a fusion layer, a first self-attention layer, a decoding layer, and an output layer.

[0054] Among them, the coding layer can be used to aggregate the point-level semantic feature set corresponding to each basic element to obtain the line-level semantic features corresponding to the basic element.

[0055] The fusion layer is used to fuse the line-level semantic features and point-level semantic feature sets corresponding to the basic elements to obtain the target point-level semantic feature set.

[0056] The first self-attention layer is used to fuse the semantic feature datasets at the target point level corresponding to different basic elements to obtain fused features;

[0057] The decoding layer is used to decode the fused features and road query features to obtain the decoded features of the roads;

[0058] The decoded features are input to the output layer to obtain the road topology, which includes at least the centerline location information of the road.

[0059] In this embodiment, a basic element can be represented based on multiple sampling points and connecting lines between those sampling points. By aggregating the point-level semantic feature set corresponding to the basic element through an encoding layer, the semantic features of the multiple sampling points used to represent the basic element can be considered and synthesized to obtain the global semantic features of the basic element, i.e., the line-level semantic features of the basic element. As an example, the line-level semantic features corresponding to the basic element may include at least one of the basic element's shape, length, direction, and type.

[0060] In this embodiment, line-level semantic features can reflect the global semantic features of basic elements, while point-level semantic feature sets, including point-level semantic features of multiple sampling points, can reflect the local semantic features of basic elements. By fusing the line-level semantic features corresponding to the basic elements and the point-level semantic features of multiple sampling points through a fusion layer, point-level semantic features corresponding to each sampling point that comprehensively consider the global and local information of the basic elements can be obtained, i.e., target point-level semantic features, which can provide more comprehensive and accurate point-level semantic information; the target point-level semantic feature set can be obtained from the target point-level semantic features corresponding to multiple sampling points.

[0061] In this process, by using the first self-attention layer to fuse the semantic feature datasets at the target point level corresponding to different basic elements, the correlation between each sampling point in each basic element and the remaining sampling points in the corresponding frame data can be obtained, thus obtaining the global semantic features of the road in the corresponding frame data, i.e., the fused features.

[0062] In some embodiments, such as Figure 2 As shown, road query features can include predefined point-level query features and predefined line-level query features, and the decoding layer can include a residual connection layer, a second self-attention layer, and a cross-attention layer.

[0063] Among them, the residual connection layer can be used to fuse predefined point-level query features and predefined line-level query features to obtain fused query features;

[0064] The fused query features are input into the second self-attention layer to obtain the output features of the self-attention layer;

[0065] The output features of the self-attention layer and the fused features are input into the cross-attention layer to obtain the decoded features of the road.

[0066] In this embodiment, the basic elements of a road can be represented by multiple sampling points and connecting lines between the sampling points. The road query features can be obtained by training a transformer model. As an example, the road query features can be composed of L learnable feature vectors of length C, where L represents the number of predefined basic elements and C represents the number of sampling points in each predefined basic element. Each sampling point in each predefined basic element can correspond to a learnable feature vector. L and C can be set according to actual needs.

[0067] In this embodiment of the application, by using the residual connection layer to fuse the predefined point-level query features and the predefined line-level query features, the predefined point-level query features and the predefined line-level query features can be weighted and summed. Thus, based on the different proportions of the predefined point-level query features and the predefined line-level query features, features that integrate the local and global information of the basic elements can be obtained, namely, fused query features.

[0068] In some embodiments, the output layer may include an MLP network. Decoding features can obtain the centerline location information of the road through the output layer. The topology of the road can be characterized by the centerline location information, where the centerline can be represented in the form of points and lines.

[0069] In some embodiments, centerline location information can be represented by multiple sampling points and the lines connecting the sampling points. As an example, by predicting the centerline location information of a lane, at least one piece of information such as the road geometry, orientation, and relationships between different lanes can be obtained, thereby yielding the lane topology results.

[0070] In other embodiments, after decoding the feature input / output layer, at least one of the following prediction results can be obtained: the type of each road, the color of the corresponding basic element for each road, and the coordinate difference between the centerline of each road and the corresponding basic element. For example, the road type may include double yellow line lanes and tidal flow lanes, etc.

[0071] The above describes a road topology prediction method provided in this application. It involves acquiring at least one frame of data containing roads, with the basic elements of the roads marked in the at least one frame; extracting semantic features from the at least one frame of data; inputting the semantic features into a network model; and obtaining the road topology based on the output of the network model. This method, by employing a network model, can be applied to more complex and diverse road scenarios, automatically and effectively predicting the road's topology.

[0072] In other embodiments, to obtain the road topology more accurately, the data can be preprocessed and / or the obtained topology can be post-processed, as described in the following embodiments. It should be noted that this embodiment can be implemented based on obtaining one frame or multiple frames of data containing roads; here, the implementation using multiple frames of time-series data containing roads is described as an example.

[0073] See Figure 3 As shown, Figure 3 This is a schematic flowchart of a road topology prediction method provided in a preferred embodiment of this application, which may include:

[0074] Step S31: Obtain multi-frame data containing the road, the multi-frame data including at least one historical frame data arranged in time sequence and the current frame data;

[0075] Step S32: Perform at least one preprocessing operation on the multi-frame data, including specified frame extraction, resampling, coordinate transformation, and construction of higher-order features;

[0076] Step S33: Extract semantic features from multiple frames of data to obtain semantic features;

[0077] Step S34: Input the semantic features into the network model, and obtain the road topology based on the output of the network model;

[0078] Step S35: Perform at least one post-processing operation on the road topology, including filtering, suppression, and completion.

[0079] In some embodiments, step S31 may specifically involve: collecting perception data containing roads at different times to obtain multiple frames of perception data; using a basic element detection model to detect the multiple frames of perception data to obtain multiple frames of data, with each frame of data labeled with the basic element detection result of the road.

[0080] In some embodiments, the basic element detection results may include basic element type and / or location information.

[0081] As an example, basic element types can include at least one of lane lines, curbs, and stop lines.

[0082] In some embodiments, each frame of data may include multiple basic elements of a road, each basic element being represented by multiple sampling points, and the basic element detection result may be represented by the location information of multiple sampling points.

[0083] In some embodiments, step S32 may specifically involve performing at least one of the following preprocessing operations:

[0084] Based on the acquisition time and / or acquisition location, specified frames are extracted from multiple frames of data to obtain different frames of data with differences, which serve as the target's multi-frame data. In this embodiment, specifically, specified frames are extracted from multiple historical frames of data, and the target's multi-frame data is obtained based on the extracted historical frames and the current frames. The road shape, color, or structure contained in each frame of the target's multi-frame data differs. By performing specified frame extraction, more representative and diverse information can be obtained as input, avoiding similar information contained in the multi-frame data and improving the accuracy of subsequent prediction results.

[0085] Resampling multiple frames of data ensures that the sampling points used to represent each basic element are evenly spaced, or that the distance between any two adjacent sampling points is approximately the same. As an example, resampling can be performed based on existing sampling points using conventional interpolation methods. Resampling allows for a more effective representation of the basic elements of a road.

[0086] Coordinate transformation is performed on multi-frame data. Specifically, when the historical frame data in multi-frame data is obtained in the coordinate system of the sensor at its corresponding acquisition time, coordinate transformation can be performed to convert the historical frame data from the coordinate system of its corresponding acquisition time to the coordinate system of the sensor at the current frame data's acquisition time, achieving coordinate system unification and facilitating subsequent data processing. As an example, the sensor can be a vehicle-mounted sensor used to acquire perception data including road conditions. Different acquisition times can correspond to coordinate systems constructed with the vehicle's own position at the corresponding time as the origin.

[0087] To construct high-order features from multiple frames of data, specifically, for the basic elements of a road in each frame, at least one of the following is constructed: 3D coordinate information of each sampling point among multiple sampling points representing the basic element; the coordinate difference between each sampling point and the previous sampling point; the index of each sampling point in its representation of the basic element; and the index of the basic element to which each sampling point belongs in the corresponding frame of data. Before constructing high-order features, it can be determined whether the annotation information of each sampling point is valid. If valid, high-order features can be constructed based on the annotation information of each sampling point. For example, the annotation information of a sampling point may include the two-dimensional coordinates corresponding to the sampling point.

[0088] In some embodiments, step S33 may employ an MLP (Multi-Layer Perceptron) to extract semantic features from the preprocessed multi-frame data. In other embodiments, other conventional networks may be used for semantic feature extraction, such as a Convolutional Neural Network (CNN) to extract semantic features from multi-frame data.

[0089] In some embodiments, the network model can adopt a transformer model. Step S34 can specifically involve inputting the semantic features corresponding to multiple frames of data into the transformer model. The semantic features corresponding to each frame of data can include a set of point-level semantic features corresponding to each basic element in the corresponding data. The topology of the road is obtained according to the output of the transformer model.

[0090] The transformer model can be adopted as follows: Figure 2 The architecture shown, for each frame of data, utilizes an encoding layer to aggregate the point-level semantic feature sets corresponding to each basic element, obtaining line-level semantic features corresponding to the basic elements. A fusion layer then fuses the line-level and point-level semantic feature sets corresponding to the basic elements to obtain the target point-level semantic feature set. A first self-attention layer fuses the target point-level semantic feature datasets corresponding to different basic elements across multiple frames, obtaining fused features for the multi-frame data. A decoding layer decodes the fused features and road query features from the multi-frame data, obtaining decoded features. Finally, the decoded features are input to the output layer to obtain the road topology, which includes at least the centerline location information of the road.

[0091] In this embodiment, a basic element can be represented based on multiple sampling points and connecting lines between those sampling points. By aggregating the point-level semantic feature set corresponding to the basic element through an encoding layer, the semantic features of the multiple sampling points used to represent the basic element can be considered and synthesized to obtain the global semantic features of the basic element, i.e., the line-level semantic features of the basic element. As an example, the line-level semantic features corresponding to the basic element may include at least one of the basic element's shape, length, direction, and type.

[0092] In this embodiment, line-level semantic features can reflect the global semantic features of basic elements, while point-level semantic feature sets, including point-level semantic features of multiple sampling points, can reflect the local semantic features of basic elements. By fusing the line-level semantic features corresponding to the basic elements and the point-level semantic features of multiple sampling points through a fusion layer, point-level semantic features corresponding to each sampling point that comprehensively consider the global and local information of the basic elements can be obtained, i.e., target point-level semantic features, which can provide more comprehensive and accurate point-level semantic information; the target point-level semantic feature set can be obtained from the target point-level semantic features corresponding to multiple sampling points.

[0093] In this process, the first self-attention layer is used to fuse the target point-level semantic feature datasets corresponding to different basic elements in multi-frame data. This allows us to obtain the correlation between each sampling point in each basic element and the corresponding frame data and the remaining sampling points in other frames. Based on the multi-frame data, we obtain the global semantic features of the road, which are the fused features of the multi-frame data.

[0094] In some embodiments, such as Figure 2 As shown, road query features can include predefined point-level query features and predefined line-level query features, and the decoding layer can include a residual connection layer, a second self-attention layer, and a cross-attention layer.

[0095] Among them, the residual connection layer can be used to fuse predefined point-level query features and predefined line-level query features to obtain fused query features;

[0096] The fused query features are input into the second self-attention layer to obtain the output features of the self-attention layer;

[0097] The output features of the self-attention layer and the fused features are input into the cross-attention layer to obtain the decoded features of the road.

[0098] In this embodiment, the basic elements of a road can be represented by multiple sampling points and connecting lines between the sampling points. The road query features can be obtained by training a transformer model. As an example, the road query features can be composed of L learnable feature vectors of length C, where L represents the number of predefined basic elements and C represents the number of sampling points in each predefined basic element. Each sampling point in each predefined basic element can correspond to a learnable feature vector. L and C can be set according to actual needs.

[0099] In this embodiment of the application, by using the residual connection layer to fuse the predefined point-level query features and the predefined line-level query features, the predefined point-level query features and the predefined line-level query features can be weighted and summed. Thus, based on the different proportions of the predefined point-level query features and the predefined line-level query features, features that integrate the local and global information of the basic elements can be obtained, namely, fused query features.

[0100] In some embodiments, the output layer may include an MLP network. Decoding features can obtain the centerline location information of the road through the output layer, and the road topology can be characterized by the centerline location information.

[0101] In some embodiments, centerline location information can be represented by multiple sampling points and the lines connecting the sampling points. As an example, by predicting the centerline location information of a lane, at least one piece of information such as the road geometry, orientation, and relationships between different lanes can be obtained, thereby yielding the lane topology results.

[0102] In other embodiments, after the fused decoded features are input to the output layer, at least one of the following prediction results can be obtained: the type of each road, the color of the corresponding basic element for each road, and the coordinate difference between the centerline of each road and the corresponding basic element. For example, the road type may include double yellow line lanes and tidal flow lanes, etc.

[0103] In some embodiments, step S35 may specifically involve performing at least one of the following post-processing operations:

[0104] Based on the centerline location information, determine whether there are two roads with a high degree of overlap. If so, non-maximum suppression can be used to filter out the road with lower confidence and retain the road with higher confidence.

[0105] Based on the centerline position information, determine whether it is the edge of the road or whether the centerline intersects with the basic elements of the road. If so, suppress the output of the road corresponding to the current centerline position information.

[0106] When multiple frames of data are acquired based on onboard sensors, the road category is completed by combining the centerline position information with the vehicle's own position coordinates. This category may include the main lane, the first lane to the left of the main lane, or the first lane to the right of the main lane.

[0107] By performing at least one post-processing operation—filtering, suppressing, or completing—on the road's topology, the road's topology can be further improved.

[0108] The above is another embodiment of the road topology prediction method provided in this application, which can achieve the same results as described above. Figure 1The same beneficial effects as the corresponding embodiments can be achieved, and the accuracy of prediction can be further improved by preprocessing the data and postprocessing the obtained topology.

[0109] The road topology prediction method provided in this application can effectively and accurately predict road topology, and has a wide range of applications. It can be applied to various scenarios, as detailed in the following specific examples.

[0110] See some specific examples. Figures 4a-4e As shown, Figures 4a-4e This is a schematic diagram illustrating the road topology prediction results for a lane merging scenario provided in a specific example of this application. Figure 4a The results include lane line detection results represented by black dots and lines, which are obtained in advance by the lane line detection model. Figure 4a It also includes the lane centerline represented by white dots and lines, which is predicted based on the lane line detection results using the road topology prediction method provided in the embodiments of this application. Figure 4b A comparison chart is provided to verify the prediction results of the road topology structure. The lane centerline, represented by white dots and lines, is predicted based on the lane line detection results using the road topology structure prediction method provided in the embodiments of this application. Figure 4b The data may also include lane centerlines pre-marked manually to verify the accuracy of the topology predictions. These lane centerlines can be represented by black dots and lines. Figure 4b It can be seen that the predicted road centerline and the marked lane centerline have a high degree of overlap, which verifies that the road topology prediction method provided in this application can effectively and accurately predict the road topology and accurately obtain the lane merging structure in the lane merging scenario.

[0111] Figures 4c-4e These correspond to three images obtained from different angles of the driving device after mapping the predicted road topology to the actual road merging scenario. Figure 4c It was obtained from the narrow-range head-mounted camera of the driving device. Figure 4d It was obtained from the head-mounted wide-angle camera of the driving device. Figure 4e The data is obtained from a narrow-range rear camera of the driving device. In the image, black lines represent lane lines, and white lines represent the predicted road topology. The accuracy of the road topology prediction method in this application is verified by comparing the prediction results with the actual scene. Figures 4c-4e It is evident that the road topology prediction method provided in this application can effectively and accurately predict lane merging scenarios.

[0112] In some specific examples, the road topology prediction method provided in this application can be applied to road topology prediction in scenarios where some lane lines are missing. See details below. Figures 5a-5e As shown. Figure 5a The method includes lane lines detected in advance by a lane line detection model and lane center lines predicted by the road topology prediction method provided in this application based on the lane line detection results. Lane lines are represented by black dots and lines, and lane center lines are represented by white dots and lines. As shown in the figure, even when some lane lines are missing in the rightmost lane line, a complete lane center line prediction result can still be obtained. Figure 5b The image shows a comparison between the predicted lane centerline and the lane centerline marked manually in advance. The marked lane centerline is represented by black dots and lines. As can be seen from the image, the predicted lane centerline and the marked lane centerline have a high degree of overlap. The road topology prediction method provided in this application can still effectively predict the road topology even in scenarios where lane lines are missing. Figures 5c-5e These three images, taken from different angles of the driving device, correspond to the road topology prediction results mapped to a real-world scenario with missing lane lines. Figure 5c It was obtained from the narrow-range head-mounted camera of the driving device. Figure 5d It was obtained from the head-mounted wide-angle camera of the driving device. Figure 5e The data is obtained from a narrow-range camera at the rear of the driving device. In the figure, the black line represents the lane line and the white line represents the road topology prediction result. The degree of conformity between the prediction result and the real scene can further verify that the road topology prediction method of this application can make predictions effectively and accurately.

[0113] In some specific examples, the road topology prediction method provided in this application can be applied to road topology prediction in lane bifurcation scenarios, see details below. Figures 6a-6c As shown. Figure 6a The process includes lane lines detected in advance by a lane line detection model and lane center lines predicted based on the lane line detection results using the road topology prediction method provided in this application. Lane lines are represented by black dots and lines, and lane center lines are represented by white dots and lines. As can be seen from the three lane lines on the right side of the figure, a road fork has occurred. Even so, lane center lines representing the road fork structure can still be obtained. Figure 6b This is a comparison diagram of the predicted lane centerline and the lane centerline marked manually in advance. The marked lane centerline is represented by black dots and lines. As can be seen from the diagram, the predicted lane centerline and the marked lane centerline have a high degree of overlap. The road topology prediction method provided in this application can still effectively predict the road topology in the scenario of lane bifurcation. Figure 6cThe method involves mapping the predicted road topology onto a real lane bifurcation scenario, then using a narrow-range head-mounted camera on the driving device to obtain an image. The degree of agreement between the predicted results and the real scene further verifies that the road topology prediction method of this application can make effective and accurate predictions.

[0114] In some specific examples, the road topology prediction method provided in this application can be applied to road topology prediction in scenarios with large curvature curves. See details... Figures 7a-7c As shown. Figure 7a The method includes lane lines detected in advance by a lane line detection model and lane center lines predicted based on the lane line detection results using the road topology prediction method provided in this application. Lane lines are represented by black dots and lines, and lane center lines are represented by white dots and lines. Figure 7b This is a comparison diagram of the predicted lane centerline and the lane centerline marked manually in advance. The marked lane centerline is represented by black dots and lines. As can be seen from the diagram, the predicted lane centerline and the marked lane centerline have a high degree of overlap. The road topology prediction method provided in this application can still effectively predict the road topology in the scenario of large curvature curves. Figure 7c The method involves mapping the predicted road topology onto a real-world high-curvature curve scenario, then using a narrow-range head-mounted camera on the driving device to obtain images. The degree of agreement between the predicted results and the real-world scenario further verifies that the road topology prediction method of this application can make effective and accurate predictions and has a wide range of applications.

[0115] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0116] In another aspect, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the road topology prediction method described in any of the above embodiments. This computer-readable storage medium may be a storage device comprising various electronic devices; optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0117] Another aspect of this application provides an electronic device that may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program that, when executed by the at least one processor, implements the road topology prediction method described in any of the above embodiments.

[0118] See Figure 8 As shown, Figure 8 The example shows a configuration where memory 81 and processor 82 are connected via a bus, and each of memory 81 and processor 82 is configured with only one instance.

[0119] In other embodiments, the electronic device may include multiple memories 81 and multiple processors 82. The program executing the road topology prediction method of any of the above embodiments may be divided into multiple subroutines, each of which may be loaded and run by a processor 82 to perform different steps of the road topology prediction method of the above method embodiments. Specifically, each subroutine may be stored in a different memory 81, and each processor 82 may be configured to execute programs in one or more memories 81 to jointly implement the road topology prediction method of the above method embodiments.

[0120] In another aspect of this application, a driving device is provided, which may include a driving device body and the electronic devices described above.

[0121] The relevant user personal information that may be involved in the various embodiments of this application is processed in strict accordance with the requirements of laws and regulations, following the principles of legality, legitimacy, and necessity, based on the reasonable purpose of the business scenario, and includes personal information that users actively provide or that is generated as a result of using the product / service, as well as personal information obtained with user authorization.

[0122] The personal information processed in this application will vary depending on the specific product / service scenario and will be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. This application will treat the user's personal information and its processing with the utmost diligence.

[0123] This application attaches great importance to the security of users' personal information and has taken reasonable and feasible security protection measures that comply with industry standards to protect users' information and prevent unauthorized access, disclosure, use, modification, damage or loss of personal information.

[0124] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.

Claims

1. A method for predicting road topology, characterized in that, include: Acquire at least one frame of data containing a road, wherein the at least one frame of data is labeled with the detection results of the basic elements of the road; Semantic features are extracted from at least one frame of data to obtain semantic features; The semantic features are input into the network model, and the topology of the road is obtained according to the output of the network model; each frame of data in the at least one frame of data includes multiple basic elements of the road; the network model includes a transformer model, which includes an encoding layer, a fusion layer, a first self-attention layer, a decoding layer and an output layer; The point-level semantic feature set corresponding to each basic element is aggregated using the coding layer to obtain the line-level semantic feature corresponding to the basic element. The fusion layer is used to fuse the line-level semantic features and point-level semantic feature sets corresponding to the basic elements to obtain the target point-level semantic feature set. The first self-attention layer is used to fuse the semantic feature datasets at the target point level corresponding to different basic elements to obtain fused features. The fused features and road query features are decoded using a decoding layer to obtain the decoded features of the roads; The decoded features are input into the output layer to obtain the topology of the road, which includes at least the centerline location information of the road.

2. The method according to claim 1, characterized in that, The acquisition of at least one frame of data containing the road includes: Acquire multi-frame data containing roads, the multi-frame data including at least one historical frame data arranged in time sequence and the current frame data.

3. The method according to claim 2, characterized in that, Before extracting semantic features from the at least one frame of data, the method further includes at least one preprocessing operation among specified frame extraction, resampling, coordinate transformation, and construction of higher-order features for the multiple frames of data.

4. The method according to any one of claims 1 to 3, characterized in that, Each basic element is represented by multiple sampling points to indicate the detection result of the basic element. Semantic features are extracted from the at least one frame of data to obtain semantic features including: Semantic features are extracted from the sampling points corresponding to each basic element in the at least one frame of data to obtain a point-level semantic feature set corresponding to the basic element.

5. The method according to claim 1, characterized in that, The road query features include predefined point-level query features and predefined line-level query features, and the decoding layer includes a residual connection layer, a second self-attention layer, and a cross-attention layer. The predefined point-level query features and the predefined line-level query features are fused using the residual connection layer to obtain fused query features; The fused query features are input into the second self-attention layer to obtain the output features of the self-attention layer; The self-attention layer output features and the fused features are input into the cross-attention layer to obtain the road's decoded features.

6. The method according to claim 1, characterized in that, After obtaining the road topology based on the output of the network model, the method further includes performing at least one post-processing operation on the road topology, such as filtering, suppressing, and completing.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the road topology prediction method according to any one of claims 1 to 6.

8. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program, which, when executed by the at least one processor, implements the road topology prediction method according to any one of claims 1 to 6.

9. A driving device, characterized in that, It includes the driving equipment body and the electronic device as described in claim 8.

Images