Method and apparatus for processing road conditions

Abstract

The application relates to a method for processing road conditions, comprising: detecting a trajectory of an object contained in an image based on the image; generating a first road condition feature map based on at least one of the trajectory of the object and the image; generating a lane center line of the lane based on the first road condition feature map; and generating a lane topology based on the lane center line.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, and more specifically, to methods and apparatus for determining road topology relationships, methods and apparatus for predicting vehicle trajectories, and methods and apparatus for training neural network (NN) models. Background Technology

[0002] Accurate modeling of complex traffic scenarios is a key component of Advanced Driver Assistance Systems (ADAS) for performing correct planning and control. Lane topology reasoning aims to understand road lane scenarios and present feasible driving routes in autonomous driving. This requires detecting lanes in the scenario and further reasoning about their topological relationships.

[0003] In existing lane topology prediction neural networks, training the network requires lane topology labels as ground truth data. However, obtaining such ground truth data is extremely costly, and the annotation process is both laborious and inefficient for large datasets. Therefore, improved lane topology prediction methods and corresponding training methods are needed to reduce the requirement for lane topology label data while maintaining the accuracy of lane topology prediction. Summary of the Invention

[0004] The following brief introduction is provided to present some of the selected concepts in a simplified manner, which will be further described in the detailed description that follows. This brief introduction is not intended to highlight the key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0005] To address the aforementioned issues, this application provides a novel lane topology prediction method and a corresponding neural network (NN) model training method. Furthermore, it also provides a novel vehicle future trajectory prediction method and a corresponding NN model training method. By employing the methods described in this application, the reliance on fully annotated lane topology data is reduced, overcoming the challenges of limited labeled data and domain-specific limitations, while maintaining the accuracy of lane topology prediction.

[0006] According to one aspect of this application, a method for processing traffic conditions is provided, comprising: detecting the trajectory of an object contained in an image based on an image; generating a first traffic condition feature map based on at least one of the trajectory of the object and the image; generating a lane centerline of the lane based on the first traffic condition feature map; and generating a lane topology based on the lane centerline.

[0007] According to one aspect of this application, a method is provided for training a neural network (NN) model for handling road conditions, comprising: determining lane centerline pseudo-labels of the lanes based on lane boundary labels in a first training set, wherein the first training set includes images and the boundary labels of lanes contained in the images; generating lane topology pseudo-labels based on the lane centerline pseudo-labels; processing the images in the first training set using the method described in embodiments of this disclosure to predict lane centerlines and lane topologies of lanes contained in the images, as well as trajectories of objects contained in the images; and updating the learning parameters of the NN model based on the lane centerline pseudo-labels, the lane topology pseudo-labels, the predicted lane centerlines, and the predicted lane topologies.

[0008] According to one aspect of this application, an apparatus for processing road conditions is provided, comprising: a detection module for detecting the trajectory of an object contained in an image; a first road condition feature extraction module for generating a first road condition feature map based on at least one of the trajectory of the object and the image; a centerline prediction module for generating a lane centerline of the lane based on the first road condition feature map; and a topology prediction module for generating a lane topology based on the lane centerline.

[0009] According to one aspect of this application, an apparatus is provided for training a neural network (NN) model for handling road conditions, comprising: a lane centerline pseudo-label generation module for determining lane centerline pseudo-labels of lanes based on lane boundary labels in a first training set, wherein the first training set includes images and lane boundary labels contained in the images; a lane topology pseudo-label generation module for generating lane topology pseudo-labels based on the lane centerline pseudo-labels; wherein the NN model processes the images in the first training set using the method described in embodiments of this disclosure to predict lane centerlines and lane topologies of lanes contained in the images; and an update module for updating the learning parameters of the NN model based on the lane centerline pseudo-labels, the lane topology pseudo-labels, the predicted lane centerlines, and the predicted lane topologies.

[0010] According to one aspect of this application, a processing apparatus is provided, comprising: one or more processors; and one or more memories storing computer-executable instructions that, when executed by the one or more processors, perform operations according to the methods described in embodiments of this application.

[0011] According to one aspect of this application, a machine-readable storage medium is provided that stores executable instructions that, when executed, cause one or more processors to perform operations of the methods described in embodiments of this application.

[0012] According to one aspect of this application, a computer program product is provided, which includes executable instructions that, when executed, cause one or more processors to perform operations of the methods described in embodiments of this application.

[0013] The technical solution disclosed herein utilizes information about road structure provided by ground truth data of lane boundaries and information about dynamic changes in the road environment provided by dynamic object trajectories, making the lane topology prediction model more adaptable to new environments. This allows the use of readily available lane boundaries as weak supervision signals to train the neural network model while ensuring predictive performance. This effectively overcomes the problem of requiring large amounts of topologically labeled data for training in existing technologies, reducing the need for expensive and labor-intensive fully labeled data. Other advantages of the technical solution disclosed herein will be described below. Attached Figure Description

[0014] A further understanding of the nature and advantages of this application can be achieved by referring to the accompanying drawings. In the drawings, similar components or features may have the same reference numerals.

[0015] Figure 1 A schematic diagram of road topology relationships according to one embodiment is shown.

[0016] Figures 2A to 2C Block diagrams of an apparatus for determining road topology relationships according to one embodiment are shown.

[0017] Figure 3A An example schematic diagram of trajectory tracking according to one embodiment is shown.

[0018] Figure 3B An example schematic diagram of a lane boundary according to one embodiment is shown.

[0019] Figure 3C An example schematic diagram of a lane centerline according to one embodiment is shown.

[0020] Figure 3D An example schematic diagram of a labeled lane boundary is shown according to one embodiment.

[0021] Figure 4 and 5 Schematic diagrams of a process for training a neural network model according to one embodiment are shown.

[0022] Figure 6 A flowchart of a method for handling traffic conditions according to one embodiment is shown.

[0023] Figure 7 A flowchart of a method for training a neural network model according to one embodiment is shown.

[0024] Figure 8 A block diagram of an apparatus for handling road conditions according to one embodiment is shown.

[0025] Figure 9 A block diagram of an apparatus for training a neural network model according to one embodiment is shown.

[0026] Figure 10 A block diagram of a processing apparatus according to one embodiment is shown. Detailed Implementation

[0027] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed merely to enable those skilled in the art to better understand and implement the subject matter described herein, and are not intended to limit the scope, applicability, or examples set forth in the claims. The function and arrangement of the elements discussed may be changed without departing from the scope of this application. Various processes or components may be omitted, substituted, or added as needed in the various examples. For example, the described methods may be performed in a different order than described, and steps may be added, omitted, or combined. Furthermore, features described in some examples may be combined in other examples.

[0028] As used herein, the term "comprising" and its variations are open terms meaning "including but not limited to". The term "based on" means "at least partially based on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other definitions, whether explicit or implicit, may be included below. Unless explicitly indicated by the context, the definition of a term remains consistent throughout the specification.

[0029] Figure 1 A schematic diagram of road topology relationships according to one embodiment is shown.

[0030] Lane line 110 is a real-world traffic marking used to divide lanes on a road. In this paper, lane line 110 is referred to as the lane boundary to distinguish it from the lane centerline. The lane topology indicated by label 120 represents the relationships between lane segments, which do not actually exist in the real world. For example, the four arrows shown in label 120 represent the relationship between a lane segment to the lower right and two lane segments to the upper right and lower right, respectively, such as a drivability relationship. The relationships between lane segments can be called lane topology relationships or road condition topology relationships. For example, this lane topology can be represented by an adjacency matrix, where each node in the adjacency matrix represents a lane segment, and the matrix elements between two nodes represent the relationship between them; for example, an element of 1 indicates a drivability relationship between the two, and 0 indicates no drivability relationship.

[0031] Figure 2A A block diagram of an apparatus 200A for determining road condition topology relationships according to one embodiment is shown.

[0032] The device 200A includes a detection module 210, a road condition feature extraction module 220, a lane centerline generation module 230, a topology generation module 240, and an optional vehicle trajectory prediction module 250. It is understood that the device 200A may include other modules. Figure 2A Only modules relevant to embodiments of this disclosure are shown. In one example, these modules may be deployed in a vehicle. For instance, the vehicle may have autonomous driving capabilities and may determine road topology relationships to perform these capabilities. In another example, some or all of these modules may be deployed in a server, where the vehicle and the server can communicate to exchange data.

[0033] The detection module 210 can obtain a road condition image I as input. For example, the road condition image I can include a multi-view image acquired from image sensors on the vehicle, containing multiple images of the panoramic view around the vehicle, such as six images acquired by six image sensors on the vehicle. Alternatively, image I can be one or more images corresponding to partial angles around the vehicle, such as one or more images acquired by one or more image sensors located in front of the vehicle. Another example is that image I can be an image acquired by other types of sensors, such as a three-dimensional depth image acquired by a lidar sensor. Yet another example is that image I can be a bird's-eye view (BEV) image, for example, by converting image data acquired by sensors such as image sensors and / or lidar into a BEV image in BEV coordinates.

[0034] The detection module 210 detects the trajectories OT of objects contained in the input image I and the boundaries LE of lanes contained in the image. For example, the detection module 210 includes a trajectory tracking module 212 and a lane boundary detection module 214. The trajectory tracking module 212 detects the trajectories of objects contained in the image I, and the lane boundary detection module 214 detects the boundaries of lanes contained in the image I.

[0035] The trajectory tracking module 212 can track moving objects such as vehicles and pedestrians on the lane, and the obtained object trajectories provide dynamic information about the road conditions. For example, the trajectory tracking module 212 can track the trajectory of each object over a period of time based on a series of images I that are continuous in time. Figure 3A An example of trajectory tracking is shown, involving the trajectories of two objects. For example, in the case of tracking trajectories over 5 seconds at a frequency of 10Hz, a tracked trajectory includes 50 coordinate points (or sampling points). The two object trajectories shown in the example include, for example, a trajectory identifier (ID) and coordinate points. It is understood that the trajectory may also include other information, such as trajectory type, velocity at each coordinate point, acceleration, etc. It is understood that the trajectory tracking module 212 can be implemented using any object tracking method.

[0036] The lane boundary detection module 214 detects the lane boundaries contained in image I, and the obtained lane boundaries provide static information about the road conditions. For example, the lane boundary detection module 214 detects lane boundaries in the current image I. Figure 3B An example of a lane boundary is shown, which includes multiple lane boundaries. For example, a lane boundary includes coordinate points. It is understood that lane boundaries may also include other information, such as lane boundary type, lane boundary ID, etc. It is understood that the lane boundary detection module 214 can be implemented using any object tracking method.

[0037] The road condition feature extraction module 220 generates a road condition feature representation FM1 based on the object trajectory OT and lane boundaries LE. This feature representation FM1 can also be called a road condition feature map or feature sequence FM1. The road condition feature map FM1 is a spatial representation of the road environment that includes static information (lane boundaries) and dynamic information (object trajectory). For example, the road condition feature map FM1 can be a tensor of shape (d1, d2, dc), which represents a d1×d2 grid with dc channels. For example, d1 = d2 = 16, dc = 512. In this example, each of the 16×16 grid points includes a 512-dimensional feature vector.

[0038] In one embodiment, for example Figure 3A and 3BAs shown, the object trajectory (OT) and lane boundary (LE) are included in the BEV map and used as input to the road condition feature extraction module 220. For example, as... Figure 2A As shown, the BEV map, which serves as input to the road condition feature extraction module 220, includes information on the object trajectory OT and lane boundaries LE, but not the input image I. For example, as... Figure 2A As shown by the dashed arrow from input I to feature extraction module 220, the BEV map, which serves as input to the road condition feature extraction module 220, can contain information from the input image I and information from the object trajectory OT and lane boundaries LE. For example, the object trajectory OT and lane boundaries LE can be superimposed on the BEV map of the input image I to obtain the BEV map that serves as input to the road condition feature extraction module 220. It is understood that the object trajectory OT and lane boundaries LE can be used as input to the road condition feature extraction module 220 in any suitable manner, and are not limited to the BEV map method described above.

[0039] In one embodiment, the road condition feature extraction module 220 includes a feature extraction module 222 and a feature output module 224. The feature extraction module 222 processes the input object trajectory OT and lane boundary LE to generate a feature map FM2, which has spatial dimensions d1×d2×dc and represents high-level spatial features extracted from the object trajectory OT and lane boundary LE. In one embodiment, the feature extraction module 222 can be implemented using a convolutional neural network (CNN). For example, the feature extraction module 222 can be implemented using a ResNet18 backbone network, which is a CNN network for feature extraction with the ability to extract hierarchical high-level spatial features. It is understood that any suitable backbone network can be used to implement the feature extraction module 222.

[0040] The feature output module 224 processes the feature map FM2 to obtain a feature map FM1 that is more suitable for downstream tasks. In one embodiment, the feature output module 224 can be implemented using a multilayer perceptron (MLP) network. It is understood that any suitable neural network can be used to implement the feature output module 224.

[0041] The lane centerline generation module 230 generates the lane centerline LC based on the road condition feature map FM1. Figure 3C An example of a lane centerline (LC) is shown, as illustrated by the dashed line in the figure. For instance, a lane centerline is a collection of lane centerline segments, each of which may include coordinates. It is understood that lane centerline segments may also include other information, such as lane centerline segment ID, lane centerline segment type, lane centerline segment length, and the number of sampling points contained within the lane centerline segment.

[0042] In one embodiment, the lane centerline generation module 230 includes a convolution module 232 and a centerline prediction module 234. For example, the convolution module 230 may include multiple convolutional layers for predicting a heatmap of the lane centerline. In one example, taking a 16×16×512 feature map FM1 as an example, the convolutional module 230 may include three convolutional layers. Convolutional layer 1 has a 3×3 kernel, a stride of 1, and padding of 1, and reduces the number of channels in feature map FM1 from 512 to 256, thus outputting a 16×16×256 feature map FMc1. Convolutional layer 2 has a 3×3 kernel, a stride of 1, and padding of 1, and reduces the number of channels in feature map FMc1 from 256 to 128, thus outputting a 16×16×128 feature map FMc2. Convolutional layer 3 has a 3×3 kernel, a stride of 1, and padding of 1, and convolutional layer 1 reduces the number of channels in feature map FMc2 from 128 to 1, thus outputting a 16×16×1 feature map FMc3, which serves as a heatmap of the predicted lane centerline. It is understood that the convolutional module 232 can be implemented using any suitable structure.

[0043] The centerline prediction module 234 determines the lane centerline (LC) based on the heatmap FMc3. The heatmap FMc3 has the same spatial dimension of 16×16 as the input feature map FM1 and is a single-channel two-dimensional matrix, where higher element values ​​correspond to higher confidence in the lane centerline location. For example, the centerline prediction module 234 can process the heatmap FMc3 using the non-maximum suppression (NMS) method to extract the coordinates of the lane centerline segment. In one example, an extracted coordinate point can represent a lane centerline segment. Further processing can be performed based on the extracted lane centerline segment. For example, curve fitting can be performed on the extracted coordinate points to generate a smooth lane centerline. For a lane centerline segment, the coordinates of the sampling points corresponding to that lane centerline segment can be sampled on the fitted lane centerline, forming a set of sampling points for that lane centerline, which can serve as an attribute of the lane centerline segment. Other attributes, such as ID and length, can also be determined for the lane centerline segment.

[0044] The topology generation module 240 generates the lane topology LT based on the road feature map FM1 and the lane centerline LC. The lane topology LT represents the relationships between lane segments, such as the accessibility relationship between two lanes. For example, the lane centerline LC includes n lane segments, and the lane topology LT determined by the topology generation module 240 may include a lane segment-lane segment matrix of size n×n. For example, this lane segment-lane segment matrix could be... Where, the element A in the matrix ij =1 indicates that it is possible to enter lane j from lane i, and the element A in the matrix is... ij=0 indicates that lanes cannot enter lane segment j from lane segment i. It can be understood that the lane topology LT does not necessarily have to be represented in matrix form; as long as the corresponding lane segment-lane segment relationships can be represented, it can be represented in any suitable form.

[0045] In one embodiment, the topology generation module 240 is implemented using a Graph Convolutional Network (GCN). In the illustrated example, the topology generation module 240 includes a graph construction module 242, a graph convolution module 244, and a topology output module 246. The graph construction module 242 constructs a graph containing nodes and edges based on the lane centerline LC and the feature graph FM1, where each node corresponds to a lane centerline segment, and each edge corresponds to a relationship between a pair of lane centerline segments. In one embodiment, each node can be identified by the coordinates of the lane centerline segment corresponding to that node, or each node can have its ID and be associated with the coordinates of the lane centerline segment corresponding to that node. The node features of each node can be extracted from the feature graph FM1 to obtain the node feature matrix of the graph. Where n is the number of nodes. dc represents the feature vector dimension. Taking the 16×16×512 shape of the feature map FM1 as an example, the feature vector dimension of each node is 512. This means that each node carries rich feature information and can effectively characterize the characteristics of the lane centerline at that location. In one embodiment, spatial proximity can be used to assign a portion of the feature map FM1 to the corresponding node. For example, region of interest (ROI) pooling can be used to construct the feature vector of each node from the feature map FM1. Taking node i corresponding to lane segment i as an example, based on the principle of spatial proximity, the feature vector corresponding to the region adjacent to lane segment i is selected from the 16×16 feature vectors of the feature map FM1. For example, the feature vectors corresponding to region i and multiple surrounding regions of lane segment i are pooled to obtain the node feature vector of node i, which in this example is a 1×512 dimensional feature vector. By aggregating the values ​​in the feature map FM1 to calculate the feature vector of each node, each node can summarize the information of its corresponding region, thereby more effectively representing the characteristics of the lane centerline segment in the graph structure.

[0046] In one embodiment, the initial adjacency matrix can be constructed based on the distance between nodes. For example, the distance between any two nodes can be determined based on the coordinates of the lane centerline segments corresponding to each node, such as Euclidean distance or Manhattan distance. An adjacency matrix can be constructed based on a function of distance. For instance, a linear function can be used to assign weights A to the edges between nodes i and j in the adjacency matrix A. ij Set as the distance d between the two ij The reciprocal of, that is For example, a Gaussian function can be used to assign weights A to the edges between nodes i and j in the adjacency matrix A. ij Set as the distance d between the two ij The Gaussian function, i.e. Where σ is a hyperparameter. In one example, the adjacency matrix obtained above can be normalized to obtain an initial adjacency matrix A. It is understood that other functions can also be used to determine the relationship elements A between nodes i and j in adjacency matrix A. ij .

[0047] After the graph construction module 242 constructs a graph represented by the node feature matrix H and the adjacency matrix A, the graph convolution module 244 performs graph convolution operation on the graph to update the node features based on the information of adjacent nodes and capture the topological information of how the lane center line segments are connected.

[0048] According to one embodiment, the GCN of the graph construction module 242 includes L GCN layers, where each GCN convolutional layer can be represented as: Where H (1) The node feature matrix H is constructed by graph construction module 242. W is the normalized adjacency matrix of adjacency matrix A constructed based on graph building module 242. (l) σ is the weight matrix of the l-th GCN layer, and σ is the activation function. After GCN processing, the node features are obtained after graph convolution. It is understandable that any suitable graph convolution processing method can be used to perform the graph convolution operation of graph convolution module 244. For example, during the processing at each layer, the adjacency matrix... It can be fixed or updated accordingly, for example, based on the H output at layer l. (l+1) Update the adjacency matrix

[0049] The topology output module 246 is based on the feature map H output by the graph convolution module 244. (L+1) Generate the topology LT between lane centerline segments, which can be represented as a matrix. In one embodiment, the topology output module 246 can be implemented using a multilayer perceptron (MLP) network. It is understood that any suitable neural network can be used to implement the topology output module 246. In one embodiment, in addition to outputting the topology LT, the topology output module 246 may also output other information, such as the type of lane segment (left-turn lane, right-turn lane, straight lane, etc.). The properties of the lane centerline can be further updated based on the information output by the topology output module 246, for example, updating the topology and type information of lane centerline segment i. For example, the topology information may include one or more subsequent lanes that can be entered from lane centerline segment i and one or more preceding lanes that can be entered from lane centerline segment i.

[0050] In one embodiment, the vehicle trajectory prediction module 250 predicts the future trajectory of the vehicle based on lane topology LT, object trajectory OT, and road condition feature map FM1. In one embodiment, as indicated by the arrows from module 230 to module 250, the properties of the lane centerline LC are updated based on the lane topology LT as described above to obtain an updated lane centerline LC, such that the properties of the updated lane centerline segment include the lane topology information output by the topology generation module 240. The vehicle trajectory prediction module 250 predicts the future trajectory of the vehicle based on the lane centerline LC containing lane topology information LT, the object trajectory OT, and the road condition feature map FM1.

[0051] In one embodiment, the vehicle trajectory prediction module 250 includes an attention module 252 and a vehicle trajectory output module 254. In one embodiment, the attention module 252 performs cross-attention processing based on the road feature map FM1, the lane centerline LC containing lane topology information LT, and the object trajectory OT. For example, the attention module 252 generates a query feature sequence or query matrix (Q) based on the road feature map FM1, a key feature sequence or key matrix (K) based on the lane centerline LC, and a value feature sequence or value matrix (V) based on the object trajectory OT. As mentioned above, the road feature map FM1 is a feature sequence of dimension (d1×d2,dc), in other words, it includes d1×d2 dc-dimensional feature vectors. The lane centerline LC includes n lane segments, and a lane segment feature sequence FMlc of dimension (n,dk) is obtained by encoding the n lane segments. The object trajectory OT includes n1 objects, each object including trajectory identifier, trajectory sampling points, and other information, and an object trajectory feature sequence FMot of dimension (n1,dv) is obtained by encoding the n1 object trajectories. For each feature sequence FM1, FMlc, and FMot, it is projected onto the Q, K, and V matrices through a learnable linear layer: Q = Wq·FM1, K = Wk·FMlc, and V = Wv·FMot. It is understood that the generated Q, K, and V matrices have the same dimension d, which can be equal to, for example, FM1 having dimension dc, or any suitable value. Attention module 252 performs cross-attention processing on the Q, K, and V matrices to obtain a cross-attention feature map FMC with dimension (d1×d2,d). In one embodiment, the number of lane segments n included in the lane centerline LC differs from the number of objects n1 included in the object trajectory OT; therefore, the number of vectors in the two matrices is aligned when generating the K and V matrices. Alignment can be achieved through various methods, such as interpolating, padding, or repeating the V matrix to make the number of vectors in the V matrix the same as that in the K matrix. Alternatively, interpolating, padding, or repeating can be performed on both the V and K matrices to make the number of vectors in the Q, K, and V matrices the same. It is understood that the attention module 252 may include multiple attention layers, each of which iteratively performs cross-attention processing, and may include other necessary operations in addition to the cross-attention operations described above. It is understood that the attention processing of the road feature map FM1, lane centerline LC, and object trajectory OT by the attention module 252 can be implemented in any suitable manner.

[0052] The vehicle trajectory output module 254 generates the vehicle's future trajectory FT based on the feature map FMC output by the attention module 252. For example, the vehicle trajectory output module 254 generates the vehicle's future trajectory FT based on the feature map FMC output by the attention module 252 and the lane segment information where the vehicle is located. For example, the future trajectory FT includes the coordinates of predicted sampling points over a future period of time. For example, the coordinates of sampling points in the next 5 seconds; when 10 sampling points are included per second, the future trajectory includes the coordinates of 50 sampling points in the next 5 seconds. In one embodiment, the vehicle trajectory output module 254 can be implemented using a multilayer perceptron (MLP) network. It is understood that any suitable neural network can be used to implement the vehicle trajectory output module 254.

[0053] Figure 2B A block diagram of an apparatus 200B for determining road topology relationships according to one embodiment is shown. Components as shown in 2A and 2B are indicated by the same reference numerals.

[0054] Figure 2B In this embodiment, the detection module 210 includes a trajectory tracking module 212, but does not include... Figure 2A The lane boundary detection module 214 in this embodiment. The road condition feature extraction module 220 receives the trajectory OT of the object and the input image I, and generates a road condition feature map FM1 based on the trajectory OT and the input image I. For example, the BEV map as input to the road condition feature extraction module 220 may contain information about the input image I and the trajectory OT of the object. For example, the trajectory OT of the object is superimposed on the BEV map of the input image I to obtain the BEV map as input to the road condition feature extraction module 220. Figure 2B Other operations of the embodiments and Figure 2A The operation of the illustrated embodiments is the same and will not be repeated.

[0055] Figure 2C A block diagram of an apparatus 200C for determining road topology relationships according to one embodiment is shown. Components as shown in 2A to 2C are indicated by the same reference numerals.

[0056] Figure 2C In this embodiment, the object trajectory OT generated by the detection module 210 is not used as input to the road condition feature extraction module 220. The road condition feature extraction module 220 receives the input image I and generates a road condition feature map FM1 based on image I. Figure 2C Other operations of the embodiments and Figure 2B The operation of the illustrated embodiments is the same and will not be repeated.

[0057] Figure 4 A schematic diagram illustrating a process for training a neural network model according to one embodiment is shown. Figure 4 China and Israel Figure 2BThe training process is illustrated using the model structure shown as an example, and the described process is also applicable to... Figure 2A and Figure 2C The model structure shown.

[0058] During training, the ground truth data used as supervision data is the lane boundary GT-LE in image I. This ground truth lane boundary GT-LE... Figure 3D As shown, this represents the ground truth location, or label location, of the lane boundary in image I. The candidate pseudo-label determination module 410 calculates initial candidates for the lane centerline pseudo-label PGT-LC based on the lane boundary GT-LE. For example, the midpoint between two lane boundaries can be used as an initial candidate for the lane centerline pseudo-label PGT-LC. For instance, the midpoint can be obtained by interpolation between each pair of sampling points on the two lane boundaries. It is understood that any suitable matching method can be used to determine the midpoint between pairs of sampling points on the two lane boundaries.

[0059] During the pseudo-tag generation stage, modules 210 to 250 perform the actions described above. Figures 2A to 2C Especially Figure 2A and 2B The operation is similar to that described above, except that the ground truth lane boundary GT-LE corresponding to image I is used as the input to the road condition feature extraction module 220. Accordingly, the lane centerline generation module 230 generates a pseudo-label PGT-LC for the lane centerline, the topology generation module 240 generates a pseudo-label PGT-LT for the lane topology, and the optional vehicle trajectory prediction module 250 generates a pseudo-label PGT-FT for the vehicle's future trajectory.

[0060] During the inference phase, modules 210 to 250 perform actions in conjunction with the above. Figures 2A to 2C The operation described differs from the pseudo-label generation stage in that the ground truth lane boundary GT-LE corresponding to image I is not used as input to the road condition feature extraction module 220. Accordingly, the lane centerline generation module 230 generates the lane centerline LC, the topology generation module 240 generates the lane topology LT, and the optional vehicle trajectory prediction module 250 generates pseudo-labels FT for the vehicle's future trajectory.

[0061] Loss module 420 determines the loss L based on pseudo-labels PGT-LC, PGT-LT, and PGT-FT and the corresponding predicted values ​​LC, LT, and FT. According to one embodiment, in the initial stage of training, the loss can be determined based on the initial candidates of lane centerline pseudo-labels PGT-LC output by candidate pseudo-label determination module 410, and after the initial stage, the loss is determined based on the pseudo-labels PGT-LC of the lane centerline generated by lane centerline generation module 230.

[0062] According to one embodiment, in order to generate accurate lane centerline pseudo-labels and align them with the trajectory of dynamic objects, the centerline-related loss may include boundary loss L. LC1 And optionally include dynamic trajectory consistency loss L LC2 and / or alignment loss L LC3 .

[0063] Boundary loss L LC1 Weak supervision from the ground truth lane boundary GT-LE is used to ensure the predicted centerline is close to the midpoint between the left and right lane boundary lines. For example, the boundary loss can be calculated using binary cross-entropy (BCE), L... LC1 =∑ i,j BCE(LC(i,j),PGT_LC(i,j)), where LC(i,j) and PGT_LC(i,j) are the coordinates of the sampling points of the predicted center line and the pseudo-label center line, respectively.

[0064] Dynamic trajectory consistency loss L LC2 This aligns the predicted centerline with the trajectory of the dynamic object. For example, the dynamic trajectory consistency loss, L, can be calculated based on the distance between the predicted centerline and the trajectory of the dynamic object. LC2 =∑ t Dist(LC(t),OT(t)), where LC(t) is the predicted center line at time point t, and OT(t) is the dynamic object trajectory at time point t output by the detection module 210.

[0065] Alignment loss L LC3 The goal is to align dynamic object trajectories with lane centerlines even when they are not perfectly aligned. This loss factor takes into account the spatial transformation required to align the dynamic object trajectories with the lane centerlines. For example, a spatial transformation network or an optical flow network can be used to compute the alignment loss, L. LC3 =∑ i,j ‖LC(i,j)-Transform(OT(i,j))‖, where Transform() represents a spatial transformation network or an optical flow network.

[0066] According to one embodiment, the loss related to lane topology can be calculated using binary cross-entropy (BCE). Among them LT ij and PGT_LT ij These are the adjacency matrix element values ​​for the predicted lane topology and the pseudo-label lane topology, respectively.

[0067] According to one embodiment, the loss associated with the vehicle trajectory can be calculated using the minimum mean square error (MSE). Among them (PGT_x) i ,PGT_y i(FT_x) represents the coordinates of the pseudo-label PGT-FT of the vehicle's future trajectory at time step i. i FT_y i ) is the coordinate of the predicted future trajectory of the vehicle at time step i.

[0068] In one embodiment, the loss module 420 can determine the loss L based on the aforementioned loss components. For example, the loss L can be the sum of the aforementioned loss components. It is understood that the loss L may include a portion of the aforementioned losses, or it may include other loss components.

[0069] In one embodiment, Figure 4 The components 210 to 250 of the NN model shown can be trained together or separately. For example, the lane centerline generation module 230 can be trained first based on a loss related to the lane centerline, and then the topology generation module 240 can be trained individually or additionally based on a loss related to the lane topology, or modules 230 and 240 can be trained jointly. Then, the vehicle trajectory prediction module 250 can be trained individually or additionally based on a loss related to the vehicle's future trajectory, or modules 230 to 250 can be trained jointly.

[0070] Figure 4 The neural network model shown learns to generate pseudo-labels under weak supervision of lane boundary ground truth (GT-LE), therefore it can also be called a pseudo-label generator. For example, Figure 2A and 2B The neural network model shown can be used as a pseudo-label generator. The pseudo-labels generated by this generator can also be used to train other neural network models for generating lane centerlines, lane topology, and / or future vehicle trajectories. For example, it can be used to train... Figure 2C The NN model shown.

[0071] Figure 5 A schematic diagram illustrating a process for training a neural network model according to one embodiment is shown. Figure 5 China and Israel Figure 2B The training process is illustrated using the model structure shown as an example, and the described process is also applicable to... Figure 2A and 2C The model structure shown.

[0072] exist Figure 5 The illustrated embodiment utilizes domain adaptation (DA) to train a neural network model in the target domain T. The training data in the source domain S includes ground truth values ​​GT_LT_S of the lane topology, and optionally ground truth values ​​GT_LC_S of the lane centerlines. Based on the training data in the source domain S, a neural network model can be trained in the source domain S, such as... Figures 2A to 2CThe NN model shown is problematic because, due to variations in factors such as camera settings, vehicle type, and geographical region, an NN model trained in the source domain cannot accurately process data in the target domain. In this embodiment, training data from the source domain is transferred to the target domain using Data Acquisition (DA) to train the NN model in the target domain. For example, the source domain (first domain) and the target domain (second domain) can correspond to two different regions, such as two countries, two cities, two districts, or two smaller regions.

[0073] According to one embodiment, the road condition feature extraction module 220 generates feature maps FM1_S and FM1_T based on the source domain image I_S and the target domain image I_T, respectively. The discriminator 510 generates domain discriminations D(FM1_S) and D(FM1_T) based on the feature maps FM1_S and FM1_T, which are, for example, the probabilities that a feature is judged to belong to either the source or target domain. The loss module 520 can generate a loss related to DA based on the domain discriminations D(FM1_S) and D(FM1_T), which can also be called adversarial loss, for example... The goal of this loss is to prevent the discriminator 510 from distinguishing between the feature maps FM1_S and FM1_T generated by the road condition feature extraction module 220, thereby enabling the road condition feature extraction module 220 to adapt the training data from the source domain to the target domain. It is understood that any suitable domain-adaptive related loss can be used.

[0074] like Figure 5 As shown, the adaptive feature FM1_S is input to the lane centerline generation module 230, which generates the lane centerline LC accordingly, and the topology generation module 240 generates the lane topology LT. The loss module 520 can generate a lane topology-related loss L based on the predicted lane topology LT and the ground truth lane topology GT_LT_S. LT And it can generate a loss L related to the lane centerline based on the predicted lane centerline LC and the true lane centerline GT_LC_S. LC .

[0075] In one embodiment, the loss module 520 can determine the loss L based on the aforementioned loss components. For example, the loss L can be the sum of the aforementioned loss components. It is understood that the loss L can include a portion of the aforementioned losses, or it can include other loss components.

[0076] In one embodiment, they can be used in combination. Figure 4 and Figure 5 The training process is used to train the NN model, for example, by alternately using Figure 4 Weakly supervised training and Figure 5 For example, in DA training, fully annotated training data with regional adaptation and weakly supervised pseudo-labeled training data can be combined into a larger dataset to train the NN model.

[0077] According to various aspects of this disclosure, the technical solution of this application utilizes information about road structure provided by ground-value data of lane boundaries and information about dynamic changes in the road environment provided by dynamic object trajectories. By using more general and less location-sensitive lane boundary lines and dynamic object trajectory detection results, the lane topology prediction model is made more adaptable to new environments. This allows the NN model to be trained using readily available lane boundaries as weak supervision signals while ensuring predictive performance, effectively overcoming the problem of requiring large amounts of topologically labeled data for training in existing technologies, and reducing the need for expensive and labor-intensive fully labeled data. Furthermore, by applying domain adaptation, the NN model can adapt to changes in camera settings, vehicle types, and geographical regions, making it highly versatile in different environments. By combining weak supervision methods, dynamic object perception, and domain adaptation, a more efficient, cost-effective, and scalable solution for lane topology prediction is provided.

[0078] Figure 6 A flowchart of a method for handling traffic conditions according to one embodiment is shown.

[0079] In step 610, the trajectory of the object contained in the image is detected based on the image.

[0080] In step 620, a first road condition feature map is generated based on the trajectory of the object and at least one of the images.

[0081] In step 630, the lane centerline of the lane is generated based on the first road condition feature map.

[0082] In step 640, a lane topology is generated based on the lane centerline.

[0083] According to one embodiment, in step 650, the future trajectory of the vehicle is predicted based on the lane topology, the trajectory of the object, and the first road condition feature map.

[0084] According to one embodiment, in step 610, the boundaries of lanes contained in the image are further detected based on the image. In step 620, the first road condition feature map is further generated based on the boundaries of the lanes.

[0085] According to one embodiment, in step 630, a heatmap is generated based on the first road condition feature map; and the lane centerline is determined based on the heatmap. For example, a convolutional neural network is used to generate the heatmap based on the first road condition feature map. For example, non-maximum suppression is applied to the heatmap to determine the position of the lane centerline.

[0086] According to one embodiment, in step 640, a lane topology is generated based on the first road condition feature map and the lane centerline. According to one embodiment, in step 640, a graph containing nodes and edges is constructed based on the lane centerline, wherein each node corresponds to a lane centerline segment in the lane centerline, and each edge corresponds to a relationship between a pair of lane centerline segments in the lane centerline; the lane topology is generated based on the graph using a graph convolutional network (GCN). According to one embodiment, constructing the graph containing nodes and edges includes: determining the node features of each node and the weights of each edge based on the first road condition feature map and the lane centerline. For example, the node features of all nodes form a node matrix, and the weights of all edges form an adjacency matrix; the node matrix and the adjacency matrix represent the graph. According to one embodiment, determining the node features of each node includes: determining partial features related to the node in the first road condition feature map based on the position of each node, and determining the node features of the node based on the partial features.

[0087] According to one embodiment, in step 650, a second feature map is obtained based on the first road condition feature map, the lane topology, and the trajectory of the object; the future trajectory of the vehicle is generated based on the second feature map.

[0088] According to one embodiment, method 600 further includes: updating the attributes of the lane centerline based on the lane topology to obtain an updated lane centerline. Obtaining the second feature map in step 650 includes: generating a query feature map based on the first feature map, generating a key feature map based on the updated lane centerline, and generating a value feature map based on the trajectory of the object; performing cross-attention processing based on the query feature map, the key feature map, and the value feature map to obtain the second feature map.

[0089] Figure 7 A flowchart of a method for training a neural network model for handling road conditions, according to one embodiment, is shown.

[0090] In step 710, a pseudo-label of the lane centerline is determined based on the lane boundary labels in the first training set, wherein the first training set includes images and the lane boundary labels contained in the images.

[0091] In step 720, a lane topology pseudo-label is generated based on the lane centerline pseudo-label.

[0092] In step 730, the NN model is combined with Figures 2A to 6 The method processes the image in the first training set to predict the lane centerline and lane topology of the lanes contained in the image, as well as the trajectory of the objects contained in the image.

[0093] In step 740, the learning parameters of the NN model are updated based on the lane centerline pseudo-label, the lane topology pseudo-label, the predicted lane centerline, and the predicted lane topology.

[0094] According to one embodiment, the NN model includes combining Figures 2A to 6 The feature extraction module 220, lane centerline prediction model 230, lane topology prediction model 240, and optional vehicle trajectory prediction model 250 are described.

[0095] According to one embodiment, in step 720, the future trajectory pseudo-label of the vehicle is determined based on the lane topology pseudo-label. In step 730, the NN model combines... Figures 2A to 6 The method processes the images in the first training set to predict the future trajectory of the vehicle. In step 740, the learning parameters of the NN model are also updated based on the future trajectory pseudo-label and the predicted future trajectory. According to one embodiment, in step 740, the learning parameters of the NN model are further updated based on the predicted trajectory of the object.

[0096] According to one embodiment, determining the lane centerline pseudo-label in step 720 includes: calculating the midpoint of the lane based on the left and right boundary labels of the lane, and using this midpoint as the lane centerline pseudo-label. For example, this pseudo-label can be used as an initial candidate for pseudo-labels in the initial stage of training.

[0097] According to one embodiment, determining the lane centerline pseudo-label in step 720 includes: generating a first road condition feature map based on the trajectory of the object and the boundary labels of the lane; and generating the lane centerline pseudo-label based on the first road condition feature map. For example, the pseudo-label can be used in subsequent stages of training. It is understood that the pseudo-label obtained by the midpoints of the left and right boundaries and the pseudo-label obtained by the centerline generation module 230 can be used in any suitable manner, such as alternating use or merging use.

[0098] According to one embodiment, the first training set corresponds to a first domain, and the second training set, corresponding to a second domain, includes images and lane topology labels. Method 700 further includes: processing images from the first training set to obtain a third feature map; processing images from the second training set to obtain a fourth feature map; generating a lane topology prediction based on the fourth feature map; generating a first domain prediction by a discriminator based on the first feature map; and generating a second domain prediction based on the second feature map. In step 740, the learning parameters of the NN model are also updated based on the lane topology prediction, the lane topology labels, the first domain prediction, and the second domain prediction.

[0099] According to one embodiment, the second training set also includes lane centerline labels. Method 700 further includes generating lane centerline predictions based on the fourth feature map. In step 740, the learning parameters of the NN model are also updated based on the lane centerline predictions and the lane centerline labels.

[0100] Figure 8 A block diagram of an apparatus for handling road conditions according to one embodiment is shown.

[0101] The device 800 includes a detection module 810, a first road condition feature extraction module 820, a centerline prediction module 830, a topology prediction module 840, and an optional trajectory prediction module 850.

[0102] According to one embodiment, detection module 810 detects the trajectory of an object contained in the image; first road condition feature extraction module 820 generates a first road condition feature map based on the trajectory of the object and at least one of the image; centerline prediction module 830 generates a lane centerline of the lane based on the first road condition feature map; and topology prediction module 840 generates a lane topology based on the lane centerline. According to one embodiment, trajectory prediction module 850 predicts the future trajectory of the vehicle based on the lane topology, the trajectory of the object, and the first road condition feature map.

[0103] According to one embodiment, the detection module 810 further detects the boundaries of lanes contained in the image. The first road condition feature extraction module 820 further generates the first road condition feature map based on the lane boundaries.

[0104] According to one embodiment, the centerline prediction module 830 generates a heatmap based on the first road condition feature map; and determines the lane centerline based on the heatmap. For example, a convolutional neural network is used to generate the heatmap based on the first road condition feature map. For example, non-maximum suppression is applied to the heatmap to determine the position of the lane centerline.

[0105] According to one embodiment, the topology prediction module 840 generates a lane topology based on the first road condition feature map and the lane centerline. According to one embodiment, the topology prediction module 840 constructs a graph containing nodes and edges based on the lane centerline, wherein each node corresponds to a lane centerline segment in the lane centerline, and each edge corresponds to a relationship between a pair of lane centerline segments in the lane centerline; the lane topology is generated based on the graph using a graph convolutional network (GCN). According to one embodiment, the topology prediction module 840 determines the node features of each node and the weight of each edge based on the first road condition feature map and the lane centerline. According to one embodiment, the topology prediction module 840 determines the partial features related to the node in the first road condition feature map based on the position of each node in the node, and determines the node features of the node based on the partial features.

[0106] According to one embodiment, the trajectory prediction module 850 obtains a second feature map based on the first road condition feature map, the lane topology, and the trajectory of the object; and generates the future trajectory of the vehicle based on the second feature map.

[0107] According to one embodiment, the properties of the lane centerline are updated based on the lane topology to obtain an updated lane centerline. The trajectory prediction module 850 generates a query feature map based on the first feature map, a key feature map based on the updated lane centerline, and a value feature map based on the trajectory of the object; cross-attention processing is performed on the query feature map, the key feature map, and the value feature map to obtain the second feature map.

[0108] Figure 9 A block diagram of an apparatus for training a neural network model for handling road conditions, according to one embodiment, is shown.

[0109] The device 900 includes a lane centerline pseudo-label generation module 910, a lane topology pseudo-label generation module 920, an update module 930, and an optional future trajectory pseudo-label generation module 940.

[0110] According to one embodiment, the lane centerline pseudo-label generation module 910 determines the lane centerline pseudo-label of a lane based on the lane boundary labels in a first training set, wherein the first training set includes images and the lane boundary labels contained in the images. The lane topology pseudo-label generation module 920 generates lane topology pseudo-labels based on the lane centerline pseudo-labels. This NN model combines... Figures 2A to 6 The method processes the image in the first training set to predict the lane centerlines and lane topology of lanes contained in the image, as well as the trajectories of objects contained in the image. The update module 930 updates the learning parameters of the NN model based on the lane centerline pseudo-label, the lane topology pseudo-label, the predicted lane centerlines, and the predicted lane topology.

[0111] According to one embodiment, the NN model includes combining Figures 2A to 6 The feature extraction module 220, lane centerline prediction model 230, lane topology prediction model 240, and optional vehicle trajectory prediction model 250 are described.

[0112] According to one embodiment, the future trajectory pseudo-label generation module 940 determines the vehicle's future trajectory pseudo-label based on the lane topology pseudo-label. This NN model combines... Figures 2A to 6 The method processes the images in the first training set to predict the future trajectory of the vehicle. The update module 930 also updates the learning parameters of the NN model based on the future trajectory pseudo-label and the predicted future trajectory. According to one embodiment, the update module 930 further updates the learning parameters of the NN model based on the predicted trajectory of the object.

[0113] According to one embodiment, the lane topology pseudo-label generation module 920 calculates the midpoint of the lane based on the left boundary label and the right boundary label of the lane, and uses it as the pseudo-label of the lane centerline.

[0114] According to one embodiment, the lane topology pseudo-label generation module 920 generates a first road condition feature map based on the trajectory of the object and the boundary label of the lane; and generates a lane centerline pseudo-label for the lane based on the first road condition feature map.

[0115] According to one embodiment, the first training set corresponds to a first domain, and the second training set, corresponding to a second domain, includes images and lane topology labels. The neural network model processes images from the first training set to obtain a third feature map; processes images from the second training set to obtain a fourth feature map; and generates a lane topology prediction based on the fourth feature map. The discriminator of the device 900 generates a first domain prediction based on the first feature map and a second domain prediction based on the second feature map. The update module 930 further updates the learning parameters of the neural network model based on the lane topology prediction, the lane topology labels, the first domain prediction, and the second domain prediction.

[0116] According to one embodiment, the second training set also includes lane centerline labels. The NN model generates lane centerline predictions based on the fourth feature map. The update module 930 further updates the learning parameters of the NN model based on the lane centerline predictions and the lane centerline labels.

[0117] Figure 10 A block diagram of a processing apparatus according to one embodiment is shown.

[0118] The processing apparatus or processing system 1000 includes one or more control units or processing units 1010 that execute one or more machine-readable instructions stored or encoded in a machine-readable storage medium (i.e., memory 1020). In one embodiment, the processing unit 1010 is configured to execute the instructions in conjunction with the above when executing program instructions. Figure 1-9 The description includes various operations and functions.

[0119] Those skilled in the art will understand that the apparatus 800 to 1000 may also include various other components, such as various communication modules, bus modules, and possible user interface modules.

[0120] According to one embodiment, a program product, such as a non-transitory machine-readable medium, is provided. This non-transitory machine-readable medium may have instructions that, when executed by the processing unit 1010, are capable of performing the above-described combinations of various embodiments of this application. Figures 1 to 9 The description includes various operations and functions.

[0121] According to one embodiment, a computer program product is provided. The computer program product includes machine-executable instructions that, when executed by a processing unit 1010, are capable of performing the above-described combinations of various embodiments of this application. Figures 1 to 9 The description includes various operations and functions.

[0122] The specific embodiments described above with reference to the accompanying drawings are exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of the claims. The term "example" as used throughout this specification means "serving as an example, instance, or illustration" and does not imply that it is "preferred" or "advantageous" compared to other embodiments. Specific details are included to provide an understanding of the described technology. However, these technologies can be practiced without these specific details. In some instances, well-known structures and apparatuses are shown in block diagram form to avoid obscuring the concepts of the described embodiments.

[0123] The foregoing description of this application is provided to enable any person skilled in the art to implement or use the application. Various modifications to the application will be apparent to those skilled in the art, and the general principles defined herein can be applied to other variations without departing from the scope of protection of this application. Therefore, this application is not limited to the examples and designs described herein, but is consistent with the widest scope of the principles and novel features disclosed herein.

Claims

1. A method for handling road conditions, comprising: Based on image detection, the trajectory of objects contained in the image is determined; A first road condition feature map is generated based on at least one of the trajectory of the object and the image; The lane centerline of the lane is generated based on the first road condition feature map; A lane topology is generated based on the lane centerline.

2. The method of claim 1, further comprising: The future trajectory of the vehicle is predicted based on the lane topology, the trajectory of the object, and the first road condition feature map.

3. The method of claim 1 or 2, further comprising: Based on image detection, the boundaries of lanes contained in the image are identified; The process of generating the first road condition feature map includes generating the first road condition feature map based on the boundaries of the lanes.

4. The method as described in claim 1 or 2, wherein, The lane centerline for generating the lane includes: A heat map is generated based on the first road condition feature map; The lane centerline is determined based on the heat map.

5. The method as described in claim 1 or 2, wherein, The generated lane topology includes: A graph containing nodes and edges is constructed based on the lane centerline, wherein each node corresponds to a lane centerline segment in the lane centerline, and each edge corresponds to a relationship between a pair of lane centerline segments in the lane centerline. The lane topology is generated based on the graph using a graph convolutional network (GCN).

6. The method of claim 5, wherein, The construction of the graph containing nodes and edges includes: determining the node features of each node and the weight of each edge based on the first road condition feature map and the lane centerline.

7. The method of claim 6, wherein, The process of determining the node features of each node in the nodes includes: determining the partial features related to the node in the first road condition feature map based on the position of each node in the nodes, and determining the node features of the node based on the partial features.

8. The method of claim 2, wherein, The predicted future trajectory of the vehicle includes: A second feature map is obtained based on the first road condition feature map, the lane topology, and the trajectory of the object; The future trajectory of the vehicle is generated based on the second feature map.

9. The method of claim 8, further comprising: The lane centerline attributes are updated based on the lane topology to obtain the updated lane centerline. The process of obtaining the second feature map includes: A query feature map is generated based on the first feature map, a key feature map is generated based on the updated lane center line, and a value feature map is generated based on the trajectory of the object. The second feature map is obtained by performing cross-attention processing based on the query feature map, the key feature map, and the value feature map.

10. A method for training a neural network (NN) model for handling road conditions, comprising: The lane centerline pseudo-label is determined based on the lane boundary labels in the first training set, wherein the first training set includes images and the lane boundary labels contained in the images. Generate lane topology pseudo-labels based on the lane centerline pseudo-labels; The NN model is used to process the images in the first training set using the method described in any one of claims 1 to 9 to predict the lane centerlines and lane topology of lanes contained in the images, as well as the trajectories of objects contained in the images. The learning parameters of the NN model are updated based on the lane centerline pseudo-label, the lane topology pseudo-label, the predicted lane centerline, and the predicted lane topology.

11. The method of claim 10, further comprising: The future trajectory pseudo-label of the vehicle is determined based on the lane topology pseudo-label; The NN model is used to process the images in the first training set according to one of claims 2 to 9 to predict the future trajectory of the vehicle; The step of updating the learning parameters of the NN model includes updating the learning parameters of the NN model based on the future trajectory pseudo-label and the predicted future trajectory.

12. The method of claim 10 or 11, wherein, The updating of the learning parameters of the NN model includes: further updating the learning parameters of the NN model based on the predicted trajectory of the object.

13. The method of claim 10 or 11, wherein, The process of determining the lane centerline pseudo-label includes: The midpoint of the lane is calculated based on the labels of both sides of the lane and is used as the pseudo label of the lane centerline.

14. The method of claim 10 or 11, wherein, The process of determining the lane centerline pseudo-label includes: A first road condition feature map is generated based on the trajectory of the object and the boundary labels of the lane; A pseudo-label for the lane centerline is generated based on the first road condition feature map.

15. The method of claim 10 or 11, wherein, The first training set corresponds to a first domain, and the second training set, corresponding to a second domain, includes images and lane topology labels. The method further includes: The images from the first training set are processed to obtain a third feature map; The images from the second training set are processed to obtain the fourth feature map; Lane topology prediction is generated based on the fourth feature map; The discriminator generates a first domain prediction based on the third feature map and a second domain prediction based on the fourth feature map; The step of updating the learning parameters of the NN model includes updating the learning parameters of the NN model based on the lane topology prediction, the lane topology label, the first domain prediction, and the second domain prediction.

16. The method of claim 15, wherein, The second training set also includes lane centerline labels, and the method further includes: generating lane centerline predictions based on the fourth feature map; The step of updating the learning parameters of the NN model includes updating the learning parameters of the NN model based on the lane centerline prediction and the lane centerline label.

17. An apparatus for handling road conditions, comprising: A detection module is used to detect the trajectory of objects contained in the image based on the image. The first road condition feature extraction module is used to generate a first road condition feature map based on at least one of the trajectory of the object and the image. The centerline prediction module is used to generate the lane centerline of the lane based on the first road condition feature map; The topology prediction module is used to generate lane topology based on the lane centerline.

18. The apparatus of claim 17, further comprising: The trajectory prediction module is used to predict the future trajectory of the vehicle based on the lane topology, the trajectory of the object, and the first road condition feature map.

19. An apparatus for training a neural network model for handling road conditions, comprising: A lane centerline pseudo-label generation module is used to determine the lane centerline pseudo-label of the lane based on the lane boundary labels in a first training set, wherein the first training set includes images and the lane boundary labels contained in the images. The lane topology pseudo-label generation module is used to generate lane topology pseudo-labels based on the lane centerline pseudo-labels. The NN model processes the images in the first training set using the method described in any one of claims 1 to 9 to predict the lane centerlines and lane topology of the lanes contained in the images. An update module is used to update the learning parameters of the NN model based on the lane centerline pseudo-label, the lane topology pseudo-label, the predicted lane centerline, and the predicted lane topology.

20. An apparatus for determining road topology relationships, comprising: One or more processors; as well as One or more memories storing computer-executable instructions that, when executed by the one or more processors, perform the operations of any one of claims 1 to 16.

21. A machine-readable storage medium storing executable instructions that, when executed, cause one or more processors to perform the method as described in any one of claims 1 to 16.

22. A computer program product comprising executable instructions that, when executed, cause one or more processors to perform the method as described in any one of claims 1 to 16.

Images