Lane topology information generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention relates to a lane topological information generation method and device, electronic equipment and a storage medium. The method comprises the steps that an image shot by a camera on a vehicle is acquired; performing feature extraction on the image by using a pre-trained image feature extraction model to obtain an image feature including the feature of the target object in the image; using a pre-trained lane topology identification model to carry out lane topology identification on the image features to obtain lane topology features; and generating lane topology information representing a lane position and a topological relation between lanes based on the lane topology features. According to the embodiment of the invention, end-to-end real-time lane line and topological relation detection can be realized, the features of the objects on the road are referred in the detection process, more reference information is provided for lane line and topological relation detection, and thus the accuracy, efficiency and robustness of lane topological detection are improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent vehicle technology, and in particular to a method, device, electronic device, and storage medium for generating lane topology information. Background Art

[0002] Autonomous driving technology has garnered increasing attention in recent years. Currently, most advanced autonomous driving technologies are based on high-precision maps (HD maps). However, HD maps are expensive, have limited coverage, and are not updated regularly. With the development of deep learning technology, lane perception methods have emerged that do not rely on HD maps. However, existing lane perception and topology determination methods typically rely on the accuracy of lane perception and fail to account for the influence of road objects such as road signs and traffic lights. Consequently, the accuracy and robustness of lane topology detection urgently need to be improved. Summary of the Invention

[0003] In view of this, in order to solve some or all of the above technical problems, the embodiments of the present application provide a lane topology information generation method, device, electronic device and storage medium.

[0004] In a first aspect, an embodiment of the present application provides a method for generating lane topology information, the method comprising: acquiring an image taken by a camera on a vehicle; performing feature extraction on the image using a pre-trained image feature extraction model to obtain image features containing features of a target object in the image; performing lane topology recognition on the image features using a pre-trained lane topology recognition model to obtain lane topology features; and generating lane topology information representing lane positions and topological relationships between lanes based on the lane topology features.

[0005] In one possible embodiment, a pre-trained image feature extraction model is used to extract features from an image to obtain image features containing features of a target object in the image, including: using a pre-trained image feature extraction model to identify the position and type of the target object in the image to obtain image features containing the position code of the target object.

[0006] In one possible implementation, a pre-trained lane topology recognition model is used to perform lane topology recognition on image features to obtain lane topology features, including: obtaining a current query vector and a historical query vector of the lane topology recognition model; using a temporal attention network included in the lane topology recognition model to operate on the current query vector and the historical query vector to obtain a current query vector after a single update; using a spatial attention network included in the lane topology recognition model to operate on the current query vector and image features after a single update to obtain a current query vector after a second update; and generating lane topology features based on the current query vector after the second update.

[0007] In one possible implementation, the temporal attention network included in the lane topology recognition model is used to operate on the current query vector and the historical query vector to obtain an updated current query vector, including: using the multi-head self-attention sub-network included in the temporal attention network to operate on the current query vector and the historical query vector to obtain a first lane query vector containing lane topology timing information; using the first graph convolution sub-network included in the temporal attention network to operate on the current query vector to obtain a second lane query vector containing topological information between lanes; and fusing the first lane query vector and the second lane query vector to obtain an updated current query vector.

[0008] In one possible implementation, the spatial attention network included in the lane topology recognition model is used to operate on the current query vector and image features after the first update to obtain the current query vector after the second update, including: using the deformable attention subnetwork included in the spatial attention network to operate on the image features and the current query vector after the first update to obtain a third lane query vector; using the second graph convolution subnetwork included in the spatial attention network to operate on the image features and the current query vector after the first update to obtain a fourth lane query vector; and fusing the third lane query vector and the fourth lane query vector to obtain the current query vector after the second update.

[0009] In one possible implementation, lane topology information representing lane positions and topological relationships between lanes is generated based on lane topology features, including: decoding the lane topology features to obtain a lane line coordinate set and a lane representative point set corresponding to the target lane, wherein each lane representative point in the lane representative point set is a representative point on other lanes that have a topological relationship with the target lane; merging the lane line coordinate set and the lane representative point set, and determining the merged point set as the lane topology information.

[0010] In one possible embodiment, the image feature extraction model is pre-trained according to the following steps: obtaining a sample image and corresponding annotation information, wherein the annotation information is used to characterize the true position and true type of the target object in the sample image; using a preset initial image feature extraction model, performing object feature detection on the sample image to obtain sample image features containing the features of the target object in the sample image; using a preset supervision head network, performing object classification operations on the sample image features to obtain predicted information of the target object, wherein the predicted information is used to characterize the predicted position and predicted type of the target object; determining the error between the predicted information and the annotation information, and adjusting the parameters of the initial image feature extraction model based on the error; in response to the initial image feature extraction model meeting the preset training end conditions, determining the current initial image feature extraction model as the trained image feature extraction model.

[0011] In the second aspect, an embodiment of the present application provides a lane topology information generation device, which includes: an acquisition module for acquiring an image taken by a camera on a vehicle; an extraction module for using a pre-trained image feature extraction model to perform feature extraction on the image, and obtain image features containing features of the target object in the image; a recognition module for using a pre-trained lane topology recognition model to perform lane topology recognition on the image features, and obtain lane topology features; a generation module for generating lane topology information representing lane positions and topological relationships between lanes based on the lane topology features.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory for storing a computer program; a processor for executing the computer program stored in the memory, and when the computer program is executed, it implements the method of any embodiment of the lane topology information generation method of the first aspect of the present application.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method of any embodiment of the lane topology information generation method of the first aspect described above is implemented.

[0014] In a fifth aspect, an embodiment of the present application provides a computer program, which includes a computer-readable code. When the computer-readable code runs on a device, the processor in the device implements a method as in any embodiment of the lane topology information generation method of the first aspect mentioned above.

[0015] The lane topology information generation method, device, electronic device, and storage medium provided in the embodiments of the present application obtain images taken by a camera on a vehicle, and use a pre-trained image feature extraction model to extract features from the image to obtain image features containing features of the target object in the image. The pre-trained lane topology recognition model is then used to perform lane topology recognition on the image features to obtain lane topology features. Finally, based on the lane topology features, lane topology information representing the lane position and the topological relationship between lanes is generated. The image feature extraction model and lane topology recognition model constructed in the embodiments of the present application can achieve end-to-end real-time lane line and topological relationship detection, and refer to the features of objects on the road during the detection process, providing more reference information for lane line and topological relationship detection, thereby improving the accuracy, efficiency, and robustness of lane topology detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0019] Figure 1 A flowchart of a method for generating lane topology information provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of the lane topology provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of a flow chart of a second lane topology information generation method provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of the model architecture provided in the embodiments of the present application;

[0023] Figure 5 A schematic diagram of a flow chart of a third lane topology information generation method provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of another model architecture provided in an embodiment of the present application;

[0025] Figure 7 A schematic flow chart of a fourth lane topology information generation method provided in an embodiment of the present application;

[0026] Figure 8 A flowchart of a fifth lane topology information generation method provided in an embodiment of the present application;

[0027] Figure 9 A schematic diagram of the process of training an image feature extraction model provided in an embodiment of the present application;

[0028] Figure 10 A schematic diagram of the structure of a lane topology information generating device provided in an embodiment of the present application;

[0029] Figure 11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It is apparent that the described embodiments are only a portion of the embodiments of the present application, rather than all of the embodiments. It should be noted that, unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions, and numerical values ​​described in these embodiments do not limit the scope of the present application.

[0031] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present application are only used to distinguish between different steps, devices, modules and other objects, and neither represent any specific technical meaning nor indicate the logical order between them.

[0032] It should also be understood that in this embodiment, “a plurality of” may refer to two or more than two, and “at least one” may refer to one, two or more than two.

[0033] It should also be understood that any component, data or structure mentioned in the embodiments of the present application can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0034] In addition, the term "and / or" in this application is simply a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the related objects are in an "or" relationship.

[0035] It should also be understood that the description of each embodiment in this application focuses on the differences between the embodiments, and the same or similar aspects can be referenced with each other. For the sake of brevity, they will not be described one by one.

[0036] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0037] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the above-mentioned technologies, methods, and equipment should be considered part of the specification.

[0038] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0039] It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of this application can be combined with each other. To facilitate understanding of the embodiments of this application, the application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0040] In order to solve the technical problems of low efficiency and low accuracy of lane topology detection in the existing technology, the present application provides a lane topology information generation method, which can realize end-to-end real-time lane line and topological relationship detection, and provide more reference information of related objects for lane line and topological relationship detection, thereby improving the accuracy, efficiency and robustness of lane topology detection.

[0041] Figure 1 A flow chart of a lane topology information generation method provided in an embodiment of the present application. This method can be applied to one or more electronic devices such as vehicles (for example, intelligent driving vehicles), smart phones, laptops, desktop computers, portable computers, servers, etc. In addition, the execution subject of this method can be hardware or software. When the above-mentioned execution subject is hardware, the execution subject can be one or more of the above-mentioned electronic devices. For example, a single electronic device can execute this method, or a plurality of electronic devices can cooperate with each other to execute this method. When the above-mentioned execution subject is software, this method can be implemented as multiple software or software modules, or it can be implemented as a single software or software module. It is not specifically limited here.

[0042] like Figure 1 As shown, the method specifically includes:

[0043] Step 101: Acquire an image captured by a camera on a vehicle.

[0044] In some embodiments, a camera on a vehicle can capture the environment in which the vehicle is located in real time, and the obtained image can be a single image or a panoramic image synthesized from multiple images.

[0045] Step 102 : Using a pre-trained image feature extraction model, extract features from the image to obtain image features containing features of the target object in the image.

[0046] In some embodiments, the image feature extraction model may include a convolutional neural network that can extract features of lane markings on the road and features of target objects in the image. The target objects may be pre-specified objects of a type associated with lane markings, such as traffic lights, road signs, and obstacles.

[0047] Step 103 : Using a pre-trained lane topology recognition model, lane topology recognition is performed on the image features to obtain lane topology features.

[0048] In some embodiments, the lane topology recognition model is used to perform a comprehensive calculation of lane line features and various object features included in image features to determine information such as the position and direction of each lane on the road, as well as the topological relationship between different lanes. Lane topology features can represent features such as the position, direction, and topological relationship of each lane. The above lane topology recognition model can be constructed based on various types of neural networks. For example, a lane topology recognition model can be constructed using a neural network with a Transformer architecture to determine the topological relationship between lanes from both spatial and temporal dimensions.

[0049] Step 104 : Generate lane topology information representing lane positions and topological relationships between lanes based on the lane topology features.

[0050] In some embodiments, a decoding network can be used to decode lane topology features and restore them to a set of specific spatial coordinate points. The generated lane topology information can represent the position, direction, and other information of each detected lane, and can also include the topological relationship between different lanes. For example, the topological relationship indicates whether a vehicle is moving between different lanes. Figure 2 As shown, for lane 1, the lanes having a topological relationship therewith include lane 2, lane 4, lane 6 and lane 8.

[0051] The lane topology information generation method provided in the embodiment of the present application obtains an image captured by a camera on a vehicle, uses a pre-trained image feature extraction model to extract features from the image, and obtains image features containing features of the target object in the image. The pre-trained lane topology recognition model is then used to perform lane topology recognition on the image features to obtain lane topology features. Finally, based on the lane topology features, lane topology information representing the lane position and the topological relationship between lanes is generated. The image feature extraction model and lane topology recognition model constructed in the embodiment of the present application can achieve end-to-end real-time lane line and topological relationship detection, and refer to the features of objects on the road during the detection process, providing more reference information for lane line and topological relationship detection, thereby improving the accuracy, efficiency, and robustness of lane topology detection.

[0052] In some optional implementations of this embodiment, step 102 may be performed as follows:

[0053] The pre-trained image feature extraction model is used to identify the location and type of the target object in the image, and obtain image features containing the location code of the target object.

[0054] Specifically, a neural network used for target detection can be used to construct an image feature extraction model. This model can identify the type and location of objects in an image and thereby determine the location and type of the target object. The image feature extraction model in this embodiment does not output the location and type information of the target object. Instead, it directly outputs the image features output by the convolutional neural network to the lane topology recognition model. The output image features carry the location code of the target object.

[0055] Alternatively, a 3D position recognition method can be used to convert the 3D position of the target object in the image into a position in 3D space. That is, the position encoding also includes data indicating the depth of the target object, thereby more accurately indicating the position of the target object.

[0056] This embodiment identifies the position code of the target object through the image feature extraction model, so that in the subsequent lane topology detection stage, not only the lane lines are detected, but also the spatial position of the target object related to the lane lines is referenced to analyze the restrictions of the target object on the lane topology, thereby helping to make the lane topology detection more accurate.

[0057] In some optional implementations of this embodiment, such as Figure 3 As shown, step 103 includes:

[0058] Step 1031: Obtain the current query vector and historical query vectors of the lane topology recognition model.

[0059] Among them, in the attention mechanism, the query vector (query) is a vector used to find relevant information in the input sequence. It represents the information that the model currently needs to pay attention to for a specific output. The current query vector is the query vector processed by the model when processing the current frame of the image, and the historical query vector is the query vector processed by the model when processing historical image frames. The historical query vector in this embodiment can be the query vector corresponding to the previous frame or multiple frames of images. Typically, the current query vector is expressed as query(t), and the historical query vector can be the query vector corresponding to the previous frame, expressed as query(t-1).

[0060] In step 1032 , the temporal attention network included in the lane topology recognition model is used to perform operations on the current query vector and the historical query vector to obtain an updated current query vector.

[0061] like Figure 4As shown, lane topology recognition model 402 includes a temporal attention network 4021 and a spatial attention network 4022. Temporal attention network 4021 uses a temporal attention mechanism to operate on the current query vector query(t) and the historical query vector query(t-1), combining the lane topology features at the current moment with the lane topology features at the historical moments to update the current query vector.

[0062] In step 1033 , the spatial attention network 4022 included in the lane topology recognition model is used to perform operations on the current query vector and image features after the first update to obtain the current query vector after the second update.

[0063] like Figure 4 As shown, the image feature extraction model 401 extracts features from the input image, using the image features containing the position encoding of the target object as the key vector (K) and the value vector (V). The current query vector after the above update is used as Q and input into the spatial attention network 4022. The spatial attention network 4022 uses the spatial attention mechanism to execute the spatial attention algorithm, operating on K, V, and Q to obtain the current query vector Q' after the second update. The temporal attention network and the spatial attention network can repeat the above steps multiple times, that is, Q' is used as the updated query (t) and input into the temporal attention network again. The cycle is repeated N times, and finally the latest current query vector after the second update is obtained, and the subsequent steps are executed.

[0064] Step 1034 : Generate lane topology features based on the second updated current query vector.

[0065] Specifically, the current query vector after the second update can be directly used as the lane topology feature, or the current query vector after the second update can be calculated according to a preset method (such as dimension transformation, etc.) to obtain the lane topology feature. Figure 4 As shown, the current query vector Q' after the second update is used as the lane topology feature and is input into the detection head network 403 for decoding to obtain the lane topology information.

[0066] This embodiment constructs a lane topology model by setting up a temporal attention network and a spatial attention network, and realizes the combination of the current query vector and the historical query vector. This allows the updated query vector to express the temporal characteristics of the vehicle movement process, avoids failure errors caused by phenomena such as object occlusion, and can spatially analyze the lane topology, so that the updated query vector can express the accurate position and topological relationship of the lane, thereby improving the accuracy of lane topology detection.

[0067] In some optional implementations of this embodiment, such as Figure 5 As shown, step 1032 includes:

[0068] In step 10321, the multi-head self-attention sub-network included in the temporal attention network is used to operate on the current query vector and the historical query vector to obtain a first lane query vector containing lane topology temporal information.

[0069] In step 10322, the first graph convolutional subnetwork included in the temporal attention network is used to operate on the current query vector to obtain a second lane query vector containing topological information between lanes.

[0070] Specifically, such as Figure 6 As shown, the temporal attention network 4021 includes a multi-head self-attention sub-network 40211 (MHA) and a first graph convolution sub-network 40212 (SGN, Spectral Graph Neural Network).

[0071] The temporal attention network combines the temporal self-attention mechanism and graph convolution, inputs the current query vector query(t) and the historical query vector query(t-1) into the multi-head self-attention sub-network, and outputs the first lane query vector q t1 ; Take the current query vector query(t) as the input of the first graph convolution subnetwork and output the second lane query vector q t2 .

[0072] Step 10323: Fuse the first lane query vector and the second lane query vector to obtain an updated current query vector.

[0073] Optionally, the current query vector query(t) and the historical query vector query(t-1) can be preprocessed to ensure that the dimensions of query(t) and query(t-1) meet the computational requirements of the multi-head self-attention sub-network and the first graph convolution sub-network. The multi-head self-attention sub-network and the first graph convolution sub-network then process the preprocessed data.

[0074] The calculation formula for fusing the first lane query vector and the second lane query vector is shown in the following formula (1):

[0075]

[0076] Where MHA() represents the operation process of the multi-head self-attention sub-network, which outputs the first lane query vector. SGN() represents the operation process of the first graph convolution sub-network, which outputs the second lane query vector. t Represents the current query vector, q t-1 Represents the historical query vector. Both parameters of SGN are q t, the purpose is to analyze and update the lane topology information contained in the current query vector. In addition, the fused data can also be normalized and processed. This embodiment does not limit the specific form of fusion.

[0077] The multi-head self-attention sub-network provided in this embodiment processes the current query vector and historical query vectors, integrating the currently detected lane position and topology with historical lane positions and topology. This enhances the processing of temporal information, reduces the problem of lane line occlusion caused by objects, and improves the robustness of lane topology detection. The first graph convolution sub-network can better capture the global structural information between lanes, thereby improving the accuracy of lane topology detection.

[0078] In some optional implementations of this embodiment, such as Figure 7 As shown, step 1033 includes:

[0079] In step 10331, the deformable attention subnetwork included in the spatial attention network is used to operate on the image features and the updated current query vector to obtain the third lane query vector.

[0080] In step 10332, the second graph convolutional subnetwork included in the spatial attention network is used to operate on the image features and the updated current query vector to obtain the fourth lane query vector.

[0081] like Figure 6 As shown, the spatial graph convolutional attention network 4022 includes a deformable attention sub-network 40221 (SCA, Spatial Cross-Attention) and a second graph convolutional sub-network 40222.

[0082] In the spatial attention network, the image features are used as the key vector (K) and the value vector (V), and the current query vector after the above update is used as Q. V and Q are input into the deformable attention sub-network, and K and Q are input into the second graph convolution sub-network to obtain the third lane query vector q t3 and the fourth lane query vector q t4 .

[0083] Execute the spatial attention algorithm, operate on K, V, Q, and obtain the current query vector after the second update

[0084] Step 10333: Fuse the third lane query vector and the fourth lane query vector to obtain a second updated current query vector.

[0085] The method for fusing the third lane query vector and the fourth lane query vector may be set according to actual needs. For example, the third lane query vector and the fourth lane query vector may be merged and then normalized.

[0086] The deformable attention subnetwork provided in this embodiment can perform attention calculations based on a subset of key points in an image, rather than on global image features. This reduces computational effort and improves lane topology detection efficiency. The second graph convolutional subnetwork combines the positions of target objects and lane markings, including image features, to more accurately determine the topological structure between lanes and improve the accuracy of lane topology detection.

[0087] In some optional implementations of this embodiment, such as Figure 8 As shown, step 104 includes:

[0088] Step 1041 : Decode the lane topology features to obtain a lane line coordinate set and a lane representative point set corresponding to the target lane.

[0089] Each lane representative point in the lane representative point set is a representative point on other lanes that have a topological relationship with the target lane. The lane topology feature decoding method can be implemented using a detection head network.

[0090] Usually, the centerline of a lane can be used to represent the position and extension direction of a lane. The lane line coordinate set corresponding to the centerline of a lane 1 is represented as {(x1,y1),(x2,y2),…,(x n ,y n )}, where n represents the number of points on the lane centerline. The lane representative point set is expressed as Here, m represents the maximum number of other lanes that have a topological relationship with lane 1. The representative points of other lanes can be set according to actual needs. Typically, the representative point of a lane can be the starting point of the lane centerline.

[0091] If a lane does not have a topological relationship with other lanes, then the above All are empty; if the number of lanes with which it has a topological relationship is less than m, the coordinates of the lane representative points without a topological relationship are set to empty. Figure 2 In the lane topology diagram shown, m is defined as 6. In this diagram, there are 4 lanes that have a topological relationship with lane 1. The lane representative point set corresponding to lane 1 is:

[0092] Step 1042: merge the lane line coordinate set and the lane representative point set, and determine the merged point set as lane topology information.

[0093] like Figure 2 In the lane topology diagram shown in the figure, the lane topology information corresponding to lane 1 is represented as

[0094] The conventional representation of lanes and topology currently consists of a point set and an adjacency matrix. The point set represents a lane line individually, while the adjacency matrix represents the topological relationships between lane lines. While this method can represent lane lines and their topological relationships, if a lane line detection anomaly occurs, the adjacency matrix may become distorted. Furthermore, lane line numbers may differ between different image frames, resulting in a confusing representation of lane lines in the adjacency matrix and impacting subsequent processing steps.

[0095] This embodiment combines the lane line coordinate set and the lane representative point set as lane topology information, eliminating the need to use a critical matrix to represent lane topology relationships. When lane numbers corresponding to different frames change, the lane numbers contained in the critical matrix will become confused. However, this embodiment only needs to use lane representative points to represent lanes with topological relationships. That is, lanes with topological relationships can be located based on the coordinates of the lane representative points. There is no need to reflect lane numbers in the lane topology information, which avoids errors in the expression of topological relationships and improves the stability of lane topology detection.

[0096] In some optional implementations of this embodiment, such as Figure 9 As shown in Figure 1, the image feature extraction model is pre-trained according to the following steps:

[0097] Step 901: Obtain a sample image and corresponding annotation information.

[0098] The annotation information is used to characterize the actual position and type of the target object in the sample image.

[0099] Step 902 : Utilize a preset initial image feature extraction model to perform object feature detection on the sample image to obtain sample image features including features of the target object in the sample image.

[0100] Among them, the initial image feature extraction model can be built based on the target detection network, which can perform the detection tasks of objects such as traffic signs and signal lights, as well as the lane line segmentation tasks.

[0101] Step 903: Use the preset supervisory head network to perform object classification operations on the sample image features to obtain prediction information of the target object.

[0102] The prediction information is used to characterize the predicted position and predicted type of the target object.

[0103] Step 904 : determining the error between the predicted information and the labeled information, and adjusting the parameters of the initial image feature extraction model based on the error.

[0104] The error can be calculated using a loss function (e.g., a cross entropy loss function for classification and a regression loss function for position prediction). The loss value is calculated based on the loss function, and then the parameters of the initial image feature extraction model are adjusted using gradient descent and backpropagation to minimize the loss value.

[0105] Step 905 : In response to the initial image feature extraction model meeting a preset training end condition, the current initial image feature extraction model is determined as the trained image feature extraction model.

[0106] The training end conditions may include but are not limited to: loss value convergence, training times reaching a preset number, training duration reaching a preset duration, etc.

[0107] When training the image feature extraction model, this embodiment uses a supervision head to supervise the position and type of the target object in the sample image, so that the image features extracted by the image feature extraction model contain the position information of the target object, which helps the lane topology recognition model to refer to the position information of the target object and improves the accuracy of lane topology detection.

[0108] Figure 10 This is a schematic diagram of the structure of a lane topology information generation device provided in an embodiment of the present application. Specifically comprising:

[0109] An acquisition module 1001 is used to acquire images captured by a camera on a vehicle;

[0110] Extraction module 1002, for performing feature extraction on the image using a pre-trained image feature extraction model to obtain image features containing features of the target object in the image;

[0111] The recognition module 1003 is used to perform lane topology recognition on the image features using a pre-trained lane topology recognition model to obtain lane topology features;

[0112] The generation module 1004 is configured to generate lane topology information representing lane positions and topological relationships between lanes based on lane topology features.

[0113] In one possible implementation, the extraction module is further configured to: utilize a pre-trained image feature extraction model to identify the position and type of the target object in the image, and obtain image features containing the position code of the target object.

[0114] In one possible embodiment, the recognition module includes: an acquisition unit for acquiring the current query vector and the historical query vector of the lane topology recognition model; a temporal attention operation unit for using the temporal attention network included in the lane topology recognition model to operate on the current query vector and the historical query vector to obtain the current query vector after one update; a spatial attention operation unit for using the spatial attention network included in the lane topology recognition model to operate on the current query vector and the image feature after one update to obtain the current query vector after two updates; and a generation unit for generating lane topology features based on the current query vector after two updates.

[0115] In one possible embodiment, the temporal attention operation unit includes: a multi-head self-attention operation subunit, which uses the multi-head self-attention subnetwork included in the temporal attention network to operate on the current query vector and the historical query vector to obtain a first lane query vector containing lane topology timing information; a first graph convolution operation subunit, which uses the first graph convolution subnetwork included in the temporal attention network to operate on the current query vector to obtain a second lane query vector containing topology information between lanes; and a first fusion subunit, which fuses the first lane query vector and the second lane query vector to obtain an updated current query vector.

[0116] In one possible embodiment, the spatial attention operation unit includes: a deformable attention operation subunit, which uses the deformable attention subnetwork included in the spatial attention network to operate on the image features and the current query vector after one update to obtain a third lane query vector; a second graph convolution operation subunit, which uses the second graph convolution subnetwork included in the spatial attention network to operate on the image features and the current query vector after one update to obtain a fourth lane query vector; and a second fusion subunit, which uses the third lane query vector and the fourth lane query vector to fuse the third lane query vector and the fourth lane query vector to obtain the current query vector after two updates.

[0117] In one possible embodiment, the generation module includes: a decoding unit, used to decode the lane topology features to obtain a lane line coordinate set and a lane representative point set corresponding to the target lane, wherein each lane representative point in the lane representative point set is a representative point on other lanes that have a topological relationship with the target lane; a merging unit, used to merge the lane line coordinate set and the lane representative point set, and determine the merged point set as the lane topology information.

[0118] In one possible embodiment, the image feature extraction model is pre-trained according to the following steps: obtaining a sample image and corresponding annotation information, wherein the annotation information is used to characterize the true position and true type of the target object in the sample image; using a preset initial image feature extraction model, performing object feature detection on the sample image to obtain sample image features containing the features of the target object in the sample image; using a preset supervision head network, performing object classification operations on the sample image features to obtain predicted information of the target object, wherein the predicted information is used to characterize the predicted position and predicted type of the target object; determining the error between the predicted information and the annotation information, and adjusting the parameters of the initial image feature extraction model based on the error; in response to the initial image feature extraction model meeting the preset training end conditions, determining the current initial image feature extraction model as the trained image feature extraction model.

[0119] The lane topology information generating device provided in this embodiment may be as follows Figure 10 The lane topology information generation device shown in can execute all the steps of the above lane topology information generation methods, thereby achieving the technical effects of the above lane topology information generation methods. Please refer to the above related description for details. For the sake of brevity, it will not be repeated here.

[0120] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 11 The electronic device 1100 shown includes: at least one processor 1101, a memory 1102, at least one network interface 1104 and another user interface 1103. The various components in the electronic device 1100 are coupled together via a bus system 1105. It is understood that the bus system 1105 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 1105 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 1105 is not shown in FIG. Figure 11 Various buses are labeled as bus system 1105.

[0121] The user interface 1103 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touchpad, or a touch screen).

[0122] It is understood that the memory 1102 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 1102 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0123] In some embodiments, the memory 1102 stores the following elements, executable units, or data structures, or a subset thereof, or an extended set thereof: an operating system 11021 and application programs 11022 .

[0124] Among them, the operating system 11021 includes various system programs, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and handle hardware-based tasks. Application programs 11022 include various application programs, such as media players and browsers, which are used to implement various application services. Programs that implement the methods of the embodiments of the present application can be included in application programs 11022.

[0125] In this embodiment, by calling the program or instructions stored in the memory 1102, specifically, the program or instructions stored in the application 11022, the processor 1101 is configured to execute the method steps provided by each method embodiment, for example, including:

[0126] An image captured by a camera on a vehicle is obtained; a pre-trained image feature extraction model is used to extract features from the image to obtain image features containing features of the target object in the image; a pre-trained lane topology recognition model is used to perform lane topology recognition on the image features to obtain lane topology features; and based on the lane topology features, lane topology information representing lane positions and topological relationships between lanes is generated.

[0127] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1101. Processor 1101 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in processor 1101 or by software instructions. The above processor 1101 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software units in the decoding processor. The software units can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 1102 , and the processor 1101 reads the information in the memory 1102 and completes the steps of the above method in combination with its hardware.

[0128] It is understood that the embodiments described herein may be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, or other electronic units or combinations thereof for performing the above-mentioned functions of the present application.

[0129] For software implementation, the techniques described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0130] The electronic device provided in this embodiment may be Figure 11 The electronic device shown in can execute all the steps of the above-mentioned method for generating lane topology information, thereby achieving the technical effect of the above-mentioned method for generating lane topology information. Please refer to the above related description for details. For the sake of brevity, it will not be repeated here.

[0131] The present application also provides a storage medium (computer-readable storage medium). The storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; and the memory may also include a combination of the aforementioned types of memory.

[0132] When one or more programs in the storage medium can be executed by one or more processors, the lane topology information generation method executed on the electronic device side can be implemented.

[0133] The processor is configured to execute a program stored in the memory to implement the following steps of a method for generating lane topology information executed on the electronic device side:

[0134] An image captured by a camera on a vehicle is obtained; a pre-trained image feature extraction model is used to extract features from the image to obtain image features containing features of the target object in the image; a pre-trained lane topology recognition model is used to perform lane topology recognition on the image features to obtain lane topology features; and based on the lane topology features, lane topology information representing lane positions and topological relationships between lanes is generated.

[0135] Professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0136] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0137] It should be understood that the terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "comprise", "include", "contain" and "have" are inclusive and therefore specify the presence of stated features, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be performed in the specific order described or illustrated, unless the order of execution is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0138] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

Claims

1. A method for generating lane topology information, characterized in that: The method comprises: Obtain images captured by a camera on a vehicle; Using a pre-trained image feature extraction model, extracting features from the image to obtain image features containing features of a target object in the image; Using a pre-trained lane topology recognition model, lane topology recognition is performed on the image features to obtain lane topology features; Based on the lane topology features, lane topology information representing lane positions and topological relationships between lanes is generated.

2. The method according to claim 1, characterized in that The method of extracting features from the image using a pre-trained image feature extraction model to obtain image features containing features of the target object in the image includes: The position and type of the target object in the image are identified using a pre-trained image feature extraction model to obtain image features containing the position code of the target object.

3. The method according to claim 1, characterized in that The lane topology recognition model trained in advance is used to perform lane topology recognition on the image features to obtain lane topology features, including: Obtaining a current query vector and a historical query vector of the lane topology recognition model; Using a temporal attention network included in the lane topology recognition model, the current query vector and the historical query vector are operated to obtain an updated current query vector; Using a spatial attention network included in the lane topology recognition model, operating on the once-updated current query vector and the image features to obtain a second-updated current query vector; The lane topology feature is generated based on the secondarily updated current query vector.

4. The method according to claim 3, characterized in that The method of utilizing the temporal attention network included in the lane topology recognition model to operate on the current query vector and the historical query vector to obtain an updated current query vector includes: Utilizing a multi-head self-attention sub-network included in the temporal attention network, the current query vector and the historical query vector are operated to obtain a first lane query vector including lane topology temporal information; Using the first graph convolutional subnetwork included in the temporal attention network, operating on the current query vector to obtain a second lane query vector containing topological information between lanes; The first lane query vector and the second lane query vector are fused to obtain the updated current query vector.

5. The method according to claim 3, characterized in that The utilizing the spatial attention network included in the lane topology recognition model to operate on the once-updated current query vector and the image features to obtain a second-updated current query vector includes: Using the deformable attention subnetwork included in the spatial attention network, the operation is performed on the image features and the updated current query vector to obtain a third lane query vector; Using the second graph convolutional subnetwork included in the spatial attention network, operating on the image features and the updated current query vector to obtain a fourth lane query vector; The third lane query vector and the fourth lane query vector are fused to obtain the second-updated current query vector.

6. The method according to claim 1, characterized in that Generating lane topology information representing lane positions and topological relationships between lanes based on the lane topology features includes: Decoding the lane topology features to obtain a lane line coordinate set and a lane representative point set corresponding to the target lane, wherein each lane representative point in the lane representative point set is a representative point on another lane having a topological relationship with the target lane; The lane line coordinate set and the lane representative point set are merged, and the merged point set is determined as the lane topology information.

7. The method according to claim 1, characterized in that The image feature extraction model is pre-trained according to the following steps: Acquire a sample image and corresponding annotation information, wherein the annotation information is used to characterize the real position and real type of the target object in the sample image; Using a preset initial image feature extraction model, performing object feature detection on the sample image to obtain sample image features containing features of the target object in the sample image; Using a preset supervisory head network, performing an object classification operation on the sample image features to obtain prediction information of the target object, wherein the prediction information is used to characterize the predicted position and predicted type of the target object; determining an error between the predicted information and the labeled information, and adjusting parameters of the initial image feature extraction model based on the error; In response to the initial image feature extraction model meeting a preset training end condition, the current initial image feature extraction model is determined as a trained image feature extraction model.

8. A lane topology information generating device, characterized in that: The device comprises: An acquisition module, used to acquire images taken by a camera on a vehicle; An extraction module, configured to perform feature extraction on the image using a pre-trained image feature extraction model to obtain image features containing features of a target object in the image; a recognition module, configured to perform lane topology recognition on the image features using a pre-trained lane topology recognition model to obtain lane topology features; A generation module is used to generate lane topology information representing lane positions and topological relationships between lanes based on the lane topology features.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program stored in the memory, and when the computer program is executed, the lane topology information generating method described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the lane topology information generating method described in any one of claims 1 to 7 is implemented.

Images