Lane map construction method, device and storage medium based on large language model

Abstract

The present invention discloses a lane map construction method, device, and storage medium based on a large language model, comprising: collecting lane lines and a surrounding image in three-dimensional space; projecting the lane lines in the three-dimensional space onto a two-dimensional image plane; converting the lane lines on the two-dimensional image into regions to obtain region boundaries; extracting image features from the surrounding image to obtain image features; extracting text features from the region boundaries based on a prompt corpus to obtain text features; projecting the image features into a text feature space, concatenating them with the text features, and adding position coding information to obtain processed features; inputting the processed features into a large language model to generate a determination result for the corresponding region type; repeating the above steps to obtain the lane type between each group of adjacent lane lines; and constructing a lane map based on the lane line connectivity and the lane types between adjacent lane lines. The present invention can accurately classify lane types in different scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving technology, and specifically relates to a lane map construction method, device and storage medium based on a multimodal large language model. Background Art

[0002] Lane maps are crucial for autonomous driving, especially in complex urban environments. However, large-scale map construction requires high manual annotation and maintenance costs, making it difficult to manually build maps for a wide range of business scenarios.

[0003] Currently, a commonly used method in the industry is to collect road environment data through sensors such as LiDAR and cameras, extract lane line information from point clouds, images and other data based on different algorithms or models, use lane lines that have been verified multiple times as boundaries, and classify area categories directly in three-dimensional space or after projecting into two-dimensional space to achieve automated mapping.

[0004] Compared with manual annotation, the advantages of automated mapping are:

[0005] 1. It can significantly improve efficiency. By leveraging powerful computing power and efficient algorithms, annotation time is greatly shortened. Especially when processing large-scale data, the speed is far faster than manual annotation.

[0006] 2. Reduce the need for a large number of labeling personnel and reduce labor costs;

[0007] 3. Maps can be constructed according to preset standards and rules to ensure consistency and standardization of results and avoid subjective differences caused by human factors;

[0008] 4. It has a certain degree of real-timeness. In certain specific scenarios, it can realize real-time or near real-time data processing to generate the latest map information.

[0009] The paper "Automatic Construction of Lane-level HD Maps for Urban Scenes" proposes a fully automated process for constructing lane-level HD maps from raw sensor data. This process leverages data from multiple sensors (such as lidar and cameras) and improves map accuracy and robustness through multi-sensor fusion technology. Specifically, a deep learning model is used to extract lane lines and area information from images. High-precision point clouds are generated from lidar data, and point cloud registration and filtering are performed. Finally, data from different sensors are aligned to the same coordinate system to ensure map consistency.

[0010] However, the above method has a flaw. The extraction of lane lines and region classification in images rely on the segmentation model DeepLab-v3+ built by convolutional neural networks. Its generalization ability is very limited. When the input data form or scene changes, the effect of the model will be greatly reduced. Figure 1 Generally, it is necessary to cover many different types of scenarios, which is difficult to meet actual usage needs. Summary of the Invention

[0011] In order to solve the above technical problems, the present invention proposes a lane map construction method, device and storage medium based on a large language model.

[0012] In order to achieve the above object, the technical solution of the present invention is as follows:

[0013] In a first aspect, the present invention discloses a lane map construction method based on a large language model, comprising:

[0014] Step S1: Collect lane lines L in three-dimensional space 3D And panoramic image I img ;

[0015] Step S2: The lane line L in the three-dimensional space 3D Projected onto the two-dimensional image plane, the lane line L on the two-dimensional image is obtained 2D ;

[0016] Step S3: The lane line L on the two-dimensional image 2D Convert to regional form and get the regional boundary r k ;

[0017] Step S4: Surrounding image I img Perform image feature extraction to obtain image feature F img ;

[0018] Based on the prompt corpus, the region boundary r k Perform text feature extraction to obtain text feature F txt ;

[0019] Step S5: Image feature F img Projected to the text feature space, the text feature F is obtained img ';

[0020] In the text feature space, the text feature F txt With text feature F img 'After splicing, add position coding information to obtain processed features;

[0021] The processed features are input into the large language model to generate the corresponding region type judgment result;

[0022] Step S6: Repeat steps S1 to S5 to obtain the lane type between each set of adjacent lane lines;

[0023] Step S7: Construct a lane map based on the connectivity of lane lines and the lane types between adjacent lane lines.

[0024] On the basis of the above technical solution, the following improvements can be made:

[0025] As a preferred solution, step S2 includes:

[0026] Step S2.1: Obtain camera intrinsic parameter C intr , camera external parameters C extr , distortion parameter C dist And the pose parameter P;

[0027] Step S2.2: Set the pose parameters P and camera external parameters C extr Convert to quaternion form respectively;

[0028] Step S2.3: The lane line L in the three-dimensional space is converted into 3D Projection onto a two-dimensional image plane;

[0029]

[0030] in: Represents matrix multiplication;

[0031] T represents matrix transpose;

[0032] Step S2.4: Use camera intrinsic reference C intr , distortion parameter C dist Lane line L 2D The two-dimensional coordinates are corrected.

[0033] As a preferred solution, the position coding information in step S5 is calculated based on the length N and feature dimension D of the concatenated serialized features.

[0034] As a preferred solution, step S7 includes:

[0035] Step S7.1: Reconstruct the data into a graph, where nodes in the graph include adjacent lane lines and corresponding lane categories, and edges between nodes are determined based on the connectivity of the lane lines.

[0036] Step S7.2: Generate a high-precision map with lane lines and lane types in a three-dimensional spatial coordinate system based on the nodes and edges.

[0037] In a second aspect, the present invention discloses a lane map construction device based on a large language model, comprising:

[0038] Acquisition module, used to collect lane lines L in three-dimensional space 3D And panoramic image I img ;

[0039] The first projection module is used to project the lane line L in the three-dimensional space 3D Projected onto the two-dimensional image plane, the lane line L on the two-dimensional image is obtained 2D ;

[0040] The area conversion module is used to convert the lane line L on the two-dimensional image 2D Convert to regional form and get the regional boundary r k ;

[0041] Image feature extraction module, used for surrounding image I img Perform image feature extraction to obtain image feature F img ;

[0042] The text feature extraction module is used to extract the region boundary r based on the prompt corpus. k Perform text feature extraction to obtain text feature F txt ;

[0043] The second projection module is used to transform the image feature F img Projected to the text feature space, the text feature F is obtained img ';

[0044] The processing module is used to transform the text feature F into txt With text feature F img 'After splicing, add position coding information to obtain processed features;

[0045] The type determination module is used to input the processed features into the large language model to generate the determination result of the corresponding region type;

[0046] a repeated execution module, configured to sequentially and repeatedly execute the methods in the acquisition module, the first projection module, the region conversion module, the image feature extraction module, the text feature extraction module, the second projection module, the processing module, and the type determination module to obtain the lane type between each set of adjacent lane lines;

[0047] The lane map construction module is used to construct a lane map based on the connectivity of lane lines and the lane types between adjacent lane lines.

[0048] As a preferred solution, the first projection module includes:

[0049] Parameter acquisition unit, used to obtain the camera internal parameter C intr , camera external parameters C extr , distortion parameter C dist And the pose parameter P;

[0050] Parameter conversion unit, used to convert pose parameters P, camera external parameters C extr Convert to quaternion form respectively;

[0051] The projection unit is used to project the lane line L in the three-dimensional space according to the following formula 3D Projection onto a two-dimensional image plane;

[0052]

[0053] in: Represents matrix multiplication;

[0054] T represents matrix transpose;

[0055] Calibration unit, used to use the camera intrinsic reference C intr , distortion parameter C dist Lane line L 2D The two-dimensional coordinates are corrected.

[0056] As a preferred solution, the position coding information in the processing module is calculated based on the length N and feature dimension D of the spliced ​​serialized features.

[0057] As a preferred solution, the lane map construction module includes:

[0058] The node and edge determination unit is used to reconstruct the data into a graph. The nodes in the graph contain adjacent lane lines and corresponding lane categories, and the edges between nodes are determined based on the connectivity of the lane lines.

[0059] The high-precision map generation unit is used to generate a high-precision map with lane lines and lane types in a three-dimensional spatial coordinate system based on nodes and edges.

[0060] In a third aspect, the present invention further discloses a storage medium storing one or more computer-readable programs, wherein the one or more programs include instructions suitable for being loaded by a memory and executing any of the above-mentioned lane map construction methods based on a large language model.

[0061] The present invention provides a lane map construction method, device, and storage medium based on a large language model, which have the following beneficial effects:

[0062] This invention uses a multimodal large language model to process images and lane information, enabling accurate classification of lane types in different scenarios. This largely avoids generalization issues caused by data discrepancies during the mapping process, allowing for more flexible application in various driving environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0064] Figure 1 Flowchart of a lane map construction method provided by an embodiment of the present invention.

[0065] Figure 2 A schematic diagram of region boundary processing provided by an embodiment of the present invention;

[0066] (a) is the original image of lane lines;

[0067] (b) is a schematic diagram of the regional boundary;

[0068] (c) is a schematic diagram of the region boundary after downsampling;

[0069] (d) is a schematic diagram of the region boundary after downsampling to 10 points.

[0070] Figure 3 A schematic diagram of the feature coding and region identification framework provided by an embodiment of the present invention.

[0071] Figure 4 A schematic diagram of a lane map construction device provided in an embodiment of the present invention.

[0072] Figure 5 A schematic diagram of a computing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0074] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0075] The use of ordinal numbers "first," "second," "third," etc. to describe common objects merely indicates that different instances of similar objects are involved and is not intended to imply that the objects so described must have a given order in time, space, ranking, or in any other manner.

[0076] In addition, the expression of “comprising” an element is an “open” expression, which merely means that corresponding components or steps exist, and should not be interpreted as excluding additional components or steps.

[0077] In order to achieve the purpose of the present invention, some embodiments of a lane map construction method, device and storage medium based on a large language model, such as Figure 1 As shown, the lane map construction method includes:

[0078] Step S101: Collect lane lines L in three-dimensional space 3D And panoramic image I img ;

[0079] Step S102: The lane line L in the three-dimensional space 3D Projected onto the two-dimensional image plane, the lane line L on the two-dimensional image is obtained 2D ;

[0080] Step S103: The lane line L on the two-dimensional image 2D Convert to regional form and get the regional boundary r k ;

[0081] Step S104: Surrounding image I img Perform image feature extraction to obtain image feature F img ;

[0082] Based on the prompt corpus, the region boundary r k Perform text feature extraction to obtain text feature F txt ;

[0083] Step S105: Image feature F img Projected to the text feature space, the text feature F is obtained img ';

[0084] In the text feature space, the text feature F txt With text feature F img 'After splicing, add position coding information to obtain processed features;

[0085] The processed features are input into the large language model to generate the corresponding region type judgment result;

[0086] Step S106: Repeat steps S101 to S105 to obtain the lane type between each set of adjacent lane lines;

[0087] Step S107: Construct a lane map based on the connectivity of lane lines and the lane types between adjacent lane lines.

[0088] To facilitate understanding of this embodiment, each step is described in detail below.

[0089] Step S101 is a data collection step. The collected data includes: lane lines L in three-dimensional space 3D and panoramic image I img Among them, the three-dimensional lane line L 3D It can be obtained through multimodal data extraction and fusion.

[0090] Step S102 is a spatial projection step. 3D Projection to the two-dimensional image plane, L 3D A point set in three-dimensional space, consisting of the x, y, and z coordinates in the world coordinate system. An additional dimension is required to aid in projection transformation. Because different onboard sensors have different exposure times, projection results can shift if the vehicle is in motion. Therefore, motion compensation is required during projection using the vehicle's pose parameters.

[0091] Specifically, step S102 includes:

[0092] Step S102.1: Obtain camera intrinsic parameter C intr , camera external parameters C extr , distortion parameter C dist And the pose parameter P;

[0093] Step S102.2: Convert the pose parameter P into quaternion form, including the rotation matrix and translation vector;

[0094] The camera external parameter C extr Also converted to quaternion form;

[0095] Step S102.3: The lane line L in the three-dimensional space is converted into 3D Projection onto a two-dimensional image plane;

[0096]

[0097] in: Represents matrix multiplication;

[0098] T represents matrix transpose;

[0099] Step S102.4: Use camera intrinsic reference C intr , distortion parameter C dist Lane line L 2D For details, please refer to the cv2.projectPoints projection function in the computer vision library OpenCV.

[0100] Step S103 is a boundary processing step. 2DConvert to region form. Compared with two independent boundary lines, closed regions are more conducive to image understanding.

[0101] Specifically, for two adjacent lane lines l i and l j , where: l i ,l j ∈L 2D ,like Figure 2 As shown in (a),

[0102] Step S103 includes the following contents:

[0103] First, the adjacent lane line l i and l j Interpolate the two ends of the region to obtain a closed region boundary r k ,like Figure 2 As shown in (b);

[0104] Then, downsampling is performed according to a certain distance threshold, such as using the Douglas-Peucker Algorithm to reduce the region boundary r k The number of points in a point set, such as Figure 2 As shown in (c);

[0105] Finally, all region boundaries in the image are downsampled to a fixed number of n points, for example, n = 10, as shown in Figure 2 As shown in (d).

[0106] Step S104 is a feature encoding step. Figure 3 As shown, this step can use the visual encoder E on the computer device vis and text encoder E txt To achieve this. Visual encoder E vis An image feature extractor of any architecture can be used, such as ResNet based on convolutional neural network or ViT based on Transformer neural network, to extract the input panoramic image I img Converted into image features F img . Text Encoder E txt It consists of a word segmenter based on Byte Pair Encoder and a linear layer, which converts the region boundary r k After normalization and combination with specific prompt corpus, it is input into the text encoder E txt Converted into text features F txt .

[0107] The prompt corpus refers to the text input provided to the model when training or using a large model. It can be any form of text such as questions and instructions. In this embodiment, it can be a description and requirement of the task.

[0108] Step S105 is the area recognition step. Figure 3 As shown, this step can be implemented using a modality adapter Φ, a position encoder Θ and a large language model LLM on a computer device.

[0109] Image feature F img and text features F txt With high inter-modality differences, the image features F are converted to img Projected to the text feature space, we can get F that can be directly embedded in the text features. img ', which can be implemented through a lightweight multi-layer perceptron (MLP) or a cross-attention layer (CrossAttention).

[0110] The F in the text feature space txt and F img After concatenation, position encoding information is added. The position encoding information is calculated by the position encoder Θ based on the length N and feature dimension D of the concatenated serialized features. For the specific calculation method, refer to Rotary Position Embedding.

[0111] The features are fed into the large language model (LLM) to generate the type determination result o corresponding to the input region r through autoregression. The formula is as follows:

[0112] o=LLM((F txt ,Φ(F img ))⊕Θ(N,D));

[0113] Among them, ⊕ represents the element-by-element addition operation.

[0114] The above-mentioned large language model LLM is composed of multiple layers of Transformer neural networks with the same structure.

[0115] The output feature size of the large language model (LLM) is (L, D), where L is the sequence length and D is the feature dimension. The output feature passes through a linear layer to map the feature dimension D to M, where M corresponds to the number of characters in the vocabulary. Similar to the classification model, softmax is used to obtain the logits corresponding to each feature vector, and finally the character corresponding to the feature is indexed based on the logits and the vocabulary.

[0116] Therefore, the large language model LLM of the present invention is finally equipped with a linear layer and a vocabulary to map the output features into a string form.

[0117] Step S107 is the map construction step. After the above steps, the lane type between each adjacent lane line can be obtained, and the data is reconstructed into the form of a graph, which consists of nodes and edges.

[0118] Specifically, step S107 includes:

[0119] Step S107.1: Reconstruct the data into a graph, where nodes in the graph include adjacent lane lines and corresponding lane categories, and edges between nodes are determined based on the connectivity of the lane lines.

[0120] Step S107.2: Generate a high-precision map with lane lines and lane types in a three-dimensional spatial coordinate system based on the nodes and edges.

[0121] In addition, in some embodiments, the present invention discloses a lane map construction device based on a large language model, such as Figure 4 As shown, including:

[0122] The acquisition module 201 is used to acquire lane lines L in three-dimensional space. 3D And panoramic image I img ;

[0123] The first projection module 202 is used to project the lane line L in the three-dimensional space 3D Projected onto the two-dimensional image plane, the lane line L on the two-dimensional image is obtained 2D ;

[0124] The area conversion module 203 is used to convert the lane line L on the two-dimensional image 2D Convert to regional form and get the regional boundary r k ;

[0125] Image feature extraction module 204, for the surrounding image I img Perform image feature extraction to obtain image feature F img ;

[0126] The text feature extraction module 205 is used to extract the region boundary r based on the prompt corpus. k Perform text feature extraction to obtain text feature F txt ;

[0127] The second projection module 206 is used to transform the image feature F img Projected to the text feature space, the text feature F is obtained img ';

[0128] The processing module 207 is used to convert the text feature F txt With text feature F img'After splicing, add position coding information to obtain processed features;

[0129] Type determination module 208, used to input the processed features into the large language model to generate a determination result of the corresponding region type;

[0130] A repeated execution module 209 is configured to repeatedly execute the methods in the acquisition module, the first projection module, the region conversion module, the image feature extraction module, the text feature extraction module, the second projection module, the processing module, and the type determination module in sequence to obtain the lane type between each set of adjacent lane lines;

[0131] The lane map construction module 210 is used to construct a lane map based on the connectivity of lane lines and the lane types between adjacent lane lines.

[0132] Furthermore, the first projection module includes:

[0133] Parameter acquisition unit, used to obtain the camera internal parameter C intr , camera external parameters C extr , distortion parameter C dist And the pose parameter P;

[0134] Parameter conversion unit, used to convert pose parameters P, camera external parameters C extr Convert to quaternion form respectively;

[0135] The projection unit is used to project the lane line L in the three-dimensional space according to the following formula 3D Projection onto a two-dimensional image plane;

[0136]

[0137] in: Represents matrix multiplication;

[0138] T represents matrix transpose;

[0139] Calibration unit, used to use the camera intrinsic reference C intr , distortion parameter C dist Lane line L 2D The two-dimensional coordinates are corrected.

[0140] Furthermore, the position encoding information in the processing module is calculated based on the length N and feature dimension D of the concatenated serialized features.

[0141] Furthermore, the lane map building module includes:

[0142] The node and edge determination unit is used to reconstruct the data into a graph. The nodes in the graph contain adjacent lane lines and corresponding lane categories, and the edges between nodes are determined based on the connectivity of the lane lines.

[0143] The high-precision map generation unit is used to generate a high-precision map with lane lines and lane types in a three-dimensional spatial coordinate system based on nodes and edges.

[0144] Furthermore, it should be noted that the lane map construction device based on the large language model provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when constructing the lane map. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the lane map construction device based on the large language model can be divided into different functional modules to complete all or part of the functions described above.

[0145] In addition, the lane map construction device based on a large language model and the lane map construction method based on a large language model provided in the above embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0146] In some embodiments, as Figure 5 As shown, the present invention also discloses a computing device, including:

[0147] One or more processors 301;

[0148] Memory 302;

[0149] and one or more programs, wherein the one or more programs are stored in the memory 302 and configured to be executed by the one or more processors 301 , and the one or more programs include instructions for the lane map construction method based on the large language model disclosed in the above embodiment.

[0150] The processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 301 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0151] Memory 302 may include one or more computer-readable storage media, which may be non-transitory. Memory 302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in memory 302 is used to store at least one instruction, which is executed by processor 301 to implement the lane map construction method based on a large language model provided in the method embodiment of the present invention.

[0152] In addition, the computing device may optionally include a peripheral device interface and at least one peripheral device. The processor 301, memory 302, and peripheral device interface may be connected via a bus or signal lines. Each peripheral device may be connected to the peripheral device interface via a bus, signal lines, or circuit boards. Illustratively, the peripheral devices include, but are not limited to, radio frequency circuitry, a touchscreen display, audio circuitry, and a power supply.

[0153] Of course, the computing device may also include fewer or more components, which is not limited in this embodiment.

[0154] In addition, an embodiment of the present invention further discloses a storage medium storing one or more computer-readable programs, wherein the one or more programs include instructions suitable for being loaded by a memory and executing the lane map construction method based on a large language model disclosed in the above embodiment.

[0155] The present invention provides a lane map construction method, device, and storage medium based on a large language model, which have the following beneficial effects:

[0156] This invention uses a multimodal large language model to process images and lane information, enabling accurate classification of lane types in different scenarios. This largely avoids generalization issues caused by data discrepancies during the mapping process, allowing for more flexible application in various driving environments.

[0157] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which shall fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A lane map construction method based on a large language model, characterized in that: include: Step S1: Collect lane lines L in three-dimensional space 3D And panoramic image I img ; Step S2: The lane line L in the three-dimensional space 3D Projected onto the two-dimensional image plane, the lane line L on the two-dimensional image is obtained 2D ; Step S3: The lane line L on the two-dimensional image 2D Convert to regional form and get the regional boundary r k ; Step S4: Surrounding image I img Perform image feature extraction to obtain image feature F img ; Based on the prompt corpus, the region boundary r k Perform text feature extraction to obtain text feature F txt ; Step S5: Image feature F img Projected to the text feature space, the text feature F is obtained img '; In the text feature space, the text feature F txt With text feature F img 'After splicing, add position coding information to obtain processed features; The processed features are input into the large language model to generate the corresponding region type judgment result; Step S6: Repeat steps S1 to S5 to obtain the lane type between each set of adjacent lane lines; Step S7: Construct a lane map based on the connectivity of lane lines and the lane types between adjacent lane lines.

2. The lane map construction method according to claim 1, characterized in that: The step S2 comprises: Step S2.1: Obtain camera intrinsic parameter C intr , camera external parameters C extr , distortion parameter C dist And the pose parameter P; Step S2.2: Set the pose parameters P and camera external parameters C extr Convert to quaternion form respectively; Step S2.3: The lane line L in the three-dimensional space is converted into 3D Projection onto a two-dimensional image plane; in: Represents matrix multiplication; T represents matrix transpose; Step S2.4: Use camera intrinsic reference C intr , distortion parameter C dist Lane line L 2D The two-dimensional coordinates are corrected.

3. The lane map construction method according to claim 1, characterized in that: The position coding information in step S5 is calculated based on the length N and feature dimension D of the concatenated serialized features.

4. The lane map construction method according to claim 1, characterized in that: The step S7 comprises: Step S7.1: Reconstruct the data into a graph, where nodes in the graph include adjacent lane lines and corresponding lane categories, and edges between nodes are determined based on the connectivity of the lane lines. Step S7.2: Generate a high-precision map with lane lines and lane types in a three-dimensional spatial coordinate system based on the nodes and edges.

5. Lane map construction device based on large language model, characterized by: include: Acquisition module, used to collect lane lines L in three-dimensional space 3D And panoramic image I img ; The first projection module is used to project the lane line L in the three-dimensional space 3D Projected onto the two-dimensional image plane, the lane line L on the two-dimensional image is obtained 2D ; The area conversion module is used to convert the lane line L on the two-dimensional image 2D Convert to regional form and get the regional boundary r k ; Image feature extraction module, used for surrounding image I img Perform image feature extraction to obtain image feature F img ; The text feature extraction module is used to extract the region boundary r based on the prompt corpus. k Perform text feature extraction to obtain text feature F txt ; The second projection module is used to transform the image feature F img Projected to the text feature space, the text feature F is obtained img '; The processing module is used to transform the text feature F into txt With text feature F img 'After splicing, add position coding information to obtain processed features; The type determination module is used to input the processed features into the large language model to generate the determination result of the corresponding region type; a repeated execution module, configured to sequentially and repeatedly execute the methods in the acquisition module, the first projection module, the region conversion module, the image feature extraction module, the text feature extraction module, the second projection module, the processing module, and the type determination module to obtain the lane type between each set of adjacent lane lines; The lane map construction module is used to construct a lane map based on the connectivity of lane lines and the lane types between adjacent lane lines.

6. The lane map construction device according to claim 5, characterized in that: The first projection module includes: Parameter acquisition unit, used to obtain the camera internal parameter C intr , camera external parameters C extr , distortion parameter C dist And the pose parameter P; Parameter conversion unit, used to convert pose parameters P, camera external parameters C extr Convert to quaternion form respectively; The projection unit is used to project the lane line L in the three-dimensional space according to the following formula 3D Projection onto a two-dimensional image plane; in: Represents matrix multiplication; T represents matrix transpose; Calibration unit, used to use the camera intrinsic reference C intr , distortion parameter C dist Lane line L 2D The two-dimensional coordinates are corrected.

7. The lane map construction method according to claim 5, characterized in that: The position coding information in the processing module is calculated based on the length N and feature dimension D of the spliced ​​serialized features.

8. The lane map construction device according to claim 5, characterized in that: The lane map construction module includes: The node and edge determination unit is used to reconstruct the data into a graph. The nodes in the graph contain adjacent lane lines and corresponding lane categories, and the edges between nodes are determined based on the connectivity of the lane lines. The high-precision map generation unit is used to generate a high-precision map with lane lines and lane types in a three-dimensional spatial coordinate system based on nodes and edges.

9. A storage medium, characterized in that The storage medium stores one or more computer-readable programs, and the one or more programs include instructions, which are suitable for being loaded by the memory and executing the lane map construction method based on the large language model described in any one of claims 1-4.

Images