Lane line generation method and device, equipment, medium and program product

Abstract

The embodiment of the invention provides a lane line generation method and device, equipment, a medium and a program product. The method comprises the following steps: acquiring a satellite image and a crowd-source vector diagram of a target area, aligning the satellite image and the crowd-source vector diagram in a space dimension and / or a time dimension, and processing the satellite image and the crowd-source vector diagram through a trained target detection model to output a lane line. Wherein the crowd-source vector diagram comprises lane line vector data generated based on a crowd-source image, the crowd-source image is a road close-shot image shot in a target area, the image comprises rich lane line information, the lane line information has dynamic timeliness, and the satellite image can provide a wide-area view angle of a lane line. And feature extraction and fusion are performed on the multi-modal information through the target detection model, so that manual intervention can be reduced, the lane line generation efficiency can be improved, the limitation of a single data source can be overcome, and the precision and comprehensiveness of the lane line can be enhanced.

Description

Technical Field

[0001] The present application relates to the field of map technology, and in particular to a lane line generation method, device, equipment, medium and program product. Background Art

[0002] With the increasing popularity of navigation applications, people are demanding higher and higher accuracy maps. Lane lines are an important component of maps, and accurately identifying and modeling lane line boundaries in road environments is key to generating high-precision maps.

[0003] Currently, lane line generation mainly relies on manual drawing, requiring professional surveyors to use sensors to mark the boundaries of lane lines along the road. This method is inefficient and cannot meet the needs of large-scale map updates. Summary of the Invention

[0004] The embodiments of the present application provide a lane line generation method, apparatus, device, medium, and program product to improve the efficiency of lane line generation.

[0005] In a first aspect, an embodiment of the present application provides a lane line generation method, comprising:

[0006] Obtaining a satellite image and a crowdsource vector map of a target area, wherein the crowdsource vector map includes lane line vector data generated based on the crowdsource image, wherein the crowdsource image is an image captured along a road within the target area by an image acquisition device;

[0007] The satellite image and the crowd-source vector map are aligned in a spatial dimension and / or a temporal dimension, and the satellite image and the crowd-source vector map are processed by a trained object detection model to output the processed lane lines.

[0008] In one possible implementation, the target detection model includes an image processing branch and a vector processing branch, and processing the satellite image and the crowd-source vector map using the trained target detection model includes:

[0009] Extracting features from the satellite image using the image processing branch to obtain visual features, wherein the visual features include a multi-scale feature map;

[0010] Performing feature extraction on the lane line vector data in the multi-source vector map by the vector processing branch to obtain vector features, wherein the vector features represent position information of the lane lines in the target area;

[0011] The visual features and the vector features are fused.

[0012] In a possible implementation, fusing the visual features and the vector features includes:

[0013] For the visual features and the vector features, respectively generate corresponding spatial attention weights;

[0014] The visual features and the vector features are weightedly fused based on the spatial attention weights.

[0015] In a possible implementation, fusing the visual features and the vector features includes:

[0016] assigning fusion weights to the vector features and the visual features respectively based on the clarity of the crowd-source image and the satellite image;

[0017] Identifying whether there are missing portions of lane markings in the crowd-source vector map and / or the satellite image;

[0018] If there is no missing part, fusing the vector feature and the visual feature based on their respective corresponding fusion weights;

[0019] If there is a missing part, the fusion weights of the vector features and the visual features corresponding to the missing part are corrected, and then the vector features and the visual features are fused based on the corrected fusion weights.

[0020] In one possible implementation, the lane line vector data includes coordinate data of multiple lane line instances. Before the vector processing branch performs feature extraction on the lane line vector data in the multi-source vector map, the method further includes:

[0021] Based on the difference between the coordinate data of different lane line instances, the coordinate data of at least one lane line instance in the lane line vector data is removed.

[0022] In one possible implementation, based on differences between coordinate data of different lane line instances, removing coordinate data of at least one lane line instance in the lane line vector data includes:

[0023] For the coordinate data of any lane line instance, calculating a normal vector of a straight line fitted to the coordinate data;

[0024] The angle between the normal vectors corresponding to each instance is calculated, and the variance of each instance is calculated based on the angle, and the coordinate data of the lane line instances with variance greater than or equal to a preset threshold are removed.

[0025] In a possible implementation, before performing feature extraction on the lane line vector data by the vector processing branch, the method further includes:

[0026] The lane line vector data is rasterized into a mask matrix aligned with the satellite image in a preset reference system.

[0027] In one possible implementation, aligning the satellite image and the crowd-source vector map in a spatial dimension and / or a temporal dimension includes:

[0028] Determining a spatial range of the lane lines represented by the crowd-source vector map based on boundary values ​​of the lane line vector data, and cropping the satellite image based on the spatial range, so that the cropped satellite image is aligned with the crowd-source vector map in a spatial dimension;

[0029] and / or,

[0030] A time range is determined according to the shooting time of the multi-source image corresponding to the lane line vector data, and a satellite image within the time range is selected.

[0031] In a second aspect, an embodiment of the present application provides a lane line generating device, comprising:

[0032] an acquisition module for acquiring satellite images and a crowdsourced vector map of a target area, wherein the crowdsourced vector map includes lane line vector data generated based on the crowdsourced image, wherein the crowdsourced image is an image captured along a road within the target area by an image acquisition device;

[0033] A lane line processing module is used to align the satellite image and the crowd-source vector map in the spatial dimension and / or the temporal dimension, process the satellite image and the crowd-source vector map using a trained object detection model, and output the processed lane lines.

[0034] In one possible embodiment, the lane line processing module is further used to: perform feature extraction on the satellite image through the image processing branch to obtain visual features, wherein the visual features include multi-scale feature maps; perform feature extraction on the lane line vector data in the multi-source vector map through the vector processing branch to obtain vector features, wherein the vector features represent the position information of the lane lines in the target area; and fuse the visual features and the vector features.

[0035] In a possible implementation, the lane line processing module is further used to: generate corresponding spatial attention weights for the visual features and the vector features respectively; and perform weighted fusion of the visual features and the vector features based on the spatial attention weights.

[0036] In a possible implementation, the lane line processing module is further configured to: assign fusion weights to the vector features and the visual features respectively based on the clarity of the multi-source image and the satellite image;

[0037] Identifying whether there are missing portions of lane markings in the crowd-source vector map and / or the satellite image;

[0038] If there is no missing part, fusing the vector feature and the visual feature based on their respective corresponding fusion weights;

[0039] If there is a missing part, the fusion weights of the vector features and the visual features corresponding to the missing part are corrected, and then the vector features and the visual features are fused based on the corrected fusion weights.

[0040] In a possible implementation, the lane line processing module is further configured to: remove coordinate data of at least one lane line instance in the lane line vector data based on a difference between coordinate data of different lane line instances.

[0041] In one possible embodiment, the lane line processing module is also used to: calculate the normal vector of the straight line fitting the coordinate data of any lane line instance; calculate the angle between the normal vectors corresponding to each instance, and calculate the variance of each instance based on the angle, and remove the coordinate data of the lane line instance whose variance is greater than or equal to a preset threshold.

[0042] In a possible implementation, the lane line processing module is further configured to: rasterize the lane line vector data into a mask matrix aligned with the satellite image in a preset reference frame.

[0043] In one possible embodiment, the lane line processing module is also used to: determine the spatial range of the lane line represented by the crowd-source vector map based on the boundary value of the lane line vector data, and crop the satellite image based on the spatial range, and align the cropped satellite image with the crowd-source vector map in the spatial dimension; and / or determine the time range based on the shooting time of the crowd-source image corresponding to the lane line vector data, and select the satellite image within the time range.

[0044] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor;

[0045] The memory stores computer-executable instructions;

[0046] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0047] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementation methods of the first aspect.

[0048] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0049] The lane line generation method, device, equipment, medium and program product provided in the embodiments of the present application can align the satellite image and the crowd-source vector map in the spatial dimension and / or time dimension by acquiring the satellite image and crowd-source vector map of the target area, and can process the satellite image and the crowd-source vector map through a trained target detection model to output lane lines. Among them, the crowd-source vector map contains lane line vector data generated based on the crowd-source image, and the crowd-source image is a close-up image of the road taken in the target area. The image contains rich lane line information, and this lane line information has dynamic timeliness. The satellite image can provide a wide-area perspective of the lane line, and then the multi-modal information is extracted and fused through the deep learning-based target detection model. It can not only automatically output the lane line, reduce manual intervention, and improve the efficiency of lane line generation, but also overcome the limitations of a single data source and enhance the accuracy and comprehensiveness of the lane line. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0051] Figure 1 A flowchart of a lane line generation method provided in this application;

[0052] Figure 2 A crowd-sourced vector diagram provided as an example for this application;

[0053] Figure 3 A schematic diagram of a satellite image cropping process provided as an example for this application;

[0054] Figure 4 A schematic diagram of the processing flow of a target detection model provided in this application;

[0055] Figure 5 A schematic diagram of the structure of an object detection model provided as an example in this application;

[0056] Figure 6 This is a schematic diagram of an example of lane line removal provided by this application;

[0057] Figure 7 A schematic diagram of the structure of a lane line generation device provided in this application;

[0058] Figure 8This is a schematic diagram of the structure of an electronic device provided in this application.

[0059] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0060] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0061] With the increasing popularity of navigation applications, people are demanding higher and higher accuracy maps. Lane markings are a crucial component of high-precision maps. Accurate lane marking data is not only used for navigation but can also be applied to smart urban traffic management, vehicle-road collaboration in connected vehicles, and other fields.

[0062] Currently, generating lane lines for high-precision maps primarily requires capturing close-up images of the road. These images contain lane information. This process can be accomplished using on-board cameras or handheld cameras. Professional technicians then identify lane line information from the images and draw them on the map using a combination of the capture point coordinates and camera parameters. This manual acquisition of lane lines is extremely time-consuming and inefficient. In particular, complex road scenes make it difficult to capture sufficient information for high-precision lane line generation in a single capture. Consequently, multiple images must be captured, further increasing the workload for manual lane line identification and making it difficult to meet the lane line requirements for large-scale map updates.

[0063] Based on this, a technical concept is proposed: first, a camera or other image acquisition device can be used to capture several close-up images along the road. These close-up images contain information related to lane lines, and the lane line information of the same section of road in different images can be aggregated into a vector map representing the position of the lane lines. Secondly, a satellite image containing the road can be obtained, and the multimodal lane line features in the satellite image and the vector map can be combined to generate the lane lines.

[0064] The process of generating lane lines by combining multimodal lane line features can be achieved using a pre-trained model. Specifically, several vector graphs and corresponding satellite images can be obtained as training samples in the manner described above. A target detection model is trained using the vector graphs and satellite images as input and the true lane line values ​​as output. Once trained, the target detection model can automatically identify and fuse multimodal lane line features based on the input vector graphs and satellite images, and generate and output lane lines based on the fusion results.

[0065] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0066] Figure 1 A schematic diagram of the process of generating lane lines provided in this application Figure 1 ,like Figure 1 As shown, the method includes:

[0067] Step S101: Obtain satellite images and crowd-source vector maps of the target area.

[0068] The target area may be the area where lane lines are to be generated. The crowdsource vector map includes lane line vector data generated based on crowdsource images, where the crowdsource images are images captured along the road in the target area by an image acquisition device.

[0069] In the embodiments of the present application, multi-source images may refer to a number of images captured near the road using an image acquisition device. These images may come from multiple data sources, for example, they may be captured by different onboard cameras of the same vehicle as the vehicle passes through the road, or by onboard cameras of different vehicles as they pass through the road, or by a person passing through the road with a handheld camera or using a motion camera or other device.

[0070] To obtain a crowd-source vector map, lane line features in the image can be extracted based on crowd-source images using image processing algorithms to automatically generate lane line instances. Each image can generate at least one corresponding lane line instance, which can be vector data representing the position of the lane line. The lane line instances corresponding to multiple images can be summarized into a single map after unifying the reference system to obtain a crowd-source vector map containing lane line instances from multiple images.

[0071] Exemplarily, for any crowd-source image, the coordinates of the points on the lane lines in the image in the preset coordinate system can be calculated based on the coordinates of the shooting points and the internal and external parameters of the camera. The calculation process can be implemented by an algorithm. In this way, the lane lines in the image can be converted into a set of vector points in the preset coordinate system. The set of vector points of the lane lines of each image in the coordinate system can be called an instance, and the crowd-source vector map can contain multiple instances.

[0072] For example, in a crowdsourced vector map, a vector point can be recorded as (index, x, y, z, t, instanceID). Index represents the sequence number of the vector point, which can be used to distinguish different vector points in the same instance. x, y, and z represent the three-dimensional coordinates of the vector point in a preset coordinate system, t represents the capture time of the image corresponding to the instance, and instanceID represents the instance number of the vector point, which can be used to distinguish different instances. The preset coordinate system can be a Gauss-Krüger 3-zone projection coordinate system.

[0073] Figure 2 This is a crowd-source vector diagram provided as an example for this application. Figure 2 As shown in the figure, dashed lines represent lane lines, and each lane line consists of several vector points. It should be noted that each multi-source image can generate a corresponding instance, and each instance can contain one lane line or multiple lane lines. The number of lane lines corresponding to an instance is related to the amount of lane line information contained in the image corresponding to the instance. For example, if an image contains lane lines on both sides of a lane, the lane lines on both sides can be converted into vector points based on the image to form an instance.

[0074] Step S102 : Align the satellite image and the crowd-source vector map in the spatial dimension and / or the temporal dimension, process the satellite image and the crowd-source vector map using a trained object detection model, and output the processed lane lines.

[0075] In some possible implementations, aligning the satellite image and the crowd-source vector map in the spatial dimension may include: determining the spatial range of the lane line represented by the crowd-source vector map based on the boundary value of the lane line vector data, and cropping the satellite image based on the spatial range, and aligning the cropped satellite image with the crowd-source vector map in the spatial dimension.

[0076] By aligning satellite imagery and the crowdsource vector map in the spatial dimension, the information contained in the satellite imagery can overlap with the lane markings represented by the crowdsource vector map. This allows lane marking features to be verified based on multimodal data from two different sources, avoiding information omissions and distortion. By aligning satellite imagery and the crowdsource vector map in the temporal dimension, it is possible to avoid differences in lane markings from images from different sources at different times, reducing the impact of differences in information collection time on the accuracy of generated lane markings. Aligning satellite imagery and the crowdsource vector map in both spatial and temporal dimensions allows subsequent feature extraction and fusion of the model to be performed under a unified benchmark, improving the accuracy and reliability of lane feature processing.

[0077] For example, the maximum and minimum coordinate values ​​of the vector points representing the lane lines in the crowd-source vector map can be calculated to determine the spatial range of the point set, which can cover all the vector points in the crowd-source vector map. After obtaining the spatial range, it can be converted to the coordinate system of the satellite image to obtain the coordinates of the corner points of the image to be cropped. For example, the satellite image uses WGS84 coordinates, and the coordinates of the four corner points of the spatial range can be converted to WGS84 coordinates to obtain the coordinates of the four corner points of the satellite image to be cropped. The corresponding image is cut out in the satellite image according to the coordinates, so that the cut image can be aligned with the crowd-source vector map in the spatial dimension.

[0078] Figure 3 This is a schematic diagram of a satellite image cropping process provided as an example for this application. Figure 3 As shown in the figure, the satellite image before cropping contains more complex road environment information. After cropping, only the areas related to the lane lines to be generated are retained. This method can reduce the amount of information that the model needs to process and improve the efficiency of lane line generation.

[0079] In some possible implementations, aligning the satellite image and the crowd-source vector map in the time dimension may include: determining a time range based on the shooting time of the crowd-source image corresponding to the lane line vector data, and selecting a satellite image within the time range.

[0080] Exemplarily, the shooting time points of the crowd-source images corresponding to each vector point in the crowd-source vector map can be obtained, and the time range can be determined based on these shooting time points. For example, satellite images can be selected as the time range from the earliest shooting time point to 60 days ago.

[0081] After aligning the satellite image and the crowd-source vector map, the aligned image can be input into a trained object detection model. The object detection model can extract lane line features from the satellite image and the crowd-source vector map and perform fusion processing to finally generate and output lane lines.

[0082] In the above embodiment, after obtaining satellite images and a crowdsource vector map of the target area, the satellite images and the crowdsource vector map can be aligned in the spatial dimension and / or the temporal dimension, and the satellite images and the crowdsource vector map can be processed by a trained target detection model to output lane lines. The crowdsource vector map includes lane line vector data generated based on crowdsource images, and the crowdsource images are close-up images of the road taken within the target area. The images contain rich lane line information, and this lane line information is dynamic and timely. The satellite images can provide a wide-area perspective of the lane lines. The multimodal information is then extracted and fused by a deep learning-based target detection model. This not only allows for automated lane line output, reduces manual intervention, and improves lane line generation efficiency, but also overcomes the limitations of a single data source and enhances the accuracy and comprehensiveness of the lane lines.

[0083] In one embodiment, the target detection model includes an image processing branch and a vector processing branch, such as Figure 4 As shown in the figure, the satellite imagery and crowdsourced vector maps are processed using the trained object detection model, including:

[0084] Step S401: extract features from satellite images through an image processing branch to obtain visual features.

[0085] Among them, visual features include multi-scale feature maps.

[0086] Step S402 : extracting features from the lane line vector data in the multi-source vector map through a vector processing branch to obtain vector features.

[0087] Among them, the vector feature represents the position information of the lane line in the target area.

[0088] Step S403: fusing the visual features and the vector features.

[0089] In the embodiment of the present application, the visual features are derived from satellite images, the vector features are derived from crowd-source vector maps, and the crowd-source vector maps are derived from crowd-source images, so the vector features are actually also derived from crowd-source images.

[0090] For the fusion of visual features and vector features, the weights of visual features and vector features in the fusion process can be defined from one or more perspectives. Alternatively, the visual features and vector features can be processed through a spatial attention mechanism to obtain their respective weights, which affect the contribution of visual features and vector features to the fused features.

[0091] The embodiment of the present application can define the weights of visual features and vector features in the fusion process by analyzing the characteristics of multi-source images and satellite images and the degree to which these two images retain lane line information.

[0092] For example, from the perspective of image clarity, crowdsource images are close-up images taken along the road using devices such as cameras. They typically have higher image clarity and contain richer lane line texture details. Accordingly, the lane line vector data in the crowdsource vector map generated from crowdsource images retains more actual lane line information. However, satellite images and crowdsource images have different clarity, so weights can be assigned based on image clarity. For example, if crowdsource images have higher clarity than satellite images, and vector features are derived from crowdsource images while visual features are derived from satellite images, the weight of vector features in the fusion process can be higher than that of visual features.

[0093] For example, from the perspective of the complementarity of images from different sources, the multi-source image is a close-up lane line image, which can provide lane line details but has a narrow field of view. The satellite image has weaker details but richer global information. In areas where the two are very different (such as where the lane lines in the multi-source image are obscured), the information of the satellite image can be emphasized, and in areas where the differences are smaller, the close-up details of the multi-source image can be emphasized.

[0094] Specifically, lane markings captured by both crowdsource and satellite imagery can be obscured by obstacles. In particular, crowdsource images are limited by their close-up shooting angles, making them more susceptible to obstruction in complex road environments such as urban areas. Furthermore, due to limitations in shooting time and efficiency, even integrating images from multiple perspectives may not capture complete lane markings. To address this, the fusion weights assigned to vector features can be locally adjusted. For pixel regions where lane markings are missing due to occlusion, the weight of these regions in the vector features is reduced.

[0095] In some possible implementations, fusing the visual features and the vector features includes:

[0096] assigning fusion weights to the vector features and the visual features respectively based on the clarity of the crowd-source image and the satellite image;

[0097] Identifying whether there are missing portions of lane markings in the crowd-source vector map and / or the satellite image;

[0098] If there is no missing part, fusing the vector feature and the visual feature based on their respective corresponding fusion weights;

[0099] If there is a missing part, the fusion weights of the vector features and the visual features corresponding to the missing part are corrected, and then the vector features and the visual features are fused based on the corrected fusion weights.

[0100] In the above implementation, based on the clarity allocation, the weights of vector features and visual features in the feature fusion process can be modified based on whether lane lines are missing in the crowdsource vector map, crowdsource image, or satellite image. For example, if lane lines are missing in the crowdsource vector map, the pixel coordinate region corresponding to the missing portion can be determined. The weight of the vector features in this region can be reduced, while the visual features (derived from the satellite image) in this region can be given a higher weight.

[0101] In some examples, whether there is a missing part can be determined based on the coordinates of the lane line vector points in the crowd-source vector map. For example, based on the mapping relationship between the coordinate system represented by the crowd-source vector map and the coordinate system of the real physical world, the coordinates of the lane line in the target area in the real physical world mapped to the crowd-source vector map should be expressed as (X1, Y1), (X2, Y2),..., (Xn, Yn). However, the crowd-source vector map actually obtained through the crowd-source image does not have corresponding vector points at some of its coordinates. In this case, the part where the vector points are missing can be determined as the missing part of the lane line.

[0102] The lane line vector data in a crowdsourced vector map is determined based on crowdsource images. In some other examples, image processing methods such as semantic recognition can be used to identify lane lines in crowdsource images, and the presence of missing lane lines can be determined based on the mapping relationship between the pixel coordinate system of the crowdsource images and the real-world coordinate system. Similarly, in some other examples, image processing methods such as semantic recognition can be used to identify missing lane lines in satellite images.

[0103] It should be noted that the crowd-source vector map can be obtained by integrating multiple crowd-source images. The missing lane lines in one crowd-source image can be compensated by the lane line information in other crowd-source images. For this, a judgment rule for the missing parts of the lane lines in the crowd-source vector map can be defined. This application does not limit the specific judgment rules.

[0104] Through the above method, the weights of visual features and vector features in the fusion process can be defined and corrected from one or more perspectives, fully considering the characteristics of satellite images and multi-source images and the degree to which images retain lane lines. This allows more real lane line information to be retained during the fusion process, making the fused features more reflective of lane line characteristics in the real physical world and improving the accuracy of lane lines generated by the model.

[0105] Figure 5 This is a schematic diagram of the structure of a target detection model provided by this application. Figure 5 The above embodiment is further described.

[0106] like Figure 5As shown in Figure 1, the target detection model can include three parts: Backbone (skeleton network), Neck (neck structure), and Head (detection head). The Backbone of the target detection model can adopt a dual-branch structure consisting of an image processing branch and a vector processing branch.

[0107] The image processing branch can use ResNet50 as the base network architecture to extract multi-scale features from satellite imagery. ResNet50 is a deep convolutional neural network with excellent feature extraction capabilities. In this branch, the outputs from conv1 to stage4 of ResNet50 are retained to generate a four-level feature map with strides of 4, 8, 16, and 32, respectively. This allows the capture of features at different scales in satellite imagery, providing rich visual information for subsequent processing.

[0108] The vector processing branch extracts features from the lane line vector data from the crowdsource vector map through a four-layer DenseNet block. The number of channels can be gradually increased from 64 to 512. The DenseNet block fully utilizes features from each layer, enhances feature propagation, and mitigates the vanishing gradient problem, resulting in more efficient feature extraction of the vector data. Fourier position encoding can then be injected into the features extracted by the DenseNet block. Fourier position encoding provides position information for the vector data, enhancing the model's understanding of the spatial structure of the vector data and helping to improve the accuracy of vector generation.

[0109] In some possible implementations, before performing feature extraction on the lane line vector data through the vector processing branch, the method further includes: rasterizing the lane line vector data into a mask matrix aligned with the satellite image in a preset reference frame.

[0110] For example, before extracting features from the crowdsource vector map through a four-layer DenseNet Block, the lane line vector data in the crowdsource vector map can be rasterized into a mask matrix aligned with the satellite image. This allows the vector data to be spatially aligned with the satellite image in advance, facilitating subsequent feature fusion.

[0111] In some possible implementations, fusing visual features and vector features may include: generating corresponding spatial attention weights for the visual features and vector features, respectively; and weighted fusing the visual features and vector features based on the spatial attention weights.

[0112] See also Figure 5, a dual-path spatial attention mechanism (DSA) can be used to perform cross-modal fusion of visual features and vector features in the P2-P5 layers of the FPN (Feature Pyramid Network) in ResNet50.

[0113] Specifically, spatial attention weights are generated for both visual features (from the image processing branch) and vector features (from the vector processing branch). These two generated weight matrices reflect the relative importance of different locations in their respective feature maps. Through cross-attention weighted fusion—multiplying the visual features by the vector attention weights plus the vector features by the visual attention weights—the two modal features can be complemented and integrated. This cross-attention mechanism enables the model to reference the feature distribution of the other modality while focusing on the features of one modality, thereby better mining the correlation information between the two modalities, improving the effectiveness of feature fusion, and enabling the fused features to more accurately represent the semantic information and spatial structure of the map.

[0114] The Neck part of the target detection model can use PANet (Path Aggregation Network) to achieve multi-scale feature aggregation, which can include three fusion methods: top-down path, bottom-up path and cross-modal skip connection.

[0115] Top-down path: Unify the stage 2-stage 4 features of ResNet50 into 256 channels. This eliminates the channel dimension differences between features at different stages, allowing features of different scales to interact and fuse within the same channel space, providing a unified feature foundation for the subsequent bottom-up path.

[0116] Bottom-up approach: Introducing deformable convolution to enhance geometric feature modeling. By learning the offsets of sampling points, deformable convolution can adaptively capture geometric transformations in feature maps. This overcomes the shortcomings of traditional convolution in processing map features with complex geometric structures, improves the model's ability to model elements of various shapes and orientations in the map, and thus better aggregates multi-scale features.

[0117] Cross-modal skip connections: Channel-dimensional splicing is performed before feature addition to preserve multimodal characteristics. Cross-modal skip connections are designed to preserve the unique information of satellite imagery and vector data. By performing channel-dimensional splicing before feature addition, the features of the two modalities complement and enhance each other during the fusion process, avoiding the loss of feature information that may be caused by simple addition operations. This ensures the integrity and complementarity of multimodal features, further improving the effectiveness of feature aggregation and providing richer feature representations for subsequent detection tasks.

[0118] The detection head of the object detection model can contain outputs of two task types, namely the classification task branch and the coordinate regression branch.

[0119] Classification branch: Consists of three convolutional layers, ultimately outputting class probabilities. Group normalization and Focal Loss are used as loss functions. Group normalization improves model training stability with small batches of data, while Focal Loss effectively addresses class imbalance, allowing the model to focus more on difficult-to-classify samples during training, improving classification accuracy and robustness.

[0120] Coordinate regression branch: This branch predicts the (x, y) coordinates of instance feature points using a distance loss function. This branch directly regresses the coordinates of instance features through convolution operations. The distance loss function measures the difference between the predicted and true coordinates, guiding the model to continuously optimize the accuracy of coordinate predictions, ensuring that vectorized map features are accurately positioned at the appropriate locations.

[0121] Among them, the classification branch can be used in the model training process, and the coordinate regression branch can be used to generate lane lines.

[0122] To train an object detection model, satellite imagery, crowdsourced vector maps, and the ground-truth lane markings for a specific road segment can be used as training samples. The ground-truth lane markings are used as the target output. The satellite imagery and crowdsourced vector maps in the training samples are fed into the object detection model to be trained, and the training loss is calculated by comparing the corresponding outputs with the ground-truth lane markings. A composite loss function can be used, for example, combining classification loss with regression loss to calculate the composite loss value.

[0123] Classification loss: Use Focal Loss with α=0.5 and γ=2.0. Focal Loss is a loss function designed to address class imbalance. By introducing adjustable parameters α and γ, it can reduce the weight of easily classified samples and make the model focus more on difficult-to-classify samples. When α=0.5 and γ=2.0, Focal Loss effectively balances the weight difference between positive and negative samples, allowing the model to prioritize positive samples (such as various vector features in a map) during training, improving classification accuracy and robustness and avoiding model bias caused by class imbalance.

[0124] Regression loss: L1 loss is used. L1 loss measures the absolute error between the predicted value and the true value. Compared with L2 loss, L1 loss is more robust to outliers. It enables the model to better fit the majority of normal data points during coordinate regression, reducing the impact of outliers on the regression results, thereby improving the accuracy of coordinate regression and making the position of map features generated by vectorization more accurate.

[0125] In some possible implementations, the lane line vector data includes coordinate data of multiple lane line instances. Before the vector processing branch performs feature extraction on the lane line vector data in the multi-source vector map, the method further includes:

[0126] Based on the difference between the coordinate data of different lane line instances, the coordinate data of at least one lane line instance in the lane line vector data is removed.

[0127] Specifically, for the coordinate data of any lane line instance, the normal vector of the fitting straight line of the coordinate data can be calculated; the angle between the normal vectors corresponding to each instance is calculated, and the variance of each instance is calculated based on the angle, and the coordinate data of the lane line instances whose variance is greater than or equal to a preset threshold are removed.

[0128] Figure 6 This is a schematic diagram of an example of removing lane lines provided by this application. Figure 6 As shown, the left side of the arrow is the initial crowd-source vector graph, in which some instances are quite different from other instances, and the right side of the arrow is the crowd-source vector graph after removing the instances with large differences.

[0129] For example, adjacent vector data can be clustered using the nearest neighbor clustering method, and then the normal vector of the fitted straight line is calculated for each instance of the lane line vectorized from the multi-source image, the angle between the normal vectors of each instance is calculated, and instances with large variance values ​​are excluded.

[0130] In the above embodiment, before using the target detection model to process the crowd-source vector map, by removing some lane line instances with large differences in the crowd-source vector map, the interference of extreme values ​​on lane line generation can be reduced, thereby improving the accuracy of lane lines.

[0131] Figure 7 This is a schematic diagram of the structure of a lane line generating device provided in this application, such as Figure 7 As shown, the lane line generating device 700 provided in this embodiment includes:

[0132] An acquisition module 701 is configured to acquire a satellite image and a crowdsource vector map of a target area, wherein the crowdsource vector map includes lane line vector data generated based on the crowdsource image, wherein the crowdsource image is an image captured along a road within the target area by an image acquisition device;

[0133] The lane line processing module 702 is used to align the satellite image and the crowd-source vector map in the spatial dimension and / or the temporal dimension, and process the satellite image and the crowd-source vector map using a trained object detection model to output the processed lane lines.

[0134] In one possible implementation, the lane line processing module 702 is further used to: perform feature extraction on the satellite image through the image processing branch to obtain visual features, wherein the visual features include multi-scale feature maps; perform feature extraction on the lane line vector data in the multi-source vector map through the vector processing branch to obtain vector features, wherein the vector features represent the position information of the lane lines in the target area; and fuse the visual features and the vector features.

[0135] In a possible implementation, the lane line processing module 702 is further configured to: generate corresponding spatial attention weights for the visual features and the vector features, respectively; and weightedly fuse the visual features and the vector features based on the spatial attention weights.

[0136] In one possible embodiment, the lane line processing module 702 is further used to: assign fusion weights to the vector features and the visual features based on the clarity of the multi-source image and the satellite image, respectively; identify whether there are missing parts in the lane lines in the multi-source vector image and / or the satellite image; if there are no missing parts, fuse the vector features and the visual features based on their respective corresponding fusion weights; if there are missing parts, correct the fusion weights of the vector features and the visual features corresponding to the missing parts, and then fuse the vector features and the visual features based on the corrected fusion weights.

[0137] In a possible implementation, the lane line processing module 702 is further configured to: remove the coordinate data of at least one lane line instance in the lane line vector data based on the difference between the coordinate data of different lane line instances.

[0138] In one possible embodiment, the lane line processing module 702 is also used to: calculate the normal vector of the straight line fitting the coordinate data of any lane line instance; calculate the angle between the normal vectors corresponding to each instance, and calculate the variance of each instance based on the angle, and remove the coordinate data of the lane line instance whose variance is greater than or equal to a preset threshold.

[0139] In a possible implementation, the lane line processing module 702 is further configured to: rasterize the lane line vector data into a mask matrix aligned with the satellite image in a preset reference frame.

[0140] In one possible embodiment, the lane line processing module 702 is also used to: determine the spatial range of the lane line represented by the crowd-source vector map based on the boundary value of the lane line vector data, and crop the satellite image based on the spatial range, and align the cropped satellite image with the crowd-source vector map in the spatial dimension; and / or determine the time range based on the shooting time of the crowd-source image corresponding to the lane line vector data, and select the satellite image within the time range.

[0141] The lane line generation device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effects are similar, and will not be described in detail in this embodiment.

[0142] Figure 8 This is a schematic diagram of the structure of an electronic device provided by this application. Figure 8 As shown, the electronic device 80 provided in this embodiment includes: at least one processor 801 and a memory 802. Optionally, the device 80 also includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus 804.

[0143] During the specific implementation process, at least one processor 801 executes the computer-executable instructions stored in the memory 802, so that the at least one processor 801 performs the above method.

[0144] The specific implementation process of the processor 801 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0145] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0146] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0147] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0148] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0149] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0150] The readable storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0151] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0152] The division of units is merely a logical functional division; actual implementations may employ alternative divisions, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, any direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units, either through an interface, electrical, mechanical, or other means.

[0153] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0154] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0155] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present invention. The aforementioned storage medium includes various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0156] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0157] Finally, it should be noted that those skilled in the art will readily identify other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The present invention is not limited to the precise structure described above and illustrated in the accompanying drawings, and various modifications and variations may be made without departing from the scope thereof. The scope of the present invention is limited solely by the appended claims.

Claims

1. A lane line generation method, characterized in that: include: Obtaining a satellite image and a crowdsource vector map of a target area, wherein the crowdsource vector map includes lane line vector data generated based on the crowdsource image, wherein the crowdsource image is an image captured along a road within the target area by an image acquisition device; The satellite image and the crowd-source vector map are aligned in a spatial dimension and / or a temporal dimension, and the satellite image and the crowd-source vector map are processed by a trained object detection model to output the processed lane lines.

2. The method according to claim 1, characterized in that The target detection model includes an image processing branch and a vector processing branch. The processing of the satellite image and the crowd-source vector map by the trained target detection model includes: Extracting features from the satellite image using the image processing branch to obtain visual features, wherein the visual features include a multi-scale feature map; Performing feature extraction on the lane line vector data in the multi-source vector map by the vector processing branch to obtain vector features, wherein the vector features represent position information of the lane lines in the target area; The visual features and the vector features are fused.

3. The method according to claim 2, characterized in that The fusing of the visual features and the vector features includes: For the visual features and the vector features, respectively generate corresponding spatial attention weights; The visual features and the vector features are weightedly fused based on the spatial attention weights.

4. The method according to claim 2, characterized in that The fusing of the visual features and the vector features includes: assigning fusion weights to the vector features and the visual features respectively based on the clarity of the crowd-source image and the satellite image; Identifying whether there are missing portions of lane markings in the crowd-source vector map and / or the satellite image; If there is no missing part, fusing the vector feature and the visual feature based on their respective corresponding fusion weights; If there is a missing part, the fusion weights of the vector features and the visual features corresponding to the missing part are corrected, and then the vector features and the visual features are fused based on the corrected fusion weights.

5. The method according to any one of claims 2 to 4, characterized in that The lane line vector data includes coordinate data of a plurality of lane line instances. Before the vector processing branch performs feature extraction on the lane line vector data in the multi-source vector map, the method further includes: Based on the difference between the coordinate data of different lane line instances, the coordinate data of at least one lane line instance in the lane line vector data is removed.

6. The method according to claim 5, characterized in that The removing the coordinate data of at least one lane line instance in the lane line vector data based on the difference between the coordinate data of different lane line instances includes: For the coordinate data of any lane line instance, calculating a normal vector of a straight line fitted to the coordinate data; The angle between the normal vectors corresponding to each instance is calculated, and the variance of each instance is calculated based on the angle, and the coordinate data of the lane line instances with variance greater than or equal to a preset threshold are removed.

7. The method according to any one of claims 1 to 3, characterized in that The aligning the satellite image and the crowd-source vector map in a spatial dimension and / or a temporal dimension comprises: Determining a spatial range of the lane lines represented by the crowd-source vector map based on boundary values ​​of the lane line vector data, and cropping the satellite image based on the spatial range, so that the cropped satellite image is aligned with the crowd-source vector map in a spatial dimension; and / or, A time range is determined according to the shooting time of the multi-source image corresponding to the lane line vector data, and a satellite image within the time range is selected.

8. A lane line generating device, characterized in that: include: an acquisition module for acquiring satellite images and a crowdsourced vector map of a target area, wherein the crowdsourced vector map includes lane line vector data generated based on the crowdsourced image, wherein the crowdsourced image is an image captured along a road within the target area by an image acquisition device; A lane line processing module is used to align the satellite image and the crowd-source vector map in the spatial dimension and / or the temporal dimension, process the satellite image and the crowd-source vector map using a trained object detection model, and output the processed lane lines.

9. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium / computer program product, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method according to any one of claims 1 to 7; and / or, The computer program product comprises a computer program, which implements the method according to any one of claims 1 to 7 when executed by a processor.

Images