Lane line detection method

Abstract

The invention discloses a lane line detection method, and relates to the technical field of intelligent driving. The method comprises the following steps: acquiring a data sequence of a road acquired by a vehicle-mounted sensor, and performing feature processing on the data sequence to obtain a target aerial view feature map sequence; performing feature fusion on the historical frame aerial view feature map and the current frame aerial view feature map in the target aerial view feature map sequence to obtain an enhanced aerial view feature map; extracting a map feature map from a road map of a vehicle driving road, and performing feature fusion on the enhanced aerial view feature map and the map feature map to obtain a target fusion feature map; and performing lane line detection on the target fusion feature map to obtain a lane line of a vehicle driving road. The lane line detection method has better road information perception capability aiming at the complex problem of rapid change of road light and road surface conditions, so that the continuity, accuracy and stability of lane line detection are improved, and the lane line detection method is suitable for lane line detection of long-distance roads.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a lane line detection method. Background Art

[0002] Lane detection is a vital part of the vehicle's intelligent system. Its main task is to obtain data on the road the vehicle is traveling on through on-board sensors, and use technologies such as image processing and machine learning to identify lane lines, thereby providing necessary information for the vehicle's path planning and control system.

[0003] However, when collecting road data, on-board sensors are affected by light, weather and road environment, making it difficult to obtain high-quality road data, which will undoubtedly reduce the accuracy of lane line detection. To address the problem of low lane line detection accuracy, current methods mainly fuse data collected by multiple on-board sensors to improve the quality of road data. However, this method has limited improvement in the quality of road data, resulting in the need to improve lane line accuracy. Summary of the invention

[0004] The present invention provides a lane line detection method, which improves the road data quality by fusing multi-frame enhanced data of a vehicle-mounted sensor with map data, so as to solve the problem that the road data quality is not high and the lane line accuracy still needs to be improved.

[0005] In a first aspect, an embodiment of the present invention provides a lane line detection method, including:

[0006] Acquire a data sequence of a vehicle driving road collected by an on-board sensor, and perform feature processing on the data sequence to obtain a target bird's-eye view feature map sequence; wherein the data sequence arranges the collected road data according to the collection time;

[0007] Performing feature fusion on the historical frame bird's-eye view feature image and the current frame bird's-eye view feature image in the target bird's-eye view feature image sequence to obtain an enhanced bird's-eye view feature image;

[0008] Extracting a map feature map from a road map of the vehicle's driving road, and performing feature fusion on the enhanced bird's-eye view feature map and the map feature map to obtain a target fused feature map;

[0009] Perform lane line detection on the target fusion feature map to obtain the lane line of the road where the vehicle is traveling.

[0010] In a second aspect, an embodiment of the present invention provides a lane line detection method, including:

[0011] A feature extraction module is used to obtain a data sequence of a road collected by a vehicle-mounted sensor, and perform feature processing on the data sequence to obtain a target bird's-eye view feature map sequence; wherein the data sequence arranges the collected road data according to the collection time;

[0012] A feature enhancement module is used to perform feature fusion on the historical frame bird's-eye view feature image and the current frame bird's-eye view feature image in the target bird's-eye view feature image sequence to obtain an enhanced bird's-eye view feature image;

[0013] A feature fusion module is used to extract a map feature map from a road map of the vehicle's driving road, and perform feature fusion on the enhanced bird's-eye view feature map and the map feature map to obtain a target fused feature map;

[0014] The lane line detection module is used to perform lane line detection on the target fusion feature map to obtain the lane line of the road where the vehicle is traveling.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the lane line detection method described in any embodiment of the present invention.

[0019] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the lane line detection method described in any embodiment of the present invention when executed.

[0020] In a fifth aspect, an embodiment of the present invention provides a computer program product including a computer program, and when the computer program is executed by a processor, the lane line detection method described in any embodiment of the present invention is implemented.

[0021] The technical solution of the embodiment of the present invention is to obtain a target bird's-eye feature map sequence by acquiring a data sequence of a road collected by an on-board sensor and performing feature processing on the data sequence; wherein the data sequence arranges the collected road data according to the collection time; performs feature fusion on the historical frame bird's-eye feature map and the current frame bird's-eye feature map in the target bird's-eye feature map sequence to obtain an enhanced bird's-eye feature map; extracts a map feature map from a road map of the road on which the vehicle is traveling, and performs feature fusion on the enhanced bird's-eye feature map and the map feature map to obtain a target fused feature map; performs lane line detection on the target fused feature map to obtain the lane line of the road on which the vehicle is traveling. The embodiment of the present invention can fuse road features perceived at multiple times and a priori road information contained in the road map, and has better road information perception capabilities for complex problems such as changes in road light and rapid changes in road conditions, solves the problem that the quality of road data is not high, resulting in the need to improve the accuracy of lane lines, improves the continuity, accuracy and stability of lane line detection, and is suitable for lane line detection on long-distance roads.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 A flow chart of a lane line detection method provided in Embodiment 1 of the present invention;

[0025] Figure 2 A flow chart of a lane line detection method provided in Embodiment 2 of the present invention;

[0026] Figure 3 Schematic diagram of a lane line detection method provided by an embodiment of the present invention

[0027] Figure 4 A schematic diagram of the structure of a lane line detection device provided in Embodiment 3 of the present invention;

[0028] Figure 5 It is a schematic diagram of the structure of an electronic device for implementing the lane line detection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 This is a flow chart of a lane line detection method provided in Embodiment 1 of the present invention. This embodiment can be applied to detecting lane line conditions of a road on which a vehicle is traveling. The method can be executed by a lane line detection device. The lane line detection device can be implemented in the form of hardware and / or software. The lane line detection device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0033] S110, obtaining a data sequence of a vehicle driving road collected by an on-board sensor, and performing feature processing on the data sequence to obtain a target bird's-eye view feature map sequence; wherein the data sequence arranges the collected road data according to the collection time.

[0034] The vehicle-mounted sensor can be understood as a sensor installed on the vehicle, which is used to collect data on the road the vehicle is traveling on. The types of vehicle-mounted sensors may include: image sensors, laser radar sensors, or millimeter wave radar sensors. The embodiment of the present invention does not limit the number and installation location of the vehicle-mounted sensors, which can be set according to actual needs.

[0035] The data sequence can be understood as a sequence of data collected by the vehicle-mounted sensor at multiple consecutive moments arranged according to the collection time. The multiple consecutive moments can include the current moment and a preset number of consecutive historical moments before the current moment. The data sequence can contain data collected by a single vehicle-mounted sensor, or it can contain fused data collected by multiple vehicle-mounted sensors. The target bird's-eye view feature map sequence can be understood as a feature sequence extracted from the data sequence under the bird's eye view (Bird's Eye View, BEV, also known as the top view).

[0036] Specifically, at least one vehicle-mounted sensor is used to obtain data of the vehicle's driving road at multiple consecutive moments, and the data is sorted in the order of acquisition time to obtain a data sequence. The lane line detection device obtains the data sequence, and performs processing operations such as data preprocessing and data feature extraction on the data sequence to obtain a target bird's-eye view feature map sequence under a bird's-eye view perspective.

[0037] In an optional embodiment, the method for obtaining the data sequence of the vehicle driving road collected by the vehicle-mounted sensor may be: obtaining the data sequence of the vehicle driving road collected by multiple vehicle-mounted sensors of the same type arranged at different positions on the vehicle, and the vehicle-mounted sensor may be, for example, a vehicle-mounted image sensor or a vehicle-mounted radar sensor. Accordingly, the method for performing feature processing on the data sequence to obtain the target bird's-eye feature map sequence may be: performing feature processing on the data sequences collected by the vehicle-mounted sensors at different positions respectively to convert them into a bird's-eye feature map sequence, and performing feature splicing on the bird's-eye feature map sequences corresponding to the vehicle-mounted sensors at different positions to obtain the target bird's-eye feature map sequence.

[0038] In another optional embodiment, the method for obtaining the data sequence of the vehicle driving road collected by the vehicle-mounted sensor may be: obtaining the data sequence of the vehicle driving road collected by multiple different types of vehicle-mounted sensors installed on the vehicle, and the vehicle-mounted sensor may include, for example, a vehicle-mounted image sensor or a vehicle-mounted radar sensor. Accordingly, the method for performing feature processing on the data sequence to obtain the target bird's-eye feature map sequence may be: performing feature processing on the data sequences collected by different types of vehicle-mounted sensors respectively, and converting them into a bird's-eye feature map sequence. Feature splicing is performed on the bird's-eye feature map sequences corresponding to different types of vehicle-mounted sensors to obtain the target bird's-eye feature map sequence.

[0039] S120, performing feature fusion on the historical frame bird's-eye view feature map and the current frame bird's-eye view feature map in the target bird's-eye view feature map sequence to obtain an enhanced bird's-eye view feature map.

[0040] The current frame bird's-eye view feature map can be understood as the bird's-eye view feature map at the current moment, or the bird's-eye view feature map of the last frame in the target bird's-eye view feature map sequence; the historical frame bird's-eye view feature map can be understood as the bird's-eye view feature map of the consecutive frames before the current frame, or other bird's-eye view feature maps except the last frame bird's-eye view feature map in the target bird's-eye view feature map sequence. The enhanced bird's-eye view feature map can be understood as the feature map obtained after the feature enhancement of the current frame bird's-eye view feature map.

[0041] In this embodiment, the feature fusion of the historical frame bird's-eye view feature image in the target bird's-eye view feature image sequence with the current frame bird's-eye view feature image is performed to enhance the feature of the current frame bird's-eye view feature image and obtain an enhanced bird's-eye view feature image. Since the enhanced bird's-eye view feature image fuses the road features at multiple times, it has better road information perception capabilities for complex problems such as changes in road light and rapid changes in road conditions, and can improve the fluency and stability of road information perception.

[0042] S130, extracting a map feature map from a road map of the vehicle's driving road, and performing feature fusion on the enhanced bird's-eye view feature map and the map feature map to obtain a target fused feature map.

[0043] The road map may be understood as a map containing road information of the road on which the vehicle is traveling, including at least lane information. The road map may be a navigation map or a high-precision map, which is not limited in the embodiments of the present invention. The map feature map may be understood as a feature map extracted from the road map. The target fusion feature map may be understood as a feature map obtained by fusion of the enhanced bird's-eye view feature map and the map feature map.

[0044] Exemplarily, the vehicle is positioned, and a road map of the road on which the vehicle is traveling is obtained based on the vehicle positioning, and a two-dimensional convolutional layer in a convolutional neural network (CNN) is used to extract features of the road map to obtain a map feature map.

[0045] This embodiment can fully integrate the priori road information contained in the road map and the actual road information obtained by observation by fusing the enhanced bird's-eye view feature map and the map feature map, thereby improving the continuity, accuracy and stability of road perception. At the same time, the long-distance road information contained in the road map is used to compensate for the weakness that the perception performance of the on-board sensor decreases significantly with increasing distance, which can improve the long-distance perception capability of the road, thereby improving the accuracy and stability of lane line detection.

[0046] S140: Perform lane line detection on the target fusion feature map to obtain the lane line of the road on which the vehicle is traveling.

[0047] Specifically, the target fusion feature map is input into the lane line detection model, the lane line detection result output by the lane line detection model is obtained, and the lane line detection result is used as the lane line of the road where the vehicle is traveling.

[0048] The technical solution of the embodiment of the present invention is to obtain a target bird's-eye feature map sequence by acquiring a data sequence of a road collected by an on-board sensor and performing feature processing on the data sequence; wherein the data sequence arranges the collected road data according to the collection time; performs feature fusion on the historical frame bird's-eye feature map and the current frame bird's-eye feature map in the target bird's-eye feature map sequence to obtain an enhanced bird's-eye feature map; extracts a map feature map from a road map of the road on which the vehicle is traveling, and performs feature fusion on the enhanced bird's-eye feature map and the map feature map to obtain a target fused feature map; performs lane line detection on the target fused feature map to obtain the lane line of the road on which the vehicle is traveling. The embodiment of the present invention can fuse road features perceived at multiple times and a priori road information contained in the road map, and has better road information perception capabilities for complex problems such as changes in road light and rapid changes in road conditions, thereby improving the continuity, accuracy and stability of lane line detection, and is suitable for lane line detection on long-distance roads.

[0049] Embodiment 2

[0050] Figure 2 A flow chart of a lane line detection method provided in the second embodiment of the present invention, based on the above embodiment, this embodiment further limits the generation method of the target bird's-eye view feature map sequence. Specifically, the vehicle-mounted sensor includes: a vehicle-mounted image sensor and a vehicle-mounted radar sensor; the data sequence includes: an image sequence of the vehicle's driving road collected by the vehicle-mounted image sensor and a radar point cloud sequence of the vehicle's driving road collected by the vehicle-mounted radar sensor. The step of extracting features from the data sequence to obtain a target bird's-eye view feature map sequence includes: performing feature processing on the image sequence to obtain a first bird's-eye view feature map sequence; performing feature processing on the radar point cloud sequence to obtain a second bird's-eye view feature map sequence; performing feature fusion on the bird's-eye view feature maps of the same frame in the first bird's-eye view feature map sequence and the second bird's-eye view feature map sequence to obtain a target bird's-eye view feature map sequence.

[0051] like Figure 2 As shown, the method includes:

[0052] S210: Acquire an image sequence of the vehicle driving road collected by the vehicle-mounted image sensor and a radar point cloud sequence of the vehicle driving road collected by the vehicle-mounted radar sensor.

[0053] The image sequence is a sequence of images collected by the vehicle-mounted image sensor and sorted in the order of collection time. The radar point cloud sequence is a sequence of radar point clouds collected by the vehicle-mounted radar sensor and sorted in the order of collection time.

[0054] S220: Perform feature processing on the image sequence to obtain a first bird's-eye view feature image sequence.

[0055] The first bird's-eye view feature image sequence may be understood as a sequence of feature images obtained by performing feature processing on images captured by the vehicle-mounted image sensor at a bird's-eye view.

[0056] Specifically, the road image captured by the vehicle-mounted image sensor is a two-dimensional image, from which a two-dimensional feature image is extracted. However, the two-dimensional feature image does not contain depth information. Therefore, it is also necessary to estimate the depth features of the image, obtain a spatial feature image containing depth features, and then convert the spatial feature map into a feature map from a bird's-eye view to obtain a first bird's-eye view feature map sequence.

[0057] As an optional implementation of the embodiment of the present application, S220, the step of performing feature processing on the image sequence to obtain a first bird's-eye view feature image sequence includes:

[0058] S221. Perform feature extraction on the image sequence through a feature extraction network cascaded by a deep residual network and a feature pyramid network to obtain a multi-scale feature map sequence.

[0059] Among them, the multi-scale feature map sequence can be understood as a sequence consisting of multi-scale feature maps. The deep residual network (ResNet) solves the gradient vanishing problem in traditional deep neural networks by introducing residual learning and a special network structure, and has better performance. For example, ResNet50 can be selected. Feature Pyramid Network (FPN). By fusing feature information at different layers, the hierarchical information is enriched.

[0060] In this embodiment, a feature extraction network is formed by cascading a deep residual network and a feature pyramid network to extract features from an image sequence captured by a vehicle-mounted image sensor, thereby obtaining a multi-scale feature map sequence with rich semantic features in a UV coordinate system.

[0061] S222. Perform depth estimation on the multi-scale feature map sequence through a depth estimation network to obtain a spatial feature map sequence.

[0062] Among them, the depth estimation network DepthNet is used to estimate the depth information. The spatial feature map sequence can be understood as a sequence of three-dimensional feature maps containing depth information.

[0063] In this embodiment, the multi-scale feature map sequence is input into the depth estimation network to obtain a spatial feature map sequence in the UVD coordinate system, which is output by the depth estimation network for performing depth estimation on the multi-scale feature maps in the multi-scale feature map sequence. The spatial feature maps in the spatial feature map sequence can be represented in the form of a cone feature map.

[0064] S223. Perform bird's-eye view pooling on the spatial feature map sequence through a pooling layer to obtain a first bird's-eye view feature sequence.

[0065] The essence of the pooling layer is feature sampling, for example, BEVPoolv2 can be used.

[0066] In this embodiment, the spatial feature maps in the UVD coordinate system in the spatial feature map sequence are sampled to the XYZ coordinate system based on the bird's-eye view perspective grid through the pooling layer, and the sampled spatial feature maps in the XYZ coordinate system are projected onto the projection surface of the bird's-eye view perspective to obtain the first bird's-eye view feature sequence.

[0067] S230: Perform feature processing on the radar point cloud sequence to obtain a second bird's-eye view feature image sequence.

[0068] The second bird's-eye view feature map sequence may be understood as a sequence of feature maps obtained by performing feature processing on the radar point cloud collected by the vehicle-mounted radar sensor from a bird's-eye view perspective.

[0069] Specifically, the radar point cloud data collected by the vehicle-mounted radar sensor is three-dimensional data. For lane line detection, the radar point cloud data needs to be converted into a feature map from a bird's-eye view.

[0070] As an optional implementation of the embodiment of the present application, S230, the step of performing feature processing on the radar point cloud sequence to obtain a second bird's-eye view feature map sequence includes:

[0071] S231. Map each frame of radar point cloud in the radar point cloud sequence to an image coordinate system of a corresponding frame image in the image sequence according to calibration parameters of the vehicle-mounted sensor.

[0072] Among them, the vehicle-mounted sensors include vehicle-mounted image sensors and vehicle-mounted radar sensors, and the calibration parameters of the vehicle-mounted sensors include calibration internal and external parameters of the vehicle-mounted image sensors and calibration parameters of the vehicle-mounted radar sensors.

[0073] Specifically, according to the calibrated internal and external parameters of the vehicle-mounted image sensor and the calibrated parameters of the vehicle-mounted radar sensor, the spatial transformation matrix from the point cloud coordinate system corresponding to the vehicle-mounted radar to the image coordinate system corresponding to the vehicle-mounted image sensor is determined, and according to the spatial transformation matrix, each frame of radar point cloud in the radar point cloud sequence is mapped to the image coordinate system of the frame image corresponding to the image sequence.

[0074] S232. Convert the radar point clouds mapped in the image coordinate systems into pseudo images through a point cloud feature processing model, and construct a pseudo image sequence based on the pseudo images.

[0075] Among them, the pseudo image can be understood as a two-dimensional data expression similar to an image obtained by converting the three-dimensional radar point cloud. The point cloud feature processing model PointPillars is used to convert the three-dimensional coordinates output by the lidar into the form of neural network input.

[0076] Specifically, the point cloud feature processing model PointPillars is used to divide the point cloud into multiple fixed-size pillars. Each pillar contains many points, and each point can be represented by a 10-dimensional feature. The dimension of the point is increased to 64 dimensions using a fully connected layer. After maximum pooling MaxPooling, the number of points in each pillar is compressed to 1, and the compressed data is mapped back to the point cloud space and converted into a 2D pseudo image for input into the feature extraction network.

[0077] S233. Perform feature extraction on the pseudo image sequence through a point cloud feature extraction network to obtain a second bird's-eye view feature image sequence.

[0078] The point cloud feature extraction network is used to extract features from the pseudo image, for example, it can be a two-dimensional convolutional network.

[0079] Specifically, feature extraction is performed on each of the images in the pseudo image sequence obtained by radar point cloud conversion through a point cloud feature extraction network to obtain a second bird's-eye view feature map sequence consisting of feature maps at a bird's-eye view.

[0080] S240 , performing feature fusion on the bird's-eye view feature images of the same frame in the first bird's-eye view feature image sequence and the second bird's-eye view feature image sequence to obtain a target bird's-eye view feature image sequence.

[0081] Specifically, the first bird's-eye view feature map sequence acquired by the vehicle-mounted image sensor and processed by feature and the second bird's-eye view feature map sequence acquired by the vehicle-mounted radar sensor and processed by feature are fused according to the frame sequence number, so that the bird's-eye view feature map of the image and the bird's-eye view feature map of the radar point cloud at the same time are fused into one feature map. Finally, a target bird's-eye view feature map sequence that fuses the features of the image and the radar point cloud is obtained.

[0082] S250, performing feature fusion on the historical frame bird's-eye view feature map and the current frame bird's-eye view feature map in the target bird's-eye view feature map sequence to obtain an enhanced bird's-eye view feature map.

[0083] As an optional implementation of the embodiment of the present application, performing feature fusion on the historical frame bird's-eye view feature map and the current frame bird's-eye view feature map in the target bird's-eye view feature map sequence to obtain an enhanced bird's-eye view feature map includes:

[0084] S251. Determine the displacement of the bird's-eye view feature images of adjacent frames in a target bird's-eye view feature image sequence according to the vehicle's driving speed and the sampling frequency of the vehicle-mounted sensor.

[0085] Specifically, the sampling time interval of adjacent frames of the bird's-eye view feature image in the target bird's-eye view feature image sequence is determined according to the sampling frequency of the vehicle-mounted sensor, and the product of the sampling time interval and the vehicle driving speed is determined as the displacement of the adjacent frames of the bird's-eye view feature image.

[0086] S252, mapping the historical frame bird's-eye view feature map in the target bird's-eye view feature map sequence to the current frame bird's-eye view feature map according to the displacement, and performing feature fusion on the historical frame bird's-eye view feature map and the current frame bird's-eye view feature map to obtain an enhanced bird's-eye view feature map.

[0087] Specifically, for each historical frame bird's-eye feature map in the target bird's-eye feature map sequence, the displacement of each historical frame bird's-eye feature map and the current frame bird's-eye feature map is determined according to the displacement of the bird's-eye feature maps of adjacent frames in the target bird's-eye feature map sequence and the time interval (or the number of interval frames) between the historical frame bird's-eye feature map and the current frame bird's-eye feature map. According to the displacement of the historical frame bird's-eye feature map and the current frame bird's-eye feature map, each historical frame bird's-eye feature map is mapped to the current frame bird's-eye feature map; and the current frame bird's-eye feature map and each historical frame bird's-eye feature map are feature fused through a feature fusion network to obtain an enhanced bird's-eye feature map. Exemplarily, the size of each frame bird's-eye feature map is (B, C1, X, Y), and the size of the enhanced bird's-eye feature map obtained after fusion is (B, nC1, X, Y); wherein B is a batch, which can be set to 1; n is the number of bird's-eye feature maps contained in the target bird's-eye feature map sequence; C1 represents the feature dimension of the bird's-eye feature map; X represents the size of the bird's-eye feature map in the X direction, and Y represents the size of the bird's-eye feature map in the Y direction.

[0088] S260: extracting a map feature map from a road map of the vehicle's driving road, and performing feature fusion on the enhanced bird's-eye view feature map and the map feature map to obtain a target fused feature map.

[0089] Specifically, the feature fusion of the enhanced bird's-eye view feature map and the map feature map may include: feature splicing, feature alignment and dimensionality reduction processing.

[0090] As an optional implementation of the embodiment of the present application, the step of performing feature fusion on the enhanced bird's-eye view feature map and the map feature map to obtain a target fused feature map includes:

[0091] S261. Perform feature splicing on the enhanced bird's-eye view feature map and the map feature map in the channel dimension to obtain a spliced ​​feature map.

[0092] Exemplarily, the size of the enhanced bird's-eye view feature map is (B, nC1, X, Y), the size of the map feature map is (B, C2, X, Y), and the size of the spliced ​​feature map obtained by splicing the enhanced bird's-eye view feature map and the map feature map in the channel dimension is (B, nC1+C2, X, Y).

[0093] S262. Perform feature alignment on the concatenated feature map through a feature alignment network to obtain an aligned feature map.

[0094] The feature alignment network may be a cross-attention module or a convolutional layer. The aligned feature map may be understood as an image after feature alignment of the map feature map and the enhanced bird's-eye view feature map.

[0095] Specifically, the concatenated feature map is aligned through a feature alignment network to obtain an aligned feature map in which lane line features contained in the map feature map and lane line features contained in the enhanced bird's-eye view feature map are aligned. When aligning features, the key points of the lane line can be aligned.

[0096] S263. Perform dimensionality reduction processing on the aligned feature map through a convolutional network to obtain a target fused feature map.

[0097] Specifically, the size of the aligned feature map is (B, nC1+C2, X, Y), and the aligned feature map is reduced in dimension through a convolutional network to obtain a target fusion feature map with a size of (B, nC1, X, Y).

[0098] In this embodiment, the target fusion feature map fuses the image features and radar point cloud features at multiple times, as well as the road map features of the vehicle's driving road.

[0099] S270: Perform lane line detection on the target fusion feature map to obtain the lane line of the road on which the vehicle is traveling.

[0100] In a specific example, Figure 3 FIG. 1 is a schematic diagram showing the principle of a lane line detection method provided by an embodiment of the present invention. Figure 3As shown, an image sequence I consisting of images collected by an on-board image sensor at T moments is obtained, and feature processing is performed on the image sequence to obtain a first bird's-eye feature map sequence FI; a radar point cloud sequence P consisting of radar point clouds collected by an on-board radar sensor at T moments is obtained, and feature processing is performed on the radar point cloud sequence P to obtain a second bird's-eye feature map sequence FP; a target bird's-eye feature map sequence TF at T moments is obtained by performing a fusion of the first bird's-eye feature map sequence FI and the second bird's-eye feature map sequence FP; the target bird's-eye feature maps at T moments in the target bird's-eye feature map sequence TF are fused to obtain an enhanced bird's-eye feature map FF at the Tth moment, where the Tth moment may be the current moment and the previous T-1 moments may be historical moments; a road map of the road on which the vehicle is traveling is obtained; feature processing is performed on the road map to obtain a road map feature FM at the Tth moment; feature fusion is performed on the enhanced bird's-eye feature map FF at the Tth moment and the road map feature FM at the Tth moment to obtain a target fused feature map; lane line detection is performed on the target fused feature map to obtain the lane line of the road on which the vehicle is traveling.

[0101] The technical solution of the embodiment of the present invention is to obtain an image sequence of the vehicle driving road collected by an on-board image sensor and a radar point cloud sequence of the vehicle driving road collected by an on-board radar sensor; perform feature processing on the image sequence to obtain a first bird's-eye feature map sequence; perform feature processing on the image sequence to obtain a first bird's-eye feature map sequence; perform feature fusion on the bird's-eye feature maps of the same frame in the first bird's-eye feature map sequence and the second bird's-eye feature map sequence to obtain a target bird's-eye feature map sequence; perform feature fusion on the bird's-eye feature maps of the historical frames and the bird's-eye feature maps of the current frame in the target bird's-eye feature map sequence to obtain an enhanced bird's-eye feature map; extract a map feature map from a road map of the vehicle driving road, and perform feature fusion on the enhanced bird's-eye feature map and the map feature map to obtain a target fused feature map; perform lane line detection on the target fused feature map to obtain the lane line of the vehicle driving road. The embodiments of the present invention can integrate the bird's-eye view feature maps corresponding to the road data sensed by the vehicle-mounted image sensor and the vehicle-mounted radar sensor at multiple times, as well as the prior road information contained in the road map. It has better road information perception capabilities for complex problems such as changes in road light and rapid changes in road conditions, thereby improving the continuity, accuracy and stability of lane line detection, and is suitable for lane line detection on long-distance roads.

[0102] Embodiment 3

[0103] Figure 4 This is a schematic diagram of the structure of a lane line detection device provided in Embodiment 3 of the present invention. Figure 4 As shown, the device includes: a feature extraction module 310, a feature enhancement module 320, a feature fusion module 330 and a lane line detection module 340; wherein,

[0104] The feature extraction module 310 is used to obtain a data sequence of a road collected by a vehicle-mounted sensor, and perform feature processing on the data sequence to obtain a target bird's-eye view feature map sequence; wherein the data sequence arranges the collected road data according to the collection time;

[0105] The feature enhancement module 320 performs feature fusion on the historical frame bird's-eye view feature image and the current frame bird's-eye view feature image in the target bird's-eye view feature image sequence to obtain an enhanced bird's-eye view feature image;

[0106] The feature fusion module 330 is used to extract a map feature map from a road map of the vehicle driving road, and perform feature fusion on the enhanced bird's-eye view feature map and the map feature map to obtain a target fused feature map;

[0107] The lane line detection module 340 is used to perform lane line detection on the target fusion feature map to obtain the lane line of the road where the vehicle is traveling.

[0108] The technical solution of the embodiment of the present invention is to obtain a target bird's-eye feature map sequence by acquiring a data sequence of a road collected by an on-board sensor and performing feature processing on the data sequence; wherein the data sequence arranges the collected road data according to the collection time; performs feature fusion on the historical frame bird's-eye feature map and the current frame bird's-eye feature map in the target bird's-eye feature map sequence to obtain an enhanced bird's-eye feature map; extracts a map feature map from a road map of the road on which the vehicle is traveling, and performs feature fusion on the enhanced bird's-eye feature map and the map feature map to obtain a target fused feature map; performs lane line detection on the target fused feature map to obtain the lane line of the road on which the vehicle is traveling. The embodiment of the present invention can fuse road features perceived at multiple times and a priori road information contained in the road map, and has better road information perception capabilities for complex problems such as changes in road light and rapid changes in road conditions, thereby improving the continuity, accuracy and stability of lane line detection, and is suitable for lane line detection on long-distance roads.

[0109] Optionally, the vehicle-mounted sensor includes: a vehicle-mounted image sensor and a vehicle-mounted radar sensor; the data sequence includes: an image sequence of the vehicle driving road collected by the vehicle-mounted image sensor and a radar point cloud sequence of the vehicle driving road collected by the vehicle-mounted radar sensor.

[0110] Optionally, the feature extraction module includes:

[0111] A first feature processing unit, used for performing feature processing on the image sequence to obtain a first bird's-eye view feature image sequence;

[0112] A second feature processing unit is used to perform feature processing on the radar point cloud sequence to obtain a second bird's-eye view feature image sequence;

[0113] The feature fusion unit is used to perform feature fusion on the bird's-eye view feature images of the same frame in the first bird's-eye view feature image sequence and the second bird's-eye view feature image sequence to obtain a target bird's-eye view feature image sequence.

[0114] Optionally, the first feature processing unit is specifically configured to:

[0115] Extracting features from the image sequence through a feature extraction network cascaded by a deep residual network and a feature pyramid network to obtain a multi-scale feature map sequence;

[0116] Performing depth estimation on the multi-scale feature map sequence through a depth estimation network to obtain a spatial feature map sequence;

[0117] The spatial feature map sequence is subjected to bird's-eye view pooling through a pooling layer to obtain a first bird's-eye view feature sequence.

[0118] Optionally, the second feature processing unit includes:

[0119] Mapping each frame of radar point cloud in the radar point cloud sequence to an image coordinate system of a corresponding frame image in the image sequence according to calibration parameters of the vehicle-mounted sensor;

[0120] The radar point clouds mapped in each of the image coordinate systems are converted into pseudo images through a point cloud feature processing model, and a pseudo image sequence is formed according to each of the pseudo images;

[0121] The pseudo image sequence is subjected to feature extraction through a point cloud feature extraction network to obtain a second bird's-eye view feature image sequence.

[0122] Optionally, the feature enhancement module is specifically used to:

[0123] Determining the displacement of the bird's-eye view feature images of adjacent frames in the target bird's-eye view feature image sequence according to the vehicle speed and the sampling frequency of the vehicle-mounted sensor;

[0124] According to the displacement, the historical frame bird's-eye view feature map in the target bird's-eye view feature map sequence is mapped to the current frame bird's-eye view feature map, and feature fusion is performed on the historical frame bird's-eye view feature map and the current frame bird's-eye view feature map to obtain an enhanced bird's-eye view feature map.

[0125] Optionally, the feature fusion module is specifically used to:

[0126] Projecting the map feature map into the image coordinate system of the enhanced bird's-eye view feature map according to the vehicle's posture information and the calibration parameters of the vehicle-mounted image sensor;

[0127] Performing feature stitching on the enhanced bird's-eye view feature map and the map feature map projected in the image coordinate system in a channel dimension to obtain a stitched feature map;

[0128] Performing feature alignment on the spliced ​​feature map through a feature alignment network to obtain an aligned feature map;

[0129] Perform dimensionality reduction processing on the aligned feature map to obtain a target fusion feature map.

[0130] The lane line detection device provided in the embodiment of the present invention can execute the lane line detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0131] Embodiment 4

[0132] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0133] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0134] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0135] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as the lane line detection method.

[0136] In some embodiments, the lane detection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the lane detection method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the lane detection method in any other appropriate manner (e.g., by means of firmware).

[0137] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] In some embodiments, the lane line detection method may be implemented as a computer program, which is invisibly included in a computer program product. The computer program implements the lane line detection method of the present invention when executed by a processor. The computer program product can be understood as a software product that implements its solution mainly through a computer program. The computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the computer program, when executed by the processor, implements the functions / operations specified in the flowchart and / or block diagram. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0139] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0140] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0141] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0142] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0143] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0144] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A lane line detection method, characterized in that: include: Acquire a data sequence of a vehicle driving road collected by an on-board sensor, and perform feature processing on the data sequence to obtain a target bird's-eye view feature map sequence; wherein the data sequence arranges the collected road data according to the collection time; Performing feature fusion on the historical frame bird's-eye view feature image and the current frame bird's-eye view feature image in the target bird's-eye view feature image sequence to obtain an enhanced bird's-eye view feature image; Extracting a map feature map from a road map of the vehicle's driving road, and performing feature fusion on the enhanced bird's-eye view feature map and the map feature map to obtain a target fused feature map; Perform lane line detection on the target fusion feature map to obtain the lane line of the road where the vehicle is traveling.

2. The lane line detection method according to claim 1, characterized in that: The vehicle-mounted sensor includes: a vehicle-mounted image sensor and a vehicle-mounted radar sensor; the data sequence includes: an image sequence of the vehicle driving road collected by the vehicle-mounted image sensor and a radar point cloud sequence of the vehicle driving road collected by the vehicle-mounted radar sensor; accordingly, the step of extracting features from the data sequence to obtain a target bird's-eye view feature map sequence includes: Performing feature processing on the image sequence to obtain a first bird's-eye view feature image sequence; Performing feature processing on the radar point cloud sequence to obtain a second bird's-eye view feature image sequence; Feature fusion is performed on the bird's-eye view feature images of the same frame in the first bird's-eye view feature image sequence and the second bird's-eye view feature image sequence to obtain a target bird's-eye view feature image sequence.

3. The lane line detection method according to claim 2, characterized in that: The step of performing feature processing on the image sequence to obtain a first bird's-eye view feature image sequence comprises: Extracting features from the image sequence through a feature extraction network cascaded by a deep residual network and a feature pyramid network to obtain a multi-scale feature map sequence; Performing depth estimation on the multi-scale feature map sequence through a depth estimation network to obtain a spatial feature map sequence; The spatial feature map sequence is subjected to bird's-eye view pooling through a pooling layer to obtain a first bird's-eye view feature sequence.

4. The lane line detection method according to claim 2, characterized in that: The step of performing feature processing on the radar point cloud sequence to obtain a second bird's-eye view feature map sequence comprises: Mapping each frame of radar point cloud in the radar point cloud sequence to an image coordinate system of a corresponding frame image in the image sequence according to calibration parameters of the vehicle-mounted sensor; The radar point clouds mapped in each of the image coordinate systems are converted into pseudo images through a point cloud feature processing model, and a pseudo image sequence is formed according to each of the pseudo images; The pseudo image sequence is subjected to feature extraction through a point cloud feature extraction network to obtain a second bird's-eye view feature image sequence.

5. The lane line detection method according to any one of claims 1 to 4, characterized in that: The step of performing feature fusion on the historical frame bird's-eye view feature image and the current frame bird's-eye view feature image in the target bird's-eye view feature image sequence to obtain an enhanced bird's-eye view feature image comprises: Determining the displacement of the bird's-eye view feature images of adjacent frames in the target bird's-eye view feature image sequence according to the vehicle speed and the sampling frequency of the vehicle-mounted sensor; According to the displacement, the historical frame bird's-eye view feature map in the target bird's-eye view feature map sequence is mapped to the current frame bird's-eye view feature map, and feature fusion is performed on the historical frame bird's-eye view feature map and the current frame bird's-eye view feature map to obtain an enhanced bird's-eye view feature map.

6. The method according to claim 2, characterized in that The step of fusing the enhanced bird's-eye view feature map and the map feature map to obtain a target fused feature map comprises: Projecting the map feature map into the image coordinate system of the enhanced bird's-eye view feature map according to the vehicle's posture information and the calibration parameters of the vehicle-mounted image sensor; Performing feature stitching on the enhanced bird's-eye view feature map and the map feature map projected in the image coordinate system in a channel dimension to obtain a stitched feature map; Performing feature alignment on the spliced ​​feature map through a feature alignment network to obtain an aligned feature map; Perform dimensionality reduction processing on the aligned feature map to obtain a target fusion feature map.

7. A lane line detection device, characterized in that: include: A feature extraction module is used to obtain a data sequence of a road collected by a vehicle-mounted sensor, and perform feature processing on the data sequence to obtain a target bird's-eye view feature map sequence; wherein the data sequence arranges the collected road data according to the collection time; A feature enhancement module is used to perform feature fusion on the historical frame bird's-eye view feature image and the current frame bird's-eye view feature image in the target bird's-eye view feature image sequence to obtain an enhanced bird's-eye view feature image; A feature fusion module is used to extract a map feature map from a road map of the vehicle's driving road, and perform feature fusion on the enhanced bird's-eye view feature map and the map feature map to obtain a target fused feature map; The lane line detection module is used to perform lane line detection on the target fusion feature map to obtain the lane line of the road where the vehicle is traveling.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the lane line detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the lane line detection method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the lane line detection method according to any one of claims 1 to 6.

Images