Lane detection method, system, equipment and medium

Abstract

The invention provides a lane detection method, system and device and a medium, and the method comprises the steps: predicting the height probability distribution of each pixel in roadside image data from a ground reference height to obtain road curved surface data, and accurately describing a real road structure; by randomly adding disturbance in an external parameter matrix and suppressing height probability distribution of a non-road area, the influence of external parameters of shooting equipment on three-dimensional lane reconstruction can be reduced, and stable and accurate road curved surface data can be generated; the characteristics of different types of lane data sets are progressively utilized in a staged mode, hierarchical optimization from geometric perception to semantic understanding is achieved, the problem that a shooting device is unclear in a long-distance lane structure can be solved, and lane semantic segmentation data are accurately generated; by combining road curved surface data and lane semantic segmentation data, high-precision detection and reconstruction of three-dimensional lane lines can be carried out on roadside image data collected by shooting equipment of roadside facilities.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a lane detection method, system, device and medium. Background Technology

[0002] In the field of autonomous driving, most existing lane detection methods are designed for vehicle-mounted forward-facing cameras. When transferring lane detection algorithms oriented towards vehicle perspectives to roadside facilities, it is often difficult to achieve ideal results. To improve the accuracy of lane detection, related technologies introduce prior knowledge of lanes to constrain the lane structure, thereby achieving lane detection in roadside scenarios. However, this method does not take into account the characteristics of roadside camera devices, such as high camera position, long distance, large depth span, and relatively fixed viewing angle, resulting in low accuracy of the lane detection results. Therefore, there is an urgent need for a lane detection method that can take into account the characteristics of roadside facility camera devices to improve the lane detection accuracy of roadside facilities. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a lane detection method, system, device and medium to improve the lane detection accuracy of roadside facilities.

[0004] To achieve the above objectives, this application provides a lane detection method applied to roadside facilities, comprising: acquiring roadside image data, wherein the roadside image data is obtained from the camera of the roadside facility; predicting the height probability distribution of each pixel in the roadside image data relative to the ground reference height through a pre-trained height estimation module to obtain road surface data; during the training of the height estimation module, adding random perturbation to the extrinsic parameter matrix of the camera corresponding to the input image, and performing supervised training on the perturbated height estimation module based on the real ground height label of the input image, thereby suppressing the height probability distribution in non-road areas during supervised training; The roadside image data is processed by a pre-trained lane segmentation module to obtain lane semantic segmentation data. During the training of the lane segmentation module, the first stage of training is performed based on the vehicle-view lane dataset, using an optimization objective constructed with mask segmentation loss and mask quality loss to obtain the first optimization parameters. Based on the roadside lane segmentation dataset and the first optimization parameters, the second stage of training is performed based on an optimization objective constructed with mask contour loss to obtain the second optimization parameters. Based on the roadside lane semantic dataset and the second optimization parameters, the lane segmentation module is performed in a third stage of training using the topological and logical relationships between lanes to obtain the third optimization parameters. The 3D lane detection results are obtained by calculating based on the road surface data and the lane semantic segmentation data.

[0005] Optionally, a pre-trained height estimation module predicts the height probability distribution of each pixel in the roadside image data relative to the ground reference height to obtain road surface data. This includes: extracting multi-scale visual features from the roadside image data to obtain high-dimensional feature information; extracting contextual information and residual features from the high-dimensional feature information to obtain enhanced feature information; processing the enhanced feature information using deformable convolution to predict the height distribution probability of each pixel in multiple height bins; the multiple height bins are obtained by dividing the road height interval according to a preset division rule; for any pixel, if the highest probability in the pixel's height probability distribution exceeds a preset confidence threshold, then the pixel's height probability distribution is used to construct the road surface data; calculating the position information of the height bins based on the road height interval, height probability distribution, number of height bins, and density of the height bins; projecting the pixels used to construct the road surface data from the enhanced feature information into three-dimensional space and transforming them into the vehicle coordinate system based on the position information of the height bins to obtain three-dimensional wedge features; and processing the three-dimensional wedge features using pooling, convolution, and regression functions to obtain the road surface data.

[0006] Optionally, based on the position information of the height buckets, the pixels used to construct road surface data in the enhanced feature information are projected into three-dimensional space and transformed into the vehicle coordinate system to obtain three-dimensional wedge features. This includes: transforming the pixels used to construct road surface data in the enhanced feature information into a virtual coordinate system based on the intrinsic parameter matrix of the shooting device and the first transformation matrix to obtain first surface data; the first transformation matrix represents the transformation relationship between the coordinate system of the shooting device and the virtual coordinate system, where the virtual coordinate system is obtained by rotating the coordinate system of the shooting device, and the vertical axis of the virtual coordinate system is perpendicular to the ground; the three-dimensional projection points of the height buckets are calculated based on the height from the origin of the virtual coordinate system to the ground, the position information of the height buckets, the vertical coordinate values ​​of the virtual coordinate system, and the first surface data; the three-dimensional wedge features are obtained by calculating based on the three-dimensional projection points and the second transformation matrix; the second transformation matrix represents the transformation relationship between the virtual coordinate system and the vehicle coordinate system, and is obtained based on the extrinsic parameter matrix of the shooting device and the first transformation matrix.

[0007] Optionally, the lane segmentation module includes an image encoder, a cue word encoder, a mask decoder, a semantic detection head, and a memory unit. The pre-trained lane segmentation module processes roadside image data to obtain lane semantic segmentation data, including: encoding the roadside image data using the image encoder to obtain high-dimensional image feature data; encoding the cue information using the cue word encoder to obtain cue embedding data; fusing the high-dimensional image feature data and the cue embedding data using the mask decoder to obtain a multi-channel feature mask; performing channel attention and convolution operations on the multi-channel feature mask using the semantic detection head to obtain the current semantic segmentation data; and fusing the current semantic segmentation data and historical semantic segmentation data using the memory unit to obtain the lane semantic segmentation data.

[0008] Optionally, lane semantic segmentation data is obtained by fusing the current semantic segmentation data and historical semantic segmentation data according to the memory unit, including: calculating the first data based on the current semantic segmentation data and its corresponding first pixel confidence; calculating the second data based on the historical semantic segmentation data and its corresponding second pixel confidence; and calculating the lane semantic segmentation data based on the first data, the second data, the first pixel confidence, and the second pixel confidence.

[0009] Optionally, the lane segmentation module includes an image encoder, a cue word encoder, a mask decoder, a semantic detection head, and a memory unit. Based on the vehicle-view lane dataset, the lane segmentation module is trained in the first stage using an optimization objective constructed with mask segmentation loss and mask quality loss to obtain first optimized parameters. Based on the roadside lane segmentation dataset and the first optimized parameters, the lane segmentation module is trained in the second stage using an optimization objective constructed with mask contour loss to obtain second optimized parameters. Based on the roadside lane semantic dataset and the second optimized parameters, the lane segmentation module is trained in the third stage using the topological and logical relationships between lanes to obtain third optimized parameters. This includes: inputting the vehicle-view lane dataset into the lane segmentation module to be trained to obtain a first feature mask output by the mask decoder; calculating the mask segmentation loss and mask quality loss based on the first feature mask, and adjusting the cue word encoder and mask... The parameters of the mask decoder are used to obtain the first optimized parameters. The roadside lane segmentation dataset is input into the lane segmentation module based on the first optimized parameters to obtain the second feature mask output by the mask decoder. The mask segmentation loss, mask quality loss, and mask contour loss are calculated based on the second feature mask, and the parameters of the lane segmentation module are adjusted to obtain the second optimized parameters. The roadside lane semantic dataset is input into the lane segmentation module based on the second optimized parameters, and the semantic detection head and memory unit are trained using the topological and logical relationships between lanes to obtain the third optimized parameters. The mask segmentation loss is constructed based on the predicted first or second feature mask and the ground truth mask. The mask quality loss is constructed based on the quality score of the predicted first or second feature mask and the degree of closeness to the actual mask. The mask contour loss is constructed based on the mask contour of the predicted second feature mask and the ground truth mask contour.

[0010] Optionally, the 3D lane detection result is obtained by calculating based on road surface data and lane semantic segmentation data, including: mapping the lane semantic segmentation data to the coordinate system of the camera based on the intrinsic parameter matrix of the camera to obtain a first ray direction vector; mapping the first ray direction vector to the vehicle coordinate system based on the extrinsic parameter matrix of the camera to obtain a second ray direction vector; obtaining ray information extending along the viewing angle of the camera in the vehicle coordinate system based on the second ray direction vector; calculating the intersection point of the road surface data and the ray information to obtain a 3D point; and mapping the lane semantic segmentation data to the corresponding 3D point to obtain the 3D lane detection result.

[0011] This application also provides an autonomous driving system, including: an acquisition module for acquiring roadside image data, the roadside image data being obtained from the camera of a roadside facility; a calculation module for predicting the height probability distribution of each pixel in the roadside image data relative to a ground reference height through a pre-trained height estimation module, so as to obtain road surface data; during the training of the height estimation module, random perturbation is added to the extrinsic parameter matrix of the camera corresponding to the input image, and supervised training is performed on the perturbated height estimation module based on the real ground height label of the input image, suppressing the height probability distribution in non-road areas during supervised training; The roadside image data is processed by a pre-trained lane segmentation module to obtain lane semantic segmentation data. During the training of the lane segmentation module, the first stage of training is performed based on the vehicle-view lane dataset, using an optimization objective constructed with mask segmentation loss and mask quality loss to obtain the first optimization parameters. Based on the roadside lane segmentation dataset and the first optimization parameters, the second stage of training is performed based on an optimization objective constructed with mask contour loss to obtain the second optimization parameters. Based on the roadside lane semantic dataset and the second optimization parameters, the lane segmentation module is performed in a third stage of training using the topological and logical relationships between lanes to obtain the third optimization parameters. The 3D lane detection results are obtained by calculating based on the road surface data and the lane semantic segmentation data.

[0012] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method.

[0013] This application also provides a computer-readable storage medium that stores computer instructions for causing a computer to perform the methods described above.

[0014] As can be seen from the above, this application provides a lane detection method. On the one hand, by predicting the height probability distribution of each pixel in the roadside image data from the ground reference height, the road surface data can accurately depict the real road structure. Even if the shooting device changes its posture or external parameters drift due to loose installation, thermal deformation, or vibration at a high installation position, causing a deviation in the ground reference height, this application provides a self-correcting capability by randomly adding controllable disturbances and suppressing the height probability distribution of non-road areas to the external parameter matrix of the shooting device corresponding to the input image. This allows the height estimation module to adaptively compensate for deviations caused by external parameters. By offsetting the volume height, the influence of the extrinsic parameters of the imaging equipment on 3D lane reconstruction is reduced, generating stable and accurate road surface data. On the other hand, by utilizing the characteristics of different types of lane datasets in a phased and progressive manner, hierarchical optimization from geometric perception to semantic understanding is achieved, which can solve the problem of unclear lane structure at long distances when imaging equipment is used, and accurately generate lane semantic segmentation data. By combining road surface data and lane semantic segmentation data, high-precision detection and reconstruction of 3D lane lines can be achieved on roadside image data collected by imaging equipment of roadside facilities, and the stability and accuracy of 3D lane reconstruction can be improved without relying on precise extrinsic parameters. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of a lane detection method according to an embodiment of this application; Figure 2 This is a schematic flowchart of a lane detection method according to an embodiment of this application; Figure 3 This is a schematic diagram of the height-divided bins according to an embodiment of this application; Figure 4 This is a schematic diagram of the virtual coordinate system in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an autonomous driving system according to an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0018] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by those skilled in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects.

[0019] With the rapid development of intelligent transportation systems, 3D lane detection technology, as an important component of road environment perception, plays a crucial role in autonomous driving, vehicle-to-infrastructure (V2I) communication, and traffic flow monitoring. Existing technologies primarily focus on the vehicle-mounted perspective, and their lane detection algorithms are designed based on the characteristics of vehicle-mounted perception, such as low camera position, small field of view, and close distance. These differ significantly from the installation methods, viewing angles, and imaging characteristics of roadside equipment. Directly transferring lane detection algorithms from vehicle-mounted scenarios to roadside scenarios leads to performance degradation.

[0020] Based on this, related technologies map forward-looking feature data from the roadside view to obtain bird's-eye view feature data according to spatial transformation relationships and prior information on lane line geometry, and obtain a three-dimensional lane detection structure based on the bird's-eye view feature data.

[0021] However, relying on prior information about lane geometry is insufficient for irregular road surfaces and non-standard lane shapes. When the road shape is complex, with large curvature variations or local undulations, detection results based on geometric priors are prone to bias and cannot accurately depict the real road structure. Furthermore, the methods mentioned above do not consider the characteristics of high-positioned and long-distance imaging equipment. Specifically, roadside imaging equipment is usually installed at a high position with a certain overhead view, covering a wide area and a large depth span. Lane lines in distant parts of the image are extremely fine, making it difficult to directly extract fine-grained semantic features. At the same time, the relatively fixed and limited perspective of roadside scenes makes it difficult to form large-scale datasets, further exacerbating the difficulty of roadside lane detection. In addition, lane detection in roadside scenes is more sensitive to geometric relationships. During the process of projecting lane features into 3D space using the extrinsic parameter matrix of the imaging equipment, even slight rotations or displacements of the extrinsic parameters of the imaging equipment will be amplified in distant regions, producing significant geometric shifts and severely affecting the estimation accuracy of the 3D road structure.

[0022] To address the problems existing in the prior art, this application provides a lane detection method. On one hand, by predicting the probability distribution of the height of each pixel in the roadside image data relative to the ground reference height, the road surface data can accurately depict the real road structure. Even if the shooting device experiences attitude changes or extrinsic parameter drift due to loose installation, thermal deformation, or vibration at a high installation position, causing deviations in the ground reference height, this application provides a self-correcting capability by randomly adding controllable perturbations and suppressing the height probability distribution of non-road areas to the extrinsic parameter matrix of the shooting device corresponding to the input image. This allows the height estimation module to adaptively compensate for deviations caused by extrinsic parameter issues. The overall height offset reduces the impact of the imaging equipment's extrinsic parameters on 3D lane reconstruction, generating stable and accurate road surface data. On the other hand, by utilizing the characteristics of different types of lane datasets in a phased and progressive manner, hierarchical optimization from geometric perception to semantic understanding is achieved, which can solve the problem of unclear lane structure at long distances when imaging equipment is used, and accurately generate lane semantic segmentation data. By combining road surface data and lane semantic segmentation data, high-precision detection and reconstruction of 3D lane lines can be achieved on roadside image data collected by imaging equipment of roadside facilities, and the stability and accuracy of 3D lane reconstruction can be improved without relying on precise extrinsic parameters.

[0023] refer to Figure 1 This is a schematic diagram illustrating an application scenario of the lane detection method provided in this application embodiment. The application scenario includes an autonomous vehicle 100 and roadside facilities 110. The autonomous vehicle 100 and the roadside facilities 110 are connected via a wired or wireless communication network.

[0024] The autonomous vehicle 100 may include onboard computing units, cameras, lidar, millimeter-wave radar, positioning modules, communication modules, and other sensors and computing devices. It can collect images, point clouds, and positioning information of the surrounding environment in real time and run autonomous driving function modules such as perception, prediction, planning, and control. In this embodiment, the autonomous vehicle can acquire vehicle image data from the camera, and based on this image data, it can obtain an onboard view lane dataset. This processing can be performed on the autonomous vehicle side or on the roadside facility side. If processed on the autonomous vehicle side, the onboard view lane dataset is sent to the roadside facility; if processed on the roadside facility side, the vehicle image data is sent to the roadside facility.

[0025] Roadside facility 110 can be deployed along the road and can be used to perform environmental perception of road sections such as intersections, curves, and obstructed areas. The perception results can be sent to autonomous vehicles or users via a communication link. In this embodiment, the roadside facility may include, but is not limited to, a camera and a computing server. The camera and computing server are communicatively connected. The camera is used to capture road scenes to obtain roadside image data. The computing server can provide lane detection services to users based on lane detection methods. Before or during the execution of this method, the computing server can process the roadside image data to obtain a roadside lane segmentation dataset and a roadside lane semantic dataset. The parameters of the viewpoint feature extraction module are optimized based on the vehicle-view lane dataset, the roadside lane segmentation dataset, and the roadside lane semantic dataset. In addition, the roadside facility may also include roadside units, LiDAR, millimeter-wave radar, and communication gateways.

[0026] The following is combined Figure 1 The application scenarios described above illustrate the lane detection method according to exemplary embodiments of this application. It should be noted that the above application scenarios are merely shown for the purpose of understanding the principles of this application, and the embodiments of this application are not limited in any way in this respect. Rather, the embodiments of this application can be applied to any applicable scenario.

[0027] refer to Figure 2 Lane detection methods may include: S1. Acquire roadside image data, which comes from the camera equipment of the roadside facilities.

[0028] S2. Predict the height probability distribution of each pixel in the roadside image data relative to the ground reference height through a pre-trained height estimation module to obtain road surface data. During the training of the height estimation module, add random perturbation to the extrinsic parameter matrix of the shooting device corresponding to the input image. Based on the real ground height label of the input image, supervise the training of the perturbated height estimation module and suppress the height probability distribution in non-road areas during supervised training.

[0029] S3. The roadside image data is processed by a pre-trained lane segmentation module to obtain lane semantic segmentation data. During the training of the lane segmentation module, the first stage of training is performed on the lane segmentation module based on the vehicle-view lane dataset and using the optimization objective constructed by mask segmentation loss and mask quality loss to obtain the first optimization parameters. Based on the roadside lane segmentation dataset and the first optimization parameters, the second stage of training is performed on the lane segmentation module based on the optimization objective constructed by mask contour loss to obtain the second optimization parameters. Based on the roadside lane semantic dataset and the second optimization parameters, the lane segmentation module is performed in the third stage of training using the topological and logical relationships between lanes to obtain the third optimization parameters.

[0030] S4. Calculate the three-dimensional lane detection results based on the road surface data and lane semantic segmentation data.

[0031] Roadside image data can refer to data collected by the camera equipment of roadside facilities. It should be noted that in the following description, the camera equipment refers to one side of the roadside facility.

[0032] In this embodiment, the roadside facilities can invoke a height estimation module and a lane segmentation module. The height estimation module can predict the height probability distribution of each pixel based on the input roadside image data, thereby obtaining road surface data. The height probability distribution of a pixel refers to the probability distribution of the spatial position corresponding to that pixel relative to the ground reference height at various heights. The road surface data refers to the three-dimensional geometric structure information of the road, which can characterize the longitudinal undulations and lateral slopes of the road, and depict the undulations and local elevation differences of the road. The lane segmentation module can perform semantic understanding on the input roadside image data and output lane semantic segmentation data. The lane semantic segmentation data refers to the pixel-level classification results of the lane area, such as lane line type, drivable area, and distinction between solid and dashed lines.

[0033] The height estimation module and lane segmentation module can be jointly trained to achieve collaborative optimization of road surface data and lane semantic segmentation data; alternatively, they can be trained independently to improve the flexibility and maintainability of module deployment. The height estimation module and lane segmentation module can be deployed on the local edge computing unit of the roadside facility, or on a remote server or cloud computing platform; this is not limited in this embodiment.

[0034] By fusing road surface data and lane semantic segmentation data, a three-dimensional lane detection result can be obtained. This three-dimensional lane detection result can refer to lane structure information with three-dimensional spatial coordinates, which may include lane location information, as well as lane geometry and topological relationships.

[0035] During the training of the height estimation module, controllable perturbations are randomly added to the extrinsic parameters (such as pitch angle, roll angle, and height offset) of the camera corresponding to the input image to simulate possible installation deviations or extrinsic parameter drifts in actual deployment. These perturbations cause the ground reference height (i.e., the zero-height baseline) to shift relative to the real ground, thus introducing height bias into the label space. However, the network in the height estimation module can adaptively adjust the height probability distribution by minimizing the loss between the predicted distribution and the perturbed labels. This is based on the prior knowledge that "road pixel height predictions should be concentrated within the road height range" and the guidance of large-scale real sample data. This ensures that pixels truly belonging to the road area tend to form high-confidence peaks near the undisturbed physical ground height, rather than simply following the label shift.

[0036] Meanwhile, the height estimation module can automatically suppress the height probability distribution in non-road areas during the optimization process. For example, it can reduce the peak value of the height probability distribution and increase the distribution entropy, thereby resulting in a lower maximum probability value or a more diffuse probability distribution at the output level.

[0037] Ultimately, this training strategy enables the height estimation module to have self-correction capabilities. Even if there are uncalibrated extrinsic parameter biases during actual inference, the height estimation module can still output a road surface based on the real ground, effectively suppressing systematic errors caused by the offset of the ground reference plane.

[0038] In this embodiment of the application, the road height range can refer to the vertical height range of the actual road surface. For example, the road height range can be [-3m, 3m].

[0039] During the processing of the lane segmentation module, in the training phase, based on the vehicle-view lane dataset, the lane segmentation module is trained in the first stage using an optimization objective constructed with mask segmentation loss and mask quality loss to obtain the first optimized parameters; based on the roadside lane segmentation dataset and the first optimized parameters, the lane segmentation module is trained in the second stage using an optimization objective constructed with mask contour loss to obtain the second optimized parameters; based on the roadside lane semantic dataset and the second optimized parameters, the lane segmentation module is trained in the third stage using the topological and logical relationships between lanes to obtain the third optimized parameters; in the inference phase, the lane segmentation module obtained based on the third optimized parameters processes the roadside image data.

[0040] In this context, the mask refers to the category labeling of each pixel in the roadside image data, which can be represented using binary semantics or probabilistic form. The mask segmentation loss is constructed based on the pixel-level semantic classification between the predicted mask and the ground truth mask, ensuring pixel-level semantic accuracy. The mask quality loss is constructed based on the degree to which the predicted mask quality score closely approximates the ground truth value, improving overall prediction confidence and completeness. The degree of closeness to the ground truth can be measured by the local or global overlap between the predicted mask and the ground truth mask, such as the IoU (Intersection over Union) coefficient. The mask contour loss is constructed based on the predicted mask contour and the ground truth mask contour, ensuring the geometric accuracy of lane boundaries. The topological and logical relationships between lanes can be modeled using current and historical roadside image data. This can be understood as a semantic topological structure for predictable behavior. For example, if historical data consistently shows that a lane maintains a change-lane relationship with the left lane before approaching an intersection, and the current frame has blurred boundaries due to occlusion, the likelihood of a legal lane change can still be inferred based on the remembered topological and logical relationships. By constructing the topological and logical relationships between lanes, the lane logical relationships become forward-looking, allowing for the fusion of information from multiple frames across frames. This continuously improves the segmentation results for occluded or distant lanes, enabling dynamic completion of lane line masks in occluded scenarios, solving the problem of unclear distances, and obtaining complete and continuous lane semantic segmentation data.

[0041] The vehicle-mounted lane dataset can include vehicle image data captured by the onboard cameras of autonomous vehicles, as well as information such as labeled lane line pixels, providing a local road structure from the vehicle's perspective. The roadside lane segmentation dataset can include roadside image data captured by cameras of roadside facilities, as well as labeled semantic segmentation masks, providing a global road structure from the roadside perspective. The roadside lane semantic dataset can include roadside image data captured by cameras of roadside facilities, as well as labeled pixel-level semantic tags, lane topology, and other information.

[0042] By utilizing an onboard lane dataset and under the joint supervision of mask segmentation loss and mask quality loss, the model learns the slender and continuous geometric features of lane lines, yielding the first set of optimized parameters. Building upon this, a roadside lane segmentation dataset is introduced, and mask contour loss is used to further refine the model's geometric modeling capabilities for lane boundaries, gradually adapting to the imaging characteristics of roadside high-position, long-distance, and fine-grained structures, resulting in the second set of optimized parameters. Finally, based on this geometric foundation and combined with the logical association information from roadside lane semantic data, an optimization objective oriented towards semantic reasoning is constructed, leading to the third set of optimized parameters with contextual understanding capabilities. This progressive optimization process not only effectively integrates multi-view, multi-granularity lane prior knowledge but also significantly improves geometric accuracy, semantic integrity, and cross-view consistency in complex traffic scenarios.

[0043] By combining road surface data and lane semantic segmentation data, it is possible to perform high-precision detection and reconstruction of 3D lane lines on roadside image data collected by roadside facility imaging equipment, and improve the stability and accuracy of 3D lane reconstruction without relying on precise external parameters.

[0044] In an optional embodiment, road surface data is obtained by predicting the height probability distribution of each pixel in the roadside image data relative to a ground reference height using a pre-trained height estimation module, including: S200. Extract multi-scale visual features from roadside image data to obtain high-dimensional feature information; extract contextual information and residual features from the high-dimensional feature information to obtain enhanced feature information.

[0045] High-dimensional feature information refers to feature information extracted from roadside image data at different spatial scales (which can be understood as different resolutions or receptive field sizes). This high-dimensional feature information can be obtained based on convolutional neural networks and feature pyramid networks. For example, the convolutional neural network can be ResNet-50, and the feature pyramid network can be SECOND FPN. For ease of subsequent description, high-dimensional feature information can be defined as F... 2d ∈ C F ×H / 16×W / 16 , where C F H represents the number of channels, W represents the height of the roadside image data, and 16 represents the width of the roadside image data. The total downsampling rate accumulated from the downsampling operation is not fixed and can be adjusted according to the actual situation.

[0046] After obtaining high-dimensional feature information, contextual information and residual features are extracted from this high-dimensional feature information to obtain enhanced feature information. In one implementation, contextual information can be extracted from the high-dimensional feature information using a compressed excitation network. This compressed excitation network can compress spatial information, learn the dependencies between channels, and adaptively reweight feature channels to enhance key information and improve overall perception capability. Residual features can be extracted from the high-dimensional feature information using at least one residual network, significantly enhancing boundary clarity and geometric accuracy. By fusing contextual information and residual features, complementary synergy between semantic understanding and detail perception can be achieved, improving the perception capability of complex scenes.

[0047] S210. Deformable convolution is used to process the enhanced feature information to predict the height distribution probability of each pixel in multiple height bins; the multiple height bins are obtained by dividing the road height range according to a preset division rule.

[0048] Deformable convolution refers to a convolution operation that adaptively adjusts the sampling position. After processing the enhanced feature information using deformable convolution, a classification function can be used to map the feature vector of each pixel in the feature data obtained by deformable convolution to a height probability distribution relative to the ground reference height.

[0049] To reduce the difficulty of optimization, in the embodiments of this application, such as Figure 3 As shown, the continuous height is divided into N height bins, where [-3,3] represents the ground reference height. Figure 3 The horizontal lines represent the lines dividing the height into bins, and adjacent horizontal lines represent one height bin. This embodiment employs a nonlinear compression discretization strategy. The height bins closer to the ground reference height have smaller intervals, allowing for more accurate estimation of minor ground undulations. The height bins farther from the ground reference height have larger intervals, reducing the processing of irrelevant areas and improving optimization efficiency. Preferably, this discretization strategy is a power-law nonlinear compression strategy. Furthermore, the height bins in this embodiment can be vertically symmetrical relative to the ground reference height, thereby enabling the fitting of road surface depressions and protrusions.

[0050] S220. For any pixel, if the highest probability in the pixel's height probability distribution exceeds a preset confidence threshold, then the pixel's height probability distribution is used to construct road surface data.

[0051] By setting a confidence threshold to filter pixel-level height probability distributions, only high-confidence predictions are retained for constructing road surface data, significantly improving the reliability and geometric consistency of the reconstruction results. Since the height estimation module uses real ground height as the supervision signal during training, non-road areas, lacking corresponding ground truth height values, typically exhibit low peak values ​​or high entropy in their predicted distributions, resulting in lower confidence. Road areas, on the other hand, tend to output concentrated and high-confidence height distributions. Therefore, by using the criterion of "the highest probability exceeding the preset confidence threshold," road and non-road pixels can be effectively distinguished without introducing an additional semantic segmentation module. This simplifies the system architecture, avoids interference from non-ground objects on the road surface, and ultimately generates smoother and more accurate road surface data.

[0052] S230. Based on the road height range, height probability distribution, number of height bins, and density of height bins, the location information of the height bins is obtained.

[0053] The location information of the height bins is calculated based on the road height range, height probability distribution, number of height bins, and density of height bins. The expression for this is:

[0054] in, It can represent the lower boundary of the ground reference height. It can represent the upper boundary of the ground reference height. This can represent the ground reference height, where 'i' represents the pixel index information. This can represent the height bin position information of pixel i, and clip(x, 0, N-1) can represent the truncation function. The range is limited to [0, N-1] to prevent out-of-bounds operations; N can represent the number of height buckets, sign(·) can represent the sign function, h can represent the height value in the height probability distribution, and α can represent the parameter that controls the density of the height buckets.

[0055] In this embodiment, the height regression is discretized into a one-hot encoding form. For example, if the height value is 1.5 meters, multiple height bins may include [-3,-1.92], [-1.92,-1.08], [-1.08,-0.48], [-0.48,-0.12], [-0.12,0], [0,0.12], [0.12,0.48], [0.48,1.08], [1.08,1.92], [1.92,3], and the corresponding position information is [0,N-1]. Through the above formula, the position information of the height bin corresponding to the pixel with a height value of 1.5 meters can be obtained as 8, and the one-hot encoding vector can be [0,0,0,0,0,0,0,0,1,0].

[0056] S240. Based on the position information of the height buckets, the pixels used to construct the road surface data in the enhanced feature information are projected into three-dimensional space and transformed into the vehicle coordinate system to obtain three-dimensional wedge features.

[0057] The vehicle coordinate system can be understood as the coordinate system under autonomous driving vehicles.

[0058] S250. Pooling, convolution, and regression functions are applied to the three-dimensional wedge features to obtain road surface data.

[0059] In this embodiment, the pooling operation can be a voxel pooling operation. By using voxel pooling to converge the 3D wedge features in the height direction, the bird's-eye view feature Fbev can be obtained. Multi-layer convolution can be used to extract local and global geometric features on the bird's-eye view plane, and these local and global geometric features can be fused. A regression function can be used to process the feature fusion result and output the road height offset and / or surface undulation information for each bird's-eye view grid, thereby predicting the 3D geometric structure of the road and obtaining road surface data.

[0060] In an optional embodiment, based on the position information of the height buckets, the pixels used to construct road surface data in the enhanced feature information are projected into three-dimensional space and transformed into the vehicle coordinate system to obtain three-dimensional wedge features, including: S240a. Based on the intrinsic parameter matrix of the shooting device and the first transformation matrix, the pixels used to construct road surface data in the enhanced feature information are transformed to the virtual coordinate system to obtain the first surface data.

[0061] The first transformation matrix represents the transformation relationship between the coordinate system of the shooting device and the virtual coordinate system. The virtual coordinate system can be obtained by rotating the coordinate system of the shooting device, and the vertical axis of the virtual coordinate system is perpendicular to the ground. In one implementation, in the coordinate system of the shooting device, the horizontal axis is positive along the right, the vertical axis is positive along the down, and the forward vertical axis is positive. The origin of the virtual coordinate system coincides with the origin of the coordinate system of the shooting device, the horizontal axis is positive along the right, the vertical axis is positive along the ground, and the forward vertical axis is positive along the ground. It should be noted that this is only an exemplary way of constructing the coordinate system of the shooting device and the virtual coordinate system, and is not limited to this.

[0062] For each pixel in the enhanced feature information A reference plane parallel to the enhanced feature information and at a fixed distance can be constructed. In one implementation, the fixed value can be 1m, but it is not limited to this. Based on this reference plane, the enhanced feature information can be transformed into the coordinate system of the imaging device using the intrinsic parameter matrix of the imaging device. Its expression can be: ,in, This represents the enhanced feature information in the coordinate system of the shooting device. This represents the intrinsic parameter matrix of the shooting device. The x-axis represents the enhanced feature information. The vertical axis represents the enhanced feature information.

[0063] Then, based on the first transformation matrix, the enhanced feature information in the shooting device coordinate system is transformed to the virtual coordinate system to obtain the first surface data, which can be expressed as follows: ,in, This represents the data of the first surface. This represents the first transformation matrix.

[0064] S240b: Calculate the three-dimensional projection points of the height buckets based on the height from the origin of the virtual coordinate system to the ground, the position information of the height buckets, the longitudinal coordinate values ​​of the virtual coordinate system, and the data of the first curved surface.

[0065] like Figure 4 As shown, the height H from the origin of the virtual coordinate system to the ground is used. Based on the principle of similar triangles, the height of each pixel in the enhanced feature information can be obtained. Corresponding three-dimensional projection points .in, This represents the vertical coordinate value of the virtual coordinate system.

[0066] S240c: The three-dimensional wedge features are obtained by calculation based on the three-dimensional projection points and the second transformation matrix; the second transformation matrix represents the transformation relationship between the virtual coordinate system and the vehicle coordinate system, and is obtained based on the external parameter matrix of the shooting device and the first transformation matrix.

[0067] The expression for calculating the three-dimensional wedge feature can be: ,in, This represents three-dimensional wedge-shaped features, i.e., three-dimensional point features with high semantic depth. Let represent the second transformation matrix, where is the second transformation matrix. First transformation matrix extrinsic matrix of shooting equipment The relationship is .

[0068] Data collected by traditional roadside cameras, projected onto the camera's coordinate system, is from a top-down perspective and has a wedge-shaped shape, which presents challenges for subsequent processing. This application's embodiment utilizes a virtual coordinate system to assist in converting the camera's coordinate system to the autonomous vehicle's coordinate system. Compared to a level view, the data features can be viewed as cuboids parallel to the road surface, facilitating not only data processing but also conversion to a bird's-eye view, thus improving computational efficiency.

[0069] Furthermore, when there is a distance between the adjusted height probability distribution and the ground reference height, the extrinsic parameter matrix of the imaging device can be corrected based on the adjusted distance information to obtain a corrected extrinsic parameter matrix of the imaging device. The three-dimensional wedge feature is then calculated based on this corrected extrinsic parameter matrix and the second transformation matrix. This improves the accuracy of road surface data, reduces the influence of the imaging device's extrinsic parameters, and ultimately enhances the accuracy of the final result.

[0070] In one optional embodiment, the lane segmentation module may include an image encoder, a cue word encoder, a mask decoder, a semantic detection head, and a memory unit. The lane segmentation module processes roadside image data through a pre-trained method to obtain lane semantic segmentation data, which may include: S300: The roadside image data is encoded using an image encoder to obtain high-dimensional image feature data.

[0071] S310. The prompt information is encoded according to the prompt word encoder to obtain the prompt embedding data.

[0072] S320. The high-dimensional image feature data and the cue embedding data are fused according to the mask decoder to obtain a multi-channel feature mask.

[0073] S330. Perform channel attention and convolution operations on the multi-channel feature mask based on the semantic detection head to obtain the current semantic segmentation data.

[0074] S340. Based on the memory unit, the current semantic segmentation data and the historical semantic segmentation data are fused to obtain lane semantic segmentation data.

[0075] In one optional embodiment, based on the vehicle-view lane dataset, the lane segmentation module is trained in a first stage using an optimization objective constructed with mask segmentation loss and mask quality loss to obtain first optimization parameters; based on the roadside lane segmentation dataset and the first optimization parameters, the lane segmentation module is trained in a second stage using an optimization objective constructed with mask contour loss to obtain second optimization parameters; based on the roadside lane semantic dataset and the second optimization parameters, the lane segmentation module is trained in a third stage using the topological and logical relationships between lanes to obtain third optimization parameters, which may include: The first stage of training involves inputting the vehicle-view lane dataset into the lane segmentation module to be trained, obtaining the first feature mask output by the mask decoder; calculating the mask segmentation loss and mask quality loss based on the first feature mask, and adjusting the parameters of the prompt word encoder and mask decoder to obtain the first optimized parameters.

[0076] The second stage of training involves inputting the roadside lane segmentation dataset into the lane segmentation module based on the first optimization parameters to obtain the second feature mask output by the mask decoder; calculating the mask segmentation loss, mask quality loss, and mask contour loss based on the second feature mask, and adjusting the parameters of the lane segmentation module to obtain the second optimization parameters.

[0077] The third stage of training involves inputting the roadside lane semantic dataset into the lane segmentation module based on the second optimization parameters, and using the topological and logical relationships between lanes to train the semantic detection head and memory unit to obtain the third optimization parameters.

[0078] In the embodiments of this application, the mask segmentation loss is constructed based on the predicted first or second feature mask and the ground truth mask, and the mask quality loss is constructed based on the relationship between the quality fraction of the predicted first or second feature mask and the actual degree of proximity to the mask. The mask contour loss can also characterize the loss between the mask contour of the predicted second feature mask and the ground truth mask contour.

[0079] In the first phase of training, the cue word encoder and mask decoder are transferred to train using the vehicle-view lane dataset. Other modules besides the cue word encoder and mask decoder, such as the image encoder, semantic detection head and memory unit, are frozen to avoid destroying the general image features learned during pre-training.

[0080] Based on the mask segmentation loss and mask quality loss, a loss function is constructed according to the one used during training, and its expression can be:

[0081] in, Indicates mask segmentation loss, Indicates mask quality loss. The weighting parameter represents the mask quality loss in the first stage.

[0082] In one implementation, the mask segmentation loss can be obtained by supervising the pixel-level mask using a binary cross-entropy loss, and its expression can be:

[0083] Where yi∈{0,1} is a mask that can represent the actual annotation. ∈[0,1] can represent the first feature mask for prediction, ɛ=10 -5 To ensure the parameters are adjustable and to prevent numerical instability, It can represent the total number of pixels.

[0084] In one implementation, the mask quality loss can be constructed by aligning the mask quality fraction with the actual proximity to the mask calculated by IoU, and its expression can be:

[0085] in, This can represent the quality score of the first feature mask in the prediction. This can represent the intersection area between the predicted first feature mask and the ground truth labeled mask. It can represent the area of ​​the union between the predicted first feature mask and the ground truth labeled mask.

[0086] In the second stage of training, the model is further fine-tuned using a roadside lane segmentation dataset to adapt it to the imaging characteristics of high camera positions, long distances, fine-grained lane lines, and complex backgrounds. The total loss function can then be expressed as:

[0087] in, It can represent the mask contour loss. The weighted adjustment parameter can represent the mask quality loss in the second stage. The weighted sky parameter can represent the mask contour loss in the second stage.

[0088] In one implementation, the mask contour loss can be expressed as:

[0089] in, This represents the Sobel gradient operator.

[0090] In the embodiments of this application, by introducing edge loss, the supervision of distant and blurred areas is strengthened, thereby enhancing the shape consistency and edge clarity of the lane line contour.

[0091] In the third training phase, while training the semantic detection head and memory unit, other modules besides the semantic detection head and memory unit, such as the image encoder, cue word encoder, and mask encoder, can be frozen.

[0092] After the roadside lane semantic dataset is input into the view feature extraction module based on the second optimization parameters, the view feature extraction module can perform the following process: The roadside lane semantic dataset is processed sequentially by an image encoder, a cue word encoder, and a mask decoder to obtain a third feature mask. This third feature mask is then processed by a semantic detection head to obtain the current semantic segmentation data. A memory module is applied to the current semantic segmentation data to fuse it with historical semantic segmentation data, resulting in predicted lane semantic segmentation data. An optimization objective is constructed based on the predicted and actual lane semantic segmentation data, and the parameters of the semantic detection head and memory module are trained to obtain the third optimization parameters. The process of fusing the current and historical semantic segmentation data is described below.

[0093] In one optional embodiment, lane semantic segmentation data is obtained by fusing current semantic segmentation data and historical semantic segmentation data according to the memory unit, including: calculating first data based on the current semantic segmentation data and its corresponding first pixel confidence; calculating second data based on the historical semantic segmentation data and its corresponding second pixel confidence; and calculating lane semantic segmentation data based on the first data, the second data, the first pixel confidence, and the second pixel confidence.

[0094] In this embodiment, lane semantic segmentation data can be obtained by processing the current semantic segmentation data and historical semantic segmentation data output by the mask decoder according to the memory unit. The historical semantic segmentation data can be understood as historical memory, referring to either a single historical semantic segmentation data point or data obtained by fusing multiple historical semantic segmentation data points. Preferably, the mask decoder has 24 output channels, with 23 channels corresponding to different lane line types and one channel serving as the background channel. Based on this, fine depiction of multiple lane line structures in complex road scenes can be achieved.

[0095] In one implementation, the expression for calculating lane semantic segmentation data can be:

[0096] in, S can represent lane semantic segmentation data, and S can represent the current semantic segmentation data. Each pixel in S corresponds to a set of segmentation probabilities. It can represent the confidence level of the first pixel corresponding to the current semantic segmentation data. It can represent historical semantic segmentation data. It can represent the confidence level of the second pixel corresponding to historical semantic segmentation data. This can represent tiny values ​​to prevent division by zero. The first pixel confidence score and the second pixel confidence score are numerical values ​​that measure how "trustworthy" the current pixel prediction is, derived from the maximum class probability output by the classification function.

[0097] The above formula achieves error suppression and memory completion by adjusting the confidence weights. When the current semantic segmentation data is mis-segmented due to noise or shadows, the corresponding... The results will also be lower, and the final result will be closer to historical semantic segmentation data. In cases where occlusion leads to loss, =0, allowing for priority selection of historical priors over reliance on the current frame prediction affected by occlusion. This results in the completion of the continuous structure of the originally occluded region after fusion. In this case, the confidence level of the second historical pixel... This plays the role of "geometric prior," ensuring the structural continuity of the segmentation results rather than being disrupted by invisible regions in the current semantic segmentation data. Based on this design, the embodiments of this application can correct mis-segmented pixels in the current semantic segmentation data and fill in missing regions caused by occlusion, thereby gradually accumulating and generating more complete, continuous, and semantically consistent lane semantic segmentation data, achieving more refined multi-class lane semantic segmentation and enhanced consistency of cross-frame masks. During the update process, lightweight convolutional smoothing and edge enhancement can be applied to historical memory to further ensure the detail and continuity of lane lines.

[0098] In an optional embodiment, the calculation based on road surface data and lane semantic segmentation data to obtain a three-dimensional lane detection result may include: S400. Based on the intrinsic parameter matrix of the shooting device, the lane semantic segmentation data is mapped to the coordinate system of the shooting device to obtain the first ray direction vector.

[0099] In one implementation, the expression for calculating the first ray direction vector can be: ,in, Represents the direction vector of the first ray. This represents the intrinsic parameter matrix of the shooting device.

[0100] S410. Based on the extrinsic parameter matrix of the shooting device, map the first ray direction vector to the vehicle coordinate system to obtain the second ray direction vector.

[0101] In one implementation, the expression for calculating the direction vector of the second ray can be: ,in, It can represent the direction vector of the second ray. It can represent the extrinsic parameter matrix of the shooting device.

[0102] S420. Based on the second ray direction vector, obtain the ray information extending along the viewing angle of the shooting device in the vehicle coordinate system; calculate the intersection point of the road surface data and the ray information to obtain the three-dimensional point; map the lane semantic segmentation data to the corresponding three-dimensional point to obtain the three-dimensional lane detection result.

[0103] In one implementation, Newton's iteration can be used to solve for the ray information. Road surface data The intersection of, where, It can represent the ray information extending along the line of sight of the shooting device in the coordinate system of the autonomous vehicle, and O can represent the coordinates of the origin of the autonomous vehicle coordinate system. It can represent the distance scale factor.

[0104] Define the height difference function: ,in, , , Ray information as a distance scale factor The position of . Taking its derivative, we get: The iterative formula is: By setting an initial value λ0 and iterating repeatedly until... This allows for the precise determination of the intersection points between the ray and the road surface, thus obtaining three-dimensional points. Mapping the two-dimensional lane semantic segmentation data to the corresponding three-dimensional points yields the three-dimensional lane detection results.

[0105] In this embodiment, by iteratively converging to the precise intersection of the ray and the surface, lane semantic segmentation data can be quickly mapped to road surface data, greatly improving computational efficiency. In addition, back-projecting the two-dimensional lane semantic segmentation results onto three-dimensional points avoids semantic drift or breakage caused by depth estimation errors, making the three-dimensional lane structure more coherent and the topology more reasonable, thus improving projection accuracy.

[0106] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method.

[0107] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0108] Based on the same technical concept, corresponding to any of the above embodiments, this application also provides an autonomous driving system.

[0109] refer to Figure 5 The autonomous driving system 500 may include: The acquisition module 501 is used to acquire roadside image data, which comes from the camera equipment of the roadside facilities.

[0110] The calculation module 502 is used to predict the height probability distribution of each pixel in the roadside image data relative to the ground reference height through a pre-trained height estimation module, so as to obtain road surface data. During the training of the height estimation module, random perturbation is added to the extrinsic parameter matrix of the shooting device corresponding to the input image. The perturbated height estimation module is trained under supervision based on the real ground height label of the input image, and the height probability distribution of non-road areas is suppressed in the supervised training. The roadside image data is processed by a pre-trained lane segmentation module to obtain lane semantic segmentation data. During the training of the lane segmentation module, the first stage of training is performed on the lane segmentation module based on the vehicle-view lane dataset, using the optimization objective constructed by mask segmentation loss and mask quality loss to obtain the first optimization parameters. Based on the roadside lane segmentation dataset and the first optimization parameters, the second stage of training is performed on the lane segmentation module based on the optimization objective constructed by mask contour loss to obtain the second optimization parameters. Based on the roadside lane semantic dataset and the second optimization parameters, the lane segmentation module is trained in the third stage using the topological and logical association between lanes to obtain the third optimization parameters. The three-dimensional lane detection result is obtained by calculation based on the road surface data and the lane semantic segmentation data.

[0111] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0112] The apparatus of the above embodiments is used to implement a lane detection method in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0113] Based on the same technical concept, corresponding to any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a lane detection method of any of the above embodiments.

[0114] Figure 6This embodiment illustrates a more specific hardware structure of an electronic device. The device may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0115] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0116] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0117] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0118] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0119] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0120] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0121] The electronic devices described above are used to implement a lane detection method in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0122] Based on the same technical concept, corresponding to any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing a computer to execute a lane detection method as described in any of the above embodiments.

[0123] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0124] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute a lane detection method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0125] It should be noted that the embodiments of this application can also be further described in the following ways: A lane detection method, applied to roadside facilities, includes: acquiring roadside image data, wherein the roadside image data comes from the camera of the roadside facility; A pre-trained height estimation module predicts the height probability distribution of each pixel in the roadside image data relative to the ground reference height to obtain road surface data. During the training of the height estimation module, random perturbations are added to the extrinsic parameter matrix of the shooting device corresponding to the input image. The perturbated height estimation module is then trained under supervision based on the real ground height label of the input image, suppressing the height probability distribution in non-road areas during supervised training. A pre-trained lane segmentation module processes the roadside image data to obtain lane semantic segmentation data. During the training of the lane segmentation module, the first stage of training is performed based on the vehicle-view lane dataset, using an optimization objective constructed with mask segmentation loss and mask quality loss to obtain the first optimization parameters. Based on the roadside lane segmentation dataset and the first optimization parameters, the second stage of training is performed using an optimization objective constructed with mask contour loss to obtain the second optimization parameters. Based on the roadside lane semantic dataset and the second optimization parameters, the lane segmentation module is trained in the third stage using the topological and logical relationships between lanes to obtain the third optimization parameters. The 3D lane detection results are obtained by calculating based on the road surface data and the lane semantic segmentation data.

[0126] Optionally, road surface data is obtained by predicting the height probability distribution of each pixel in the roadside image data relative to the ground reference height through a pre-trained height estimation module. This includes: extracting multi-scale visual features from the roadside image data to obtain high-dimensional feature information; extracting contextual information and residual features from the high-dimensional feature information to obtain enhanced feature information; processing the enhanced feature information using deformable convolution to predict the height distribution probability of each pixel in multiple height bins; the multiple height bins are obtained by dividing the road height interval according to a preset division rule; for any pixel, if the highest probability in the height probability distribution of the pixel exceeds a preset confidence threshold, the height probability distribution of the pixel is used to construct the road surface data; calculating the position information of the height bins based on the road height interval, height probability distribution, number of height bins, and density of height bins; projecting the pixels used to construct the road surface data from the enhanced feature information into three-dimensional space and transforming them into the vehicle coordinate system based on the position information of the height bins to obtain three-dimensional wedge features; and processing the three-dimensional wedge features using pooling, convolution, and regression functions to obtain the road surface data.

[0127] Optionally, based on the position information of the height buckets, the pixels used to construct road surface data in the enhanced feature information are projected into three-dimensional space and transformed into the vehicle coordinate system to obtain three-dimensional wedge features. This includes: transforming the pixels used to construct road surface data in the enhanced feature information into a virtual coordinate system based on the intrinsic parameter matrix of the shooting device and the first transformation matrix to obtain first surface data; the first transformation matrix represents the transformation relationship between the coordinate system of the shooting device and the virtual coordinate system, where the virtual coordinate system is obtained by rotating the coordinate system of the shooting device, and the vertical axis of the virtual coordinate system is perpendicular to the ground; the three-dimensional projection points of the height buckets are calculated based on the height from the origin of the virtual coordinate system to the ground, the position information of the height buckets, the vertical coordinate values ​​of the virtual coordinate system, and the first surface data; the three-dimensional wedge features are obtained by calculating based on the three-dimensional projection points and the second transformation matrix; the second transformation matrix represents the transformation relationship between the virtual coordinate system and the vehicle coordinate system, and is obtained based on the extrinsic parameter matrix of the shooting device and the first transformation matrix.

[0128] Optionally, the lane segmentation module includes an image encoder, a cue word encoder, a mask decoder, a semantic detection head, and a memory unit. The pre-trained lane segmentation module processes roadside image data to obtain lane semantic segmentation data, including: encoding the roadside image data using the image encoder to obtain high-dimensional image feature data; encoding the cue information using the cue word encoder to obtain cue embedding data; fusing the high-dimensional image feature data and the cue embedding data using the mask decoder to obtain a multi-channel feature mask; performing channel attention and convolution operations on the multi-channel feature mask using the semantic detection head to obtain the current semantic segmentation data; and fusing the current semantic segmentation data and historical semantic segmentation data using the memory unit to obtain the lane semantic segmentation data.

[0129] Optionally, lane semantic segmentation data is obtained by fusing the current semantic segmentation data and historical semantic segmentation data according to the memory unit, including: calculating the first data based on the current semantic segmentation data and its corresponding first pixel confidence; calculating the second data based on the historical semantic segmentation data and its corresponding second pixel confidence; and calculating the lane semantic segmentation data based on the first data, the second data, the first pixel confidence, and the second pixel confidence.

[0130] Optionally, the lane segmentation module includes an image encoder, a cue word encoder, a mask decoder, a semantic detection head, and a memory unit. Based on the vehicle-view lane dataset, the lane segmentation module is trained in the first stage using an optimization objective constructed with mask segmentation loss and mask quality loss to obtain first optimized parameters. Based on the roadside lane segmentation dataset and the first optimized parameters, the lane segmentation module is trained in the second stage using an optimization objective constructed with mask contour loss to obtain second optimized parameters. Based on the roadside lane semantic dataset and the second optimized parameters, the lane segmentation module is trained in the third stage using the topological and logical relationships between lanes to obtain third optimized parameters. This includes: inputting the vehicle-view lane dataset into the lane segmentation module to be trained to obtain a first feature mask output by the mask decoder; calculating the mask segmentation loss and mask quality loss based on the first feature mask, and adjusting the cue word encoder and mask... The parameters of the mask decoder are used to obtain the first optimized parameters. The roadside lane segmentation dataset is input into the lane segmentation module based on the first optimized parameters to obtain the second feature mask output by the mask decoder. The mask segmentation loss, mask quality loss, and mask contour loss are calculated based on the second feature mask, and the parameters of the lane segmentation module are adjusted to obtain the second optimized parameters. The roadside lane semantic dataset is input into the lane segmentation module based on the second optimized parameters, and the semantic detection head and memory unit are trained using the topological and logical relationships between lanes to obtain the third optimized parameters. The mask segmentation loss is constructed based on the predicted first or second feature mask and the ground truth mask. The mask quality loss is constructed based on the quality score of the predicted first or second feature mask and the degree of closeness to the actual mask. The mask contour loss is constructed based on the mask contour of the predicted second feature mask and the ground truth mask contour.

[0131] Optionally, the 3D lane detection result is obtained by calculating based on road surface data and lane semantic segmentation data, including: mapping the lane semantic segmentation data to the coordinate system of the camera based on the intrinsic parameter matrix of the camera to obtain a first ray direction vector; mapping the first ray direction vector to the vehicle coordinate system based on the extrinsic parameter matrix of the camera to obtain a second ray direction vector; obtaining ray information extending along the viewing angle of the camera in the vehicle coordinate system based on the second ray direction vector; calculating the intersection point of the road surface data and the ray information to obtain a 3D point; and mapping the lane semantic segmentation data to the corresponding 3D point to obtain the 3D lane detection result.

[0132] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the above embodiments of this application, which are not provided in detail for the sake of brevity.

[0133] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0134] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0135] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A lane detection method, applied to roadside facilities, characterized in that, include: Acquire roadside image data, wherein the roadside image data comes from the camera device of the roadside facility; The height probability distribution of each pixel in the roadside image data relative to the ground reference height is predicted by a pre-trained height estimation module to obtain road surface data. During the training of the height estimation module, random perturbation is added to the extrinsic parameter matrix of the shooting device corresponding to the input image. The perturbated height estimation module is trained under supervision based on the real ground height label of the input image. In the supervised training, the height probability distribution of non-road areas is suppressed. The roadside image data is processed by a pre-trained lane segmentation module to obtain lane semantic segmentation data. During the training of the lane segmentation module, the lane segmentation module is trained in the first stage based on the vehicle-view lane dataset and an optimization objective constructed using mask segmentation loss and mask quality loss to obtain the first optimization parameters. Based on the roadside lane segmentation dataset and the first optimization parameters, the lane segmentation module is trained in the second stage using an optimization objective constructed with mask contour loss to obtain the second optimization parameters. Based on the roadside lane semantic dataset and the second optimization parameters, the lane segmentation module is trained in the third stage using the topological and logical relationships between lanes to obtain the third optimization parameters. The three-dimensional lane detection results are obtained by calculating based on the road surface data and the lane semantic segmentation data.

2. The method according to claim 1, characterized in that, The road surface data is obtained by predicting the height probability distribution of each pixel in the roadside image data relative to the ground reference height using a pre-trained height estimation module, including: Multi-scale visual features are extracted from the roadside image data to obtain high-dimensional feature information; Contextual information and residual features are extracted based on the high-dimensional feature information to obtain enhanced feature information; The enhanced feature information is processed using deformable convolution to predict the height distribution probability of each pixel in multiple height bins; the multiple height bins are obtained by dividing the road height range according to a preset division rule; For any pixel, if the highest probability in the height probability distribution of the pixel exceeds a preset confidence threshold, then the height probability distribution of the pixel is used to construct the road surface data. The location information of the height bins is obtained by calculating based on the road height range, the height probability distribution, the number of height bins, and the density of the height bins; Based on the position information of the height buckets, the pixels in the enhanced feature information used to construct the road surface data are projected into three-dimensional space and transformed into the vehicle coordinate system to obtain three-dimensional wedge features; The three-dimensional wedge features are processed by pooling, convolution, and regression functions to obtain the road surface data.

3. The method according to claim 2, characterized in that, Based on the position information of the height buckets, the pixels in the enhanced feature information used to construct the road surface data are projected into three-dimensional space and transformed into the vehicle coordinate system to obtain three-dimensional wedge features, including: Based on the intrinsic parameter matrix of the shooting device and the first transformation matrix, the pixels in the enhanced feature information used to construct the road surface data are transformed to a virtual coordinate system to obtain the first surface data; the first transformation matrix represents the transformation relationship between the coordinate system of the shooting device and the virtual coordinate system, the virtual coordinate system is obtained by rotating the coordinate system of the shooting device, and the vertical axis of the virtual coordinate system is perpendicular to the ground; The three-dimensional projection points of the height buckets are calculated based on the height from the origin of the virtual coordinate system to the ground, the position information of the height buckets, the longitudinal coordinate values ​​of the virtual coordinate system, and the first surface data. The three-dimensional wedge feature is obtained by calculation based on the three-dimensional projection points and the second transformation matrix; the second transformation matrix represents the transformation relationship between the virtual coordinate system and the vehicle coordinate system, and is obtained based on the extrinsic parameter matrix of the shooting device and the first transformation matrix.

4. The method according to claim 1, characterized in that, The lane segmentation module includes an image encoder, a cue word encoder, a mask decoder, a semantic detection head, and a memory unit. It processes the roadside image data using a pre-trained lane segmentation module to obtain lane semantic segmentation data, including: The roadside image data is encoded using the image encoder to obtain high-dimensional image feature data. The prompt information is encoded according to the prompt word encoder to obtain the prompt embedding data; The high-dimensional image feature data and the cue embedding data are fused according to the mask decoder to obtain a multi-channel feature mask; The semantic detection head performs channel attention and convolution operations on the multi-channel feature mask to obtain the current semantic segmentation data. The lane semantic segmentation data is obtained by fusing the current semantic segmentation data and the historical semantic segmentation data according to the memory unit.

5. The method according to claim 4, characterized in that, The lane semantic segmentation data is obtained by fusing the current semantic segmentation data and historical semantic segmentation data according to the memory unit, including: The first data is obtained by calculating based on the current semantic segmentation data and its corresponding first pixel confidence score; The second data is obtained by calculating based on the historical semantic segmentation data and its corresponding second pixel confidence. Lane semantic segmentation data is obtained by calculating based on the first data, the second data, the first pixel confidence score, and the second pixel confidence score.

6. The method according to claim 1, characterized in that, The lane segmentation module includes an image encoder, a cue word encoder, a mask decoder, a semantic detection head, and a memory unit. Based on the vehicle-view lane dataset, the lane segmentation module is trained in the first stage using an optimization objective constructed with mask segmentation loss and mask quality loss to obtain the first optimization parameters. Based on the roadside lane segmentation dataset and the first optimization parameters, the lane segmentation module is trained in the second stage using an optimization objective constructed with mask contour loss to obtain the second optimization parameters. Based on the roadside lane semantic dataset and the second optimization parameters, the lane segmentation module is trained in a third stage using the topological and logical relationships between lanes to obtain the third optimization parameters, including: The vehicle-view lane dataset is input into the lane segmentation module to be trained to obtain the first feature mask output by the mask decoder; the mask segmentation loss and mask quality loss are calculated based on the first feature mask, and the parameters of the prompt word encoder and the mask decoder are adjusted to obtain the first optimized parameters; The roadside lane segmentation dataset is input into the lane segmentation module based on the first optimization parameters to obtain the second feature mask output by the mask decoder; the mask segmentation loss, mask quality loss and mask contour loss are calculated based on the second feature mask, and the parameters of the lane segmentation module are adjusted to obtain the second optimization parameters; The roadside lane semantic dataset is input into the lane segmentation module based on the second optimization parameters. The semantic detection head and the memory unit are trained using the topological and logical relationships between lanes to obtain the third optimization parameters. The mask segmentation loss is constructed based on the predicted first feature mask or second feature mask and the actual labeled mask; the mask quality loss is constructed based on the predicted quality fraction of the first feature mask or second feature mask and the degree of closeness to the actual mask; and the mask contour loss is constructed based on the predicted mask contour of the second feature mask and the actual mask contour.

7. The method according to claim 1, characterized in that, Based on the road surface data and the lane semantic segmentation data, a three-dimensional lane detection result is obtained, including: Based on the intrinsic parameter matrix of the shooting device, the lane semantic segmentation data is mapped to the coordinate system of the shooting device to obtain the first ray direction vector; Based on the extrinsic parameter matrix of the shooting device, the first ray direction vector is mapped to the vehicle coordinate system to obtain the second ray direction vector; Based on the second ray direction vector, ray information extending along the viewing angle of the shooting device in the vehicle coordinate system is obtained; Calculate the intersection points of the road surface data and the ray information to obtain three-dimensional points; The lane semantic segmentation data is mapped to the corresponding 3D points to obtain the 3D lane detection results.

8. An autonomous driving system, characterized in that, include: An acquisition module is used to acquire roadside image data, which comes from the camera device of the roadside facility; A calculation module is used to predict the height probability distribution of each pixel in the roadside image data relative to the ground reference height through a pre-trained height estimation module, so as to obtain road surface data. During the training of the height estimation module, random perturbation is added to the extrinsic parameter matrix of the shooting device corresponding to the input image, and the perturbated height estimation module is supervised training based on the real ground height label of the input image. In the supervised training, the height probability distribution of non-road areas is suppressed. The roadside image data is processed by a pre-trained lane segmentation module to obtain lane semantic segmentation data. During the training of the lane segmentation module, the lane segmentation module is trained in the first stage based on the vehicle-view lane dataset and an optimization objective constructed using mask segmentation loss and mask quality loss to obtain the first optimization parameters. Based on the roadside lane segmentation dataset and the first optimization parameters, the lane segmentation module is trained in the second stage using an optimization objective constructed with mask contour loss to obtain the second optimization parameters. Based on the roadside lane semantic dataset and the second optimization parameters, the lane segmentation module is trained in the third stage using the topological and logical relationships between lanes to obtain the third optimization parameters; the three-dimensional lane detection results are obtained by calculating based on the road surface data and the lane semantic segmentation data.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method as described in any one of claims 1 to 7.

Images