A three-dimensional road line extraction method and related device

Abstract

The application provides a three-dimensional road line extraction method and related equipment, comprising: obtaining binocular image data and inertial navigation positioning data of a research area based on a vehicle-mounted platform; performing parallax estimation on the binocular image data according to the inertial navigation positioning data to obtain a visual three-dimensional point cloud; determining a region with the binocular camera as the center and a preset distance as the radius as a target region in the research area, extracting the visual three-dimensional point cloud in the target region, and converting the visual three-dimensional point cloud in the target region to obtain a road surface bird's-eye view and a point cloud elevation map of the target region; inputting the road surface bird's-eye view and the point cloud elevation map into a lane line detection network to extract a road line, and obtaining a three-dimensional road line and a three-dimensional coordinate of the road line; compared with the prior art, the problems of low precision, poor robustness and high detection cost of the current road line recognition method in vehicle-mounted images are overcome, and the efficiency and quality of road line detection in vehicle-mounted images are improved.

Description

Technical Field

[0001] This invention relates to the fields of photogrammetry, high-precision positioning, and artificial intelligence, and particularly to a method and related equipment for extracting three-dimensional road lines. Background Technology

[0002] With rapid urbanization and increasingly sophisticated road infrastructure, emerging technology industries such as autonomous driving and intelligent transportation systems are booming. Traditional maps, with their basic spatial vector road data, primarily store and represent two-dimensional road information as line elements rather than the true three-dimensional form of roads. This is no longer sufficient to meet the needs of emerging technology industries like autonomous driving, 3D scene navigation, and digital twins. High-precision road network maps, characterized by high accuracy, timeliness, and rich semantics, are fundamental to the safe operation of related intelligent systems. Road lines define the left and right boundaries of a lane. Generally, lane boundaries are determined by road lines printed on the ground, roughly categorized into five types: single dashed lines, single solid lines, double solid lines, double dashed lines, and dashed-solid lines. These lines ensure vehicles travel within the correct lane, providing safety guarantees and representing the most crucial road element in the creation of high-precision maps for autonomous driving. However, the acquisition of 3D road line information requires high-quality, high-precision data. Previous research has been limited by data acquisition constraints. Traditional mapping techniques and LiDAR methods face numerous challenges in acquiring city-level 3D road information. Therefore, how to automate and reduce the cost of extracting 3D lane information has become a current research hotspot.

[0003] Domestic and international scholars have been researching 3D road extraction methods for many years. Current 3D road line acquisition methods can be categorized based on data sources into two types: methods based on laser point clouds and methods based on vehicle-mounted images. Vehicle-mounted LiDAR point clouds are a crucial source of 3D road element data, and intelligently extracting 3D road information from radar point clouds is a significant problem that needs to be solved in the field of 3D modeling. Dinesh et al., based on the characteristic that road surface height varies along the road cross-section, performed height histogram statistics on the point clouds obtained from each scan line to extract road surface information. Abujadrous et al. optimized the road obtained from histogram statistics by combining prior features such as road width and road curvature. Jaakkola et al. noted the significant height difference between the road edges on both sides and the road itself, extracted the road boundary based on the elevation map generated along the radar scan line direction, and finally constructed the road surface based on Delaunay triangulation and road width information. These methods are limited by the selection of ground elevation thresholds. They can extract roads well in simple and regular road environments, but have shortcomings in complex and irregular road scenes. Yuan et al. proposed a fuzzy clustering method based on maximum entropy theory to cluster point clouds along the same scan line and optimize adjacent clustering results to extract road surface elements. Milioto et al. used spherical projection to perform depth-like image transformation on laser point clouds and used convolutional neural networks to perform semantic segmentation on the transformation results, which can effectively extract lane regions.

[0004] Overall, there is still a lack of efficient 3D road line extraction algorithms. The main problems with existing methods include:

[0005] (1) Current research on automated extraction of road 3D information is mostly a separate task, that is, the 3D position information of road line is separated from the road line detection task, which makes it difficult to achieve complementary advantages;

[0006] (2) At the application level, there are many limitations such as cost constraints, environmental impact, and acquisition efficiency, which make it impossible to support the collection of three-dimensional road information on a full range and large scale.

[0007] (3) Directly detecting road lines from vehicle images faces many challenges such as occlusion and nonlinear deformation of road lines, which affect the final extraction quality of three-dimensional road lines. Summary of the Invention

[0008] This invention provides a method and related equipment for extracting three-dimensional road lines, with the aim of improving the efficiency and quality of three-dimensional road line extraction.

[0009] To achieve the above objectives, the present invention provides a three-dimensional road line extraction method, comprising:

[0010] Step 1: Acquire binocular image data and inertial navigation positioning data of the study area based on the vehicle-mounted platform;

[0011] Step 2: Based on the inertial navigation positioning data, perform disparity estimation on the binocular image data to obtain the visual 3D point cloud;

[0012] Step 3: In the study area, determine the area with the binocular camera as the center and a preset distance as the radius as the target area, extract the visual 3D point cloud in the target area, and transform the visual 3D point cloud in the target area to obtain the road surface bird's-eye view and point cloud elevation map of the target area.

[0013] Step 4: Input the road surface bird's-eye view and point cloud elevation map into the lane line detection network. Using the point cloud elevation map as auxiliary information, extract the road line elements from the road surface bird's-eye view. Calculate the 3D coordinate information of the road line in camera space based on the image coordinates in the road line elements. Transform the 3D coordinate information in camera space according to the camera space to geographic space transformation matrix to obtain the 3D road line and its 3D coordinates.

[0014] Furthermore, prior to step 2, the following steps are also included:

[0015] Multiple calibration board images of the study area were captured from different angles by the binocular camera on the vehicle platform. The calibration board images were then calibrated to obtain the camera intrinsic parameter matrix and distortion coefficients.

[0016] The rotation and translation parameters are calculated based on the absolute spatial pose of the left camera of the binocular camera and the pose of the positioning system to obtain the camera extrinsic parameter matrix.

[0017] Furthermore, step 2 includes:

[0018] Preprocessing is performed on the binocular image data and inertial navigation positioning data based on the camera's intrinsic parameter matrix, distortion coefficients, and extrinsic parameter matrix.

[0019] Based on the preprocessed inertial navigation positioning data, disparity estimation is performed on the preprocessed binocular image data.

[0020] Furthermore, the disparity value is estimated from the preprocessed binocular image data, including:

[0021] A deep learning-based stereo matching model is used to estimate the disparity of the preprocessed binocular image data to obtain the disparity value of each pixel.

[0022] The 3D coordinates of each pixel are calculated based on the disparity value of each pixel, as shown in the following formula:

[0023] z = f * b / d

[0024] x = x l *z / f

[0025] y = y l *z / f

[0026] Where, x l ,y l is the image coordinate of the binocular image, f is the camera focal length, b is the distance between the left and right cameras of the binocular camera, and d is the parallax value;

[0027] A 3D visual point cloud is obtained based on the 3D coordinates of each pixel.

[0028] Furthermore, step 3 includes:

[0029] Within the study area, a region centered on the binocular camera and with a preset distance as its radius is defined as the target region.

[0030] Extract the visual 3D point cloud within the target area;

[0031] The visual 3D point cloud of the target area in 3D space is projected onto 2D space to obtain a bird's-eye view of the road surface and a point cloud elevation map of the target area. The conversion formula is as follows:

[0032] Scal x =pic w / (x max -x min )

[0033] Scal y =pic h / (y max -y min )

[0034] X = int(Scal x *(x0-x min ))

[0035] Y = pic h -int(Scal y *(y0-y min ))

[0036] P XY =value(x,y,z)

[0037] Among them, pic w pic h The width and height of the target image to be converted are both set to 400, x max y max x represents the maximum value of the visual 3D point cloud within the target region on the coordinate axis. min y minThe minimum value of the visual 3D point cloud in the target area on the coordinate axis is XY, which are the coordinates projected onto the plane. P represents a pixel. When the corresponding pixel value is the color of the visual 3D point cloud, a bird's-eye view of the road surface is obtained; when the corresponding pixel value is the height value of the visual 3D point cloud, a point cloud elevation map is obtained.

[0038] The present invention also provides a three-dimensional road line extraction device, comprising:

[0039] The acquisition module is used to acquire binocular image data and inertial navigation positioning data of the study area based on the vehicle-mounted platform.

[0040] The disparity estimation module is used to estimate the disparity of the binocular image data based on the inertial navigation positioning data to obtain a visual 3D point cloud.

[0041] The point cloud extraction module is used to determine the target area in the study area with the binocular camera as the center and a preset distance as the radius, extract the visual 3D point cloud in the target area, and transform the visual 3D point cloud in the target area to obtain the road surface bird's-eye view and point cloud elevation map of the target area.

[0042] The road line extraction module is used to input the road surface bird's-eye view and point cloud elevation map into the lane line detection network, and extract the road line elements from the road surface bird's-eye view using the point cloud elevation map as auxiliary information; calculate the three-dimensional coordinate information of the road line in camera space based on the image coordinates in the road line elements; and transform the three-dimensional coordinate information in camera space according to the camera space to geographic space transformation matrix to obtain the three-dimensional road line and the three-dimensional coordinates of the road line.

[0043] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a three-dimensional road line extraction method.

[0044] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a three-dimensional road line extraction method.

[0045] The above-described solution of the present invention has the following beneficial effects:

[0046] This invention acquires binocular image data and inertial navigation positioning data of a research area based on an onboard platform, and performs disparity estimation on the binocular image data to obtain a visual 3D point cloud. Within the research area, a region centered on the binocular camera with a preset distance as the radius is selected as the target region. The visual 3D point cloud within the target region is extracted and transformed to obtain a road surface bird's-eye view and a point cloud elevation map. The road surface bird's-eye view and point cloud elevation map are input into a lane line detection network for road line extraction, yielding 3D road lines and their 3D coordinates. Compared with existing technologies, this invention overcomes the problems of low accuracy, poor robustness, and high detection costs associated with current onboard image road line recognition methods. It improves the efficiency and quality of road line detection in onboard images, reduces errors caused by image distortion, and can provide input for 3D road construction in road scenes. It is a key link in 3D road network construction and has significant application prospects and practical value in high-precision road network model construction and road network data production and updating.

[0047] Other beneficial effects of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0048] Figure 1 This is a flowchart of an embodiment of the present invention;

[0049] Figure 2 This is a flowchart of an embodiment of the present invention;

[0050] Figure 3 This is a diagram illustrating the disparity estimation effect of an embodiment of the present invention. Detailed Implementation

[0051] To make the technical problems, solutions, and advantages of this invention clearer, a detailed description will be provided below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0052] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0053] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a locking connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0054] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0055] This invention addresses existing problems by providing a method and related equipment for extracting three-dimensional road lines, which integrates binocular images and inertial navigation information to improve the efficiency and quality of three-dimensional road line extraction.

[0056] like Figure 1 As shown, an embodiment of the present invention provides a method for extracting three-dimensional road lines, including:

[0057] Step 1: Acquire binocular image data and inertial navigation positioning data of the study area based on the vehicle-mounted platform;

[0058] Step 2: Based on the inertial navigation positioning data, perform disparity estimation on the binocular image data to obtain the visual 3D point cloud;

[0059] Step 3: In the study area, determine the area with the binocular camera as the center and a preset distance as the radius as the target area, extract the visual 3D point cloud in the target area, and transform the visual 3D point cloud in the target area to obtain the road surface bird's-eye view and point cloud elevation map of the target area.

[0060] Step 4: Input the road surface bird's-eye view and point cloud elevation map into the lane line detection network. Using the point cloud elevation map as auxiliary information, extract the road line elements from the road surface bird's-eye view. Calculate the 3D coordinate information of the road line in camera space based on the image coordinates in the road line elements. Transform the 3D coordinate information in camera space according to the camera space to geographic space transformation matrix to obtain the 3D road line and its 3D coordinates.

[0061] This invention embodiment performs 3D route extraction by combining high-resolution vehicle image data acquired by a binocular vision system with camera attitude information provided by an inertial navigation system composed of a real-time differential positioning system and an inertial strategy unit.

[0062] Specifically, before step 2, the following is also included:

[0063] Multiple calibration board images of the study area were captured from different angles by the binocular camera on the vehicle platform. The calibration board images were then calibrated to obtain the camera intrinsic parameter matrix and distortion coefficients.

[0064] The rotation and translation parameters are calculated based on the absolute spatial pose of the left camera of the binocular camera and the pose of the positioning system to obtain the camera extrinsic parameter matrix.

[0065] Since cameras use lenses for imaging, the lens structure can cause a certain degree of distortion to the captured binocular images. Therefore, this embodiment of the invention uses the Zhang Zhengyou camera calibration method from the OpenCV library. Calibration is performed by acquiring 10 calibration board images of the study area taken by the binocular camera at different angles, obtaining the camera intrinsic parameter matrix and distortion coefficients. Simultaneously, rotation and translation parameters are calculated based on the absolute spatial pose of the left camera and the pose of the positioning system to obtain the camera extrinsic parameter matrix, ensuring that any point in the real coordinate space can be mapped to the camera's planar coordinate system. The solution formula is as follows:

[0066]

[0067] Among them, Z c f represents the distance from the camera plane to the real object corresponding to the image (u,v) in the camera coordinate system, where u and v represent the camera plane coordinates, and f is the distance from the camera plane to the real object corresponding to the image (u,v). x f y Let be the focal length of the camera on the x and y axes, u0 and v0 be the image centers, R be the rotation matrix, T be the translation matrix, and X be the focal length of the camera on the x and y axes. W Y W Z W This represents the coordinates of a real object in the world coordinate system.

[0068] Specifically, step 2 includes:

[0069] Preprocessing is performed on the binocular image data and inertial navigation positioning data based on the camera's intrinsic parameter matrix, distortion coefficients, and extrinsic parameter matrix.

[0070] Based on the preprocessed inertial navigation positioning data, disparity estimation is performed on the preprocessed binocular image data.

[0071] In this embodiment of the invention, the binocular image data and inertial navigation positioning data acquired by the vehicle platform are preprocessed, the average rate of change is calculated based on the time sequence information of the data, and a moving window smoothing algorithm is used to remove sampling points with significant offsets. The specific method is as follows: the window size is set to 5 sampling points, the mean μ and variance σ of the moving distance of the sampling points in the x, y and z axes within the window are calculated, and when the difference between the moving distance of the current sampling point and the mean of the previous sampling point reaches 1.5σ, it is regarded as an outlier and removed.

[0072] The Fusiello epipolar correction algorithm is then used to align the epipolar lines of the left and right views. This involves finding new projection coordinates that make the optical axes of the right and left cameras parallel, lie on the same plane, and be reprojected. The new coordinate system satisfies the following conditions:

[0073] The new x-axis is parallel to the baseline: r1 = (c1-c2) / ||c1-c2||;

[0074] The new y-axis is orthogonal to the new x-axis and the z-axis of the left camera before correction: r2 = k × r1;

[0075] The new z-axis is orthogonal to the new x-axis and the new y-axis: r3 = r1 × r2.

[0076] Further parallel computing was used to perform statistical analysis on the binocular image data. It was found that the road area is generally located in the lower 2 / 3 of the image. Based on the commonly used ratio of 4:3 for current image data, the road area was cropped for the binocular image data of each sampling point. The original input image and the preprocessed image of the effective road area in the image were extracted to obtain the left and right corrected views.

[0077] Specifically, the disparity value is estimated from the preprocessed binocular image data, including:

[0078] A deep learning-based stereo matching model is used to estimate the disparity of the preprocessed binocular image data to obtain the disparity value of each pixel.

[0079] The 3D coordinates of each pixel are calculated based on the disparity value of each pixel, as shown in the following formula:

[0080] z = f * b / d

[0081] x = x l *z / f

[0082] y = y l *z / f

[0083] Where, x l ,y l is the image coordinate of the binocular image, f is the camera focal length, b is the distance between the left and right cameras of the binocular camera, and d is the parallax value;

[0084] A 3D visual point cloud is obtained based on the 3D coordinates of each pixel.

[0085] This invention uses a deep learning-based disparity matching algorithm to estimate the disparity values ​​of the left and right corrected views, and extracts a visual 3D point cloud based on the principle of binocular vision. The specific implementation is as follows:

[0086] Let the resulting binocular image dataset after cropping be S={(L1,R1),(L2,R2),…,(L n ,Rn )}, where (L n ,R n ) represents the nth pair of left and right corrected views, and the disparity map D = {d1, d2, ..., dn} n}, where d n This represents the disparity of the right corrected view relative to the left corrected view in the nth pair of left and right corrected views, in pixels, where n is the total number of pairs of left and right corrected views.

[0087] The disparity map D is obtained using a deep learning-based stereo matching model. The model structure is briefly summarized below:

[0088] We use conv1 to conv4 in Resnext-34 as the initial feature learning modules, and fuse the feature maps extracted from each conv layer to predict the object edge, thus guiding the model to learn the object edge information.

[0089] Pyramid upsampling is performed on the feature maps output by the feature learning module to construct cost bodies at different scales (1 / 2, 1 / 4, and 1 / 8) of the original image size. The cost body construction formula is as follows:

[0090] C i =Concat{F Li ,F Rj ,…,F Rm}

[0091] F Ri =F R (0, row, j, column)

[0092] Where row and column are the number of rows and columns of the right corrected view feature map, and F is a function that clips the j-th column to the last column and sets the subsequent columns to 0, that is, to obtain the right view feature map under the disparity j corresponding to the i-th feature map in the left view feature map, where the disparity range m is 192.

[0093] The constructed cost volumes are regularized using 3D convolution to resolve noise interference. After regularization, the disparity results are refined by combining the initial convolution features.

[0094] Finally, disparity estimation is performed on the feature maps of the three scales from small scale to large scale, and the features from the previous step are fused together. The final predicted disparity value is predicted using soft-argmin.

[0095]

[0096] Where k is the predicted disparity value, p k The confidence level of this disparity value;

[0097] The stereo matching model was trained using a transfer learning strategy. First, it was pre-trained on the Scene Flow Datasets dataset for 100 epochs, and then retrained using the Kitti2012 and Kitti2015 datasets. The test results are as follows: Figure 3 As shown.

[0098] Specifically, step 3 includes:

[0099] Within the study area, a region centered on the binocular camera and with a preset distance as its radius is defined as the target region.

[0100] Extract the visual 3D point cloud within the target area;

[0101] The visual 3D point cloud of the target area in 3D space is projected onto 2D space to obtain a bird's-eye view of the road surface and a point cloud elevation map of the target area. The conversion formula is as follows:

[0102] Scal x =pic w / (x max -x min )

[0103] Scal y =pic h / (y max -y min )

[0104] X = int(Scal x *(x0-x min ))

[0105] Y = pic h -int(Scal y *(y0-y min ))

[0106] P XY =value(x,y,z)

[0107] Among them, pic w pic h The width and height of the target image to be converted are both set to 400, x max y max x represents the maximum value of the visual 3D point cloud within the target region on the coordinate axis. min y min The minimum value of the visual 3D point cloud in the target area on the coordinate axis is XY, which are the coordinates projected onto the plane. P represents a pixel. When the corresponding pixel value is the color of the visual 3D point cloud, a bird's-eye view of the road surface is obtained; when the corresponding pixel value is the height value of the visual 3D point cloud, a point cloud elevation map is obtained.

[0108] Specifically, in step 4, this embodiment of the invention uses an instance-segmentation-based lane line detection network to extract road lines from the road surface bird's-eye view and point cloud elevation map, as follows:

[0109] The dataset was created by manually labeling road line features from bird's-eye views using the LabelMe software. The labeling followed constraints such as equal distance between road lines, symmetry in road line distribution, and road line extension lines. The labeled data was used for training the lane line detection network.

[0110] The lane detection network is constructed by improving upon the SCNN (Spatial CNN) spatial convolutional network as the basic model. The improvements include: adding an elevation map as auxiliary information to the input data; changing the network information transmission from the original top-down and left-to-right to only retaining top-down; and setting a transmission step size to reduce the model inference cost.

[0111] The evaluation function will consist of three parts: a weighted cross-entropy function, the variance of the distance between road lines, and the rate of change of road lines along the height direction of the image, as shown in the following formula:

[0112]

[0113] in, This represents the weighted cross-entropy function. An expression representing the variance of distances between roads. p j-1 ) 2 This represents the rate of change of the road line along the height direction of the image, where α and β represent the weights.

[0114] Using the above model, road line elements can be extracted from the bird's-eye view of the road surface. By combining the elevation information in the point cloud elevation map, the three-dimensional coordinate information of the road line in camera space can be obtained. Furthermore, based on the camera space to geographic space transformation matrix, the three-dimensional road line and the true three-dimensional coordinates of the road line can be extracted at each data sampling point.

[0115] Based on the vehicle-mounted platform, binocular image data and inertial navigation positioning data of the research area are acquired, and disparity estimation is performed on the binocular image data to obtain a visual 3D point cloud. The research area with the binocular camera as the center and a preset distance as the radius is used as the target area. The visual 3D point cloud in the target area is extracted and transformed to obtain a road surface bird's-eye view and point cloud elevation map of the target area. The road surface bird's-eye view and point cloud elevation map are input into the lane line detection network for road line extraction to obtain the 3D road line and the 3D coordinates of the road line. Compared with the existing technology, this method overcomes the problems of low accuracy, poor robustness and high detection cost of current road line recognition methods in vehicle images. It improves the efficiency and quality of road line detection in vehicle images and reduces road line detection errors caused by image distortion. It can provide input for 3D road construction in road scenes and is a key link in 3D road network construction. It has great application prospects and practical value in the construction of high-precision road network models and the production and updating of road network data.

[0116] The present invention also provides a three-dimensional road line extraction device, comprising:

[0117] The acquisition module is used to acquire binocular image data and inertial navigation positioning data of the study area based on the vehicle-mounted platform.

[0118] The disparity estimation module is used to estimate the disparity of the binocular image data based on the inertial navigation positioning data to obtain a visual 3D point cloud.

[0119] The point cloud extraction module is used to determine the target area in the study area with the binocular camera as the center and a preset distance as the radius, extract the visual 3D point cloud in the target area, and transform the visual 3D point cloud in the target area to obtain the road surface bird's-eye view and point cloud elevation map of the target area.

[0120] The road line extraction module is used to input the road surface bird's-eye view and point cloud elevation map into the lane line detection network, and extract the road line elements from the road surface bird's-eye view using the point cloud elevation map as auxiliary information; calculate the three-dimensional coordinate information of the road line in camera space based on the image coordinates in the road line elements; and transform the three-dimensional coordinate information in camera space according to the camera space to geographic space transformation matrix to obtain the three-dimensional road line and the three-dimensional coordinates of the road line.

[0121] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0122] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the embodiments of the present invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0123] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a three-dimensional road line extraction method.

[0124] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the embodiments of the present invention can implement all or part of the processes in the methods described above by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a building device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0125] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a three-dimensional road line extraction method.

[0126] It should be noted that the terminal device can be a mobile phone, tablet computer, laptop computer, Ultra-mobile Personal Computer (UMPC), netbook, Personal Digital Assistant (PDA), etc. For example, the terminal device can be a station (ST) in a WLAN, a cellular phone, cordless phone, Session Initiation Protocol (SIP) phone, Wireless Local Loop (WLL) station, PDA, handheld device with wireless communication capabilities, computing device or other processing device connected to a wireless modem, computer, laptop computer, handheld communication device, handheld computing device, satellite wireless device, etc. The embodiments of this invention do not limit the specific type of terminal device.

[0127] The processor referred to can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0128] In some embodiments, the memory may be an internal storage unit of the terminal device, such as a hard drive or RAM. In other embodiments, the memory may be an external storage device of the terminal device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital Card (SD), or Flash Card. Furthermore, the memory may include both internal and external storage units of the terminal device. The memory is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory can also be used to temporarily store data that has been output or will be output.

[0129] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0130] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for extracting three-dimensional road lines, characterized in that, include: Step 1: Acquire binocular image data and inertial navigation positioning data of the study area based on the vehicle-mounted platform; Step 2: Based on the inertial navigation positioning data, perform disparity estimation on the binocular image data to obtain a visual 3D point cloud; Step 3: Determine the target area from the study area with the binocular camera as the center and a preset distance as the radius, extract the visual 3D point cloud in the target area, and transform the visual 3D point cloud in the target area to obtain the road surface bird's-eye view and point cloud elevation map of the target area. Step 4: Input the road surface bird's-eye view and the point cloud elevation map into the lane line detection network, and use the point cloud elevation map as auxiliary information to extract the road line elements from the road surface bird's-eye view; The three-dimensional coordinate information of the road line in camera space is calculated based on the image coordinates in the road line elements; the three-dimensional coordinate information in camera space is transformed according to the camera space to geographic space transformation matrix to obtain the three-dimensional road line and the three-dimensional coordinates of the road line.

2. The three-dimensional road line extraction method according to claim 1, characterized in that, Before step 2, the following is also included: Multiple calibration board images of the study area were captured from different angles by a binocular camera on a vehicle-mounted platform. The calibration board images were then calibrated to obtain the camera intrinsic parameter matrix and distortion coefficients. The rotation and translation parameters are calculated based on the absolute spatial pose of the left camera of the binocular camera and the pose of the positioning system to obtain the camera extrinsic parameter matrix.

3. The three-dimensional road line extraction method according to claim 2, characterized in that, Step 2 includes: The binocular image data and the inertial navigation positioning data are preprocessed based on the camera's intrinsic parameter matrix, the distortion coefficients, and the camera's extrinsic parameter matrix. Based on the preprocessed inertial navigation positioning data, disparity estimation is performed on the preprocessed binocular image data.

4. The three-dimensional road line extraction method according to claim 3, characterized in that, The disparity value is estimated from the preprocessed binocular image data, including: A deep learning-based stereo matching model is used to estimate the disparity of the preprocessed binocular image data to obtain the disparity value of each pixel. The 3D coordinates of each pixel are calculated based on the disparity value of each pixel, as shown in the following formula: z = f * b / d x=x l *z / f y=y l *z / f Where, x l ,y l is the image coordinate of the binocular image, f is the camera focal length, b is the distance between the left and right cameras of the binocular camera, and d is the parallax value; A 3D visual point cloud is obtained based on the 3D coordinates of each pixel.

5. The three-dimensional road line extraction method according to claim 4, characterized in that, Step 3 includes: Within the study area, a region centered on the binocular camera and with a preset distance as its radius is defined as the target region. Extract the visual 3D point cloud within the target area; The visual 3D point cloud of the target area in 3D space is projected onto 2D space to obtain a bird's-eye view of the road surface of the target area. The conversion formula is as follows: Scal x =pic w / (x max -x min ) Scal y =pic h / (y max -y min ) X=int(Scal x *(x0-x min )) Y=pic h -int(Scal y *(y0-y min )) P XY =value(x,y,z) Among them, pic w pic h The width and height of the target image to be converted are both set to 400, x max y max x represents the maximum value of the visual 3D point cloud within the target region on the coordinate axis. min y min The minimum value of the visual 3D point cloud in the target area on the coordinate axis is XY, which are the coordinates projected onto the plane. P represents a pixel. When the corresponding pixel value is the color of the visual 3D point cloud, a bird's-eye view of the road surface is obtained; when the corresponding pixel value is the height value of the visual 3D point cloud, a point cloud elevation map is obtained.

6. A three-dimensional road line extraction device, characterized in that, include: The acquisition module is used to acquire binocular image data and inertial navigation positioning data of the study area based on the vehicle-mounted platform. The disparity estimation module is used to perform disparity estimation on the binocular image data based on the inertial navigation positioning data to obtain a visual three-dimensional point cloud. The point cloud extraction module is used to determine the area with the binocular camera as the center and a preset distance as the radius as the target area in the study area, extract the visual three-dimensional point cloud in the target area, and transform the visual three-dimensional point cloud in the target area to obtain the road surface bird's-eye view and point cloud elevation map of the target area. The road line extraction module is used to input the road surface bird's-eye view and the point cloud elevation map into the lane line detection network, and use the point cloud elevation map as auxiliary information to extract the road line elements in the road surface bird's-eye view; The three-dimensional coordinate information of the road line in camera space is calculated based on the image coordinates in the road line elements; the three-dimensional coordinate information in camera space is transformed according to the camera space to geographic space transformation matrix to obtain the three-dimensional road line and the three-dimensional coordinates of the road line.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional road line extraction method as described in any one of claims 1 to 5.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the three-dimensional road line extraction method as described in any one of claims 1 to 5.

Images