Lane information detection method and system and electronic equipment
Through the monocular camera and lane information detection model, the depth information and two-dimensional coordinates of the lane image are acquired and processed, and the three-dimensional coordinates of the lane information are obtained through combined processing, which solves the problem of low detection efficiency and accuracy of existing technologies in complex environments, and achieves for more efficient and accurate lane information detection.
Patent Information
- Application Number
- CN202411405345.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2025-01-24
Smart Images
Figure CN119360336A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the technical field of lane information detection, and in particular, to a lane information detection method, system and electronic device. Background Art
[0002] Lane information detection technology is a key technology in the field of automobile driving. In the autonomous driving system, lane information detection technology uses sensors to identify lane images on the road in real time, and uses specific algorithms to process them to obtain accurate lane information data, provide navigation and other information for the vehicle, improve the accuracy of the vehicle's driving path, optimize the driving path, and improve driving efficiency.
[0003] Lane information detection technology can capture the color and texture information of the road environment and the edge images of the lane information through binocular cameras to achieve lane image acquisition, and generate point cloud data by scanning the lane with lidar to restore the shape of the lane. The image data and radar point cloud data are then fused and processed to achieve accurate lane information detection.
[0004] However, the system structure of lane information detection using binocular cameras and radars is complex, and the data obtained by the binocular cameras requires known camera parameters when performing operations such as image correction, which increases the difficulty of the algorithm. The radar has a small perception range and it is difficult to achieve accurate detection in complex environments such as uphill, turning, and large-span lanes. Summary of the invention
[0005] The present application provides a lane information detection method, system and electronic equipment to improve the efficiency and accuracy of lane information detection.
[0006] In a first aspect, some embodiments of the present application provide a lane information detection method, including:
[0007] Obtain a monocular lane image through a monocular camera;
[0008] Inputting the monocular lane image into a lane information detection model, performing depth information detection to obtain depth information; and performing lane picture detection to obtain two-dimensional coordinates of the lane picture;
[0009] Acquire depth information and two-dimensional coordinates of a lane image that have an associated relationship; wherein the associated relationship represents a relationship between the depth information and the two-dimensional coordinates of the lane image obtained after the same pixel in the monocular lane image is detected by the lane information detection model;
[0010] The depth information and the two-dimensional coordinates of the lane image having the associated relationship are combined to obtain the three-dimensional coordinates of the lane information.
[0011] The method acquires a monocular lane image through a monocular camera, and uses a lane information detection model to perform detection on the monocular lane image to obtain depth information and two-dimensional coordinates of the lane image, and then combines the associated depth information and two-dimensional coordinates of the lane image to obtain three-dimensional coordinates of the lane information. The deviation in lane information detection can be corrected, and the efficiency and accuracy of lane information detection can be improved.
[0012] In an implementable manner of the first aspect, the lane information detection model performs depth information detection according to the depth label data to obtain depth information, and the method for obtaining the depth label data includes: obtaining aircraft mark radar point cloud data; performing coordinate system transformation on the aircraft mark radar point cloud data to obtain a depth truth value; performing mapping on the depth truth value to obtain a discretized depth truth value; performing classification on the discretized depth truth value, and classifying the discretized depth truth value into a preset number of categories to obtain depth label data.
[0013] In an implementable manner of the first aspect, the lane information detection model includes a depth information detection sub-model, and a training process of the depth information detection sub-model includes: obtaining a depth information detection training data set, wherein the depth information detection training data set includes lane image data and the depth label data; inputting the lane image data into the depth information detection sub-model to be trained to obtain a predicted depth value output by the depth information detection sub-model to be trained; calculating a training loss based on the depth label data and the predicted depth value; if the training loss is greater than or equal to a loss threshold, adjusting the parameters of the depth information detection sub-model to be trained based on the training loss; if the training loss is less than the loss threshold, outputting the parameters of the depth information detection sub-model to be trained as the parameters of the depth information detection sub-model.
[0014] In an implementable manner of the first aspect, the depth information and the two-dimensional coordinates of the lane image are combined to obtain the three-dimensional coordinates of the lane information, including: performing restoration processing on the depth information to obtain a depth information prediction value; performing correction processing on the two-dimensional coordinates of the lane image according to the depth information prediction value to obtain the three-dimensional coordinates of the lane information under the camera system; performing coordinate system transformation on the three-dimensional coordinates of the lane information under the camera system according to the external parameters of the monocular camera to obtain the three-dimensional coordinates of the lane information, and the three-dimensional coordinates of the lane information include a horizontal coordinate and a height coordinate.
[0015] In an implementable manner of the first aspect, the method also includes: obtaining a first lane information three-dimensional coordinate and a second lane information three-dimensional coordinate, wherein the first lane information three-dimensional coordinate and the second lane information three-dimensional coordinate are lane information three-dimensional coordinates having a preset relationship with the monocular camera, the first lane information three-dimensional coordinate includes a first horizontal coordinate and a first height coordinate, and the second lane information three-dimensional coordinate includes a second horizontal coordinate and a second height coordinate; subtracting the first height coordinate from the second height coordinate to obtain a height coordinate difference; subtracting the first horizontal coordinate from the second horizontal coordinate to obtain a horizontal coordinate difference; and performing an inverse tangent calculation on the ratio of the height coordinate difference to the horizontal coordinate difference to obtain an angle value of a road surface inclination.
[0016] In an implementation manner of the first aspect, the method further includes: obtaining an angle value of the angle between the monocular camera and a calibrated horizontal plane based on external parameters of the monocular camera; adding the angle value of the angle between the monocular camera and the calibrated horizontal plane to the angle value of the road surface inclination to obtain an angle value of a pitch angle, wherein the pitch angle is the angle between the optical axis of the monocular camera and the lane information horizontal plane.
[0017] In an implementable manner of the first aspect, the method further includes: performing a tangent calculation on the angle value of the road surface inclination to obtain an angle fluctuation value; multiplying the angle fluctuation value by an infinite value to obtain a v-axis coordinate value of a road surface vanishing point, wherein the road surface vanishing point is a projection point of an infinite point in the lane information three-dimensional coordinates on the monocular lane image; obtaining a calibrated vanishing point, and judging a vehicle position state according to a difference between the v-axis coordinate value of the road surface vanishing point and the v-axis coordinate value of the calibrated vanishing point, wherein the calibrated vanishing point is a projection point of an infinite point in the lane information three-dimensional coordinates on the monocular lane image when there is no angle between the calibrated horizontal plane and the lane information horizontal plane in the horizontal direction.
[0018] In an implementation of the first aspect, the lane information detection model also includes a two-dimensional image detection submodel and a feature extraction layer, the two-dimensional image detection submodel is used to detect the two-dimensional coordinates of the lane image, and the feature extraction layer is used to extract the first feature and the second feature of the monocular lane image. The method also includes: inputting the first feature into the depth information detection submodel to perform depth information detection to obtain depth information; inputting the second feature into the two-dimensional image detection submodel to perform lane image detection to obtain the two-dimensional coordinates of the lane image.
[0019] In a second aspect, some embodiments of the present application further provide a lane information detection system, including:
[0020] The image acquisition module is configured to: acquire a monocular lane image through a monocular camera;
[0021] The image detection module is configured to: input the monocular lane image into a lane information detection model, perform depth information detection to obtain depth information; and perform lane picture detection to obtain two-dimensional coordinates of the lane picture;
[0022] The association relationship acquisition module is configured to: acquire depth information and two-dimensional coordinates of the lane image having an association relationship; wherein the association relationship represents the relationship between the depth information and the two-dimensional coordinates of the lane image obtained after the same pixel in the monocular lane image is detected by the lane information detection model;
[0023] The three-dimensional coordinate generation module is configured to: perform a combination process on the depth information having an associated relationship and the two-dimensional coordinates of the lane picture to obtain the three-dimensional coordinates of the lane information.
[0024] In a third aspect, some embodiments of the present application further provide an electronic device, including a processor, a memory, and a bus;
[0025] The processor and the memory are communicatively connected via the bus;
[0026] The memory stores computer program instructions executable by the processor, the processor being configured to:
[0027] Obtain a monocular lane image through a monocular camera;
[0028] Inputting the monocular lane image into a lane information detection model, performing depth information detection to obtain depth information; and performing lane picture detection to obtain two-dimensional coordinates of the lane picture;
[0029] Acquire depth information and two-dimensional coordinates of a lane image that have an associated relationship; wherein the associated relationship represents a relationship between the depth information and the two-dimensional coordinates of the lane image obtained after the same pixel in the monocular lane image is detected by the lane information detection model;
[0030] The depth information and the two-dimensional coordinates of the lane image having the associated relationship are combined to obtain the three-dimensional coordinates of the lane information.
[0031] It can be seen from the above technical solutions that the present application provides a lane information detection method, system and electronic device, the method comprising: acquiring a monocular lane image through a monocular camera; inputting the monocular lane image into a lane information detection model, performing depth information detection to obtain depth information; and performing lane picture detection to obtain two-dimensional coordinates of the lane picture; acquiring depth information and two-dimensional coordinates of the lane picture with an associated relationship; wherein the associated relationship represents the relationship between the depth information and the two-dimensional coordinates of the lane picture obtained after the same pixel in the monocular lane image is detected by the lane information detection model; performing a combination process on the depth information and the two-dimensional coordinates of the lane picture with an associated relationship to obtain three-dimensional coordinates of the lane information.
[0032] The method acquires a monocular lane image through a monocular camera, and uses a lane information detection model to perform detection on the monocular lane image to obtain depth information and two-dimensional coordinates of the lane image, and then combines the associated depth information and two-dimensional coordinates of the lane image to obtain three-dimensional coordinates of the lane information. The deviation in lane information detection can be corrected, and the efficiency and accuracy of lane information detection can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0034] Figure 1 A schematic flow chart of a lane information detection method provided in some embodiments of the present application;
[0035] Figure 2 A schematic diagram of a first monocular lane image captured by a monocular camera provided in some embodiments of the present application;
[0036] Figure 3 A two-dimensional coordinate point simulation diagram of real lane lines and virtual lane lines detected by a two-dimensional image detection sub-model when a vehicle is going downhill provided in some embodiments of the present application;
[0037] Figure 4 A schematic diagram of a horizontal road surface and an actual road surface simulation when a vehicle is going downhill provided in some embodiments of the present application;
[0038] Figure 5 A schematic diagram of a horizontal road surface and an actual road surface simulation when a vehicle goes uphill provided in some embodiments of the present application;
[0039] Figure 6 A two-dimensional coordinate point simulation diagram of real lane lines and virtual lane lines detected by a two-dimensional image detection sub-model when a vehicle goes uphill provided in some embodiments of the present application;
[0040] Figure 7 A flow chart of a method for obtaining deep tag data provided in some embodiments of the present application;
[0041] Figure 8 A schematic diagram of a flow chart of performing a combined process on depth information and two-dimensional coordinates of a lane image having an associated relationship provided in some embodiments of the present application;
[0042] Fig. 9 A lane information simulation image obtained by combining the depth information obtained by the depth information detection model of the first monocular lane image and the two-dimensional coordinates of the lane image provided in some embodiments of the present application;
[0043] Fig.10 A schematic diagram of a method for calculating a pitch angle provided in some embodiments of the present application;
[0044] Fig.11 A schematic diagram of a road surface vanishing point and a calibrated vanishing point during vehicle driving provided in some embodiments of the present application;
[0045] Fig.12 Schematic diagram of a lane information detection system provided for some embodiments of the present application. DETAILED DESCRIPTION
[0046] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0047] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.
[0048] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there a limitation on the quantity and execution order. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms.
[0049] These terms are only used to distinguish one element from another element. For example, without departing from the scope of various examples, a first action can be referred to as a second action, and similarly, a second action can also be referred to as a first action. Both the first action and the second action can be actions, and in some cases, can be separate and different actions.
[0050] Here, at least one means one or more than one, for example, at least one action can be one action, two actions, three actions, or any other action that is an integer greater than or equal to one. And multiple means two or more than two, for example, multiple actions can be two actions, three actions, or any other action that is an integer greater than or equal to two.
[0051] Lane information detection technology uses high-precision sensors installed on vehicles, such as high-definition cameras, to capture real-time images of the road surface, including lane lines, curbs, crosswalks, intersections, and other information. Image processing and machine learning algorithms are then used to analyze and identify the collected images to form lane information data. Lane information data can provide navigation guidance for vehicles, improve the accuracy of vehicle driving path prediction, and thus improve driving efficiency and safety.
[0052] Lane information detection technology can use a binocular camera system, which captures and compares road images obtained by the left and right cameras by simulating the stereoscopic imaging principle of human vision, and then identifies lane lines, curbs and other information. At the same time, laser radar can also be used to emit laser beams and receive the reflected signals to generate three-dimensional point cloud data, thereby depicting the ups and downs of the road surface, the shape of the lane and other information.
[0053] However, the system structure of lane information detection using binocular cameras and radars is complex, and the data obtained by the binocular cameras requires known camera parameters when performing operations such as image correction, which increases the difficulty of the algorithm. The radar has a small perception range and it is difficult to achieve accurate detection in complex environments such as uphill, turning, and large-span lanes.
[0054] Some embodiments of the present application provide a lane information detection method, system and electronic device, which can solve the problem of low detection efficiency and accuracy of lane information detection technology in complex environments such as uphill and turning. Figure 1 A schematic flow chart of a lane information detection method provided in some embodiments of the present application, such as Figure 1 As shown, the lane information detection method includes steps S100-S400.
[0055] S100: Acquire a monocular lane image through a monocular camera.
[0056] In some embodiments, the lane information detection method can directly use a monocular camera to capture a monocular lane image through a single optical lens, and convert the analog image into a digital image. The monocular lane image captured by the monocular camera includes information such as lane lines, crosswalks, and curbs. For example, Figure 2 As shown, the first monocular lane image schematic diagram captured by the monocular camera includes a real lane line 201, a virtual lane line 202 and a curb 203.
[0057] In some embodiments, in order to improve the quality of the monocular image and the efficiency of subsequent processing, the monocular lane image may be preprocessed, including image denoising, edge enhancement, grayscale stretching, image segmentation, morphological processing, etc., to improve the visual effect of the monocular lane image and highlight key features.
[0058] S200: Inputting a monocular lane image into a lane information detection model, performing depth information detection to obtain depth information; and performing lane picture detection to obtain two-dimensional coordinates of the lane picture.
[0059] In some embodiments, after acquiring a monocular lane image, the monocular lane image can be input into a lane information detection model to perform lane information detection. The lane information detection model is a neural network model, which is trained by a lane information training data set. The lane information training data set includes lane image data and lane information label data. The lane information detection model is iteratively trained with the lane image data as input data and the lane information label data as output data.
[0060] It should be noted that the lane information detection model includes a depth information detection sub-model, a two-dimensional image detection sub-model, etc. The output data of different types of sub-models are different. Therefore, the types of lane information label data used when training the depth information detection sub-model and the two-dimensional image detection sub-model are different.
[0061] In some embodiments, the lane information detection model further includes a feature extraction layer for extracting a first feature and a second feature of a monocular lane image. The first feature is a feature required for the depth information detection submodel to perform detection, and the second feature is a feature required for the two-dimensional image detection submodel to perform detection. The first feature is input into the depth information detection submodel to perform depth information detection to obtain depth information; the second feature is input into the two-dimensional image detection submodel to perform lane image detection to obtain the two-dimensional coordinates of the lane image.
[0062] For example, Figure 2 As shown, the first monocular lane image is input into the lane information detection model, and after the feature extraction layer performs feature extraction on the first monocular lane image to obtain the first feature and the second feature, the second feature is input into the two-dimensional picture detection sub-model to obtain the two-dimensional coordinate point 204 of the real lane line, the two-dimensional coordinate point 205 of the virtual lane line and the two-dimensional coordinate point 206 of the curb in the first monocular lane image.
[0063] In some embodiments, when a vehicle is traveling on a road surface that is not parallel to the calibrated horizontal plane, such as an uphill, downhill, or turning road, the two-dimensional coordinates of the lane image output by the two-dimensional image detection sub-model deviate from the actual situation. Figure 3The 2D image detection sub-model uses the first monocular lane image as input to detect the 2D coordinate points of the real lane line 204 and the virtual lane line 205. Since the 2D image detection sub-model assumes that there is no angle between the horizontal road surface and the road surface in the actual lane image, when the 2D coordinate points output by the 2D image detection sub-model are combined into simulated lane lines, they will not align with the actual lane lines. Figure 3 It can be seen that the simulated lane line composed of the two-dimensional coordinate points output by the two-dimensional image detection sub-model has the "inward-facing" phenomenon. It should be understood that the reason why the simulated lane line and the actual lane line are not aligned is that when the vehicle is going downhill, the actual road surface is lower than the horizontal road surface, and there is an inclination angle between the two, such as Figure 4 shown.
[0064] In some embodiments, similar to the case of a vehicle going downhill, during the process of the vehicle going uphill, the actual road surface is higher than the horizontal road surface, and the inclination angle between the two is as follows: Figure 5 As shown in the figure. The simulated lane line composed of the two-dimensional coordinate points output by the two-dimensional image detection sub-model is shown in the figure. Figure 6 As shown, there is an "outward eight" phenomenon.
[0065] In some embodiments, in order to correct the two-dimensional coordinates of the lane image output by the two-dimensional image detection sub-model, the depth information in the same monocular lane image can be detected, the monocular lane image is input into the lane information detection model, and the feature extraction layer performs feature extraction on the monocular lane image to obtain the first feature and the second feature, and then the depth information detection sub-model performs detection on the first feature to obtain the depth information in the monocular lane image.
[0066] Exemplarily, the first feature extracted by the feature extraction layer for the first monocular lane image is input into the depth information detection sub-model. The depth information detection sub-model can output the depth information of each pixel in the first monocular lane image. The depth information corresponds to the two-dimensional coordinates of the lane image obtained after the two-dimensional image detection sub-model performs detection on the second feature of the first monocular lane image.
[0067] In some embodiments, the depth information detection submodel in the lane information detection model performs depth information detection based on the depth label data to obtain depth information. Figure 7 As shown, when acquiring the depth label data, step S701 may be performed first: acquiring the aircraft mark radar point cloud data.
[0068] Then execute step S702: perform coordinate system transformation on the radar point cloud data of the aircraft mark to obtain the true depth value. After obtaining the radar point cloud data of the aircraft mark, the depth value in the camera system is obtained. The depth value in the camera system is transformed into the pixel coordinate system by using formula (1) and formula (2), and the true depth value corresponding to each pixel in the monocular lane image can be obtained.
[0069]
[0070] Among them, u and v are pixel systems, is the unit transformation matrix, f is the camera intrinsic parameter, is the camera internal parameter matrix, R is the rotation matrix, T is the translation matrix, X w , Y w , Z w , for the world system, X c , Y c , Z c For the camera system.
[0071] Then execute step S703: map the true depth value to obtain the discretized true depth value. In order to simplify the learning task of the depth information detection sub-model and improve the robustness of the depth information detection sub-model, the true depth value can be mapped by formula (3) to map the true depth value to the range of (0,1), and then multiplied by category (80) by formula (4) and then rounded to obtain the discretized true depth value.
[0072] Z n =1-e (-a*Z) ……(3)
[0073] Z_n=[Z n *80]……(4)
[0074] Among them, Z n is the discretized true depth value, a is a hyperparameter, Z is the true depth value, Z_n is the discretized true depth value after rounding, and [*] is rounding.
[0075] Finally, step S704 is performed: the discretized depth truth value is classified into a preset number of categories to obtain depth label data. In the embodiment of the present application, a classification training method is adopted for the depth information detection sub-model, so the discretized truth value needs to be classified into a preset number of categories. Exemplarily, in order to improve the accuracy of the depth information detection sub-model and reduce data noise, the discretized depth truth value is divided into 80 categories to obtain depth label data.
[0076] Exemplarily, the discretized depth truth value is classified by means of equal width binning, and the category labels of the discretized depth truth value include (P1, P2, ..., P80 ), the range value of each category is 0.0125. For example, if the discretized depth truth value is 0.19, then the category label of the discretized depth truth value is P 16 , that is, the category value is 16.
[0077] In some embodiments, after the depth label data is obtained, the depth information detection sub-model can be trained, and the training process includes: obtaining a depth information detection training data set, and the depth information detection training data set includes lane image data and depth label data. That is, the lane image data is used as the input of the depth information detection sub-model, and the depth label data is used as the true value corresponding to the output of the depth information detection sub-model for training. The lane image data is input into the depth information detection sub-model to be trained to obtain the predicted depth value output by the depth information detection sub-model to be trained. And the training loss is calculated based on the depth label data and the predicted depth value. If the training loss is greater than or equal to the loss threshold, the parameters of the depth information detection sub-model to be trained are adjusted according to the training loss. If the training loss is less than the loss threshold, the parameters of the depth information detection sub-model to be trained are output as the parameters of the depth information detection sub-model.
[0078] S300: Acquire the depth information and the two-dimensional coordinates of the lane image that have an associated relationship.
[0079] In some embodiments, the monocular lane image captured by the monocular camera is detected by the trained lane information detection model, and the depth information of each pixel in the monocular lane image and the two-dimensional coordinates of the lane image can be obtained. The depth information and the two-dimensional coordinates of the lane image have an associated relationship, that is, the relationship between the depth information and the two-dimensional coordinates of the lane image obtained after the same pixel in the monocular lane image is detected by the lane information detection model.
[0080] Therefore, after obtaining the associated depth information and the two-dimensional coordinates of the lane image, the two-dimensional coordinates of the associated lane image can be corrected by the depth information to obtain more accurate lane image coordinate points, thereby improving the accuracy of lane information detection.
[0081] S400: performing a combination process on the depth information and the two-dimensional coordinates of the lane image that have an associated relationship to obtain the three-dimensional coordinates of the lane information.
[0082] In some embodiments, after the depth information and the two-dimensional coordinates of the lane image with an associated relationship are obtained, the two are combined to obtain the three-dimensional coordinates of the lane information. It should be understood that the three-dimensional coordinates of the lane information are the coordinates obtained after the depth information corrects the two-dimensional coordinates of the lane image associated with it.
[0083] In some embodiments, a flowchart of performing a combined process on the depth information and the two-dimensional coordinates of the lane image having an associated relationship is shown in FIG. Figure 8 As shown, including:
[0084] Step S801: performing restoration processing on the depth information to obtain a depth information prediction value.
[0085] In some embodiments, the depth information output by the depth information detection sub-model is a classified discretized depth value, which needs to be restored using formula (5) and formula (6).
[0086] Z n =(z_dp) / 80……(5)
[0087] Z=-a*log(1-Z n )……(6)
[0088] Among them, z_dp is the discretized depth truth value after classification, Zn is the discretized depth truth value after classification restored to the discretized depth truth value before classification, Z is the predicted value of depth information, and a is a hyperparameter.
[0089] Step S802: Correct the two-dimensional coordinates of the lane image according to the depth information prediction value to obtain the three-dimensional coordinates of the lane information in the camera system.
[0090] In some embodiments, after the depth information prediction value is obtained, the two-dimensional coordinates of the lane image can be corrected according to the depth information prediction value. By combining the depth information prediction value of the same pixel with the two-dimensional coordinates of the lane image, the three-dimensional coordinates of the lane information under the camera system are obtained.
[0091] Step S803: performing coordinate system transformation on the lane information three-dimensional coordinates in the camera system according to the external parameters of the monocular camera to obtain the lane information three-dimensional coordinates.
[0092] It should be understood that since the lane information obtained by the lane information detection technology needs to provide a reference for vehicle driving, the three-dimensional coordinates of the lane information need to maintain the same reference system as the vehicle. In some embodiments, after the three-dimensional coordinates of the lane information in the camera system are obtained, the three-dimensional coordinates of the lane information in the camera system can be transformed into a coordinate system in combination with the external parameters of the monocular camera to obtain the three-dimensional coordinates of the lane information with actual reference value.
[0093] Exemplarily, the three-dimensional coordinates of the lane information in the camera system can be transformed using formulas (7) and (8).
[0094]
[0095] x,y,dh,1=[x,y,z,1]*rtT ……(8)
[0096] Among them, x and y are the horizontal and vertical coordinates of the lane information in the camera system and after conversion, z is the depth coordinate of the lane information in the pixel system, u and v are in the pixel system, and c x ,y x 、f x , fy is the conversion parameter, d h is the height coordinate of the converted lane information, rt T are the external parameters of the monocular camera.
[0097] After the above conversion, the corrected three-dimensional coordinates of the lane information can be obtained. Fig. 9 The lane information simulation diagram is obtained by combining the depth information obtained by the depth information detection model of the first monocular lane image with the two-dimensional coordinates of the lane image, and Figure 3 In comparison, the angle deviation between the real lane line 204 and the virtual lane line 205 and the actual lane line is corrected, thereby improving the accuracy of lane information detection.
[0098] It should be noted that, since the lane information detection model in some embodiments of the present application only performs detection and processing on the monocular lane image acquired by the monocular camera, the corrected three-dimensional coordinates of the lane information can be obtained, thereby improving the efficiency of lane information detection.
[0099] In some embodiments, if the monocular lane image shows a vehicle turning or going uphill or downhill, there is an angle difference between the two corrected lane information three-dimensional coordinates obtained by the above method. Therefore, the angle value of the road surface inclination can be calculated by the corrected lane information three-dimensional coordinates, and then the angle value of the pitch angle can be calculated. The pitch angle is the angle between the optical axis of the monocular camera and the horizontal plane of the lane information, which can provide more accurate information for subsequent lane information detection.
[0100] First, the first lane information 3D coordinates and the second lane information 3D coordinates are obtained, wherein the first lane information 3D coordinates and the second lane information 3D coordinates are lane information 3D coordinates that have a preset relationship with the monocular camera, that is, two lane information 3D coordinates that are relatively close to the vehicle equipped with the monocular camera.
[0101] Exemplarily, if lane information detection is performed on the first monocular lane image, and the maximum distance between the three-dimensional coordinates of the lane information output after combining the detected depth information and the two-dimensional coordinates of the lane image and the vehicle is greater than or equal to 15 meters, the three-dimensional coordinates of the lane information closest to the vehicle are selected as the first three-dimensional coordinates of the lane information, and the three-dimensional coordinates of the lane information that are 15 meters away from the vehicle are selected as the second three-dimensional coordinates of the lane information;
[0102] If lane information detection is performed on the first monocular lane image, and the lane information three-dimensional coordinates output after combining the detected depth information with the two-dimensional coordinates of the lane image are less than 15 meters away from the vehicle, such as 13 meters, then the lane information three-dimensional coordinates closest to the vehicle are selected as the first lane information three-dimensional coordinates, and the lane information three-dimensional coordinates at a distance of 13 meters from the vehicle are selected as the second lane information three-dimensional coordinates.
[0103] Secondly, the three-dimensional coordinates of the first lane information and the three-dimensional coordinates of the second lane information are calculated. It should be understood that the three-dimensional coordinates of the first lane information include the first horizontal coordinate and the first height coordinate, and the three-dimensional coordinates of the second lane information include the second horizontal coordinate and the second height coordinate. The first height coordinate is subtracted from the second height coordinate to obtain the height coordinate difference; the first horizontal coordinate is subtracted from the second horizontal coordinate to obtain the horizontal coordinate difference; the ratio of the height coordinate difference to the horizontal coordinate difference is calculated by inverse tangent to obtain the angle value of the road surface inclination.
[0104] For example, if the three-dimensional coordinates of the first lane information are [x a ,y a ,h a ], the three-dimensional coordinates of the second lane information are [x b ,y b ,h b ], the angle value θ1 of the road surface inclination is calculated by formula (9).
[0105] θ1=arctan((h b -h a ) / (x b -x a ))……(9)
[0106] In some embodiments, after obtaining the angle value of the road surface inclination angle, the angle value of the pitch angle can be calculated according to the angle value of the road surface inclination angle. The angle value of the angle between the monocular camera and the calibrated horizontal plane is obtained according to the external parameters of the monocular camera; the angle value of the angle between the monocular camera and the calibrated horizontal plane is added to the angle value of the road surface inclination angle to obtain the angle value of the pitch angle.
[0107] For example, Fig.10As shown, line segment DE represents the calibrated horizontal plane, line segment BD represents the actual road surface during vehicle driving, and the angle ∠(angle) BDE between the two is the road surface inclination angle obtained above. The position of the monocular camera is point A. Since line segment BC is parallel to line segment DE, ∠ABC represents the angle between the optical axis of the monocular camera and the calibrated horizontal plane, and the angle values of ∠BCD and ∠BDE are equal, that is, the angle value of ∠BCD is the angle value of the road surface inclination. Therefore, the angle values of ∠BCD and ∠ABC can be added to obtain the angle value of ∠ABD, that is, the angle value of the pitch angle.
[0108] In some embodiments, the road surface vanishing point, that is, the projection point of the infinite point in the lane information three-dimensional coordinates on the monocular lane image, can also be calculated based on the corrected lane information three-dimensional coordinates to determine the vehicle's position status.
[0109] First, take the infinite point (inf, 0, inf*tanθ) of the actual road surface in the ego vehicle system, project the infinite point (inf, 0, inf*tanθ) to the monocular lane image plane, and get the ordinate v2 of the road vanishing point. Similarly, take the infinite point (inf, 0, 0) of the calibration horizontal plane in the ego vehicle system, that is, the calibration infinite point, and project the calibration infinite point to the monocular lane image plane to get the ordinate v1 of the calibration vanishing point.
[0110] After obtaining the ordinate v2 of the road vanishing point and the ordinate v1 of the calibration vanishing point, the vehicle position state can be determined according to the difference between the ordinate v1 of the calibration vanishing point and the ordinate v2 of the road vanishing point. The calibration vanishing point is the projection point of the infinite point in the lane information three-dimensional coordinates on the monocular lane image when there is no angle between the calibration horizontal plane and the lane information horizontal plane in the horizontal direction, that is, the projection point when the road surface on which the vehicle is traveling is in an ideal state such as no turns, uphill or downhill. Fig.11 Schematic diagram of the road surface vanishing point and the calibrated vanishing point during vehicle driving, wherein point 1101 is the calibrated vanishing point and point 1102 is the road surface vanishing point.
[0111] It can be seen from the above technical solutions that the present application provides a lane information detection method, system and electronic device, the method comprising: acquiring a monocular lane image through a monocular camera; inputting the monocular lane image into a lane information detection model, performing depth information detection to obtain depth information; and performing lane picture detection to obtain two-dimensional coordinates of the lane picture; acquiring depth information and two-dimensional coordinates of the lane picture with an associated relationship; wherein the associated relationship represents the relationship between the depth information and the two-dimensional coordinates of the lane picture obtained after the same pixel in the monocular lane image is detected by the lane information detection model; performing a combination process on the depth information and the two-dimensional coordinates of the lane picture with an associated relationship to obtain three-dimensional coordinates of the lane information.
[0112] The method acquires a monocular lane image through a monocular camera, and uses a lane information detection model to perform detection on the monocular lane image to obtain depth information and two-dimensional coordinates of the lane image, and then combines the associated depth information and two-dimensional coordinates of the lane image to obtain three-dimensional coordinates of the lane information. The deviation in lane information detection can be corrected, and the efficiency and accuracy of lane information detection can be improved.
[0113] Based on the lane information detection method provided by some embodiments of the present application, some embodiments of the present application also provide a lane information detection system. Fig.12 This is a schematic diagram of the structure of the lane information detection system provided in some embodiments of the present application. Fig.12 As shown, the lane information detection system includes:
[0114] The image acquisition module 1201 is configured to: acquire a monocular lane image through a monocular camera;
[0115] The image detection module 1202 is configured to: input the monocular lane image into a lane information detection model, perform depth information detection to obtain depth information; and perform lane picture detection to obtain two-dimensional coordinates of the lane picture;
[0116] The association relationship acquisition module 1203 is configured to: acquire depth information and two-dimensional coordinates of the lane image that have an association relationship; wherein the association relationship represents the relationship between the depth information and the two-dimensional coordinates of the lane image obtained after the same pixel in the monocular lane image is detected by the lane information detection model;
[0117] The three-dimensional coordinate generation module 1204 is configured to perform a combination process on the depth information having an associated relationship and the two-dimensional coordinates of the lane image to obtain the three-dimensional coordinates of the lane information.
[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
[0119] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.
Claims
1. A lane information detection method, characterized in that: include: Obtain a monocular lane image through a monocular camera; Inputting the monocular lane image into a lane information detection model, performing depth information detection to obtain depth information; and, performing lane image detection to obtain two-dimensional coordinates of the lane image; Acquire depth information and two-dimensional coordinates of a lane image that have an associated relationship; wherein the associated relationship represents a relationship between the depth information and the two-dimensional coordinates of the lane image obtained after the same pixel in the monocular lane image is detected by the lane information detection model; The depth information and the two-dimensional coordinates of the lane image having the associated relationship are combined to obtain the three-dimensional coordinates of the lane information.
2. The lane information detection method according to claim 1, characterized in that: The lane information detection model performs depth information detection according to the depth label data to obtain depth information, and the method for obtaining the depth label data includes: Obtain aircraft radar point cloud data; Performing coordinate system transformation on the radar point cloud data of the aircraft mark to obtain a true depth value; Performing mapping on the true depth value to obtain a discretized true depth value; Classification is performed on the discretized depth truth value, and the discretized depth truth value is classified into a preset number of categories to obtain depth label data.
3. The lane information detection method according to claim 2, characterized in that: The lane information detection model includes a depth information detection sub-model, and the training process of the depth information detection sub-model includes: Acquire a depth information detection training data set, wherein the depth information detection training data set includes lane image data and the depth label data; Inputting the lane image data into the depth information detection sub-model to be trained to obtain a predicted depth value output by the depth information detection sub-model to be trained; Calculating a training loss based on the depth label data and the predicted depth value; If the training loss is greater than or equal to the loss threshold, adjusting the parameters of the depth information detection sub-model to be trained according to the training loss; If the training loss is less than the loss threshold, the parameters of the depth information detection sub-model to be trained are output as the parameters of the depth information detection sub-model.
4. The lane information detection method according to claim 1, characterized in that: The combining process of the depth information and the two-dimensional coordinates of the lane image to obtain the three-dimensional coordinates of the lane information includes: Performing restoration processing on the depth information to obtain a depth information prediction value; Performing correction processing on the two-dimensional coordinates of the lane image according to the depth information prediction value to obtain the three-dimensional coordinates of the lane information under the camera system; A coordinate system transformation is performed on the lane information three-dimensional coordinates under the camera system according to the external parameters of the monocular camera to obtain the lane information three-dimensional coordinates, where the lane information three-dimensional coordinates include a horizontal coordinate and a height coordinate.
5. The lane information detection method according to claim 1, characterized in that: The method further comprises: Acquire a first lane information three-dimensional coordinate and a second lane information three-dimensional coordinate, wherein the first lane information three-dimensional coordinate and the second lane information three-dimensional coordinate are lane information three-dimensional coordinates having a preset relationship with the monocular camera, the first lane information three-dimensional coordinate includes a first horizontal coordinate and a first height coordinate, and the second lane information three-dimensional coordinate includes a second horizontal coordinate and a second height coordinate; subtracting the first height coordinate from the second height coordinate to obtain a height coordinate difference; Subtracting the first horizontal coordinate from the second horizontal coordinate to obtain a horizontal coordinate difference; An inverse tangent calculation is performed on the ratio of the height coordinate difference to the horizontal coordinate difference to obtain an angle value of the road surface inclination.
6. The lane information detection method according to claim 5, characterized in that: The method further comprises: Obtaining an angle value between the monocular camera and a calibrated horizontal plane according to the external parameters of the monocular camera; The angle value of the angle between the monocular camera and the calibrated horizontal plane is added to the angle value of the road surface inclination to obtain the angle value of the pitch angle, where the pitch angle is the angle between the optical axis of the monocular camera and the lane information horizontal plane.
7. The lane information detection method according to claim 6, characterized in that: The method further comprises: Performing tangent calculation on the angle value of the road surface inclination to obtain an angle fluctuation value; Multiplying the angle fluctuation value by the infinite value to obtain the z-axis coordinate value of the road surface vanishing point, where the road surface vanishing point is the projection point of the infinite point in the three-dimensional coordinates of the lane information on the monocular lane image; A calibrated vanishing point is obtained, and the vehicle position state is determined according to the difference between the z-axis coordinate value of the road surface vanishing point and the z-axis coordinate value of the calibrated vanishing point, wherein the calibrated vanishing point is the projection point of the infinite point in the three-dimensional coordinates of the lane information on the monocular lane image when there is no angle between the calibrated horizontal plane and the lane information horizontal plane in the horizontal direction.
8. The lane information detection method according to claim 3, characterized in that: The lane information detection model further includes a two-dimensional image detection sub-model and a feature extraction layer, wherein the two-dimensional image detection sub-model is used to detect the two-dimensional coordinates of the lane image, and the feature extraction layer is used to extract the first feature and the second feature of the monocular lane image. The method further includes: Inputting the first feature into the depth information detection sub-model to perform depth information detection to obtain depth information; The second feature is input into the two-dimensional image detection sub-model to perform lane image detection to obtain the two-dimensional coordinates of the lane image.
9. A lane information detection system, characterized in that: include: The image acquisition module is configured to: acquire a monocular lane image through a monocular camera; The image detection module is configured to: input the monocular lane image into a lane information detection model, perform depth information detection, and obtain depth information; and, performing lane image detection to obtain two-dimensional coordinates of the lane image; The association relationship acquisition module is configured to: acquire depth information and two-dimensional coordinates of the lane image having an association relationship; wherein the association relationship represents the relationship between the depth information and the two-dimensional coordinates of the lane image obtained after the same pixel in the monocular lane image is detected by the lane information detection model; The three-dimensional coordinate generation module is configured to: perform a combination process on the depth information having an associated relationship and the two-dimensional coordinates of the lane picture to obtain the three-dimensional coordinates of the lane information.
10. An electronic device, characterized in that: Includes processor, memory and bus; The processor and the memory are communicatively connected via the bus; The memory stores computer program instructions executable by the processor, the processor being configured to: Obtain a monocular lane image through a monocular camera; Inputting the monocular lane image into a lane information detection model, performing depth information detection to obtain depth information; and, performing lane image detection to obtain two-dimensional coordinates of the lane image; Acquire depth information and two-dimensional coordinates of a lane image that have an associated relationship; wherein the associated relationship represents a relationship between the depth information and the two-dimensional coordinates of the lane image obtained after the same pixel in the monocular lane image is detected by the lane information detection model; The depth information and the two-dimensional coordinates of the lane image having the associated relationship are combined to obtain the three-dimensional coordinates of the lane information.