3D lane line detection method, electronic equipment and storage medium

Abstract

The invention provides a 3D lane line detection method, electronic equipment and a storage medium, and relates to the technical field of lane detection. In order to improve the generalization ability of 3D lane line detection, the provided 3D lane line detection method comprises the steps that a target image is acquired, and the target image is an image containing a lane line; inputting the target image into a pre-trained 3D lane line detection model to obtain a 3D lane line detection result of the target image; a training sample of the 3D lane line detection model is an image after pose disturbance, and the image after pose disturbance is used for simulating an image collected by a camera for a lane line on a road surface under the pose after disturbance. When the 3D lane line detection model is trained, lane line acquisition images in different poses are used, so that the 3D lane line detection model can perform more accurate 3D lane line detection on target images shot by a camera in different poses, and the influence of different camera mounting positions on 3D lane line detection is avoided.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of lane detection technology, and in particular to a 3D lane line detection method, electronic equipment, and storage medium. Background Art

[0002] 3D lane detection plays a vital role in planning vehicle routes and has become a rapidly developing topic in the field of autonomous driving in recent years. At present, domestic and foreign manufacturers generally use single-purpose input, that is, only use the images collected by the on-board forward-looking camera to predict the 3D lane perception results in the self-vehicle coordinate system (ie, the ego coordinate system).

[0003] However, the extrinsic parameters of the forward-looking cameras of different vehicles (i.e., the installation position of the cameras) are often different. Different camera extrinsics will lead to large differences in the images captured by the cameras, which will have a great impact on the accuracy of the predicted lane line perception results, resulting in poor generalization ability of 3D lane line detection.

[0004] Therefore, there is an urgent need for a new 3D lane line detection method to improve the generalization ability of 3D lane line detection and avoid the impact of different camera installation positions on 3D lane line detection. Summary of the invention

[0005] Embodiments of the present invention provide a 3D lane line detection method, an electronic device, and a storage medium to at least partially solve the problems existing in the related art.

[0006] A first aspect of an embodiment of the present invention provides a 3D lane line detection method, the method comprising:

[0007] Acquire a target image, wherein the target image is an image including lane lines;

[0008] Inputting the target image into a pre-trained 3D lane line detection model to obtain a 3D lane line detection result of the target image;

[0009] The training samples of the 3D lane line detection model are posture-perturbed images, and the posture-perturbed images are used to simulate images collected by a camera for lane lines on a road surface in a posture-perturbed state.

[0010] Optionally, the pose perturbed image is generated according to the following steps:

[0011] Obtaining an original position and posture of the camera, and obtaining a sample image captured by the camera for a lane line on a road surface at the original position and posture;

[0012] Perturbing the original posture to obtain the perturbed posture;

[0013] The pose-perturbed image is obtained according to the original pose, the perturbed pose and the sample image.

[0014] Optionally, the sample image carries a 3D lane line annotation result; and the training step of the 3D lane line detection model at least includes:

[0015] According to the original posture and the perturbed posture, the 3D lane line marking result is transformed to obtain the 3D lane line marking result after posture perturbation;

[0016] According to the 3D lane line annotation result after the posture disturbance and the image after the posture disturbance, the target model is trained to obtain the 3D lane line detection model.

[0017] Optionally, training a target model according to the pose-perturbed 3D lane line annotation result and the pose-perturbed image to obtain the 3D lane line detection model includes:

[0018] Inputting the pose-perturbed image into the target model to obtain a first 3D lane line prediction result of the pose-perturbed image;

[0019] Determining a first loss value of the target model according to the 3D lane line annotation result after the posture disturbance and the first 3D lane line prediction result;

[0020] According to the first loss value, the model parameters of the target model are updated to obtain the 3D lane line detection model.

[0021] Optionally, the sample image carries a 3D lane line annotation result; and the training step of the 3D lane line detection model at least includes:

[0022] According to the original posture and the perturbed posture, parameters of a feature extraction module in the target model are transformed to obtain a feature extraction module after posture perturbation;

[0023] Inputting the pose-perturbed image into the pose-perturbed feature extraction module to obtain 3D lane line features of the pose-perturbed image;

[0024] Processing the 3D lane line features of the pose-perturbed image by a 3D lane line prediction module in the target model to obtain a second 3D lane line prediction result of the pose-perturbed image;

[0025] Determining a second loss value of the target model according to the second 3D lane line prediction result and the 3D lane line annotation result;

[0026] According to the second loss value, the model parameters of the target model are updated to obtain the 3D lane line detection model.

[0027] Optionally, the feature extraction module in the target model includes: a 2D lane line feature extraction submodule and a feature conversion submodule, wherein the feature conversion submodule is used to convert the 2D lane line features extracted by the 2D lane line feature extraction submodule into 3D lane line features;

[0028] According to the original posture and the perturbed posture, parameters of a feature extraction module in the target model are transformed to obtain a feature extraction module after posture perturbation, including:

[0029] According to the original posture and the perturbed posture, the parameters of the feature conversion submodule are transformed to obtain the feature conversion submodule after the posture perturbation;

[0030] Inputting the pose perturbed image into the pose perturbed feature extraction module to obtain 3D lane line features of the pose perturbed image, including:

[0031] By means of the 2D lane line feature extraction submodule, 2D lane line feature extraction is performed on the pose perturbed image to obtain the 2D lane line feature;

[0032] The 2D lane line features are converted through the post-pose perturbation feature conversion submodule to obtain the 3D lane line features of the post-pose perturbation image.

[0033] Optionally, obtaining the pose perturbed image according to the original pose, the perturbed pose and the sample image is performed according to a homography transformation matrix, and the homography transformation matrix is ​​determined according to the following steps:

[0034] Obtaining coordinates of n position points on the road surface in a self-vehicle coordinate system, where the self-vehicle coordinate system is a coordinate system of the vehicle where the camera is located, and n is an integer greater than or equal to 4;

[0035] Projecting the n position points to the image coordinate system corresponding to the original posture, and projecting the n position points to the image coordinate system corresponding to the perturbed posture, to obtain n point pairs, each point pair comprising: a coordinate of a position point projected to the image coordinate system corresponding to the original posture, and a coordinate of the position point projected to the image coordinate system corresponding to the perturbed posture;

[0036] The homography transformation matrix is ​​determined according to the n point pairs.

[0037] Optionally, the n position points on the road surface are determined based on a 3D lane line marking result carried by the sample image.

[0038] Optionally, perturbing the original posture to obtain the perturbed posture includes:

[0039] Randomly select the disturbance amount within the target range;

[0040] The original posture is disturbed according to the disturbance amount to obtain a disturbed posture.

[0041] A second aspect of an embodiment of the present invention provides a 3D lane line detection device, the device comprising:

[0042] A first image acquisition module, used to acquire a target image, wherein the target image is an image including lane lines;

[0043] A lane line detection module is used to input the target image into a pre-trained 3D lane line detection model to obtain a 3D lane line detection result of the target image;

[0044] The training samples of the 3D lane line detection model are posture-perturbed images, and the posture-perturbed images are used to simulate images collected by a camera for lane lines on a road surface in a posture-perturbed state.

[0045] A third aspect of an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the 3D lane line detection method as described in the first aspect of the present invention are implemented.

[0046] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps in the 3D lane line detection method as described in the first aspect of the present invention are implemented.

[0047] A fifth aspect of an embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the 3D lane line detection method as described in the first aspect of the present invention.

[0048] In an embodiment of the present invention, 3D lane line detection is performed on a target image based on a pre-trained 3D lane line detection model. Since the training samples of the 3D lane line detection model are images after posture perturbation, and the images after posture perturbation are used to simulate the images collected by the camera for the lane lines on the road surface in the perturbed posture, the 3D lane line detection model in the embodiment of the present invention uses images collected for the lane lines on the road surface in various different postures (i.e., different camera installation positions) during training. The trained 3D lane line detection model can perform more accurate 3D lane line detection on target images taken by the camera in various different postures, thereby avoiding the influence of different camera installation positions on the 3D lane line detection and improving the generalization ability of 3D lane line detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0050] Figure 1 is a flow chart of a 3D lane line detection method according to an embodiment of the present invention;

[0051] Figure 2 is a flow chart of a method for generating a posture-perturbed image according to an embodiment of the present invention;

[0052] Figure 3 is a schematic diagram of determining and applying a homography transformation matrix according to an embodiment of the present invention;

[0053] Figure 4 is a flow chart of a 3D lane detection model training method according to an embodiment of the present invention;

[0054] Figure 5 is a flowchart of another 3D lane detection model training method according to an embodiment of the present invention;

[0055] Figure 6 is a training schematic diagram of a 3D lane detection model according to an embodiment of the present invention;

[0056] Figure 7 is a schematic diagram of a 3D lane line detection device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] Currently, in order to solve the problem of poor lane line perception accuracy caused by different camera installation positions, the current strategy is to simulate the image into an image with fixed external parameters during model training / testing. However, this strategy has at least the following disadvantages:

[0059] 1. During testing, the original image needs to be simulated under fixed external parameters, which increases the time consumption of the system. 2. As the number of vehicle models in the training data increases, it is difficult to find a fixed external parameter that is applicable to all vehicle models. This will cause the training data of some vehicle models to be greatly distorted after simulation. 3. The looseness of the camera will cause the installation position of the camera to change. The installation positions of cameras of different vehicle models are also different. The images collected by cameras with different installation positions for the lane lines on the road will be very different. As a result, the 3D lane line detection model based on the 3D lane line prediction training based on the images taken by the camera with fixed external parameters under this strategy still does not have the generalization ability.

[0060] Therefore, in order to at least partially solve one or more of the above problems and other potential problems, an embodiment of the present invention proposes a 3D lane line detection method, which detects 3D lane lines on a target image through a pre-trained 3D lane line detection model. Since the 3D lane line detection model uses images collected of lane lines on the road surface at various different postures during training, the trained 3D lane line detection model can perform more accurate 3D lane line detection on target images taken by the camera at various different postures, thereby avoiding the impact of different camera installation positions on 3D lane line detection and improving the generalization ability of 3D lane line detection. Figure 1 , shows a flow chart of a 3D lane line detection method according to an embodiment of the present invention. The 3D lane line detection method provided by the embodiment of the present invention can be used to process a single image or to process each frame of an image in a video. The 3D lane line detection method provided by the embodiment of the present invention may include the following steps:

[0061] S101, acquiring a target image, where the target image is an image including lane lines.

[0062] In the embodiment of the present invention, the target image may be an image containing lane lines taken by a vehicle-mounted front-view camera in real time or in non-real time, or an image containing lane lines taken by a common camera placed on the vehicle in real time or in non-real time. The target image of this embodiment is used for detecting 3D lane lines.

[0063] S102, inputting the target image into a pre-trained 3D lane line detection model to obtain a 3D lane line detection result of the target image.

[0064] In an embodiment of the present invention, a 3D lane line detection model is pre-trained, and the training samples of the 3D lane line detection model are posture-perturbed images, which are used to simulate images captured by a camera of lane lines on the road surface in the perturbed posture.

[0065] Specifically, the posture perturbed image of the embodiment of the present invention can be a simulated image of the lane lines on the road surface captured by the camera in a variety of postures after disturbance, that is, the lane line images taken by the camera under a variety of different external parameters are simulated, where the external parameters represent the installation position of the camera.

[0066] Therefore, in the embodiment of the present invention, the lane line detection result of the target image can be predicted based on the pre-trained 3D lane line detection model. Since the training samples of the 3D lane line detection model are the images after posture perturbation, and the images after posture perturbation are used to simulate the images collected by the camera for the lane lines on the road surface in the perturbed posture, the 3D lane line detection model in the embodiment of the present invention uses images collected for the lane lines on the road surface in various different postures (i.e., different camera installation positions) during training. The trained 3D lane line detection model can perform more accurate 3D lane line detection on the target images taken by the camera in various different postures, thereby avoiding the influence of different camera installation positions on the 3D lane line detection and improving the generalization ability of 3D lane line detection.

[0067] In combination with the above embodiments, refer to Figure 2 , shows a flow chart of a method for generating a posture perturbed image according to an embodiment of the present invention. Specifically, the method for generating a posture perturbed image may include the following steps:

[0068] S201, obtaining an original position and posture of the camera, and obtaining a sample image captured by the camera for a lane line on a road surface in the original position and posture.

[0069] In the embodiment of the present invention, the original posture of the camera can be obtained, and the sample image can be obtained. The original posture refers to the posture before the camera installation position is disturbed, and the sample image refers to the image collected by the camera at the original posture for the lane line on the road surface.

[0070] S202, perturbing the original posture to obtain the perturbed posture.

[0071] In the embodiment of the present invention, after obtaining the original posture of the camera, the original posture needs to be disturbed to obtain a disturbed posture. The original posture may be randomly disturbed to obtain the disturbed posture, thereby enriching the posture diversity of the disturbed posture.

[0072] Specifically, in an optional implementation manner, step S202 may include the following sub-steps:

[0073] S2021, randomly select a disturbance amount within the target range.

[0074] S2022: perturb the original posture according to the disturbance amount to obtain a perturbed posture.

[0075] In an embodiment of the present invention, a disturbance amount can be randomly selected within a target range, wherein the target range can be a user-defined disturbance range, which is not limited. Generally speaking, the larger the disturbance range, the more robust the 3D lane line detection model is, but the more difficult it is to converge, so it is necessary to set a suitable target range.

[0076] Specifically, three Euler angles can be used<pitch,yaw,roll> And the translation of the corresponding coordinate axis<x,y,z> The disturbance on the camera pose is used to generate the disturbance of the camera pose, that is, the disturbance of the camera pose δ=<δ pitch ,δ yaw ,δ roll ,δ x ,δ y ,δ z >∈R 4×4 , where δ pitch , δ yaw , δ roll , δ x , δ y , δ z That is, the user-defined disturbance range. In an optional implementation, the original posture is H e2c , then the posture after disturbance can be H e2c ′=H e2c @δ, where @ represents matrix multiplication.

[0077] In the embodiment of the present invention, after randomly selecting a perturbation amount, the original pose can be perturbed according to the perturbation amount to obtain a perturbed pose. For an original pose, multiple perturbations can be randomly selected to obtain multiple different perturbations to enrich the perturbation of the sample image and improve the generalization effect of the 3D lane line detection model to camera looseness robustness and easy crossing of vehicles.

[0078] S203, obtaining the pose-perturbed image according to the original pose, the perturbed pose and the sample image.

[0079] In the embodiment of the present invention, after obtaining the original pose and the perturbed pose, the training sample of the 3D lane line detection model can be obtained according to the original pose, the perturbed pose and the obtained sample image: the pose perturbed image. Specifically, the sample image can be processed accordingly according to the original pose and the perturbed pose to obtain the pose perturbed image.

[0080] In fact, for one original pose, multiple perturbed poses can be obtained, so that the sample image can be processed accordingly according to each perturbed pose and the original pose to obtain the pose perturbed image corresponding to each perturbed pose, so as to enrich the training samples of the 3D lane line detection model.

[0081] In an optional implementation, step S203 is performed according to a homography transformation matrix, wherein the method for determining the homography transformation matrix may include the following steps:

[0082] S2031, obtaining coordinates of n position points on the road surface in the vehicle coordinate system, where the vehicle coordinate system is the coordinate system of the vehicle where the camera is located, and n is an integer greater than or equal to 4.

[0083] In the embodiment of the present invention, the coordinates of n position points on the road surface expressed in a homogeneous form in the ego coordinate system (ie, the ego coordinate system) can be obtained, namely, P ego ∈R 4×n , where the ego-vehicle coordinate system is the coordinate system of the vehicle where the camera is located, n is an integer greater than or equal to 4, and these n position points are not on the same straight line, and the n position points on the road surface are n points on the road surface in the real physical space.

[0084] S2032, projecting the n position points to the image coordinate system corresponding to the original posture, and projecting the n position points to the image coordinate system corresponding to the perturbed posture, to obtain n point pairs, each point pair including: the coordinates of a position point projected to the image coordinate system corresponding to the original posture, and the coordinates of the position point projected to the image coordinate system corresponding to the perturbed posture.

[0085] In the embodiment of the present invention, after determining the coordinates of n position points in the vehicle coordinate system, the n position points can be projected to the image coordinate system corresponding to the original posture to obtain n first projection coordinates; and the n position points can be projected to the image coordinate system corresponding to the perturbed posture to obtain n second projection coordinates; thereby obtaining n point pairs consisting of n first projection coordinates and second projection coordinates. Each of the n point pairs includes: the coordinates of a position point projected to the image coordinate system corresponding to the original posture, and the coordinates of the position point projected to the image coordinate system corresponding to the perturbed posture.

[0086] For example, the set of n first projection coordinates may be: Among them, U ori is the set of n position point coordinates in the image coordinate system corresponding to the original pose, K is the intrinsic parameter of the camera, @ is the matrix multiplication, H e2c Represents the transformation matrix from the camera coordinate system to the vehicle coordinate system, represents the transformation matrix from the ego-vehicle coordinate system to the camera coordinate system. The transformation matrix from the ego-vehicle coordinate system to the camera coordinate system can be used to project the points in the ego-vehicle coordinate system to the camera coordinate system. ego are the coordinates of n positions in the vehicle coordinate system, so, The representative operation is to first transform the point P in the vehicle coordinate system ego Convert to the camera coordinate system, and then project it onto the image through the intrinsic parameter K. When expressed in homogeneous coordinates, the two-dimensional point on the image is represented by (u,v,1), so (u1,v1,1),…,(u n ,v n ,1) is the image coordinates of n position points projected to the image coordinate system corresponding to the original pose.

[0087] For example, the set of n second projection coordinates may be: Among them, U warped is the coordinate set of n position points in the image coordinate system corresponding to the perturbed pose, K is the intrinsic parameter of the camera, @ is the matrix multiplication, H e2c Represents the transformation matrix from the camera coordinate system to the vehicle coordinate system, represents the transformation matrix from the ego-vehicle coordinate system to the camera coordinate system. The transformation matrix from the ego-vehicle coordinate system to the camera coordinate system can be used to project the points in the ego-vehicle coordinate system to the camera coordinate system. δ is the perturbation amount. P ego are the coordinates of n positions in the vehicle coordinate system. When expressed in homogeneous coordinates, the two-dimensional points on the image are represented by (u,v,1), so (u′1,v′1,1),…,(u′ n ,v′ n ,1) is the image coordinates of n position points projected to the image coordinate system corresponding to the perturbed posture.

[0088] Since the lane lines fall on the road surface, in a preferred embodiment, the sample image carries the 3D lane line marking results, and the n position points on the road surface can be replaced by the GT of the lane lines (i.e., the 3D lane line marking), that is, the n position points on the road surface are determined based on the 3D lane line marking results carried by the sample image.

[0089] Correspondingly, when the n position points on the road surface are determined according to the 3D lane line marking results carried by the sample image, since the 3D lane line marking results carried by the sample image are in the image coordinate system corresponding to the original posture, there is no need to perform a projection transformation on the image coordinate system corresponding to the original posture when performing a projection transformation. The coordinates of the position point projected to the image coordinate system corresponding to the original posture are: the coordinates of the position point on the road surface in the sample image itself. In this case, it is only necessary to project the position point to the coordinates in the image coordinate system corresponding to the perturbed posture, thereby reducing the number of projections and saving computing resources.

[0090] S2033: Determine the homography transformation matrix according to the n point pairs.

[0091] In the embodiment of the present invention, after obtaining n point pairs, the homography transformation matrix can be determined by using the RANSAC algorithm (ie, random sampling consensus algorithm).

[0092] In an embodiment of the present invention, after the homography transformation matrix is ​​determined, the sample image can be subjected to a corresponding homography transformation based on the homography transformation matrix to transform the sample image into an image after posture perturbation, that is, the sample image can be simulated into an image taken by a camera after posture perturbation, thereby simulating sample images collected for lane lines on the road surface in various postures.

[0093] See also Figure 3 , shows a schematic diagram of determining and applying a homography transformation matrix according to an embodiment of the present invention. Figure 3 In, 1. Figure 3 As shown in the left figure, the GT points (i.e., 3D lane line marking result) of the lane line in the ego coordinate system (i.e., ego coordinate system) are selected, and the number of n position points on the road surface selected is at least 4;

[0094] 2. If Figure 3 As shown in the upper middle figure, the original external parameters of the camera (i.e., the original pose) are used to project these n position points into the 2D image system (i.e., the image coordinate system corresponding to the original pose);

[0095] 3. If Figure 3 As shown in the middle and lower figure, the transformed external parameters (i.e., the perturbed posture) are used to project these n position points into the 2D image system (i.e., the image coordinate system corresponding to the perturbed posture) to form n point pairs;

[0096] 4. According to the correspondence between the points in the n point pairs, calculate the homography transformation matrix H between the images. Specifically, you can simply call cv2.findHomography(src_pixels,dst_pixels) to solve H, and then convert the original sample image ( Figure 3The upper right figure in the figure) performs the corresponding homography transformation based on the homography transformation matrix H to generate the image after pose disturbance ( Figure 3 , lower right figure).

[0097] In combination with any of the above embodiments, for the training of the 3D lane line detection model, one of the two training methods can be selected: training based on camera extrinsic parameter (camera installation posture) transformation and training based on GT (3D lane line annotation) transformation. Also, during the training process of the 3D lane line detection model, each training can select any of the above two training methods until the training is completed, so that the trained 3D lane line detection model can perform more accurate 3D lane line detection for target images captured by the camera in various different postures.

[0098] Reference Figure 4 , shows a flow chart of a 3D lane line detection model training method according to an embodiment of the present invention. The embodiment of the present invention performs model training based on GT (3D lane line annotation) transformation. The 3D lane line detection model training method provided by the embodiment of the present invention may specifically include the following steps:

[0099] S401, transforming the 3D lane line marking result according to the original posture and the perturbed posture to obtain the 3D lane line marking result after posture perturbation.

[0100] In an embodiment of the present invention, the sample image carries a 3D lane line marking result. After obtaining the original pose and the perturbed pose of the camera, the 3D lane line marking result (original 3D lane line marking result) carried in the sample image can be transformed based on the original pose and the perturbed pose to obtain the 3D lane line marking result after pose perturbation.

[0101] In one implementation, the 3D lane line marking result may be transformed based on the following formula 1 to obtain the 3D lane line marking result after posture disturbance:

[0102] GT ego ′=H e2c @δ@inv(H e2c )@GT ego Formula 1

[0103] Among them, GT ego ′ is the 3D lane line marking result after pose disturbance, H e2c is the original pose, δ is the perturbation amount, H e2c @δ is the pose after perturbation, inv(H e2c )for Represents the transformation matrix from the vehicle coordinate system to the camera coordinate system, @ is the multiplication matrix, GT egois the 3D lane line annotation result carried in the sample image. In the training process of a model, the perturbation amount used to generate the image after pose perturbation is the same as the perturbation amount used to generate the 3D lane line annotation result after pose perturbation.

[0104] S402, training a target model according to the pose-perturbed 3D lane line annotation result and the pose-perturbed image to obtain the 3D lane line detection model.

[0105] In the embodiment of the present invention, the target model is an initial model for generating 3D lane line detection results based on images during model training. After obtaining the 3D lane line labeling results after pose disturbance, in the training process of this target model, the 3D lane line labeling results after pose disturbance are used as labels for this model training, the original external parameters of the camera, i.e., the original pose, are used to configure the hyperparameters of the target model, and the image after pose disturbance is used as the input of the target model to train the target model to obtain a 3D lane line detection model. In each model training process, the hyperparameters configured in the target model will not change with the update of the model parameters, but in the process of model training, based on the selection of training methods, the hyperparameters are configured before each model training.

[0106] In the model training process of the embodiment of the present invention, in addition to the model input being the image after posture perturbation, the perturbation change is further reflected in the GT, so that the change of the GT can simulate different numerical ranges of the GT, that is, simulate the 3D lane line marking when the car changes lanes or bumps, thereby enhancing the model's robustness to camera looseness and generalization across vehicles.

[0107] Specifically, in the embodiment of the present invention, S402 specifically includes the following steps:

[0108] S4021, input the pose-perturbed image into the target model to obtain a first 3D lane line prediction result of the pose-perturbed image.

[0109] S4022: Determine a first loss value of the target model according to the 3D lane line annotation result after the posture disturbance and the first 3D lane line prediction result.

[0110] S4023: Update the model parameters of the target model according to the first loss value to obtain the 3D lane line detection model.

[0111] In an embodiment of the present invention, firstly, the hyperparameters of the target model are configured with the original posture, and the image after the posture perturbation is input into the target model to obtain the first 3D lane line prediction result output by the target model and predicted based on the image after the posture perturbation. Among them, the first 3D lane line prediction result is a 3D lane line prediction result obtained by model training based on the transformation of the 3D lane line annotation result. After obtaining the output first 3D lane line prediction result, the loss can be calculated according to the first 3D lane line prediction result and the 3D lane line annotation result after the posture perturbation to obtain the first loss value of the target model, and the model parameters of the target model are updated based on the first loss value, so as to obtain the 3D lane line detection model. Specifically, all model parameters of the target model can be updated based on the first loss value, but in each model training process, when the model is trained in the GT transformation mode, the hyperparameters configured based on the original external parameters (i.e., the original posture) of the target model will not change with the update of the model parameters, but the hyperparameters are configured before each model training.

[0112] Reference Figure 5 , shows a flow chart of another 3D lane line detection model training method according to an embodiment of the present invention. The embodiment of the present invention performs model training based on camera extrinsic parameter (camera installation posture) transformation. The 3D lane line detection model training method provided by the embodiment of the present invention may specifically include the following steps:

[0113] S501, transforming the parameters of a feature extraction module in a target model according to the original posture and the perturbed posture to obtain a feature extraction module after posture perturbation.

[0114] In an embodiment of the present invention, the sample image carries a 3D lane line annotation result, which is the label for this model training. The target model is an initial model for generating 3D lane line detection results based on the image during the model training process, and the target model includes at least: a feature extraction module and a 3D lane line prediction module.

[0115] The embodiment of the present invention can transform the parameters of the feature extraction module based on the original posture and the perturbed posture to obtain the feature extraction module after the posture perturbation after the parameter transformation. In the training process of a model, the perturbed posture used to generate the posture perturbed image is the same as the perturbed posture used to transform the parameters of the feature extraction module.

[0116] S502, inputting the pose-perturbed image into a pose-perturbed feature extraction module to obtain 3D lane line features of the pose-perturbed image.

[0117] In an embodiment of the present invention, after determining the feature extraction module after the pose disturbance, the pose-perturbed image is input into the feature extraction module after the pose disturbance to obtain the output of the feature extraction module after the pose disturbance: the 3D lane line features of the pose-perturbed image.

[0118] Specifically, in an optional implementation, the feature extraction module in the target model includes: a 2D lane line feature extraction submodule and a feature conversion submodule, and the feature conversion submodule is used to convert the 2D lane line features extracted by the 2D lane line feature extraction submodule into 3D lane line features. In the embodiment of the present invention, S501 specifically includes the following S5011, and S502 specifically includes the following S5021 and S5022:

[0119] S5011, transforming the parameters of the feature conversion submodule according to the original posture and the perturbed posture to obtain the feature conversion submodule after posture perturbation.

[0120] In the embodiment of the present invention, the parameters of the feature conversion submodule in the feature extraction module are transformed based on the original posture and the posture after disturbance, so as to obtain the feature conversion submodule after the parameter transformation and the posture disturbance. It can be understood that before the parameter transformation, the parameters of the feature conversion submodule are configured based on the original posture; after the parameter transformation, the parameters of the feature conversion submodule are configured based on the posture after disturbance, so as to obtain the feature conversion submodule after the posture disturbance.

[0121] S5021: Perform 2D lane line feature extraction on the pose perturbed image through the 2D lane line feature extraction submodule to obtain the 2D lane line feature.

[0122] In an embodiment of the present invention, the pose perturbed image is input into a target model, and firstly, 2D lane line features are extracted from the pose perturbed image through a 2D lane line feature extraction submodule in the target model to obtain the extracted 2D lane line features of the pose perturbed image.

[0123] S5022: The 2D lane line features are converted through the pose perturbed feature conversion submodule to obtain 3D lane line features of the pose perturbed image.

[0124] In an embodiment of the present invention, after obtaining the 2D lane line features of the image after posture disturbance, the 2D lane line features are converted by a feature conversion submodule after posture disturbance to obtain the 3D lane line features of the image after posture disturbance.

[0125] For example, in an optional embodiment, the feature conversion submodule may be an LSS model to convert 2D lane line features to 3D lane line features. For example, the 2D lane line features may be RV features, and the 3D lane line features may be BEV features; wherein the RV features are distance view features, and the BEV features are bird's-eye view features.

[0126] S503, processing the 3D lane line features of the pose-perturbed image through the 3D lane line prediction module in the target model to obtain a second 3D lane line prediction result of the pose-perturbed image.

[0127] In an embodiment of the present invention, after obtaining the 3D lane line features of the pose perturbed image, the 3D lane line features of the pose perturbed image can be processed by the 3D lane line prediction module in the target model to obtain the output of the 3D lane line prediction module: the second 3D lane line prediction result of the pose perturbed image. The second 3D lane line prediction result is a 3D lane line prediction result obtained by model training based on parameter transformation of the feature extraction module.

[0128] S504: Determine a second loss value of the target model according to the second 3D lane line prediction result and the 3D lane line marking result.

[0129] In an embodiment of the present invention, after obtaining the second 3D lane line prediction result, the loss can be calculated based on the second 3D lane line prediction result and the 3D lane line annotation result carried by the sample image to obtain a second loss value of the target model.

[0130] S505: Update the model parameters of the target model according to the second loss value to obtain the 3D lane line detection model.

[0131] In an embodiment of the present invention, after determining the second loss value of the target model, the model parameters of the target model can be updated based on the second loss value to obtain a trained 3D lane line detection model. Specifically, all model parameter parts of the target model can be updated based on the second loss value, for example, the model parameters of the 2D lane line feature extraction submodule, the model parameters of the feature conversion submodule, and the model parameters of the 3D lane line prediction module in the target model are updated to obtain a 3D lane line detection model. In each model training process, when the model is trained in a camera extrinsic parameter transformation manner, the hyperparameters of the target model configured based on the perturbation posture will not change with the update of the model parameters, but the hyperparameters are configured before each model training.

[0132] In the model training process of the embodiment of the present invention, in addition to the model input being the image after posture perturbation, the disturbance change is further reflected in the extrinsic parameters of the model parameters, so that the change of the model parameters can simulate the different extrinsic parameter values ​​involved in the training, thereby achieving robustness when the extrinsic parameters need to be replaced after the camera is loose, and enhancing the generalization of the model for easy-to-cross vehicles.

[0133] See also Figure 6 , shows a training schematic diagram of a 3D lane line detection model according to an embodiment of the present invention. For each model training in the 3D lane line detection model training process, training mode 1 or training mode 2 can be randomly selected for model training until the model training is completed.

[0134] Among them, training method 1 is:

[0135] First, randomly sample the camera Pose perturbation (i.e., the camera perturbation amount) within the target range δ1=<δ pitch ,δ yaw , δ roU ,δ x , δ y , δ z >, and based on the camera Pose perturbation δ1, calculate the homography transformation matrix H corresponding to the sample image Image, and then perform the corresponding homography transformation cv2.warpPespective(Image,H) on the sample image Image to obtain the pose perturbed image Image′ as the input of the target model.

[0136] Secondly, based on the camera external parameter H e2c (i.e. original pose) is used to configure parameters for the feature conversion submodule (e.g. LSS module) in the target model. And based on the camera external parameter H e2c (i.e. original pose) and camera Pose perturbation δ1, determine the pose after perturbation H e2c ′, based on the perturbed pose H e2c ′, original pose H e2c And the 3D lane line annotation result GT carried by the sample image ego , determine the 3D lane line annotation result GT after pose disturbance ego ', specifically, it can be GT ego ′=H e2c @δ1@inv(H e2c )@GT ego .

[0137] Then, the pose perturbed image Image′ is input into the 2D lane line feature extraction submodule of the target model to extract the 2D lane line features, and then the 2D lane line features are converted through the feature conversion submodule to obtain 3D lane line features (such as BEV Feature). Then, the 3D lane line features are processed by the 3D lane line prediction module to obtain the predicted first 3D lane line prediction result.

[0138] Finally, the first 3D lane line prediction result and the 3D lane line annotation result GT after pose perturbation are combined. ego 'Calculate the loss LOSS, and update the model parameters based on the calculated first loss value, thereby completing this model training.

[0139] Training method 2 is:

[0140] First, randomly sample the camera Pose perturbation (i.e., the camera perturbation amount) within the target range δ2=<δ pitch , δ yaw , δ rou ,δ x , δ y , δ z >, and based on the camera Pose perturbation δ2, calculate the homography transformation matrix H corresponding to the sample image Image, and then perform the corresponding homography transformation cv2.warpPespective(Image,H) on the sample image Image to obtain the pose perturbed image Image′ as the input of the target model.

[0141] Secondly, based on the camera external parameter H e2c (i.e. original pose) and camera Pose perturbation δ2, determine the perturbed pose H e2c ′, specifically, it can be H e2c ′=H e2c @δ2. And based on the perturbed posture H e2c ′ is the configuration parameter of the feature conversion submodule (such as LSS module) in the target model, that is, the original external parameter H in the feature conversion submodule (such as LSS module) in the target model is converted to e2c Replaced by the perturbed pose H e2c ′.

[0142] Then, the pose perturbed image Image′ is input into the 2D lane line feature extraction submodule of the target model to extract the 2D lane line features, and then the 2D lane line features are converted by the feature conversion submodule after replacing the external parameters to obtain 3D lane line features (such as BEV Feature), and then the 3D lane line features are processed by the 3D lane line prediction module to obtain the predicted second 3D lane line prediction result.

[0143] Finally, the second 3D lane line prediction result is compared with the 3D lane line annotation result GT carried by the sample image ego The loss LOSS is calculated, and the model parameters are updated based on the calculated second loss value, thereby completing this model training.

[0144] Based on the same inventive concept, an embodiment of the present invention provides a 3D lane line detection device, referring to Figure 7 , Figure 7 is a schematic diagram of a 3D lane line detection device provided by an embodiment of the present invention, the device comprising:

[0145] A first image acquisition module, used to acquire a target image, wherein the target image is an image including lane lines;

[0146] A lane line detection module is used to input the target image into a pre-trained 3D lane line detection model to obtain a 3D lane line detection result of the target image;

[0147] The training samples of the 3D lane line detection model are posture-perturbed images, and the posture-perturbed images are used to simulate images collected by a camera for lane lines on a road surface in a posture-perturbed state.

[0148] Optionally, the device further includes: a second image acquisition module, configured to generate the posture perturbed image; the second image acquisition module includes:

[0149] A first acquisition submodule is used to obtain an original position and posture of the camera, and to obtain a sample image collected by the camera for a lane line on a road surface in the original position and posture;

[0150] A first perturbation submodule, used for perturbing the original posture to obtain the perturbed posture;

[0151] The second perturbation submodule is used to obtain the pose perturbed image according to the original pose, the perturbed pose and the sample image.

[0152] Optionally, the sample image carries a 3D lane line annotation result; the device further includes: a first model training module, used to train the 3D lane line detection model; the first model training module includes:

[0153] A first transformation submodule, configured to transform the 3D lane line marking result according to the original posture and the perturbed posture to obtain the 3D lane line marking result after posture perturbation;

[0154] The first training submodule is used to train the target model according to the 3D lane line annotation result after the posture disturbance and the image after the posture disturbance to obtain the 3D lane line detection model.

[0155] Optionally, the first training submodule includes:

[0156] A second acquisition submodule is used to input the pose-perturbed image into the target model to obtain a first 3D lane line prediction result of the pose-perturbed image;

[0157] A first loss determination submodule, configured to determine a first loss value of the target model according to the 3D lane line annotation result after the posture disturbance and the first 3D lane line prediction result;

[0158] The first updating submodule is used to update the model parameters of the target model according to the first loss value to obtain the 3D lane line detection model.

[0159] Optionally, the sample image carries a 3D lane line annotation result; the device further includes: a second model training module, used to train the 3D lane line detection model; the second model training module includes:

[0160] A second transformation submodule, used for transforming the parameters of the feature extraction module in the target model according to the original posture and the perturbed posture, to obtain the feature extraction module after the posture perturbation;

[0161] A third acquisition submodule is used to input the pose-perturbed image into the pose-perturbed feature extraction module to obtain the 3D lane line features of the pose-perturbed image;

[0162] A fourth acquisition submodule is used to process the 3D lane line features of the pose perturbed image through the 3D lane line prediction module in the target model to obtain a second 3D lane line prediction result of the pose perturbed image;

[0163] A second loss determination submodule, configured to determine a second loss value of the target model according to the second 3D lane line prediction result and the 3D lane line annotation result;

[0164] The second updating submodule is used to update the model parameters of the target model according to the second loss value to obtain the 3D lane line detection model.

[0165] Optionally, the feature extraction module in the target model includes: a 2D lane line feature extraction submodule and a feature conversion submodule, wherein the feature conversion submodule is used to convert the 2D lane line features extracted by the 2D lane line feature extraction submodule into 3D lane line features;

[0166] The second transformation submodule includes:

[0167] A parameter transformation submodule, used for transforming the parameters of the feature transformation submodule according to the original posture and the perturbed posture to obtain a feature transformation submodule after posture perturbation;

[0168] The third acquisition submodule includes:

[0169] A first feature determination submodule is used to extract 2D lane line features from the pose perturbed image through the 2D lane line feature extraction submodule to obtain the 2D lane line features;

[0170] The second feature determination submodule is used to convert the 2D lane line features through the feature conversion submodule after the posture disturbance to obtain the 3D lane line features of the image after the posture disturbance.

[0171] Optionally, the second perturbation submodule includes: a homography transformation matrix determination submodule, used to determine the homography transformation matrix, and the homography transformation matrix determination submodule includes:

[0172] A first coordinate acquisition submodule is used to obtain the coordinates of n position points on the road surface in the vehicle coordinate system, where the vehicle coordinate system is the coordinate system of the vehicle where the camera is located, and n is an integer greater than or equal to 4;

[0173] A second coordinate acquisition submodule is used to project the n position points to the image coordinate system corresponding to the original posture, and project the n position points to the image coordinate system corresponding to the perturbed posture, to obtain n point pairs, each point pair including: the coordinates of a position point projected to the image coordinate system corresponding to the original posture, and the coordinates of the position point projected to the image coordinate system corresponding to the perturbed posture;

[0174] The matrix acquisition submodule is used to determine the homography transformation matrix according to the n point pairs.

[0175] Optionally, the n position points on the road surface are determined based on a 3D lane line marking result carried by the sample image.

[0176] Optionally, the first disturbance submodule includes:

[0177] The disturbance amount determination submodule is used to randomly select the disturbance amount within the target range;

[0178] The posture determination submodule is used to perturb the original posture according to the disturbance amount to obtain a disturbed posture.

[0179] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0180] Based on the same inventive concept, an embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the 3D lane line detection method described in any of the above embodiments are implemented.

[0181] Based on the same inventive concept, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps in the 3D lane line detection method described in any of the above embodiments are implemented.

[0182] Based on the same inventive concept, an embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps in the 3D lane line detection method described in any of the above embodiments.

[0183] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0184] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0185] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable image processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable image processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0186] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable image processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0187] These computer program instructions can also be loaded onto a computer or other programmable image processing terminal device, so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0188] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0189] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0190] The 3D lane line detection method, electronic device and storage medium provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A 3D lane line detection method, characterized in that: The method comprises: Acquire a target image, wherein the target image is an image including lane lines; Inputting the target image into a pre-trained 3D lane line detection model to obtain a 3D lane line detection result of the target image; The training samples of the 3D lane line detection model are posture-perturbed images, and the posture-perturbed images are used to simulate images captured by a camera of lane lines on a road surface in a posture-perturbed state.

2. The 3D lane line detection method according to claim 1, characterized in that: The pose perturbed image is generated according to the following steps: Obtaining an original position and posture of the camera, and obtaining a sample image captured by the camera for a lane line on a road surface in the original position and posture; Perturbing the original posture to obtain the perturbed posture; The pose-perturbed image is obtained according to the original pose, the perturbed pose and the sample image.

3. The 3D lane line detection method according to claim 2, characterized in that: The sample image carries a 3D lane line annotation result; the training step of the 3D lane line detection model at least includes: According to the original posture and the perturbed posture, the 3D lane line marking result is transformed to obtain the 3D lane line marking result after posture perturbation; According to the 3D lane line annotation result after the posture disturbance and the image after the posture disturbance, the target model is trained to obtain the 3D lane line detection model.

4. The 3D lane line detection method according to claim 3, characterized in that: According to the 3D lane line annotation result after the posture disturbance and the image after the posture disturbance, the target model is trained to obtain the 3D lane line detection model, including: Inputting the pose-perturbed image into the target model to obtain a first 3D lane line prediction result of the pose-perturbed image; Determining a first loss value of the target model according to the 3D lane line annotation result after the posture disturbance and the first 3D lane line prediction result; According to the first loss value, the model parameters of the target model are updated to obtain the 3D lane line detection model.

5. The 3D lane line detection method according to claim 2, characterized in that: The sample image carries a 3D lane line annotation result; the training step of the 3D lane line detection model at least includes: According to the original posture and the perturbed posture, parameters of a feature extraction module in the target model are transformed to obtain a feature extraction module after posture perturbation; Inputting the pose-perturbed image into the pose-perturbed feature extraction module to obtain 3D lane line features of the pose-perturbed image; Processing the 3D lane line features of the pose-perturbed image through the 3D lane line prediction module in the target model to obtain a second 3D lane line prediction result of the pose-perturbed image; Determining a second loss value of the target model according to the second 3D lane line prediction result and the 3D lane line annotation result; According to the second loss value, the model parameters of the target model are updated to obtain the 3D lane line detection model.

6. The 3D lane line detection method according to claim 5, characterized in that: The feature extraction module in the target model includes: a 2D lane line feature extraction submodule and a feature conversion submodule, wherein the feature conversion submodule is used to convert the 2D lane line features extracted by the 2D lane line feature extraction submodule into 3D lane line features; According to the original posture and the perturbed posture, parameters of a feature extraction module in the target model are transformed to obtain a feature extraction module after posture perturbation, including: According to the original posture and the perturbed posture, the parameters of the feature conversion submodule are transformed to obtain the feature conversion submodule after the posture perturbation; Inputting the pose perturbed image into the pose perturbed feature extraction module to obtain 3D lane line features of the pose perturbed image, including: By means of the 2D lane line feature extraction submodule, 2D lane line feature extraction is performed on the pose perturbed image to obtain the 2D lane line feature; The 2D lane line features are converted through the post-pose perturbation feature conversion submodule to obtain the 3D lane line features of the post-pose perturbation image.

7. The 3D lane line detection method according to any one of claims 2 to 6, characterized in that: According to the original pose, the perturbed pose and the sample image, the pose perturbed image is obtained according to a homography transformation matrix, and the homography transformation matrix is ​​determined according to the following steps: Obtaining coordinates of n position points on the road surface in a self-vehicle coordinate system, where the self-vehicle coordinate system is a coordinate system of the vehicle where the camera is located, and n is an integer greater than or equal to 4; Projecting the n position points to the image coordinate system corresponding to the original posture, and projecting the n position points to the image coordinate system corresponding to the perturbed posture, to obtain n point pairs, each point pair comprising: a coordinate of a position point projected to the image coordinate system corresponding to the original posture, and a coordinate of the position point projected to the image coordinate system corresponding to the perturbed posture; The homography transformation matrix is ​​determined according to the n point pairs.

8. The 3D lane line detection method according to claim 7, characterized in that: The n position points on the road surface are determined according to the 3D lane line marking results carried by the sample image.

9. The 3D lane line detection method according to any one of claims 2 to 8, characterized in that: The original posture is disturbed to obtain the disturbed posture, including: Randomly select the disturbance amount within the target range; The original posture is disturbed according to the disturbance amount to obtain a disturbed posture.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the 3D lane line detection method according to any one of claims 1 to 9 are implemented.

11. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the 3D lane line detection method described in any one of claims 1-9 are implemented.

12. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the steps of the 3D lane line detection method as described in any one of claims 1 to 9.

Images