Lane detection network training method, driving control method and device

Abstract

The present disclosure provides a lane line detection network training method, a driving control method and an apparatus, wherein the method comprises: obtaining a sample image of a forward-looking angle and a supervisory mask image determined based on three-dimensional lane line data corresponding to the sample image; wherein the supervisory mask image is used to characterize the overhead position information of the lane line in the sample image; inputting the sample image into a lane line detection network to be trained, and determining an overhead feature map of the sample image; determining a predicted mask image of the sample image based on the overhead feature map, and training the lane line detection network to be trained based on the predicted mask image and the supervisory mask image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a lane line detection network training method, a driving control method, and a device. Background Art

[0002] In the field of autonomous driving technology, it is necessary to collect a variety of information to generate decision-making information and control the autonomous driving vehicle. Among them, lane line information is crucial to the generation of decision-making information.

[0003] In related technologies, lane line detection typically involves first detecting two-dimensional lane line information from an image. This information is then projected onto the road surface based on a predetermined homography matrix to determine the three-dimensional lane line information. The homography matrix is determined under the assumption that the road surface is horizontal. However, in real life, many roads are not horizontal, such as uphill, downhill, or uneven surfaces. This causes the three-dimensional lane line information projected from the horizontal road surface to deviate from the actual lane line information, leading to errors in the generated decision information. Therefore, determining lane line information more accurately has become a pressing issue. Summary of the Invention

[0004] The embodiments of the present disclosure at least provide a lane detection network training method, a driving control method, and a device.

[0005] In a first aspect, an embodiment of the present disclosure provides a lane detection network training method, comprising:

[0006] Acquire a sample image of a forward-looking angle and a supervisory mask image determined based on three-dimensional lane line data corresponding to the sample image; wherein the supervisory mask image is used to represent the overhead position information of the lane line in the sample image;

[0007] Inputting the sample image into a lane detection network to be trained to determine a top-view feature map of the sample image;

[0008] A predicted mask image of the sample image is determined based on the overhead feature map, and the lane line detection network to be trained is trained based on the predicted mask image and the supervised mask image.

[0009] In the above method, the overhead feature map of the sample image can be determined by the lane line detection network, and the supervision data of the lane line detection network is a supervision mask image determined based on the three-dimensional lane line data corresponding to the sample image. The three-dimensional lane line data contains the height information of the lane line, so the supervision mask image contains accurate overhead position information of the lane line, which provides more accurate supervision for the distal lane lines. Therefore, the trained lane line detection network has a strong extraction capability for the distal features of the lane lines in the image, and the trained network has higher lane line detection accuracy.

[0010] In one possible implementation, the method further includes determining the supervision mask image according to the following method:

[0011] Sampling three-dimensional lane line data corresponding to the two-dimensional lane lines in the sample image to determine three-dimensional lane key points;

[0012] projecting the three-dimensional lane key points onto the sample image based on extrinsic parameter data of an image acquisition device that acquires the sample image, and determining a projected image corresponding to the three-dimensional lane key points;

[0013] The projection image is converted to a top-down angle to obtain the supervision mask image.

[0014] Using this method, an accurate supervision mask image containing the overhead position information of the lane line can be obtained, so that a lane line detection network with higher accuracy can be trained based on the supervision mask image.

[0015] In one possible implementation, inputting the sample image into a lane detection network to be trained to determine a top-view feature map of the sample image includes:

[0016] Inputting the sample image into a lane detection network to be trained, and extracting a front view feature map of the sample image;

[0017] Based on the front view feature map and external parameter data of an image acquisition device that acquires the sample image, a top view feature map of the sample image is generated.

[0018] In one possible implementation, the lane line detection network includes a two-dimensional lane line detection network and a three-dimensional lane line detection network;

[0019] Inputting the sample image into a lane detection network to be trained and extracting a front view feature map of the sample image includes:

[0020] Inputting the sample image into the two-dimensional lane detection network to be trained, and extracting a front view feature map of the sample image;

[0021] The method further comprises:

[0022] Inputting the predicted mask image into the three-dimensional lane detection network to be trained to determine a first prediction result of the three-dimensional lane line corresponding to the sample image;

[0023] The training of the lane line detection network to be trained based on the predicted mask image and the supervised mask image includes:

[0024] The lane line detection network to be trained is trained based on the predicted mask image, the supervised mask image, the first prediction result and the three-dimensional lane line data.

[0025] Using this method, the two-dimensional lane line detection process and the three-dimensional lane line detection process can be separated. The three-dimensional lane line detection network and the two-dimensional lane line prediction network can both be lightweight networks. Compared with the end-to-end network, the internal structure of the network is simpler, the scale is smaller, and the calculation speed is faster.

[0026] In one possible implementation, the training of the lane detection network to be trained based on the predicted mask image, the supervisory mask image, the first prediction result, and the three-dimensional lane line data includes:

[0027] Determining a classification loss based on the predicted mask image and the supervised mask image, and training the two-dimensional lane detection network to be trained based on the classification loss; and

[0028] A three-dimensional reconstruction loss is determined based on the first prediction result and the three-dimensional lane line data, and the three-dimensional lane line detection network to be trained is trained based on the three-dimensional reconstruction loss.

[0029] Using this method, the two-dimensional lane line detection network and the three-dimensional lane line detection network can be trained simultaneously, which improves the training speed of the lane line detection network.

[0030] In one possible implementation, the first prediction result includes an offset of the lane line in the three-dimensional grid corresponding to the sample image, a lane line height, and visibility;

[0031] The determining of the three-dimensional reconstruction loss based on the lane line offset in the three-dimensional grid corresponding to the sample image, the lane line height, the visibility, and the three-dimensional lane line data includes:

[0032] Determining a grid prediction loss based on the first prediction result and the three-dimensional lane line data; and

[0033] A geometric prediction loss for characterizing the geometric relationship between the lane lines is determined based on the first prediction result and the prediction mask image.

[0034] The 3D reconstruction loss determined using this method takes into account the geometric relationship between lane lines. Therefore, the lane line information extracted by the lane line detection network trained based on the 3D reconstruction loss is more accurate.

[0035] In one possible implementation, the extrinsic parameter data of the image acquisition device that acquires the sample image is determined by the lane detection network;

[0036] The method further comprises:

[0037] Obtaining supervised extrinsic reference data corresponding to the sample image; wherein the supervised extrinsic reference data corresponding to the sample image is the true value extrinsic reference data of the image acquisition device when acquiring the sample image;

[0038] Determining an external reference prediction loss based on the external reference data and the supervised external reference data;

[0039] The training of the lane line detection network to be trained based on the predicted mask image and the supervised mask image includes:

[0040] The lane line detection network to be trained is trained based on the predicted mask image, the supervised mask image and the external parameter prediction loss.

[0041] Using this method, the lane line detection network can determine the external parameter data of the image acquisition device, so that no additional equipment is required to determine the external parameters of the image acquisition device, saving equipment resources, and training the lane line detection network based on the supervised external parameter data, thereby improving the lane line detection network's prediction accuracy for the external parameter data.

[0042] In one possible implementation, the front view feature map includes feature points corresponding to lane lines in the sample image;

[0043] The determining of the predicted mask image of the sample image based on the top-view feature map includes:

[0044] Decoding the overhead feature map to determine position information of overhead key points corresponding to the feature points;

[0045] According to the position information and the preset lane line width, the overhead key points of the lane line are connected to obtain the predicted mask image.

[0046] Using this method, the position information of the overhead key points can be accurately determined. By connecting the overhead key points of the lane lines with a preset lane line width, the lane lines in the predicted mask image can be presented more clearly, thereby improving the accuracy of the distant lane lines.

[0047] In a possible implementation manner, the method further includes:

[0048] Performing data enhancement processing on the three-dimensional lane line data corresponding to the sample image;

[0049] Determine a sample mask image of the three-dimensional lane line data after data enhancement processing;

[0050] Inputting the sample mask image into the three-dimensional lane detection network to be trained to determine a second prediction result;

[0051] The three-dimensional lane line detection network to be trained is trained based on the second prediction result and the three-dimensional lane line data after data enhancement processing.

[0052] Using this method, the amount of 3D lane line data can be greatly increased. Using richer 3D lane line data to train the 3D lane line detection network can improve the training effect of the 3D lane line detection network.

[0053] In a second aspect, an embodiment of the present disclosure provides a driving control method, including:

[0054] Obtain the image to be detected collected by the autonomous driving equipment;

[0055] A lane line detection network trained using the lane line detection network training method according to the first aspect or any possible implementation manner of the first aspect detects lane line information in the image to be detected;

[0056] The autonomous driving device is controlled to travel based on the lane line information.

[0057] The driving control method provided by the disclosed embodiments can obtain images to be detected from an autonomous driving device, determine lane information based on the images to be detected, and then control the driving of the autonomous driving device based on the lane information. This method enables the autonomous driving device to identify road conditions, thereby automatically controlling vehicle driving and other operations. Furthermore, because the determined lane information is more accurate, the autonomous driving device can operate more safely during driving.

[0058] In a third aspect, the present disclosure further provides a lane detection network training device, comprising:

[0059] A first acquisition module is configured to acquire a sample image of a forward-looking angle and a supervisory mask image determined based on three-dimensional lane line data corresponding to the sample image; wherein the supervisory mask image is used to represent overhead position information of the lane line in the sample image;

[0060] a determination module, configured to input the sample image into a lane detection network to be trained, and determine a top-view feature map of the sample image;

[0061] A training module is used to determine a predicted mask image of the sample image based on the overhead feature map, and to train the lane line detection network to be trained based on the predicted mask image and the supervised mask image.

[0062] In a possible implementation, the first acquisition module is further configured to determine the supervision mask image according to the following method:

[0063] Sampling three-dimensional lane line data corresponding to the two-dimensional lane lines in the sample image to determine three-dimensional lane key points;

[0064] projecting the three-dimensional lane key points onto the sample image based on extrinsic parameter data of an image acquisition device that acquires the sample image, and determining a projected image corresponding to the three-dimensional lane key points;

[0065] The projection image is converted to a top-down angle to obtain the supervision mask image.

[0066] In one possible implementation, when inputting the sample image into the lane detection network to be trained and determining the top-view feature map of the sample image, the determination module is configured to:

[0067] Inputting the sample image into a lane detection network to be trained, and extracting a front view feature map of the sample image;

[0068] Based on the front view feature map and external parameter data of an image acquisition device that acquires the sample image, a top view feature map of the sample image is generated.

[0069] In one possible implementation, the lane line detection network includes a two-dimensional lane line detection network and a three-dimensional lane line detection network;

[0070] The determination module is configured to, when inputting the sample image into the lane detection network to be trained and extracting the front view feature map of the sample image,:

[0071] Inputting the sample image into the two-dimensional lane detection network to be trained, and extracting a front view feature map of the sample image;

[0072] The determining module is further configured to:

[0073] Inputting the predicted mask image into the three-dimensional lane detection network to be trained to determine a first prediction result of the three-dimensional lane line corresponding to the sample image;

[0074] The training module is further configured to: when training the lane detection network to be trained based on the predicted mask image and the supervised mask image;

[0075] The lane line detection network to be trained is trained based on the predicted mask image, the supervised mask image, the first prediction result and the three-dimensional lane line data.

[0076] In one possible implementation, the training module, when training the lane detection network to be trained based on the predicted mask image, the supervisory mask image, the first prediction result, and the three-dimensional lane line data, is configured to:

[0077] Determining a classification loss based on the predicted mask image and the supervised mask image, and training the two-dimensional lane detection network to be trained based on the classification loss; and

[0078] A three-dimensional reconstruction loss is determined based on the first prediction result and the three-dimensional lane line data, and the three-dimensional lane line detection network to be trained is trained based on the three-dimensional reconstruction loss.

[0079] In one possible implementation, the first prediction result includes an offset of the lane line in the three-dimensional grid corresponding to the sample image, a lane line height, and visibility;

[0080] The training module is configured to, when determining a 3D reconstruction loss based on the first prediction result and the 3D lane line data,:

[0081] Determining a grid prediction loss based on the lane line's offset in the three-dimensional grid corresponding to the sample image, the lane line's height, the visibility, and the three-dimensional lane line data; and

[0082] A geometric prediction loss for characterizing the geometric relationship between the lane lines is determined based on the first prediction result and the prediction mask image.

[0083] In one possible implementation, the extrinsic parameter data of the image acquisition device that acquires the sample image is determined by the lane detection network;

[0084] The training module is further used to:

[0085] Obtaining supervised extrinsic reference data corresponding to the sample image; wherein the supervised extrinsic reference data corresponding to the sample image is the true value extrinsic reference data of the image acquisition device when acquiring the sample image;

[0086] Determining an external reference prediction loss based on the external reference data and the supervised external reference data;

[0087] The training module is configured to: when training the lane detection network to be trained based on the predicted mask image and the supervised mask image;

[0088] The lane line detection network to be trained is trained based on the predicted mask image, the supervised mask image and the external parameter prediction loss.

[0089] In one possible implementation, the front view feature map includes feature points corresponding to lane lines in the sample image;

[0090] The training module is configured to, when determining the predicted mask image of the sample image based on the top-view feature map,:

[0091] Decoding the overhead feature map to determine position information of overhead key points corresponding to the feature points;

[0092] According to the position information and the preset lane line width, the overhead key points of the lane line are connected to obtain the predicted mask image.

[0093] In a possible implementation, the training module is further configured to:

[0094] Performing data enhancement processing on the three-dimensional lane line data corresponding to the sample image;

[0095] Determine a sample mask image of the three-dimensional lane line data after data enhancement processing;

[0096] Inputting the sample mask image into the three-dimensional lane detection network to be trained to determine a second prediction result;

[0097] The three-dimensional lane line detection network to be trained is trained based on the second prediction result and the three-dimensional lane line data after data enhancement processing.

[0098] In a fourth aspect, an embodiment of the present disclosure further provides a driving control device, comprising:

[0099] The second acquisition module is used to acquire the image to be detected collected by the autonomous driving equipment;

[0100] A detection module, configured to detect lane line information in the image to be detected based on a lane line detection network trained by the lane line detection network training method according to the first aspect or any possible implementation manner of the first aspect;

[0101] A control module is used to control the driving of the automatic driving device based on the lane line information.

[0102] In a fifth aspect, an embodiment of the present disclosure further provides a computer device, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect, or the steps of the above-mentioned second aspect, or any possible implementation of the second aspect are executed.

[0103] In a sixth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the steps of the above-mentioned first aspect, or any possible implementation of the first aspect, or executes the above-mentioned second aspect, or any possible implementation of the second aspect.

[0104] For a description of the effects of the lane line detection network training device and the driving control device, please refer to the description of the lane line detection network training method and the driving control method, which will not be repeated here.

[0105] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure.

[0106] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0108] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without inventive effort.

[0109] Figure 1 A schematic diagram showing a projection effect provided by an embodiment of the present disclosure is shown;

[0110] Figure 2 A flowchart of a lane detection network training method provided by an embodiment of the present disclosure is shown;

[0111] Figure 3 A schematic diagram of lane lines in a three-dimensional grid provided by an embodiment of the present disclosure is shown;

[0112] Figure 4 A schematic diagram showing key points in three-dimensional lane line data provided by an embodiment of the present disclosure is shown;

[0113] Figure 5 A schematic diagram showing the overall workflow of a lane detection network provided by an embodiment of the present disclosure is shown;

[0114] Figure 6 A flow chart of a driving control method provided by an embodiment of the present disclosure is shown;

[0115] Figure 7 A schematic diagram of the architecture of a lane detection network training device provided by an embodiment of the present disclosure is shown;

[0116] Figure 8 A schematic diagram of the architecture of a driving control device provided by an embodiment of the present disclosure is shown;

[0117] Figure 9 A schematic structural diagram of a computer device provided by an embodiment of the present disclosure is shown;

[0118] Figure 10 A structural diagram of another computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0119] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0120] In related technologies, lane detection typically involves inputting a 2D image containing lane lines. A pre-trained neural network then predicts the lane line positions in the 2D image. This predicted lane line positions are then projected onto a 3D scene to determine the 3D lane line information. However, this projection assumes a horizontal road surface, leading to inaccurate 3D lane line information when the road surface is non-horizontal, such as uphill or downhill.

[0121] For example, Figure 1 As shown, Figure 1 The first and second pictures are the projection effects in uphill and downhill scenes respectively, when the road surface is assumed to be horizontal. Figure 1 The third and fourth pictures are real projection effects. It can be seen that the projection effect when the road surface is non-horizontal is different from the projection effect when the road surface is horizontal.

[0122] Based on the above research, the present disclosure provides a lane line detection network training method, driving control method and device, which can determine the overhead feature map of the sample image through the lane line detection network, and the supervision data of the lane line detection network is a supervision mask image determined based on the three-dimensional lane line data corresponding to the sample image. The three-dimensional lane line data contains the height information of the lane line, so the supervision mask image contains accurate overhead position information of the lane line, providing more accurate supervision for the far-end lane lines. Therefore, the trained lane line detection network has a strong extraction capability for the far-end features of the lane lines in the image, and the trained network has higher lane line detection accuracy.

[0123] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0124] The term "and / or" herein simply describes an association relationship, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, and the existence of B alone. In addition, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0125] To facilitate understanding of this embodiment, a detailed introduction to a lane detection network training method disclosed in an embodiment of the present disclosure is provided. The lane detection network training method provided in this embodiment is generally executed by a computer device with certain computing capabilities, such as a client, server, or other processing device. The client may include, for example, a personal computer or tablet computer. In some possible implementations, the lane detection network training method can be implemented by a processor invoking computer-readable instructions stored in a memory.

[0126] See also Figure 2 FIG. 2 is a flowchart of a lane detection network training method provided by an embodiment of the present disclosure, wherein the method includes steps 201 to 203, wherein:

[0127] Step 201: Acquire a sample image of a forward-looking angle and a supervisory mask image determined based on three-dimensional lane line data corresponding to the sample image; wherein the supervisory mask image is used to represent the overhead position information of the lane line in the sample image;

[0128] Step 202: Input the sample image into a lane detection network to be trained to determine a top-view feature map of the sample image;

[0129] Step 203: Determine a predicted mask image of the sample image based on the top-view feature image, and train the lane detection network to be trained based on the predicted mask image and the supervised mask image.

[0130] The following are detailed instructions for the above steps:

[0131] For step 201,

[0132] Among them, the sample image of the forward-looking angle is an image of the lane line containing the forward-looking angle. Exemplarily, the sample image is collected by an image acquisition device on the vehicle, and the image acquisition device can be deployed at a position such as the top of the vehicle.

[0133] The supervisory mask image is used to represent the overhead position information of the lane line in the sample image. Exemplarily, the supervisory mask image can be a binary image, and the pixel value of the pixel point corresponding to the lane line in the supervisory mask image can be 1, and the pixel value of the pixel point corresponding to the non-lane line can be 0.

[0134] In one possible implementation, when determining the supervisory mask image, the three-dimensional lane line data corresponding to the two-dimensional lane lines in the sample image can be sampled first to determine the three-dimensional lane key points. Then, based on the external parameter data of the image acquisition device that acquires the sample image, the three-dimensional lane key points can be projected onto the sample image to determine the projection image corresponding to the three-dimensional lane key points. Finally, the projection image can be converted to a bird's-eye view to obtain the supervisory mask image.

[0135] Specifically, the three-dimensional lane line data may include the position information of the lane line in the three-dimensional grid, which may be exemplified as follows: Figure 3 As shown, the 3D lane data can be acquired by scanning the lane lines using a LiDAR. In the 3D lane data, the lane lines are continuous. Therefore, the lane lines in the 3D lane data can be sampled according to a preset sampling algorithm to obtain the 3D lane key points, which are then projected.

[0136] The external parameter data of the image acquisition device may be the external parameter data when the sample image is acquired, and the external parameter data is used to ensure the relative position relationship between the position information of the image acquisition device and the laser radar.

[0137] After projecting the three-dimensional lane keypoints onto the sample image, the two-dimensional lane keypoints corresponding to the three-dimensional lane keypoints on the sample image can be obtained. The projected image is an image containing the two-dimensional lane keypoints. Exemplarily, when determining the two-dimensional lane keypoints corresponding to the projected three-dimensional lane keypoints, the following formula can be used for calculation:

[0138]

[0139] Among them, x 2D represents the x-axis coordinate of the two-dimensional lane key point in the two-dimensional coordinate system of the sample image, 2D represents the y-axis coordinate of the two-dimensional lane key point in the two-dimensional coordinate system of the sample image, h cam represents the height of the image acquisition device in the three-dimensional coordinate system, x 3D Indicates the x-axis coordinate of the key point of the three-dimensional lane in the three-dimensional coordinate system, y 3D represents the y-axis coordinate of the 3D lane key point in the 3D coordinate system, and z represents the z-axis coordinate of the 3D lane key point in the 3D coordinate system.

[0140] Then, based on a preset angle conversion algorithm, the two-dimensional overhead key points corresponding to the two-dimensional lane key points can be determined, and the two-dimensional overhead key points can be connected according to the preset lane line width to generate the supervision mask image, wherein the preset algorithm can exemplarily be a homography transformation (Homography).

[0141] Using this method, an accurate supervision mask image containing the overhead position information of the lane line can be obtained, so that a lane line detection network with higher accuracy can be trained based on the supervision mask image.

[0142] For step 202,

[0143] In one possible implementation, when inputting the sample image into the lane line detection network to be trained and determining the top-view feature map of the sample image, the sample image can be first input into the lane line detection network to be trained to extract the front-view feature map of the sample image; then, based on the front-view feature map and the external parameter data of the image acquisition device that acquires the sample image, the top-view feature map of the sample image is generated.

[0144] Exemplarily, the lane detection network may perform feature extraction on the sample image using an image encoder in the lane detection network. Exemplarily, the image encoder may be an efficient residual decomposition convolutional neural network (ERFNet) for real-time semantic segmentation.

[0145] Here, to reduce the computational complexity of the lane detection network and speed up its operation, the image encoder can extract feature information of all objects in the sample image when performing feature extraction on the sample image, and then perform feature screening. That is, the front-view feature map can only include feature information of lane lines, and exclude feature information of other objects in the sample image (such as trees, streetlights, walls, etc.). Therefore, the image encoder can be compact, saving storage space and improving work efficiency.

[0146] In a possible embodiment, the lane line detection network may include a two-dimensional lane line detection network and a three-dimensional lane line detection network. When the sample image is input into the lane line detection network to be trained and the forward-view feature map of the sample image is extracted, the sample image may be input into the two-dimensional lane line detection network to be trained and the forward-view feature map of the sample image is extracted.

[0147] That is, the image encoder in the two-dimensional lane line monitoring network extracts the forward-looking feature map of the sample image. The method for extracting the forward-looking feature map is the same as the method for extracting the forward-looking feature map by the above-mentioned lane line detection network, which will not be repeated here. The specific implementation method of the three-dimensional lane line detection network will be described in detail later.

[0148] Here, the extrinsic parameter data may include the height information of the image acquisition device and the pitch angle of the image acquisition device. The extrinsic parameter data may be directly obtained through an extrinsic parameter estimation device, such as an inertial measurement unit (IMU).

[0149] Alternatively, in a possible embodiment, the extrinsic parameter data of the image acquisition device of the sample image is determined by the extrinsic parameter prediction module of the lane line detection network. Specifically, when the lane line detection network includes an extrinsic parameter prediction module, the training process of the extrinsic parameter prediction module will be described in detail below.

[0150] Step 203:

[0151] In one possible embodiment, the forward-view feature map includes feature points corresponding to the lane lines in the sample image. Specifically, the following method can be used to determine the predicted mask image of the sample image: the overhead feature map can be decoded to determine the position information of the overhead key points corresponding to the feature points, and then the overhead key points of the lane lines are connected according to the position information and the preset lane line width to obtain the predicted mask image.

[0152] Among them, the image decoder used in decoding corresponds to the encoder used in encoding. After decoding, the position information of the overhead key point corresponding to the feature point in the overhead feature map can be determined. The overhead key point can be a pixel point. The preset lane line width can be, for example, a pixel length, such as 2 pixels wide. The lane line is composed of multiple lane curves. When connecting the lane lines, the predicted overhead key points located on the same curve can be connected according to the preset lane line width.

[0153] Compared to directly using the front view as supervision information, the method provided by this disclosure uses a supervision mask image directly determined based on three-dimensional lane line data. Therefore, the lane line position information in the supervision mask image is more accurate. However, if the front view is used directly as supervision information, the position information of the front lane lines in the supervision information is more accurate, while the position information of the far lane lines is relatively less accurate. Therefore, if the front view is used as supervision information, the feature extraction accuracy of the far lane lines is lower. Accordingly, the method provided by this disclosure determines more accurate lane line position information in the predicted mask image.

[0154] Furthermore, by using a preset lane line width to connect the top-down key points of the lane line, the lane lines in the predicted mask image can be presented more clearly, thereby improving the accuracy of the distant lane lines.

[0155] In one possible implementation, when the lane line detection network to be trained is trained based on the predicted mask image and the supervised mask image, the classification loss can be calculated based on the feature values of the overhead key points in the predicted mask image and the feature values of the key points in the supervised mask image, and the lane line detection network can be adjusted based on the classification loss.

[0156] Using this method, the position information of the overhead key points can be accurately determined. By connecting the overhead key points of the lane lines with a preset lane line width, the lane lines in the predicted mask image can be presented more clearly, thereby improving the accuracy of the distant lane lines.

[0157] In another possible implementation, when the lane detection network predicts the extrinsic parameter data, when training the lane detection network to be trained based on the predicted mask image and the supervised mask image, the supervised extrinsic parameter data corresponding to the sample image can be first obtained, and the extrinsic parameter prediction loss can be determined based on the extrinsic parameter data and the supervised extrinsic parameter data. The lane detection network to be trained can then be trained based on the predicted mask image, the supervised mask image, and the extrinsic parameter prediction loss. The supervised extrinsic parameter data corresponding to the sample image is the true extrinsic parameter data of the image acquisition device when the sample image is acquired.

[0158] Specifically, the predicted extrinsic parameter data of the sample image can be first determined based on the extrinsic parameter prediction module of the lane detection network. Then, the supervised extrinsic parameter data of the image acquisition device corresponding to the sample image can be obtained. The supervised extrinsic parameter data is actual data of the image acquisition device obtained through measurement. Exemplary supervised extrinsic parameter data can be obtained through manual measurement, lidar, or IMU measurement. Finally, the extrinsic parameter prediction loss is calculated based on the supervised extrinsic parameter data and the predicted extrinsic parameter data, and the extrinsic parameter prediction module is adjusted based on the extrinsic parameter prediction loss.

[0159] Exemplarily, the external parameter prediction loss can be calculated by the following formula:

[0160]

[0161] Among them, L cam represents the external parameter prediction loss, θ represents the pitch angle predicted by the lane detection network, represents the actual pitch angle of the image acquisition device of the sample image, hcam represents the height information of the image acquisition device predicted by the lane detection network, Indicates the actual height information of the image acquisition device.

[0162] When training the lane line prediction network, the extrinsic parameter prediction module can be adjusted based on the extrinsic parameter prediction loss, and other related modules in the lane line detection network can be adjusted based on the image prediction loss calculated from the predicted mask image and the supervised mask image. Alternatively, the extrinsic parameter prediction loss and the image prediction loss can be weighted and summed according to their respective weights to obtain a classification loss, and the lane line detection network can be adjusted based on the classification loss. Exemplarily, the calculation formula of the classification loss is as follows:

[0163] L fea =L seg +λ cam L cam

[0164] Among them, L fea represents the classification loss, L seg represents the image prediction loss, λ cam Represents the weight of the external parameter prediction loss, L cam Represents the external parameter prediction loss.

[0165] Using this method, the lane line detection network can determine the external parameter data of the image acquisition device, so that no additional equipment is required to determine the external parameters of the image acquisition device, saving equipment resources, and training the lane line detection network based on the supervised external parameter data, thereby improving the lane line detection network's prediction accuracy for the external parameter data.

[0166] In a possible implementation, when the lane line detection network includes a two-dimensional lane line detection network and a three-dimensional lane line detection network, the predicted mask image can also be input into the three-dimensional lane line detection network to be trained to determine a first prediction result of the three-dimensional lane line corresponding to the sample image, and then the lane line detection network to be trained is trained based on the predicted mask image, the supervision mask image, the first prediction result and the three-dimensional lane line data.

[0167] Here, the three-dimensional lane line detection network is used to determine the three-dimensional data corresponding to the lane line in the sample image based on the predicted mask image.

[0168] Using this method, the two-dimensional lane line detection process and the three-dimensional lane line detection process can be separated. The three-dimensional lane line detection network and the two-dimensional lane line prediction network can both be lightweight networks. Compared with the end-to-end network, the internal structure of the network is simpler, the scale is smaller, and the calculation speed is faster.

[0169] In a possible implementation, when training the lane line detection network to be trained based on the predicted mask image, the supervised mask image, the first prediction result, and the three-dimensional lane line data, in addition to determining the classification loss based on the predicted mask image and the supervised mask image as described above, and training the two-dimensional lane line detection network to be trained based on the classification loss, the three-dimensional reconstruction loss can also be determined based on the first prediction result and the three-dimensional lane line data, and training the three-dimensional lane line detection network to be trained based on the three-dimensional reconstruction loss.

[0170] Using this method, the two-dimensional lane line detection network and the three-dimensional lane line detection network can be trained simultaneously, which improves the training speed of the lane line detection network.

[0171] In one possible embodiment, the first prediction result may include the offset of the lane line in the three-dimensional grid corresponding to the sample image, the lane line height and the visibility; when determining the three-dimensional reconstruction loss based on the first prediction result and the three-dimensional lane line data, the grid prediction loss can be determined based on the offset of the lane line in the three-dimensional grid corresponding to the sample image, the lane line height, the visibility and the three-dimensional lane line data; and, based on the first prediction result and the predicted mask image, a geometric prediction loss for characterizing the geometric relationship between the lane lines is determined.

[0172] Here, the three-dimensional grid corresponding to the sample image may refer to the three-dimensional grid of the scene corresponding to the sample image. Exemplarily, for the three-dimensional prediction key points in any column of the grid, the grid prediction loss can be calculated by the following formula:

[0173]

[0174] Among them, L anchor Represents the grid prediction loss, N represents the number of three-dimensional prediction key points in each column, represents the actual possibility, p i represents the possibility of the output of the three-dimensional lane detection network, represents the actual visibility, v i represents the visibility output by the three-dimensional lane detection network, Indicates the actual height, z irepresents the height of the three-dimensional lane detection network output, Indicates the actual offset, x i Represents the offset of the 3D lane detection network output.

[0175] In practical applications, the distance between different lane lines should change smoothly. Therefore, based on this, the geometric relationship constraints between lane lines are added to the loss value calculation formula.

[0176] For any three-dimensional prediction key point on a curve, the geometric prediction loss can be calculated using the following formula:

[0177]

[0178] Among them, L geo Represents the geometric prediction loss, d represents the dimension, N represents the number of key points, when d is 2, N represents the number of key points in the predicted mask image, when d is 3, N represents the number of key points in the three-dimensional lane line data, v i represents the visibility output by the three-dimensional lane detection network, p represents the probability output by the three-dimensional lane detection network, represents the closest distance between the i-th key point and the adjacent left lane line in the predicted mask image, Indicates the shortest distance between the i-1th key point on the same lane line as the i-th key point and the adjacent left lane line, It represents the shortest distance between the i+1th key point on the same lane line as the i-th key point and the adjacent left lane line.

[0179] When d is 2, for the above formula (That is, D in the following formula 2D ) can be calculated using the following formula:

[0180]

[0181]

[0182] in, represents the horizontal coordinate of the i-th key point, represents the horizontal coordinate of the key point on the adjacent left lane line closest to the i-th key point, represents the ordinate of the i-th key point, represents the ordinate of the key point on the adjacent left lane line closest to the i-th key point, h cam represents the height of the image acquisition device, represents the height of the key point in the three-dimensional lane line data corresponding to the i-th key point, represents the height of the key point on the adjacent left lane line closest to the i-th key point and the corresponding key point in the three-dimensional lane line data, Represents the estimated height of the three-dimensional key point corresponding to the i-th key point.

[0183] When d is 3, for the above formula The three-dimensional coordinates of the key points in the three-dimensional lane line data can be directly used to calculate the above-mentioned closest distance.

[0184] For example, Figure 4 As shown, point C represents the image acquisition device, represents the i-1th key point, represents the i-th key point, represents the key point on the adjacent left lane line closest to the i-1th key point, represents the key point on the adjacent left lane line closest to the i-th key point, Indicates the shortest distance between the i-1th key point and the adjacent left lane line, Represents the shortest distance between the i-th key point and the adjacent left lane line.

[0185] After determining the grid prediction loss and the geometric prediction loss, the grid prediction loss and the geometric prediction loss may be weighted and summed according to their respective preset weights to obtain a 3D lane detection loss value, and the 3D lane detection network may be adjusted based on the 3D lane detection loss value. Exemplarily, the 3D lane detection loss value may be calculated using the following formula:

[0186] L rec =L anchor +λ geo L geo

[0187] Among them, L rec Represents the three-dimensional lane line detection loss value, L anchor represents the grid prediction loss, L geo represents the geometric prediction loss, λ geo Represents the weight corresponding to the geometric prediction loss.

[0188] The 3D reconstruction loss determined using this method takes into account the geometric relationship between lane lines. Therefore, the lane line information extracted by the lane line detection network trained based on the 3D reconstruction loss is more accurate.

[0189] In one possible implementation, since the three-dimensional lane line data is relatively small, in order to avoid the impact of the data volume on the network accuracy of the three-dimensional lane line detection network, the three-dimensional lane line data corresponding to the sample image can be data enhanced, and a sample mask image of the three-dimensional lane line data after data enhancement processing can be determined. The sample mask image is then input into the three-dimensional lane line detection network to be trained, and a second prediction result is determined. The three-dimensional lane line detection network to be trained is trained based on the second prediction result and the three-dimensional lane line data after data enhancement processing.

[0190] Exemplarily, when performing data enhancement processing on the three-dimensional lane line data, data after rotating the three-dimensional key points corresponding to the three-dimensional lane line data may be obtained.

[0191] Using this method, the amount of 3D lane line data can be greatly increased. Using richer 3D lane line data to train the 3D lane line detection network can improve the training effect of the 3D lane line detection network.

[0192] The lane line detection network may include a two-dimensional lane line detection network and a three-dimensional lane line detection network. Figure 5 As shown in the figure, the overall workflow of the lane line detection network is introduced:

[0193] After inputting the forward-looking image to be detected into the 2D lane detection network, the 2D lane detection network can extract the forward-looking feature map of the image to be detected and, based on the forward-looking feature map and the extrinsic parameter data of the image acquisition device for the image to be detected, generate a top-down feature map of the image to be detected. Finally, based on the top-down feature map, a predicted mask image for the image to be detected is determined. Then, after determining the predicted mask image for the image to be detected, the predicted mask image for the image to be detected is input into the 3D lane detection network to obtain a predicted result for the 3D lane line corresponding to the image to be detected.

[0194] The lane line detection network training method provided by the embodiment of the present disclosure has the following characteristics: the supervision data of the lane line detection network is a supervision mask image determined based on the three-dimensional lane line data corresponding to the sample image. The three-dimensional lane line data contains the height information of the lane line, so the supervision mask image contains accurate bird's-eye view position information of the lane line, providing more accurate supervision for the far-end lane lines. Therefore, the trained lane line detection network has a strong extraction capability for the far-end features of the lane lines in the image, and the trained network has higher lane line detection accuracy.

[0195] Based on the same inventive concept, the present disclosure also provides a driving control method, which is applied to a user terminal or a server. The user terminal may be, for example, an autonomous driving device, a personal computer, etc. Figure 6 FIG. 6 is a flow chart of a driving control method provided by an embodiment of the present disclosure, wherein the method includes steps 601 to 603, wherein:

[0196] Step 601: Acquire an image to be detected collected by the autonomous driving device;

[0197] Step 602: Detect lane line information in the image to be detected using a lane line detection network trained by the lane line detection network training method described in the above embodiment.

[0198] Step 603: Control the autonomous driving device to drive based on the lane line information.

[0199] For steps 601 to 603,

[0200] The image to be detected is used to represent a front-view image of the lane line, and the lane line information may include position information of the lane line (including height information of the lane line).

[0201] In a possible implementation, when controlling the driving of the autonomous driving device based on the lane line information, the road surface condition of the lane line in the image to be detected can be first determined based on the lane line information, and then the driving of the autonomous driving device can be controlled based on the road surface condition.

[0202] Among them, the road surface conditions can exemplarily include uphill, downhill, flat, and steep. For example, when the height information in the lane line information gradually increases, it can be judged that the road surface condition is uphill. When controlling the automatic driving device to drive, the automatic driving device can be controlled to accelerate, decelerate, turn, etc. For example, when it is detected that the road surface condition is downhill, the automatic driving device can be controlled to decelerate.

[0203] In one possible implementation, prompt information may be determined and displayed based on the lane line information. Specifically, the road surface condition of the lane line in the image to be detected may be determined based on the lane line information, and then prompt information matching the road surface condition may be determined, such as "There is a steep slope ahead, please hold on tight."

[0204] In one possible implementation, the autonomous driving device can generate a three-dimensional lane model of the lane lines in the image to be detected based on the lane line information, and then display the three-dimensional lane model. For example, the three-dimensional lane model can be displayed on a display device of the autonomous driving device, or the three-dimensional lane model can be sent to a server. Using this method, the autonomous driving device can transmit the detected road conditions to the user, saving manpower.

[0205] The driving control method provided by the disclosed embodiments can obtain images to be detected from an autonomous driving device, determine lane information based on the images to be detected, and then control the driving of the autonomous driving device based on the lane information. This method enables the autonomous driving device to identify road conditions, thereby automatically controlling vehicle driving and other operations. Furthermore, because the determined lane information is more accurate, the autonomous driving device can operate more safely during driving.

[0206] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0207] Based on the same inventive concept, the embodiment of the present disclosure also provides a lane line detection network training device corresponding to the lane line detection network training method. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned lane line detection network training method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0208] Reference Figure 7 , which is a schematic diagram of the architecture of a lane detection network training device provided by an embodiment of the present disclosure, the device includes: a first acquisition module 701, a determination module 702, and a training module 703; wherein,

[0209] A first acquisition module 701 is configured to acquire a sample image of a forward-looking angle and a supervisory mask image determined based on three-dimensional lane line data corresponding to the sample image; wherein the supervisory mask image is used to represent overhead position information of the lane line in the sample image;

[0210] A determination module 702 is configured to input the sample image into a lane detection network to be trained and determine a top-view feature map of the sample image;

[0211] The training module 703 is configured to determine a predicted mask image of the sample image based on the overhead feature map, and train the lane detection network to be trained based on the predicted mask image and the supervisory mask image.

[0212] In a possible implementation, the first acquisition module 701 is further configured to determine the supervision mask image according to the following method:

[0213] Sampling three-dimensional lane line data corresponding to the two-dimensional lane lines in the sample image to determine three-dimensional lane key points;

[0214] projecting the three-dimensional lane key points onto the sample image based on extrinsic parameter data of an image acquisition device that acquires the sample image, and determining a projected image corresponding to the three-dimensional lane key points;

[0215] The projection image is converted to a top-down angle to obtain the supervision mask image.

[0216] In one possible implementation, when inputting the sample image into the lane detection network to be trained and determining the top-view feature map of the sample image, the determination module 702 is configured to:

[0217] Inputting the sample image into a lane detection network to be trained, and extracting a front view feature map of the sample image;

[0218] Based on the front view feature map and external parameter data of an image acquisition device that acquires the sample image, a top view feature map of the sample image is generated.

[0219] In one possible implementation, the lane line detection network includes a two-dimensional lane line detection network and a three-dimensional lane line detection network;

[0220] The determination module 702 is configured to: when inputting the sample image into the lane detection network to be trained and extracting the front view feature map of the sample image;

[0221] Inputting the sample image into the two-dimensional lane detection network to be trained, and extracting a front view feature map of the sample image;

[0222] The determining module 703 is further configured to:

[0223] Inputting the predicted mask image into the three-dimensional lane detection network to be trained to determine a first prediction result of the three-dimensional lane line corresponding to the sample image;

[0224] The training module 703 is further configured to: when training the lane detection network to be trained based on the predicted mask image and the supervised mask image;

[0225] The lane line detection network to be trained is trained based on the predicted mask image, the supervised mask image, the first prediction result and the three-dimensional lane line data.

[0226] In one possible implementation, the training module 703, when training the lane detection network to be trained based on the predicted mask image, the supervisory mask image, the first prediction result, and the three-dimensional lane data, is configured to:

[0227] Determining a classification loss based on the predicted mask image and the supervised mask image, and training the two-dimensional lane detection network to be trained based on the classification loss; and

[0228] A three-dimensional reconstruction loss is determined based on the first prediction result and the three-dimensional lane line data, and the three-dimensional lane line detection network to be trained is trained based on the three-dimensional reconstruction loss.

[0229] In one possible implementation, the first prediction result includes an offset of the lane line in the three-dimensional grid corresponding to the sample image, a lane line height, and visibility;

[0230] The training module 703 is configured to:

[0231] Determining a grid prediction loss based on the lane line's offset in the three-dimensional grid corresponding to the sample image, the lane line's height, the visibility, and the three-dimensional lane line data; and

[0232] A geometric prediction loss for characterizing the geometric relationship between the lane lines is determined based on the first prediction result and the prediction mask image.

[0233] In one possible implementation, the extrinsic parameter data of the image acquisition device that acquires the sample image is determined by the lane detection network;

[0234] The training module 703 is further configured to:

[0235] Obtaining supervised extrinsic reference data corresponding to the sample image; wherein the supervised extrinsic reference data corresponding to the sample image is the true value extrinsic reference data of the image acquisition device when acquiring the sample image;

[0236] Determining an external reference prediction loss based on the external reference data and the supervised external reference data;

[0237] The training module 703 is configured to:

[0238] The lane line detection network to be trained is trained based on the predicted mask image, the supervised mask image and the external parameter prediction loss.

[0239] In one possible implementation, the front view feature map includes feature points corresponding to lane lines in the sample image;

[0240] The training module 703 is configured to: when determining the predicted mask image of the sample image based on the top-view feature map;

[0241] Decoding the overhead feature map to determine position information of overhead key points corresponding to the feature points;

[0242] According to the position information and the preset lane line width, the overhead key points of the lane line are connected to obtain the predicted mask image.

[0243] In a possible implementation, the training module 703 is further configured to:

[0244] Performing data enhancement processing on the three-dimensional lane line data corresponding to the sample image;

[0245] Determine a sample mask image of the three-dimensional lane line data after data enhancement processing;

[0246] Inputting the sample mask image into the three-dimensional lane detection network to be trained to determine a second prediction result;

[0247] The three-dimensional lane line detection network to be trained is trained based on the second prediction result and the three-dimensional lane line data after data enhancement processing.

[0248] Corresponding to Figure 6 The driving control method shown in the embodiment of the present disclosure provides a driving control device, referring to Figure 8 FIG. 8 is a schematic diagram of the architecture of a driving control device provided by an embodiment of the present disclosure, wherein the device includes: a second acquisition module 801, a detection module 802, and a control module 803; wherein,

[0249] The second acquisition module 801 is used to acquire the image to be detected collected by the autonomous driving device;

[0250] A detection module 802 is configured to detect lane line information in the image to be detected based on a lane line detection network trained by the lane line detection network training method described in the above embodiment;

[0251] The control module 803 is used to control the driving of the automatic driving device based on the lane line information.

[0252] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0253] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Figure 9900 , which includes a processor 901 , a memory 902 , and a bus 903 . The memory 902 is used to store execution instructions and includes a memory 9021 and an external memory 9022 . The memory 9021 , also referred to as internal memory, is used to temporarily store computational data in the processor 901 and data exchanged with an external memory 9022 , such as a hard disk. The processor 901 exchanges data with the external memory 9022 via the memory 9021 . When the computer device 900 is running, the processor 901 communicates with the memory 902 via the bus 903 , enabling the processor 901 to execute the following instructions:

[0254] Acquire a sample image of a forward-looking angle and a supervisory mask image determined based on three-dimensional lane line data corresponding to the sample image; wherein the supervisory mask image is used to represent the overhead position information of the lane line in the sample image;

[0255] Inputting the sample image into a lane detection network to be trained to determine a top-view feature map of the sample image;

[0256] A predicted mask image of the sample image is determined based on the overhead feature map, and the lane line detection network to be trained is trained based on the predicted mask image and the supervised mask image.

[0257] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Figure 10 1 is a schematic diagram of the structure of a computer device 1000 provided in an embodiment of the present disclosure, including a processor 1001, a memory 1002, and a bus 1003. The memory 1002 is used to store execution instructions and includes a memory 10021 and an external memory 10022. The memory 10021 is also referred to as internal memory and is used to temporarily store operation data in the processor 1001 and data exchanged with an external memory 10022 such as a hard disk. The processor 1001 exchanges data with the external memory 10022 through the memory 10021. When the computer device 1000 is running, the processor 1001 communicates with the memory 1002 via the bus 1003, so that the processor 1001 executes the following instructions:

[0258] Obtain the image to be detected collected by the autonomous driving equipment;

[0259] Detecting lane line information in the image to be detected using a lane line detection network trained by the lane line detection network training method described in the above embodiment;

[0260] The autonomous driving device is controlled to travel based on the lane line information.

[0261] The present disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, executes the lane detection network training method and driving control method described in the aforementioned method embodiments. The storage medium may be volatile or non-volatile, computer-readable.

[0262] The embodiments of the present disclosure also provide a computer program product, which carries program code. The instructions included in the program code can be used to execute the steps of the lane line detection network training method and the driving control method described in the above method embodiments. For details, please refer to the above method embodiments and will not be repeated here.

[0263] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0264] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0265] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0266] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0267] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0268] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The scope of protection of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present disclosure, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure shall be subject to the scope of protection of the claims.

[0269] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A lane detection network training method, characterized in that: The lane line detection network includes a two-dimensional lane line detection network and a three-dimensional lane line detection network, and the method includes: Acquire a sample image of a forward-looking angle and a supervisory mask image determined based on three-dimensional lane line data corresponding to the sample image; wherein the supervisory mask image is used to represent the overhead position information of the lane line in the sample image; Inputting the sample image into the two-dimensional lane detection network to be trained, and extracting a front view feature map of the sample image; generating a top-view feature map for determining the sample image based on the front-view feature map and external parameter data of an image acquisition device that acquires the sample image; Determining a predicted mask image of the sample image based on the overhead feature map, and training the lane detection network to be trained based on the predicted mask image and the supervised mask image; The method further comprises: Inputting the predicted mask image into the three-dimensional lane detection network to be trained to determine a first prediction result of the three-dimensional lane line corresponding to the sample image; The training of the lane line detection network to be trained based on the predicted mask image and the supervised mask image includes: The lane line detection network to be trained is trained based on the predicted mask image, the supervised mask image, the first prediction result, and the three-dimensional lane line data.

2. The method according to claim 1, characterized in that The method further comprises determining the supervisory mask image according to the following method: Sampling three-dimensional lane line data corresponding to the two-dimensional lane lines in the sample image to determine three-dimensional lane key points; projecting the three-dimensional lane key points onto the sample image based on extrinsic parameter data of an image acquisition device that acquires the sample image, and determining a projected image corresponding to the three-dimensional lane key points; The projection image is converted to a top-down angle to obtain the supervision mask image.

3. The method according to claim 1, characterized in that The training of the lane line detection network to be trained based on the predicted mask image, the supervised mask image, the first prediction result, and the three-dimensional lane line data includes: Determining a classification loss based on the predicted mask image and the supervised mask image, and training the two-dimensional lane detection network to be trained based on the classification loss; and A three-dimensional reconstruction loss is determined based on the first prediction result and the three-dimensional lane line data, and the three-dimensional lane line detection network to be trained is trained based on the three-dimensional reconstruction loss.

4. The method according to claim 3, characterized in that The first prediction result includes an offset of the lane line in the three-dimensional grid corresponding to the sample image, a lane line height, and visibility; The determining of the three-dimensional reconstruction loss based on the first prediction result and the three-dimensional lane line data includes: Determining a grid prediction loss based on the lane line's offset in the three-dimensional grid corresponding to the sample image, the lane line's height, the visibility, and the three-dimensional lane line data; and A geometric prediction loss for characterizing the geometric relationship between the lane lines is determined based on the first prediction result and the prediction mask image.

5. The method according to any one of claims 2 to 4, characterized in that: The extrinsic parameter data of the image acquisition device that acquires the sample image is determined by the lane detection network; The method further comprises: Obtaining supervised extrinsic reference data corresponding to the sample image; wherein the supervised extrinsic reference data corresponding to the sample image is the true value extrinsic reference data of the image acquisition device when acquiring the sample image; Determining an external reference prediction loss based on the external reference data and the supervised external reference data; The training of the lane line detection network to be trained based on the predicted mask image and the supervised mask image includes: The lane line detection network to be trained is trained based on the predicted mask image, the supervised mask image and the external parameter prediction loss.

6. The method according to any one of claims 1 to 4, characterized in that: The front view feature map includes feature points corresponding to lane lines in the sample image; The determining of the predicted mask image of the sample image based on the top-view feature map includes: Decoding the overhead feature map to determine position information of overhead key points corresponding to the feature points; According to the position information and the preset lane line width, the overhead key points of the lane line are connected to obtain the predicted mask image.

7. The method according to claim 1 or 3, characterized in that The method further comprises: Performing data enhancement processing on the three-dimensional lane line data corresponding to the sample image; Determine a sample mask image of the three-dimensional lane line data after data enhancement processing; Inputting the sample mask image into the three-dimensional lane detection network to be trained to determine a second prediction result; The three-dimensional lane line detection network to be trained is trained based on the second prediction result and the three-dimensional lane line data after data enhancement processing.

8. A driving control method, characterized in that: include: Obtain the image to be detected collected by the autonomous driving equipment; Detecting lane line information in the image to be detected based on a lane line detection network trained by the lane line detection network training method according to any one of claims 1 to 7; The autonomous driving device is controlled to travel based on the lane line information.

9. A lane detection network training device, characterized in that: The lane line detection network includes a two-dimensional lane line detection network and a three-dimensional lane line detection network, and the device includes: A first acquisition module is configured to acquire a sample image of a forward-looking angle and a supervisory mask image determined based on three-dimensional lane line data corresponding to the sample image; wherein the supervisory mask image is used to represent overhead position information of the lane line in the sample image; a determination module, configured to input the sample image into the two-dimensional lane detection network to be trained, extract a front view feature map of the sample image, and generate a top view feature map of the sample image based on the front view feature map and extrinsic parameter data of an image acquisition device that acquires the sample image; a training module, configured to determine a predicted mask image of the sample image based on the overhead feature map, and train the lane detection network to be trained based on the predicted mask image and the supervisory mask image; The determining module is further configured to: Inputting the predicted mask image into the three-dimensional lane detection network to be trained to determine a first prediction result of the three-dimensional lane line corresponding to the sample image; The training module is configured to: when training the lane detection network to be trained based on the predicted mask image and the supervised mask image; The lane line detection network to be trained is trained based on the predicted mask image, the supervised mask image, the first prediction result, and the three-dimensional lane line data.

10. A driving control device, characterized in that: include: The second acquisition module is used to acquire the image to be detected collected by the autonomous driving equipment; A detection module, configured to detect lane line information in the image to be detected based on a lane line detection network trained by the lane line detection network training method according to any one of claims 1 to 7; A control module is used to control the driving of the automatic driving device based on the lane line information.

11. A computer device, characterized in that: include: A processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the memory communicate via the bus. When the machine-readable instructions are executed by the processor, the steps of the lane detection network training method according to any one of claims 1 to 7 are performed, or the steps of the driving control method according to claim 8 are performed.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the lane line detection network training method according to any one of claims 1 to 7, or executes the steps of the driving control method according to claim 8.

Images