Method and system for lane detection

Abstract

The invention is a method for lane detection, comprising the steps of - generating, by means of an image processing module (12) trained by machine learning, based on an image pair of a first and a second image (10a, 10b), correspondence mapping data determining a correspondence between a first lane boundary group of the first image and a second lane boundary group of the second image as output data, - generating, based on the output data, a data block of an image space detection pair, and - generating, by means of triangulation, a 3D lane detection data block based on first and second data block portions of the data block of the image space detection pair corresponding to the respective first and second members of the image space detection pair. The invention is also a system for lane detection, as well as a method and a system for training an image processing module.

Description

Technical Field

[0001] This invention relates to a method and system for lane detection using an image processing module trained by machine learning. The invention also relates to a method and system for training the image processing module. Background Technology

[0002] Perception of the three-dimensional environment of road vehicles is crucial in autonomous driving applications. Surface estimation and detection of lane markings or boundaries are necessary for lane keeping and lane-changing maneuvers, as well as for locating lane-level obstacles. Lane geometry and obstacle information are key inputs to the equipment responsible for controlling the vehicle. Therefore, several methods and attempts have been made to address the lane detection problem.

[0003] Existing lane detection methods have utilized traditional computer vision techniques, which search for lane markings based on object width, orientation, alignment, and other criteria. For example, such lane detection methods are disclosed in US6819779B1, US6813370B1, US9286524B2, CN105975957A, and JP 2002150302A. More computer vision-based methods have been described below:

[0004] • András Bódis Szomorú et al.: “A lane detection algorithm based on wide-baseline stereo vision for advanced driver assistance,” KéPAF 2009, 7th Conference of the Society for Image Processing and Pattern Recognition, Budapest, Hungary, 2009. • Sergiu Nedevschi et al.: “3D Lane Detection System Based on Stereovision,” IEEE Intelligent Transportation Systems Conference 2004, Washington, D.C., October 3-6, 2004.

[0005] Rui Fan et al., “Real-time stereo vision-based lane detection system”, Meas.Sci. Technology, Vol. 29, 074005, 2018.

[0006] US 2018 / 0067494A1 discloses another type of lane detection method in which a 3D point cloud from a LiDAR (light detection and ranging) device is used for lane detection. The main drawback of this method is the use of LiDAR equipment; that is, it requires specialized and expensive equipment to achieve lane detection. The use of LiDAR is also disadvantageous from the perspective of the "visible" object, as the LiDAR point cloud does not contain information about occluded objects and / or about the occluded side of the object, meaning the point cloud contains limited spatial information. A similar method using depth information is disclosed in US 2018 / 131924A1.

[0007] US2018 / 283892A1 discloses lane marking detection related to the use of high-resolution maps. In this method, lane marking detection is accomplished by semantic segmentation, which results in a wide region being associated with lane markings on the image. This method produces high noise in 3D projection even when performing Douglas Peucker polygonization for line segment extraction.

[0008] US 10055650B2 discloses a similar approach, in which lane detection is performed as part of object detection, which is used to verify lanes between other objects (e.g., peripheral vehicles, traffic signs, danger zones, or tunnels).

[0009] Similar techniques based on segmentation and object classification, used for identifying, for example, lane markings, are disclosed in WO2018 / 104563A2, WO2018 / 172849A2, US9902401B2, US9286524B1, CN107092862A, EP3171292A1, and US10007854B2. US9884623B2 discloses a method for determining lane boundaries using, for example, a neural network.

[0010] Given the known methods, there is a need for a lane detection method and system that is more efficient than existing technologies.

[0011] Invention Description

[0012] The main objective of this invention is to provide a method and system for lane detection that avoids the disadvantages of prior art methods to the greatest extent possible.

[0013] Another objective of the methods and systems for lane detection is to provide an improved method for lane detection that is more efficient than existing methods. The objective of this invention is to provide 3D (three-dimensional) lane detection using machine learning based on first and second images.

[0014] The object of the present invention can be achieved by the following: the method and system for lane detection according to claims 1 and 13 respectively; the method and system for training a neural network according to claims 25 and 26 respectively; and the non-transitory computer-readable medium according to claim 27. Preferred embodiments of the invention are defined in the dependent claims.

[0015] The invention is disclosed below to illustrate the invention with reference to exemplary embodiments. In an exemplary embodiment of the method and system according to the invention (see also the possible summary below), 3D lane detection is performed based on the original detection of lane boundaries in the input image (defining the position of lane boundaries on the image) and the correspondence mapping that gives the correspondence between the original detections of two images (defining corresponding lane boundary pairs on the stereo image pair). The original detection and correspondence mapping are the outputs of the image processing module (unit) applied in this embodiment of the method and system according to the invention. Therefore, the direct identification of lane boundary localization is performed according to the invention (e.g., based on the centerline of the lane boundary marking). This constitutes a method different from the aforementioned known segmentation-based techniques (see Davy Neven et al., “Towards End-to-End Lane Detection: an Instance Segmentation Approach”, 2018, arXiv:1802.05591; Yen Chang Hsu et al., “Learning to Cluster for Proposal-Free Instance Segmentation”, 2018, arXiv:1803.06459, also hereinafter; preferably, the problem of how to efficiently process stereo information has been solved according to the present invention compared to these methods), because in these known methods, lane boundary markers have been searched using segmentation methods performed by machine learning algorithms, i.e., the boundary markers have been studied as patches on the image, similar to any other object on the image.

[0016] Based on the preceding sections, image processing modules trained using machine learning are generally used in the methods and systems according to the present invention. However, the use of this image processing module is explained by using a neural network as an exemplary implementation. The explanation of the neural network also shows several parameters and other details to be set for the selected machine learning implementation.

[0017] Brief description of the attached figures

[0018] Preferred embodiments of the present invention will now be described by way of example with reference to the accompanying drawings, wherein:

[0019] Figure 1 This is a flowchart illustrating embodiments of the method and system according to the present invention.

[0020] Figure 2 This is a diagram illustrating various correspondence mappings in one embodiment and their relationship to different types of raw detections.

[0021] Figures 3A-3C This is a schematic diagram illustrating the unification of an exemplary left and right image in one embodiment, illustrating various pairing masks and search directions.

[0022] Figures 4A-4F In the context of Figures 3A-3C The left and right images, which illustrate paired masks and lane boundaries in similar embodiments, are schematic examples.

[0023] Figure 4G It shows the relationship with Figures 4A-4F The corresponding left image space detection scheme,

[0024] Figure 5A and 5B It is based on an embodiment of the method according to the invention. Figure 4A-4G The Y(Z) and X(Z) plots of the results obtained from the scenario shown in the image are as follows.

[0025] Figure 6A and 6B It is based on known methods Figure 4A-4G The Y(Z) and X(Z) plots of the results obtained from the scenario shown in the image are as follows.

[0026] Figure 7A and 7B This is a schematic diagram illustrating exemplary first and second camera images showing lane boundaries and correspondence mapping in another embodiment, and

[0027] Figure 8A and 8B These are illustrated exemplary left and right images showing lane boundaries and corresponding mappings obtained through stereo instance segmentation.

[0028] Modes for implementing inventions

[0029] This invention relates to a method and system for lane detection. Within the framework of this invention, this means that, as a result of the method or system according to the invention, 3D (three-dimensional) lane detection data is obtained based on an input image. More specifically, preferably, the 3D representation of the 3D lane boundary is obtained based on image pairs (preferably taken by vehicles traveling along the lanes corresponding to the lane boundaries). Figure 1The flowcharts illustrating embodiments of the method and system are explained (with regard to the method, the boxes corresponding to stages or data blocks, and with regard to the system, the boxes corresponding to modules or data blocks).

[0030] The method according to the present invention includes the following steps:

[0031] - In the image processing step, based on the first image and the second image ( Figure 1 In this embodiment, the image pairs of the first image 10a and the second image 10b are used to generate output data by an image processing module trained with machine learning. The correspondence mapping data determines the correspondence between the first lane boundary group of the first image and the second lane boundary group of the second image.

[0032] - In the pairing step (also known as the image spatial detection data generation step), data blocks for image spatial detection pairs are generated based on the output data, and

[0033] - In the 3D lane detection data generation step, calibration data corresponding to the first and second images are used to generate 3D lane detection data blocks based on triangulation.

[0034] The first data block portion corresponding to the first member of the image spatial detection pair, and the data block of the image spatial detection pair.

[0035] The second data block portion of the data block of the image spatial detection pair, corresponding to the second member of the image spatial detection pair.

[0036] Of course, 3D can be written as three-dimensional in the corresponding name.

[0037] The first or second lane boundary group is a group of lane boundaries that can be found in the corresponding image. A lane boundary group may include one or more lane boundaries, but in edge cases, the number of lane boundaries in the image may even be zero. The name "lane boundary group" also reflects this; the name does not indicate that it contains multiple lane boundaries or zero lane boundaries. Therefore, correspondence mapping data gives the correspondence between lane boundary groups, with the number of lane boundaries included in these groups unknown in advance.

[0038] According to the above description of the present invention, correspondence mapping data is typically generated in the method according to the present invention. As will be shown below, in some embodiments, the correspondence mapping data is merged into a separate correspondence mapping data block (see also the next embodiment). Typically, in embodiments with separate correspondence mapping data blocks, the original detection data block is also defined as the output of the image processing module.

[0039] However, different implementations of the correspondence mapping data are conceivable. In one embodiment, only the correspondence mapping data is used in further calculations of the 3D lane detection data, without using the original detection data at all. For illustrative purposes, an example is presented by Tables 8a-8d below (however, in Tables 8a-8d, the original detection is defined in addition to the correspondence mapping data): the coordinates corresponding to the non-zero values ​​of the correspondence mapping data give a good approximation of the lane boundary arrangement, and, if necessary, this data is used alone for image-space detection and 3D coordinates of the lane boundaries using a coordinate grid. The quality of this approximation is limited by the resolution of the correspondence mapping data. If the resolution of the correspondence mapping data is similar to the resolution of the input image, the original detection will be interpreted as a small correction to the neglect of the coordinates of the lane boundaries inferred from the correspondence mapping data. In this case, in addition to the primary purpose of determining correspondence, the correspondence mapping data also defines the arrangement of each first member of the first lane boundary group and the arrangement of each second member of the second lane boundary group.

[0040] Furthermore, it is conceivable that the correspondence mapping data is incorporated into the original detection from the outset, so only those original detection data elements with high correspondence are included in the original detection data block. This filtering can also be performed when separate correspondence mapping data blocks are defined.

[0041] In summary, in all embodiments of the present invention, the correspondence mapping data (and other outputs in other embodiments, such as raw detections) is generated in a suitable form by means of an image processing module trained by machine learning (i.e., an image processing module implemented as a neural network, which can be trained by machine learning to generate the correspondence mapping data and other possible related outputs). In other words, data responsible for determining the correspondence between lane boundaries of the first and second images is defined in all embodiments. Therefore, in other words, the trained image processing module is a trained machine learning model.

[0042] In the above description of the invention, a correspondence between first and second lane boundary detection groups of corresponding first and second images is determined. These groups may include one or more lane boundaries (corresponding to each lane boundary that is observable on the corresponding first or second image), but may also include zero detection in the case of boundary lines when no lane boundaries are detected on the image (then all information on the image corresponds to the background, see below to distinguish foreground and background).

[0043] It is also noted here that in the pairing step, (all) output data generated in the image processing step is used to generate data blocks for image spatial detection pairs; that is, if only correspondence mapping data exists, it is used only, but if original detection data blocks exist, these data blocks are also used. In some embodiments below, what is included in the output data forwarded to the pairing step is also specified.

[0044] As shown below, triangulation is the most common method upon which 3D lane detection data blocks are computed. Triangulation is a method for calculating 3D position based on point correspondences found in multiple images of a point-like object (e.g., a specific point of the object). The corresponding set of points defines a set of rays starting from the corresponding camera center (the origin of the camera coordinate system). The 3D coordinates of the object are computed as the intersections of the rays, which produces a system of linear equations. If the point correspondences are not perfect, the rays may not intersect. In this case, the estimated 3D position can be defined as the point located at the point closest to the ray in perpendicular distance. Alternatively, a minimum refinement of the correspondences can be determined numerically so that the refined correspondences satisfy the epipolar constraint, i.e., the corresponding rays intersect (see Section 12.5 of the book, which is incorporated herein by reference). Disparity-based computation is a special variant of triangulation. For more details on triangulation, see Richard Hartley and Andrew Zisserman (2003). Multiple View Geometry in Computer Vision, Cambridge University Press, ISBN 978-0-521-54051-3. Calibration data corresponding to the first and second images will also be used for triangulation, because if the calibration data is taken into account, the arrangement of the images being compared can be satisfactorily defined.

[0045] In one embodiment, an initial detection is also defined during the method. In this embodiment of the method, the correspondence mapping data consists of correspondence mapping data blocks (e.g., Figure 1 The output data consists of correspondence data blocks 16 marked as "correspondence mapping", and also includes...

[0046] - The first image raw detection data block 14a used to define the arrangement of each first member of the first lane boundary detection group, and

[0047] - The second image raw detection data block 14b is used to define the arrangement of each second member of the second lane boundary detection group.

[0048] See Figure 1The embodiment refers to a first image raw detection data block 14a labeled "First Image Raw Detection". Typically, a raw detection data block includes data from one or more raw detections. In the case of zero raw detections, the result of this method is that lane boundaries cannot be found, and therefore 3D coordinates cannot be assigned to them. A single detection means a single finite-value parameter or another parameter corresponding to a distance, as shown below. See also... Figure 1 The second image raw detection data block 14b is labeled "Second Image Raw Detection". These data blocks are called raw detection data blocks because they correspond to the raw detection of the corresponding data to be processed (data defining the lane boundary arrangement). As described above, the lane boundary arrangement can be defined by the raw detection data blocks, and alternatively by means of the correspondence mapping data itself. These two alternatives provide two main possibilities. Furthermore, when the arrangement is defined by means of raw detection, two subclasses can be introduced as described below. In the first subclass, the raw detection and correspondence mapping are independent outputs of the image processing module (see Tables 1a-1c below). However, in the second subclass, the raw detection and correspondence mapping data are obtained as outputs in such a way that the raw detection is essentially used as input to the correspondence mapping data (see the model-based embodiment below).

[0049] In the above description of the method according to the invention, the computation of data blocks is defined. For an explanation of "data block", please refer to the following details (e.g., a data block can be represented by a tensor); generally, a data block is a block of data (where the block has an arbitrary shape), and the members of the data block are usually numerical values, as explained by the examples below.

[0050] It is also mentioned here that, as will be used in many embodiments, in one embodiment, the image pair of the first image and the second image is a stereoscopic image pair of the left image and the right image (for capturing stereoscopic images, it is generally limited to the two images being captured simultaneously, and preferably the two imaging cameras being located in the same plane). Nevertheless, if the first image and the second image generated in this way have sufficient overlap for detection, the first image and the second image can come from any relative orientation of the imaging device and have different timings.

[0051] Based on the details given above, for lane detection purposes, lane boundaries are preferably defined as continuous curves dividing the drivable road surface into lanes. When lanes are bounded by lane markings, the lane boundaries preferably follow the centerline of the lane boundary markings, which can be continuous (single or double) or can have some simple or more complex dashed lines. Alternatively, the lane boundary can also be another specific line corresponding to the sequence of lane markings (it can be along its right or left side, or anywhere else). Therefore, lane boundaries are preferably single lines (without width dimensions), for example, double closed (continuous) lane markings are also projected as single lines. Depending on national regulations, it may be the case that the side lane boundaries (i.e., the side of the lane located on the side of the road itself) are not marked by lane boundary markings. Therefore, this lane boundary (the roadside is also covered by the name "lane boundary") is a simple transition between the road and the area beside the road (which can be a sidewalk, grass-covered area, gravel, etc.).

[0052] A significant advantage of the solution according to the invention is that it can also handle this type of lane boundary. During the learning process, the image processing module, or in a concrete implementation, the neural network (with the aid of an appropriate type of real-world ground image), is also able to learn this type of lane boundary (the network searches for such transitions, which can also be approximated by a single straight line on the image), and it can recognize this in use. Therefore, this different type of lane boundary can be identified in a manner similar to lane boundaries with lane boundary markings, without requiring a conceptual change in the approach (the annotation technique is the same). Thus, the two boundaries of a lane (i.e., the boundaries on either side of it) can preferably be modeled using the invention.

[0053] In the pairing step and the pairing module below, detection pairs are identified. These can be represented, for example, in a single block using channel-like parameters for the first and second detections (see [link to module]). Figure 1 The output (where the output is given as a single output to the next module) can be represented, for example, as more separate data blocks for the first and second images, see the following example (separate first and second image spatial detection data blocks for the correspondence mapping data block and corresponding first or second image raw detection data blocks; the image spatial detection data blocks can also be part of a larger data block obtained in the pairing step). In the pairing step, such detections are identified, which constitute pairings, and disparity can be calculated, for example, based on these pairings.

[0054] Image spatial detections are obtained in the pairing step (and therefore in pairing module 20). These can have single-tensor or dual-tensor representations, as given above. There are corresponding first and second data block portions of the data blocks of the image spatial detection pair, corresponding to the corresponding first and second members of the image spatial detection pair. From the data blocks of the image spatial detection pair, it is always possible to derive which portions correspond to the corresponding members of the pair, i.e., its first and second data block portions can be defined (e.g., see the two tensor representations in Tables 2b and 2c or 6b and 6c; data separation can also be performed on a single tensor representation).

[0055] The corresponding system according to the invention (preferably adapted to perform the method according to the invention) is also applicable to lane detection, including...

[0056] - Image processing module trained by machine learning (by...) Figure 1 The image processing module 12 in this embodiment is implemented by labeling the main component of the module in this embodiment as a "neural network". It is adapted to generate correspondence mapping data as output data based on the image pairs of the first image and the second image, which determines the correspondence between the first lane boundary detection group of the first image and the second lane boundary detection group of the second image (in the embodiment of the system, the original detection data block can also be introduced).

[0057] - Pairing module ( Figure 1 In the embodiments, the pairing module 20 (labeled "pairing") is adapted to generate data blocks for image spatial detection pairs based on the output data, and

[0058] -3D projection module ( Figure 1 The 3D projection module 24 in the embodiment, labeled "3D projection"; this module is adapted for 3D projection, and is adapted to generate 3D lane detection data blocks by means of triangulation, using calibration data corresponding to the first and second images, based on the following

[0059] -The first data block portion of the data block of the image spatial detection pair, corresponding to the first member of the image spatial detection pair, and

[0060] - The second data block portion of the data block of the image spatial detection pair, corresponding to the second member of the image spatial detection pair.

[0061] Preferably, the system (or equivalently, it can be considered as a device) can be implemented by including the following:

[0062] - One or more processors; and

[0063] - A memory coupled to one or more processors and including program code, which, when executed by one or more processors, causes the system to perform tasks (functions, i.e. steps defined in the corresponding embodiments of the method according to the invention) by the modules introduced according to the above definition of the system, and more specifically, to generate the various quantities generated in the above modules.

[0064] The above modules provide a task-oriented allocation of computer sub-components, which can also be considered from the perspective of the system (device) itself being responsible for various tasks.

[0065] It is also clear from this method that disparity data needs to be calculated in not all cases. For the system, this means that it does not necessarily need to include a disparity module. If the calculation of the 3D lane detection data block is also based on this data (preferably arranged in advance), then the system includes a disparity module.

[0066] The following points provide an overview of the methods and systems according to embodiments of the present invention (further summaries are given in the description for some points):

[0067] 1. Input:

[0068] a. Image pairs (preferably stereoscopic and corrected)

[0069] b. Calibration

[0070] 2. Image processing module (arbitrary, except for constraints on input and output, such as implementation via neural networks):

[0071] a. Input (The image processing module can be fed stereo input)

[0072] b. Output:

[0073] i. Correspondence mapping

[0074] ii. Raw detection on individual images

[0075] iii. Optional output for additional tasks

[0076] 3. Pairing Module

[0077] 4.3D projection module

[0078] Some embodiments of the present invention relate to methods and systems for training an image processing module (e.g., a neural network) applied to the lane detection method and system according to the present invention as described above. This training method and system are described in parallel with methods and systems for lane detection; the training method and system use the loss function described below. Since each loss module (or stage) is also... Figure 1The explanation in the text thus describes both the methods and systems used for lane detection, as well as the methods and systems used for training the image processing module.

[0079] A key component of this invention is the machine learning implementation used in the image processing module, which introduces the use of annotations and helps avoid manual parameter setting. The image processing module trained via machine learning is preferably implemented using a neural network (i.e., it is used in the image processing module), but it can also be implemented using, for example, decision trees, support vector machines, random forests, or other types of machine learning. Generally, the trained machine learning model (and therefore the image processing module trained via machine learning) represents a broader category of artificial intelligence than neural networks, which includes methods that use neural networks.

[0080] By minimizing the loss function, the (learnable) parameters of the image processing module trained by machine learning are optimized during training. Preferably, the parameters are updated to result in lower loss values, thus ensuring that the loss is controllable during training. The loss function corresponds to the objective of the image processing module trained by machine learning; that is, the loss function depends on the type of output expected after the machine learning procedure. For example, it is not preferred that the image processing module trained by machine learning places detections far from lane boundaries (also known as separators), as a high loss would correspond to detections placed far from the ground. In another example, the error can be considered smaller when the image processing module trained by machine learning cannot determine whether a marker is a dashed line or continuous, compared to the error in determining a point (whether that point is part of the foreground or background). Therefore, the members of the loss function responsible for different errors can be weighted.

[0081] The loss function quantifies the difference between the prediction (the output of the image processing module) and the reference output (the ground truth) that we want to achieve. The loss function can include various elements, such as detection loss in image space, detection loss in 3D, etc., and can be introduced at different blocks of the lane detection system.

[0082] Advantageously, in a preferred embodiment using a neural network as an image processing module trained via machine learning, the system according to the invention is preferably fully differentiable. Then, all gradients of the loss function can be computed via backpropagation, i.e., each gradient can flow through various modules and return to each parameter and the input parameters of the neural network. Figure 1(The dashed lines in the diagram). The neural network, neural network output, parallax module, and 3D projection module are naturally differentiable. Conversely, the pairing module must be constructed to make it differentiable. In one embodiment, gradients from other modules and through the pairing module are propagated back to the modules of the first image raw detection data block 14a and the second image raw detection data block 14b, and optionally to the module of the correspondence mapping data block 16. Thus, the learning process for the entire system can be performed in a uniform manner.

[0083] The gradient of the loss function is used to update the values ​​of the learnable parameters of the neural network. As the gradient flows back through various modules, the loss function can optimize not only the learnable parameters of the modules assigned to it (see various loss modules for details). Figure 1 Furthermore, it can optimize the entire neural network preceding this module. Some neural networks are designed so that different parts must be trained separately to achieve good performance. In contrast, this invention provides an end-to-end trainable solution that eliminates the need for complex training processes.

[0084] Figure 1 The text explains the main blocks and the information flow between them, namely... Figure 1 A modular description of the 3D lane detection system according to the present invention is given, as well as a system based on... Figure 1 This reveals the corresponding 3D lane detection method according to the present invention. Figure 1 The text also explains ground real data 28 through blocks (i.e., modules suitable for forwarding ground real data). The modules for ground real data 28 are preferably connected to all loss modules for calculating the losses in those modules.

[0085] In summary, it is preferable to assign the loss module to each of the modules that drives the neural network during the learning process to produce such an output as the ground truth.

[0086] exist Figure 1 In the inference mode, the information flow during inference (i.e., the use of methods and systems for 3D lane detection) is explained by solid arrows (forward path). During training, information propagates backward from the loss module to the neural network along dashed lines. Dashed lines correspond to data connections through which data is passed back from the loss module. In the modules along this data path, dashed lines are interconnected via dashed circles; connections via dashed circles serve only an interpretive purpose, and data packets from the corresponding loss modules travel in parallel to the image processing module 12. Loss modules have a special role because they are only used during the learning process of the neural network. Therefore, it is possible that... Figure 1 The explanation in the text separates the individual loss modules (e.g., using another type of line to connect them to the corresponding modules and / or the loss modules themselves may have been explained by an explanation tool, such as by using...). Figure 1The diagram shows other types of rectangles (e.g., dashed lines) used to separate the loss module.

[0087] Additional output block 18 refers to additional tasks performed by the neural network, which may be closely or weakly related to lane detection, such as road segmentation, traffic object detection, lane type determination, etc. Additional regularization losses (not shown in the accompanying figure) may be added to the neural network. Additional output block 18 preferably has its own loss module 30.

[0088] The output of the 3D projection module (i.e., the entire 3D lane detection system) can be the input of other modules in an autonomous driving application (as explained by the arrow labeled "Next Module" exiting from the 3D projection module 24). Furthermore, data from the additional output block 18 is also available to other modules.

[0089] For example, the extracted 3D lane boundaries can be used for online camera parameter estimation, motion planning or trajectory planning, localization, or any other module that may benefit from the 3D location of the lane boundaries. Optionally, the 3D detection of the lane detection system can be input to a sensor fusion algorithm that determines a (3D) road model from various sources, such as lane detection, road segmentation, and object detection, and various sources, such as cameras, LiDAR, or radar.

[0090] Figure 1 The input of the image processing module 12 is shown (in Figure 1 The header is simply labeled "Input"; the header shows the successive stages of the system and the corresponding methods. It has two image inputs (typically, a first image 10a and a second image 10b, which can be a left image and a right image). These image inputs preferably give a pair of calibrated stereo images, of arbitrary size and channels (grayscale or RGB, etc.). In principle, the calibration can be moved to the lane detection system (i.e., implemented therein), and then the input to the image processing module 12 is the stereo image pair and calibration data 26, and the first step is calibration.

[0091] Calibration data 26 is the input to the 3D projection module 24. The calibration data describes the internal and external parameters of the camera. External parameters characterize the relative position and orientation of the two cameras. Internal parameters, along with the camera model used to interpret the image, determine how 3D points in the camera coordinate system are projected onto the image plane. The calibration data is required for correction, but correction can be performed in a module that is not part of the lane detection system (performed before entering the system). Therefore, the lane detection system and method can receive corrected camera images, as well as calibration data describing the corrected camera images. Furthermore, as... Figure 1 As explained herein, calibration data describing the corrected camera images is preferably used for 3D projection.

[0092] Stereo camera and corresponding image calibration is a standard method used in stereo image processing. It transforms the camera and corresponding images into simplified epipolar geometry, where the two camera image planes coincide and their epipolar lines coincide. Points on the image planes represent rays in the camera coordinate system passing through the origin. For any point on one of the image planes, the corresponding epipolar line is the projection of the corresponding ray onto the other image plane. This means that if a 3D point corresponding to a real object is found on one of the images, its projection onto the other image plane lies on the corresponding epipolar line. For more details on calibration, see Bradski, G., Kaehler, A.: O'Reilly learning OpenCV. First (edn), ISBN: 978-0-596-51613-0. O'Reilly Media Inc., New York, USA (2010).

[0093] Introducing image processing module 12 (in) Figure 1 When a separate label "NN" is provided in the header, the architecture of the neural network used is first described. The neural network applies a series of transformations to the input to produce the desired output. These transformations may depend on learnable parameters, which are determined during the training of the neural network. In most applications, these transformations form a hierarchical structure, where the input of one layer is the output of the previous layer. These inputs and outputs of each layer are called features. The neural network used in a 3D lane detection system can contain any number and type of layers, including but not limited to convolutional layers, activation layers, normalization layers, etc. The sequence of layers (or more generally, transformations) is defined by the neural network architecture. In this invention, preferably, the so-called stereo lane detection neural network receives a stereo image pair (consisting of a first image and a second image) as input, which must be obeyed (processed) by the neural network architecture.

[0094] In the image processing module, processing can begin on the first image 10a and the second image 10b respectively. This can be beneficial for low-level filters to focus on a single image, such as forming edge or color detectors. The two branches of the network may or may not use the same filters (weights). Filter sharing reduces the number of learnable parameters (weights). Therefore, shared filters can preferably be used in parallel in both branches to search for the same features. However, it is also allowed for filters to be learned independently on both branches. The latter is preferred in cases where the first and second cameras are not equivalent.

[0095] In an embodiment of the lane detection system according to the invention, a neural network is used in the image processing module trained by machine learning (of course, if there are no obstacles, those system-related features can also be applied within the framework of the method). Before the output layer of the image processing module, the features of the two images are combined in the network at least once (i.e., a blending step is inserted). Feature combination (i.e., blending) may have already been done on the input image or in a subsequent layer. Combination (blending) can be achieved, for example, as feature concatenation, addition, multiplication, or nonlinear functions of the input features.

[0096] Suppose the first and second features are blended; these features can be represented as tensors, which can have height, width, channel dimensions (or additional batch and time dimensions). In many cases, blending only occurs in the channel dimension. In principle, more complex blends can be defined where the tensor elements to be combined are blended not only in the channel dimension but also in the spatial dimension.

[0097] It's important to note that in the hybrid approach introduced above, the neural network's output is compared (adversarially) with ground-based data. Since hybridization helps obtain a plausible correspondence mapping, it is the neural network's task to achieve an output close to ground reality based on the hybrid output (performed during the learning process by selecting, for example, appropriate weights).

[0098] For neural networks in a 3D lane detection system, this mixing step (also known as combination) is preferred to determine the output (which is a correspondence map) depending on the two images, such as the correspondence map data block defined in the Output section. The mixing of first and second camera features also enhances the correlation between the errors (difference between prediction and ground reality) of the outputs corresponding to the first and second images (e.g., the first and second camera detections defined in the Output section), which is crucial for improving the accuracy of the predicted 3D coordinates by the lane detection system according to the invention. This can be explained by the principle that positive correlation between random variables reduces the variance of their differences.

[0099] The correlation between the enhancement errors is as follows. During the use of the neural network, it finds lane boundary markers, but inevitably some errors (to the left or right). Without blending, there is no explicit enhancement, meaning the errors are consistent across the first and second images.

[0100] Therefore, in embodiments of this method, a hybrid approach is applied, wherein the image processing module is implemented by a neural network having a first branch for (i.e., adapted to be applied to or adapted to be used for) a first image and a second branch for a second image, and during the image processing steps, the method includes a combination step of combining first branch data as the output of the first branch of the first branch and second branch data as the output of the second branch of the second branch (instead of this combination step, in the corresponding system: combining the first branch data as the output of the first branch of the first branch and the second branch data as the output of the second branch of the second branch in the image processing module). More generally, it can be said that, preferably, all outputs of the image processing module trained by machine learning can access both input images (it can access, i.e., both images may affect the output, but not necessarily both), i.e., combining the processing of the first image and the second image in the image processing module.

[0101] The 3D distance (depth) between the detected lane boundary points and the camera plane is inferred based on disparity, which is the difference between the detections in the corresponding first and second images. Without mixing, the detection errors on the two images are independent (uncorrelated), while mixing reduces the variance of the inferred depth values.

[0102] The following description relates to the preceding sections. Image pairs consisting of multiple consecutive frames can be processed similarly to the per-frame case without altering the output of the neural network and subsequent modules. The simplest implementation uses the original per-frame architecture and cascades the RGB or grayscale channels of the consecutive frames in the left and right image inputs. The target can be selected as the original target of the last frame. Information from previous frames (first and second images) makes the output more robust to per-frame noise (e.g., noise caused by windshield wipers) (i.e., preferably, the image processing module considers at least one previous frame to generate its output for the frame). More generally, it can be processed like a stereo image pair, that is, through a mixture of parallel processing and feature combination.

[0103] In summary, in one embodiment, during the image processing step (or in the system: generating output data by means of an image processing module), in addition to the actual frames of the first and second images that serve as input data, at least one additional frame of the first and second images preceding or following the actual frames is used. If subsequent frames are used, it is necessary to wait for these frames.

[0104] according to Figure 1 In an embodiment, the neural network of the 3D lane detection system preferably has three characteristic outputs (in Figure 1 (The header is labeled "NN output"):

[0105] 1. First camera detection (first camera raw detection data block 14a)

[0106] 2. Second camera detection (second camera raw detection data block 14b)

[0107] 3. Correspondence Mapping (Correspondence Mapping Data Block 16)

[0108] Preferably, these are also the feature outputs of other embodiments.

[0109] Additional outputs can be added for supplementary tasks, such as clustering, lane type segmentation, road segmentation, vehicle or pedestrian segmentation, etc. Further outputs are included in... Figure 1 It is in additional output block 18. See below for details about the output.

[0110] The original detection is a (neural) representation of lane boundaries on an image. Lane boundaries can be represented in various ways by neural networks. For example, the representation can be dense (e.g., pixel classification or regression of lane boundary points) or model-based (e.g., polynomial, spline parameter; see the description of the model-based embodiments for details). The output of the image processing module of the 3D lane detection system can be any representation; some exemplary representations of the original detection are described below.

[0111] Note that, according to the invention, the representation of the original detection is not limited, and the exemplary embodiments are intended to show that various representations can be conceived within the framework of the invention. The architecture described in the previous section produces correlated first and second camera detections (see above, where correlation enhancement is refined) because they can access features and / or blended features from other viewpoints. The possibility of accessing information from other cameras can improve the quality of image detection. However, if the correlation is not explicitly enhanced during training (e.g., through an appropriate loss function), the correlation between the detections remains small. To obtain 3D consistent detection, a loss function is used that strengthens the correlation between detection errors.

[0112] The raw detections may only indicate the location of lane boundaries (centerlines), or they may describe other characteristics, such as the direction of lane boundary segments or the type of separator lines (see below, for example, to show how this information can be selectively incorporated into the framework of this invention). Note that some characteristics of the separator lines (e.g., lane type) can be more naturally incorporated into the correspondence mapping output.

[0113] The following is a commentary on the loss for the original detection. During training, the original detection can acquire a representation loss (see...). Figure 1The loss (representation) modules 32 and 34 in the model characterize the difference between the ground truth representation of the lane boundary and the original detection. The mathematical form of the loss function is chosen based on the representation. For example, if the representation corresponds to a regression problem (see below), a natural choice could be Euclidean (L2) distance or Manhattan (L1) distance (L1 and L2 losses can also be used in model-based embodiments, see below; however, in this case, the difference in model parameters occurs in the loss, not the distance between image points). Here, L... p The distance in the pointer field that is related to the p-norm. For example, if a and b are derived from a... i and b i If the vectors formed by a and b are denoted by b, then the distance between them can be characterized by the p-norm of their difference.

[0114]

[0115] In the loss function, the distance raised to the power of p is typically used, related to L. p For an example of loss, see equation (6).

[0116] If the original detection originates from a classification problem, standard classification losses can be applied, such as softmax loss (Equation (9)) or SVM loss (Support Vector Machine loss). In general, any convex function can be used as the loss, which has a minimum value when the predicted value equals the ground truth value.

[0117] Compared to parallax loss and 3D projection loss, this representation loss does not directly encourage the network to produce relevant detection errors.

[0118] Another characteristic output of the 3D lane detection network is the correspondence mapping, which will be described below. The correspondence mapping is preferably represented as the correspondence between lane boundaries on the first and second images of the original detection input, eliminating the need to run a separate stereo matching algorithm and thus significantly reducing computation time. Various solutions may exist for the neural representation of the correspondence between the detections in the first and second images; several different examples of the correspondence mapping are described below with reference to some embodiments.

[0119] Correspondence mapping can be viewed as a correspondence between abstract entities (e.g., lane boundaries) that may or may not have a visual appearance in the input image; however, contextual information provides the probability of predicting descriptors representing these abstract entities (e.g., the original detections). Therefore, this is a completely different approach compared to traditional vision-based stereo matching solutions. Furthermore, correspondence mapping is done within the neural network itself (provided as output; these principles are generally true for image processing modules trained via machine learning), as it provides correspondences between detections on stereo image pairs. It differs from traditional explicit stereo matching in the following ways.

[0120] • No manual rules are used for matching; rules for correspondence are learned by the neural network during the training phase (given as a separate output).

[0121] • Provide the correspondence between the detections (which represent lane boundaries) rather than between image patches (note that standard stereo matching methods search for correspondences between local segments of an image; see also the next point);

[0122] The correspondence is not based on local similarity of image segments, but on lane boundary descriptors (which can be selected in various ways), thus it is reliable even in the lane boundary portion between occluded areas (regions) and lane markings, and is more robust to noise in the image. According to the invention, preferably, lane markings (e.g., when dashed and discontinuous) are not searched in themselves, but rather they are connected, and the resulting continuous lane boundaries are ground truth annotations, and searched and output by a neural network (e.g., see examples where sample outputs of the neural network are given; in these examples, the results of continuous lane boundaries can also be observed). This approach is also advantageous in more distant parts of the image (where dashed boundary markings are not easily captured). Therefore, for the position between previous and subsequent lane boundary markings, a high probability or label of the lane boundary can be given in the corresponding embodiments.

[0123] Lane boundaries that are considered to be simple shapes are searched, which usually do not have overly complex runs on the map; most importantly, lane boundaries are usually straight or gently curved; this characteristic of lane boundaries is also helpful in the case of occluded areas.

[0124] In principle, a correspondence mapping can be a dense one-to-one match between pixels in two images. However, as detailed below, preferably for the 3D lane detection system according to the invention, a much simpler mapping is sufficient. For example, it can be chosen to describe the correspondence between regions around lane boundaries in two images (see the introduction to paired masks and stereo instance segmentation).

[0125] For an explanation of the loss on correspondence mappings, see below. Correspondence mappings (i.e., neural networks capable of outputting correspondence mapping data) are trained using ground-real data during the training process (see Correspondence Mapping Loss Module 36). The loss function compares the ground-real correspondences with the predicted correspondences and has a minimum value when the two correspondences are equivalent. Naturally, based on ground reality, the correspondence can be accurately given in any representation of the correspondence mapping. Correspondence mappings depend on two images; therefore, the corresponding loss function encourages the network to generate filters sensitive to the features of both images.

[0126] Accordingly, the system for training the image processing module applied to the system for lane detection thus includes the following steps (in the corresponding method for training the image processing module, the following control steps are performed; given in the following sections) Figure 1 The reference numerals in the accompanying drawings, in which a neural network is used (summing the loss values ​​and minimizing them during training), indicate that the 3D projection loss module 42 is adapted to control the 3D lane detection data blocks during the training of the image processing module by modifying the learnable parameters of the image processing module. Optionally, it also includes the step of a correspondence mapping loss module 36, which is adapted to control the correspondence mapping loss of the correspondence mapping data during the training of the image processing module by modifying the learnable parameters of the image processing module.

[0127] The system may include an image spatial loss module 38, which is adapted to control a first image spatial detection data block and a second image spatial detection data block during training.

[0128] Based on the above construction, in addition to the correspondence mapping loss, image spatial loss and / or 3D projection loss can also be controlled. Furthermore, for example, in... Figure 1 In one embodiment (i.e., in which parallax is used), the system for training the image processing module further includes a parallax loss module 40 adapted to control parallax data blocks based on ground-based real data 28 during training. In the following embodiments, the system (and method) for training the image processing module applied to a system for lane detection is fully differentiable. In this embodiment, the correspondence mapping data consists of correspondence mapping data blocks, and the correspondence mapping loss module is adapted to control the correspondence mapping loss of the correspondence mapping data blocks. The image processing module 12 is implemented by a neural network, and the system also includes…

[0129] - The first representation loss module 32 is adapted to control the first representation loss of the first image raw detection data block 14a during training, and

[0130] - The second representation loss module 34 is adapted to control the second representation loss of the second image raw detection data block 14b during training.

[0131] In summary, in this embodiment, the machine learning algorithm is implemented by a neural network, where all transformations are differentiable and depend on the predictions of the neural network used during training. The pairing step and the 3D projection step are differentiable with respect to their inputs, thereby allowing the gradients of the 3D projection loss and the image space loss to be computed with respect to the neural network parameters.

[0132] Furthermore, in this embodiment, the system for lane detection is fully differentiable, meaning that during training, all loss functions (including but not limited to 3D projection loss and image space loss) are differentiable with respect to all learnable parameters of the machine learning algorithm. This is achieved through a differentiable image processing module trained by machine learning, for example, through a neural network, characterized by all transformations (layers) being differentiable. Additionally, depending on the predictions of the neural network used during training, the pairing step and the 3D projection step are differentiable with respect to their inputs.

[0133] The image processing module (neural network) can use any representation for detection and for describing the correspondence between detections on two images (see exemplary embodiments below). The output is forwarded to the pairing module 20 (i.e., the first image raw detection data block 14a, the second image raw detection data block 14b, and the correspondence mapping data block 16 in the NN output, as illustrated by the arrows from these blocks to the pairing module 20). The pairing module is adapted to map the representation to a standardized form, and its output is an image spatial representation of the detection pair (e.g., see the table below). This standardized representation is the input to the disparity block.

[0134] As mentioned above, the representation is arbitrary, and tensor (matrix)-based embodiments are described below (e.g., for representations different from tensor-based representations, see model-based embodiments). In such embodiments of the method and system, output data is generated for a coordinate grid (i.e., the output data can be interpreted on a coordinate grid), which is represented by a coordinate grid tensor with multiple grid tensor elements (in tensor-based methods, the coordinate grid is naturally represented by a corresponding tensor), and

[0135] - The first image raw detection data block corresponds to a coordinate grid tensor (as specified here and in the next section, the raw detection is also represented by a corresponding tensor, which is preferably downsampled or upsampled compared to the original image under investigation, for example, as shown in the example), and has a first image raw detection tensor with multiple first tensor elements, including, in the first search direction in the first image (as detailed below, in the tensor-based approach, distance is used to characterize the arrangement of lane boundaries), the nearest first lane boundary point of the first lane boundary group (i.e., the point of the lane boundary closest in the search direction), and the corresponding first distance value measured from the corresponding grid tensor element; the distance between the corresponding point of the coordinate grid and the nearest lane boundary point in the search direction is measured: therefore, in the left-right image pair, these are horizontal distances).

[0136] - The second image raw detection data block is a second image raw detection tensor corresponding to the coordinate network tensor and having multiple second tensor elements. These multiple second tensor elements include the nearest second lane boundary point in the second lane boundary group along the corresponding second search direction in the second image, and the corresponding second distance value measured from the corresponding grid tensor element (in the case of correction, the search direction is preferably given along the epipolar line, which can be in the left or right direction; see also R and L search directions below).

[0137] - A correspondence mapping data block is a correspondence mapping tensor corresponding to a coordinate grid tensor and having multiple third tensor elements. These multiple third tensor elements characterize whether the nearest first lane boundary point (as described above, in the first search direction) and the nearest second lane boundary point (in the second search direction) correspond to the same lane boundary (i.e., correspond to the same lane marker on the first and second images, thereby determining the correspondence between the first lane boundary group of the first image and the second lane boundary group of the second image; these correspond to the same lane boundary in reality, i.e., correspond to the same lane boundary marker if the basis of the lane boundary is given).

[0138] Using coordinate grid tensors, tensors are generated for image spatial detection pairs, serving as data blocks for these pairs. That is, these data blocks are also represented by tensors.

[0139] The original detection tensors of the first and second images, as well as the correspondence mapping tensor, all correspond to coordinate grid tensors. Therefore, these tensors have the same size and define the values ​​of corresponding points in the coordinate grid tensor (see also the exemplary tensors in the table below). The preferred parallax data blocks and 3D lane detection data blocks can be represented in any form; as shown in Tables 3a and 3b, 7a and 7b, these data blocks are represented, for example, as tensors. However, Tables 10a and 10b show that these data blocks can also be simply represented as lists (although data blocks of different sizes, such as matrices and lists, are considered tensors).

[0140] In tensor-based approaches, the correspondence mapping tensor—for example, see the embodiment with paired masks for a single correspondence mapping tensor, and the embodiment with stereo instance segmentation for two (i.e., independent) correspondence mapping tensors for each of the first and second images—includes information about whether the closest lane boundary point investigated is the same lane boundary point. Properties in the third tensor element can be accomplished using probability values ​​as shown in some examples; other representations are also conceivable.

[0141] The following describes the function of the pairing module 20. The pairing module 20 derives the image space representation of the detection pair from the output of the image processing module 12 (i.e., from the original detection and correspondence mapping on the two images).

[0142] How the image space representation is determined depends on the original detection and correspondence mapping used in a particular embodiment. However, it is important that the transformation to the image space be differentiable (to preserve end-to-end differentiable features; as mentioned above, this expectation can be achieved). The gradient then flows naturally through the pairing module to features that are beneficial to the prediction of the detection pair (therefore, the training of the neural network can also be improved with loss data generated in stages after the pairing module; see the dashed line for the flow of loss data).

[0143] Depending on the specific implementation, gradients from the pairing module may or may not flow to the correspondence map. Figure 1 (This connection, which can be utilized accordingly, is shown.) In the example given below, the pairing module uses ground-based correspondence mapping data during training, so gradients flow only to the original detections on the left and right images. Note that the correspondence mapping output is trained directly via the correspondence mapping loss. However, other examples could be constructed where gradients flow to the correspondence mapping. The difference is that the 3D loss in the 3D projection module is used only to refine the image spatial detection, or similarly to refine the correspondence mapping.

[0144] Optionally, the image spatial loss function can be assigned to the image spatial representation corresponding to the first and second images (see [link to image space representation]). Figure 1The loss (image space) module 38 in the model represents the distance between the original detected image space representation and the true image location of the lane boundary. In some embodiments, the representation loss of the original detection is equivalent to the image space loss.

[0145] The image spatial detection pair output by the pairing module 20 is processed by the parallax module 22, as described below.

[0146] Parallax is calculated as the difference between the image coordinates of the first and second image detections along the epipolar direction (in most cases, the horizontal direction). Therefore, in a horizontal stereo setting,

[0147] parallax = u 左 -u 右 (1)

[0148] Where u is the image coordinate of the detection along the horizontal axis. The disparity for all detection pairs is calculated and forwarded to the 3D projection module along with, for example, the image coordinates of the left detection.

[0149] A loss function can be introduced into the disparity module to enable the network to produce the correct disparity values ​​(see [link]). Figure 1 The loss (parallax) module 40 in the middle implements the correlation between the detections of the first and second cameras.

[0150] The features of the 3D projection module 24 are also introduced below. The 3D projection module 24 calculates the detected 3D coordinates (e.g., in the camera coordinate system corresponding to the car holding the camera). The input is an image space representation of the detection, disparity, and camera calibration parameters (the latter being calibration data 26 from the calibration module). The 3D position of the lane boundary point can be determined by triangulation based on the corresponding image space position. In the simplest case with a calibrated camera, the corresponding image space position differs only along the epipolar direction, and this difference is characterized by disparity. For the purpose of illustrating embodiments of the invention, the relevant equations for disparity-based calculations are described below, but the triangulation method is equally applicable to more general (uncalibrated) settings (see above).

[0151] Below, you can find the expression used to determine the coordinates. The indices used below correspond to the horizontal stereo arrangement (see below for horizontal / vertical arrangements). Depth (coordinate Z) is defined as the distance to the detected calibrated camera plane and is calculated as follows:

[0152] Z = Bf x / Parabolic distortion, (2)

[0153] Among them, f xB is the calibrated focal length of the camera along the epipolar direction, B is the distance (baseline distance) between the two cameras that produce the first image 10a and the second image 10b (in many cases: the left image and the right image), and "parallax" is the parallax corresponding to this detection.

[0154] If the principal points of the two calibrated cameras are different, then compensation must be made in the parallax:

[0155] Z = Bf x / (parallax–(c x,左 -c x,右 ))(3)

[0156] Among them (c x,左 ,c y,左 ) and (c x,右 ,c y,右 The points on the left and right cameras are the principal points. Points on the image plane represent rays in the camera coordinate system that pass through the origin. A principal point corresponds to a ray perpendicular to the camera plane and is represented as a real number in pixels (it is not guaranteed to coincide with a pixel, as it can have a non-integer value in the coordinates used for pixels).

[0157] The X and Y coordinates of the detections are calculated based on, for example, the detection coordinates of the left camera and the depth (already calculated).

[0158] X = (u 左 -c x ) / f x ·Z

[0159] Y = (v 左 -c y ) / f y ·Z (4)

[0160] Among them (c x ,c y () is the principal point of the calibrated left camera.

[0161] 3D projection is calculated for all detections and is the main output of the 3D lane detection system.

[0162] Regarding 3D projection loss, see below. The primary objective of a 3D lane detection system is to provide accurate 3D information about lane boundaries. Therefore, it is preferable to assign a loss function to 3D projection detection (see [link to documentation]). Figure 1 The 3D projection loss module 42 is used to train the neural network to predict the correct 3D location to be projected. This step is crucial for reducing noise from the neural network's 3D detection. Since the 3D projection is differentiable (and the parallax module), the loss diffuses back to the early layers of the neural network (i.e., noise reduction in 3D detection can be trained into the neural network).

[0163] For information on obtaining ground truth annotations, see below. The application of neural networks (generally, image processing modules trained via machine learning) requires a large amount of training data to calibrate the network's learnable parameters. The training data consists of stereo images labeled with descriptions of lane boundary locations in three-dimensional space, or equivalently, locations projected onto a two-dimensional camera plane. These ground truth labels are called ground truth. These can be obtained through manual annotation, that is, by determining lane boundaries on ground truth images under human supervision, possibly using some classical or neural network-based detection algorithms a priori.

[0164] Additionally, training data for neural networks suitable for 3D lane detection can be generated in a simulation environment, such as through a simulator capable of generating photorealistic images (in which lane boundaries are parameterized, for example, and thus labeled). 3D lane boundary locations can also be derived from processing LiDAR point clouds (since lane boundaries are objects projected from the road surface), classical stereo algorithms, high-density maps, or a fusion of these methods (in other words, any combination).

[0165] Below, we will discuss some aspects of training neural network models. The parameters of a neural network are optimized by using a suitable optimization method to minimize the weighted sum of all loss functions introduced into the system (and the corresponding method). Any initialization scheme can be used for the parameters before optimization.

[0166] The following provides some details of certain further embodiments of the 3D lane detection system. Several slightly different embodiments can be implemented using the architecture described above. The following embodiments may differ in how lane boundaries are represented by the neural network and in the representation of the correspondence mapping. In addition, several additional tasks may be assigned to the neural network.

[0167] The orientation of the stereo camera can be horizontal, vertical, or unrestricted. Therefore, the input images are often referred to as the first and second images. Without loss of generality, the embodiments described below assume a horizontal stereo setup, wherein the epipolar lines are selected as horizontal during correction. In this embodiment, the first and second images are the left and right images.

[0168] The embodiments related to the raw detection are described in detail below. Referring to the table below, exemplary tensors are specified for the actual data in specific stages of the system and method in each embodiment. In this embodiment, the raw detection of lane boundaries on the left and right images is performed using a 2D mesh D of scalar values ​​(or more generally, tensors). 右 / 左 The subscript indicates the image (see Tables 1b and 1c for example).

[0169] Each element (i, j) of the 2D mesh represents the image position I. i，j=(u i，j v i，j This forms a coordinate grid (i.e., the coordinate grid can also be described by a data matrix, typically by a tensor in this embodiment). Based on downsampling or upsampling of the neural network, for example through pooling layers, strided convolutions, dilated convolutions, the image coordinates (u) are... i，j v i，j (i, j) may or may not be equal to (i, j). Therefore, the coordinates at certain points on the coordinate grid are determined by the structure of the neural network, more specifically by its downsampling or upsampling factor. Furthermore, an offset can be manually introduced into the target, which also modifies (i, j) -> (u). i，j v i，j Mapping. For example, in a stride along the horizontal direction... x (stride) x and the vertical direction of the stride y (stride) y In a network (where stride gives the upsampling / downsampling factor),

[0170] u i，j =i·stride x +offset x

[0171] v i，j =j·stride y +offset y (5)

[0172] I i，j Indicates at offset x (offset) x ) = stride x / 2 and offset y =stride y In the case of / 2, the (i, j)th [stride] x ×stride y The center of the cell. Table 2a provides an example of the coordinate grid.

[0173] Original detection D 右 / 左 This describes the signed relative distances between lane boundary points on the image measured from (points) along the epipolar direction on the coordinate grid. Four independent primitive detection tensors can be defined on the left and right images (subscripts) in the left (L) and right (R) search directions (superscripts): D R 左 D R 右 D L 左 D L 右Its elements describe the relative position of the nearest lane boundary in the search direction. The neural network of the 3D lane detection system according to an embodiment of the invention can predict all four independent detection tensors, or in another embodiment, only two or three of them. For example, only D... R 左 and D L 右 That is, one for the left (first) image and one for the right (second) image; the correspondence mapping is also determined by selecting the original detection tensor to use. When targeting D R 左 and D L 右 When making a choice, the correspondence mapping P must be used. RL (See below for definitions of correspondence mappings with various directional indices). Therefore, in one embodiment, the first search direction and the second search direction are selected as the left-hand direction or the right-hand direction on the corresponding first and / or second images, respectively. This can be interpreted as any first and second images, not just the left and right images. According to this definition, all cases where two, three, or four original detection tensors are used in the computation.

[0174] (and Figure 1 The loss function of the loss (representation) modules 32 and 34 corresponding to the original detections 14a and 14b in the first and second images penalizes the difference between the predicted and actual relative positions on the ground. The simplest choice of loss function is...

[0175]

[0176] For example, when p = 2 and the same function can be applied to both the image and the direction, i.e., D pred / GT It can refer to D R 左 D R 右 D L 左 D L 右 The predicted value or GT value of any one of them. The sum of the subset F of all possible indices (i,j) is called the foreground. Applying the foreground region near the lane boundary helps the neural network focus on the area where it can correctly predict the location of the lane divider (i.e., only providing relevant information to the neural network). For example, the foreground can be defined as

[0177] F = {(i, j): |D} GT|i，j| <W}, (7)

[0178] Where W is an arbitrarily chosen window size; these points on the coordinate grid are selected for the foreground, for which the absolute value of the corresponding tensor element of the ground truth detection is less than W, i.e., a tensor element with a finite value. This means that only one region is considered by the neural network, and the nearest lane boundary is not too far from this region (less than W). The foreground region can depend on the location of the lane boundary, and for the four original detection tensors (D... R 左 D R 右 D L 左 D L 右 The values ​​may also be different, therefore there may exist an F corresponding to the original detection tensor with the same subscripts and superscripts. R 左 F R 右 F L 左 F L 右 :

[0179]

[0180] For the network to predict reliable detections, in principle, it should also predict the learned foreground output corresponding to each relative detection tensor. In one embodiment, the original detection is a pair of D and F tensors, which can be represented as a single output with multiple channels or as multiple outputs. However, a compact solution is introduced where this additional foreground output is not needed, as it can be incorporated into the correspondence mapping (see below). In this case, the original detection is simply the original detection tensor D.

[0181] Notes related to the above details: The above assumes that the original detection has only one component, which corresponds to the distance from the coordinate grid to the lane boundary along the epipolar line. The relative detection may have another component corresponding to the orthogonal direction. If the orthogonal component does not regress but is chosen to be constant, it can be incorporated into the coordinate grid offset. y In the parameters. This allows for the introduction of parameters corresponding to different offsets. y Multiple D outputs of s are used to increase the prediction density under a fixed stride. In the following sections, scalar relative detection with a single offset is considered. Below, some details of an embodiment of the correspondence mapping are given. The correspondence mapping finds a correspondence between the original detections on the left image and the original detections on the right image. There are four possible combinations of the original detection tensors of the left and right images, resulting in four different correspondences (P...). RR P RL P LL P LR), where the two superscripts indicate the search directions on the left and right images, respectively:

[0182] -P RR Description D R 左 and D R 右 Correspondence between them

[0183] -P LL Description D L 左 and D L 右 Correspondence between them

[0184] -P RL Description D R 左 and D L 右 Correspondence between them

[0185] -P LR Description D L 左 and D R 右 The correspondence between them.

[0186] This is Figure 2 The explanation will be provided in the middle. Figure 2 This is an explanation of the various correspondence mappings in this embodiment, which specifies which correspondence mappings will be used for which original detections.

[0187] In one embodiment, the correspondence mapping data block is a binary mask (paired mask) described by tensors defined on the same 2D mesh as the original detection, informing which tensor coordinates produce detections corresponding to the physically identical lane boundaries. Binary refers to the number of classifications represented by the paired mask: it describes a set of two elements, one interpreted as background and the other as foreground. For example, a convenient choice for the binary mask values ​​is {0,1}, but any other values ​​or representations (e.g., probabilities, see below) can be chosen. In this example, the mask is D... 左 and D 右 The mask takes a value of zero when pointing to different physical lane boundaries, and a value of 1 when they point to the same instance. A zero in the mask is interpreted as background, meaning no pairs were detected from these indices (in other words, the corresponding original detection tensor elements do not point to the same lane boundary). These are interpreted as foreground, since the detection pairs are predicted based on these indices.

[0188] Although in one embodiment the pairing mask takes discrete values ​​(ground truth) in the annotations, the network output can be chosen to be real values, which are interpreted as a score describing the network's determinism about whether a point is foreground in the sense of pairing prediction (this approach is illustrated in the example below). In this case, if the neural network output in the correspondence map exceeds a certain threshold, the point is accepted as foreground. In a probabilistic approach, the elements of the pairing mask are real numbers in the probability range (typically 0 to 1), which describe D 左 and D 右 The probability of pointing to the same physical lane boundary. Foreground and background classification is recovered by a threshold selected from the probability range (it is typically greater than or equal to 0.5, for example, it is chosen as 0.5 or 0.75), such that probabilities above the threshold are considered foreground, and probabilities below the threshold are considered background. This parameter allows fine-tuning of the neural network for the precision and recall function. A higher threshold is expected to give lower recall and higher precision for foreground regions. The raw probabilities before thresholding can be forwarded to the following modules, and they can be used as confidence in detection.

[0189] Learning paired masks can be viewed as a regression or classification problem. In a regression problem, the prediction is a single true value on a 2D grid, which can be regressed using simple L1, L2, or other regression losses. In a classification approach, the output is a 2D vector defined on a 2D grid, where the first component (e.g., represented in the channel dimension) describes the probability of having the background and the second component corresponds to the foreground probability. As a dependent output, the correspondence is well defined by the foreground or background probability. In many cases, a classification approach (giving a normalized score for the foreground or background at the top of the classification) is preferred, but a regression approach can also be used. This can be assigned a traditional classification loss, such as softmax loss, focus loss, etc. In one example, a classification approach with softmax loss is chosen.

[0190] The softmax loss looks like

[0191]

[0192] The sum is obtained by passing through all spatial indices of the 2D mesh, p GT,fg / bg The location is the true ground probability (usually binary) of the foreground / background predicted in pairs, while p pred,fg / bg This is the predicted value. If the predicted value equals the actual ground value, the loss is minimized.

[0193] Therefore, the original detection preferably provides the distance from the nearest lane boundary point (along the epipolar direction) to the corresponding coordinate grid point. A value is given in each tensor element of the original detection tensor because the foreground is not specified using window size, but rather through a correspondence mapping tensor. In this case, the tensor is a binary mask describing which original detection values ​​at the same point on the coordinate grid correspond to the same lane boundary.

[0194] Tables 1a-1c provide examples for paired masks and the original detection tensor.

[0195] Based on the above, in this embodiment, the correspondence mapping tensor is a paired correspondence mapping tensor (this is a paired mask), with a paired probability value in each of its third tensor elements (the probability values ​​below in the correspondence mapping data are paired probability values). The paired correspondence mapping tensor represents the probability that the nearest first lane boundary point and the nearest second lane boundary point correspond to the same lane boundary (i.e., the lane boundaries on the first and second images (whose points are under investigation) correspond to the same lane boundary; in other words, determining the same lane boundary based on the images) can be observed in the first and second images and are located at a distance from the grid corresponding to the corresponding third tensor element. (Distance values ​​are also measured from this) Within the window size of the tensor elements, the third tensor element whose corresponding probability value is higher than a predetermined first threshold is selected as the foreground third tensor element (see details of foreground designation in each embodiment; embodiments with binary pairing masks are also considered such embodiments, see the next section). In the pairing step (in the pairing module), which foreground third tensor element and the corresponding first and second tensor elements are used as output data in the image processing step (in the image processing module) is maintained for further computation (see also the example illustrated in the table; the foreground is selected for further computation, and other values ​​of the tensor are not taken into consideration).

[0196] The optimal window size depends on the resolution of the input image and is typically between 1 pixel and half the image width (in pixels). Preferably, it is chosen to be less than half the typical lane width (the lateral distance between adjacent lane boundaries) seen at the bottom of the image, depending on camera calibration.

[0197] Therefore, if the same boundary line can be found in both the first and second images within a window size from the corresponding coordinate grid, then the aforementioned pairing probability value is therefore high. Thus, the given distance value in the original detection tensor is maximized by the window size (see also expressions (7) and (8) above).

[0198] Examples of paired correspondence mapping tensors are shown in Tables 1a and 4a-4c (in the examples, these are simply labeled "correspondence mapping"). The sole purpose of the "paired" attribute in the name is to identify the method; tensors can also be named in different ways.

[0199] The RR, LL, and RL types of the foreground region of the paired mask described above have a transparent visual interpretation, and their combination Figures 3A-3C To explain. In Figures 3A-3C In the middle, left, and right images, the lanes can be observed in the same attached figure. This type of attached figure is derived from the unification of these images. The foreground regions of the paired masks are specified in the unified attached figure. The two images giving the starting point (the first and second images) have the same size, i.e., frames of the same size correspond to them. In this frame, both images can be displayed, especially the lane boundaries, and because the images are loaded into the same frame, the content of the original images is slightly offset from each other. The lane boundaries can be described using different parameters in the first and second images. According to the invention, not only can the coordinates of the lane boundaries be derived in two dimensions of the image, but also the depth coordinates can be determined.

[0200] The correspondence mapping can also be interpreted within the frames defined above. It preferably provides the corresponding value for each grid point based on its definition, taking into account, for example, the search direction and window as described in the embodiments. In the stereo instance segmentation embodiment, correspondence information is defined using labels; other ways of representing correspondence information are given below.

[0201] Therefore, for example, in Figure 3B In the example, it will be closer to the lane boundary (continuous line) on the left image than the lane boundary (dashed line) on the right image.

[0202] Figure 3A The foreground regions 100a and 100b (each for a pair of lane boundaries from the first and second images) corresponding to the RR-type paired mask are shown to be regions on a 2D grid from which the lane boundary closest to the right in both the left and right images corresponds to the same real number, 3D lane boundary (the center of the lane boundary is shown in the figure as the lane boundary projection, i.e., the projection of the 3D lane boundary onto the first and second images, see below).

[0203] Therefore, in Figure 3AIn the first image, lane boundary projections 102a and 104a (shown as dashed lines; in the case of lane boundaries, the lines on the graph preferably correspond to the center lines of lane boundary markers of finite widths, and in the visualization, whether the original lane boundary is a dashed line (e.g., with long intervals), continuous, or of another type is irrelevant) correspond to the lane boundaries on the right (second) image, and lane edge projections 102b and 104b (shown as continuous lines; other lane boundary projections 102a and 104a are also continuous, but shown as dashed lines to distinguish them from continuous illustrations) correspond to the lane boundaries on the left (first) image. From these lines, lane boundary projections 102a and 102b correspond to the same lane boundary (dashed lines and continuity do not correspond to the type of lane boundary, but these visualizations represent the same lane boundary); both lane boundary projections 102a and 102b are located to the right of the foreground region 100a (as are lane boundary projections 104a and 104b and the foreground region 100b).

[0204] exist Figure 3A In the image, distances 106a and 106b, as well as distances 108a and 108b, can also be observed. These distances 106a, 106b, 108a, and 108b point from points 100a and 100b in the foreground regions; this starting point corresponds to a point on the coordinate grid. The endpoints of the distances are located on the observable lane boundaries in the left and right images, respectively. Therefore, distance 106a gives D R 右 The original detection value, and the distance of 106 bits gives D. R 左 Value. For example... Figure 3A As shown, when more lane boundaries are visible in the camera, more foreground areas and greater distances can be specified for the paired masks. The distance between the boundaries of foreground areas 100a and 100b and the projections of the two closer lane boundaries must be less than the grid resolution (i.e., the distance between grid points). However, in Figures 3A-3C In the diagram, for illustrative purposes, the gap between the lane boundary projection and the foreground area is shown. Note that at the top of the maximum distance, the minimum distance can also be defined.

[0205] exist Figure 3B Other types of foreground regions 110a and 110b can be observed. In this case, an LL-type mask is explained, where a region can be specified from which the nearest lane boundary on the left corresponds to the same physical lane boundary on both the left and right images. In this case, lane boundary projections 112a and 114a (marked with dashed lines) correspond to the lane boundaries on the right image, and their paired lane boundary projections 112b and 114b (continuous lines) are observable in the left image. These are related to... Figure 3AThe lane boundaries are different lane boundaries, and these pairs are clearly located to the left of the foreground regions 110a, 110b of the paired mask. As explained by multiple foreground regions 110a, 110b, only the lane boundaries corresponding to a pair must be located to the left of the respective foreground regions 110a, 110b. Other reference numerals in this figure are similar. Figure 3A That is, distances 116a and 116b point from points on the coordinate grid to a pair of corresponding lanes. Distances 116a and 118a correspond to D. L 右 The distances of 116b and 118b correspond to D. L 左 .

[0206] exist Figure 3C The third type of foreground regions 120a and 120b of the paired mask can be observed. The foreground regions 120a and 120b of the RL-type paired mask are regions corresponding to the closest lane boundary on the right side of the left image and the closest lane boundary on the left side of the right image. This region appears as a slice (a set of slices) bounded by lanes in the left and right images. Depending on the search direction, the distances from 126a and 126b (D...) L 右 and D R 左 ) and distances of 128a and 128b at Figure 3C The middle finger points in opposite directions. For example... Figure 3C As shown, foreground regions 120a and 120b are designated between corresponding lane boundary pairs. Therefore, foreground region 120a lies between lane boundary projection 122a (dashed line, lane boundary in the right image) and lane boundary projection 122b (continuous line, the same lane boundary in the left image), and foreground region 120b lies between lane boundary projection 124a (dashed line, another lane boundary in the right image) and lane boundary projection 124b (continuous line, another lane boundary pair in the left image).

[0207] The foreground region structure of an LR-type mask is similar to that of an RL-type mask. An LR-type mask is the region corresponding to the nearest lane boundary on the left side of the left image and the nearest lane boundary on the right side of the right image. If the camera's principal point is equal to or equal to c... x,左 -c x,右 If ≥0, then this is a null mask. This is related to c. x,左 ≥c x,右 The parallax is the same; in this case, the parallax is equal to or greater than zero, meaning that in the right camera, all objects are more to the left compared to the left camera. Therefore, there is no LR region, or more precisely, it is an empty mask.

[0208] Additionally, a maximum distance limit for either lane can be added to the pairing mask to predict pairings only from areas close to the lanes, rather than from areas far from lane markings (e.g., vegetation, buildings, etc.). If the neural network does not predict the foreground region of the original detection, this maximum distance should be chosen based on the window size applied in training the original detection.

[0209] The foreground regions of RR, LL, and RL type masks are illustrated not only in Figure 3 but also in Figure 4. Figures 3 and 4 show case c. x,左 =c x,右 This is the standard choice during calibration. Increase c. x,左 The value of shifts the projection of the lane boundary on the left image to the right. Therefore, the distance between the left and right projections of the lane boundary increases; for example, the RL-type foreground mask becomes wider. The minimum disparity seen on the image is greater than or equal to c. x,左 -c x,右 Decrease c x,左 The value of shifts the projection of the lane boundary on the left image to the right. Therefore, the distance between the left and right projections of the lane boundary decreases; for example, the RL-type foreground mask becomes narrower. Additionally, in the far region, the right projection of the lane boundary lies to the right of the left projection of the same lane boundary (if c...). x,左 ≥c x,右 (If this doesn't happen, then an LR-type foreground mask will appear.) Similarly, increasing or decreasing c... x,右 The value of will shift the projection of the lane boundary on the right image to the right or left, respectively.

[0210] Figures 4A-4F The following is given in c x,左 =c x,右 Explanation of the foreground region of the RL, RR, LL type paired masks (highlighted by white stripes) in the actual image. In a single line, the same region is highlighted on both the left and right images, corresponding to the foreground region of the paired mask. It can be observed that the position of the paired mask is absolute, i.e., identical in both photographs, while the content of the photographs differs depending on whether they were taken by the left or right camera (e.g., see the vehicles in the next lane and the lane boundary portion observable in front of the photographing vehicle). Figure 4A and 4B In this context, the foreground of the RL mask lies between the left and right projections of the same 3D lane boundary line (the left and right projections refer to the visualization of the left lane boundary, while the right image is a projection of the same real 3D boundary marker). Therefore, the shape of the paired mask is constrained by the lane boundary markers to its left and right (see [link to image]). Figure 4C-4F ).exist Figure 4A and 4BIn the RL mask, the foreground regions 202a and 202b are the regions corresponding to lane markings 204 and 206 (in... Figure 4A and 4B There aren't many lane markings visible. For the same lane boundary markings 204 and 206, please also refer to [link / reference needed]. Figure 4C-4F Lane markings 204 and 206 mark the boundaries of the lane the camera vehicle is traveling in. Other foreground areas of the RL mask, such as 200... Figure 4A and 4B As shown, the explanation preferably identifies all lane boundaries located in front of the camera vehicle (and also identifies the obscured portion of the foreground 200) by means of the method according to the invention.

[0211] The foreground regions 200a, 200b, and 200 of the paired mask are in Figure 4A and 4B China and Belgium in Figure 4C-4F The foreground width is narrower, especially at a distance from the car being imaged (see its reference numerals below). This is because parallax limits the width of the foreground area in the RL mask. However, the width of the foreground area in the RR and LL masks is limited by the distance to the lane boundary. Furthermore, the width of all foreground areas is also limited by the window size.

[0212] As another result, the paired mask is very close to the lane boundary marker. In this example, the foreground region is precisely the area between the two projections of the lane boundary marker (from the left and right images, this interpretation and other interpretations can also be applied to the first and second images in general), that is, the foreground region is roughly bounded by the centerline of the lane boundary marker (for a better explanation, see...). Figure 4B Lane boundary marker 206 closest to the car). "Generally limited" means the foreground area starts from the grid point closest to the center line of the lane boundary marker, therefore it cannot be observed near the center line (from this perspective, Figures 3A-3C Use a schematic diagram to distinguish between the foreground area and the lane boundary projection.

[0213] exist Figure 4C and 4D In the foreground, it can be observed that the foreground regions 208a, 208b and 208, 210a, 210b and 210b of the RR mask (the method for foreground regions 208a, 208b, 210a and 210b is the same as that for foreground regions 200a and 200b mentioned above; the areas labeled "a" and "b" in the attached diagram are the areas surrounding the camera vehicle) are located to the left of the corresponding left and right lane boundary markings 204 and 206 (most of which are observable for the center lane occupied by the camera vehicle). Depending on the camera's perspective corresponding to the left and right images, the foreground of the paired mask is closer to the lane boundary in the right image than in the left image (see [link to image]). Figure 3A ).

[0214] exist Figure 4E and 4F In the image, the foreground regions 212a, 212b and 212, 214a, 214b and 214 of the LL mask are located to the right of both the left and right projections. The foreground of the paired mask is closer to the lane boundary in the left image than in the right image.

[0215] As an example, Figure 4G It shows the relationship with Figures 4A-4F The results given correspond to the left image space detection averaged from the three foreground regions RL, RR, and LL. Figure 4G It can be observed that the good detection (dense and close to the center line of the lane marking) 220a and 220b are for lanes occupied by camera vehicles; the detection every other lane boundary 220 also yields good results, and occlusion is also resolved.

[0216] exist Figure 5A and 5B In the diagram, corresponding to the functions Y(Z) (side view) and X(Z) (top view), the graphs show the flow from... Figure 4A-4G The above explanation shows the 3D lane detection data obtained. Figure 5A and 5B The results clearly show that good quality results can be obtained through the corresponding embodiments of the present invention. Figure 5A The results shown illustrate the slowly ascending path in the camera coordinate system due to the camera's pitch angle, while... Figure 5B The text describes roughly parallel lanes. The lane boundaries corresponding to lane markings 204 and 206 are... Figure 5B The point is located at an interval X in the range [-4 meters, 4 meters]. The optical axis of the camera is preferably not parallel to the road surface (which is fixed by using the desired coverage of the camera's environment). Therefore, the present invention is preferably suitable for online calibration of the camera's pitch and roll angles. Thus, in the present invention, the pitch and roll motion of the vehicle can be compensated.

[0217] exist Figure 5A and 5B In this process, due to perspective distortion, the detection is organized in an approximate quantization pattern (with increasingly larger gaps between detections at greater distances).

[0218] The good quality of the results is even more evident. Figure 6A and 6BThe support is based on the results of 3D projections from SGM for Y(Z) and X(Z) (semi-global matching, a widely used stereo matching algorithm, Hirschmuller, Heiko) "Accurate and efficient stereo processing by semi-global matching and mutual information", IEEE Conference on Computer Vision and Pattern Recognition (CVPR'05), 2005. For a fair comparison, [the following is used]. Figure 4G The left image space detection, only the disparity estimated according to the present invention is replaced by the result of the SGM algorithm in equation (3-4) to evaluate the 3D position of the lane boundary. Figure 6A and 6B The data shows a large amount of noise (with many outliers), i.e. Figure 5A and 5B The results are much better. Using SGM parallax projects lane boundaries in the occluded area to incorrect locations in the camera coordinate system, and this is consistent with the present invention (…). Figure 5B Compared to the 60-80m range in the middle, the effective detection range is much shorter. Figure 6B (20-40m in the middle).

[0219] Since the hidden portion of the lane boundary can be resolved by the method and system according to the invention, it is also explained that the lane boundary running behind the vehicle corresponds to the next lane and is hidden by the vehicle while it is in the hidden position.

[0220] Occlusion resolution (lane boundary positioning, e.g., behind a car) is based on ground reality and a neural network learning process (generally, an image processing module trained via machine learning). By identifying occlusion, such ground reality can be obtained from multiple sources: maps, previous or next frames without occlusion, simulated images, etc. (in ground reality, foreground areas are also displayed behind the car).

[0221] In another embodiment, the correspondence mapping is a mask of discrete values ​​obtained from a set of size C, where the indices of C are k = 0...C-1. For example, where D... 左 and D 右 At indices pointing to different physical lane boundaries, the mask is treated as background represented by, for example, k=0, if D 左 and D 右If both point to the same lane boundary with an attribute indexed by k, then the value of k is taken (k = 1...C-1). For example, this attribute might be the lane instance label, or it might distinguish between dashed and solid lines, etc. Non-background indices are treated as foreground indices. The correspondence mapping output of the neural network is a 2D grid of C-dimensional vectors, meaning the output has C components (channels) at each position in the 2D grid. The first channel can reference the background probability, while the other C-1 channels can describe the probabilities of foreground regions with attributes indexed by k = 1...C-1.

[0222] If the lane attributes represented by the paired mask can be formulated as a classification problem, such as in lane type classification, which describes the dashes and / or colors of lane dividers, then training can be driven by a softmax loss of class C, similar to equation (9).

[0223]

[0224] If the lane attributes represented by the paired masks are lane instance labels, then the loss function can be one of the functions used in instance segmentation tasks. The difference between classification and instance segmentation is that, in instance segmentation, the arrangement of ground truth labels may be considered a similarly good output (the labels are equivalent), which should be obeyed by the loss function. A more detailed description of instance segmentation is given below.

[0225] The details of the pairing module in this embodiment are given below. The detected image positions in the original image depend on the coordinate grid and the original detection tensor D (typically, this is a data block, but in this embodiment, it is a tensor); the original detection data preferably includes values ​​available within the frame of the original image that are unaffected by downsampling / upsampling of the neural network; if scaling is also applied in the prediction, the neural network can also be implemented, as it is a product of constants, and this detail related to the scaling method does not lead to any efficiency-related differences. As discussed earlier, only detections from the foreground region are accepted. The pairing module has two tasks: generating image spatial representations of the left and right detections and pairing them according to the correspondence mapping.

[0226] In this embodiment, not only are they paired, but the foreground region is represented by a correspondence map P (in another embodiment, the foreground region is part of the original detection tensor).

[0227] Image spatial detection is computed by adding the original detection tensor D to the horizontal component of the coordinate grid. (Alternatively, if the original detection tensor also has a vertical component, it should also be added to the vertical component of the coordinate grid.)

[0228] Detection i,j =(u i,j +D i,j ,v i,j(10)

[0229] Therefore, detection i,j The first component is equal to u i,j +D i,j The second component is v i,j The original detection section provides a coordinate grid (u). i,j ,v i,j The definition of ) is given in Table 2, which provides an example of a mesh.

[0230] The image spatial detection tensor can be computed for all indices, but only detections from the foreground region will be accepted. Alternatively, the image spatial detection can be computed only at the indices of the foreground region corresponding to the paired mask.

[0231] The left and right image space detections from the same foreground index (i,j) of the paired mask are paired and can be further processed by the parallax module and the 3D projection module.

[0232] If the neural network generates multiple correspondence mappings of types such as RR and RL, the pairing module outputs image-space detection pairs corresponding to all types of RR, RL, etc. Optionally, if image-space detections from different pairing masks represent the same 3D lane boundary points, they can be averaged in the pairing module or forwarded as independent detections.

[0233] In this embodiment, the representation loss introduced for the original detection is equivalent to the image spatial loss, and it is not necessary to introduce additional image spatial loss at the pairing module. Alternatively, only the image spatial loss in the pairing module can be used, and the representation loss for the original detection can be turned off. These two approaches differ slightly in how foreground regions are handled in the loss calculation. In the image spatial loss introduced in the pairing module, a natural choice is to sum the indices considered as foreground in the pairing mask. In the representation loss introduced for the original detection, the foreground depends only on lane boundaries on a single image, so the loss might be assigned to regions that ultimately are not used for paired prediction.

[0234] The details of the disparity module described below are a continuation of the embodiments introduced above. The disparity tensor is calculated as the difference between the horizontal components of the left and right image space detection tensors.

[0235] Parallax i,j =u 左检测,i,j -u 右检测,i,j (11)

[0236] Note that in this embodiment, the original detection tensor D is directly subtracted. 左 -D 右To obtain the same result, calculate the disparity of all detection pairs from various types of paired masks, and obtain the resulting disparity value by averaging or by processing the disparity value individually.

[0237] For the original detection, a similar loss function can be used. The disparity loss is assigned only to the foreground index of the relevant paired mask.

[0238] In the 3D projection module, 3D projection is performed on all the above detection pairs according to equation (3-4) above.

[0239] For example, the Huber loss function can be used for 3D projection; however, any suitable loss function can be applied.

[0240] L 3D =∑f(|r pred -r GT |), (12)

[0241] Where r (bold "r") is a 3D vector pointing to the predicted or actual location of the lane boundary.

[0242]

[0243] x0 is an arbitrary threshold, and the summation is performed using the detections corresponding to the ground truth paired masks. The 3D projection loss is assigned only to the foreground index of the relevant paired mask.

[0244] The table below illustrates the tensors and computations occurring in the various modules of the system according to the invention, for a highly simplified example. For simplicity, only the single left and right detection tensor D is shown in the table below. R 左 and D L 右 Provide the values ​​for these tensors from P RL Provide the corresponding mapping. The coordinate grid in this example is composed of stride. x = stride y =8, offset x =Offset y =4 is defined. Considering Table 2a below, it can be observed that these parameters mean that the x or y coordinate jumps by 8 units between adjacent grid points. Based on the offset value, the coordinate grid starts at (4,4) instead of (0,0); this is a more realistic choice for the starting point. For the principal point, the camera parameter is c. x =30, c y =3, for focal length, f=200, for distance between two cameras, B=1.

[0245] Tables 1a-1c explain the raw outputs of the neural network, namely the correspondence mapping and the raw detection tensors of the left and right images. This information is the output of the trained neural network. Next, Table 2 (more precisely, Tables 2a, 2b, and 2c) shows the tensors and computations that occur in the pairing module, while Table 3 (Tables 3a, 3b, and 3c) shows the outputs of the disparity module and the 3D projection module.

[0246]

[0247] Table 1a

[0248]

[0249] Table 1b

[0250]

[0251] Table 1c

[0252] The raw outputs of the neural network are given in Tables 1a-1c above. Table 1a shows the correspondence mapping data blocks in tensor (more specifically, matrix) form in this embodiment. All data blocks in this embodiment are in tensor (matrix) form.

[0253] The correspondence mapping tensors given in Table 1a correspond to the foreground channels of RL-type paired masks. The background channels are undecoded because they do not contain additional information (the foreground and background probabilities sum to 1). RL masks describe D... R 左 and D L 右 The correspondence between them is thus established, and therefore, these original detections are presented in Tables 1b and 1c.

[0254] In Table 1a, foreground regions of paired masks whose outputs exceed a threshold (e.g., 0.5) are highlighted in bold. In this case, corresponding to the R and L search directions, the original detection of the left image takes a positive value, while the original detection of the right image takes a negative value (see Table 1a). Figure 3C It shows D pointing to the right from the foreground area. R 左 Therefore, D has a positive value, while D has the opposite direction. L 右 These distances have negative values; correspondingly, they are signed.

[0255] In Tables 1b and 1c, the lowercase 'x' represents values ​​predicted by the network but ignored due to invalid correspondences. Only those values ​​exceeding the threshold corresponding to the probability values ​​in Table 1a are preserved (see other sections of this description for more details on the predetermined threshold). In other words, the original left or right image detection described above only shows those values ​​selected as foreground, i.e., not the "original" original detection (which is preferably the output of the image processing module), but those values ​​that have already been filtered by the correspondence mapping.

[0256] Tables 1b and 1c show that, in this example, three pairs are still retained after the filtering. In the original output, downsampling / upsampling is naturally applied compared to the original left and right images. Therefore, in this example, lane boundaries are detected based on the valid values ​​in Tables 1b and 1c.

[0257]

[0258] Table 2a

[0259]

[0260] Table 2b

[0261]

[0262] Table 2c

[0263] Tables 2a-2c illustrate the calculations performed in the pairing module on the inputs obtained from Tables 1a-1c (stride length of 8 in both directions and offset of 4). The coordinate grid is calculated based on Equation (5); these values ​​indicate the applied stride length and offset. Detection pairs predicted according to the foreground index are indicated in bold (the same positions as in the correspondence mapping in Table 1a). The pairing module calculates the image positions of the detection pairs by adding the original detection tensors of the left and right images from Tables 1b and 1c to the coordinate grid according to Equation (10) (correspondingly, only the x-coordinate of the coordinate grid is changed to obtain image spatial detection). Depending on the embodiment, the image positions of the detections may be calculated only for the foreground region (as in this example) or anywhere else, but only valid pairs will contribute to the final output.

[0264]

[0265] Table 3a

[0266] As explained in Table 3a, according to equation (11), the disparity module calculates the horizontal distance between the left and right detections by subtracting two corresponding tensors from Tables 2b and 2c. In this embodiment, the disparity value can also be calculated based on the values ​​in Tables 1b and 1c. The disparity values ​​in Table 3a show that, since the images taken from the vehicle are natural, these points are closest to the lower part of the image (i.e., the bottom of the table), while lane boundaries tend to be further away from the upper part of the image (according to the minimum disparity 2.1).

[0267]

[0268]

[0269] Table 3b

[0270] As explained in Table 3b, according to equation (3-4), the 3D projection module maps the detection pair from the detected left image coordinates and disparity to the 3D space of the camera coordinate system, where c x =30, c y =3, f=200, B=1. Lowercase x represents the value predicted by the network, but is ignored due to invalid correspondence. The values ​​in Table 3b reinforce what is written as being related to the disparity values ​​in Table 3a. The Z coordinate (depth) increases from the point closest to the bottom of the table (i.e., from 12.4 to 95.2).

[0271] Below, a similar example of merging lane type segments into a paired mask is described. In this simple example, solid and dashed lanes are distinguished, but similarly, more types can be included (e.g., dashed dividing lines with different frequencies, lane marking colors, roadside lanes, etc.). For roadside lanes, see above, where some aspects of lane boundaries for this special class are discussed.

[0272] In this example, we only consider RR-type paired masks, but it applies to all types of paired masks. The calculations in Tables 6-7 (Tables 6a-6c, 7a-7c) are the same as those in Tables 2-3 (Tables 2a-2c, 3a-3b), but also provide detection information for lane type. Unlike the previous example, a slightly larger window size is applied here (the window size may depend on distance, for example, decreasing at larger distances, but a fixed window size is sufficient for the entire image) to allow for a wider foreground region and to predict the location of the same lane boundary point from multiple adjacent indices of the 2D grid. These detections may or may not be averaged in the post-processing steps.

[0273]

[0274] Table 4a

[0275]

[0276]

[0277] Table 4b

[0278]

[0279] Table 4c

[0280]

[0281] Table 4d

[0282] Tables 4a-4d present the raw outputs of the neural network. In the outputs, there exists an RR-type paired mask as a mapping to multiple foreground channels corresponding to lane type classifications (i.e., background, solid line, dashed line: from channel 0 to channel 2). The foreground regions of the paired masks, where the background value is less than a threshold (e.g., 0.5), are highlighted in bold in Table 4a.

[0283] The preferred approach is to introduce a correspondence mapping for the background, since both channel 1 and channel 2 correspondence mappings may contain low-probability values ​​that do not constitute background. In an image showing solid (continuous) and dashed lane boundaries, the background can be separated based on the channel 0 correspondence mapping, and the remainder is either solid or dashed. The contents of Table 4d can be derived from Tables 4a-4c, which mark which cells of the tensor (matrix) correspond to solid and dashed lines.

[0284] In an alternative approach, the background can be separated by considering the correspondence mapping between channel 1 and channel 2, separating solid lines from channel 1 and dashed lines from channel 2 based on probability values, and the remaining part will be assigned as the background.

[0285] In the embodiments described in Tables 4a-4d, 5a-5b, 6a-6c, and 7a-7c, the correspondence mapping tensor (since there is typically at least one such tensor) comprises at least two lane boundary type selective correspondence mapping tensors (the tensor contained in the other tensor may be referred to as a subtensor) for at least two different lane types respectively. The pairing probability value is a lane boundary type selective probability value that further characterizes the probability that a lane boundary belongs to a certain lane boundary type, and the system is adapted to generate corresponding 3D lane detection data blocks based on the at least two lane boundary type selective correspondence mapping tensors for at least two different lane types. As shown in the table below, lane boundary type selective 3D projection can be obtained in this embodiment.

[0286] RR mask describes D R 左 and D R 右The correspondence between them is explained in Tables 5a and 5b. The detected lane type is determined by comparing the values ​​of channel 1 and channel 2 at the foreground index. Alternatively, lane type probabilities can be assigned to the detectors.

[0287]

[0288] Table 5a

[0289]

[0290] Table 5b

[0291] The further raw outputs of the neural network are explained in Tables 5a and 5b (the values ​​of the solid and dashed lines are given in the same tensor (matrix) because the lane type is given in Table 4d and can also be extracted in other ways). RR mask description D R 左 and D R 右 The correspondence between them. In this case, corresponding to the search direction, both the original detection of the left image and the original detection of the right image take positive values ​​(see...). Figure 3A The lowercase 'x' represents the value predicted by the network, but is ignored due to invalid correspondence.

[0292]

[0293] Table 6a

[0294]

[0295] Table 6b

[0296]

[0297] Table 6c

[0298] Tables 6a-6c explain the calculations performed in the pairing module for the inputs given in Tables 4a-4d and 5a-5b. The calculations for the values ​​in Tables 6a-6c are similar to those in Tables 2a-2c; therefore, the coordinate grid is calculated based on Equation (5). Detection pairs are predicted based on the foreground index indicated by bold letters. According to Equation (10), the pairing module calculates the image positions of the detection pairs by adding the original detection tensors of the left and right images from Tables 5a-5b to the coordinate grid. Depending on the embodiment, the image positions of the detections may be calculated only for the foreground region (as given in this example) or anywhere else, but only valid pairs will contribute to the final output.

[0299]

[0300]

[0301] Table 7a

[0302]

[0303] Table 7b

[0304] Channel type 3D Inspection Reality (-5.4,0.5,95.2) Reality (-5.6,0.5,100.0) Reality (-2.7,1.4,30.8) virtual (0.8,1.4,30.3) virtual (1.2,1.4,16.7) virtual (0.9,1.4,11.4) virtual (0.9,1.4,11.1)

[0305] Table 7c

[0306] According to equation (11), the disparity module calculates the horizontal distance between the left and right detections by subtracting two corresponding tensors from tables 6b and 6c.

[0307] According to equation (3-4), the 3D projection module maps the detection pair from the detected left image coordinates and disparity to the 3D space of the camera coordinate system, where c x =30, c y =3, f=200, B=1. Lowercase x represents the value predicted by the network, but is ignored due to invalid correspondences. Table 7c lists the valid 3D detections with the corresponding lane types.

[0308] As observed in Table 7b, the results in the same original data correspond to the same 3D points. This is a result of the fact that a wider window has already been considered in the current calculations. Only one of the equivalent results must be taken into account (or these can be averaged); therefore, the calculations show that considering a wider window is not disadvantageous.

[0309] Note that, similar to Tables 6b and 6c, the data is not separated based on the lane type in Tables 7a and 7b. For example, this could be done using Table 4d, and the final results also given in Table 7b can be provided using the lane boundary type selection method in Table 7c.

[0310] The above describes some compact implementations of using paired masks as correspondence mappings.

[0311] Four different pairing masks have been listed, and any subset of these can be used in a 3D lane detection system with the corresponding original detection. Various types of original detection and pairing masks can be implemented as different outputs of a neural network, or as different channels of unique left and right image original detection and corresponding mapping outputs.

[0312] A simple equivalent implementation is that the original detection tensor gives the unsigned relative distance from the coordinate grid to the nearest lane boundary along the search direction. Then, the pairing module is modified accordingly to predict the image spatial detection (transforming the original detection D...). R 左 / 右 Add to the coordinate grid and subtract the original detection D. L 左 / 右 ).

[0313] A search direction is defined for the original detection tensor. Conversely, in another embodiment, D can be... 左 / 右 Defined as the relative signed distance from the coordinate grid to the nearest lane boundary projection on the left or right. Then, slightly different pairing masks should be constructed to pair the relevant detections.

[0314] In the following section, another embodiment is described, which utilizes the stereo instance segmentation method developed by the inventors for this invention. In this embodiment, the correspondence mapping is not a single 2D grid of the left and right images (which may also have more channels), but rather two 2D grids corresponding to the first and second images respectively (see, for example, Tables 8a and 8b below).

[0315] Known instance segmentation techniques (see, for example, Davy Neven et al., Towards End-to-End LaneDetection: An Instance Segmentation Approach, 2018, arXiv: 1802.05591; Yen ChangHsu et al., Learning to Cluster for Proposal-Free Instance Segmentation, 2018, arXiv: 1803.06459) aim to find objects in an image by labeling all pixels of the image, such that pixels corresponding to the same object receive the same label and pixels corresponding to different objects receive different labels.

[0316] Using some unconventional modifications (including the application of the two-channel correspondence mapping framework detailed below), the instance segmentation technique has been extended to stereo instance segmentation to be applicable to image pairs in the correspondence mapping framework of the 3D lane detection system in this embodiment. In the stereo instance segmentation method applied in this embodiment, pixels that correspond to the same physical instance receive the same label, and pixels that do not come from the same object receive different labels (in this embodiment, the neural network is trained to index the first and second images in this manner), regardless of which image the pixel comes from (therefore, labels are applied to the images in a uniform and consistent manner). In other words, stereo instance segmentation is consistent labeling of two images (consistent labeling refers to the consistent use of the same label in both channels of the correspondence mapping tensor). In summary, in this embodiment, during the learning process of the neural network, it has been taught to output these consistent labels (i.e., to perform consistent labeling).

[0317] In the known instance segmentation methods described above, model fitting is performed based on the results of the instance segmentation process. Furthermore, according to other aspects, the results to be displayed are obtained in a manner drastically different from the stereo instance segmentation applied in the embodiments of this invention.

[0318] In the case of the 3D lane detection method and system according to the present invention, the object (to be labeled) can be the area around the projected lane boundary line. The width of this area is a parameter that can be chosen as a constant, or it can depend on the properties of the lane boundary (distance, type, etc.). Areas that do not correspond to any lane boundary can obtain unique background labels.

[0319] These regions can be constructed from known locations of the lanes during training. Using an appropriate loss function, the neural network learns to find these regions and assign labels consistent with the ground truth data. Consistency means that the arrangement of the ground truth labels can be accepted as a perfect prediction (since the values ​​of the labels are not special, any value can be chosen as the label). During inference (i.e., when using 3D lane detection methods and systems), a prediction mask is used to obtain detection pairs on the first (left) and second (right) images.

[0320] This type of correspondence mapping is highly flexible, allowing it to be used for paired detection regardless of how they are predicted by a neural network, such as through classification, dense regression, or by predicting model parameters like splines. This flexibility stems from the fact that stereo instance segmentation defines the correspondence between detections in the image space, rather than between the original detections. Therefore, the pairing module first computes the image locations corresponding to the original detections and selects detection pairs through stereo instance segmentation, independent of the representations used in the original detections. Some original detections may not be paired by stereo instance segmentation, and these are then ignored (or discarded) by the pairing module.

[0321] In one embodiment, stereo instance segmentation is used as the foreground region (i.e., the foreground region is specified based on the result of stereo instance segmentation), from which the precise location of the lane boundary is predicted, just as in the previous example with a single paired mask. In other words, a correspondence mapping between two images is used to determine the location (instead of a single paired mask as described above, see also the examples given in Tables 8-10 below). Detection cannot be predicted from the background. Original detection output D 左 and D 右 It can be the signed distance from the lane boundary along the epipolar direction to the foreground position of the stereo instance segmentation on the left and right images, respectively.

[0322] An embodiment based on the stereo instance segmentation introduced within the framework of this invention can typically be constructed, i.e., based on correspondence mapping data not limited to (individual) correspondence mapping data blocks (however, this embodiment is explained by means of correspondence mapping data blocks).

[0323] Therefore, in summary, the correspondence mapping data includes first-channel correspondence mapping data elements corresponding to the first image (generally, these can be data blocks, tensors, or any general form of data similar to correspondence mapping data) and second-channel correspondence mapping data elements corresponding to the second image (i.e., for both images, separate data elements—which can be considered as channels of the main correspondence mapping data, as shown in the examples in Tables 8a-8d, 9a-9d, and 10a-10b). Each third tensor element of the first-channel and second-channel correspondence mapping tensors is a label that characterizes whether the corresponding third tensor element corresponds to a lane boundary (if not, the value is zero; and when a lane boundary is found, the value is—as shown in Tables 8a and 8b below—a certain value). Labels (e.g., "1" and "2") are used, where the same label corresponds to the same lane boundary, and different labels correspond to different lane boundaries in the first and second channel correspondence mapping data elements, respectively. Data elements of the first and second channel correspondence mapping data elements corresponding to the lane boundaries are selected as foreground correspondence mapping data elements. Which foreground correspondence mapping data elements are used as output data in the pairing step (in the pairing module) and thus maintained for further calculation (as shown in the table below, the foreground of each starting image may be different, but this fact does not cause computational problems). If raw detection data is used, the corresponding raw detection data can also be forwarded to the pairing step (see also...). Figure 1 The methods and examples described in Tables 8a-8d).

[0324] In this context, the correspondence mapping data is a label indicating whether an element of the first and second channel correspondence mapping data corresponds to a lane boundary; in other words, whether it is a lane boundary point or a background point. Note that the correspondence mapping data is initially required to determine the correspondence between the first lane boundary group of the first image and the second lane boundary group of the second image. Therefore, the label of each channel correspondence mapping data will show its correspondence with the other channels (for a particular lane boundary that is observable in both images, the label will be the same). Examples of channel correspondence mapping data are given in Tables 8a and 8b; these examples are in tensor form, which is a possible representation of channel correspondence mapping data.

[0325] As explained in Tables 8a-8d below, the instance segmentation method based on the general definition given in the preceding paragraphs can be used in this embodiment based on the original detection tensor, coordinate grid tensor, etc. However, as also given below, the instance segmentation method can be combined with model-based embodiments.

[0326] In one embodiment, the various labels can be represented by orthogonal vectors, similar to the embodiment with paired masks representing lane attributes, as described above. That is, the correspondence mapping tensor elements are vectors whose components describe the probabilities corresponding to various labels. In Tables 8a and 8b, the labels corresponding to the highest probabilities are given as correspondence mappings.

[0327] The names of the first and second channel correspondence mapping data elements (e.g., (sub)tensors) are intended to emphasize that these elements constitute channels of the original tensor and that they collectively function as correspondence mapping data. See Tables 8a and 8b for details; these tables also show that the correspondence between two data items can be determined based on the information contained in the two channels of the correspondence mapping tensor (data block). Although Tables 8a and 8b below are simply labeled “Correspondence Mapping,” the remainder of the label (“Left / Right Instance Segmentation”) is used to implement a channel-based approach (different labels may also be used for Tables 8a and 8b).

[0328] The two channels of the original tensor can also become part of a single tensor by using separate indices for the channels. This unification of subtensors can also be used in other embodiments where a tensor contains more tensors (which may be called subtensors); this possibility stems from the general mathematical framework of tensors.

[0329] Tables 8a-8d explain the raw outputs of the neural network, namely the correspondence mapping and the raw detection tensors of the left and right images. Tables 9a-9d show the tensors and computations that occur in the pairing module, and Tables 10a and 10b show the outputs of the disparity and 3D projection modules.

[0330]

[0331] Table 8a

[0332]

[0333] Table 8b

[0334]

[0335] Table 8c

[0336]

[0337] Table 8d

[0338] Tables 8a-8d provide an explanation of the raw output of the neural network in one embodiment. This embodiment uses stereo instance segmentation for the correspondence mapping (defining the region around the line boundary; exemplary results are obtained with a stride of 8), and the raw detections are relative distances to coordinates represented by tensor indices (coordinate grids), similar to the previous embodiment with paired masks. In this example, the raw detections are relative distances, but the correspondence mapping based on stereo instance segmentation can be applied to detections from arbitrary representations.

[0339] like Figures 3A-3C It can be observed that if c x,左 -c x,右 If ≥0, then the lane boundary point on the left image is located to the right of the corresponding lane boundary point on the right image. By comparing the contents of Tables 8a and 8b, a good estimate of the positions of lane boundaries “1” and “2” can be given, which is refined by the values ​​in Tables 8c and 8d.

[0340] Preserving D based on the results of left instance segmentation and right instance segmentation respectively 左 and D 右 The valid results in D. In other words, in D 左 and D 右 There is a lowercase 'x', and Tables 8a and 8b do not contain "1" and "2" respectively. Therefore, at this stage, for D... 左 and D 右 The prospect estimates are separate (and also remain separate in Tables 9b and 9c, see below).

[0341] Please note that since the original detection tensor is chosen as a signed relative distance, the sign of the original detection will change if both sides of the lane boundary are in the foreground region.

[0342]

[0343]

[0344] Table 9a

[0345]

[0346] Table 9b

[0347]

[0348] Table 9c

[0349]

[0350] Table 9d

[0351] Tables 9a-9d provide examples of calculations for the paired modules corresponding to the representations shown in Tables 8a-8d. A coordinate grid is generated according to equation (5), where the stride... x = stride y =8, offset x =Offset y =4. Image spatial detection is given by adding a coordinate grid to the original detection, as shown in Tables 2a-2c and Equation (10). Lowercase x refers to the unused predicted original detection, as it is treated as background according to stereo instance segmentation.

[0352] Table 9d contains the image spatial detection pairs. Before forming pairs, the image spatial detections corresponding to the same label and the same vertical coordinate on the left and right images were averaged separately (in this way, the first coordinate of the left side of the first pair is 22.1). Therefore, these results help improve accuracy.

[0353] Parallax 2.2 9.35 16.1 17.6 19.5

[0354] Table 10a

[0355] 3D projection (-3.6,0.5,90.9) (-1.5,1.0,21.4) (-1.5,1.1,12.4) (0.3,1.0,11.4) (0.2,1.3,10.3)

[0356] Table 10b

[0357] According to equation (11), the disparity module calculates the horizontal distance between the left and right detections by subtracting two corresponding tensors from Table 9, see the disparity results in Table 10a. Different rows correspond to different detections, such as the image space detection pairs in Table 9d. In Table 10b, according to equations (3-4), the 3D projection module maps the detection pairs from the detected left image coordinates and disparity to the 3D space of the camera coordinate system, where c x =30, c y =3, f=200, B=1. Table 10b may also have a column for a specified result label.

[0358] In different embodiments, the image processing module does not have any raw detection output, only stereo instance segmentation as a correspondence mapping. As discussed earlier, stereo instance segmentation on the left and right images encloses the lane boundaries and provides a good estimate of their positions. The precise location of the lane boundaries can be determined by averaging or least-squares model fitting. Thus, in one embodiment, the pairing module computes the image spatial location of the detections corresponding to various labels by weighted averaging along each original coordinate grid, where the weights are a difference function of the probability of a particular label at a grid location. In another embodiment, the pairing module fits the model to a point cloud defined by the coordinate grid and weights the label probability of each label by least-squares fitting. Samples are extracted from this model to obtain detection pairs, which are projected into 3D space in the 3D projection module. Since the least-squares fit and sampling from the model are differentiable, this embodiment is suitable for the general framework of a 3D lane detection system. More details about the model are described below in model-based embodiments, which differ from this approach in that they utilize raw detections represented by model parameters.

[0359] exist Figure 7A and 7B The diagram explains the stereo instance segmentation used in embodiments of the present invention, showing images from the first and second cameras respectively. In each image, three lane boundaries can be observed, namely... Figure 7A The first lane boundary projection 150a, the second lane boundary projection 152a, and the third lane boundary projection 154a, and their corresponding... Figure 7B The first lane boundary projection 150b, the second lane boundary projection 152b, and the third lane boundary projection 154b are shown in the image. It can be observed that these images have different viewpoints.

[0360] All lane boundaries have surrounding areas marked by 3D instance segmentation technology. Figure 7A In the middle, region 156a surrounds lane boundary projection 150a; region 156a maintains "1" as the instance segmentation index; a surrounding region 158a is specified for lane boundary projection 152a (labeled "2"), and a region 160a (labeled "3") is specified for lane boundary projection 154a. Figure 7B The corresponding regions 156b, 158b, and 160b (labeled “1”, “2”, and “3”) were also specified. Pixels around the same lane boundary lines received the same labels in both the first and second camera images.

[0361] In this technique, these region designations are used for correspondence mapping, as explained in the examples above. The correspondence mapping tensors explained in Tables 8a-8b give area representations of the first and second images. Therefore, by taking labels from the corresponding elements of the tensors, the specified surrounding regions (areas) in the images can be represented in these tensors (in the examples in Tables 8a-8b, two different labels, "1" and "2", are used; however, in... Figure 7A and 7B In the example, there are three different indices.

[0362] Figure 8A and 8B The diagram illustrates the stereo instance segmentation predicted by a trained neural network specified on a real image. Figure 8A and 8B The first image from the first camera and the second image from the second camera are given. The specified regions 230a, 232a, 234a, 236a, 238a and 230b, 232b, 234b, 236b, 238b are segmented by the instance. Figure 8A and 8B The areas highlighted in white (corresponding areas have the same numbering as "a" and "b" and are represented by the same pattern, such as left / right stripes, stars, "+" and "x" shaped grid lines; that is, the pattern represents the labels specified by the 3D instance segmentation) are the starting instance segmentation areas (similar to...). Figure 7A and 7B (and the regions explained in Tables 8a-8b), corresponding embodiments of the methods and systems according to the invention are performed based on these regions. Figure 8A and 8B In some ways similar to Figures 4A-4F This is because both sets of figures show corresponding mappings in their respective examples.

[0363] In the following text, another embodiment is specified, referred to as the model-based embodiment.

[0364] In this embodiment, the lane boundaries on the left and right images can also be represented by model parameters predicted by the neural network (generally, an image processing module trained by machine learning), such as spline parameters. Therefore, the original detection specifying the lane boundary arrangement is not a set of parameters similar to the distances described above, but rather spline parameters output by the neural network, such as spline parameters that characterize lane boundaries in different ways.

[0365] The lane boundary model can be represented by a variable number of parameters (N). In the case of a spline model, N depends on the number of control points and the degree of the spline polynomial. N can be fixed by the architecture, for example, when the number of control points is fixed. Alternatively, the number of control points used to represent the lane boundary can also be determined by a neural network.

[0366] Generally, the original detection is a different set of model parameters for each lane boundary. In the embodiments described in the table above, these parameters are a set of distance-type parameters; these distances are lane boundary-coordinate grid point distances. However, it is not necessary for the coordinate grid to specify the lane boundaries; in this model-based embodiment, when the lane boundaries are represented by splines, a coordinate grid is not required.

[0367] For example, a neural network can find the m of the lane boundary model on the left / right images. l / r The quantity (i.e., m) l / r (This refers to the number of detections), and each model has different parameters a. l / r,k , where k is the index of the detection. The number of model parameters can also differ between different detections. For example, this different number of detections can be achieved using LSTM units (Long Short-Term Memory units). The original detections of the left and right images are formatted as follows:

[0368]

[0369] These original detections D l / r These are more generalized data blocks than the tensors (matrices) described above. Since the number of required model parameters (N1, N2, Nm1) may differ in the rows, the rows of the above data blocks are not necessarily of equal length (there is no reason for them to be of equal length, but theoretically they can be).

[0370] A representation loss can be assigned to the original detection, which compares the model parameters of the lane boundaries with their ground-based true model parameters. However, various model parameters can produce similarly good detections, which can be better captured by an image spatial loss.

[0371] The detected image coordinates are calculated based on the lane boundary model predicted by the neural network; this step is performed in the pairing module. The lane boundary model M is an implicit function of the image coordinates (u, v) and the model parameters:

[0372] M(u,υ;a) l / r，k ) = 0

[0373] In a simpler case, when the model is parameterized by v (the above equations are rearranged):

[0374] u=M′(υ;a l / r，k )

[0375] In this embodiment, the correspondence mapping is a mapping between detected indices, for example...

[0376]

[0377] That is, the correspondence between the left and right parameter sets is given. Therefore, the original detection preferably provides input essentially to the correspondence mapping data block. In another embodiment, the lane boundary m l and m r The number of models is fixed by the architecture (regardless of how many lane boundaries are visible on the image), and the order between lane boundaries is defined. In this case, the raw detection is not necessarily the input to the correspondence mapping data block.

[0378] The pairing module iterates through the correspondences in C and collects samples from model M or M', for example, at different horizontal lines indexed by v. This provides detection pairs (for a... l，i -a r，j Yes, the equation given by M' above provides two different values ​​for 'u', which is processed by the disparity module and the 3D projection module by the detection (i.e., in this embodiment, the disparity is also calculated based on the two different values ​​of 'u', more specifically, based on its left and right values).

[0379] Training can be driven by image spatial loss. Sampling from the lane boundary model is a differentiable operation, so gradients can be propagated backward through the modules into the neural network.

[0380] In summary, in the above-described model-based method and system embodiments:

[0381] - The arrangement of each first member of the first index of the first lane boundary group in the first image (i.e., each lane boundary has a corresponding index) is defined by the first model parameters in the original detection data block of the first image.

[0382] - The arrangement of each second member with a second index of the second lane boundary group in the second image is defined by the second model parameters in the original detection data block of the second image, and

[0383] - The correspondence mapping data block determines the correspondence between the first index and the second index based on whether the first index and the second index correspond to the same lane boundary.

[0384] The methods for the first and second indexes described above can also be combined with the stereo instance segmentation method described above (not only with the correspondence mapping method mentioned in the previous paragraph), that is, the correspondence mapping method for stereo instance segmentation can be implemented to process the indexed lane boundaries.

[0385] In this scenario, the pairing module iteratively generates image spatial detections by sampling from the original detected elements in the left and right images (e.g., at different horizontal lines indexed by v). These image spatial detections obtain labels from stereo instance segmentation. Detection pairs are defined by points on the left and right images that lie on the same horizontal line and are characterized by the same label. Preferably, points on the same horizontal line with the same label can be individually averaged on the left and right images before generating pairs. Image spatial detections labeled as background do not constitute image spatial detection pairs.

[0386] Embodiments of the present invention relate to a non-transient computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform the method according to the present invention.

[0387] The advantages of the system and method according to the present invention are summarized here:

[0388] The solution of this invention is based on the above-described correspondence mapping method. Therefore, an explicit stereo matching step is not applied in this invention.

[0389] According to the present invention, a reduced noise level can be achieved in 3D detection. In contrast, in prior art methods, stereo matching generates noise on uniform surfaces (such as road surfaces) and produces poor quality results at lane boundaries.

[0390] The method and system according to the invention can also be operated in obscured areas.

[0391] Preferably, in embodiments applied to neural networks, the methods and systems used for training are end-to-end learnable and differentiable.

[0392] This invention is by no means limited to the preferred embodiments described in the above description, but rather various variations, modifications, and developments are possible within the scope of protection defined by the claims. Furthermore, all embodiments that may be defined by any combination of dependent claims are part of this invention.

Claims

1. A method for lane detection, the method comprising the steps of: generating, in an image processing step, image processing output data by means of an image processing module trained by machine learning, based on an image pair comprising a first image (10a) of a first lane boundary group and a second image (10b) of a second lane boundary group, wherein the image pair of the first image (10a) and the second image (10b) is a stereo image pair of a left image and a right image taken by a pair of cameras, and wherein the image processing output data comprises: correspondence mapping data consisting of a correspondence mapping data block (16) determining a correspondence between the first lane boundary group of the first image (10a) and the second lane boundary group of the second image (10b), a first image raw detection data block (14a) for defining an arrangement of each first member of the first lane boundary group, and a second image raw detection data block (14b) for defining an arrangement of each second member of the second lane boundary group, and wherein the image processing output data is generated for a coordinate grid represented by a coordinate grid tensor having a plurality of grid tensor elements, and the first image raw detection data block (14a) is a first image raw detection tensor corresponding to the coordinate grid tensor and having a plurality of first tensor elements comprising respective first distance values of a closest first lane boundary point of the first lane boundary group in a first search direction in the first image (10a) measured from a respective grid tensor element, the second image raw detection data block (14b) is a second image raw detection tensor corresponding to the coordinate grid tensor and having a plurality of second tensor elements comprising respective second distance values of a closest second lane boundary point of the second lane boundary group in a second search direction in the second image (10b) measured from a respective grid tensor element, and the correspondence mapping data block is a correspondence mapping tensor corresponding to the coordinate grid tensor and having a plurality of third tensor elements characterizing whether the closest first lane boundary point and the closest second lane boundary point correspond to the same lane boundary, and generating, in a pairing step, data blocks of image space detection pairs of lane boundaries based on the image processing output data to obtain image positions, wherein a tensor of image space detection pairs is generated as data blocks of image space detection pairs using the coordinate grid tensor, and generating, in a detection data generation step, 3D lane detection data blocks based on the following by means of triangulation for calculating 3D positions using calibration data (26) corresponding to the first image and the second image of internal and external parameters of the pair of cameras: a first data block portion of the tensor of image space detection pairs corresponding to a first member of the image space detection pairs, and a second data block portion of the tensor of image space detection pairs corresponding to a second member of the image space detection pairs. a second data block portion of the tensor corresponding to a second member of the image space detection pair, wherein the extrinsic parameters characterize a relative position and orientation of the pair of cameras and the intrinsic parameters are used together with a camera model of each of the pair of cameras to interpret an image to determine a projection of a 3D point in a camera coordinate system onto an image plane.

2. The method of claim 1, wherein, The correspondence mapping tensor is a pairwise correspondence mapping tensor having in each third tensor element thereof a pairwise probability value characterizing that the closest first lane boundary point and the closest second lane boundary point correspond to a same lane boundary and lie within a window size of a grid tensor element corresponding to the respective third tensor element, and third tensor elements having a respective probability value above a predetermined first threshold value are selected as foreground third tensor elements, the foreground third tensor elements and the corresponding first tensor elements and second tensor elements being used as output data of the image processing step in the pairing step.

3. The method of claim 2, wherein, The first search direction and the second search direction are selected as a left side direction on the respective first image and / or second image or a right side direction on the respective first image and / or second image, respectively.

4. The method of claim 2, wherein, The correspondence mapping tensor comprises at least two lane boundary type selective correspondence mapping tensors for at least two different lane types, respectively, the pairwise probability value is a lane boundary type selective probability value further characterizing a probability that a lane boundary corresponding to the closest first lane boundary point and the closest second lane boundary point is a certain lane boundary type, and the respective 3D lane detection data block is generated based on the at least two lane boundary type selective correspondence mapping tensors for the at least two different lane types.

5. The method of claim 1, wherein, The correspondence mapping data comprises first channel correspondence mapping data elements corresponding to the first image (10a) and second channel correspondence mapping data elements corresponding to the second image (10b), each of the first channel correspondence mapping data elements and the second channel correspondence mapping data elements is a label, the label characterizing whether a respective data element of the first channel correspondence mapping data elements and the second channel correspondence mapping data elements corresponds to a lane boundary, wherein identical labels correspond to a same lane boundary and different labels correspond to different lane boundaries in the first channel correspondence mapping data elements and the second channel correspondence mapping data elements, respectively, and data elements of the first channel correspondence mapping data elements and the second channel correspondence mapping data elements corresponding to lane boundaries are selected as foreground correspondence mapping data elements, the foreground correspondence mapping data elements being used as output data of the image processing step in the pairing step.

6. The method of claim 1, wherein: an arrangement of each second member of the first lane boundary group in the first image (10a) having a first index is defined by first model parameters in the first image raw detection data block, the arrangement of each second member of the second set of lane boundaries in the second image (10b) is defined by second model parameters in the second image raw detection data block, and the correspondence mapping data block (16) determines a correspondence between the first indices and the second indices based on whether the respective first indices and second indices correspond to the same lane boundary.

7. The method of claim 1, wherein, The image processing module (12) is implemented by a neural network.

8. The method of claim 1, wherein, In the image processing step, the image processing output data is generated by means of the image processing module based on at least one additional frame of the first image and the second image before or after the actual frames of the first image (10a) and the second image (10b) in addition to the actual frames.

9. A system for lane detection, comprising: an image processing module (12) trained by machine learning, which is adapted to generate image processing output data based on an image pair comprising a first set of lane boundaries of a first image (10a) and a second set of lane boundaries of a second image (10b), wherein the image pair of the first image (10a) and the second image (10b) is a stereo image pair of a left image and a right image taken by a pair of cameras, and wherein the image processing output data comprises: correspondence mapping data consisting of a correspondence mapping data block (16) determining a correspondence between the first set of lane boundaries of the first image (10a) and the second set of lane boundaries of the second image (10b), a first image raw detection data block (14a) for defining an arrangement of each first member of the first set of lane boundaries, and a second image raw detection data block (14b) for defining an arrangement of each second member of the second set of lane boundaries, and wherein the image processing output data is generated for a coordinate grid represented by a coordinate grid tensor having a plurality of grid tensor elements, and the first image raw detection data block (14a) is a first image raw detection tensor corresponding to the coordinate grid tensor and having a plurality of first tensor elements comprising respective first distance values of a closest first lane boundary point of the first set of lane boundaries in a first search direction in the first image (10a) measured from the respective grid tensor element, the second image raw detection data block (14b) is a second image raw detection tensor corresponding to the coordinate grid tensor and having a plurality of second tensor elements comprising respective second distance values of a closest second lane boundary point of the second set of lane boundaries in a second search direction in the second image (10b) measured from the respective grid tensor element, and the second image raw detection data block (14b) is a second image raw detection tensor corresponding to the coordinate grid tensor and having a plurality of second tensor elements comprising respective second distance values of a closest second lane boundary point of the second set of lane boundaries in a second search direction in the second image (10b) measured from the respective grid tensor element, and The correspondence mapping data block is a correspondence mapping tensor corresponding to the coordinate grid tensor and having a plurality of third tensor elements characterizing whether the closest first lane boundary point and the closest second lane boundary point correspond to the same lane boundary, and a pairing module (20) adapted to generate, based on the image processing output data, a data block of image space detections pairs to obtain image positions, wherein using the coordinate grid tensor, a tensor of image space detections pairs is generated as a data block of image space detections pairs, and a 3D projection module (24) adapted to generate, based on the following, a 3D lane detection data block by means of triangulation for computing 3D positions using calibration data (26) corresponding to the first and second images describing internal and external parameters of the pair of cameras: a first data block portion of the tensor of image space detections pairs corresponding to the first member of the image space detections pair, and a second data block portion of the tensor of image space detections pairs corresponding to the second member of the image space detections pair, wherein the external parameters characterize a relative position and orientation of the pair of cameras and the internal parameters together with a camera model of each of the pair of cameras are used to interpret an image to determine a projection of a 3D point in a camera coordinate system to an image plane.

10. The system of claim 9, wherein, The correspondence mapping tensor is a pair-wise correspondence mapping tensor having, in each third tensor element thereof, a pair-wise probability value characterizing that the closest first lane boundary point and the closest second lane boundary point correspond to the same lane boundary and lie within a window size of the grid tensor element corresponding to the respective third tensor element, and a third tensor element having a respective probability value above a predetermined first threshold value is selected as a foreground third tensor element, the foreground third tensor element and the corresponding first and second tensor elements being used in the pairing module (20) as output data of the image processing module.

11. The system of claim 10, wherein, The first and second search directions are selected as a left side direction on the respective first and / or second image or a right side direction on the respective first and / or second image, respectively.

12. The system of claim 10, wherein, The correspondence mapping tensor comprises at least two lane boundary type selective correspondence mapping tensors for at least two different lane types, respectively, the pair-wise probability value is a lane boundary type selective probability value further characterizing a probability that the lane boundary corresponding to the closest first lane boundary point and the closest second lane boundary point is a certain lane boundary type, and the system is adapted to generate a respective 3D lane detection data block based on the at least two lane boundary type selective correspondence mapping tensors for the at least two different lane types.

13. The system of claim 9, wherein, The correspondence mapping data comprises a first lane boundary group of the first image (10a) and a second lane boundary group of the second image (10b), each of the first lane boundary group and the second lane boundary group being a set of lane boundary elements, and the correspondence mapping data comprises a first lane correspondence mapping data element corresponding to the first image (10a) and a second lane correspondence mapping data element corresponding to the second image (10b), each of the first lane correspondence mapping data element and the second lane correspondence mapping data element being a label, the label characterizing whether a respective data element of the first lane correspondence mapping data element and the second lane correspondence mapping data element corresponds to a same lane boundary, wherein a same label corresponds to a same lane boundary, and different labels respectively correspond to different lane boundaries in the first lane correspondence mapping data element and the second lane correspondence mapping data element, and the data elements of the first lane correspondence mapping data element and the second lane correspondence mapping data element corresponding to a lane boundary are selected as foreground correspondence mapping data elements, the foreground correspondence mapping data elements being used in the pairing module as image processing output data of the image processing module.

14. The system of claim 9, wherein: an arrangement of each second member of the first lane boundary group in the first image (10a) having the first index is defined by first model parameters in the first image raw detection data block, an arrangement of each second member of the second lane boundary group in the second image (10b) having the second index is defined by second model parameters in the second image raw detection data block, and the correspondence mapping data block (16) determines a correspondence between the first index and the second index based on whether the respective first index and second index correspond to a same lane boundary.

15. The system of claim 9, wherein, The image processing module (12) is implemented by a neural network.

16. The system of claim 9, wherein, The generation of the output data of the image processing module (12) is based on at least one additional frame of the first image and the second image before or after the actual frames of the first image (10a) and the second image (10b) in addition to the actual frames.

17. A non-transitory computer readable medium storing instructions that, when executed by a computer, cause the computer to: generate, in an image processing step, image processing output data by means of an image processing module trained by machine learning based on an image pair comprising a first image (10a) of a first lane boundary group and a second image (10b) of a second lane boundary group, wherein the image pair of the first image (10a) and the second image (10b) is a stereo image pair of a left image and a right image taken by a pair of cameras, and wherein the image processing output data comprises: correspondence mapping data consisting of a correspondence mapping data block (16) determining a correspondence between the first lane boundary group of the first image (10a) and the second lane boundary group of the second image (10b), a first image raw detection data block (14a) for defining an arrangement of each first member of the first lane boundary group, and a second image raw detection data block (14b) for defining an arrangement of each second member of the second lane boundary group. a second image raw detection data block (14b) for defining an arrangement of each second member of the second lane boundary group, and wherein the image processing output data is generated for a coordinate grid represented by a coordinate grid tensor having a plurality of grid tensor elements, and the first image raw detection data block (14a) is a first image raw detection tensor corresponding to the coordinate network tensor and having a plurality of first tensor elements comprising respective first distance values of closest first lane boundary points of the first lane boundary group in the first search direction in the first image (10a) measured from respective grid tensor elements, the second image raw detection data block (14b) is a second image raw detection tensor corresponding to the coordinate network tensor and having a plurality of second tensor elements comprising respective second distance values of closest second lane boundary points of the second lane boundary group in the second search direction in the second image (10b) measured from respective grid tensor elements, and the correspondence mapping data block is a correspondence mapping tensor corresponding to the coordinate grid tensor and having a plurality of third tensor elements characterizing whether the closest first lane boundary point and the closest second lane boundary point correspond to the same lane boundary, and generating a data block of image space detection pairs of lane boundaries in a pairing step based on the image processing output data to obtain image positions, wherein using the coordinate grid tensor, a tensor of image space detection pairs is generated as a data block of image space detection pairs, and in a 3D lane detection data generation step, using calibration data (26) corresponding to the first image and the second image describing internal and external parameters of the pair of cameras, generating a 3D lane detection data block based on: a first data block portion of the tensor of image space detection pairs corresponding to the first member of the image space detection pair, and a second data block portion of the tensor of image space detection pairs corresponding to the second member of the image space detection pair, wherein the external parameters characterize a relative position and orientation of the pair of cameras and the internal parameters together with a camera model of each of the pair of cameras are used to interpret an image to determine a projection of a 3D point in a camera coordinate system to an image plane.

Images