Method and system for lane detection - Patents.com

Abstract

The present invention relates to a method for lane detection, the method including: generating, as output data, correspondence mapping data based on an image pair of first and second images (10a, 10b) using an image processing module (12) trained by machine learning, the correspondence mapping data determining a correspondence between a first lane boundary group in the first image and a second lane boundary group in the second image; generating data blocks of image space detection pairs based on the output data; and generating 3D lane detection data blocks by triangulation based on first and second data block portions of the data blocks of the image space detection pairs corresponding to the first and second members, respectively, of the image space detection pair. The present invention also relates to a system for lane detection, as well as a method and system for training an image processing module.

Description

[Technical field]

[0001] The present invention relates to a method and a system for lane detection by an image processing module trained by machine learning. The present invention also relates to a method and a system for training an image processing module. [Background technology]

[0002] Perception of the 3D environment for road vehicles is crucial for autonomous driving applications. Road surface estimation and lane marker or boundary detection are necessary for lane keeping and lane changing maneuvers, as well as for obstacle localization at lane level. Lane shape and obstacle information are key inputs for devices responsible for controlling the vehicle. Therefore, several approaches and attempts have been made to tackle the problem of lane detection.

[0003] Among prior art lane detection approaches, traditional computer vision techniques are used, where lane markers are searched for according to object width, orientation, alignment, and other criteria. Such lane detection approaches are disclosed, for example, in U.S. Patent No. 6,819,779 (B1), U.S. Patent No. 6,813,370 (B1), U.S. Patent No. 9,286,524 (B2), CN105975957A, and JP2002150302 (A).

[0004] Further computer vision-based approaches have been introduced: Andras Bodis-Szomoru et al.: "A lane detection algorithm based on wide-baseline stereo vision for advanced driver assistance", KEPAF2009. 7th conference of Hungarian Association for Image Processing and Pattern Recognition. Budapest, 2009. Sergiu Nedevschi et al.: "3D Lane Detection System Based on Stereovision", 2004 IEEE Intelligent Transportation Systems Conference, Washington, DC,USA, October 3-6,2004. Rui Fan et al.: “Real-time stereo vision-based lane detection system”, Meas. Sci. Technol. Vol. 29, 074005, 2018.

[0005] A further type of lane detection approach is disclosed in US Patent Application Publication No. 2018 / 0067494(A1), where a 3D point cloud from a LIDAR (Light Detection and Ranging) device is utilized for lane detection. The main drawback of this approach is the use of a LIDAR device; that is, this approach requires a dedicated and expensive device to realize lane detection. The use of LIDAR is also disadvantageous from the perspective of "visible" objects. The LIDAR point cloud does not contain information about occluded objects and / or occluded sides of objects; that is, the point cloud contains limited spatial information. A similar approach using depth information is disclosed in US Patent Application Publication No. 2018 / 131924(A1).

[0006] US Patent Application Publication No. 2018 / 283892(A1) discloses the detection of lane markers involving the use of high-resolution maps. In this approach, the detection of lane markers is performed with the help of semantic segmentation, therefore large areas on the image are associated with lane markers. This approach produces high noise in the 3D projection, even when Douglas-Peucker divergence is performed to extract the lane segments.

[0007] A similar approach to the above is disclosed in U.S. Pat. No. 10,055,650 (B2), where lane detection is performed as part of object detection performed to verify lanes among other objects, e.g. surrounding vehicles, traffic signs, danger zones, or tunnels.

[0008] Segmentation and object classification based techniques similar to the above approaches are disclosed in WO2018 / 104563A2, WO2018 / 172849A2, U.S. Patent No. 9,902,401(B2), U.S. Patent No. 9,286,524(B1), CN107092862A, EP3171292A1 and U.S. Patent No. 10,007,854(B2), for example, to identify lane markers. A method for determining lane boundaries, for example with the help of neural networks, is disclosed in U.S. Patent No. 9,884,623(B2).

[0009] In view of known approaches, there is a need for a lane detection method and system that is more efficient than the prior art approaches. Summary of the Invention [Problem to be solved by the invention]

[0010] The main object of the present invention is to provide a method and system for lane detection which, as far as possible, does not suffer from the disadvantages of the prior art approaches.

[0011] A further object of the method and system for lane detection is to provide an improved approach for lane detection that is more efficient than prior art approaches. It is an object of the present invention to provide 3D (three dimensional) lane detection based on a first image and a second image using machine learning. [Means for solving the problem]

[0012] The object of the invention can be achieved by a method and a system for lane detection according to claims 1 and 13, respectively, a method and a system for training a neural network according to claims 25 and 26, respectively, and a non-transitory computer readable medium according to claim 27. Preferred embodiments of the invention are defined in the dependent claims.

[0013] In order to explain the invention with the help of exemplary embodiments, the following is disclosed herein. In an exemplary embodiment of the method and system according to the invention (see also possible generalizations below), 3D lane detection is performed based on raw detection of lane boundaries in the input image (defining the location of the lane boundaries on the image) and a correspondence mapping that gives a correspondence between the raw detections of the two images (defining corresponding pairs of lane boundaries in a stereo image pair). The raw detection and the correspondence mapping are the output of an image processing module (unit) applied in this embodiment of the method and system according to the invention. Thus, a direct identification of the localization of the lane boundaries is performed according to the invention (e.g. based on the centerline of the lane boundary markers). This constitutes a different approach to the segmentation-based known techniques introduced above, since in these known approaches the lane boundary markers are searched for with the help of segmentation approaches carried out with the help of machine learning algorithms, i.e. the boundary markers are explored as patches on the image, like any other object (see Davy Neven et al., Towards End-to-End Lane Detection: an Instance Segmentation Approach, 2018, arXiv:1802.05591; Yen-Chang Hsu et al., Learning to Cluster for Proposal-Free Instance Segmentation, 2018, arXiv:1803.06459, see also below. In comparison with these approaches, a method capable of efficiently processing stereo information is preferably solved according to the present invention).

[0014] According to the above sections, an image processing module trained by machine learning is generally used in the methods and systems according to the present invention. However, the use of this image processing module is illustrated by the use of its exemplary implementation, a neural network. The description of the neural network also indicates that some parameters and other details of the selected machine learning implementation are set.

[0015] Preferred embodiments of the invention will now be described, by way of example only, with reference to the following drawings, in which: [Brief description of the drawings]

[0016] [Figure 1] 1 is a flow chart illustrating an embodiment of a method and system according to the present invention. [Diagram 2] FIG. 1 illustrates various correspondence mappings and their relationship to different types of raw detections in one embodiment. [Figure 3A] FIG. 1 is a schematic diagram illustrating an exemplary left and right image merging in one embodiment, showing various pairing masks and search directions. [Figure 3B] FIG. 1 is a schematic diagram illustrating an exemplary left and right image merging in one embodiment, showing various pairing masks and search directions. [Figure 3C] FIG. 1 is a schematic diagram illustrating an exemplary left and right image merging in one embodiment, showing various pairing masks and search directions. [Figure 4A] 3A-3C are stylized exemplary left and right images showing pairing masks and lane boundaries in an embodiment similar to that of FIGS. [Figure 4B] 3A-3C are stylized exemplary left and right images showing pairing masks and lane boundaries in an embodiment similar to that of FIGS. [Figure 4C] 3A-3C are stylized exemplary left and right images showing pairing masks and lane boundaries in an embodiment similar to that of FIGS. [Figure 4D]3A-3C are stylized exemplary left and right images showing pairing masks and lane boundaries in an embodiment similar to that of FIGS. [Figure 4E] 3A-3C are stylized exemplary left and right images showing pairing masks and lane boundaries in an embodiment similar to that of FIGS. [Figure 4F] 3A-3C are stylized exemplary left and right images showing pairing masks and lane boundaries in an embodiment similar to that of FIGS. [Figure 4G] Shown on the left is image space detection corresponding to the scheme in Figures 4A-4F. [Figure 5A] 5A-5B are Y(Z) and X(Z) diagrams of results obtained based on the scenario depicted in FIGS. 4A-4G with the aid of an embodiment of the method according to the invention. [Figure 5B] 5A-5B are Y(Z) and X(Z) diagrams of results obtained based on the scenario depicted in FIGS. 4A-4G with the aid of an embodiment of the method according to the invention. [Figure 6A] 6A-6B are Y(Z) and X(Z) diagrams of results obtained based on the scenario shown in FIGS. 4A-4G with the aid of known approaches. [Figure 6B] 6A-6B are Y(Z) and X(Z) diagrams of results obtained based on the scenario shown in FIGS. 4A-4G with the aid of known approaches. [Figure 7A] 7A-7B are schematic diagrams illustrating exemplary first and second camera images showing lane boundaries and corresponding mapping in a further embodiment. [Figure 7B] 7A-7B are schematic diagrams illustrating exemplary first and second camera images showing lane boundaries and corresponding mapping in a further embodiment. [Figure 8A] 8A-8B are stylized example left and right images showing lane boundaries and the correspondence mapping obtained by stereo instance segmentation. [Figure 8B]8A-8B are stylized example left and right images showing lane boundaries and the correspondence mapping obtained by stereo instance segmentation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] The present invention is a method and a system for lane detection. In the framework of the present invention, this means that as a result of the method or system according to the present invention, 3D (three-dimensional) lane detection data is obtained based on an input image. More specifically, preferably, a 3D representation of the 3D lane boundaries is the result obtained based on an image pair (preferably taken by a vehicle traveling along the lane to which the lane boundaries correspond). A flow diagram of an embodiment of the method and system (blocks corresponding to stages or data blocks, considering the method, as well as blocks corresponding to modules or data blocks, considering the system) is shown in Fig. 1.

[0018] The method according to the invention comprises the steps of: In an image processing step, using an image processing module trained by machine learning, based on an image pair of a first image and a second image (first image 10a and second image 10b in the embodiment of FIG. 1), generating as output data correspondence mapping data determining correspondence between a first lane boundary group of the first image and a second lane boundary group of the second image; generating image space detection pair data blocks based on the output data in a pairing step (which may also be referred to as an image space detection data generation step); In the 3D lane detection data generation step, a first data block portion of the data block of the image space detection pair corresponding to a first member of the image space detection pair; and A second data block portion of the data block of the image space detection pair corresponding to the second member of the image space detection pair and generating a 3D lane detection data block by triangulation based on the calibration data corresponding to the first image and the second image.

[0019] Of course, 3D can be written as three dimensions with each name.

[0020] A lane boundary group - first or second - is a group of possible lane boundaries in the respective image. A lane boundary group may contain one or more lane boundaries, but in a few cases the number of lane boundaries may be zero in an image. This is also reflected in the name "lane boundary group", which does not indicate that there are multiple lane boundaries or that there is zero lane boundary in it. Thus, the correspondence mapping data indicates the correspondence between groups of lane boundaries, without knowing in advance which groups and how many lane boundaries they contain.

[0021] According to the above introduction of the invention, in general, the correspondence mapping data is generated by the method according to the invention. As shown below, in some embodiments, the correspondence mapping data is incorporated into a separate correspondence mapping data block (see also the next embodiment). Typically, in the embodiment with a separate correspondence mapping data block, a raw detection data block is also defined as the output of the image processing module.

[0022] However, various realizations of the correspondence mapping data are conceivable. In one embodiment, only the correspondence mapping data is utilized for further calculation of the 3D lane detection data, and no raw detection data is utilized at all. For illustrative purposes, an example is given in Tables 8a-8d below (although in Tables 8a-8d raw detection is also defined in addition to the correspondence mapping data): Coordinates corresponding to non-zero values ​​of the correspondence mapping data provide a good approximation of the location of the lane boundaries, and this data is used alone, utilizing a coordinate grid if necessary, to obtain image space detection and 3D coordinates of the lane boundaries. The quality of this approximation is limited by the resolution of the correspondence mapping data. If the resolution of the correspondence mapping data is similar to that of the input image, the raw detection is interpreted as a ignored small correction to the coordinates of the lane boundaries inferred from the correspondence mapping data. In this case, the correspondence mapping data, in addition to its main purpose of determining the correspondence, further defines the location of each first member of the first lane boundary group and the location of each second member of the second lane boundary group.

[0023] It is also conceivable that the corresponding mapping data is incorporated into the raw detection from the beginning, so that the highly corresponding raw detection data block contains only these raw detection data elements. This filtering can also be done if a separate corresponding mapping data block is defined.

[0024] In summary, the correspondence mapping data (and other outputs in other embodiments such as raw detection) are generated with the help of an image processing module trained by machine learning in a suitable form in all embodiments of the invention (i.e., the image processing module - implemented for example by a neural network - is trainable by machine learning and generates the correspondence mapping data and other possibly relevant outputs). In other words, the data involved in determining the correspondence between lane boundaries in the first image and the second image is defined in all embodiments. Thus, the trained image processing module is, in other words, a trained machine learning model.

[0025] In the above introduction of the present invention, a correspondence is determined between the first lane boundary detection group and the second lane boundary detection group of the respective first image and second image. These groups may include one or more lane boundaries (corresponding to each lane boundary observable in the respective first or second image), but may also include zero detection in case of boundary line when no lane boundary is detected in the image (then all information of the image corresponds to the background; see below for the distinction between foreground and background).

[0026] It should also be noted that the pairing steps (all of them) use the output data generated in the image processing steps to generate the data blocks of the image space detection pairs. That is, if there is only correspondence mapping data, then that is all that is used. However, if there are raw detection data blocks, then these are also used. In some of the following embodiments, it is also specified what is included in the output data forwarded to the pairing step.

[0027] As shown below, triangulation is the most common approach based on which a 3D lane detection data block can be computed. Triangulation is a method of computing 3D positions based on the correspondence of points found in multiple images of a point-like object (e.g., a specific point of the object). The set of corresponding points defines a set of rays starting from the corresponding camera centers (the origin of the camera coordinate system). The 3D coordinates of the object are computed as the intersection points of the rays, resulting in a linear simultaneous equation. If the point correspondence is not perfect, it is possible that the rays do not intersect. In this case, the estimated 3D position can be defined as the point that is closest in distance perpendicular to the rays. Alternatively, a minimum refinement of the correspondences can be determined numerically such that the refined correspondences satisfy the epipolar constraint, i.e., the corresponding rays intersect (see section 12.5 of the following reference book, which is incorporated herein by reference). Disparity-based computation is a special variant of triangulation. For more information on triangulation, see, for example, Richard Hartley and Andrew Zisserman (2003). Multiple View Geometry in computer vision. Cambridge University Press. ISBN978-0-521-54051-3. Taking into account the calibration data, the geometry for the mutual comparison of the images can be well defined, so that the calibration data corresponding to the first and second images are also used for triangulation.

[0028] In one embodiment, raw detection is also defined during the method. In this embodiment of the method, the correspondence mapping data is included in a correspondence mapping data block (e.g., correspondence data block 16 in FIG. 1 labeled "Correspondence Mapping"), and the output data is further a first image raw detection data block 14a for defining the arrangement of each first member of the first lane boundary detection group; and and a second image raw detection data block 14b for defining the arrangement of each second member of the second lane boundary detection group.

[0029] For the first image raw detection data block 14a labeled "first image raw detection", please refer to the embodiment in FIG. 1. Typically, the raw detection data block contains data of one or more raw detections. If there are zero raw detections, the method results in no lane boundaries being found, and therefore no 3D coordinates can be assigned to them. A single detection means a single finite-valued parameter, which corresponds to a distance or another parameter, as shown below. Also refer to FIG. 1 for the second image raw detection data block 14b labeled "second image raw detection". These data blocks are called raw detection data blocks, because the raw detection corresponds to a raw detection of the respective data (data defining the placement of the lane boundaries) on which the data is processed.

[0030] According to the introduction above, the location of lane boundaries can be defined by raw detection data blocks or with the help of the correspondence mapping data itself. These two options result in two main possibilities. Furthermore, when defining the location with the help of raw detection, two sub-cases can be introduced as detailed below. In the first sub-case, the raw detection and the correspondence mapping are independent outputs of the image processing module (see, for example, Tables 1a-1c below). However, in the second sub-case, the raw detection and the correspondence mapping data are obtained as outputs such that the raw detections essentially serve as inputs for the correspondence mapping data (see model-based embodiment below).

[0031] In the introduction of the method according to the invention above, the computation of data blocks is defined. For the interpretation of "data block", please see the details below (e.g., a data block can be represented by a tensor). In general, a data block is a block of data (if the block has an arbitrary shape) and the members of a data block are usually numbers, as shown in the example below.

[0032] It is also stated herein that, as used in many embodiments, in one embodiment the image pair of the first image and the second image is a stereo image pair of a left image and a right image (to capture a stereo image, one is typically limited to creating two images simultaneously, and preferably the two imaging cameras are in the same plane). However, the first image and the second image may be from any relative orientation of the imaging devices and may be created at different times, provided that the first image and the second image so created have sufficient overlap for detection.

[0033] According to the above details, for the purpose of lane detection, a lane boundary is preferably defined as a continuous curve that divides the drivable road surface into lanes. If the lanes are bounded by lane markers, the lane boundary preferably follows the center line of the lane boundary markers, which can be continuous (single or double) or have several simple or more complex dashed lines. Alternatively, the lane boundary may be another specific line corresponding to the sequence of lane markers (it can run along its right or left side or somewhere else therein). Thus, the lane boundary is preferably a single line (without width dimension), e.g., doubly closed (continuous) lane markers are also projected onto a single line. Depending on national regulations, for example, there may be cases where the lateral lane boundary of a lane (i.e. the side of the lane that is on the side of the road itself) is not marked with a lane boundary marker. This lane boundary (the roadside is also covered by the name "lane boundary") is therefore a simple transition between the road and the area adjacent to the road (which may be pavement, grassy area, gravel, etc.).

[0034] Being able to handle this type of lane boundary as well is a major advantage of the present solution according to the invention. During the learning process, the image processing module, or in a particular implementation, the neural network, can also learn (with the help of application of a suitable type of ground truth image) such type of lane boundary (the network searches for this type of transition, which can also be approximated by a single line on the image) and recognize it during use. This different type of lane boundary can therefore be identified in a similar way as lane boundaries using lane boundary markers (the annotation technique is also the same) without changing the concept of the approach. Thus, both boundaries of a lane (i.e. its boundaries on both sides) can preferably be modeled with the help of the present invention.

[0035] In the pairing step (and the pairing module below), detection pairs are identified. These can be represented for example in a single block with parameters like channels for the first and second detection (see FIG. 1, this output is given as a single output to the next module) or they can be represented in more separated data blocks for example of the first and second images, see the example below (separate first and second image space detection data blocks for the corresponding mapping data block and the respective first or second image raw detection data block. The image space detection data block can also be part of a larger data block obtained in the pairing step). In the pairing step, such detections that constitute a pair are identified, for example based on which a disparity can be calculated.

[0036] In the pairing step (hence the pairing module 20), image space detections are obtained. These can have a single tensor representation or two tensor representations as described above. There is a first data block part and a second data block part of each of the data blocks of the image space detection pair that corresponds to the first and second members of each of the image space detection pairs. From the data blocks of the image space detection pair it is always possible to derive which parts correspond to the respective members of the pair, i.e. its first data block part and its second data block part can be defined (see for example the two tensor representations of tables 2b and 2c or 6b and 6c. The separation of the data can also be performed for a single tensor representation).

[0037] A corresponding system according to the invention, which is also suitable for lane detection, preferably adapted to carry out the method according to the invention, comprises: an image processing module trained by machine learning (implemented by image processing module 12 in the embodiment of FIG. 1 and labeled “neural network” according to the main elements of the module in the embodiment), adapted to generate as output data correspondence mapping data (raw detection data blocks may also be introduced in the embodiment of the system) determining correspondence between a first lane boundary detection group in the first image and a second lane boundary detection group in the second image based on an image pair of a first image and a second image; a pairing module (labeled “pairing” in the embodiment of FIG. 1 ) adapted to generate image space detection pair data blocks based on the output data; a first data block portion of the data block of the image space detection pair corresponding to a first member of the image space detection pair; and The second data block portion of the data block of the image space detection pair that corresponds to the second member of the image space detection pair. and a 3D projection module (in the embodiment of FIG. 1 , 3D projection module 24, labeled “3D projection”, this module is adapted for 3D projection) adapted to generate 3D lane detection data blocks by triangulation using calibration data corresponding to the first image and the second image based on the above.

[0038] Preferably, the system (or equivalently, it can be considered as an apparatus) comprises: one or more processors; This can be realized by including a memory containing program code which, when coupled to one or more processors and executed by the one or more processors, causes the system to perform the tasks (functions, i.e. steps defined in the corresponding embodiments of the method according to the invention) performed by the modules introduced in accordance with the above system definition, and more specifically, to generate the various quantities generated by the above modules.

[0039] The above modules provide a task-oriented distribution of computer subassemblies. The system itself can also be considered as being responsible for various tasks.

[0040] As is evident from the method, it is not necessary to calculate disparity data in all cases. For the system, this means that it is not necessary to include a disparity module. If the calculation of the 3D lane detection data block is also based on this data (which is preferably scheduled in advance), then the system includes a disparity module.

[0041] A summary of the method and system according to one embodiment of the present invention is given in the following points (for some points further generalizations are given in the description). 1. Input: a. Image pair (preferably stereo, rectified) b. Calibration 2. Image processing module (optional, excluding input and output constraints, for example implemented by a neural network): a. Input (the image processing module can be fed with stereo input) b.Output: i. Correspondence Mapping ii. Raw detection on individual images iii. Optional optional outputs for additional tasks 3. Pairing Module 4.3D Projection Module

[0042] A particular embodiment of the present invention relates to a method and system for training an image processing module (e.g., a neural network) applied in the method and system for lane detection according to the present invention introduced above. This training method and system is introduced in parallel with the method and system for lane detection. The training method and system uses a loss function introduced below. The loss module (or stage) is also shown in Figure 1, thus illustrating both the method and system for lane detection and the method and system for training the image processing module.

[0043] An important building block of the present invention is the implementation of machine learning utilized in the image processing module, which introduces the use of annotations and helps to avoid hand-crafted parameter settings. The image processing module trained by machine learning is preferably implemented by a neural network (i.e. utilized in the image processing module), but e.g. decision trees, support vector machines, random forests, or other types of machine learning implementations can also be utilized. In general, trained machine learning models (and thus image processing modules trained by machine learning) are a broader class of artificial intelligence than neural networks, this class being composed of approaches using neural networks.

[0044] The (learnable) parameters of the image processing module trained by machine learning are optimized during training by minimizing a loss function. The parameters are preferably updated to have low loss values, and therefore the loss is also controlled during training. The loss function corresponds to the objective of the image processing module trained by machine learning. That is, the loss function depends on the type of output expected after the machine learning procedure is performed. For example, it is not good if the image processing module trained by machine learning places detections far from lane boundaries (sometimes also called separators). Thus, a high loss corresponds to detections that are placed far from the ground truth. In another example, an error can be considered smaller if the image processing module trained by machine learning cannot decide whether a marker is dashed or continuous than if there is an error in the decision of whether it is part of the foreground or background. Thus, the members of the loss function that are responsible for the different errors can be weighted.

[0045] The loss function quantifies the difference between the prediction (output of the image processing module) and the reference output (ground truth) we want to achieve. The loss function can include various components such as detection loss in image space, detection loss in 3D, etc. and can be introduced in different blocks of the lane detection system.

[0046] Advantageously, the system according to the invention is fully differentiable, preferably in the preferred embodiment using a neural network as image processing module trained by machine learning. Then, all gradients of the loss function can be calculated by backpropagation. That is, the gradients can pass through the various modules back to all parameters and to the input parameters of the neural network (dashed lines in FIG. 1). The neural network, the neural network output, the disparity module and the 3D projection module are naturally differentiable. In contrast, the pairing module needs to be constructed to be differentiable. In one embodiment, the gradients coming from the other modules and passing through the pairing module can be propagated back to the modules of the first image raw detection data block 14a and the second image raw detection data block 14b and, optionally, can also be propagated to the module of the corresponding mapping data block 16. Thus, the learning procedure can be performed in a unified way for the whole system.

[0047] The gradient of the loss function is used to update the values ​​of the learnable parameters of the neural network. As the gradient flows back through the various modules, the loss function not only optimizes the learnable parameters of the module to which it is assigned (see Figure 1 for the various loss modules), but also the entire neural network that precedes the module. Some neural networks are designed such that different parts need to be trained separately to achieve good performance. In contrast, the present invention provides an end-to-end trainable solution and does not require complex training procedures.

[0048] The main blocks and the information flow between them are shown in Fig. 1, i.e. Fig. 1 shows a modular description of the 3D lane detection system according to the invention and also the corresponding 3D lane detection method according to the invention can be interpreted on the basis of Fig. 1. In Fig. 1, the ground truth data 28 is also shown by a block (i.e. a module adapted to transfer ground truth data). The ground truth data 28 module is preferably connected to all loss modules in order to calculate the losses of these modules.

[0049] In summary, a loss function is preferably assigned to each module, which drives the neural network during the training procedure to give an output, e.g., ground truth.

[0050] In FIG. 1, the information flow during the inference mode (i.e., the use of the method and system for 3D lane detection) is indicated by solid arrows (forward path). During training, information is propagated backwards from the loss module to the neural network along the dashed lines. The dashed lines correspond to the data connections where data propagates back from the loss module. In the modules along the way of this data, the dashed lines are connected to each other via dashed circles. The connections via dashed circles are for illustrative purposes only, the data packets coming from the respective loss modules travel in parallel to the image processing module 12. The loss modules play a special role, since they are only used during the learning procedure of the neural network. Thus, the loss modules could have been separated using the explanation in FIG. 1 (e.g., by using another type of line to connect them to the corresponding modules and / or to the loss modules themselves, which could have resulted in a separate explanation tool, using for example other types of rectangles - e.g. dashed - for the loss modules in the diagram of FIG. 1).

[0051] The additional output block 18 refers to additional tasks performed by the neural network, which may be closely or weakly related to lane detection, e.g. road segmentation, traffic object detection, lane type determination, etc. Additional regularization losses may be added to the neural network, but are not shown in this figure. The additional output block 18 preferably has its own loss module 30.

[0052] The output of the 3D projection module (i.e., the entire 3D lane detection system) can be the input of other modules in the autonomous driving application (as indicated by the arrow labeled "Next Module" from the 3D projection module 24). Additionally, data in the additional output block 18 can also be made available for other modules.

[0053] For example, the extracted 3D lane boundaries can be used in online camera parameter estimation, motion or trajectory planning, localization, or any other module that may benefit from the 3D location of lane boundaries. Optionally, the 3D detections of a lane detection system can be input to sensor fusion algorithms that determine a (3D) road model from different sources such as lane detection, road segmentation, object detection, and from different sources such as cameras, lidar, radar, etc.

[0054] FIG. 1 shows the inputs of the image processing module 12 (simply labeled "input" in the header of FIG. 1; the header shows the successive stages of the system and the corresponding method). It has two image inputs (generally a first image 10a and a second image 10b, which can be the left and right images). These image inputs preferably provide rectified stereo image pairs, any size and channel (greyscale or RGB, etc.). In principle, the rectification could be moved (i.e. realized) to a lane detection system, and then the inputs of the image processing module 12 would be the stereo image pairs and the calibration data 26, the first step being the rectification.

[0055] Calibration data 26 is an input of the 3D projection module 24. The calibration data describes the intrinsic and extrinsic parameters of the cameras. The extrinsic parameters characterize the relative position and orientation of the two cameras. The intrinsic parameters together with the model of the cameras used to interpret the images determine how 3D points in the camera coordinate system are projected onto the image plane. Although the calibration data is necessary for correction, the correction can be performed in a module that is not part of the lane detection system (performed before entering the system). Thus, the lane detection system and method can receive the corrected camera images as well as the calibration data describing the corrected camera images. Furthermore, as shown in FIG. 1, the calibration data describing the corrected camera images is preferably used for the 3D projection.

[0056] Rectification of stereo cameras and corresponding images is a standard method used in stereo image processing, which transforms the cameras and corresponding images into a simplified epipolar geometry where the two camera image planes coincide and the epipolar lines coincide. Points on the image planes represent rays in the camera coordinate system that pass through the origin. For any point on one image plane, the corresponding epipolar line is the projection of the corresponding ray on the other image plane. This means that if a 3D point corresponding to a real object is found in one image, its projection onto the other image plane will be on the corresponding epipolar line. For more details on rectification, see e.g. Bradski, G., Kaehler, A.: O'Reilly learning OpenCV. 1st (edn), ISBN: 978-0-596-51613-0. O'Reilly Media Inc., NY, USA (2010).

[0057] Introducing the image processing module 12 (with a separate label "NN" in the header of Fig. 1), we first describe the architecture of the neural network applied therein. A neural network applies a sequence of transformations to the input in order to generate the desired output. These transformations may depend on learnable parameters determined during the training of the neural network. In most applications, these transformations form a layered structure, where the input of a layer is the output of the previous layer. These inputs and outputs of a layer are called features. A neural network used in a 3D lane detection system may include any number and type of layers, including but not limited to convolutional layers, activation layers, normalization layers, etc. The sequence of layers (or more generally, transformations) is defined by the neural network architecture. In the present invention, preferably a so-called stereo lane detection neural network receives as input a stereo image pair (consisting of a first image and a second image), which must be taken into account (processed) by the neural network architecture.

[0058] In the image processing module, the processing can start separately on the first image 10a and the second image 10b, which may be beneficial for low-level filters to focus on a single image, for example to form an edge or color detector. The two branches of the network may or may not use the same filters (weights). Filter sharing reduces the number of learnable parameters (weights). It may therefore be preferable to use a common filter in parallel in the two branches to search for the same features. However, it is also possible that the filters are learned independently in the two branches. This latter is recommended when the first and second cameras are not equivalent.

[0059] In an embodiment of the lane detection system according to the invention using a neural network with an image processing module trained by machine learning (naturally, these features introduced in the context of the system are also applicable to the method framework, without impediments), the features of the two images are combined in the network at least once before the output layer of the image processing module (i.e., a mixing step is inserted). The combination (i.e., mixing) of the features may have already been performed on the input images or in a later layer. The combination (mixing) may be implemented, for example, as a concatenation, addition, multiplication of the features or as a nonlinear function of the input features.

[0060] Assuming that a first feature and a second feature are to be mixed, these features can be represented as tensors that can have height, width, and channel dimensions (or, for example, additional batch, and time dimensions). Often, the mixing occurs only in the channel dimension. In principle, more complex mixing can be defined. In this case, the elements of the tensors to be combined are mixed not only in the channel dimension but also in the spatial dimension.

[0061] In relation to the blending approach introduced above, it is important to note that the output of the neural network is compared (face-to-face) with the ground truth data: blending helps to obtain a reasonable correspondence mapping, so it is the task of the neural network to arrive at an output close to the ground truth based on the data that is the output of the blending (performed during the training process, for example, by selecting appropriate weights).

[0062] This mixing step (sometimes called combination) preferably determines an output of the neural network of the 3D lane detection system that depends on both images (the correspondence mapping is such an output), for example the correspondence mapping data block defined in the output section. The mixing of the first and second camera features also strengthens the correlation between the errors (difference between prediction and ground truth) of the outputs corresponding to the first and second images (for example the first camera detection and the second camera detection defined in the output section). This is essential to improve the accuracy of the 3D coordinates predicted by the lane detection system according to the invention. This can be explained by the principle that a positive correlation between random variables reduces the variance of their difference.

[0063] To strengthen the correlation between errors, we can do the following: while using the neural network, lane boundary markers are detected, but (inevitably) errors occur (left or right direction). Without mixing, it is not explicitly highlighted that the errors are consistent in the first and second images.

[0064] Thus, in an embodiment of the method in which the image processing module applies the blending implemented by a neural network, the neural network has a first branch for the first image (i.e. adapted to be applied to or adapted for use on the first image) and a second branch for the second image. The method includes a combination step in the course of the image processing step, combining the first branch data, which is the first branch output of the first branch, with the second branch data, which is the second branch output of the second branch (instead of this combination step, in a corresponding system, the first branch data is the first branch output of the first branch and the second branch data is the second branch output of the second branch, which are combined in the image processing module). More generally, it can be said that it is preferable that all outputs of the image processing module trained by machine learning have access to both input images (accessible, i.e. both images can affect the output, but not necessarily both affect the output), i.e. the processing of the first image and the second image is combined in the image processing module.

[0065] The 3D distance (depth) of the detected lane boundary points from the camera plane is inferred from the disparity, which is the difference between the detections in the corresponding first and second images. Without blending, the detection errors in the two images are independent (uncorrelated), while blending can reduce the variance of the inferred depth values.

[0066] In relation to the above section, the following remarks are given: Image pairs from several consecutive frames can be processed in the same way as in the frame-by-frame case, without modifying the output of the neural network and subsequent modules. The simplest implementation is to use the original frame-by-frame architecture and concatenate the RGB or grayscale channels of consecutive frames at the left and right image inputs. The target can be chosen to be the original target of the last frame. Information from previous frames (first and second images) can make the output more robust against frame-by-frame noise, e.g. caused by windshield wipers (i.e. the image processing module preferably generates its output for a frame taking into account at least one previous frame). More generally, it can be processed like stereo image pairs, i.e. with a mixed parallel processing and feature combination.

[0067] In summary, in one embodiment, the image processing step (or in the system, the generation of output data by the image processing module) is based on the input data, the actual frames of the first and second images, plus at least one additional frame of the first and second images preceding or following the actual frames, for which subsequent frames must be waited for if they are to be used.

[0068] According to the embodiment of FIG. 1, the neural network of the 3D lane detection system preferably has three characteristic outputs (labeled "NN Outputs" in the header of FIG. 1): 1. First camera detection (first image raw detection data block 14a) 2. Second Camera Detection (Second Image Raw Detection Data Block 14b) 3. Correspondence Mapping (Correspondence Mapping Data Block 16)

[0069] Preferably, these are also characteristic outputs of other embodiments.

[0070] Optional further outputs can be added for additional tasks such as clustering, lane type segmentation, road segmentation, vehicle or pedestrian segmentation, etc. The further outputs are included in the additional outputs block 18 of Figure 1. See details of the outputs below.

[0071] The raw detection is a (neural) representation of the lane boundaries on the image. The lane boundaries can be represented in different ways by the neural network. For example, the representation can be dense (e.g. pixel-wise classification or regression of lane boundary points) or model-based (e.g. polynomials, spline parameters, see details in the introduction of the model-based embodiment). The output of the image processing module of the 3D lane detection system can be any representation, and several embodiments showing exemplary representations of the raw detection are introduced below.

[0072] It should be noted that the representation of the raw detection is not limited by the present invention, but is intended to show that, with the help of exemplary embodiments, various representations are possible within the framework of the present invention.

[0073] The architectures introduced in the previous section produce first and second camera detections that are correlated (see above for details on correlation enhancement) since they have access to features from other views and / or mixed features. Having access to information from the other camera allows us to improve the detection quality of the images. However, this correlation between detections remains small unless the correlation is explicitly enhanced during training, for example by an appropriate loss function. To obtain 3D consistent detections, we use a loss function that enhances the correlation between detection errors.

[0074] The raw detections may represent only the location of the (centerlines of) the lane boundaries, or may describe other properties such as the orientation of the lane boundary segments and the type of separator (see below for an example showing how this information is optionally built into the inventive framework). Note that some properties of the separators, e.g. lane type, can be more naturally incorporated into the corresponding mapping output.

[0075] A comment on the loss for raw detection is given below: During training, the raw detection may obtain a representation loss (see Loss (Representation) modules 32 and 34 in Fig. 1) that characterizes the difference between the ground truth representation of lane boundaries and the raw detection. The mathematical form of the loss function is selected according to the representation. For example, if the representation corresponds to a regression problem (see also below), a natural choice might be the Euclidean (L2) distance or the Manhattan (L1) distance (L1 and L2 losses can also be used in model-based embodiments, see below; however, in that case the difference in model parameters occurs in the loss and not in the distance of the image points). Here, L p refers to the distance related to the p-norm of a vector field. For example, if a and b have components a i and b i If a and b are vectors with p-norm, then the distance between a and b can be characterized by the p-norm of their difference.

[0076]

number

[0077] In the loss function, the pth power of the distance is usually used. p See equation (6) in the loss example.

[0078] If the raw detection results from a classification problem, a standard classification loss can be applied, for example the softmax loss (Equation (9)) or the SVM loss (Support Vector Machine loss). In general, any convex function can be applied as the loss, which is minimized when the predictions are equal to the ground truth values.

[0079] This representation loss, in contrast to disparity loss and 3D projection loss, does not directly encourage the network to produce correlated detection errors.

[0080] Another characteristic output of the 3D lane detection network is the correspondence mapping, which is introduced below. The correspondence mapping determines the correspondence between lane boundaries in the input first and second images, preferably represented as raw detections, thus eliminating the need to run a separate stereo matching algorithm and significantly reducing computation time. There may be various solutions for the neural representation of the correspondence between the detections in the first and second images. Some clear examples of correspondence mapping are described below in connection with some embodiments.

[0081] Correspondence mapping can be seen as a correspondence between abstract entities (e.g. lane boundaries) that may or may not be visually visible in the input images, but with contextual information we can predict the descriptors (e.g. raw detections) that represent these abstract entities. It is therefore a completely different approach from traditional vision-based stereo matching solutions. Moreover, the correspondence mapping is done within the neural network itself (provided as output) to provide the correspondence between detections in the stereo image pair. These principles are generally applicable to image processing modules trained by machine learning. It differs from traditional explicit stereo matching in the following ways: No hand-crafted rules are used for matching: the correspondence rules are learned by the neural network during the training phase (given as separate outputs). Correspondences are given between detections (characterizing lane boundaries), not between image patches (note that standard stereo matching approaches search for correspondences between local patches of images, see also next point). · The correspondence is based on lane boundary descriptors (which can be chosen in various ways) and not on local similarities of image patches, so it is more reliable even in occluded regions (areas) and lane boundary parts between lane markers and is more robust against image noise. According to the invention, preferably lane markers are not searched for themselves - for example, if they are dashed and not continuous. Rather, they are connected and the continuous lane boundaries thus obtained are ground truth annotations and are therefore searched for and output by the neural network (see, for example, the example where a sample output of the neural network is shown, where the results for continuous lane boundaries can also be observed). This approach is also advantageous in distant parts of the image where dashed boundary markers are difficult to capture. Thus, a high probability or label of a lane boundary can be given in the respective embodiment for the positions between the previous lane boundary marker and the following lane boundary marker. Lane boundaries that are considered to be of simple shape are searched for. This is usually not too complicated to implement on the diagram. The most important separator lane boundaries are usually straight or gently curving. This property of lane boundaries can also be useful in case of occlusion areas.

[0082] In principle, the correspondence mapping could be a dense one-to-one matching between pixels on the two images. However, as detailed below, preferably, a much simpler mapping suffices for our 3D lane detection system. For example, it could be chosen to describe the correspondence between regions around lane boundaries in the two images (see introduction to pairing masks and stereo instance segmentation).

[0083] Regarding the loss of correspondence mapping, see below. The correspondence mapping (i.e. the neural network that can output the correspondence mapping data) is trained on the ground truth data during the training procedure (see the Correspondence Mapping Loss module 36). The loss function compares the ground truth correspondences with the predicted correspondences and has a minimum if the two correspondences are comparable. Naturally, based on the ground truth, the correspondences can be given exactly in any representation of the correspondence mapping. The correspondence mapping depends on both images. The correspondence loss function therefore encourages the network to generate a filter that is sensitive to the features of both images.

[0084] Thus, the system for training an image processing module applied in a lane detection system consequently comprises a step of a 3D projection loss module 42 adapted to control the 3D lane detection data blocks during the training of the image processing module by modifying the learnable parameters of the image processing module (in a corresponding method for training an image processing module the following control steps are performed; in the following sections reference numbers of the embodiment of FIG. 1 are given, in which embodiment a neural network is used. The loss values ​​are summed and need to be minimized during training). Furthermore, optionally, it also comprises a step of a correspondence mapping loss module 36 adapted to control the correspondence mapping loss of the correspondence mapping data during the training of the image processing module by modifying the learnable parameters of the image processing module.

[0085] The system may comprise an image space loss module 38 adapted to control the first image space sensed data block and the second image space sensed data block during training.

[0086] According to the above configuration, beyond the correspondence mapping loss, the image space loss and / or the 3D projection loss can be controlled. Furthermore, for example, in the embodiment of Fig. 1 (i.e., in an embodiment where a situation exists for using disparity), the system for training the image processing module further comprises a disparity loss module 40 adapted to control the disparity data block based on the ground truth data 28 during training.

[0087] In the following embodiment of the system (and method) for training, the image processing module applied to the system for lane detection is fully differentiable. In this embodiment, the correspondence mapping data is contained in a correspondence mapping data block, the correspondence mapping loss module is adapted to control the correspondence mapping loss of the correspondence mapping data block, and the image processing module 12 is implemented by a neural network. The system comprises: a first representation loss module 32 adapted to control a first representation loss of the first image raw detection data block 14a during training; and The second representation loss module 34 is adapted to control a second representation loss of the second image raw detection data block 14b during training.

[0088] In summary, in this embodiment, the machine learning algorithm is implemented by a neural network, all transformations of which are differentiable, and the pairing and 3D projection steps are differentiable with respect to their inputs according to the predictions of the neural network. These are exploited during training, allowing the computation of the gradients of the 3D projection loss and the image space loss with respect to the parameters of the neural network.

[0089] Furthermore, in this embodiment, the system for lane detection is fully differentiable in the sense that all loss functions (including but not limited to 3D projection loss and image space loss) are differentiable with respect to all learnable parameters of the machine learning algorithm during training. This is achieved, for example, by a differentiable image processing module trained by machine learning, implemented by a neural network characterized in that all transformations (layers) are differentiable and further, the pairing step and the 3D projection step are differentiable with respect to their inputs according to the predictions of the neural network utilized during training.

[0090] The image processing module (neural network) can use any representation for the detections and for describing the correspondence between the detections on the two images (see below for an exemplary embodiment). The output is forwarded to the pairing module 20 (i.e., as indicated by the arrows, the NN outputs 1st image raw detection data block 14a, 2nd image raw detection data block 14b and correspondence mapping data block 16 run from these blocks to the pairing module 20). The pairing module is adapted to map the representations into a standardized form, the output of which is an image space representation of the detection pairs (see the table below for an example). This standardized representation is the input of the disparity block.

[0091] As noted above, the representation is arbitrary, and tensor (matrix)-based embodiments are described below (see, e.g., model-based embodiments for representations that differ from tensor-based representations). In such method and system embodiments, the output data is generated for (i.e., the output data can be interpreted in terms of) a coordinate grid represented by a coordinate grid tensor having multiple grid tensor elements (in a tensor-based approach, the coordinate grid is naturally represented by its respective tensor), the first image raw detection data block is a first image raw detection tensor, the first image raw detection tensor corresponds to a coordinate grid tensor and has a plurality of first tensor elements, the plurality of first tensor elements including respective first distance values ​​measured from corresponding grid tensor elements of the closest first lane boundary points (i.e. the points of the closest lane boundary in the search direction) of the first lane boundary group in the first search direction of the first image (here and in the following section it is specified that the raw detections are also represented by respective tensors, the resulting tensors being preferably downsampled or upsampled compared to the original image examined, e.g. as shown in the example) (as disclosed in detail below, in the tensor-based approach distances are used to characterize the placement of the lane boundaries. The distances are measured between the respective points of the coordinate grid and the closest lane boundary points in the search direction. Thus, in a left and right image pair these are horizontal distances); the second image raw detection data block is a second image raw detection tensor, the second image raw detection tensor corresponding to a coordinate grid tensor and having a plurality of second tensor elements, the plurality of second tensor elements including respective second distance values ​​measured from corresponding grid tensor elements of nearest second lane boundary points of the second lane boundary group in a second search direction in the second image (in case of modification, the search direction is preferably given along an epipolar line and may be left or right direction thereon, see also R, L search directions below); the correspondence mapping data block is a correspondence mapping tensor that corresponds to the coordinate grid tensor and has a number of third tensor elements that characterize whether the closest first lane boundary point (in the first search direction, as described above) and the closest second lane boundary point (in the second search direction) correspond to the same lane boundary (i.e. correspond to the same lane marker in the first and second images, thus determining the correspondence between the first lane boundary group in the first image and the second lane boundary group in the second image, which in fact correspond to the same lane boundary, i.e. correspond to the same lane boundary marker if that lane boundary marker provides the basis for the lane boundaries); Using the coordinate grid tensor, an image space detection pair tensor is generated as a data block of image space detection pairs, i.e., this data block is also represented as a tensor.

[0092] The raw detection tensors of the first and second images, as well as the corresponding mapping tensors, all correspond to coordinate grid tensors, i.e. they have the same dimensions and define the values ​​for each point of the coordinate grid tensor (see also the example tensors in the table below).

[0093] The preferably used disparity data blocks, as well as the 3D lane detection data blocks, can be represented in any format: as Tables 3a and 3b, 7a and 7b show, they are represented, for example, in the form of tensors, but Tables 10a and 10b also show that they can simply be represented as lists (although data blocks of various dimensions, such as matrices, lists, etc., are considered tensors).

[0094] In the tensor-based approach, the correspondence mapping tensor - see for example the embodiment with a pairing mask of one correspondence mapping tensor and the embodiment with stereo instance segmentation of two, i.e. separate correspondence mapping tensors for each of the first and second images - consists of information about whether the investigated closest lane boundary points are points of the same lane boundary or not. The characterization of the third tensor element can be done for example with the help of probability values ​​as shown in some examples. Other characterization methods are also conceivable.

[0095] Below, we introduce the characteristics of the pairing module 20. The pairing module 20 derives an image space representation of the detection pairs from the output of the image processing module 12 (i.e., from the raw detections and correspondence mapping of the two images).

[0096] How the image space representation is determined depends on the raw detections and the corresponding mappings used in a particular embodiment. However, it is important that the transformation to image space must be differentiable (to maintain end-to-end differentiable functionality; as mentioned before, this expectation can be met). The gradients then flow naturally through the pairing module to arrive at features that contribute to predicting the detection pairs (thus, the training of the neural network can also be improved with loss data that occurs in later stages of the pairing module; see the dashed lines for the flow of loss data).

[0097] Depending on the particular embodiment, the gradients from the pairing module may or may not flow into the correspondence mapping (Figure 1 shows such connections that may be utilized accordingly). In the example below, the pairing module uses ground truth correspondence mapping data during training, so the gradients only flow into the raw detections of the left and right images. Note that the correspondence mapping output is trained directly by the correspondence mapping loss. However, other examples can be created where the gradients flow into the correspondence mapping. The difference is whether the 3D loss of the 3D projection module is used only to refine the image space detection or to refine the correspondence mapping as well.

[0098] Optionally, an image space loss function can be assigned to the image space representations corresponding to the first and second images (see Loss (Image Space) module 38 in FIG. 1). This loss characterizes the distance between the image space representation of the raw detections and the actual image locations of the lane boundaries. In some embodiments, the representation loss of the raw detections is equivalent to the image space loss.

[0099] The image space detection pairs output by the pairing module 20 are processed by a disparity module 22, which is described below.

[0100] The disparity is calculated as the difference between the image coordinates of the first and second image detection along the epipolar direction, which in most cases is the horizontal direction. So in a horizontal stereo setup, parallax=u left -u right (1) where u is the image coordinate of the detection along the horizontal axis. The disparity is calculated for every detection pair and forwarded to the 3D projection module together with the image coordinate of the detection on the left side, for example.

[0101] A loss function can be introduced into the disparity module to force the network to generate correct disparity values ​​(see Loss (Disparity) Module 40 in FIG. 1). This loss enforces a correlation between the first and second camera detections.

[0102] In the following, the characteristics of the 3D projection module 24 are also introduced. The 3D projection module 24 calculates the 3D coordinates of the detections (for example in the camera coordinate system corresponding to the car holding the camera). The inputs are the image space representation of the detections, the disparity and the calibration parameters of the camera (the latter obtained from the calibration module as calibration data 26). The 3D positions of the lane boundary points can be determined from the corresponding image space positions by triangulation. In the simplest case of a rectified camera, the corresponding image space positions differ only along the epipolar direction, and this difference is characterized by the disparity. To illustrate an embodiment of the invention, the relevant equations of the calculation based on disparity are introduced below, but the method of triangulation is also applicable to a more general (unrectified) setting (see above).

[0103] Below are the formulas for determining the coordinates. The indices used below correspond to the horizontal stereo configuration (for horizontal / vertical configurations see below). Depth (coordinate Z) is defined as the distance of the detection from the rectified camera plane and is calculated as follows: Z=Bf x / parallax(2) where f x is the focal length of the rectified camera along the epipolar direction, B is the distance between the two cameras (baseline distance), generating a first image 10a and a second image 10b (often: left and right images), and “disparity” is the disparity corresponding to the detection.

[0104] If the principal points of the two rectified cameras are different, then parallax correction is required. Z=Bf x / (parallax-(c x,left -c x,right ))(3) Here, (c x,left ,c y,left ) and (c x,right ,c y,right) are the principal points of the left and right cameras. The points on the image plane represent a ray in the camera coordinate system that passes through the origin. The principal points correspond to a ray perpendicular to the camera plane and are represented as real numbers in pixels (which may or may not coincide with pixels, since the coordinates used for pixels can have non-integer values).

[0105] The X and Y coordinates of the detection are calculated, for example, from the left camera detection coordinates and depth (already calculated). X=(u left -c x ) / f x Z Y=(v left -c y ) / f y Z(4) Here, (c x ,c y ) is the rectified left camera principal point.

[0106] A 3D projection is calculated for every detection and is the primary output of a 3D lane detection system.

[0107] Regarding the 3D projection loss, see below: The main goal of a 3D lane detection system is to provide accurate 3D information about lane boundaries. Therefore, it is preferable to assign a loss function to the 3D projection detection (see 3D projection loss module 42 in Figure 1) and train the neural network to make predictions that are projected to the correct 3D location. This step is important to reduce noise in the 3D detection from the neural network. Since the 3D projection is differentiable (and the disparity module), the loss can be propagated back to the early layers of the neural network (i.e., the neural network can be trained to reduce noise in the 3D detection).

[0108] Obtaining ground truth annotations can be found in: Applying neural networks (generally image processing modules trained by machine learning) requires a large amount of training data to tune the network's learnable parameters. The training data consists of stereo images with labels describing the lane boundary positions in 3D space, or equivalently their projected positions in the 2D camera plane. These true labels are called ground truth. They can be obtained by manual annotation, i.e. by determining the lane boundary lines in real images by human supervision, possibly using some conventional or neural network-based detection algorithms as before.

[0109] Moreover, training data for neural networks adapted for 3D lane detection can be generated in simulated environments, for example by simulators capable of producing photorealistic images (e.g., in simulated environments lane boundaries are parameterized so that they can be labeled). The 3D lane boundary locations can also come from processing of lidar point clouds (as lane boundaries are objects protruding from the road surface), traditional stereo algorithms, dense maps, or a fusion of the above methods (i.e., any combination).

[0110] Below we refer to some aspects of training a neural network model. The parameters of the neural network are optimized by minimizing the weighted sum of all loss functions introduced in the system (and the corresponding method) using an appropriate optimization technique. Any initialization scheme can be used for the parameters before optimization.

[0111] Below some details of certain further embodiments of the 3D lane detection system are given. Using the architecture introduced above, several slightly different embodiments can be realized. The following embodiments may differ in how the lane boundaries are represented by the neural network and in the representation of the correspondence mapping.

[0112] Furthermore, some additional tasks can be assigned to the neural network.

[0113] The orientation of the stereo cameras may be horizontal, vertical, or unconstrained. Therefore, the input images are generally referred to as the first image and the second image. Without loss of generality, the embodiment introduced below assumes a horizontal stereo setup, and during rectification, the epipolar line is chosen to be horizontal. In this embodiment, the first image and the second image are the left and right images.

[0114] The embodiments related to raw detection are described in more detail below. See the table below, which specifies example tensors of actual data at particular stages of the systems and methods in various embodiments. In this embodiment, the raw detection of lane boundaries in the left and right images is represented by a scalar value (or more generally a tensor) D with subscripts referring to the images. right / left (see, for example, Tables 1b and 1c).

[0115] Each element (i,j) of the 2D grid corresponds to an image location I i,j =(u i,j ,v i,j ), which forms a coordinate grid (i.e., the coordinate grid can be described by a matrix of data, generally also a tensor in this embodiment). i,j ,v i,j) may or may not be equal to (i,j) based on the downsampling or upsampling of the neural network introduced, for example, by pooling layers, strided convolutions, or dilated convolutions. Thus, the coordinate at a particular point of the coordinate grid is determined by the structure of the neural network, in particular its downsampling or upsampling factor. Additionally, an offset can also be manually introduced to the target, which makes it possible to change the (i,j)->(u i,j ,v i,j ) mapping is also changed. For example, stride along the horizontal direction x There is a stride along the vertical direction y For a network with (stride indicates the upper and lower sampling factors), u i,j =i·Stride x +Offset x v i,j =j·Stride y +Offset y (5) It is.

[0116] I i,j is the offset x =Stride x / 2 and offset y =Stride y / 2, the (i,j)th [stride x ×Stride y ] represents the center of a cell. An example coordinate grid is shown in Table 2a.

[0117] Raw detection D right / left represents the (signed) relative distance of the lane boundary points on the image, measured from the coordinate grid (points) along the epipolar direction. We can define four independent raw detection tensors for the left and right images (subscripts) with left (L) and right (R) search directions (superscripts): R left , D R right , D L left , D Lright , whose elements represent the relative positions of the nearest lane boundaries in the search direction. A neural network for a 3D lane detection system according to one embodiment of the present invention can predict all four independent detection tensors, or in another embodiment, only two or three of them. For example, D R left and D L right Only one detection tensor is used: one for the left (first) image and one for the right (second) image. By choosing which raw detection tensor to use, the correspondence mapping is also determined. D R left and D L right If you select the corresponding mapping P RL must also be used (see below for the definition of the correspondence mapping with various direction indices). Thus, in one embodiment, the first search direction and the second search direction are selected as the left direction in the respective first image and / or second image, or as the right direction in the respective first image and / or second image, respectively. This can be interpreted not only for left and right images, but also for any first and second images. According to this definition, all cases using two, three or four raw detection tensors are used for the calculation.

[0118] The loss function (of the loss (representation) modules 32 and 34 corresponding to the first and second image raw detections 14a, 14b in FIG. 1) penalizes the difference between the predicted relative position of the detection and the ground truth relative position. The simplest choice of loss function is:

[0119]

number

[0120]

number

[0121] To force the network to predict reliable detections, in principle we need to also predict the learned foreground output corresponding to each relative detection tensor. In one embodiment, the raw detection is a pair of D and F tensors, which can be represented as a single output with multiple channels or as multiple outputs. However, we introduce a compact solution where this additional foreground output is not necessary, as it can be incorporated into the correspondence mapping (see below). In this case, the raw detection is simply the raw detection tensor D.

[0122] A note related to the above detail: In the above it is assumed that the raw detections have only one component corresponding to the distance from the coordinate grid to the lane boundary along the epipolar line. Relative detections may have another component corresponding to the orthogonal direction. If the orthogonal component is not regressed but is chosen as a constant, then the offset of the coordinate grid y This allows you to set different offsets y We can increase the density of predictions at a fixed stride by introducing multiple D outputs corresponding to,. In what follows, we consider scalar relative detection with a single offset.

[0123] Below are some details of an embodiment of the correspondence mapping. The correspondence mapping finds the correspondence between the left image raw detections and the right image raw detections. There are four possible combinations of left and right image raw detection tensors, resulting in four different correspondence mappings (P RR , P RL , P LL , P LR ), where the two superscripts indicate the search directions in the left and right images, respectively. - P RR D R left and D R right Describe the correspondence between. - P LL D L left and D L right Describe the correspondence between. - P RL D R left and D L right Describe the correspondence between. - P LR D L left and D R right Describe the correspondence between.

[0124] This is illustrated in Figure 2, which is an illustration of the various correspondence mappings in this embodiment, specifying which correspondence mappings are used for which raw detections.

[0125] In one embodiment, the correspondence mapping data block is a binary mask (pairing mask) described by a tensor defined on the same 2D grid as the raw detections, indicating which tensor coordinates produce detections that physically correspond to the same lane boundaries. The binary refers to the number of classes represented by the pairing mask. It describes a 2-element set where one element is interpreted as background and the other as foreground. For example, a convenient choice for the binary mask value is {0,1}, but any other value or representation (e.g., probabilistic, see below) could be chosen. In this example, D left and D right The mask takes values ​​of zero at the indices where the tensor elements point to different physical lane boundaries, and 1 when the tensor elements point to the same instance. The mask values ​​are interpreted as background, i.e., no pairs are detected from these indices (i.e., each raw detection tensor element does not point to the same lane boundary), and 1 as foreground, since detection pairs are predicted from these indices.

[0126] In one embodiment, the pairing mask takes discrete values ​​in the annotations (ground truth), but the network output can be chosen to be the actual value, which is interpreted as a score representing how certain the network is about a point being foreground in the pair prediction sense (such an approach is shown in the example below). In this case, a point is accepted as foreground if the neural network output of the correspondence mapping exceeds a certain threshold.

[0127] In the probabilistic approach, the elements of the pairing mask are real numbers in the probability range (usually 0 to 1), D left and D right represents the probability that points to the same physical lane boundary. Foreground and background classes are recovered by a threshold (usually chosen to be greater than or equal to 0.5, e.g. 0.5 or 0.75) selected from a probability range, so that probabilities higher than the threshold are considered foreground; probabilities lower than the threshold are considered background. This parameter allows fine-tuning the neural network with a precision-recall function. A higher threshold is expected to result in smaller recall for foreground regions and higher precision. The raw probability before thresholding can be transferred to the next module and serves as the confidence of the detection.

[0128] Learning pairing masks can be viewed as either a regression or a classification problem. In a regression problem, the prediction is a single real value on a 2D grid, which can be regressed, for example, by a simple L1, L2 regression loss, etc. In a classification approach, the output is a 2D vector defined on a 2D grid, whose first component (e.g., represented in the channel dimension) represents the probability of having background, and the second component corresponds to the probability of foreground. Being a dependent output, the correspondence is clearly defined by either the probability of foreground or background. Often a classification approach is recommended (giving a normalized score on top of the classification to foreground or background), but a regression approach can also be used. This can be assigned a traditional classification loss, for example, softmax loss, focal loss, etc. In our example, we chose a classification approach using softmax loss.

[0129] The softmax loss is:

[0130]

number

[0131] Therefore, the raw detections preferably give the distance (along the epipolar direction) of the nearest lane boundary point from the respective coordinate grid point. Since the foreground is not specified using a window size, but with the help of a correspondence mapping tensor, every tensor element of the raw detections tensor is given a value. In this case, this tensor is a binary mask, describing which raw detections at the same point of the coordinate grid correspond to the same lane boundary.

[0132] Examples of pairing masks and raw detection tensors are shown in Tables 1a–1c.

[0133] Based on the above, in this embodiment, the correspondence mapping tensor is a pairing correspondence mapping tensor (which is a pairing mask), has a pairing probability value (the probability value under the correspondence mapping data is the pairing probability value) in each of its third tensor elements, is characterized by - i.e. corresponds to -, the closest first lane boundary point and the closest second lane boundary point correspond to the same lane boundary (i.e., the lane boundaries in the first and second images - the point under investigation - correspond to the same lane boundary. In other words, it is determined based on the images that the same lane boundary can be observed in the first and second images), and is located within a window size from the grid tensor element corresponding to each third tensor element. the probability that a third tensor element will be selected as the foreground third tensor element (for which the distance value is also measured), the respective probability value of which exceeds a predefined first threshold (see details of the foreground designation in the embodiments; also the embodiment with a binary pairing mask is considered as such an embodiment, see next section), and the foreground third tensor element as well as the corresponding first and second tensor elements are used in the pairing step (in the pairing module) as output data of the image processing step (of the image processing module), i.e. are kept for further calculations (see also the example shown in the table; the foreground is selected for further calculations, the other values ​​of the tensors are not taken into account).

[0134] The optimal value of the window size depends on the resolution of the input image. It is usually between 1 pixel and half the image width (in pixels). Preferably, it is chosen to be smaller than half the typical lane width (the lateral distance between adjacent lane boundaries) seen at the bottom of the image, which depends on the camera calibration.

[0135] Therefore, the pairing probability value above is high if the same border can be seen in both the first and second image within a window size from their respective coordinate grids. Hence, the distance value given in the raw detection tensor is maximized by the window size (see also equations (7) and (8) above).

[0136] Examples of pairing correspondence mapping tensors are shown in Tables 1a and 4a-4c (in the examples, these are simply labeled "correspondence mapping"). The "pairing" attribute in the name has the sole purpose of identifying the approach; tensors could also be named in other ways.

[0137] The RR, LL and RL types of foreground areas of the pairing mask described above have a transparent visual interpretation, which is interpreted in relation to Fig. 3A-3C. In Fig. 3A-3C, lanes from the left and right images are observable in the same view. This type of view comes from the integration of these images. The foreground areas of the pairing mask are shown in a unified view. The two images (first and second images) that give the starting point are of the same size. That is, a frame of the same size corresponds to them. In this frame, both images can be displayed, especially the lane boundaries. Also, since the images are loaded into the same frame, the contents of the original images are slightly shifted from each other. The lane boundaries can be represented by various parameters of the first and second images. According to the invention, the coordinates of the lane boundaries can be derived not in the two dimensions of the image, but also the depth coordinates can be determined.

[0138] The correspondence map can also be interpreted in the frame defined above. It preferably gives for every grid point a respective value according to its definition, i.e. taking into account the search direction and window in one embodiment, for example. In the stereo instance segmentation embodiment, the correspondence information is defined with the help of labels. Other ways of representing the correspondence information are given below:

[0139] So, for example, in the example of FIG. 3B, the lane boundary in the left image (solid line) will be closer than the lane boundary in the right image (dashed line).

[0140] Figure 3A shows that the foreground regions 100a, 100b corresponding to the RR-type pairing masks (for pairs of lane boundaries from the first and second images, respectively) are regions on a 2D grid, from which the lane boundary closest to the right side in both the left and right images corresponds to the same actual 3D lane boundary (the centers of the lane boundaries are shown in the figure as lane boundary projections, which are projections of the 3D lane boundary onto the first and second images, see below).

[0141] Thus, in FIG. 3A , lane boundary projections 102a and 104a (shown as dashed lines; in the case of lane boundaries, the lines in the diagram preferably correspond to the centerlines of lane boundary markers of finite width; whether the original lane boundaries are dashed - e.g., with long interruptions - or continuous or other type is irrelevant for visualization) correspond to the lane boundaries of the right (second) image, and lane boundary projections 102b and 104b (shown as solid lines; the other lane boundary projections 102a, 104a are also continuous, but are shown as dashed lines to distinguish them from the continuous views) correspond to the lane boundaries of the left (first) image. From these lanes, lane boundary projections 102a and 102b correspond to the same lane boundary (the dashed and continuous lines do not correspond to the type of lane boundary, but they visualize the same lane boundary); both of these lane boundary projections 102a, 102b are to the right of the foreground region 100a (the same applies to lane boundary projections 104a, 104b and foreground region 100b).

[0142] In FIG. 3A, distances 106a and 106b, as well as distances 108a and 108b, can also be observed. These distances 106a, 106b, 108a, 108b are directed from points in the foreground regions 100a, 100b. The starting points correspond to points in the coordinate grid. The end points of the distances lie on the lane boundaries that can be observed in the left and right images, respectively. Thus, distance 106a is D R right , and distance 106b is D R leftvalues ​​of . As shown in Fig. 3A, if more lane boundaries are visible to the camera, more foreground regions and more distances in the pairing mask can be specified. The distance between the boundary of the foreground regions 100a, 100b and the closer lane boundary projection from the two projections must be smaller than the resolution of the grid (i.e., the distance between grid points). However, in Fig. 3A-3C - for illustration purposes - a gap is shown between this lane boundary projection and the foreground region. Note that on top of the maximum distance, a minimum value for the distance can also be defined.

[0143] Other types of foreground regions 110a, 110b are observable in FIG. 3B. In this case, a LL type mask is shown where regions can be assigned whose nearest lane boundary to the left corresponds to the same physical lane boundary in both the left and right images. In this case, lane boundary projections 112a and 114a (marked with dashed lines) correspond to lane boundaries in the right image, and their pair, lane boundary projections 112b and 114b (solid lines), are observable in the left image. These are different lane boundaries from those in FIG. 3A, and the pair is clearly to the left of the foreground regions 110a, 110b of the pairing mask. As indicated by the multiple foreground regions 110a, 110b, only the lane boundary corresponding to the pair must be to the left of the respective foreground regions 110a, 110b. Other reference symbols in this figure are similar to those in FIG. 3A, namely the distances 116a and 116b that refer to the respective lanes of the pair from the points of the coordinate grid. Distance 116a and 118a is D L right while the distances 116b and 118b correspond to D L left Corresponds to.

[0144] The foreground regions 120a and 120b of the third type of pairing mask are observable in FIG. 3C. The foreground regions 120a, 120b of the RL type pairing mask are regions where the lane boundary closest to the right in the left image corresponds to the lane boundary closest to the left in the right image. This region looks like a slice (set of slices) bounded by the lanes in the left and right images. According to the search direction, the distances 126a and 126b (DL right and D R left ), and distances 128a and 128b point in different (opposite) directions in Fig. 3C. As shown in Fig. 3C, foreground regions 120a and 120b are shown between each pair of lane boundaries. Thus, foreground region 120a is between lane boundary projection 122a (dashed line, lane boundary in right image) and lane boundary projection 122b (solid line, same lane boundary in left image), and foreground region 120b is between lane boundary projection 124a (dashed line, further lane boundary in right image) and lane boundary projection 124b (solid line, further pair of lane boundaries in left image).

[0145] The structure of the foreground region of the LR type mask is similar to that of the RL type mask: the region where the lane boundary closest to the left in the left image corresponds to the lane boundary closest to the right in the right image. x,left -c x,right If >= 0, this is an empty mask. This is c x,left >=c x,right In this case, the disparity is greater than or equal to zero, which means that for the right camera, all objects are further to the left than for the left camera, so there is no LR region, or more precisely, an empty mask.

[0146] Additionally, a restriction on the maximum distance to any of the lanes can be added to the pairing mask, to ensure that pairs are predicted only from regions close to the lane markers and not from regions far from the lane markers (e.g., vegetation, buildings, etc.). This maximum distance should be chosen according to the window size applied in training the raw detections, if the neural network does not predict foreground regions for the raw detections.

[0147] The foreground regions of the RR, LL and RL type masks are shown in Fig. 3 as well as Fig. 4. Figs. 3 and 4 show the x,left =c x,right This is the standard choice for correction. x,leftIncreasing the value of c shifts the projection of the lane boundary in the left image to the right. Hence, the distance between the left and right projections of the lane boundary increases, e.g., the foreground mask of the RL type becomes wider. The minimum disparity that can be displayed in an image is c x,left -c x,right That's all. x,left Decreasing the value of shifts the projection of the lane boundary in the left image to the right. Therefore, the distance between the left and right projections of the lane boundary decreases, for example, narrowing the foreground mask of the RL type. Furthermore, the right projection of the lane boundary shifts the far region (c x,left >=c x,right In the case of , which does not occur, an LR type foreground mask appears. Similarly, in the case of c x,right Increasing or decreasing will shift the projection of the lane boundaries in the right image to the right or left, respectively.

[0148] Figures 4A to 4F show x,left =c x,right Figure 1 shows an illustration of the foreground regions of pairing masks of type RL, RR and LL (highlighted by white bands) on a realistic image of a car. Corresponding to the foreground regions of the pairing masks, the same regions are successively highlighted in the left and right images. It can be seen that the localization of the pairing masks is absolute, i.e. it is the same in the two pictures, whereas the content of the picture differs depending on whether the picture was taken with the left or right camera (see for example the car in the next lane, and the lane boundary section that can be observed in front of the car taking the picture).

[0149] In Fig. 4A and 4B, the foreground of the RL mask is between the left and right projections of the same 3D lane boundary (left and right projections mean the visualization of lane boundaries in the left and right images are projections of the same real 3D boundary markers). Hence, the shape of the pairing mask is constrained by the lane boundary markers to its left and right (see Fig. 4C-4F). In Fig. 4A and 4B, the foreground regions 202a and 202b of the RL mask correspond to the lane markers 204 and 206 (not many lane markers are visible in Fig. 4A and 4B, see also Fig. 4C-4F for the same lane boundary markers 204 and 206), which are the lane boundaries along which the imaging vehicle travels. Another foreground region 200 of the RL mask is shown in Fig. 4A and 4B. It is shown that preferably all lane boundaries in front of the imaging vehicle are identified by the method according to the invention (occluded parts of the foreground 200 are also identified).

[0150] The foreground regions 200a, 200b, and 200c of the pairing masks are narrower in Figures 4A and 4B than in Figures 4C-4F (see their reference symbols below), especially when they are at a far distance from the car creating the image. This is because the foreground width of the RL mask is limited by the disparity. However, the width of the foreground regions of the RR and LL masks is limited by the distance of the lane boundaries. Furthermore, the width of all foreground regions is also limited by the window size.

[0151] Another consequence is that the pairing mask will be very close to the lane boundary markers. In this example, the foreground region is exactly the region between the two projections of the lane boundary markers (from the left and right images, this interpretation and others can be used for the general first and second images as well). That is, the foreground region is approximately bounded by the centerline of the lane boundary markers (see lane boundary marker 206 in Fig. 4B, close to the car, as a good example). Approximately bounded means that the foreground region starts at the grid point closest to the lane boundary marker centerline, and therefore cannot be observed near the centerline (in this respect, Figs. 3A-3C use schematic diagrams, allowing us to distinguish between the foreground region and the lane boundary projections).

[0152] In Figures 4C and 4D, it is observable that the foreground regions 208a, 208b and 208, 210a, 210b and 210 of the RR mask (same approach for the foreground regions 208a, 208b, 210a and 210b as above for the foreground regions 200a and 200b, the regions with reference numbers "a" and "b" are the regions surrounding the image capture vehicle) are located to the left with respect to both the respective left and right lane boundary markers 204 and 206 (observable mainly in the center lane occupied by the image capture vehicle). The foreground of the pairing mask is closer to the lane boundary in the right image than in the left image according to the viewpoint of the cameras corresponding to the left and right images (see also Figure 3A).

[0153] In Figures 4E and 4F, the foreground regions 212a, 212b and 212, 214a, 214b and 214 of the LL mask are on the right side of both the left and right projections. The foreground of the pairing mask is located closer to the lane boundary in the left image than in the right image.

[0154] Figure 4G shows, as an example, the left image space detection averaged from three foreground regions RL, RR and LL, which corresponds to the results given in Figures 4A-4F. In Figure 4G, good detections 220a and 220b of the lanes occupied by the image capturing vehicle (clustered close to the centerline drawn by the lane markers) are observable. The detections 220 of all other lane boundaries are also good and occlusions are also resolved.

[0155] In Fig. 5A and 5B, graphs are shown showing 3D lane detection data obtained from the results shown in Fig. 4A-4G corresponding to the functions Y(Z) (side view) and X(Z) (top view). It is clear from the results of Fig. 5A and 5B that good quality results can be obtained by the respective embodiments of the present invention. The result shown in Fig. 5A shows a slowly rising road in the camera coordinate system due to the pitch angle of the camera, and in Fig. 5B shows almost parallel lanes. The lane boundaries corresponding to the lane markers 204 and 206 correspond to the points in the interval X of [-4 meters, 4 meters] in Fig. 5B. The optical axis of the camera is preferably not parallel to the road surface (fixed by the desired coverage of the environment by the camera). Therefore, the present invention is preferably suitable for online calibration of the pitch and roll angles of the camera. Thus, the present invention can compensate for the pitch and roll movements of the vehicle.

[0156] In Figures 5A and 5B, the detections are organized in an approximately quantized pattern (with increasingly larger gaps between detections at greater distances) due to perspective distortion.

[0157] The high quality of the results is further supported by Figures 6A and 6B. These are the results of 3D projections from the SGM with respect to Y(Z) and X(Z) (Hirschmuller, Heiko. Accurate and efficient stereo processing by semi-global matching and mutual information. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) 2005). For a fair comparison, the image space detection on the left of Figure 4G is used, where only the disparity estimated from our invention is replaced by the results of the SGM algorithm in equations (3-4) to evaluate the 3D location of the lane boundaries. Figures 6A and 6B show a lot of noise (there are many outlier points), i.e. the results of Figures 5A and 5B are much better. The lane boundaries in the occluded region are projected to incorrect positions in the camera coordinate system using SGM disparity, and the effective detection range is much shorter (20-40 m in Figure 6B) compared to our invention (60-80 m in Figure 5B).

[0158] It has also been shown that the lane boundary extending behind the vehicle corresponds to the next lane and is hidden by the vehicle when it is in the hidden position, since the hidden portion of the lane boundary can be resolved by the method and system according to the invention.

[0159] The resolution of occlusions (lane boundaries extending behind the car, etc.) is based on the ground truth and the learning process of a neural network (generally an image processing module trained by machine learning). Such ground truth, which can help by recognizing occlusions, can come from several sources: maps, preceding or subsequent frames without occlusions, simulated images, etc. (In the ground truth, foreground areas are also visible behind the car).

[0160] In a further embodiment, the correspondence mapping is a mask that takes discrete values ​​from a set of size C, which is indexed by k=0...C-1. For example, the mask may be, for example, D left and D right is an index that indicates different physical lane boundaries, and is considered as the background represented by k=0, and D left and D right takes value k (k=1...C-1) if both point to the same lane boundary with a property indexed by k. For example, this property may be the instance label of the lane, or it may distinguish dashed from solid, etc. Non-background indices are considered foreground. The correspondence mapping output of the neural network is a 2D grid of C-dimensional vectors. That is, the output has C components (channels) at each location of the 2D grid. The first channel may refer to the background probability, and the other C-1 channels may represent the probability of a foreground region having the property indexed by k=1...C-1.

[0161] If the lane properties represented by the pairing mask can be formulated as a classification problem, such as lane type classification representing dashed lines and / or color of lane separators, training can be driven by a softmax loss on C categories, similar to equation (9).

[0162]

number

[0163] If the lane properties represented by the pairing mask are lane instance labels, then the loss function could be one of those used in instance segmentation tasks. The difference between classification and instance segmentation is that in instance segmentation, permutations of the ground truth labels could be considered as equally good outputs (the labels are equivalent). This needs to be taken into account by the loss function. A more detailed explanation on instance segmentation is given below.

[0164] The details of the pairing module in one embodiment are given below: The image location of the detections in the original image depends on the coordinate grid and the raw detection tensor D (generally this is a data block, but in this embodiment it is a tensor. The raw detection data preferably contains values ​​available in the framework of the original image, which are not affected by the down / upsampling of the neural network. The neural network can also be implemented if scaling is also applied to the prediction, since this detail related to the approach to scaling does not make any difference related to efficiency, since it is a multiplication with a constant). As explained before, detections are only accepted from the foreground region. The pairing module has two tasks: to generate an image space representation of the left and right detections and to pair them according to a correspondence mapping.

[0165] In this embodiment, not only the pairings but also the foreground regions are represented by a correspondence mapping P (in another embodiment, the foreground regions are part of the raw detection tensor).

[0166] The image space detection is computed by adding the raw detection tensor D to the horizontal component of the coordinate grid (or, if the raw detection tensor also has a vertical component, it should also be added to the vertical component of the coordinate grid). detection i,j =(u i,j +D i,j ,v i,j )(10)

[0167] Therefore, detection i,j The first component of is u i,j +D i,j and the second component is v i,j The coordinate grid (u i,j ,v i,j ) are defined in the Raw Detection section. An example grid is shown in Table 2.

[0168] The image space detection tensor can be computed for all indices, but only detections from foreground regions are accepted. Alternatively, the image space detection can be computed only at indices that correspond to foreground regions in the pairing mask.

[0169] The left and right image space detections arising from the same foreground index (i,j) in the pairing mask are paired and can be further processed by the disparity and 3D projection modules.

[0170] If the neural network produces multiple correspondence mappings of types RR, RL, etc., the pairing module outputs image space detection pairs corresponding to all types RR, RL, etc. Optionally, image space detections from various pairing masks can be averaged in the pairing module or forwarded as independent detections if they represent the same 3D lane boundary points.

[0171] In this embodiment, the representation loss introduced for raw detection is equivalent to the image space loss, and there is no need to introduce an additional image space loss in the pairing module. Alternatively, we can only use the image space loss in the pairing module and turn off the representation loss for raw detection. There is a slight difference between the two approaches in how foreground regions are treated in the loss computation. For the image space loss introduced in the pairing module, a natural choice is to sum the indices that are considered foreground in the pairing mask. For the representation loss introduced for raw detection, the foreground only depends on the lane boundaries in a single image, so the loss may end up being assigned to regions that are not used in the paired prediction.

[0172] Details of the disparity module are specified below as a continuation of the embodiment introduced above: The disparity tensor is computed as the difference between the horizontal components of the left and right image space detection tensors. parallax i,j =u left detection,i,j -u right detection,i,j (11)

[0173] In this embodiment, the raw detection tensor D left -D right Note that the same result can be obtained by directly subtracting . Disparities are calculated for all detection pairs originating from the various types of pairing masks, and the resulting disparity values ​​are either obtained by averaging or the disparity values ​​are processed individually.

[0174] We can use the same loss function as for raw detection: the disparity loss is assigned only to the foreground indices in the relevant pairing mask.

[0175] In the 3D projection module, 3D projection is performed for all detection pairs according to the above equations (3-4).

[0176] For example, the Huber loss function can be used for the 3D projection, but any suitable loss function can be applied.

[0177]

number

[0178]

number

[0179] The following table shows, as a very simplified example, the tensors and computations that appear in the various modules of the system according to the invention. For simplicity, in the table below, the corresponding mapping is RL A single left-right detection tensor D given by R left and D. L right The coordinate grid in this example is given with stride x =Stride y =8, offset x =Offset y = 4. Considering also Table 2a below, we can observe that these parameters mean that for adjacent grid points, the x or y coordinate jumps by 8. Due to the offset values, the coordinate grid starts at (4,4) instead of (0,0), which is a more practical choice as a starting point. The camera parameters are the c of the principal point x =30, c y = 3, focal length f = 200, and distance between the two cameras B = 1.

[0180] Tables 1a-1c show the raw output of the neural network, i.e. the correspondence mapping and the raw detection tensors for the left and right images. This information is the output of the trained neural network. Then, Table 2 (more precisely, Tables 2a, 2b, and 2c) shows the tensors and calculations that appear in the pairing module, and Table 3 (Tables 3a, 3b, and 3c) shows the output of the disparity and 3D projection modules.

[0181] [Table 1]

[0182] [Table 2]

[0183] [Table 3]

[0184] The raw output of the neural network is shown in Tables 1a-1c above. Table 1a shows the corresponding mapping data blocks having a tensor (more specifically, a matrix) format in this embodiment. All data blocks in this embodiment have a tensor (matrix) format.

[0185] The corresponding mapping tensors shown in Table 1a correspond to the foreground channel of the RL-type pairing mask. The background channel does not contain any additional information and is therefore not shown in the figure (foreground and background probabilities sum to 1). The RL mask is R left and D L right These raw detections are therefore presented in Tables 1b and 1c.

[0186] In Table 1a, foreground regions of the pairing mask whose output exceeds a threshold (e.g. 0.5) are highlighted in bold. In this case, the raw detections in the left image take positive values ​​and the raw detections in the right image take negative values. This corresponds to the search directions of R and L (see Figure 3C). Therefore, this corresponds to the search directions D pointing from the foreground region to the right. R left has a positive value, and the opposite direction D L right has a negative value, so these distances are signed).

[0187] A lowercase x in Tables 1b and 1c indicates values ​​that were predicted by the network but were ignored because the correspondence was invalid. Only values ​​that have a corresponding probability value above a threshold in Table 1a are kept (see other parts of this description for further details on the predetermined threshold). In other words, the left or right image raw detections above only show the values ​​selected as foreground, i.e. the values ​​filtered by the correspondence mapping, and not the "original" raw detections (which are preferably the output of the image processing module).

[0188] Tables 1b and 1c show that in this example, three pairs remain after this filtering. In the raw output, of course, down / upsampling is applied compared to the original left and right images. Hence, in this example, lane boundaries are detected that follow the valid values ​​in Tables 1b and 1c.

[0189] [Table 4]

[0190] [Table 5]

[0191] [Table 6]

[0192] Tables 2a-2c show the calculations performed by the pairing module on the inputs obtained from Tables 1a-1c (stride 8 and offset 4 in both directions). The coordinate grid is calculated based on equation (5); the values ​​indicate the applied stride and offset. The predicted detection pairs from the foreground index are shown in bold (same position as in the corresponding mapping in Table 1a). The pairing module calculates the image positions of the detection pairs by adding the left and right image raw detection tensors from Tables 1b and 1c to the coordinate grid according to equation (10) (thus, only the x-coordinate of the coordinate grid is changed to obtain image-space detections). Depending on the embodiment, the image positions of the detections are calculated only for the foreground regions (as in this example) or everywhere, but only valid pairs contribute to the final output.

[0193] [Table 7]

[0194] As shown in Table 3a, the disparity module calculates the horizontal distance between the left and right detections by subtracting the two corresponding tensors from Tables 2b and 2c according to equation (11). In this embodiment, the disparity values ​​can also be calculated based on the values ​​in Tables 1b and 1c. The disparity values ​​in Table 3a show that, as is natural for images taken from a vehicle, these points are the closest points closer to the bottom of the image (i.e., the bottom of the table), while lane boundaries tend to be at a higher distance at the top of the image (due to the smallest disparity of 2.1).

[0195] [Table 8]

[0196] As shown in Table 3b, the 3D projection module determines the left image coordinate of the detection and c x =30, c yFrom the disparities according to equations (3-4) with =3, f=200, and B=1, we map the detection pairs into the 3D space of the camera coordinate system. Lowercase x denotes values ​​predicted by the network, but are ignored since the correspondence is invalid. The values ​​in Table 3b reinforce those written in relation to the disparity values ​​in Table 3a. The Z coordinates (depth) get increasingly higher from the points closest to the bottom of the table (i.e., from 12.4 to 95.2).

[0197] Below we describe a similar example using lane type segmentation built into a pairing mask. In this simple example, we distinguish between solid and dashed lanes, but similarly many more types can be included (e.g. various frequencies of dashed separators, lane marker colors, road sides, etc.). For road sides, see also above where some aspects of this special type of lane boundary are described.

[0198] In this example, we consider only RR type pairing masks, but they work similarly for all types of pairings. The calculations made in Tables 6-7 (Tables 6a-6c, Tables 7a-7c) are the same as in Tables 2-3 (Tables 2a-2c, Tables 3a-3b), but now also provide lane type information for the detection pairs. In contrast to the previous example, a slightly larger window size is applied here (the window size may be distance dependent (e.g., decreasing with increasing distance), but it is sufficient to use a fixed window size for the whole image), so that the foreground region is larger and the location of the same lane boundary point can be predicted from several neighboring indices in the 2D grid. These detections may or may not be averaged in a post-processing step.

[0199] [Table 9]

[0200] [Table 10]

[0201] [Table 11]

[0202] [Table 12]

[0203] Tables 4a-4d show the raw output of the neural network. The output includes the RR type pairing mask as a correspondence mapping with multiple foreground channels corresponding to the lane type classification (i.e., background, solid line, dashed line: channel 0 to channel 2). In Table 4a, the foreground regions of the pairing mask with background values ​​smaller than a threshold (e.g., 0.5) are highlighted in bold.

[0204] It is preferable to introduce a correspondence mapping for the background since there may be low probability values ​​in the correspondence mappings for both channel 1 and channel 2. However, these do not behave as background. In an image showing solid (continuous) and dashed lane boundaries, the background can be separated based on the correspondence mapping for channel 0, while the rest of the image is either solid or dashed. The contents of Table 4d can be derived from Tables 4a-4c, where the cells of the tensor (matrix) are markers corresponding to solid and dashed lines.

[0205] In another approach, the background can be separated by considering the corresponding mapping between channel 1 and channel 2, and based on the probability values, we separate the solid lines from channel 1 and the dashed lines from channel 2, and assign the remaining part as the background.

[0206] In such an embodiment shown in Tables 4a-4d, 5a-5b, 6a-6c and 7a-7c, the correspondence mapping tensor (as there is generally at least one such tensor) includes at least two lane boundary type selective correspondence mapping tensors (tensors included in another tensor may be referred to as sub-tensors) for at least two different lane types, respectively. The pairing probability value is a lane boundary type selection probability value, further characterizing the probability that a lane boundary is of a particular lane boundary type. The system is adapted to generate a respective 3D lane detection data block based on the at least two lane boundary type selective correspondence mapping tensors for the at least two different lane types. As shown in the table below, in this embodiment, selective 3D projections of lane boundary types can be obtained.

[0207] RR mask is D R left and D R right This shows the correspondence between the lane types, which are shown in Tables 5a and 5b. The lane type of a detection is determined by comparing the channel 1 and channel 2 values ​​in the foreground index. Alternatively, a probability of lane type can be assigned to the detection.

[0208] [Table 13]

[0209] [Table 14]

[0210] Further raw outputs of the neural network are shown in Tables 5a and 5b (the lane types are shown in Table 4d and can be extracted in other ways, so the solid and dashed line values ​​are shown in the same tensor). The RR mask is R left and D R rightWe describe the correspondence between the left and right images. In this case, both the raw detections in the left and right images take positive values ​​that correspond to the search direction (see Figure 3A). The lowercase x indicates a value predicted by the network, but is ignored since the correspondence is invalid.

[0211] [Table 15]

[0212] [Table 16]

[0213] [Table 17]

[0214] Tables 6a-6c show the calculations performed by the pairing module for the inputs shown in Tables 4a-4d, 5a-5b. The values ​​in Tables 6a-6c are calculated similarly to Tables 2a-2c, so that the coordinate grid is calculated based on equation (5). Detection pairs are predicted from the foreground indices shown in bold. The pairing module calculates the image positions of the detection pairs by adding the left and right image raw detection tensors from Tables 5a-5b to the coordinate grid according to equation (10). Depending on the embodiment, the image positions of the detections are calculated only for the foreground regions (as given in this example) or everywhere, but only valid pairs contribute to the final output.

[0215] [Table 18]

[0216] [Table 19]

[0217] [Table 20]

[0218] The disparity module calculates the horizontal distance between the left and right detections by subtracting the two corresponding tensors in Tables 6b and 6c according to equation (11).

[0219] The 3D projection module maps a detection pair to the left image coordinate of the detection and c x =30, c y = 3, f = 200, B = 1, we map from the disparity to the 3D space of the camera coordinate system according to equations (3-4). Lowercase x indicates a value predicted by the network, but is ignored since the correspondence is invalid. Table 7c shows the valid 3D detections for the corresponding lane types.

[0220] As can already be observed in Table 7b, results from the same row correspond to the same 3D points. This is a consequence of the fact that a wider window is considered in the current calculation. Only one of the comparable results needs to be considered (or these can be averaged). Thus, the calculation shows that considering a wider window is not a disadvantage.

[0221] Note that, as with Tables 6b and 6c, the data is not separated according to lane type in Tables 7a and 7b. This could be done, for example, using Table 4d, with the end result, also shown in Table 7b, being shown in a selective manner of lane boundary type in Table 7c.

[0222] Some compact embodiments that use pairing masks as correspondence mappings are introduced above.

[0223] Four different pairing masks are listed, from which any subset can be used along with the corresponding raw detections in the 3D lane detection system. Different types of raw detections and pairing masks can be implemented as different outputs of a neural network, or as different channels of the raw detection and correspondence mapping outputs for unique left and right images.

[0224] A trivial equivalent embodiment is where the raw detection tensor gives the unsigned relative distance from the coordinate grid to the nearest lane boundary along the search direction. Then the pairing module is modified accordingly and image space detections are predicted (raw detection D R left / right is added to the coordinate grid, and the raw detection D L left / right is deducted).

[0225] A search direction for the raw detection tensor is defined. Alternatively, in another embodiment, D left / right We can define,x,as the relative signed distance from the coordinate grid to the,nearest lane boundary projection on the left or right.,We then need to create a slightly different pairing mask to,pair related detections.

[0226] In the following, a further embodiment is introduced that builds on the stereo instance segmentation approach developed for the present invention by the inventors, in which the correspondence mapping is not a single 2D grid for the left and right images (which can also have more channels), but two 2D grids corresponding to the first and second images, respectively (see, for example, Tables 8a and 8b below).

[0227] The goal of known instance segmentation techniques (see, e.g., Davy Neven et al., Towards End-to-End Lane Detection: an Instance Segmentation Approach, 2018, arXiv:1802.05591; Yen-Chang Hsu et al., Learning to Cluster for Proposal-Free Instance Segmentation, 2018, arXiv:1803.06459) is to find objects in an image by labeling every pixel in the image, such that pixels corresponding to the same object get the same label and pixels corresponding to different objects get different labels.

[0228] With some important modifications (including the application of a two-channel correspondence mapping framework detailed below), the instance segmentation technique is generalized as a stereo instance segmentation technique applicable to image pairs in the correspondence mapping framework in a 3D lane detection system of one embodiment. In the stereo instance segmentation approach applied in this embodiment, regardless of which image the pixel is taken from, it gets the same label if it corresponds to the same physical instance and gets a different label if the pixel is not from the same object (in this embodiment, the neural network is taught to index the first and second images in this manner) (hence the labeling is applied to the images in a uniform and consistent manner). That is, in other words, stereo instance segmentation is a consistent labeling of the two images (consistent labeling means that the same labels are consistently used in the two channels of the correspondence mapping tensor). In summary, in this embodiment, the neural network is taught to output these consistent labels (i.e., perform consistent labeling) in the neural network training procedure.

[0229] In the known instance segmentation approaches referenced above, model adaptation is performed based on the results of the instance segmentation process, and according to another aspect, the results shown are obtained in a very different way than the stereo instance segmentation applied in the present embodiment.

[0230] In the case of the 3D lane detection method and system according to the invention, the object (to be labeled) can be the region around the projected lane boundary line (projection means visualization on the first or second image). The width of this region is a parameter and can be chosen as a constant or depend on the properties of the lane boundary (distance, type, etc.). Regions that do not correspond to any of the lane boundaries can get a unique background label.

[0231] These regions can be constructed from the known positions of the lanes during training. Using an appropriate loss function, the neural network learns to find these regions and assign them labels that are consistent with the ground truth data. Consistent means that a permutation of the ground truth labels can be accepted as a perfect prediction (the values ​​of the labels are not special, so any value can be chosen as a label). At inference time (i.e. when the method and system for 3D lane detection is used), the predicted masks are used to obtain detection pairs in the first (left) and second (right) images.

[0232] This type of correspondence mapping is very flexible and can be used to pair detections regardless of how they are predicted by the neural network, for example by classification, dense regression, or prediction of model parameters such as splines. This flexibility is due to the fact that stereo instance segmentation defines correspondences between image space detections and not the raw detections. Thus, the pairing module first computes the image locations that correspond to the raw detections, and detection pairs are selected by stereo instance segmentation regardless of the representation used in the raw detections. Some raw detections may not be paired by stereo instance segmentation and are ignored (or dropped) by the pairing module.

[0233] In one embodiment, the stereo instance segmentation is used as the foreground region (i.e., the foreground region is specified based on the results of the stereo instance segmentation) and the exact location of the lane boundary is predicted from it, as in the previous example with a single pairing mask. That is, the correspondence mapping of both images is used to determine the location (not a single pairing mask as above, see also the examples in Tables 8-10 below). Detections are not predicted from the background. Raw detection output D left and D right can be the signed distance of the lane boundary from the foreground location of the stereo instance segmentation along the epipolar direction in the left and right images, respectively.

[0234] The embodiment based on stereo instance segmentation introduced within the framework of the present invention can be built based on correspondence mapping data in general, i.e. not limited to (separate) correspondence mapping data blocks (however, this embodiment is shown with the help of correspondence mapping data blocks).

[0235] Thus, in summary, the correspondence mapping data includes first channel correspondence mapping data elements corresponding to a first image (these could be data in any common format like data blocks, tensors, or general correspondence mapping data) and second channel correspondence mapping data elements corresponding to a second image (i.e., individual data elements of both images - which can be considered as channels of primary correspondence mapping data - as in the examples shown in Tables 8a-8d, 9a-9d, and 10a-10b), and each third tensor element of the first channel correspondence mapping tensor and the second channel correspondence mapping tensor is a label characterizing whether the respective third tensor element corresponds to a lane boundary (the value is, for example, zero if not, and a specific label (e.g., "1" and "2") if a specific lane boundary is found, as shown in Tables 8a and 8b below), where the same label corresponds to the same lane boundary and different labels correspond to different lane boundaries for the first channel correspondence mapping data elements and the second channel correspondence mapping data elements, respectively. The data elements of the first and second channel corresponding mapping data elements that correspond to the lane boundaries are selected as foreground corresponding mapping data elements. The foreground corresponding mapping data elements are used as output data of the image processing step (Image Processing Module) in the pairing step (in the pairing module), i.e. are kept for further calculations (the table below shows that the foreground may differ for each starting image, but this fact does not cause problems for the calculations). In case of utilizing raw detection data, it is also possible to transfer the corresponding raw detection data to the pairing step (see also the approach in Fig. 1 and the examples introduced in Tables 8a-8d).

[0236] In this case, the correspondence mapping data is a label indicating whether the elements of the first channel correspondence mapping data and the second channel correspondence mapping data correspond to lane boundaries or not, i.e., whether they are lane boundary points or background points. Note that originally, the correspondence mapping data needs to determine the correspondence between the first lane boundary group in the first image and the second lane boundary group in the second image. Thus, the label of each channel correspondence mapping data indicates its correspondence with the other channel (the label will be the same for a particular lane boundary that can be observed in both images). Examples of channel correspondence mapping data are shown in Tables 8a and 8b. These examples are in tensor format and are possible representations of channel correspondence mapping data.

[0237] As shown in Tables 8a-8d below, an instance segmentation approach based on the general definition given in the previous paragraph can be used in such embodiments based on raw detection tensors, coordinate grid tensors, etc. However, as also shown below, the instance segmentation approach can be combined with model-based embodiments.

[0238] In one embodiment, the various labels can be represented by orthogonal vectors similar to the embodiment with the pairing mask representing the lane properties introduced earlier. That is, the correspondence mapping tensor elements are vectors whose components represent the probabilities corresponding to the various labels. In Tables 8a and 8b, the label corresponding to the maximum probability is shown as the correspondence mapping.

[0239] The names of the first channel correspondence mapping data elements and the second channel correspondence mapping data elements (e.g. (sub)tensors) are intended to emphasize that they constitute the channels of the original tensor and serve as common correspondence mapping data. See Tables 8a and 8b for these. These tables also show that the information contained in both channels of the correspondence mapping tensor (data block) can determine whether two data correspond. In the following, Tables 8a and 8b are simply labeled "Correspondence Mapping", but the rest of the label ("Left / Right Instance Segmentation") is used to implement a channel-based approach (different labels could be used for Tables 8a and 8b).

[0240] These two channels of the original tensor can also be made part of a single tensor by using separate indexes for the channels. This sub-tensor integration can also be used in other embodiments where a tensor contains more tensors (sometimes called sub-tensors). This possibility comes from the general mathematical framework of tensors.

[0241] Tables 8a-8d show the raw outputs of the neural network, i.e. the correspondence mappings and the raw detection tensors for the left and right images. Tables 9a-9d show the tensors and computations presented in the pairing module, while Tables 10a and 10b show the outputs of the disparity and 3D projection modules.

[0242] [Table 21]

[0243] [Table 22]

[0244] [Table 23]

[0245] [Table 24]

[0246] Tables 8a-8d show the raw output of the neural network for one embodiment using stereo instance segmentation for correspondence mapping (regions around line boundaries are defined; example results are obtained with a stride of 8). The raw detections are relative distances to coordinates represented by tensor indices (coordinate grids), as in the previous embodiment using pairing masks. In this example, the raw detections are relative distances, but correspondence mapping based on stereo instance segmentation can be applied to detections from any representation.

[0247] As can be seen in Figures 3A-3C, the lane boundary points in the left image are c x,left -c x , right If >= 0, it is placed to the right of the corresponding lane boundary point in the right image. Comparing the contents of Tables 8a and 8b gives a good estimate of the location of lane boundaries "1" and "2", which is refined by the values ​​in Tables 8c and 8d.

[0248] D left and D right The valid results of D are maintained based on the results of left and right instance segmentation, respectively. In other words, left and D right has a lowercase x, and Table 8a and Table 8b do not have "1" or "2", respectively. Therefore, the foreground estimation is D left and D right are separated for (and remain separated in Tables 9b and 9c, see below).

[0249] Note that because the raw detection tensor was chosen as a signed relative distance, the sign of the raw detection changes if both sides of the lane boundary are in the foreground region.

[0250] [Table 25]

[0251] [Table 26]

[0252] [Table 27]

[0253] [Table 28]

[0254] Tables 9a to 9d show examples of computations of the pair matching modules corresponding to the representations shown in Tables 8a to 8d. The coordinate grid is x =Stride y =8, offset x =Offset y = 4 and is generated according to equation (5). The image space detection is done by adding a coordinate grid to the raw detections as per Tables 2a-2c and equation (10). Lowercase x denotes predicted raw detections that are not used since they are considered background according to stereo instance segmentation.

[0255] In Table 9d, image space detection pairs are collected. Image space detections corresponding to the same label and the same vertical coordinate are averaged separately in the left and right images before forming pairs (thus, the first coordinate on the left in the first pair is 22.1). Thus, these results help to increase the accuracy.

[0256] [Table 29]

[0257] [Table 30]

[0258] The disparity module calculates the horizontal distance between the left and right detections by subtracting the corresponding two tensors from Table 9 according to equation (11). See the disparity results in Table 10a. As in the image space detection pairs in Table 9d, different rows correspond to different detections. In Table 10b, the 3D projection module calculates the horizontal distance between the left and right detections by subtracting the corresponding two tensors from Table 9. x =30, c y = 3, f = 200, B = 1, and the detected pairs are mapped to the 3D space of the camera coordinate system from the disparity according to equations (3-4). Table 10b may also include a column for specifying the resulting labels.

[0259] In another embodiment, the image processing module does not have raw detection output, but only stereo instance segmentation as a correspondence mapping. As explained before, the stereo instance segmentation of the left and right images encloses the lane boundaries and estimates their location appropriately. The exact location of the lane boundaries can be determined by averaging or least squares model fitting. Thus, in one embodiment, the pairing module calculates the image space locations of detections corresponding to various labels by weighted averaging of the coordinate grid along all rows, where the weights are differentiable functions of the probability of a particular label at the grid location. In another embodiment, the pairing module fits a model to the point cloud defined by the coordinate grid and weighted by the label probabilities of all labels by least squares fitting. Samples are taken from this model to obtain detection pairs that are projected into 3D space in the 3D projection module. Since the least squares fitting and sampling from the model are differentiable, this embodiment fits into the general framework of 3D lane detection systems. Further details regarding the model are introduced below in the model-based embodiment, which differs from this approach in that the raw detections represented by model parameters are utilized.

[0260] The stereo instance segmentation used in one embodiment of the present invention is illustrated in Figures 7A and 7B, which show images from a first camera and a second camera, respectively. In each image, three lane boundaries are observable: a first lane boundary projection 150a, a second lane boundary projection 152a, and a third lane boundary projection 154a in Figure 7A. Correspondingly, a first lane boundary projection 150b, a second lane boundary projection 152b, and a third lane boundary projection 154b in Figure 7B are shown. It can be observed that the viewpoints of the images are different.

[0261] Every lane boundary has a surrounding region labeled by stereo instance segmentation techniques. In FIG. 7A, region 156a surrounds lane boundary projection 150a. Region 156a holds an instance segmentation index of “1”. Surrounding region 158a is assigned to lane boundary projection 152a (region 158a is labeled “2”), and region 160a (having label “3”) is assigned to lane boundary projection 154a. Respective regions 156b, 158b, and 160b (labeled “1”, “2”, and “3”) are also shown in FIG. 7B. Pixels around the same lane boundary line get the same labels in the first and second camera images.

[0262] As shown in the examples above, these region designations are used for the correspondence mapping of this technique. The correspondence mapping tensors shown in Tables 8a-8b represent regions in the first and second images. Thus, the surrounding regions (areas) specified in the images can be represented in these tensors by labeling the respective elements of the tensors (in the examples of Tables 8a-8b, two different labels are used, namely "1" and "2", while in the illustrative examples of Figures 7A and 7B, there are three different indices).

[0263] Figures 8A and 8B show the stereo instance segmentation on a real image, on which the predictions of the trained neural network are specified. Figures 8A and 8B give a first image of the first camera and a second image of the second camera. The regions 230a, 232a, 234a, 236a, 238a and 230b, 232b, 234b, 236b, 238b specified by the instance segmentation are highlighted in white in Figures 8A and 8B, respectively (regions corresponding to each other have the same number of "a" and "b" and are represented by the same pattern, such as left and right stripes, stars, grid lines in the shape of "+" and "x", i.e. the pattern represents the label assigned by the stereo instance segmentation). These are the starting instance segmentation regions (similar to the regions shown in Figures 7A and 7B and Tables 8a-8b), based on which the respective embodiments of the method and system according to the invention are performed. 8A and 8B are in some way similar to FIGS. 4A-4F as both groups of figures show the corresponding mappings for each example.

[0264] In the following, a further embodiment is specified, called the model-based embodiment.

[0265] In this embodiment, the lane boundaries in the left and right images can also be represented by model parameters predicted by a neural network (typically an image processing module trained by machine learning), e.g., spline parameters. Thus, rather than a raw detection specifying the placement of lane boundaries being a group of distances like the parameters above, the neural network outputs spline parameters, etc., that characterize the lane boundaries in a different way.

[0266] The model of lane boundaries can be represented by a variable number of parameters (N). For spline models, N depends on the number of control points and the polynomial degree of the spline. N may be fixed by the architecture, e.g., the number of control points is fixed. Alternatively, the number of control points used to represent the lane boundaries may also be determined by the neural network.

[0267] In general, the raw detection is a set of model parameters that are different for each lane boundary. In these embodiments shown in the table above, these parameters are a set of distance type parameters. These distances are lane boundary-coordinate grid point distances. However, it is not necessary to have a coordinate grid to specify the lane boundaries. In the current model-based embodiment, if the lane boundaries are represented by splines, it is not necessary to use a coordinate grid.

[0268] For example, the neural network can model lane boundaries in the left / right images. l / r number (i.e., m l / r is the number of detections), each of which has a different parameter a l / r、k where k is the index of the detection. The number of model parameters may also vary from detection to detection. Such a variable number of detections can be realized, for example, by LSTM cells (Long Short-Term Memory cells). The raw detections of the left and right images are formalized as follows:

[0269]

number

[0270] These raw detection D l / r is a more generalized block of data than the tensors (matrices) introduced above. The number of model parameters required (N1, N2, N ml ) may be different for the rows, so the rows in the above data block are not necessarily the same length (there is no reason for them to be the same length, but in theory they could be).

[0271] A representation loss can be assigned to the raw detections, which compares the model parameters of the lane boundaries to their ground truth model parameters, but different model parameters may produce equally good detections, which is better captured by an image space loss.

[0272] The image coordinates of the detections are calculated from the model of lane boundaries predicted by the neural network. This step is performed in the pairing module. The lane boundary model M is an implicit function of the image coordinates (u,v) and the model parameters. M(u,v;a l / r,k )=0

[0273] A simpler case is when the model is parameterized by v (the above equations are rearranged). u = M'(v;a l / r,k )

[0274] The correspondence mapping in this embodiment is a mapping between the indices of detections, e.g. C = {(l,i)⇔(r,j)|i∈[1,m l ],j∈[1,m r ],a l,i is a r,j corresponds to the same physical lane} That is, it indicates the correspondence between the left and right parameter sets. Thus, the raw detection preferably substantially provides the input for the correspondence mapping data block. In another embodiment, the lane boundary m l and m r The number of models is fixed by the architecture (regardless of the number of lane boundaries visible on the image) and an ordering is defined among the lane boundaries. In this case, the raw detections are not necessarily the input of the corresponding mapping data block.

[0275] The pairing module examines the correspondences in C and obtains samples from models M or M', e.g., at various horizons indexed by v. This provides detection pairs (a l,i -a r,j In the case of a pair, the above equation given by M' gives two different values ​​for "u"), which are processed by the disparity module and the 3D projection module (i.e., in this embodiment, the disparity is calculated from the two different values ​​of "u", and more specifically also from its left and right values).

[0276] Training can be facilitated by loss in the image space: since sampling from a model of lane boundaries is a differentiable operation, gradients can be passed backwards through the module to reach the neural network.

[0277] In summary, in the above model-based embodiments of the method and system: a location of each first member having a first index (i.e., each lane boundary has a respective index) of the first lane boundary group in the first image is defined by first model parameters of the raw detection data block in the first image; a location of each second member having a second index in the second lane boundary group in the second image is defined by second model parameters of the raw detection data block in the second image; The correspondence mapping data block determines a correspondence between each first index and second index based on whether the first index and the second index correspond to the same lane boundary.

[0278] The above approach of introducing a first index and a second index can also be combined with the stereo instance segmentation approach introduced above (as well as the correspondence mapping approach mentioned in the previous paragraph), i.e., the correspondence mapping approach of stereo instance segmentation can be implemented for the processing of indexed lane boundaries.

[0279] In this case, the pairing module generates image space detections by iteratively processing elements of raw detections in the left and right images and taking samples from the corresponding model, e.g., at various horizons indexed by v. These image space detections get their labels from the stereo instance segmentation. Points on the left and right images that are on the same horizon and characterized with the same label define a detection pair. Preferably, points on the same horizon with the same label can be averaged separately on the left and right images before generating the pairs. Image space detections labeled as background do not contribute to the image space detection pair.

[0280] One embodiment of the present invention relates to a non-transitory computer readable medium having stored thereon instructions that, when executed by a computer, cause the computer to perform a method according to the present invention.

[0281] The advantages of the system and method according to the invention are summarized here: Our solution is based on the correspondence mapping approach described above. Therefore, we do not apply an explicit stereo matching step. The present invention allows to reduce the noise level in 3D detection. In contrast, in prior art approaches, stereo matching introduces noise on homogeneous surfaces (such as road surfaces) and produces poor quality results at lane boundaries. The method and system according to the invention can also operate in occluded areas. Preferably, in embodiments applied to neural networks, the method and system for training is end-to-end learnable and differentiable.

[0282] Of course, the present invention is not limited to the preferred embodiments described in detail above, but further variations, modifications and developments are possible within the scope of protection defined by the claims. Moreover, all embodiments that can be defined by any combination of dependent claims are included in the present invention.

Claims

1. 1. A method for lane detection, comprising: The image processing module is trained by machine learning. generating output data based on an image pair comprising a first image (10a) having a first lane boundary group and a second image (10b) having a second lane boundary group, said output data including correspondence mapping data defining a correspondence between said first lane boundary group and said second lane boundary group; generating image space detection pair data blocks based on the output data; generating a 3D lane detection data block using calibration data (26) corresponding to the first image and the second image based on a first data block portion of the data block of the image space detection pair corresponding to a first member of the image space detection pair and a second data block portion of the data block of the image space detection pair corresponding to a second member of the image space detection pair and triangulation; the image processing module is implemented by a computing device including at least one processor; said corresponding mapping data being contained in a corresponding mapping data block (16); the output data includes a first image raw detected data block (14a) defining a location of each first member of the first lane boundary group, and a second image raw detected data block (14b) defining a location of each second member of the second lane boundary group; The output data is generated for a coordinate grid represented by a coordinate grid tensor having a plurality of grid tensor elements; the first image raw detection data block (14a) is a first image raw detection tensor, the first image raw detection tensor corresponding to the coordinate grid tensor and having a plurality of first tensor elements, each of the plurality of first tensor elements including a respective first distance value measured from a corresponding grid tensor element of a closest first lane boundary point of the first lane boundary group in a first search direction in the first image (10a); the second image raw detection data block (14b) is a second image raw detection tensor, the second image raw detection tensor corresponding to the coordinate grid tensor and having a plurality of second tensor elements, each of the plurality of second tensor elements including a respective second distance value measured from a corresponding grid tensor element of a closest second lane boundary point of the second lane boundary group in a second search direction in the second image (10b); the corresponding mapping data block is a corresponding mapping tensor corresponding to the coordinate grid tensor and having a plurality of third tensor elements characterizing whether the nearest first lane boundary point and the nearest second lane boundary point correspond to the same lane boundary; the data blocks of the image space detection pairs are generated by generating tensors of the image space detection pairs using the coordinate grid tensors. method.

2. 2. The method of claim 1 , wherein the correspondence mapping tensor is a pairing correspondence mapping tensor having a pairing probability value for each of its third tensor elements, the pairing probability value characterizing a probability that the closest first lane boundary point and the closest second lane boundary point correspond to the same lane boundary and are located within a window size from the grid tensor element corresponding to the respective third tensor element, the third tensor elements whose respective probability values ​​exceed a predetermined first threshold are selected as foreground third tensor elements, and the foreground third tensor elements and the corresponding first and second tensor elements are used as the output data in the step of generating the image space detection pairs data block.

3. 3. The method of claim 2, characterized in that the first and second search directions are selected as a leftward direction in the respective first and / or second image or as a rightward direction in the respective first and / or second image.

4. 3. The method of claim 2, wherein the correspondence mapping tensor comprises selective correspondence mapping tensors of at least two lane boundary types for at least two different lane types, the pairing probability values ​​being lane boundary type selection probability values ​​further characterizing a probability that the lane boundaries corresponding to the nearest first lane boundary point and the nearest second lane boundary point are of a particular lane boundary type, and each 3D lane detection data block is generated based on the selective correspondence mapping tensors of at least two lane boundary types for the at least two different lane types.

5. 2. The method of claim 1, wherein the corresponding mapping data further defines a location of each first member of the first lane boundary group and a location of each second member of the second lane boundary group.

6. 2. The method of claim 1, wherein the corresponding mapping data comprises a first channel corresponding mapping data element corresponding to the first image (10a) and a second channel corresponding mapping data element corresponding to the second image (10b), each of the first channel corresponding mapping data element and the second channel corresponding mapping data element being a label characterizing whether the respective data element of the first channel corresponding mapping data element and the second channel corresponding mapping data element corresponds to a lane boundary, with same labels corresponding to the same lane boundary and different labels corresponding to different lane boundaries of the first channel corresponding mapping data element and the second channel corresponding mapping data element, respectively, and wherein the data element of the first channel corresponding mapping data element and the second channel corresponding mapping data element corresponding to a lane boundary is selected as a foreground corresponding mapping data element, and the foreground corresponding mapping data element is used as the output data in the step of generating the data block of the image space detection pair.

7. a location of each first member having a first index of the first lane boundary group in the first image (10a) is defined by first model parameters of the first image raw detection data block; a location of each second member having a second index of the second lane boundary group in the second image (10b) is defined by second model parameters of the second image raw detected data block; 2. The method of claim 1, wherein the correspondence mapping data block (16) determines the correspondence between the first index and the second index based on whether each first index and second index corresponds to the same lane boundary.

8. 2. The method of claim 1, wherein the image processing module (12) is implemented by a neural network.

9. 2. The method according to claim 1, characterized in that in the step of generating output data, the output data is generated by the image processing module based on the actual frames of the first image (10a) and the second image (10b) as well as on at least one additional frame of the first image and the second image preceding or following the actual frame.

10. 2. The method of claim 1, characterized in that the image pair of the first image (10a) and the second image (10b) is a stereo image pair of a left image and a right image.

11. 1. A system for lane detection, comprising: one or more processors; a memory coupled to the one or more processors and containing program code for execution by at least one of the one or more processors; generating output data based on an image pair of a first image (10a) having a first lane boundary group and a second image (10b) having a second lane boundary group by an image processing module (12) trained by machine learning; the output data includes correspondence mapping data defining a correspondence between the first lane boundary group and the second lane boundary group; generating image space sensed data blocks of image space sensed pairs based on the output data; generating a 3D lane detection data block based on a first portion of the image space detection data block corresponding to a first member of the image space detection pair and a second portion of the image space detection data block corresponding to a second member of the image space detection pair using calibration data (26) and triangulation corresponding to the first image and the second image; said corresponding mapping data being contained in a corresponding mapping data block (16); the output data includes a first image raw detected data block (14a) for defining a location of each first member of the first lane boundary group, and a second image raw detected data block (14b) for defining a location of each second member of the second lane boundary group; The output data is generated for a coordinate grid represented by a coordinate grid tensor having a plurality of grid tensor elements; the first image raw detection data block (14a) is a first image raw detection tensor, the first image raw detection tensor corresponding to the coordinate grid tensor and having a plurality of first tensor elements, the plurality of first tensor elements including respective first distance values ​​measured from corresponding grid tensor elements of nearest first lane boundary points of the first lane boundary group in a first search direction in the first image (10a); the second image raw detection data block (14b) is a second image raw detection tensor, the second image raw detection tensor corresponding to the coordinate grid tensor and having a plurality of second tensor elements, the plurality of second tensor elements including respective second distance values ​​measured from corresponding grid tensor elements of nearest second lane boundary points of the second lane boundary group in a second search direction in the second image (10b); the corresponding mapping data block is a corresponding mapping tensor corresponding to the coordinate grid tensor and having a plurality of third tensor elements characterizing whether the nearest first lane boundary point and the nearest second lane boundary point correspond to the same lane boundary; generating an image space detection pair tensor using the coordinate grid tensor as the data block of an image space detection pair; A system configured as follows.

12. 12. The system of claim 11, wherein the correspondence mapping tensor is a pairing correspondence mapping tensor having a pairing probability value for each of its third tensor elements, the pairing probability value characterizing a probability that the closest first lane boundary point and the closest second lane boundary point correspond to the same lane boundary and are located within a window size from the grid tensor element corresponding to the respective third tensor element, the third tensor elements whose respective probability values ​​exceed a predetermined first threshold are selected as foreground third tensor elements, and the foreground third tensor elements and the corresponding first and second tensor elements are used as the output data of the image processing module in a pairing module (20) that generates the image space detection data block of the image space detection pair.

13. 13. The system of claim 12, wherein the first and second search directions are selected as a leftward direction in the respective first and / or second image, or as a rightward direction in the respective first and / or second image.

14. 13. The system of claim 12, wherein the correspondence mapping tensor comprises at least two lane boundary type selective correspondence mapping tensors for at least two different lane types, the pairing probability values ​​being lane boundary type selection probability values ​​further characterizing a probability that the lane boundaries corresponding to the closest first lane boundary point and the closest second lane boundary point are of a particular lane boundary type, and the system is adapted to generate each 3D lane detection data block based on the at least two lane boundary type selective correspondence mapping tensors for the at least two different lane types.

15. 12. The system of claim 11, wherein the corresponding mapping data further defines a location of each first member of the first lane boundary group and a location of each second member of the second lane boundary group.

16. 12. The system of claim 11, wherein the corresponding mapping data comprises a first channel corresponding mapping data element corresponding to the first image (10a) and a second channel corresponding mapping data element corresponding to the second image (10b), each of the first channel corresponding mapping data element and the second channel corresponding mapping data element being a label characterizing whether the respective data element of the first channel corresponding mapping data element and the second channel corresponding mapping data element corresponds to a lane boundary, with same labels corresponding to same lane boundary and different labels corresponding to different lane boundaries of the first channel corresponding mapping data element and the second channel corresponding mapping data element, respectively, wherein the data element of the first channel corresponding mapping data element and the second channel corresponding mapping data element corresponding to a lane boundary is selected as a foreground corresponding mapping data element, and the foreground corresponding mapping data element is used as the output data of the image processing module in a pairing module that generates the image space detection data block of the image space detection pair.

17. a location of each first member having a first index of the first lane boundary group in the first image (10a) is defined by first model parameters of the first image raw detection data block; a location of each second member having a second index of the second lane boundary group in the second image (10b) is defined by second model parameters of the second image raw detected data block; 12. The system of claim 11, wherein the correspondence mapping data block (16) determines the correspondence between the first index and the second index based on whether each first index and second index corresponds to the same lane boundary.

18. The system according to claim 11, characterized in that the image processing module (12) is implemented by a neural network.

19. 12. The system of claim 11, characterized in that the generation of the output data by the image processing module (12) is based on the actual frames of the first image (10a) and the second image (10b) as well as at least one additional frame of the first image and the second image preceding or following the actual frames.

20. The system according to claim 11, characterized in that the image pair of the first image (10a) and the second image (10b) is a stereo image pair of a left image and a right image.

21. 2. A method for training an image processing module to be applied in the method of claim 1, comprising the step of controlling a 3D projection loss of the 3D lane detection data block during training of the image processing module by modifying a learnable parameter of the image processing module.

22. 12. A system for training an image processing module to be applied in the system of claim 11, comprising a 3D projection loss module (42) adapted to control the 3D lane detection data blocks during training of the image processing module by modifying learnable parameters of the image processing module.

23. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a computer, cause the computer to: an image processing step using an image processing module trained by machine learning to generate as output data correspondence mapping data based on an image pair of a first image (10a) and a second image (10b), the correspondence mapping data determining a correspondence between a first lane boundary group in the first image (10a) and a second lane boundary group in the second image (10b); generating image space detection pair data blocks based on the output data in a pairing step; In the 3D lane detection data generation step, a first data block portion of the data block of the image space detection pair corresponding to a first member of the image space detection pair; and a second data block portion of said data block of said image space detection pair corresponding to a second member of said image space detection pair; and generating a 3D lane detection data block by triangulation using calibration data (26) corresponding to the first image and the second image based on: said corresponding mapping data being contained in a corresponding mapping data block (16); the output data includes a first image raw detected data block (14a) defining a location of each first member of the first lane boundary group, and a second image raw detected data block (14b) defining a location of each second member of the second lane boundary group; The output data is generated for a coordinate grid represented by a coordinate grid tensor having a plurality of grid tensor elements; the first image raw detection data block (14a) is a first image raw detection tensor, the first image raw detection tensor corresponding to the coordinate grid tensor and having a plurality of first tensor elements, each of the plurality of first tensor elements including a respective first distance value measured from a corresponding grid tensor element of a closest first lane boundary point of the first lane boundary group in a first search direction in the first image (10a); the second image raw detection data block (14b) is a second image raw detection tensor, the second image raw detection tensor corresponding to the coordinate grid tensor and having a plurality of second tensor elements, each of the plurality of second tensor elements including a respective second distance value measured from a corresponding grid tensor element of a closest second lane boundary point of the second lane boundary group in a second search direction in the second image (10b); the corresponding mapping data block is a corresponding mapping tensor corresponding to the coordinate grid tensor and having a plurality of third tensor elements characterizing whether the nearest first lane boundary point and the nearest second lane boundary point correspond to the same lane boundary; the data blocks of the image space detection pairs are generated by generating tensors of the image space detection pairs using the coordinate grid tensors.

23. A non-transitory computer-readable medium configured to:

Images