METHOD AND DEVICE FOR LANE DETECTION ON A VEHICLE ROAD

Abstract

Method for detecting a lane near a vehicle (100), the method comprising: Capturing a field of view (FOV) image of a visible area (34) encompassing a driving surface (10) via an image sensor (32); Conversion of the FOV image into a multitude of feature maps via an artificial neural network (430); Projecting the plurality of feature maps (32) via an inverse perspective mapping routine onto a bird's-eye view orthographic grid (BEV), wherein the orthographic BEV grid contains a plurality of grid sections (510) representing a ground truth of the road surface (10), wherein the feature maps (32) comprise road segments and feature embeddings in the plurality of grid sections (510) of the orthographic grid in the BEV; Linking the lane segments (520) for the plurality of grid sections (510) based on the feature embeddings to form a predicted lane (502, 504) in the orthographic BEV grid, wherein the predicted lane (502, 504) is associated with the lane near the vehicle (100); and Control of the operation of the vehicle (100) based on the predicted lane (502, 504).

Description

INTRODUCTION

[0001] Vehicles, including those with advanced driver assistance systems (ADAS) and autonomous systems, can use lane location information to inform an operator and / or to control the operation of one or more of the vehicle's systems. Vehicle control systems can benefit from dynamic lane location information and use this information as input to control one or more systems such as braking, cornering, and acceleration.

[0002] Accurate lane detection plays a crucial role in autonomous driving for several reasons. These include providing guidance on available vehicle maneuvers, precisely locating the vehicle relative to a digitized map, and enabling the automatic creation of maps in conjunction with accurate vehicle localization. Therefore, there is a need for precise three-dimensional lane detection and localization. Furthermore, it is desirable to be able to quickly, accurately, and precisely detect, monitor, and react to lanes on a road surface that lie within a vehicle's trajectory, using information from an imaging sensor.

[0003] DE 10 2017 113 572 A1 describes systems and methods for lane detection. One method receives an image from a forward-facing vehicle camera and applies a geometric transformation to the image to generate a bird's-eye view. The method analyzes this bird's-eye view using a neural network that was previously trained using side-facing vehicle camera images to determine a lane position corresponding to the bird's-eye view of the image.

[0004] DE 10 2019 101 938 A1 describes the creation of cognitive maps for vehicles. A system comprises a processor and a memory, wherein the memory contains instructions to be executed by the processor to capture images of the vehicle's surroundings, determine a cognitive map, which includes a top view of the vehicle's surroundings, based on the image, and operate the vehicle based on the cognitive map.

[0005] DE 10 2017 116 213 A1 describes a method for determining lane information. The method includes receiving perception data from at least two sensors, wherein the at least two sensors include a rear-facing camera of a vehicle. The method includes determining the number of lanes on a roadway within a field of view captured by the perception data, based on the perception data and using a neural network. The method includes providing the number of lanes to an automated driving system or driver assistance system.

[0006] DE 10 2017 105 661 A1 describes a method for determining the position of a vehicle in a lane, comprising receiving perceptual information from a first camera positioned on one side of the vehicle and a second camera positioned on the other side. The method includes determining, using one or more neural networks, the vehicle's position relative to lane markings on both sides. The method further includes notifying a driver or control system of the vehicle's position.

[0007] DE 10 2017 108 248 A1 describes techniques for road feature recognition using a vehicle camera system. A computer-implemented method comprises receiving an image from a camera attached to a vehicle on a road via a processing unit. The computer-implemented method further comprises generating a top-down view of the road based at least partially on this image. The computer-implemented method further comprises detecting lane boundaries of a lane of the road based at least partially on this top-down view. The computer-implemented method further comprises using machine learning to detect a road feature within the lane boundaries of the lane of the road.

[0008] DE 11 2019 000 122 T5 describes the reception of sensor data representing an image of a vehicle sensor's field of view, where the sensor data is applied to a machine learning model. The machine learning model can calculate a segmentation mask that is representative of parts of the image corresponding to the lane markings on the vehicle's road surface. An analysis of the segmentation mask can be performed to determine the types of lane markings, and lane boundaries can be generated by curve fitting of the lane markings corresponding to each lane marking type. The data representative of the lane boundaries can then be sent to a vehicle component to use when navigating the road surface. SUMMARY

[0009] A method, device, and system capable of providing a three-dimensional digital lane representation encompassing various lane topologies, including but not limited to urban situations, junctions, merges, etc., which can be generalized to unseen cameras and scenes. This includes a vehicle with an image sensor and a control unit, the image sensor being, as non-limiting examples, a camera or a LiDAR sensor. The image sensor is positioned to monitor a field of view (FOV) encompassing a road surface near the vehicle. A method and associated system for detecting a lane near the vehicle are described and include acquiring an FOV image of a visible portion of the road surface via the image sensor. The FOV image is transformed into a variety of feature maps using an artificial neural network.The feature maps are projected onto an orthographic BEV grid using an inverse perspective mapping algorithm. The orthographic BEV grid contains a multitude of grid sections representing the ground truth of the road surface. The feature maps include lane segments and feature embeddings, which are represented in the multitude of grid sections of the orthographic BEV grid. The lane segments are represented as line segments in the multitude of grid sections.

[0010] The line segments are concatenated for the multitude of grid sections based on the feature embeddings to form a predicted lane in the orthographic BEV grid. This concatenation, or clustering, is performed via the feature embeddings. The feature embeddings of each grid section are assigned vectors that have been trained such that vectors belonging to the same lane are embedded close together in the same embedded space, and vectors belonging to different lanes are embedded far apart. Clustering can then be applied to these vectors to achieve separation into different lanes.

[0011] Vehicle operation is controlled based on the predicted lane in the orthographic BEV grid. This can include, for example, controlling vehicle operation, controlling the vehicle's navigation and route planning, transmitting the predicted lane to nearby vehicles, and updating the vehicle's internal and / or external map data accordingly. Controlling vehicle operation based on the predicted lane includes controlling a propulsion system, a steering system, or a braking system via an advanced driver assistance system.

[0012] One aspect of the disclosure involves that each of the line segments is parameterized as a normal vector defined with respect to a center point of the grid section, where the normal vector is defined by a magnitude, a direction, and a height with respect to the center point of the grid section. Another aspect of the disclosure involves that the lane segments are represented as straight segments in the plurality of grid sections, and where each of the predicted lanes comprises a linked set of the straight segments in adjacent parts of the plurality of grid sections.

[0013] Another aspect of the disclosure involves representing the lane segments as non-linear line segments in the multitude of grid sections, with each of the predicted lanes comprising a linked set of the non-linear line segments in adjacent parts of the multitude of grid sections.

[0014] Another aspect of disclosure involves linking the lane segments for the multitude of grid sections by clustering the feature embeddings in order to achieve separation into individual lanes.

[0015] Another aspect of disclosure includes capturing the image of the field of view (FOV) of the visible area, which encompasses the road surface in front of the vehicle.

[0016] The above features and advantages, as well as other features and advantages of the present teaching, are readily apparent from the following detailed description of some of the best modes and other embodiments for carrying out the present teaching, as defined in the attached claims, when considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] One or more embodiments are now described by way of example with reference to the attached drawings, in which: Fig. Figure 1 schematically shows a side view of a vehicle comprising a space monitoring system, wherein the vehicle is arranged on a driving surface in accordance with the disclosure. Fig. Figure 2-1 shows an image of the field of view (FOV) of a road surface with a flat, multi-lane road surface according to the disclosure. Fig. Figure 2-2 shows a pictorial representation of the road surface from a bird's-eye view (BEV) including the flat, multi-lane road surface, which is based on Fig. 2-1 is shown, according to the revelation. Fig. Figure 2-3 graphically illustrates an orthographic BEV grid of the flat multi-lane road surface, which is based on Fig. 2-1 is shown, including the orthographic representation of a multitude of road edges, according to the disclosure. Fig. Figure 3 shows a visual FOV image of a road surface with a non-level, curved multi-lane road surface as disclosed. Fig. Figure 4 schematically shows a method for processing images from a vehicle-side image sensor to form one or more predicted lanes that digitally represent a vehicle driving surface, in accordance with the disclosure. Fig. Figure 5 schematically shows an example of an orthographic BEV grid associated with an example FOV image, including an exploded view of an orthographically arranged grid section containing a lane segment that includes part of a lane edge, according to the disclosure. Fig. Figure 6 schematically shows part of a method for processing images from a vehicle-side image sensor to form one or more predicted driving lanes that digitally represent a vehicle driving surface, including a layer of an artificial neural network (ANN) including an ANN classification step with a plurality of convolution layers and connected layers, according to the disclosure. Fig. Figure 7 schematically shows an example of a BEV feature map containing a variety of predicted lanes, according to the disclosure. Fig. Figure 8 shows an orthographic BEV grid, corresponding three-dimensional clusters representing embedded features, and corresponding lanes of an exemplary FOV image as disclosed.

[0018] It should be understood that the accompanying drawings are not necessarily to scale and represent a somewhat simplified depiction of various preferred features of the present disclosure as disclosed herein, including, for example, certain dimensions, orientations, positions, and shapes. Details associated with such features are partly determined by the specific intended application and operating environment. DETAILED DESCRIPTION

[0019] The components of the disclosed embodiments, as described and illustrated herein, can be arranged and implemented in a multitude of different configurations. Therefore, the following detailed description is not intended to limit the scope of the claimed disclosure, but is merely representative of possible embodiments thereof. Furthermore, while numerous specific details are included in the following description to facilitate a comprehensive understanding of the embodiments disclosed herein, some embodiments can also be implemented without some of these details. In addition, for the sake of clarity, a detailed description of certain technical matters that are self-evident in the related field has been omitted in order to avoid unnecessarily complicating the disclosure. Finally, the drawings are presented in simplified form and not to scale.For the sake of clarity and convenience, directional terms such as above, below, left, right, over, above, below, behind, and in front may be used in the drawings. These and similar directional terms are not to be interpreted as limiting the scope of the disclosure. Furthermore, the disclosure as presented and described herein may be carried out even without any element not specifically disclosed herein. As used herein, the term "system" may refer to one or a combination of mechanical and electrical hardware, sensors, controllers, application-specific integrated circuits (ASICs), combinational logic circuits, software, firmware, and / or other components arranged to provide the described functionality.

[0020] The concepts described here offer a lane representation that enables the learning of general three-dimensional (3D) lane segments from the BEV perspective. This provides an interpretable intermediate representation for lane prediction, enables efficient and accurate prediction of a high-resolution lane curve using a low-resolution BEV grid, allows robust lane prediction even in new, unseen scenarios, and enables continuous prediction of entire lane curves in a machine-learning manner through feature embedding. This also allows for better fusion with other modalities and tasks such as LiDAR and free space, which can be represented as an occupancy grid. In addition to the top-down representation of the segments described above, the height of the segments relative to the ground plane is also output in each grid section. This results in a 3D representation of the lane points, i.e.,the (x,y,z) position of the lane point and the orientation of the road segment passing through this point.

[0021] Referring to the drawings, in which identical reference numerals correspond to identical or similar components in the various figures, shows Fig. Figure 1 schematically depicts a vehicle 100, arranged on a driving surface 10, in accordance with the embodiments disclosed herein. The vehicle 100 comprises a drive system 15, a steering system 16, a wheel brake system 17, an advanced driver assistance system (ADAS) 20, a space monitoring system 30, and a navigation system 40. A side view of the vehicle 100 is shown. The vehicle 100 is arranged on a driving surface 10, e.g., a paved road, and is capable of driving on it.

[0022] The road surface 10 can be configured as a single-lane roadway, a two-lane highway, or a multi-lane divided highway and can include, without restriction, entry ramps, exit ramps, junctions, merges, intersections, roundabouts, etc. The vehicle 100 and the road surface 10 define a three-dimensional coordinate system with a longitudinal or y-axis 11, a transverse or x-axis 12, and a longitudinal or z-axis 13. The longitudinal axis 11 is defined as corresponding to a direction of travel of the vehicle 100 on the road surface 10. The transverse axis 12 is defined as being orthogonal to the direction of travel of the vehicle 10 on the road surface 10. The longitudinal axis 13 is defined as being orthogonal to a plane defined by the longitudinal axis 11 and the transverse axis 12, i.e., that it is perpendicular to the road surface 10 at the current position of the vehicle 100.Other in-vehicle systems may include, as non-limiting examples, a computer-readable storage device or medium (memory) containing a digitized road map, a telematics system 60, a GNSS (Global Navigation Satellite System) sensor 52, and a HMI (Human / Machine Interface) device 55. The concepts described herein may be employed on various systems that may benefit from information obtained by an embodiment of the spatial monitoring system 30 in a manner described herein. The vehicle 10 may, but is not limited to, comprise a mobile platform in the form of a commercial vehicle, an industrial vehicle, an agricultural vehicle, a passenger car, an aircraft, a watercraft, a train, an all-terrain vehicle, a personal mobility device, a robot, and the like, to fulfill the purposes of this disclosure.

[0023] In one embodiment and as described herein, the spatial monitoring system 30 comprises an image sensor 32 having a field of view (FOV) of a visible area 34 encompassing a driving surface located near and in front of the vehicle 100, and communicating with a spatial monitoring controller 35. Alternatively or additionally, the image sensor 32 has a field of view (FOV) of a visible area 34 encompassing a driving surface 10 located beside and / or behind the vehicle 100. The image sensor 32 can be a 2D digital camera, a 3D digital camera, or a light detection and radar (LiDAR) device. The image sensor 32 has a vehicle-side position and orientation that can be characterized, including height h. Nocken 36 and angle θ Nocken37, which are defined relative to the longitudinal axis 13. One or more image sensors 32 can be used. The concepts described here can be used in various systems that can benefit from the information obtained by an embodiment of the spatial monitoring system 30. In combination with the information from the GNSS sensor 52, the spatial monitoring controller 30 determines the spatial positions of objects located in the visible area 34 of the vehicle 100. The spatial monitoring system 30 can include other spatial sensors and systems arranged to monitor the visible area 34 in front of the vehicle 100, e.g., an ambient camera or a radar sensor, which can be used to replace, supplement, or complete spatial information generated by the image sensor 32.Each of the spatial sensors is arranged in the vehicle to monitor the visible area 34 in order to detect nearby and distant objects, such as road features, lane markings, buildings, pedestrians, road signs, traffic lights and signals, other vehicles, and geographical features located near the vehicle 100. The spatial monitoring controller 35 can evaluate inputs from the spatial sensors to determine a linear range, relative velocity, and trajectory of the vehicle 100 with respect to each nearby and distant object. The spatial sensors can be located at various points on the vehicle 100, including the front corners, rear corners, rear sides, and mid-sides. In one embodiment, the spatial sensors may include a front radar sensor and a camera, although the disclosure is not limited to this.The placement of the spatial sensors enables the spatial monitoring controller 35 to monitor traffic flow, including nearby vehicles, intersections, lane markings, and other objects around the vehicle 100. As used herein, the terms "near," "close," and related terms refer to stationary and moving objects located in the vicinity of the vehicle 100 such that they can be detected by one or more of the spatial sensors connected to the spatial monitoring controller 35, including the spatial monitoring system 30.

[0024] The Telematics System 60 offers off-vehicle communication, including communication with a communication network system that can encompass both wireless and wired communication capabilities. The Telematics System 60 includes a telematics controller capable of off-vehicle communication, which includes Vehicle-to-Everything (V2X) communication. V2X communication encompasses short-range vehicle-to-vehicle (V2V) communication and communication with one or more roadside units, enabling localized communication between a large number of vehicles in similar locations. Alternatively or additionally, the Telematics System 60 is capable of short-range wireless communication with a handheld device, such as a mobile phone, satellite phone, or other telephone device.In one embodiment, the handheld device is loaded with a software application that includes a wireless protocol for communication with the telematics controller, and the handheld device performs the vehicle-external communication, including communication with an off-board controller via a communication network. Alternatively or additionally, the telematics system 60 performs the vehicle-external communication directly by communicating with the off-board control unit via the communication network.

[0025] The ADAS 20 is arranged to provide driver assistance functions by controlling one or more of the propulsion systems 15, the steering system 16, and the braking system 17 with little or no direct operator interaction. The ADAS 20 comprises a control unit and one or more subsystems that provide operator assistance functions, including one or more adaptive cruise control (ACC) systems, a lane keeping assist (LKY) system, a lane change assist (LCC) system, an autonomous braking / collision avoidance system, and / or other systems configured to command and control autonomous vehicle operation separately from or in conjunction with operator requests.The ADAS 20 can interact with and access an onboard map database to plan the route and control the operation of the vehicle via the lane keeping system, the lane centering system, and / or other systems configured to command and control autonomous vehicle operation. Autonomous operating commands can be generated to control the ACC system, the LKY system, the LCC system, the autonomous braking / collision avoidance system, and / or other systems. Vehicle operation includes operation in a propulsion mode in response to desired commands, which may include operator requests and / or autonomous vehicle requirements. Vehicle operation, including autonomous vehicle operation, includes accelerating, braking, steering, stationary driving, coasting, and idling.Operator requests can be generated based on operator inputs at an accelerator pedal, a brake pedal, a steering wheel, a gear range selector switch, the ACC system, etc.

[0026] The term "controller" and related terms such as microcontroller, control unit, processor, and similar terms refer to one or more combinations of application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), electronic circuits, central processing units (CPUs), such as microprocessors, and associated non-transitory memory components in the form of storage devices (read-only, programmable read-only, random access, disk drives, etc.). The non-transitory memory component is capable of storing machine-readable instructions in the form of one or more software or firmware programs or routines, combinational logic circuits, input / output circuits and devices, signal conditioning, buffering, and other components that one or more processors can access and execute to provide a described functionality.Input / output circuits and devices include analog-to-digital converters and related equipment that monitor sensor inputs, either at a preset sampling rate or in response to a triggering event. Software, firmware, programs, instructions, control routines, code, algorithms, and similar terms refer to sets of instructions executable by controllers, including calibrations and lookup tables. Each controller executes control routine(s) to provide desired functionality. The routines can be executed at regular intervals, such as every 100 microseconds during operation. Alternatively, the routines can be executed in response to the occurrence of a triggering event.Communication between controllers, actuators, and / or sensors can occur via a directly wired point-to-point connection, a networked communication bus connection, a wireless connection, or another suitable communication link. This communication involves the exchange of data signals in a suitable form, such as electrical signals over a conductive medium, electromagnetic signals over air, optical signals over fiber optics, and the like. The data signals can include discrete analog or digitized analog signals representing sensor inputs, actuator commands, and communication between controllers. The term "signal" refers to a physically perceptible indicator that conveys information and can be a suitable waveform (e.g., a waveform, a ripple, or a saturation).Electrical, optical, magnetic, mechanical, or electromagnetic phenomena (such as direct current, alternating current, sine wave, triangle wave, square wave, vibration, and the like) that can propagate through a medium. A parameter is defined as a measurable quantity that represents a physical property of a device or other element, detectable by one or more sensors and / or a physical model. A parameter can have a discrete value, e.g., either "1" or "0", or it can have a continuously variable value.

[0027] Fig. Figure 2-1 shows a pictorial FOV image 200 of a visible area 34, which is provided by an embodiment of the image sensor 32 of an embodiment of the vehicle 100. Fig. 1 was detected, which travels on a multi-lane road surface 210 that is planar in the x and y dimensions. Actual lane edges 212, 214, 216, 218 and 220 and detected lane edges 222, 224, 226, 228 and 230 are shown.

[0028] Fig. Figure 2-2 shows a bird's-eye view image 240 (BEV), which was created by performing an inverse perspective mapping on the FOV image 200 of the visible area 34 of Fig. 2-1 is generated.

[0029] Fig. 2-3 graphically shows an orthographic Fig. the actual lane edges, where lines 252, 254, 256, 258 and 260 represent the detected lane edges 222, 224, 226, 228 and 230 respectively, which are in Fig. 2-1 are shown and are derived from the BEV image 240, which is in Fig. 2-2 is shown. Since the multi-lane road surface 210 is flat in the x and y dimensions, there is no deformation of the orthographic representation of the detected lane edges 222, 224, 226, 228 and 230 with respect to the actual lane edges 212, 214, 216, 218 and 220.

[0030] Fig. Figure 3 shows a pictorial FOV image 300 of a visible area 34, which is provided by an embodiment of the image sensor 32 of an embodiment of the vehicle 100. Fig. 1 was detected, traveling on a multi-lane road surface 310, where the multi-lane road surface 310 has a difference in elevation and / or is curved, i.e., it changes in either the x-dimension or the y-dimension, or both. The actual lane edges 322, 324, 326, and 328 and the detected lane edges 312, 314, 316, and 318 are displayed, respectively. This result indicates that at least some of the detected lane edges 312, 314, 316, and 318 are distorted relative to the actual lane edges 322, 324, 326, and 328. The actual lane edges 322, 324, 326, and 328 represent the ground truth. The perspective projection, which assumes a plane, distorts the lane as it appears in the image. If a lane does not lie in the plane, the distorted perceptual projection can be used to derive a 3D perspective and predict the lane's position in 3D.

[0031] With continued reference to Fig. 1 will now be in Fig. Section 4 ff. describes a lane detection method 400 for processing FOV images 33 from the vehicle's onboard image sensor 32 of the vehicle 100 to form one or more predicted lanes that digitally represent the driving surface 10 located near the vehicle 100. The predicted lanes can be used for vehicle control, mapping, and other purposes. The lane detection method 400 comprises a concept, a framework, methods, and algorithms for processing FOV images 33 acquired by the vehicle's onboard image sensor 32. The image sensor 32 is positioned to monitor the visible area 34 encompassing the driving surface 10 near the vehicle 100 and periodically acquires the FOV images 33.Detecting a lane near vehicle 100 involves acquiring a field-of-view (FOV) image 33 of the road surface 10 in front of vehicle 100 via image sensor 32 and converting the FOV image 33 into a multitude of feature maps associated with the road surface 10 using an artificial neural network. The feature maps are projected onto an orthographic BEV grid using an inverse perspective mapping algorithm. The orthographic BEV grid contains a multitude of grid sections representing the ground truth. An example of a BEV orthographic grid is shown in [reference missing]. Fig. Figure 5 illustrates this. The feature maps contain lane segments represented in the plurality of grid sections of the orthographic BEV grid, and in one embodiment, the lane segments are represented as straight line segments in the plurality of grid sections. Alternatively, the lane segments can be represented as another parametric function in the plurality of grid sections, for example, by a second-degree polynomial equation. Each of the straight line segments can be parameterized as a normal vector defined with respect to a center point of the orthographic grid section, the normal vector being defined by a magnitude, direction, and height with respect to the center point of the orthographic grid section.The straight line segments are chained for the multiple orthographic grid sections to form one or more predicted lanes in the BEV orthographic grid, with chaining performed via learned feature embeddings. Chaining the line segments involves connecting lane segment edges identified in adjacent orthographic grid sections. The operation of vehicle 100 is controlled based on the predicted lane(s) in the BEV orthographic grid. This can include, for example, controlling the operation of vehicle 100, controlling the navigation and route planning of vehicle 100, transmitting the predicted lane(s) to nearby vehicles, updating the vehicle's internal and / or external map data based on this data, and generating a new map based on this data.

[0032] Referring again to Fig. Figure 4 describes the lane detection method 400 in an embodiment arranged as a backbone architecture comprising a training phase 410 for an artificial neural network (ANN) and an ANN inference phase 450, which uses an ANN 460 to extract BEV feature maps from the images. The ANN inference phase 450 involves identifying straight line segments in orthographic grid sections of the BEV feature maps. The straight line segments are concatenated to form lane edges in the orthographic BEV grids.

[0033] The ANN training phase 410 comprises the conversion, via an artificial neural network 420, of each of a multitude of training images 415 into a multitude of feature maps 430 and the projection, via an inverse perspective mapping algorithm, of the multitude of feature maps 430 onto a corresponding orthographic BEV grid, which is defined in the context of the ground truth. The implementation and execution of an inverse perspective mapping algorithm is self-evident and is not described in detail here.

[0034] The feature maps 430 contain a first feature map 431, which contains a segment rating for each of the grid sections of the orthographic BEV grid.

[0035] The feature maps 430 include a second feature map 432, which contains for each of the grid sections of the orthographic BEV grid a size (length) of the orthogonal line, e.g. an offset size 517 of an orthogonal line 515 with respect to a center point 512, which is in Fig. 5 is shown.

[0036] The feature maps 430 include a third feature map 433, which contains for each of the grid sections of the orthographic BEV grid an orientation (angle) of the orthogonal line, e.g. the angle 516 of the orthogonal line 515 with respect to the x-axis 11, which is in Fig. 5 is shown.

[0037] The feature maps 430 include a fourth feature map 434, which contains for each of the grid sections of the orthographic BEV grid a height (z-axis height) of the lane segment, e.g. a height of lane segment 520, which is in Fig. 5 is shown.

[0038] The feature maps 430 contain a fifth feature map 435, which provides a probability value for each of the grid sections of the orthographic BEV grid, indicating the probability that an edge of the lane segment passes through the respective orthographic grid section. The feature embedding is trained such that grid sections of the same lane segment have features that are close to and far from features of grid sections belonging to a different lane in an embedded space. This is done with reference to Fig. 8 described.

[0039] The training images 415 and the feature maps 430 can be used to train the artificial neural network 420, which generates a variety of coefficients 425 that are transferred to the ANN inference phase 450. In one embodiment, an image analysis process can be based on image processing that includes a manual feature analysis approach, which may involve manually extracting features and then separately training classifiers based on machine learning. Alternatively or additionally, a deep learning approach can be used to unify the feature extraction process and the classification step through multiple layers of an artificial neural network, such as the training artificial neural network 420.During the execution of a training process for an artificial neural network, the coefficients of the artificial neural network can be learned (425), and then the FOV image (33) is fed into a trained artificial neural network in real time in the form of the ANN inference phase (450). Offline training and online analysis are based on a common approach that includes training to learn the unknown parameters, with online analysis being performed to feed images into the parameter-learned approach for classification.

[0040] Fig. Figure 5 schematically shows an example of an orthographic BEV grid 500 associated with a sample FOV image 33 and illustrates features learned for each grid section 510. The orthographic BEV grid 500 represents a field of view located in front of the vehicle 100 and includes a first and a second predicted lane 502 and 504, respectively, delimited by a first, a second, and a third lane edge 501, 503, and 505. The orthographic BEV grid 500 comprises a variety of orthographically arranged grid sections 510 that represent the ground truth, which can be defined relative to the x-axis 12 and the y-axis 11. Each FOV image 33 is evaluated to extract information, including lane segments indicated by lane markings, colored lines, etc., which can indicate one of the lane edges, e.g. B.One of the first, second, and third lane edges 501, 503, and 505, which are used to form the first and second predicted lanes 502 and 504. Sections of the first, second, and third lane edges 501, 503, and 505 are captured in corresponding grid sections 510.

[0041] One of the orthographically arranged grid sections 510, labelled 510', is shown in an exploded view and includes a lane segment 520, which, as shown, is part of the third lane edge 505. The orthographic grid section 510' contains a center point 512, and the lane segment 520 is shown as a straight line segment. An orthogonal line 515 is specified from the center point 512 to the lane segment 520. The orthogonal line 515 is a line segment that is orthogonal to the lane segment 520 and contains the center point 512. The orthogonal line 515 may include features that comprise an angle 516 with respect to the x-axis 12 and an offset size 517 with respect to the center point 512. The lane segment 520 can contain features that include a z-axis height 518 with respect to a vehicle z-axis height of 100.The lane segment 520 can also contain features that include a parameter 519, which indicates the probability that one of the lanes passes through the respective grid section 510'. The aforementioned features can be recorded in a variety of feature maps.

[0042] A general curve representation is used for lane detection and localization tasks. Two path architectures are used to process the input image into a bird's-eye view (BEV) representation, where the BEV grid is divided into coarse grid sections and the parameters of each lane segment passing through these grid sections are regressed.

[0043] This representation is both compact and enables real-time calculations, as well as precise lane localization through regression of the exact lane position within each grid section. Furthermore, it is robust against unseen cameras and scenes with varying lane topologies and geometries. Camera robustness is achieved through the use of the BEV, which normalizes both camera internals and externals. Robustness against different topologies and geometries is achieved because lane segments are learned independently across different grid sections. This increases the size of the training set, as the stack size is determined based on the number of lane segments.In the extreme case, the task of lane detection as semantic segmentation is more robust in this sense; however, working on larger grid sections results in both faster computation and a deeper understanding of the nature of the lane line. Thus, each of the orthographically arranged grid sections 510 of the orthographic BEV grid 500 can be characterized by a center point, a linear lane segment, an orthogonal line between the linear lane segment and the center point (where the orthogonal line has features of an angle and an offset), a z-axis height, and a probability that the lane segment passes through it.

[0044] Referring again to Fig. 4 includes the ANN inference phase 450, the conversion of each FOV image 33 via the ANN 460 into a multitude of BEV feature maps 470, which are projected onto an orthographic BEV grid 500 via the inverse perspective mapping routine, an example of which is shown in Fig. 5 is shown.

[0045] The ANN inference phase 450 uses the ANN 460 and the multitude of coefficients 425 to extract BEV feature maps 470 from the FOV images 33 by projecting the multitude of feature maps onto the orthographic BEV grid via the inverse perspective mapping routine. The grid sections of the BEV orthographic grid of the BEV feature maps are interpreted to construct straight line segments (471), which are clustered and concatenated (472) to form lane(s) 475 predicted via learned features in the BEV orthographic grids. An example of a BEV feature map 500 containing a multitude of predicted lanes 502, 504 is shown schematically with reference to Fig. 5 shown.

[0046] Fig. Figure 8 shows a BEV orthographic grid 800, analogous to the BEV orthographic grid 500, which is based on Fig. Figure 5 shows that the BEV orthographic grid 800 contains a multitude of orthographically arranged grid sections representing the ground truth. Each grid section contains a feature vector represented in a third dimension. An element of the learning process is described with reference to the ANN training phase 410 of Fig. 4 shown and described. The feature vectors assigned to the orthographic BEV grid 800 are shown as clusters 811, 812, 813 and 814 respectively, and clusters 811, 812, 813 and 814 correspond to tracks 821, 822, 823 and 824 respectively of the example FOV image 820.

[0047] The concatenation of lane segments for the multitude of orthographic grid sections involves clustering the feature embeddings to achieve separation into individual lanes. The learning process directs the vectors of grid sections belonging to the same lane into the embedded space near the center and the vectors of grid sections belonging to different lanes into the embedded space farther away. This is illustrated by clusters 811, 812, 813, and 814 of three-dimensional element 810, which represent three principal components that are subjected to a linear decomposition such as principal component analysis (PCA) of the tile feature vectors. Principal component analysis (PCA) is a statistical technique that uses an orthogonal transformation to convert a set of observations of potentially correlated variables into a set of values ​​of linearly uncorrelated variables called principal components.The dimension of the feature vector is 16 in one embodiment and is a design choice. Three of the main components are shown.

[0048] FOV image 820 also shows clusters 811, 812, 813, and 814, which are projected onto lanes 821, 822, 823, and 824, respectively. A clustering algorithm can be applied to the embeddings. Clustering is a machine learning technique that involves grouping data points. Given a set of data points, a clustering algorithm classifies each data point into a specific group. Clustering, i.e., concatenation, requires that the grid sections are proximal, but not necessarily adjacent, to each other. The feature vectors can therefore be derived from lanes 821, 822, 823, and 824 by forming corresponding clusters 811, 812, 813, and 814, respectively.

[0049] In one embodiment, an image analysis process can be based on image processing that includes a manual feature analysis approach, which may involve the manual extraction of features and the subsequent separate training of machine learning-based classifiers. Alternatively or additionally, a deep learning approach can be used to combine the feature extraction process and the classification step through multiple layers of the artificial neural network. During the execution of a training process for the artificial neural network, the parameters of the artificial neural network are learned, and then the real-time image is fed into the trained artificial neural network.Offline training and online analysis are based on a common approach that includes training to learn the unknown parameters, with online analysis being performed to feed images into the parameter-learned approach for classification.

[0050] The predicted lane(s) 475, which were determined by the ANN inference phase 450, with reference to Fig. The data described in section 4 can be transmitted to the vehicle control unit 50, which can use the predicted lane(s) 475 to generate warning or advisory information or for dynamic vehicle control with regard to acceleration, braking, and cornering. The predicted lane(s) 475 can also be transmitted to the driver via the human-machine interface (HMI) 55.

[0051] The predicted lane(s) 475 can also be transmitted to the telematics system 60 for short-distance vehicle-to-vehicle (V2V) communication, communication with an intelligent motorway system, or communication with another external vehicle system.

[0052] In an embodiment of the vehicle 100 with autonomous functionality, the results of the predicted lane(s) 475 from the ADAS 20 can be used to autonomously actuate the vehicle brakes. Furthermore, the predicted lane(s) 475 from the ANN inference phase 450 can be used by the ADAS 20 to autonomously actuate a traction control system to reduce condensation on the vehicle brakes. Additionally, the predicted lane(s) 475 can be communicated to other vehicles via the telematics system 60. Finally, the predicted lane(s) 475 from the ADAS 20 and the HMI device 55 can be used to warn a driver.

[0053] Fig. Figure 6 schematically shows an embodiment of a lane detection method 650, analogous to the lane detection method 400, which refers to Fig. 4 is described. The lane detection method 650 is described for processing BEV images 633, which are obtained as FOV images from the vehicle-side image sensor 32 of the vehicle 100. The lane detection method 650 is designed for general lane detection and localization in three dimensions, together with the prediction of the positional uncertainty of each lane point. The input to the system is a single FOV image from a forward-facing imaging device, which is described in Fig. Figure 4 uses the ANN architecture shown. Each image is passed through an encoder, where the feature maps are projected onto a bird's-eye view (BEV) perspective using inverse perspective mapping to form the BEV image 633 (labeled 632). The output of the lane detection method 650 is a coarse-resolution top-view feature map, which is a downscaled version of the full-resolution BEV image. Each pixel in the coarse feature map is referred to as a grid section and has real-world dimensions in meters.

[0054] The dual-pathway architecture uses an encoder and inverse perspective mapping to project the feature maps onto the BEV in real dimensions. The network comprises two main processing pipelines, one in the FOV image view and the other in the BEV. The image view encoder consists of residual mesh (ResNet) blocks C1–C5 (660), with each ResNet block multiplying the number of channels of its predecessor. The BEV backbone contains projected image view feature maps and concatenates each projected image view feature map with the folded feature map from the previous block. The final BEV feature map is the input for the lane prediction header (670). The lane detection method (650) approximates a lane segment passing through one of the BEV grid sections as a straight line with parameters that can be regressed.The regressed parameters are the offset (in the xy-plane), angle, and height (z offset) of the segment relative to the grid centers and are chained together. An example from the BEV feature map 700, which contains a variety of predicted lanes 712, 714, 716, and 718, is shown schematically with reference to [reference missing]. Fig. 7 shown.

[0055] Segment Offsets: The prediction process includes the following. For each grid segment, roadway points are found that fall within the respective grid segment. The line passing through these points is parameterized, and its distance from the grid segment center is extracted. This distance is the magnitude of the segment's normal to the grid segment center ||Nxy| and is denoted as r. The height offset Δz is calculated relative to the BEV plane, which is linked to the feature maps and is a function of the image sensor's tilt angle and height. This feature enables the system's robustness across different cameras, as the lane height is output relative to the camera mounting height hcam and orientation θcam. The loss for the offset prediction is as follows: LOffsets=‖r^−r‖1+‖Δ^z−Δz‖1 where ^. denotes the estimated attribute.

[0056] Predicting segment orientation involves the following. Simple regression of an angle value is not trivial due to the cyclic nature of continuous angles. Another challenging property for angle prediction is the fact that the angle distribution is usually multimodal. Approaches to angle estimation range from regressing the sin and cos of the angle to representing the problem as a classification problem [3]. Due to the multimodal nature of the angle, a classification procedure is used in which the angle θseg is classified into one of four bins with center α = (0, π, π, 3π), and an offset relative to the chosen bin center is regressed. The angle-bin estimation is optimized using a soft multi-label target, and the ground truth is calculated as the angular proximity of the segment to the α-bin centers, e.g.,For θseg = 0, the class probability vector of the ground truth would be pα = (1, 0, 0, 0), and for θseg = π, the probability vector would be pα = (0.5, 0.5, 0, 0). The offsets δα are calculated between the ground truth angle and the bin centers. The angle offsets are monitored not only on the ground truth angle bin but also on the bins adjacent to it. This ensures that even if the bin class prediction is incorrect, the delta offset would be able to correct it. The angle loss is the sum of the classification and offset regression losses, which is determined as follows: Langle=∑a[pa⋅log S(pa^)+(1−pa)⋅log(1−S(pa^))+ma⋅‖δa^−δa‖1] where S(p^α) is the sigmoid function and mα is the indicator function that only masks the relevant bins for which the offset is to be learned. Tiles Point Prediction:

[0057] In addition to the offsets and orientation, the probability that a lane passes through each grid section is also output. The loss for this attribute is the binary cross-entropy loss, as follows: LResult = l ⋅ log PLane^ + (1 − l) ⋅ log 1 − PLane^ where l indicates whether the grid section is a background grid section, and Lane^ the network estimates for l is.

[0058] The network section loss is the sum across all network sections in the BEV network and can be determined as follows: Ltiles=∑i=1hbev∑j=1wbev(Lscoreij+lij⋅Langleij+lij⋅Loffsetsij) where hbev and wbev are the decimated BEV grid height and width, respectively, and where the lane grid section indicator lij multiplies the angle and offset losses because they are only monitored on lane grid sections. From grid sections to points

[0059] To arrive at lane points from the prediction of the segment score, offsets, and orientation in each grid segment, the grid segment scores are subjected to a threshold to identify only the lane grid segments. The polar terms r̂ and θseg^ are converted into Cartesian coordinates and the points from the BEV plane are transformed into the camera coordinate system by... Nocken and rotation by -θ cam transformed as follows. [x^y^z^]=[1000cos θcamsin θcam0−sin θcamcos θcam]⋅[r^⋅cos θseg^r^⋅sin θseg^Δ^z−hcam]

[0060] The current representation of grid segments assumes that there is no relationship between different grid segments. However, spatial relationships are present in the data and are captured by the receptive field, and each grid segment is learned independently. Nevertheless, there is a need to cluster the individual lane grid segments into whole lane units. A clustering mechanism can be applied directly to the lane segments, or another heuristic clustering method can be used. Another possibility is to use learned features that distinguish between grid segments belonging to different lanes.In one embodiment, a discriminative loss function is used that pulls the features of grid sections belonging to the same lane closer together in an embedded space, while shifting the features of grid sections belonging to different lanes further apart in that space. The discrimination loss on the decimated grid section grid requires far fewer computations than working at the pixel level. Considering the learned feature, a simple mean-shift algorithm is used to extract the grid sections belonging to individual lanes. The loss can be formalized as follows. Linen beds = Lvar + Ldist where Lvar is the pulling force and Ldist is the pushing force.

[0061] These terms are defined as follows: Lvar=1C∑c=1C1C∑i=1hbev∑j=1wbev[‖μc−lijc⋅fij‖−δv]+2 Ldist=1C(C−1)∑cA=1C∑cB=1CcA≠cB[δd−‖μcA−μcB‖]+2 where: C is the number of lanes, which can vary; Nc is the number of grid sections belonging to lane c, and lcij specifies the grid sections belonging to lane c, such that the following relationship holds: ∑c=1C1C∑i=1hbev∑j=1wbevlijc=Nc

[0062] The losses force the distances up to δv and the cross-cluster distance to be larger than δd.

[0063] The result is a new lane display that leads to accurate and robust lane detection and localization.

[0064] The concepts described here provide a method for determining a representation for accurate general 3D lane detection, including curved lanes. The representation uses grid sections positioned on a Bird's Eye View (BEV) grid, which in one embodiment includes a stipulation that the lane is locally linear in the BEV at each grid section. Alternatively, it can be determined that the lane is non-linear in the BEV, e.g., represented by a parametric equation such as a second-degree polynomial. This leads to the derivation of a compact and efficient lane representation for each grid section. This enables an overall efficient, low-resolution representation that maintains the fine resolution required for lane representation.

[0065] The block diagrams in the flowcharts illustrate the architecture, functionality, and operation of possible implementations of systems, procedures, and computer program products according to various embodiments of this disclosure. In this respect, each block in the block diagrams can represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function(s). It is also noted that each block in the block diagrams and / or flowchart representations, and combinations of blocks in the block diagrams and / or flowchart representations, can be implemented by dedicated hardware systems that perform the specified functions or actions, or by combinations of dedicated hardware and computer instructions.These computer program instructions may also be stored in a computer-readable medium that can instruct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable medium produce a manufactured item that implements the function / action specified in the flowchart and / or block diagram block or blocks.

[0066] The detailed description and the drawings or figures are supporting and descriptive of the present teaching, but the scope of the present teaching is defined exclusively by the claims. While some of the best modes and other embodiments for carrying out the present teaching have been described in detail, various alternative designs and embodiments exist for implementing the present teaching as defined in the appended claims.

Claims

[1] Method for detecting a lane near a vehicle (100), the method comprising: Capturing a field of view (FOV) image of a visible area (34) encompassing a driving surface (10) via an image sensor (32); Conversion of the FOV image into a multitude of feature maps via an artificial neural network (430); Projecting the plurality of feature maps (32) via an inverse perspective mapping routine onto a bird's-eye view orthographic grid (BEV), wherein the orthographic BEV grid contains a plurality of grid sections (510) representing a ground truth of the road surface (10), wherein the feature maps (32) comprise road segments and feature embeddings in the plurality of grid sections (510) of the orthographic grid in the BEV; Linking the lane segments (520) for the plurality of grid sections (510) based on the feature embeddings to form a predicted lane (502, 504) in the orthographic BEV grid, wherein the predicted lane (502, 504) is associated with the lane near the vehicle (100); and Control of the operation of the vehicle (100) based on the predicted lane (502, 504). [2] Method according to claim 1, wherein the linking of the lane segments (520) comprises connecting the lane segments (520) which are included in adjacent grid sections (510). [3] Method according to claim 1, wherein the lane segments (520) are represented as straight segments in the plurality of grid sections (510), and wherein each of the predicted lanes (502, 504) comprises a linked set of the straight segments in adjacent of the plurality of grid sections (510). [4] Method according to claim 1, wherein the lane segments (520) are represented as nonlinear line segments in the plurality of grid sections (510), and wherein each of the predicted lanes (502, 504) comprises a linked set of the nonlinear line segments in adjacent of the plurality of grid sections (510). [5] Method according to claim 1, wherein the linking of the lane segments (520) for the plurality of grid sections (510) comprises clustering the feature embeddings to achieve separation into individual lanes. [6] Method according to claim 1, wherein each of the lane segments (520) is parameterized as a normal vector defined with respect to a center point of the grid section (510), the normal vector being defined by a magnitude, a direction and a height with respect to the center point of the grid section (510). [7] Method according to claim 1, wherein the control of the operation of the vehicle (100) on the basis of the predicted lane (502, 504) comprises the control of a drive system (15), a steering system (16) or a braking system (17) via an advanced driver assistance system (20). [8] Method according to claim 1, wherein controlling the operation of the vehicle (100) based on the predicted lane (502, 504) comprises updating an in-vehicle map based on the predicted lane (502, 504). [9] Method according to claim 1, wherein the control of the operation of the vehicle (100) based on the predicted lane (502, 504) comprises the control of a vehicle-integrated navigation system (40) based on the predicted lane (502, 504). [10] Method according to claim 1, wherein capturing the image of the field of view (FOV) of the visible area (34) comprising the driving surface (10) comprises capturing the image of the field of view (FOV) of the visible area (34) comprising the driving surface (10) located in front of the vehicle (100).

Images