Image annotation system and computer program product

Abstract

An image annotation system includes: an input configured to receive a sequence of two-dimensional images captured by an image capture device of a vehicle; a modeling component configured to generate a three-dimensional structure model of the vehicle driving environment while capturing the image; a path determination component configured to determine, for each of the captured images, a corresponding position and orientation of the image capture device within the environment when the image is captured; an annotation section configured to generate two-dimensional image annotation data for annotating a region of an image corresponding to the three-dimensional structure model by geometrically projecting the three-dimensional structure model onto an image plane of each image defined by the corresponding image capture device position and orientation; a presentation component configured to present at least one of the images and two-dimensional image annotation data of the at least one of the images via the user interface; and a model adaptation component configured to adapt the three-dimensional structure model according to instructions received via the user interface.

Description

[0001] This application is a divisional application with an application date of March 13, 2019, application number 201980016667.1, and invention name “Image Annotation”. The parent application is an application entering the Chinese national phase of international application number PCT / EP2019 / 056356 submitted to the International Patent Office on March 13, 2019, claiming priority to application number 1804082.4 submitted to the UK Patent Office on March 14, 2018. The above application is fully incorporated herein by reference. Technical Field

[0002] The present disclosure relates to image annotation, such as the creation of annotated street road images that can be used (among other things) to train a machine learning road detection component. The present disclosure also relates to related systems and methods that can be used in this context, but the related systems and methods also have applications in other contexts. Background Art

[0003] A rapidly emerging technology is autonomous vehicles (AVs), which can navigate themselves on urban and other roads. Not only must such vehicles perform complex maneuvers between people and other vehicles, but they must do so frequently while ensuring strict constraints on the probability of adverse events, such as collisions with these other agents in the environment. In order for an AV to plan safely, it is critical to be able to accurately and reliably observe its environment. This includes the need for accurate and reliable detection of road structures in the vicinity of the vehicle.

[0004] Therefore, in the field of autonomous driving, a common requirement is to have a structure detection component (also called a machine vision component) that is able to determine real-world structure such as road or lane structure when given visual input, for example, which part of the image is the road surface, which part of the image constitutes a lane on the road, etc.

[0005] This is often accomplished through machine learning using convolutional neural networks. Such networks require large numbers of training images. These training images are similar to the images that would be seen by a camera in an autonomous vehicle, but they have been annotated with the information that the neural network needs to learn. For example, they will have annotations that mark which pixels on the image are road surface and / or which pixels of the image belong to lanes. At training time, the network is fed thousands or preferably hundreds of thousands of such annotated images, and the network learns on its own what features of the image indicate that a pixel is part of a lane or road surface. At runtime, the network can then make this decision on its own using images it has never seen before, i.e., images it has not encountered during training.

[0006] The conventional technique for creating annotated training images is to have humans manually annotate the images. Each image may take tens of minutes (and therefore incur very significant time and cost to obtain the large number of images required). Creating hundreds of thousands of training images using this manual process requires a lot of manpower, which in turn makes the manual process an expensive exercise. In fact, the manual process imposes a limit on the number of training images that can actually be provided, which in turn may be detrimental to the performance of the trained structure detection component.

[0007] An autonomous vehicle, also called a self-driving vehicle, refers to a vehicle that has a sensor system for monitoring its external environment and a control system that is able to use those sensors to automatically make and implement driving decisions. This includes in particular the ability to automatically adapt the speed and direction of travel of the vehicle based on input from the sensor system. A fully autonomous or "driverless" vehicle has sufficient decision-making capabilities to operate without any input from a human driver. However, the term autonomous vehicle as used herein also applies to semi-autonomous vehicles, which have more limited autonomous decision-making capabilities and therefore still require a certain degree of supervision from a human driver. Summary of the invention

[0008] The core problem addressed in this paper is to quickly and efficiently, but still accurately, annotate road images, where the annotations indicate visual road structures. Such images can be used, for example, to prepare a very large training set of annotated road images suitable for training a convolutional neural network (CNN) or other state-of-the-art machine learning (ML) structure detection component.

[0009] This problem is solved by using a novel automatic or semi-automatic image annotation technique, aspects and embodiments of which are described below. The technique allows a large number of training images to be generated more efficiently and faster than such manual annotation by removing or at least significantly reducing the need for human labor.

[0010] The performance of such a trained structure detection component is strongly dependent on both the quality and size of the training set. The ability to generate large training sets of accurately annotated road images (whose size may simply be impractical using conventional annotations) in turn means that it is possible to increase the accuracy of the trained structure detection component when applied (at runtime) to images it has not encountered in training.

[0011] A first aspect of the present invention provides a method for annotating road images, the method comprising implementing the following steps at an image processing system: receiving a time series of two-dimensional images captured by an image capture device of a moving vehicle; processing the images to reconstruct a path traveled by the vehicle in three-dimensional space; using the reconstructed vehicle path to determine expected road structures extending along the reconstructed vehicle path; and generating road annotation data by performing a geometric projection of the expected road structure in the three-dimensional space onto a two-dimensional plane of the image, for marking at least one of the images with the expected road structure location.

[0012] Preferably, only information derived from the image data captured by the image capture device (i.e. the vehicle path derived from the image data of the captured images, and optionally one or more reference parameters derived from the image data captured by the image capture device) is used to determine the expected road structure, without using any other vehicle sensors and without requiring any calibration of the image capture device. For example, one or more parameters of the image capture device (image capture device) and / or one or more parameters of the expected road structure may be derived (only) from the image data captured by the image capture device. This may be derived from the image data using one or more of the techniques described below. This provides a low-cost solution in which the only equipment required to acquire the images is a standard vehicle and a basic camera, such as a dash cam.

[0013] In an embodiment, the step of determining the expected road structure using the reconstructed vehicle path includes: determining the expected road structure boundary, which is defined as a curve in three-dimensional space extending parallel to the reconstructed vehicle path, wherein road annotation data is used to mark the location of the expected road structure boundary in the image.

[0014] The expected road structure locations in the image may be determined based on the path traveled by the vehicle after the image was captured.

[0015] The expected road structure locations in the image may be determined based on the path traveled by the vehicle before the image was captured.

[0016] Annotated road images can be used to train automatic road detection components for autonomous vehicles.

[0017] The expected road structure may be determined based on at least one of: a road width or a lane width, and an offset of the vehicle from a road boundary or a lane boundary.

[0018] A constant road or lane width or offset may be applied to at least a portion of the time series of images.

[0019] The road width or lane width may be determined based on one or more manual width adjustment inputs.These may be received according to the fourth aspect of the disclosure as described below.

[0020] Manual width adjustment input may be received with respect to one image and used to modify the expected road structure and project the modified road structure into one or more other of the images.

[0021] The expected road structure may be determined based on the road normal vector.

[0022] The road normal vector n estimated from the image is:

[0023] The expected road structure may be determined based on the forward direction vector.

[0024] The forward direction vector estimated from the image is:

[0025] f and n can be used to determine the boundary points of the expected road structure.

[0026] Boundary points can be determined as:

[0027] R i is the camera pose determined for image i when reconstructing the vehicle path, g i is the waypoint below the image capture device.

[0028] The road point under the camera can be calculated as g i =c i +hR i n

[0029] Where h is the height of the camera, c i is the 3D position of the image capture device determined when reconstructing the vehicle path.

[0030] The vehicle path can be reconstructed using a simultaneous localization and mapping (SLAM) process.

[0031] When performing the SLAM process, the camera pose R can be determined for each image i i .

[0032] When performing the SLAM process, the 3D camera position c can be determined for each image i i .

[0033] The expected road structure may be determined based on the relative angular orientation of the image capture device measured from images according to the fourth aspect set out below or any embodiment thereof.

[0034] The expected road structure may be determined based on the relative rotational orientation of the image capture device measured from images according to the third aspect set out below or any embodiment thereof.

[0035] The projection may be performed based on the height of the image capture device in the vehicle.

[0036] The height may be estimated based on an average height of the reconstructed path of the image capture device and a 3D road structure mesh calculated from the images.

[0037] Furthermore, it is also recognized that the image-based measurement techniques presented above can also be used in other situations besides the core issues presented above. Therefore, these constitute separate aspects of the present disclosure that can be applied and implemented independently.

[0038] A second aspect of the present invention provides a method for measuring the relative angular orientation of an image capture device within a vehicle, the method comprising implementing the following steps at an image processing system: receiving a time series of two-dimensional images captured by the image capture device of the vehicle; processing the images to reconstruct a path traveled by the vehicle in three-dimensional space, and estimating the absolute angular orientation of the image capture device for at least one reference point on the reconstructed path; determining a forward direction vector, the forward direction vector being defined as the vector interval between a point before a reference point on the reconstructed path and a point after the reference point on the reconstructed path; and determining an estimated angular orientation of the image capture device relative to the vehicle using the forward direction vector and the absolute angular orientation of the image capture device determined for the reference point.

[0039] In an embodiment, in the processing step, the absolute angular orientation of the image capture device can be determined for multiple reference points on the reconstructed path, and the determination and use steps are performed for each reference point to determine the estimated angular orientation of the image capture device relative to the vehicle for each reference point, and the method also includes a step of determining an average value of the relative image capture device angular orientation.

[0040] The average may be a weighted average.

[0041] Each relative image capture device angular orientation may be weighted according to the change in direction presented in the path between the earlier reference point and the later reference point.

[0042] The method may also include the following steps: determining a first separation vector between the reference point and a point before the reference point on the reconstructed path; and determining a second separation vector between the reference point and a point after the reference point on the reconstructed path; wherein each of the angular orientations relative to the image capture device is weighted according to the vector dot product of the first separation vector and the second separation vector.

[0043] The estimated angular orientation of the image capture device relative to the vehicle may be in the form of an estimated longitudinal axis of the vehicle in a reference frame of the image capture device.

[0044] The or each reference point may be equidistant from points preceding the reference point and points following the reference point.

[0045] The estimated angular orientation of the image capture device relative to the vehicle may be used together with an estimate of the angular orientation of the image capture device at another point on the path to determine an estimate of the absolute angular orientation of the vehicle at the other point on the path.

[0046] A three-dimensional surface map may be created when processing the captured images, wherein the absolute angular orientation of the vehicle at another point on the path is used to determine the distance between two points on the three-dimensional surface map along the axis of the vehicle at that point.

[0047] A three-dimensional surface map may be created when processing the captured images, wherein the absolute angular orientation of the vehicle at another point on the path may be used to determine the distance between the image capture device and a point on the three-dimensional surface map along the axis of the vehicle at that point.

[0048] A third aspect of the present invention provides a method for measuring the relative rotational orientation of an image capture device within a vehicle, the method comprising implementing the following steps at an image processing system: receiving a time series of two-dimensional images captured by the image capture device of the vehicle; processing the images to reconstruct the path traveled by the vehicle in three-dimensional space, and estimating the absolute rotational orientation of the image capture device for at least one reference point on the reconstructed path; determining a first separation vector between the reference point and a point before the reference point on the reconstructed path; determining a second separation vector between the reference point and a point after the reference point on the reconstructed path; and determining an estimated rotational orientation of the image capture device relative to the vehicle using the first separation vector and the second separation vector and the absolute rotational orientation of the image capture device determined for the reference point.

[0049] In an embodiment, the method may comprise the step of determining a normal vector to the reference point, the normal vector being perpendicular to the first and second separation vectors, wherein the normal vector and the absolute rotational orientation are used to determine an estimated rotational orientation of the image capture device relative to the vehicle.

[0050] The normal vector may be determined by calculating the vector cross product of the first separating vector and the second separating vector.

[0051] In the processing step, the absolute rotational orientation of the image capture device can be determined for multiple reference points on the reconstructed path, and the determination and use steps can be performed for each reference point to determine the estimated rotational orientation of the image capture device relative to the vehicle for each reference point, and the method can also include the step of determining an average value of the relative image capture device rotational orientation.

[0052] The average may be a weighted average.

[0053] The relative image capture device rotational orientation may be weighted for each of the reference points according to an offset angle between the first and second separation vectors at that reference point.

[0054] The relative image capture device rotational orientation may be weighted according to the magnitude of the vector cross product.

[0055] The relative image capture device rotational orientation may be weighted for each of the reference points according to a change in curvature of a path between a point before the reference point and a point after the reference point.

[0056] The estimated rotational orientation of the image capture device relative to the vehicle may be in the form of an estimated vertical axis of the vehicle in a reference frame of the image capture device.

[0057] The estimated rotational orientation of the image capture device relative to the vehicle may be used along with an estimate of the rotational orientation of the image capture device at another point on the path to determine an estimate of the absolute rotational orientation of the vehicle at another point on the path.

[0058] A three-dimensional surface map may be created when processing the captured images, wherein the absolute rotational orientation of the vehicle at another point on the path may be used to determine the distance between two points on the three-dimensional surface map along the axis of the vehicle at that point.

[0059] A three-dimensional surface map can be created when processing the captured images, wherein the absolute rotational orientation of the vehicle at another point on the path can be used to determine the distance between the image capture device and a point on the three-dimensional surface map along the axis of the vehicle at that point.

[0060] In the above-mentioned first aspect of the invention, a 3D road model is created based on the reconstructed vehicle path, which in turn can be projected into the image plane of the image to automatically generate 2D annotation data. As described above, embodiments of this aspect provide fast manual "fine-tuning" (which can be effectively applied to multiple images). In addition, it is also recognized that this same technique can be implemented in an image annotation tool with a 3D road model obtained by other means to obtain the same benefits of fast, manual modification across multiple images at the same time.

[0061] On the other hand, a method for determining an estimated orientation of an image capture device relative to a vehicle comprises: performing the steps of the second aspect or any embodiment thereof to estimate the angular orientation of the image capture device within the vehicle; and performing the steps of the third aspect or any embodiment to estimate the rotational orientation of the image capture device within the vehicle; wherein the estimated orientation of the image capture device relative to the vehicle comprises an estimated rotational orientation and an estimated angular orientation.

[0062] The estimated orientation of the image capture device relative to the vehicle can be used together with an estimate of the absolute orientation of the image capture device at another point on the path to determine the distance between two points on a three-dimensional surface map or between the image capture device and a point on the three-dimensional surface map along the axis of the vehicle at that point.

[0063] A fourth aspect of the present invention provides an image annotation system, comprising: an input configured to receive a sequence of two-dimensional images captured by an image capture device of a vehicle; a modeling component configured to generate a three-dimensional structural model of the environment in which the vehicle is traveling while capturing the images; a path determination component configured to determine, for each of the captured images, the corresponding position and orientation of the image capture device within the environment when the image was captured; an annotation component configured to generate two-dimensional image annotation data for marking an area of ​​the image corresponding to the three-dimensional structural model by geometrically projecting the three-dimensional structural model onto an image plane of each image defined by the corresponding image capture device position and orientation; a presentation component configured to present at least one of the images and its two-dimensional image annotation data via a user interface; and a model adaptation component configured to adapt the three-dimensional structural model according to instructions received via the user interface, wherein the annotation component is configured to modify the two-dimensional image annotation data for each of the captured images based on the adapted three-dimensional structural model, and the presentation component is configured to present the modified annotation data for at least one image via the user interface, thereby allowing a user to use at least one image and its two-dimensional image annotation data as a reference to simultaneously implement the modification of the two-dimensional image annotation data for multiple captured images.

[0064] In an embodiment, the three-dimensional structure model may include an expected road structure determined according to the first aspect or any claim dependent thereon.

[0065] The path determination component may be configured to determine the image capture position and orientation by processing the sequence of captured images.

[0066] The modeling component may be configured to determine an expected structure of the three-dimensional structural model based on a path through the environment defined by the determined image capture device position.

[0067] It is contemplated that the structure may extend parallel to the path defined by the image capture device position.

[0068] The modeling component may be configured to assign a label to at least one structure of the three-dimensional structural model, and the labeling component may be configured to associate the label with a corresponding region marked in the image.

[0069] The modeling component may be configured to assign parallel structure labels to expected parallel structures.

[0070] The modeling component may be configured to assign one of the following labels to the expected parallel structure: a road label, a lane label, a non-drivable label, an ego lane label, a non-ego lane label, a bicycle or bus lane label, a parallel road marking, or a boundary label.

[0071] It is contemplated that road structure may extend perpendicular to the path defined by the image capture location.

[0072] The modeling component may be configured to assign vertical structure labels to expected vertical structures.

[0073] The modeling component may be configured to assign intersection labels, vertical road marking or boundary labels, yield line labels to prospective vertical structures.

[0074] The three-dimensional structural model may have at least one structure created by a user via a user interface.

[0075] The three-dimensional structure model may have at least one structure corresponding to a stationary object.

[0076] The structure corresponding to the stationary object can be user created.

[0077] The modeling component may be configured to generate the three-dimensional structural model using one or more of the following reference parameters: a position of the image capture device relative to the vehicle, an orientation of the image capture device relative to the vehicle, a distance between two real-world elements measured along an axis of the vehicle.

[0078] The image annotation system may include a parameter calculation component configured to estimate one or more reference parameters by processing image data captured by an image capture device of the vehicle.

[0079] The image data may include image data of a captured sequence of images.

[0080] The model adaptation component can be configured to create a new structure of the three-dimensional structural model in response to a parallel structure creation instruction received via a user interface, the new structure being oriented to be parallel to at least one of: a path defined by the position of the image capture device, and an existing structure of the three-dimensional structural model, wherein the orientation of the new structure is automatically determined by the model adaptation component based on the orientation of the path or the existing structure.

[0081] The model adaptation component may be configured to adapt an existing parallel structure of the three-dimensional structure model in response to a parallel structure adaptation instruction received via the user interface.

[0082] The model adaptation component can be configured to change the width of an existing parallel structure, or to change the position of an existing parallel structure.

[0083] The pattern adaptation component may be configured to change the width of the second portion of the existing structure while keeping the width of the first portion of the parallel structure fixed.

[0084] The model adaptation component may be configured to interpolate an intermediate of the adapted structure between the first portion and the second portion.

[0085] A fifth aspect of the present invention provides a positioning system for an autonomous vehicle, the positioning system comprising: an image input configured to receive a captured image from an image capture device of the autonomous vehicle; a road map input configured to receive a predetermined road map; a road detection component configured to process the captured image to identify the road structure therein; and a positioning component configured to determine the position of the autonomous vehicle on the road map by matching the road structure identified in the image with the corresponding road structure of the predetermined road map.

[0086] A vehicle control system is provided that includes a positioning system and a vehicle control component configured to control operation of an autonomous vehicle based on a determined vehicle position.

[0087] A sixth aspect of the present invention provides a road structure detection system for an autonomous vehicle, the road structure detection system comprising: an image input configured to receive a captured image from an image capture device of the autonomous vehicle; a road map input configured to receive predetermined road map data; and a road detection component configured to process the captured image to identify the road structure therein; wherein the road detection component is configured to merge the predetermined road map data with the road structure identified in the image.

[0088] A vehicle control system is provided that includes road structure detection and a vehicle control component configured to control operation of an autonomous vehicle based on the merged data.

[0089] A seventh aspect of the present invention provides a control system for an autonomous vehicle, the control system comprising: an image input configured to receive a captured image from an image capture device of the autonomous vehicle; a road map input configured to receive a predetermined road map; a road detection component configured to process the captured image to identify the road structure therein; and a map processing component configured to select a corresponding road structure on the road map; and a vehicle control component configured to control the operation of the autonomous vehicle based on the road structure identified in the captured image and the corresponding road structure selected on the predetermined road map.

[0090] In an embodiment of the seventh aspect, the control system may include a positioning component configured to determine a current position of the vehicle on a road map. The road detection component may be configured to determine the position of the identified road structure relative to the vehicle. The map processing component may select a corresponding road structure based on the current position of the vehicle, for example by selecting an area of ​​the road map containing the corresponding road structure based on the current vehicle position (e.g., corresponding to the expected field of view of the image capture device), for example, so as to merge the area of ​​the map with the identified road structure. Alternatively, the map processing component may select the corresponding vehicle structure by comparing the road structure identified in the image with the road map to match the identified road structure with the corresponding road structure, for example to allow the positioning component to determine the current vehicle position based thereon (e.g., based on the position of the identified road structure relative to the vehicle).

[0091] Various embodiments of the invention are defined in the dependent claims. Features described or claimed in relation to any one of the above aspects of the invention may be implemented in embodiments of any other aspect of the invention.

[0092] A computer program product is provided, comprising code stored on a computer readable storage medium and configured to implement the method or system of any preceding claim when executed on one or more processors. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] For a better understanding of the invention, and in order to show how embodiments of the invention may be implemented, reference is made by way of non-limiting examples to the following drawings, in which:

[0094] Figure 1 shows a highly schematic functional block diagram of a training system for training a structure detection component;

[0095] Figure 2 A highly schematic block diagram of an autonomous vehicle is shown;

[0096] Figure 3 A highly schematic block diagram of a vehicle for capturing annotated road images is shown;

[0097] Figure 3A shows a schematic front view of a vehicle;

[0098] Figure 4 A schematic block diagram of an image processing system is shown;

[0099] Figure 4A yes Figure 4 Expansion of the system;

[0100] Figure 5 On the left hand side a flow chart for an automatic image annotation process is shown, on the right hand side an example illustration of the corresponding steps of the method is shown;

[0101] Figure 6 A schematic block diagram indicating the functions of an image system to facilitate adjustments during a human correction phase is shown;

[0102] Figure 7 The principle of simultaneous localization and mapping (SLAM) technology for reconstructing vehicle paths is shown;

[0103] Fig. 7A and Figure 7B shows the 3D orientation of the camera expressed using pitch, roll and yaw angles;

[0104] Figure 8 shows a schematic perspective view of a vehicle;

[0105] Fig. 9 shows a schematic perspective view of a vehicle equipped with at least one image capture device;

[0106] Fig.10 Certain principles of image-based measurement of angular orientation of an image capture device are shown;

[0107] Fig.11 Some principles of image-based measurement of road width are shown;

[0108] Fig.12 An example of manual adjustments during the human correction phase is shown;

[0109] Fig.13 An example of an annotated road image is shown;

[0110] Fig.14 Example SLAM output is shown;

[0111] Fig.15 Some principles of image-based measurement in the forward direction are shown;

[0112] Fig.16 Some principles of image-based measurement of road normals are shown;

[0113] Fig.17 Example image showing SLAM reconstruction of a camera position moving through a point cloud world;

[0114] Figures 18 to 24 An example of an annotation tool user interface for annotating an image based on the techniques disclosed herein is shown;

[0115] Fig.25 shows continuous annotation applied to a road image;

[0116] Fig.26 showing a reference point defined relative to an image capture device of the vehicle;

[0117] Fig. 27 shows further details of the road normal estimation process; and

[0118] Fig.28 A top view of the ego lane estimation is shown. DETAILED DESCRIPTION

[0119] Autonomous vehicles require knowledge of the surrounding road layout which can be predicted by state-of-the-art CNNs. This work addresses the current lack of data for determining lane entities, which is required for various driving maneuvers. The main problem is the time-consuming manual annotation process for each image that is typically applied.

[0120] The present disclosure recognizes that driving a car is a form of annotation of the car itself. This is advantageous in a semi-automatic approach that allows an image sequence to be efficiently labeled by utilizing an estimated road plane in 3D and projecting labels from that plane into all images of the sequence. The average labeling time per image is reduced to 5 seconds, and only an inexpensive dashcam is required for data capture.

[0121] Autonomous vehicles have the potential to revolutionize urban transportation. Mobility will be safer, always available, more reliable and provided at a lower cost.

[0122] An important issue is to give autonomous systems knowledge about the surrounding space: self-driving cars need to know the road layout around them in order to make smart driving decisions. The present disclosure addresses the problem of detecting driving lane instances from a camera mounted on the vehicle. Separate spatially constrained lane instance regions are required to perform various challenging driving maneuvers, including lane changes, overtaking, and crossing intersections.

[0123] Typical state-of-the-art CNN models require a large amount of labeled data to reliably detect lane instances. However, few labeled datasets are publicly available, primarily due to the time-consuming annotation process; for semantic segmentation tasks, fully annotating images can take anywhere from several minutes to over an hour per image. In contrast, our semi-automatic annotation process reduces the average time per image to seconds. This speedup is achieved by (1) noting that driving a car is a form of annotation of the car itself and that cars primarily drive along lanes, (2) propagating manual label adjustments from a single view to all images in the sequence, and (3) accepting unlabeled parts in ambiguous cases.

[0124] Some previous work has aimed to create semi-automatic object detection in autonomous driving scenarios. Reference

[27] proposes to detect and project the future driving path in the image, but does not address the problem of lane annotation. This means that the path is not adaptive to lane widths and crosses lanes and intersections. In addition, it requires an expensive sensor set including calibrated cameras and lidars. In contrast, the present method can be applied to data from GPS-enabled dashcams. The contributions of the present disclosure include:

[0125] - A semi-automatic annotation method for lane instances in 3D that requires only inexpensive dashcam equipment;

[0126] -Road surface annotation in dense traffic scenarios despite obstruction;

[0127] - Experimental results on road, ego-lane and lane instance segmentation using CNN.

[0128] A method is described below that provides a fully automated way to generate training data that is of only slightly lower quality than the traditional manual process.

[0129] An extension to this is also described, which introduces a manual correction ("human correction") stage. With this extension, the method becomes a semi-automatic method that produces quality as good as traditional manual methods, but takes about 100 times less human effort in terms of measured annotation time per image. This is based on the observation that typical annotation times - using fully manual annotation of the type currently used - can be anywhere from 7 minutes to 90 minutes per image; whereas using the method described below, annotation times of about 12 seconds per image can be achieved.

[0130] In the method described below, the training images are frames of video images. As will become apparent, the method described is particularly suitable for batch annotation of video frames captured by a moving training vehicle. For typical training videos (consisting of a sequence of static 2D images (i.e. frames)), it has been possible to achieve an annotation time of 7 minutes per typical training video sequence, which amounts to about 12 seconds per image, while still achieving good quality results.

[0131] Figure 1 A highly schematic functional block diagram of a training system for training a structure detection component 102 (road detector) based on annotated street scene images (also referred to herein as road images) (i.e., street scene images with associated image annotation data) is shown. Figure 1 In the example, a street scene image is labeled 104, and its corresponding image annotation data is labeled 106. The annotation data 106 marks the location (multiple locations) of specific structures (e.g., roads, lanes, intersections, non-drivable areas, etc.) within the image 104 and possible targets (e.g., other vehicles, pedestrians, street signs or other infrastructure, etc.) within the image.

[0132] The images may be divided into a training dataset, a testing dataset, and a validation dataset, labeled 108a, 108b, and 108c, respectively.

[0133] The detection component 102 takes as input the image data of a street scene image labeled i and generates an output y(i), where y is a function defined by a set of model parameters of the detection component 102. This may be in the form of a feature vector derived from the image, for example, which may be formed from raw or transformed pixel values.

[0134] The detection component 102 is trained based on the training image 108a to match its output y(i) with the corresponding labeled data. This is a recursive process in which the input component 112 of the training system 110 systematically inputs the image data i of the training image 108b to the road detector 102, and the training component 114 of the training system 110 adapts the model parameters to try to minimize the objective function, which provides a measure of the difference between the output y(i) of each training image and the corresponding labeled data 106.

[0135] The detection component 102 may be, for example, a convolutional neural network, where the model parameters are the weights between neurons.

[0136] The test data 108b is used to minimize overfitting, which is defined as the point beyond which increasing the accuracy of the detection component 102 on the training data set 108a is detrimental to its ability to generalize to images not encountered during training. Overfitting can be identified as the point at which increasing the accuracy of the detection component 102 on the training data 108 reduces (or does not increase) its accuracy on the test data, where accuracy is measured according to an objective function. The goal of training is to minimize the objective function to the extent that it can be minimized without overfitting.

[0137] If desired, the validation data set 108c may be used to provide a final assessment of the performance of the inspection component.

[0138] Such machine learning techniques are known per se and are therefore not described in further detail here.

[0139] The method described below may be used to automatically or semi-automatically generate such annotated data 106 for use in training, testing, and / or validating the detection component 102 .

[0140] Figure 2 A highly schematic block diagram of an autonomous vehicle 200 is shown, which is shown to include an instance of a trained detection component 102 having an input connected to an image capture device 202 of the vehicle 200 and an output connected to an autonomous vehicle controller 204. In use, the trained structure detection component 102 of the autonomous vehicle 200 detects structures within images captured by the image capture device 102 in real time according to its training, and the autonomous vehicle controller 204 controls the speed and direction of the vehicle based on the results, without or with limited input from any human.

[0141] Despite Figure 2 Only one image capture device 202 is shown in FIG, but the autonomous vehicle may include multiple such devices. For example, a pair of image capture devices may be arranged to provide a stereoscopic view, and the road structure detection method may be applied to the images captured by each of the image capture devices.

[0142] As will be appreciated, this is a highly simplified description of certain autonomous vehicle functionality. The general principles of autonomous vehicles are known and therefore not described in further detail. 2Video Collection

[0143] For the purposes of the experiments detailed later, video and associated GPS data were captured using a standard Nextbase 402G professional dash cam at 1920×1080 resolution at 30 frames per second and compressed using the H.264 standard (however, any suitable low-cost image capture device may be used to achieve the same benefits). The camera was mounted on the inside of the car windshield, roughly along the centerline of the vehicle and approximately aligned with the axis of motion.

[0144] Fig.26 (Top left) shows an example image from the collected data. To remove parts where the car was moving very slowly or stationary (common in urban environments), only frames that were at least 1 m apart according to GPS were included. Finally, the recorded data was segmented into sequences of length 200 m.

[0145] Fig.25 An example road image (top left) is shown with annotations for the road (top right), ego lane (bottom left), and lane instances (bottom right). The road and lane under the vehicle are annotated, although they are occluded. Uncolored portions are not annotated, i.e., the category is unknown.

[0146] Figure 3 A simplified block diagram of a vehicle 300 is shown, which can be used to capture road images to be annotated, i.e., as described above with reference to Figure 1 Preferably, these images are captured as frames of short video segments recorded as the vehicle 300 travels along the road, in such a way as to allow reconstruction of the path traveled by the vehicle during the recording of each video segment from the frames of the segment. The vehicle 300 may be referred to as a training vehicle, as it is used to compare it with the vehicle 300. Figure 2 300 is a convenient shorthand for distinguishing autonomous vehicle 200 from training vehicle 300. Training vehicle 300 is shown as including image capture device 302, which may be a forward-facing or rearward-facing image capture device, and coupled to processor 304. Processor 304 receives captured images from image capture device 302 and stores them in memory 306, from which the images may be retrieved for use in the manner described below.

[0147] In this example, vehicle 300 is a car, but it may be any type of vehicle.

[0148] The basis of the present invention is the assumption that the path travelled by a human-driven training vehicle 300 extends along a road, and therefore the location of the road can be inferred from any path taken by the training vehicle 300. When it comes to annotating a particular image in a captured sequence of training images, it is the hindsight of the path subsequently taken by the training vehicle 300 after that image was captured that enables automatic annotation. In other words, hindsight of the vehicle's behaviour after the captured image is exploited in order to infer the location of the road within the image. The annotated road location is therefore the expected and given path subsequently travelled by the training vehicle 300 and potential assumptions about how this relates to the location of the road.

[0149] As described in further detail below, the path is determined by processing the captured images themselves. Thus, when annotating a particular image with an expected road location, for the (corresponding to rearward facing) front-facing image capture device 302, the expected road location in the image is determined from the path traveled by the vehicle after (corresponding to before) the image is captured, such as reconstructed using at least one of the subsequently (corresponding to previously) captured images in the captured image sequence. That is, each annotated image is annotated using path information derived from one or more of the captured images after (corresponding to before) the image is annotated. 3. Video Annotation

[0150] Fig. 27 shows the estimated lane boundary point at frame i The principle of i is the camera position at frame i (obtained via SfM), f is the forward direction, n is the normal vector to the road plane (both are oriented relative to the camera), h is the height of the camera above the road, r is the horizontal vector across the lane, are the distances to the left ego lane boundary and to the right ego lane boundary, respectively.

[0151] The initial annotation step is automatic and provides an estimate of the road surface in 3D space, as well as an estimate of the ego lane (see Section 3.1). The estimates are then manually corrected and further annotations are added in road surface space. The labels are then projected into the 2D camera view to allow all images in the sequence to be annotated at once (see Section 3.2). 3.1 Automatic 3D Ego Lane Estimation

[0152] Given a video sequence of N frames from a camera with unknown intrinsic and extrinsic parameters, the goal is to determine the road surface in 3D and project an estimate of the ego lane onto this surface. To do this, we first apply OpenSfM

[28] , a “structure from motion” algorithm, to obtain the 3D camera position c for each frame i∈{1,...,N} in the global coordinate system.i and posture R i , as well as the camera focal length and distortion parameters.

[0153] The road is assumed to be a 2D manifold embedded in a 3D world. Furthermore, the local curvature of the road is low, so the orientation of the wheels provides a good estimate of the local surface gradient. The camera is fixed inside the vehicle with a static translation and rotation from the current road plane (i.e., the body is assumed to follow the road plane and suspension motion is neglected). Thus, the ground point g on the road below the camera at frame i is i It can be calculated as g i =c i +hR i n

[0154] where h is the height of the camera above the road and n is the surface normal of the road relative to the camera (see Fig.26 , left). Then, the left ego lane boundary and the right ego lane boundary can be derived as

[0155] where r is a vector in the road plane, perpendicular to the direction of travel, are the offsets to the left and right ego lane boundaries, respectively. See for illustration Fig.26 (Right). A simplifying assumption is made that the road surface is flat and perpendicular to the direction of the car's motion (but no assumption is made that the road is generally flat - if the ego-path drives over a hill, this is captured in the ego-path).

[0156] Given a frame i, all future lane boundaries

[0157] It can be projected into the local coordinate system by the following formula

[0158] Then, lane annotations can be drawn as polygons in adjacent future frames, i.e., with corner points

[0159] This implicitly assumes that the lane is piecewise straight and flat between the captured images. The following sections describe how to measure or otherwise obtain the following quantities:

[0160] Note that h, n and r only need to be estimated once for all sequences with the same camera position.

[0161] The camera height ft above the road is easy to measure manually. However, in cases where this cannot be done (e.g., for dashcam videos downloaded from the web), the camera height can also be obtained using an estimated grid of the road surface obtained from OpenSfM. A rough estimate of h is sufficient, as it will be corrected by manual annotation, see the following section.

[0162] Fig. 27 shows the estimation of the road normal n and the forward direction f at a single frame i i The principle is that the final estimate is the set of all frames.

[0163] The road normal n is estimated based on the fact that as the car moves around a turn, the vector m representing its motion will all lie in the plane of the road, and so taking their cross product will yield the road normal, see Fig. 27 .

[0164] Let m i,j is the normalized motion vector between frames i and j, i.e.

[0165] The estimated road normal at frame i (in camera coordinates) is

[0166] in represents the cross product (see Fig. 27 ). The quality of this estimate is highly dependent on the correctness of the previous assumptions. To obtain a more reliable estimate, n i can be averaged over the entire trip, and implicitly weighted by the magnitude of the cross product:

[0167] The normal may be estimated only during turns, and therefore the weighting scheme emphasizes sharp turns and ignores the straight portion of travel.

[0168] r is perpendicular to the forward direction f and in the plane of the road, so

[0169] The only quantity remaining is f, which can be derived using the following fact:

[0170] If the turn rate is low, then m i-1，i+1 Approximately parallel to c i Therefore, the forward point at frame i can be estimated

[0171] (See Fig. 27 ). For normal conditions, f ican be averaged over the entire trip to get a more reliable estimate:

[0172] In this case, the motion is weighted according to the inner product so that parts with low turning rates are emphasized, while the maximum value ensures forward motion.

[0174] quantity and, This is important to obtain correct alignment of the labeled lane boundaries with the visible boundaries.

[0175] To estimate these, it can be assumed that the ego lane has a fixed width w and the car is driving exactly in the center, i.e.

[0176] and is constant for all frames. In the extensions (see the following section), this assumption is relaxed to obtain improved estimates through manual annotations.

[0177] In practice, sequences with many turns in the road plane are chosen to estimate n, and straight sequences are chosen to estimate f. The same value is then reused for all sequences with the same static camera position. Only the first part of the sequence is annotated, until 100m from the end, because otherwise not enough future lane boundary points can be projected. An outline of the automatic ego lane annotation procedure is provided in Appendix A (Algorithm 1) and in Fig.28 A visualization of the automatic boundary point estimates is shown in (see below) and annotated.

[0178] Further details are described below.

[0179] Figure 4 A schematic block diagram of an image processing system is shown, which operates according to the present invention to automatically generate a Figure 3 Annotation data for training images captured in the described manner. The annotation data marks expected road locations within the images, where these locations are inferred using the above assumptions. Figure 4A An extension of this system is shown and will be described below. Figure 4A The system includes Figure 4 all parts of the Figure 4 All descriptions also apply to Figure 4A .

[0180] Figure 5 On the left hand side a flow chart for an automatic image annotation process implemented by the image capture system is shown, and on the right hand side example illustrations of the corresponding steps are shown.

[0181] Now we will describe Figure 4 and 4A Image system and Figure 5 process.

[0182] Figure 4 The image processing system is shown to include a path reconstruction component 402 , a road modeling component 404 , and an image annotation component 406 .

[0183] The path reconstruction component 402 receives the sequence of captured two-dimensional (2D) images 400 and processes them to create a three-dimensional (3D) reconstruction of the path traveled by the vehicle from the sequence of captured images (step 502, Figure 5 ).

[0184] On the right side of step 502, Figure 5 Two reconstructed paths in three-dimensional space are shown.

[0185] The first path, labeled CP (Camera Path), is a 3D reconstruction of the path traveled by the image capture device (camera) 302 of the training vehicle 300. Techniques for reconstructing the path traveled by a moving camera from a sequence of images captured as the camera moves are known in the art and are therefore not described in detail here.

[0186] The path can be recreated from other data (accelerometers, highly accurate GPS, etc.), however, since a basic dash cam can be used for data collection rather than an expensive car containing expensive accelerometers, expensive and accurate GPS, etc., reconstruction from video alone advantageously makes the cost of capturing the data very cheap.

[0187] The second path, labeled VP (Vehicle Path), is a reconstruction of the path traveled by the training vehicle 300, which is defined as a path consisting of a vehicle at approximately half the width of the training vehicle 300 (i.e., equidistant from the right-hand and left-hand sides of the training vehicle) and below the image capture device 300 at approximately road level (at Figure 3 and 3A That is, at the height of the road and along the center line (C, Figure 3A ), which extends vertically and is located at a midpoint along the width of the vehicle 300.

[0188] Note that it is not necessary that the points are exactly equidistant - this method has been found to give accurate results even when the points are not equidistant.

[0189] If the camera 302 is at a height above the road surface (at Figure 3 , Figure 3A and Figure 5 H), the horizontal offset of the camera 302 from the center line C (in Figure 3A and Figure 5 If the camera path CP is known, then the vehicle path VP at ground level is determined directly from the camera path CP. The relative orientation of the camera can be captured as a "forward point" and a "horizontal line" as defined below.

[0190] Notice, Figure 5 The example shown to the right of is illustrative rather than exhaustive and is intended only to illustrate by way of example the more general principle according to which a 3D camera path CP can be used to infer the position of a road structure using a small number of reference parameters.

[0191] The road modeling component 404 creates a 3D model of the road that is assumed to have been traveled by the vehicle based on the 3D path reconstruction (602, Figure 6 )(Step 504, Figure 5 ).

[0192] In the example shown on the right side of step 504, the road model is formed by two geometric curves R1, R2 corresponding to the assumed lane boundary positions, the two geometric curves R1, R2 are defined to be placed parallel to the vehicle path VP on either side of the vehicle path VP and at the same height as the vehicle path VP, and each geometric curve is a distance W / 2 away from the vehicle path VP, where W is the assumed road width or lane width.

[0193] This is based on the assumption that the training vehicle 300 is traveling along a road lane, that the traveling direction of the training vehicle 300 is substantially parallel to the actual lane boundary of the lane, and that it is traveling in the approximate center of the lane.

[0194] The expected road position may, for example, correspond to the edge of the road itself (in order to mark a non-drivable area on or beside the road), or to a lane position within the road.

[0195] In general, automatic annotation techniques can be used to mark the locations of what are referred to herein as "parallel structures", i.e., structures that are expected to extend at least approximately parallel to the path traveled by the vehicle, or more generally, structures that are expected to have a particular orientation relative to the vehicle path. This can be a road structure, such as a road, a lane, a non-drivable area that is assumed to be parallel to the vehicle path, an intersection that is assumed to extend perpendicular to the vehicle path, or a real-world parallel structure that a vehicle other than a car may encounter (e.g., a runway (in the case of an autonomous drone, airplane, etc.)). Therefore, all descriptions of the 3D road model are equally applicable to any other form of 3D structure model of the environment in which the vehicle travels, for which the expected parallel structure can be automatically determined using the methods described herein.

[0196] Based on the basic assumption that the parallel structure is parallel to the path that the training vehicle 300 actually travels, different assumptions may be used to accommodate different types of parallel structures.

[0197] The description herein may specifically refer to roads or lanes, and related parameters such as road or lane width. However, it should be understood that the description is applicable to any parallel road structure, and any reference to road or lane width (etc.) is generally applicable to the width parameters determined for the intended parallel structure.

[0198] The image annotation component 406 uses the 3D road model to generate 2D annotation data so as to individually label individual 2D training images with expected road locations (step 506, Figure 5 ).

[0199] like Figure 5 As shown on the right side of step 506 in , the annotation data A(n) for the nth 2D image in the sequence (image n, which can be any image in the captured training image sequence) is generated by performing a geometric projection of the 3D road model onto the plane of that image (labeled I(n)). In this example, this means projecting the 3D road boundaries R1, R2 onto the image plane I(n) so that the projection of those boundaries in the image can mark where the road boundaries are assumed to be located in image n.

[0200] Image plane I(n) is the plane corresponding to the field of view of camera 302 when image n was captured, and is therefore located at a point corresponding to the position of the vehicle when image n was captured.

[0201] The parameter calculation component 408 calculates various “reference parameters” used to construct a 3D model from the reconstructed vehicle path and perform geometric projection. These reference parameters include the following image capture parameters related to the position and orientation of the image capture device 302 of the training vehicle 300: 1) The height H of the camera 302 above the road; 2) The horizontal offset S between the camera and the center line C; 3) The orientation of the camera 302 relative to the vehicle, which can be captured as: a. the "forward point" of camera 302 (defined below), and b. The camera's "horizon" (defined below); as well as: 4) The width W of the road / lane.

[0202] These are used in the manner briefly described above, and further details of how their calculation is used are described below. It is sufficient for now to say that these parameters are calculated from the captured training image 400 itself, which has various benefits described later.

[0203] The annotation data is stored in electronic storage 414 from which the annotation data may be accessed or retrieved for use in the above references. Figure 1 In this process, finally, the automatic structure detection component 102 learns from the ability of a human driver to keep the vehicle 300 at approximately the center of the road most of the time when the training images are captured. 3.2 Manual Correction and Additional Annotation

[0204] The annotation interface provides human annotators with the ability to view batches of frames individually, while projecting the currently presented lanes into the image. These lanes can be widened, narrowed, and moved by the annotator in the most convenient frame, and these changes to the lane structure will be projected to all other frames in the batch, providing a clear advantage over annotating all images individually.

[0205] Fig.28 A visualization of the ego lane estimation from above is shown. The automatic estimation and manual correction are marked as

[0206] Initially, all and is set to a constant value, so the estimate follows the travel path, but as can be seen, the lane boundaries are not labeled correctly. The annotator then corrects and This estimate has since been improved.

[0207] Figures 18 to 24 An example of an annotation interface used by annotators is shown. Fig.18In the center of the image, the ego-path can be seen projected into the frame. At the bottom left, the annotator is provided with controls to manipulate the currently presented lanes (narrow, widen, move left or right, move the borders of the lanes, etc.) and add new lanes. Fig.19 In , annotators have tools for adjusting the camera height to match the reconstruction to the road surface and adjusting the crop height to exclude the dashboard or hood of a vehicle. All annotations are performed in the estimated 3D road plane, but with instant feedback provided via projections in the 2D camera view. Annotators can easily skip forward and backward in the sequence to determine if the labels are aligned with the image and correct them if necessary. Fig. 27 An example of a corrected sequence is shown in (see above).

[0208] In addition to the lanes on the road, a strip along the side of the road indicated by reference numeral NR1 is labeled as “non-road.” In addition, the entire upper portion of each image (top strip NR2) is labeled as non-road, with the size manually adjusted by the annotator. Fig.25 An example of the presented annotations is shown.

[0209] Above Figure 4A , wherein a presentation component 416 is provided which can present a separate training image (image n) marked with the expected road annotations determined for the image via a user interface (UI) 412. Thus, the user can check how closely the expected road structure corresponds to the actual road structure visible in the image and make corrections if necessary.

[0210] The output of UI 412 is shown connected to the input of the image annotation component via connector 418 to indicate that the user can modify the annotation data A(n) of an individual image via UI 412 to better align the assumed lane boundaries R1, R2 with the actual lane boundaries in (only) that image. This is useful to accommodate "one-off" differences caused by, for example, unusual local road structure.

[0211] A model adaptation component 410 is also provided to allow efficient manual adaptation of annotation data across multiple images simultaneously via the UI 412. This works by allowing the user to adapt the 3D road model with reference to a single image and the annotation data presented therein to better align the road boundaries in that image. However, because the 3D road model (rather than just the annotation data A(n) of an individual image) is being adapted, these adaptations can also be applied to the annotation data of other images without the user having to manually adapt the annotation data of those other images. For example, a user can adapt an assumed lane width W to better fit one image, and because the road model is adapted to the new lane width, the annotation data of other images can be automatically adapted based on the adapted model with the new lane width W.

[0212] Figure 4 and 4A The above components of the image processing system are functional components of the image processing system, representing functions performed by the image processing system. These functions are implemented in software, i.e., by executable instructions (code) stored in a memory and executed by one or more processors of the image processing system (not shown), which may be formed by one or more general processing units (e.g., CPUs), more specialized processing units (e.g., GPUs), or any combination thereof. As will be appreciated, these components represent high-level functions that may be implemented using a variety of code structures. Figure 4 The arrows in the diagram generally indicate high-level interrelationships between these high-level functions and do not necessarily correspond to any particular arrangement of physical or logical connections. Figure 5 The steps are computer-implemented steps that, when the instructions are executed on one or more processors, cause the image processing system to perform these steps. Details of the example implementation:

[0213] Fig.13 shows an example of annotated images, in Fig.13 In the figure, two drivable lanes are marked: the ego lane towards the center of the image, and another drivable lane on the left side of the image. Two non-drivable areas are marked on either side of these lanes, such as the lane for traffic traveling in the other direction on the far right, which is separated from the ego lane by one of the non-drivable areas.

[0214] As mentioned above, the method uses the following observations and assumptions: 1) When humans drive cars, they typically follow a path that stays close to the center of a lane on the road. So if a video camera is placed in a car driven by a human facing forward (or alternatively, backward), then the path traced by the camera, as it projects down onto the ground and extends to the width of the lane, will (mostly) outline where the lane is. 2) After finding the outline of where the lane is, then for each image in the video, this outline of the lane under the car can be projected back into the image and provide (almost) exactly the annotations needed to mark where the lane is.

[0215] The preferred embodiment of the present invention achieves this using low cost equipment, as explained later. Determine the path and reference parameters:

[0216] In order to generate the lane positions and project them back to each image, as described above, the following needs to be known or inferred: The path in 3D space followed by the camera 302 (CP, Figure 5 ); The orientation of the camera 302 within the car 300, which can be represented using the following parameters: ο forward points, which are defined as the pixels towards which the car appears to be traveling when the car is traveling in a straight line, ο horizontal line, which is defined as the line on the image that appears parallel to the road; The height of the camera above the road surface (H, Figure 3 , 3A and 5) · Left / right position of the camera in the car (in Figure 3A and 5 as interval S in ); and Lane width (W, Figure 3A and 5 ).

[0217] The image processing techniques described below allow all of the above to be determined using cheap and uncalibrated equipment, such as a mobile phone or dashcam costing a few hundred pounds, or other low cost consumer equipment. This is because they can be calculated from images captured by the training vehicle 300 using the image processing techniques described below. 3) The 3D path CP of the camera can be inferred by post-processing the captured video from the uncalibrated camera using existing visual SLAM (Simultaneous Localization and Mapping) techniques. Software packages such as OpenSFM and ORB-SLAM can be used to generate accurate camera path information.

[0218] Such techniques may be used to construct a 3D map of the environment in which the training vehicle 300 is traveling, while simultaneously tracking the position of the vehicle within that environment.

[0219] Some basic principles of image-based SLAM are described by reference Figure 7 The examples are briefly described. Figure 7The top of shows three images of a real-world environment captured at different times and from different but close locations - img(n-1), img(n), and img(n+1). By identifying distinctive reference features that appear in the images (e.g., those labeled F1 and F2), and matching them across images, triangulation can then be used to construct a 3D model of these features, as Figure 7 For each of the captured images img(n-1), img(n), and img(n+1), the 3D position of the camera 302 when the image was captured (respectively labeled x n-1 ,x n ,x n+1 ), as the orientation of the camera's field of view in 3D space. For each of the images img(n-1), img(n), img(n+1), the corresponding image plane I(n-1), I(n), I(n+1) is defined at a distance x from the camera position n-1 ,x n ,x n+1 The plane at any non-zero distance is perpendicular to and intersects the field of view of the camera when the image was captured. The camera position and orientation are selected based on ray tracing principles applied to reference features F1 and F2 so that for the camera position x n-1 ,x n ,x n+1 For each of the reference features F1, F2, a ray (straight line) from each of the reference features F1, F2 to the camera position intersects the corresponding image plane I(n-1), I(n), I(n+1) at a position that at least approximately corresponds to the position where the reference feature appears in the corresponding image (i.e., the pixel position of the feature within the image). Then, the camera position x can be obtained from n-1 ,x n ,x n+1 The camera path CP is reconstructed from the sequence.

[0220] Note that this 3D model built as part of the SLAM process is different from the 3D model built by the modeling component 404. The former is in the form of a surface map built based on image triangulation, while the latter represents the expected structure based on the vehicle path inference, which in turn is determined from the image capture device positions derived in performing the SLAM process.

[0221] refer to Fig. 7A and 7B , the 3D orientation of the camera 302 can be expressed using the pitch (P'), roll (R'), and yaw (Y') angles. L ' can be conveniently defined as being parallel to the optical axis of the camera, the horizontal axis and the vertical axis a' T , a V' corresponds to the horizontal and vertical orientations in the image plane (i.e., to the directions that appear horizontal and vertical in the captured image), and the pitch P', roll R', and yaw Y' of the camera are defined as the rotations around the horizontal, longitudinal, and vertical axes of the camera, respectively. T ′、a′ L 、a V ′ (camera axis). Vector notation is used here, a T ′、a′ L 、a V ' is a unit vector in the direction of the corresponding axis. According to this definition, the image plane I(n) is perpendicular to the horizontal axis of the camera and is horizontally / vertically aligned with the horizontal / vertical camera axis.

[0222] This SLAM technique actually does generate a surface reconstruction of the environment. While in theory this surface reconstruction could be used to determine the road / lane position directly, the quality of the surface reconstruction is usually not sufficient to do so in practice. However, the camera's motion through the 3D scene is of good quality and therefore provides a sufficiently accurate reconstruction of the camera path CP (this is because the camera position is based on hundreds of feature points, while the surface position is based on only a small number of feature points), which in turn allows the present technique to be used to infer the road / lane position with sufficient accuracy.

[0223] Note that some of these techniques can also automatically infer camera characteristics such as focal length and even lens distortion parameters. These parameters can be used to optimize the present technique, but they are not required.

[0224] Fig.14 An example image of a SLAM reconstruction of a camera position moving through a point cloud world is shown, with a sequence of pyramid structures labeled 1400 representing the position and orientation of the image capture device when taking different images, as estimated via the SLAM process. These collectively define the camera path CP. The blocks labeled 1402 constitute a 3D surface map generated as part of the SLAM process. These two aspects will be described in more detail later.

[0225] The remaining reference parameters are calculated by the parameter calculation component 408 via image processing, as described below. First, some basic considerations are briefly introduced.

[0226] refer to Figure 8 The 3D orientation of the training vehicle 300 can be defined in the same way as the camera. Here, the pitch, roll, and yaw of the vehicle are labeled P, R, and Y, respectively, and the lateral, longitudinal, and vertical axes (vehicle axes) of the training vehicle are represented by unit vectors a and b, respectively. T ,a L ,a VThe longitudinal axis corresponds to the direction of travel (forward direction) of the vehicle when traveling in a straight line. The vertical axis is the direction perpendicular to the road surface (i.e., perpendicular to the plane where the contact points between the wheels and the road are located). The horizontal axis extends along the width of the vehicle 300 and is perpendicular to the longitudinal axis and the vertical axis.

[0227] Note that, unless otherwise indicated, the term "absolute" as used herein refers to orientation and position in the environment in which the vehicle is traveling. That is, defined relative to an (arbitrary) global coordinate system of the global reference system of the environment in which the vehicle is moving. This can be defined as an inherent part of the SLAM process. This is the coordinate system that defines the reconstructed vehicle path, the orientation of the image capture device in the environment (absolute orientation), and the surface reconstruction.

[0228] The term "relative" is used to refer to a position and orientation relative to the vehicle, i.e., relative to a coordinate system defined by the vehicle, e.g., by the vehicle axes a and b. T ,a L ,a V Limited coordinate system.

[0229] refer to Fig. 9 , this method does not require that the orientation of camera 302 be perfectly aligned with the orientation of training vehicle 300. Angular misalignment of the camera axis relative to the vehicle axis is acceptable because such misalignment can be detected and accounted for using the techniques described below. 4) The angular orientation of the camera in the car, i.e. the longitudinal axis a of the camera L ' and the longitudinal axis a of the vehicle L (ie, the longitudinal axis a of the vehicle as measured in the reference frame of the image capture device (ie, in a reference frame in which the orientation of the image capture device remains fixed) L ), can be automatically derived by exploiting the fact that the motion vector of the car will be parallel to the body of the car (i.e., the longitudinal axis of the vehicle) when the car is traveling in a straight line, and thus the orientation of the camera relative to the body of the car can be calculated. This orientation is captured as the camera's forward point, which is defined as the pixel that the car appears to be facing when traveling in a straight line (as described above).

[0230] It is not possible to rely on there being a portion of the video where the car is driving in a perfectly straight manner. To overcome the fact that a car can never drive in a perfectly straight manner, the following approach is adopted.

[0231] Take three equidistant points on the path, connect the two outside points, and use that vector as an approximation of the forward direction at the center point. This method continuously weights the average by the dot product of the vector between the center point and the two outside points. Therefore, the time when the car is close to driving in a straight line is weighted much more than the time when the car is turning a corner. Using this method can produce sub-pixel accuracy. This is Fig.15 Shown in.

[0232] In other words, this utilizes the vector difference x when the vehicle 300 is traveling along a (nearly) straight line. t+1 -x t-1 (approximately) parallel to the longitudinal axis of the vehicle a L 300 to estimate the angular offset of camera 302 relative to vehicle 300. This is captured in the weighted average by assigning greater weights to intervals of more linear (straight) motion.

[0233] Note that strictly speaking, as long as the rate of turn is approximately constant (which does not require the vehicle to be traveling in a straight line), the vector difference x t+1 -x t-1 Approximately parallel to the longitudinal axis of the vehicle. However, as the vehicle approaches traveling in a straight line, the assumption of an approximately constant turn rate will be more likely to be correct.

[0234] The calculation is performed as a vector average, where the vector represents the average forward motion of the car as expressed in the camera coordinate system. When projected forward, this vector will cut the image plane at a specific pixel (the forward point pixel). This pixel is the point towards which the car would appear to move if it were driving on a perfectly straight and flat road (as the term "forward point" is used herein).

[0235] For example, the forward direction of the vehicle can be calculated as a weighted average of the vector differences between the longitudinal axis of the camera and the longitudinal axis of the vehicle, respectively, according to Equation 1: where, as evaluated at time t, w L,t =v0·v1.

[0236] Assume that at time t the longitudinal axis of the vehicle a L Parallel to vector difference x t+1 -x t-1 .

[0237] Note that in this case, x needs to be evaluated in the reference frame of the image capture device 302 t+1 -x t-1 , in order to give a meaningful result, i.e., so that the weighted average of equation (1) provides the longitudinal axis a of the vehicle L Relative to the longitudinal axis of the camera a′ L Accurate estimate of the direction.

[0238] The SLAM process will export x t+1 and x t-1The absolute value of , i.e. in the global coordinate system. However, the direction of the camera axis in the global coordinate system, i.e. the absolute orientation of the image capture device, at time t is also derived via the SLAM process itself, which in turn can be used to transform the vector in the global reference system to the reference system of the image capture device 302.

[0239] The dot product v0·v1 depends on the magnitude of the change in direction between t-1 and t+1, such that the smaller the change in direction magnitude, the greater the weight given to the measurement at time t.

[0240] As mentioned above, strictly speaking, as long as the turning rate is approximately constant (which does not actually require the vehicle to be moving in a straight line), the vector difference x t+1 -x t-1 Approximately parallel to the longitudinal axis of the vehicle. Therefore, an alternative weighting scheme may be used, in which the vector difference at each time is weighted according to the change in curvature present in the vehicle path between t-1 and t+1.

[0241] Of course, typically the orientation of the vehicle axes remains fixed in the image capture device's reference frame as the vehicle moves, since camera 302 is expected to remain in a substantially fixed orientation relative to vehicle 300, the above time dependency stems from the fact that estimates of the vehicle and camera axes are imprecise.

[0242] The time t corresponds to a specific point on the camera path CP and may be the time of one of the captured images or the time between captured images to which the camera pose may be assigned by interpolation of the camera poses of different images.

[0243] In an alternative embodiment, a L and a L The offset between ′ can be expressed as a pair of angles.

[0244] refer to Fig.10 Once the angular orientation of the camera 302 relative to the vehicle 300 is calculated in this manner, the camera 302 can be oriented based on the longitudinal axis a′ of the camera. L The orientation at this point determines the longitudinal axis a of the vehicle. L The orientation at any point along the camera path CP (including when the vehicle is not traveling in a straight line) is known from the SLAM calculations. 5) By noting that most roads are locally flat, the rotational orientation of the camera inside the car can be automatically derived, i.e., the vertical axis a of the camera V ′ and the vertical axis a of the vehicle V(i.e. the vertical axis of the vehicle measured within the reference frame of the image capture device), so that when the vehicle turns a corner, the vector perpendicular to the surface described by the camera path will point directly down from the camera towards the road. In more detail, three equidistant points on the path are taken (although again, these points do not have to be equidistant), the cross product of the two vectors from the center to the two outer points is taken, and this is continuously weighted averaged by the magnitude of the cross product. Thus, the time when the car is driving in a curve is weighted more strongly than the time when the car is driving in a straight line (at which point there is no information about the normal).

[0245] This is Fig.16 Shown in.

[0246] In other words, this exploits the observation that when the vehicle exhibits angular acceleration such that the vehicle path exhibits local curvature in 3D space, the plane in which the curved portion of the vehicle path lies is at least approximately parallel to the plane of the road surface under normal driving conditions; or equivalently, the normal to the local curved portion of the path is at least approximately parallel to the vertical axis a of the vehicle 300. V .

[0247] The average normal vector of the path is represented in camera coordinates. A plane perpendicular to this road surface normal vector can be intersected with the image plane, and this will provide a line across the image that will match the horizon (as the term is used in this article) if the car is driving along a perfectly straight and level road.

[0248] For example, the rotational orientation of a camera within a vehicle can be calculated as a weighted average of the offset angles between the camera and the vertical axis of the vehicle, respectively, according to equation (2): in: As evaluated at time t, w V,t =|v0×v1|, v0 = x t -x t-1 , v1 = x t+1 -x t .

[0249] Assume that the vehicle vertical axis a at time t V Parallel to the vector cross product v0×v1.

[0250] As mentioned above, in this case, v0×v1 needs to be evaluated in the reference frame of the image capture device 302 in order to give a meaningful result, i.e., so that the weighted average of the equation provides the vertical axis a of the vehicle V Relative to the longitudinal axis of the camera a′ LAs described above, the absolute orientation of the image capture device determined via the SLAM process can be used to transform the vector from the global reference frame to the reference frame of the image capture device.

[0251] As the component vectors move toward the perpendicular direction, the magnitude of the cross product |v0×v1| increases, producing the most accurate results as expected. More generally, they are weighted by a factor that increases as the offset angle α between these vectors approaches ninety degrees, where |v0×v1|=|v0||v1|sinα.

[0252] As will be appreciated, many of the observations made above with respect to angular orientation measurements also apply to rotational orientation measurements. 6) The height of the camera above the road (H) can be calculated by generating a surface mesh of the road from the SLAM process (even if this is low precision), and then averaging the height of all points on the camera path above this mesh. Averaging over the entire path offsets the low precision of the mesh produced by the SLAM process.

[0253] The height H at a given point on the camera path CP is defined as the height H between the point on the camera path CP and the vertical axis a of the vehicle at that point. V The reason why the rotational orientation of the camera 302 within the vehicle 300 is needed is to determine where the vertical axis of the vehicle is located at each point on the path so that a height estimate can be calculated. This is done in Fig.11 , where the reconstructed surface map is labeled SM. 7) By using various computer vision techniques to detect lane boundaries, the left-to-right position of the camera within the lane (S) and the width of the lane (W) can be estimated.

[0254] Detecting lane edges is obviously a hard problem (and the whole purpose of generating this lane training data is to train a neural network to solve it). However, detecting lane edges in this particular case can become a significantly easier problem due to having two pieces of additional information. First, the known path of the camera (and therefore the shape of the lane) can be used to create a transformation that effectively straightens the road before any lane boundary detection is done. Second, searching for the location of the lane boundary is significantly reduced because the centerline of the lane is (almost) known from the camera path, and the lane has a relatively fixed width.

[0255] For example, the image can be projected into a top-down view, transformed to straighten the road (using the camera path information), and then a Hough transform can be used to detect the main lines with the correct approximate position and orientation.

[0256] The road width at each point on the camera path CP is defined as the width of the road along the transverse vehicle axis a at that point.T The reason why it is necessary to determine the angular orientation of the camera 302 in the training vehicle 300 in addition to the rotational orientation is to determine the lateral axis a of the vehicle. T where is located in order to estimate W and C at each point along the path. This is also Fig.11 Shown in.

[0257] By utilizing all of the above, annotations of the lanes traveled by the car can be fully automatically generated in all images of a video using only images captured from a low-cost and uncalibrated image capture device.

[0258] It is also desirable to extend these techniques to annotate multiple lanes. One way to do this is as follows: 8) Multiple lanes (and therefore the entire road surface) can be automatically annotated by driving along each lane and creating a single 3D SLAM reconstruction that includes the path of each lane. It is necessary to merge the separate videos driving along each line into a single reconstruction in order to obtain the relative positioning of the lanes and the entire road surface.

[0259] Fig.17 An image showing a SLAM reconstruction of a camera position moving through a point cloud world, where there are two sets of camera images. 9) It is difficult to merge images from multiple videos into a single 3D reconstruction unless the camera images are all facing the same direction; i.e., it is not practical to merge videos of cars driving in opposite directions along a road into a single 3D reconstruction. One way to solve this problem is to drive in one direction with a forward-facing camera, and drive in the other direction (in the other lane) with a rear-facing camera. This gives a single set of images that are all pointing in the same direction along the road. Extension-Human Correction Phase:

[0260] The described fully automatic annotation system produces usable annotations, however, it still has many errors that can be corrected by human annotators with very little effort. For example: The calculated camera height may not be exactly correct The left / right position within the lane may not be exactly correct The lane width may not be exactly right

[0261] In addition to these imperfectly calculated corrections, there are additional annotations that people can add that require very little effort to add but provide significant benefits if they are present in the training data. For example: Adjacent lanes can be added (without having to drive in them). They are easy to add because adjacent lanes usually have borders parallel to the lane the car is already driving in. Non-drivable parallel road features can be added. Areas of the road reserved for bicycles or buses / taxis can simply be added (again, they are usually just parallel areas of the ground along the driving lane). · The pedestrian area adjacent to the road can be increased. 10) A tool with a simple user interface has been developed that allows a human annotator to make these "corrections" and also add these additional annotations. The main benefit of this tool is that corrections and additional annotations can be made in one frame of the video, and the system can automatically propagate these changes to all frames in the video.

[0262] This effective saving comes from the fact that the user can use one frame and its annotation data as a reference to adapt the 3D road model (via the model adaptation component 410), and the adapted 3D road model can then be applied to multiple images. It should also be noted that the present technique is not limited to manual adjustments of automatically generated road / lane models. Any stationary object in the world can be marked, and the position of that object in each image of the video can be calculated (since the motion of the camera is known). This way, humans can be provided with one frame of a video that has been pre-populated with almost correct annotations, and with minimal corrections, they can generate about 50-100 well-annotated images. If longer video clips are used, more images can be generated, but in our experience, videos that are convenient for 3D reconstruction and are also fairly similar in their construction are long enough to contain 50-100 images. 11) Note that the images used as training data are sampled from the entire video so that they are spaced reasonably far apart in space (not time) in order to provide some diversity in the training data for the network. There is little benefit in taking every image from the video since most of them are very similar to each other.

[0263] Figure 6 A schematic block diagram illustrating an example of the types of adjustments that may be performed during the human correction phase is shown. The presentation component may present data of any of the captured images on a display of the user interface 412, overlaid with its associated annotation data A(n). One or more selectable options 600 are also presented that the user may select in order to cause the model adaptation component 410 to adapt the 3D road model 602 in a quick and intuitive manner. Figure 6As shown, when the road model 602 is updated, not only the currently presented annotation data A(n) is modified according to the adapted model 602, but the adapted model 602 can also be used to update the annotation data 604 for other images in the same video sequence. Figure 6 The top portion of the 3D model shows the marked road or lane within the image. While the right-hand boundary R2 coincides exactly with the real-world right boundary visible in the image, the left boundary R1 is slightly off. The user can quickly correct this by moving the position of the left boundary in the 3D model, using the current image and the annotation data presented on the current image as a reference, to manually adapt the position of the left boundary R1 until it corresponds to the real-world left boundary visible in the image. This change will be applied to multiple images in the sequence, meaning that as long as the road / lane remains the same width, the left road boundary will be accurately placed in those images.

[0264] The fact that large portions of the road structure of interest are parallel to the vehicle path is exploited to provide a range of extremely fast manual annotation options: - The width or position of the automatically determined road structure can be easily changed ("nudged"); - Additional road structures can be easily created parallel to the vehicle path / automatically determined road, and their width and position can be easily nudged in the same way. Additional road structures are automatically oriented parallel to the vehicle path / existing road structures. For example, an automatically determined road structure could be an "ego lane", i.e. the lane in which the vehicle is travelling. Additional structures can be added to multiple images very easily, additional structures such as: ο other lanes (e.g. non-self lanes, bus lanes, cycle lanes), centre or left / right road markings, or any other parallel road markings, non-drivable areas (such as pavements, grass, obstacles, roadside growth (trees, hedges, etc.)); - Perpendicular road structures, i.e. road structures that extend perpendicular to the automatically determined road structure / vehicle path, can also be easily added, and their width and position can be nudged in the same way. Examples include things like intersections, give way lines or other perpendicular road markings, or other structures that are at an assumed angle to the path (e.g. branching points in the road). - Changes in road width, e.g. due to a change in the number of lanes, can be accommodated by allowing the user to "split" the video sequence at a point of his / her choice. For the video after the split point, the user's adaptation will only be applied to the 3D road model at the point after the split point - the part of the model before the split point will remain unchanged, which in turn means that the annotation data of the images before the split point remain unchanged.

[0265] As an example, consider the following workflow. The user starts at the beginning of the training sequence. The 3D ego lane is automatically determined using the techniques described above. In addition, the positions of the center and left / right lane markers have been determined by assuming that they have a certain width and are located at the center and rightmost / leftmost of the ego lane. At this point the user can adapt the ego lane width and position, and if necessary, the width and position of the road markings to align them with the actual ego lane / road markings visible in the image. The user can also add additional parallel structures at this point, such as non-ego lanes, no-drive areas, etc.

[0266] Fig.18 An example of an automatically pre-corrected annotated image is shown in , where only the ego lane (and its assumed centerline and lane boundaries) is marked. As can be seen, the markings are not exactly aligned with the actual lane boundaries. Fig.19 The revised image is shown, in which the width and position of the ego lane have been adjusted, and other lanes and non-drivable areas have been marked.

[0267] These changes are applied to the base 3D model and will therefore also be applied to subsequent images in the sequence. The user can then cycle through subsequent images and quickly verify that all images still match.

[0268] When the user reaches a point in the video where the intersection is clearly visible in one of the images, he / she can add the intersection structure at that point and nudge its position / width as needed. Again, this is applied to the 3D model and will therefore apply to all images in which the intersection is visible. Non-drivable road structures at the edge of the road will automatically adapt to accommodate the parallel intersection (i.e. any part of the non-drivable road structure that overlaps the new intersection will be automatically removed).

[0269] When an intersection is first added, Fig. 22 An example of a UI is shown in , where the user has defined the intersection as a narrow passage perpendicular to the lanes and crossing the left and right lanes.

[0270] refer to Fig.12 , when the user reaches a point in the video where the road / lane width changes (e.g., lane narrows / widens, the number of lanes changes or decreases, etc.), she / he can segment the video at that point and use that frame as a reference to modify the underlying model (step 1, Fig.12). The position of the current video frame is mapped to a point on the vehicle's path, which in turn is used to restrict the fitting of the 3D model to only the portion of the model after that point - points before that point (where the user has verified that the annotations match the image) are not affected. At this point, the user can change the road width, add lanes, etc. The system is able to automatically interpolate parallel structures around the segmentation point, such as linear interpolation. For example, when adding a lane, typically the road will gradually widen until it can accommodate the new lane, and the user can advance to the point in the video where the new lane reaches its full width, and add the lane at that point (step 2, Fig.12 ). The system automatically interpolates between intermediate regions, assuming that the width of the new lane increases linearly until it reaches its final width (for example) - Fig.12 Step 3.

[0271] Fig.23 and 24 An example of this is shown in Fig.23 In the Fig. 22 The option to split the video shortly after the intersection to accommodate the fact that there is now a non-drivable area of ​​the wider road on the right-hand side, the user has Fig.24 The non-drivable areas of the wider road are marked accordingly.

[0272] As an extension, the user can also label sky areas in the images. This can be a simple case where the user defines a skyline above which all images are explicitly sky. Even though this will not label all of the sky in the images, this can still be useful in preventing a trained neural network from identifying sky as road structures. More sophisticated techniques can use the 3D road structures to infer the skyline in each image, for example based on the highest point of the road in 3D space. Extension of training data beyond lanes:

[0273] Any features in the video that are parallel to the path being followed can benefit from the same speedup in annotation and single-click addition by human annotators. 12) For example, road markings can be annotated in this way (both in position and type), including center dashed or solid lines, road edge lines, double yellow lines, etc. For example, bicycle lanes can be added in this way, and the same goes for sidewalks.

[0274] Fig. 20 An example of an image is shown, where a bicycle lane is marked on the left hand side.

[0275] Any features in a video that is stationary in the world can benefit from some level of speedup. Even if they cannot be created directly because their shape is parallel to the driving path, they can still benefit from the fact that they can be labeled by a human in one frame of the video and automatically propagated to all frames of the video. 13) For example, road signs can be annotated in this way. Signs can be specifically marked by a human in one frame (with or without any initial cues provided by low-quality detection) and then automatically propagated to the entire set of images in the video.

[0276] Annotations for stationary vehicles can be generated in this same manner. Although the technique does not automatically allow annotation of moving objects to be accelerated in the same manner, the technique can be used for parked vehicles. Thus, it allows a subset of vehicles in the scene to be annotated with minimal effort. Moving vehicles can be annotated by marking the same object in multiple (at least two) images and assuming a constant speed between annotations. The labels can then be propagated to all images between the annotated images. 14) Note that stationary vehicles look the same as moving vehicles, so a neural network trained on annotated images of stationary vehicles can be used at runtime to detect moving vehicles.

[0277] Fig.21 An image is shown with the 3D bounding box of the vehicle added.

[0278] Another benefit of this technique is that when the camera is close to an object in the video (i.e., when the object is large), a human can choose to annotate the object in that image. This accurate annotation is then propagated to all images, even when the object is small within the image. 15) Since there is a 3D model underlying the entire sequence of images in the video, the annotations automatically generated when the object is farther from the camera will have the same annotation accuracy as when the object is large and annotated by a human. This is usually more accurate than what a person trying to annotate a small image of the object can achieve. Example use case:

[0279] Trained road detection component 102 has a number of useful applications within autonomous vehicle 200 .

[0280] The ability of the trained road detection component 102 to identify road structures within captured images based on machine vision can be used in conjunction with predetermined road map data. Predetermined road map data refers to data or road maps currently used in GPS-based navigation units (such as smartphones or "satellite navigation") or pre-created maps.

[0281] One such application is localization, where road structures identified by trained road detection component 102 can be used to more accurately find the vehicle's position on a road map. This works by matching road structures identified via machine vision with corresponding road structures of a predetermined map. For example, a pair of stereoscopically arranged image capture devices can be used to determine the position of autonomous vehicle 200 relative to the identified road structures in three dimensions, which in turn can be used to determine the position of the autonomous vehicle on a road map relative to the corresponding road structures on the map.

[0282] Another such application merges visually identified road structures with corresponding road structures of a road map. For example, a road map can be used to resolve uncertainty about visual road structures detected in an image (e.g., distant or slightly occluded visual structures). By merging a road map with uncertain visual structures, the confidence of structure detection can be increased. Experiments and statistics

[0283] Table 1 provides a comparison of the disclosed method with the prior art benchmarks identified in the reference section below. Table 1. Comparison of available datasets. If provided by the authors, only the labeling time per image is shown. a Only a single image is annotated, but additional (unannotated) image sequences are provided b Road areas are implicitly labeled by given lanes c Mark the ground instead of the road, i.e. include non-drivable areas d Limited to three instances: ego lane and left / right of ego lane 4 Dataset Statistics and Segmentation

[0284] The method has been tested on a dataset (the “current dataset”) consisting of 402 sequences with a total of 23,979 images, thus an average of 60 images per sequence. Table 2(a) shows the classification of the included annotation types. In total, 47,497 lane instances were annotated, i.e., 11 8.2 per sequence. The instance IDs are consistent across sequences, i.e., consecutive frames will use the same instance ID for the same lane. In addition, the annotators were instructed to classify each sequence according to the scene type (urban, highway, or rural). The classification of the sequences is shown in Table 2(b). Table 2. Classification of datasets according to scene type (a) and annotation coverage (b). The coverage of scene types and instances is measured as a percentage of the total number of sequences, while the coverage of annotations is measured as a percentage of the total number of pixels.

[0285] The data is split into two sets, for training and testing. The training set includes 360 sequences and a total of 21,355 frames, while the test set includes 42 sequences and 2,624 frames. The test set is chosen to include the same urban / highway / rural distribution as the training set. The frames of the training set are available along with the images and annotations, while only images are provided for the test set.

[0286] Furthermore, the average annotation time for each scene type has been measured and it has been found that there is a large variation, with urban scenes taking approximately three times longer than similar lengths of highway or rural scenes (see Table 3). This is due to the complexity of the variation in road layouts, which is caused by a variety of factors: the frequency of intersections and side roads, the overall complexity of the lane structure, and additional features such as traffic islands and bike lanes that are not typically found outside of urban environments.

[0287] The quality of annotations was measured by the agreement between two annotators for twelve randomly selected sequences. 84.3% of the pixels had been labeled by at least one annotator, of which 67.3% were labeled by two annotators; i.e., 56.8% of all pixels were labeled by two annotators. The agreement of these overlapping labels was measured by the intersection over union (IoU). The results are shown in Table 4. The standard deviation was calculated for the 12 sequences. Table 3. Measured average annotation times in seconds Table 4. Annotator agreement measured by IoU Experiment 5

[0288] To demonstrate the results achievable using our annotation method, we provide evaluation procedures, models, and results for two example tasks: semantic segmentation and instance segmentation. 5.1 Road and Ego Lane Segmentation

[0289] The labels and data described in 3.2 directly allow for two segmentation tasks: road / off-road detection and self / non-self / off-road lane detection. The well-studied SegNet

[29] model is used as a baseline, trained for both self and road experiments separately.

[0290] When training on the training data of the current dataset, all available training data is randomly split into training and validation sets in order, and 10% of the available training batches are randomly selected as validation sets. During training, each input image is reprocessed by changing its size to have a height of 330px and extracting a random crop of 320×320px; the crop is randomly selected each time an image involved in a mini-batch is selected. To train the SegNet model, ADAM with a learning rate of 0.001 is used, which decays to 0.0005 after 25000 steps and then decays to 0.0001 after 50000 steps. Training is performed for 100,000 training steps, and the results on the model that converges to the validation set are reported. A mini-batch of size 2 is used, and the optimization is performed on a per-pixel cross entropy loss.

[0291] For evaluation, multiple SegNet models were trained with the above configuration. Models were trained on the training set, CityScapes coarse, Mapillary, and KITTI lanes. Road / off-road models were trained for all datasets, but ego / non-ego / off-road were trained for the UM portion of KITTI lanes and the current dataset.

[0292] For each model, IoU and FI scores are provided. Each model is measured on held-out data from each dataset. For CityScapes and Mapillary, this held-out set is the predefined validation set of the two datasets, and for the current dataset, the held-out set is the corresponding test set. The exception to this scheme is the KITTI lanes for which no annotated held-out set is available, and further reducing the KITTF size by holding out a validation set would reduce the size of the already small dataset, making the results more difficult to interpret. Therefore, the entire KITTI training set is targeted by the models Mapillary and CityScapes. IoU and FI are measured for each class of each task, and the average values ​​for the classes of each task are provided. These results are visible in Tables 5 and 6.

[0293] Before analyzing these results, it should be noted that due to the current processing of lanes and roads under cars on the road, these ratings are not directly comparable to the expected ratings of CityScatpes, Mapillary, or KITTI lanes. Also, in order to obtain the results reported below on models trained on this dataset alone, the hood and trim had to be cropped out from the CityScates validation set. All numbers reported for the CityScapes validation set have this processing applied. Without this processing, Mapillary's IoU to CityScates is reduced by 2% with the current processing, while the IoU is reduced to 56%.

[0294] Nevertheless, clear trends can be observed between datasets. First, it should be noted that the highest IoU achieved in the experimental setting comes from models trained on the same data that they were evaluated against. This points to a general generalization problem in vision datasets, with no dataset performing as well on out-of-dataset data as on in-dataset data.

[0295] The remaining analysis concerns the results achieved by models trained on data other than the target evaluation datasets (i.e., the diagonal lines in the results table). The smallest population dataset, KITTI Lanes2, achieves the worst IoU across all tasks. Cityscapes3 is the next largest dataset and achieves the next highest IoU across most tasks, and finally, Mapillary, the dataset most similar in size to the current dataset, achieves the next highest IoU while still performing significantly worse than the current methods themselves across all tasks. 5.2 Lane Instance Segmentation

[0296] The annotation of multiple distinct lanes per image naturally suggests an instance segmentation task for the current dataset, where the number of distinct lanes varies between images and potential sequences. Although the "fillers" have been assumed to be uncountable and therefore have no instances, this lane instance segmentation task operating system is presented as a counterexample. In fact, many categories that look similar to fillers (parking spaces, lanes in swimming pools, fields in satellite imagery) can have meaningful depictions and therefore apply instances.

[0297] Providing a useful baseline for the lane instance segmentation task presents its own challenges. For example, the current state of the art for instance segmentation on Cityscapes is MaskRCNN

[32] . This approach is based on the RCNN object detector and is therefore optimized for the detection of compact objects that fit inside extensively non-overlapping bounding boxes, traditionally referred to as “things”. In the case of lanes detected in perspective, the bounding box of any given lane overlaps greatly with neighboring lanes, making the task potentially challenging for standard bounding box methods. This becomes even more apparent when the road undergoes even slight curvature, in which case the bounding box is almost on top of another even if the instance pixels are completely disjoint. Recently, several works have explored an alternative approach based on the RCNN algorithm that uses pixel embeddings to perform instance segmentation [33–36]; the use of pixel embeddings provides a baseline for the current dataset.

[0298] Specifically, we train a model based on

[33] , using their approach to learn per-pixel embeddings, where the value of each pixel embedding is optimized so that pixels within the same training instance are given similar embeddings, while the average embeddings of individual instances are pushed apart simultaneously. The cost function for learning such pixel embeddings can be written exactly and is given in Equations 1-4 of

[33] . For comparison, the same hyperparameters reported in that work are used. This loss is imposed as an additional output along with the road / non-road SegNet model trained from the sporadic segmentation task.

[0299] At run time, an embedding is predicted for each pixel following a variation of the method proposed in

[33] . The prediction of the road is used to filter out pixels that are unlikely to be lanes. Pixels are then uniformly sampled in the road region and the embeddings of the pixels are clustered using the mean shift

[37] algorithm to identify the centers of the detected lane instances. Finally, all pixels in the road region are assigned to a single lane instance using Euclidean distance.

[0300] To evaluate lane instances, the AP measure is computed for each MS-COCO instance segmentation task. Specifically: the average precision is computed across images and across IoU thresholds for detected lanes (pixels assigned to embedding cluster centroids) and ground truth lanes. A detection is a true positive when it overlaps a ground truth instance with an IoU above a certain threshold. The ground truth is a false negative when it is not assigned a lane in this way, and a detection is a false positive when it does not sufficiently overlap any ground truth instance. Using these definitions, the average precision at 50% IoU and the average AP for multiple thresholds from 50% to 95% in 5% increments are obtained, as stated in Table 7. This method does not provide a score for any given lane instance detection, but rather sorts the lane detections by their size, choosing to assign larger lane instances to the ground truth before smaller ones. Table 5. Results on road / off-road tasks measured by IoU and F1 score. The table describes the training / testing settings for each number. Table 6. Results on the self / non-self / off-road tasks measured by IoU and F1 score. The table describes the training / testing settings for each number. Table 7. Lane instance segmentation results

[0301] Initially proposed experiments showed promising generalization results across the entire dataset.

[0302] Extensions of the disclosed technology include the following:

[0303] (1) It does not include the annotation of many other object types in the static road layout, such as buildings, traffic signs, and traffic lights.

[0304] (2) All annotated lanes are parallel to the future driving path, so lane splits and perpendicular lanes (e.g., at intersections) are currently excluded.

[0305] (3) The locations of dynamic objects (e.g., vehicles, pedestrians, and cyclists) can be included. In future work, these limitations can be addressed by adding further annotations of different objects in 3D.

[0306] The annotator tool can be extended to handle non-parallel lanes to allow for variable angles of lanes in the road plane. Additionally, the position of dynamic objects can be estimated by including additional modalities such as stereo vision or radar. References 5. Brostow, G.J., Fauqueur, J., Cipolla, R.: Semantic object classes in video: A high-definition ground truth database. Pattern Recognition Letters 30(2) (2009) 88-97) 6. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Pranke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. CVPR (2016) 7. Neuhold, G., Ollmann, T., Bulo, SR, Kontschieder, P.: The mapillary vistas dataset for semantic understanding of street scenes. In: Proceedings of the Inter Proceedings of the International Conference on Computer Vision (ICCV), Venice, Italy. (2017) 22-29 14. Brostow, GJ, Shotton, J., Fauqueur, J., Cipolla, R.: Segmentation and recognition using structure from motion point clouds. In: European conference on computer vision, Springer (2008) 44-57 15. Sengupta, S., Sturgess, P., Torr, PHS, et al.: Automatic dense visual semantic mapping from street-level imagery. In: Intelligent Robots and Systems (IROS), 2012 IEEE / RSJ International Conference on, IEEE (2012) 857-862. 16. Scharwachter, T., Enzweiler, M., Franke, U., Roth, S.: Efficient multi-cue scene segmentation. In: German Conference on Pattern Recognition, Springer (2013) 17. Fritsch, J., Kuehnl, T., Geiger, A.: A new performance measure and evaluation benchmark for road detection algorithms. In: 16th International IEEE Conference on Intelligent Transportation Systems (ITSC2013), IEEE (2013) 1693-1700 18. Aly, M.: Real time detection of lane markers in urban streets. In: IEEE Intelligent 675 676 Vehicles Symposium, Proceedings IEEE (2008) 7-12 19. TuSimple: Lane Detection Challenge (Dataset) (Lane Detection Challenge (Dataset)). http: / / benchmark.tusimple.ai (2017) 27. Barnes, D., Maddern, W., Posner, I.: Find Your Own Way: Weakly-Supervised Segmentation of Path Proposals for Urban Autonomy. ICRA (2017) 28. Mapillary: OpenSfM (software). https: / / github.com / mapillary / OpenSfM(2014) 29. Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: A deep convolutional encoder-decoder architecture for image segmentation. arXiv preprint: arXiv:1511.00561 (2015) 32. He, K., Gkioxari, G., Dollar, P., Girshick, RB: Mask R-CNN. CoRR abs / 1703.06870(2017) 33. Brabandere, BD, Neven, D., Gool, LV: Semantic instance segmentation with a aki discriminative loss function. CoRR abs / 1708.02551 (2017) 34. Li, S., Seybold, B., Vorobyov, A., Fathi, A., Huang, Q., Kuo, CCJ: Instance embedding transfer to unsupervised video object segmentation (2018) 35.Fathi,A.,Wojna,Z.,Rathod,V.,Wang,P.,Song,HO,Guadarrama,S.,Murphy,KP: Semantic instance segmentation via deep metric learning. CoRR713 abs / 1703.10277(2017) 36. Kong, S., Fowlkes, O: Recurrent pixel embedding for instance grouping (2017) 37. Comaniciu, D., Meer, P.: Mean shift: A robust approach toward featurespace analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24. (5) (May 2002) 603-619. Appendix A

Claims

1. An image annotation system, comprising: an input configured to receive a sequence of two-dimensional images captured by an image capture device of the vehicle; a modeling component configured to generate a three-dimensional structural model of an environment in which the vehicle is traveling while capturing the image; a path determination component configured to determine, for each of the images captured, a corresponding position and orientation of the image capture device within the environment when the image was captured; an annotation component configured to generate two-dimensional image annotation data for marking a region of the image corresponding to the three-dimensional structural model by geometrically projecting the three-dimensional structural model onto an image plane of each image defined by a corresponding image capture device position and orientation; a presentation component configured to present at least one of the images and two-dimensional image annotation data of at least one of the images via a user interface; as well as A model adaptation component, wherein the model adaptation component is configured to adapt the three-dimensional structure model according to instructions received via the user interface, wherein the annotation component is configured to modify the two-dimensional image annotation data for each of the captured images based on the adapted three-dimensional structure model, and the presentation component is configured to present the modified annotation data for the at least one image via the user interface, thereby allowing a user to use the at least one image and the two-dimensional image annotation data of the at least one image as a reference to simultaneously implement the modification of the two-dimensional image annotation data for multiple captured images.

2. The image annotation system according to claim 1, wherein: The path determination component is configured to determine the image capture position and orientation by processing the sequence of captured images.

3. The image annotation system according to claim 1 or 2, wherein: The modeling component is configured to determine an expected structure of the three-dimensional structural model based on a path through the environment defined by the determined image capture device position.

4. The image annotation system according to claim 3, wherein: The expected structure extends parallel to the path defined by the image capture device position, or the expected road structure extends perpendicular to the path defined by the image capture position.

5. An image annotation system according to any one of the preceding claims, wherein: The modeling component is configured to assign a label to at least one structure of the three-dimensional structural model, and the annotation component is configured to associate the label with a corresponding region marked in the image.

6. An image annotation system according to any one of the preceding claims, in, The three-dimensional structural model has at least one structure created by a user via the user interface.

7. An image annotation system according to any one of the preceding claims, wherein: The three-dimensional structure model has at least one structure corresponding to a stationary object.

8. An image annotation system according to any one of the preceding claims, wherein: The modeling component is configured to generate the three-dimensional structural model using one or more of the following reference parameters: a position of the image capture device relative to the vehicle, an orientation of the image capture device relative to the vehicle, a distance between two real-world elements measured along an axis of the vehicle. 9 . The image annotation system according to claim 8 , comprising a parameter calculation component configured to estimate the one or more reference parameters by processing image data captured by the image capture device of the vehicle.

10. An image annotation system according to any one of the preceding claims, wherein: The model adaptation component is configured to create a new structure of the three-dimensional structure model in response to a parallel structure creation instruction received via the user interface, the new structure being oriented to be parallel to at least one of: a path defined by the position of the image capture device, and an existing structure of the three-dimensional structure model, wherein the orientation of the new structure is automatically determined by the model adaptation component based on the path or the orientation of the existing structure.

11. An image annotation system according to any one of the preceding claims, wherein: The model adaptation component is configured to adapt an existing parallel structure of the three-dimensional structure model in response to a parallel structure adaptation instruction received via the user interface.

12. The image annotation system according to claim 10, wherein: The model adaptation component is configured to change the width of the existing parallel structure, or change the position of the existing parallel structure.

13. The image annotation system according to claim 11, wherein: The pattern adaption component is configured to change the width of the second portion of the existing structure while keeping the width of the first portion of the parallel structure fixed.

14. An image annotation system according to any one of the preceding claims, wherein: The three-dimensional structure model includes an expected road structure determined by the following steps: receiving a time sequence of two-dimensional images captured by an image capture device of a moving vehicle; processing the images to reconstruct in three-dimensional space a path traveled by the vehicle; using the reconstructed vehicle path to determine an expected road structure extending along the reconstructed vehicle path; as well as Road annotation data is generated by performing a geometric projection of the expected road structure in three-dimensional space onto a two-dimensional plane of the image for marking at least one of the images with the expected road structure position.

15. A computer program product comprising code stored on a computer readable storage medium, the code being configured to implement the system according to any one of the preceding claims when executed on one or more processors.

Images