Structural Annotations
By determining the three-dimensional road model in the image processing system and automatically generating annotation data, the problem of insufficient image annotation speed and quality in the training of the structure detection component of autonomous driving vehicle is solved, and the performance and training efficiency of the detection component are improved.
Patent Information
- Application Number
- CN201980057614.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-07-19
- Filing Date
- 2019-09-26
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2039-09-26
AI Technical Summary
In the prior art, the annotated road image generation speed and quality required for training structural detection components of autonomous driving vehicles are insufficient, resulting in high training costs and limited performance of detection components.
By determining the three-dimensional road model in the image processing system and automatically generating annotation data using known object locations, road images are quickly annotated, manual intervention is reduced, and high-quality large number of training images are generated.
It realizes the rapid and accurate generation of large number of annotated road image training sets, improving the performance and accuracy of structural detection components and reducing training costs.
Smart Images

Figure CN112714913B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to structural annotation, such as creating annotated road images, which can be used, among other things, to train a machine learning object detection component. Background Art
[0002] A rapidly developing technology is autonomous vehicles (AVs) that navigate cities and other roads on their own. Such vehicles must not only perform complex maneuvers between humans and other vehicles, but must do so frequently while maintaining tight constraints on the probability of adverse events occurring (such as collisions with other agents in the environment). In order to allow the AV to plan safely, it is critical that it can accurately and reliably observe its environment. This includes the need to accurately and reliably detect road structures in the vicinity of the vehicle.
[0003] Therefore, in the field of autonomous driving, a common requirement is to have a structure detection component (also known as a machine vision component) that, given visual input, can determine the structure of the real world, such as road or lane structure, e.g., which part of the image is the road surface, which part of the image constitutes the lanes of the road, and so on.
[0004] This is typically accomplished through machine learning using convolutional neural networks. Such networks require large numbers of training images. These training images are like the images seen from the camera of an autonomous vehicle, but they have been annotated with the information needed to learn the neural network. For example, these images will have annotations that mark which pixels on the image are road surface and / or which pixels on the image belong to lanes. At training time, the network is presented with thousands or preferably hundreds of thousands of such annotated images and learns on its own which features of the image indicate whether a pixel is part of the road surface or lane. Then, at runtime, the network can make this determination on its own using images it has never seen before (i.e., images it has not seen during the training process).
[0005] The conventional technique for creating annotated training images is to manually annotate the images manually. This can take tens of minutes per image (hence the large number of images required). Creating hundreds of thousands of training images using this manual process requires a lot of labor costs, which in turn makes it a high-cost training. In practice, it limits the number of training images that can actually be provided, which in turn can be detrimental to the performance of the trained structure detection component.
[0006] An autonomous vehicle (also known as a self-driving vehicle) is a vehicle that has a sensor system for monitoring its external environment and a control system that is able to use these sensors to automatically make and implement driving decisions. This includes in particular the ability to automatically adapt the vehicle's speed and direction of travel based on input from the sensor system. Fully autonomous or "driverless" vehicles have sufficient decision-making capabilities to operate without any input from the driver. However, the term "autonomous vehicle" as used herein also applies to semi-autonomous vehicles, which have greater autonomous decision-making capabilities and therefore still require a certain degree of driver supervision. Summary of the invention
[0007] The core problem to be solved by the present invention is to annotate road images quickly and efficiently but still accurately, where the annotations represent visual road structures. Such images can be used, for example, to prepare a very large training set of annotated road images suitable for training convolutional neural networks (CNNs) or other state-of-the-art machine learning (ML) structure detection components.
[0008] One aspect of the present disclosure relates to techniques that allow for rapid annotation of large numbers of road images in a semi-automated manner.
[0009] This technique allows large numbers of training images to be generated more efficiently and quickly than such manual annotation by eliminating or at least drastically reducing the need for human intervention.
[0010] The performance of such a trained structure detection component is strongly dependent on the quality and size of the training set. Being able to generate a large, accurately annotated training set of road images that is not practical to generate using traditional annotations, in turn, means that the accuracy of the trained structure detection component can be improved when applied to images that were not encountered in training (at runtime).
[0011] A first aspect of the present invention provides a method for annotating frames in a frame time sequence captured by at least one moving vehicle, the method comprising: determining a three-dimensional (3D) road model of an area captured in the frame time sequence; receiving first annotation data representing a known 3D position of a moving object in a first frame of the frame time sequence; and automatically generating second annotation data for marking the expected moving object position in at least a second frame of the frame time sequence by assuming that the moving object moves along an expected path determined based on the known 3D position and the 3D road model.
[0012] The terms "frame" and "further frame" may be used interchangeably with the terms "first frame" and "second frame", respectively. Similarly, the terms "annotation data" and "further annotation data" may be used interchangeably with "first annotation data" and "second annotation data". The terms "image" and "frame" may be used interchangeably and in a broad sense to cover any 2D or 3D structure representation (including point clouds, RGBD images, etc.). The term "2D image" may be used to specifically refer to, for example, an RGB image, etc.
[0013] A second aspect of the present invention provides a method for annotating a road image, the method comprising the following steps implemented in an image processing system:
[0014] receiving a time sequence of images captured by an image capture device of a moving vehicle;
[0015] determining a three-dimensional (3D) road model of the area captured in the image time series;
[0016] receiving annotation data representing a known position of a moving object in at least one image of a time sequence of images;
[0017] By assuming that the mobile object moves along an expected path determined from the known position and the 3D road model, further annotation data is automatically generated for marking the expected mobile object position in at least one other image of the image time series.
[0018] The present invention has particular advantages in the context of annotating 2D images based on 3D road models. The efficiency in this context comes from using object annotation elements (such as cuboids or other 3D volumes) in 3D space, and then geometrically projecting them back into the individual image planes of the 2D image to generate 2D annotation data for moving objects. As long as the position and movement of the object annotation elements are accurately captured, the burden of manual annotation can be greatly reduced.
[0019] In order to take into account the movement of objects between images, the object annotation elements can be moved in 3D space under the assumption that the object annotation elements follow the road shape of the 3D road model (i.e., so that the expected path matches the road shape). To do this, for at least one image, the position of the 3D object annotation elements in 3D space needs to be known. This can be established by manual annotation (i.e., annotation data representing known positions can be manually generated), but the problem in this case is that the annotator can only use a single 2D view, which corresponds to the 2D image plane of the relevant image. The problem in this case is also that the freedom to place the 3D annotation elements in 3D space is too high: the 3D annotation elements may be placed in multiple positions in 3D space, all of which will appear correct when projected back to the image plane of the 2D image, but because the actual starting point of the object in 3D space has not been correctly established, incorrect results will be obtained when the object annotation elements move between frames.
[0020] This problem can be overcome by incorporating geometric constraints on the elevation of the object annotation feature relative to the road model road surface (which can be defined in a direction perpendicular to the road surface). For example, for a moving vehicle, it can be assumed that the moving vehicle always stays on the road surface when moving along the path, for example, the vertical spacing between one or more predetermined reference points of the object annotation feature (e.g., the four bottom corners of a cuboid) and the road surface is zero or substantially zero.
[0021] In this way, even in the context of a 2D image, the annotator can define first (3D) annotation data that represent the known 3D position (in 3D space) of the moving object. In other words, the advantage of the geometric constraints imposed based on the 3D road model is that it allows the annotator to accurately position the 3D annotations relative to the 2D image, which in turn allows the automatic determination of 3D and 2D annotations for further images. For example, the 2D annotations can be determined by projecting the automatically determined 3D annotations into the respective image planes of the further images.
[0022] In other words, in the embodiment of the present invention, by assuming that the mobile object moves along a predetermined path at a predetermined elevation (which may be zero) in the road surface of the 3D road model, additional annotation data may be automatically generated.
[0023] Further exemplary embodiments of the first and second aspects are described below.
[0024] Additional annotation data may be generated for other images by moving an object annotation feature (e.g., a cuboid or other 3D volume) from a known position in 3D space along an expected path to an expected position in 3D space, and geometrically projecting the object annotation feature at the expected position into the 2D image plane of the other images.
[0025] The known position and the expected position may be such that a predetermined elevation from the road surface is maintained at both positions.
[0026] In response to annotation input received at a user interface of the image processing system, annotation data representing the known position of the moving object can be generated, wherein the annotation input can cause the object annotation element to move while maintaining a predetermined elevation from the road surface. In other words, a geometric constraint (restriction) can be imposed so that the annotation object can only be moved in the road plane by the annotator so as to always maintain the predetermined elevation.
[0027] Annotation data may be generated in response to annotation input by geometrically projecting the object annotation features into the 2D image plane of the image. The annotation data may be displayed with the image (e.g., overlaid on the image) and updated as annotation input is received so that the annotator can see when the object annotation features have been correctly positioned based on the 2D projection, i.e., he can see when a known position has been accurately established relative to the image (and where annotation features of the correct size and shape are applicable, etc.). Built-in geometric constraints allow the annotator to accurately establish the correct known position in 3D space using only the 2D projection into the image plane of the associated image.
[0028] Preferably, the expected road model is determined by reconstructing the path of the vehicle while capturing the image time series. That is, the 3D road model can show a road shape corresponding to the shape of the reconstructed vehicle path (that is, the expected road shape can be the shape of the reconstructed vehicle path).
[0029] Further example embodiments are described below. These embodiments are framed in the context of the first aspect, but those same features can also be applied to any other aspects described herein unless otherwise noted.
[0030] The frames may be 2D images.
[0031] Based on geometric constraints imposed on the elevation of the known 3D position relative to the road surface of the 3D road model, the 3D position of the moving object may be determined for the first frame.
[0032] The geometric constraints may constrain the known 3D positions to a predetermined zero elevation or non-zero elevation relative to the road surface of the 3D road model.
[0033] The 3D annotation element may be located at an initial 3D position, the 3D annotation element may be geometrically projected into an image plane of the first frame, and the 3D position may be adjusted while maintaining the geometric constraints to match the projection of the 3D annotation element with the moving object in the first frame, thereby determining a known 3D position of the moving object.
[0034] The 3D position can be manually adjusted while maintaining the geometric constraints to manually match the projection to the moving object in the image plane.
[0035] The second annotation data may be determined by transforming the 3D annotation element so as to move it along the expected path to the expected moving object position in the 3D space.
[0036] The second annotation data may include 2D annotation data for marking the expected mobile object position in the image plane of the second frame, the 2D annotation data being determined by projecting a 3D annotation element at the expected mobile object position into the image plane of the second frame.
[0037] First 2D annotation data may be determined for the first frame by projecting 3D annotation elements at known 3D locations into an image plane of the first frame.
[0038] The expected path may be determined based on the speed of the moving object and the time interval between the first frame and the second frame.
[0039] It can be assumed that this speed matches the speed of the moving vehicle.
[0040] An additional known 3D position of the moving object may be determined for one-third of the frames, and based on the time interval between the first frame and the third frame, the known 3D position and the additional known 3D position may be used to estimate the velocity.
[0041] The expected road model may be determined by reconstructing a path traveled by the vehicle while capturing a time series of images, and the 3D road model exhibits a road shape corresponding to a shape of the reconstructed vehicle path.
[0042] The path can be reconstructed by processing the frame time sequence.
[0043] Alternatively or additionally, the path may be determined based on satellite positioning data or other sensor data captured simultaneously with the frames.
[0044] The expected path of the moving object can be defined as having zero vertical offset in the road surface normal direction relative to the reconstructed vehicle path.
[0045] The expected path of the moving vehicle can be defined with a fixed lateral offset relative to the reconstructed vehicle path.
[0046] The direction of movement of a moving object can be inferred based on its position relative to the road structure of the 3D road model - for example, its position can be used to infer whether it belongs to the same traffic flow (moving in the same direction) or the upcoming traffic flow (moving in the opposite direction).
[0047] The method may further comprise generating classification data for classifying the frames based on the positions of the mobile objects in the frames relative to the road structure of the 3D road model.
[0048] Classification frames can be used to validate machine learning models.
[0049] Alternatively or additionally, the annotated frames can be used to train a machine learning model in a supervised training process.
[0050] The position data captured in association with the frame time sequence may be used to match the frame time sequence to a map area in order to determine a road shape for a 3D road model.
[0051] The second annotation data may take the form of 3D annotation elements located at the locations of the expected moving objects in the 3D space.
[0052] As an alternative to 2D images, the frames may be 3D frames. All of the above disclosures are equally applicable to such frames unless otherwise noted. All descriptions of 2D images herein are equally applicable to other types of frames unless otherwise required by the context.
[0053] Each 3D frame may include at least one of a depth map and a point cloud.
[0054] The 3D annotation element may be a 3D bounding box. Alternatively, the 3D annotation element may be a 3D model.
[0055] The 3D model may be derived from one or more 3D frames.
[0056] Further example embodiments framed in the context of the second aspect are described below. However, the teachings are equally applicable to any other aspects or embodiments described herein.
[0057] The second annotation may include 2D annotation data generated by moving an object annotation feature from a known position in the 3D space along an expected path to an expected position in the 3D space and geometrically projecting the object annotation feature at the expected position to a two-dimensional image plane of the second frame.
[0058] The known position and the expected position may be such that both positions maintain a predetermined zero elevation or non-zero elevation from the road surface.
[0059] Annotation data representing a known position of a moving object may be generated in response to annotation input received at a user interface of the image processing system, wherein the annotation input may cause the object annotation element to move while maintaining a predetermined elevation from a road surface.
[0060] Annotation data may be generated in response to the annotation input by geometrically projecting the object annotation features into a 2D image plane of the image.
[0061] The annotation data may be superimposed on or otherwise displayed with the image and updated as annotation input is received so that the annotator can see when the object annotation features are correctly positioned based on the projection of the 2D annotation features displayed with the image.
[0062] The annotation data may be generated in response to annotation input received at a user interface of the image processing system.
[0063] Additional annotation data may be generated based on the expected speed of the moving object along the expected path.
[0064] It can be assumed that the expected speed of the moving object matches the speed of the moving vehicle.
[0065] The annotation data may represent respective known positions of the moving object in the at least two images, and an expected speed of the moving object may be determined based thereon.
[0066] For images between two images in the sequence, the moving object position may be interpolated along the expected path based on the expected velocity to generate additional annotation data for these images.
[0067] The above applies equally to frames other than 2D images, unless the context requires otherwise.
[0068] The path may be reconstructed by processing the image time series (sequence).
[0069] The expected path of the moving object can be defined as having zero vertical offset in the road surface normal direction relative to the reconstructed vehicle path.
[0070] The road surface normal can be determined by processing a time series of images.
[0071] The expected path of the moving vehicle can be defined with a fixed lateral offset relative to the reconstructed vehicle path.
[0072] The annotation data may be generated in response to annotation input received at a user interface of the image processing system.
[0073] Additional annotation data may be generated based on the expected speed of the moving object along the expected path.
[0074] It can be assumed that the expected speed of the moving object matches the speed of the moving vehicle.
[0075] Alternatively, the annotation data may represent respective known positions of the moving object in the at least two images, and the expected speed of the moving object may be determined based thereon. In this case, the at least two images may each be manually annotated as described above.
[0076] For images between two images in the sequence, the moving object position may be interpolated along the expected path based on the expected velocity to generate additional annotation data for these images.
[0077] The moving direction of the moving object may be inferred based on the position of the moving object relative to the road structure of the 3D road model.
[0078] The method may further comprise: generating classification data for classifying the image based on the position of the mobile object in the image relative to the road structure of the 3D road model. As described later, the classified image may, for example, be used as a basis for M1 verification.
[0079] It should also be noted that while the reconstructed vehicle path may be used to infer the expected road shape, the knowledge of the expected road shape may come from any source. For example, position data captured in association with the image time series may be used to match the image time series to an area of a map (such as an HD map) in order to determine the expected road shape from the map. In general, it may be assumed that the expected path of a moving object matches the expected road shape, but this knowledge may be obtained.
[0080] Object detection can take various forms, such as bounding box detection, image segmentation of relevant object categories, etc. It should be appreciated that in the context of ML training, the form of object detection to be learned determines the form of annotation data to be generated for the training images.
[0081] Another aspect of the present invention provides a method for annotating a road image, the method comprising the following steps implemented in an image processing system:
[0082] receiving a time sequence of images captured by an image capture device of a moving vehicle;
[0083] Processing the images to reconstruct the path of the vehicle in three-dimensional space;
[0084] receiving annotation data representing a known position of a moving object in at least one image of a time sequence of images;
[0085] The known position and the reconstructed vehicle path are used to automatically generate additional annotation data for marking the expected position of the mobile object in at least another image of the image time series by assuming that the mobile object moves along an expected path determined from the known path and the reconstructed vehicle path.
[0086] The road image may be a street view image.
[0087] Yet another aspect of the present invention provides a method for annotating frames in a frame time sequence captured by at least one moving vehicle, the method comprising: determining, in a frame processing system, a three-dimensional (3D) road model of an area captured in the frame time sequence; and generating annotation data for marking a 3D object position in at least one frame of the frame time sequence, the annotation data being generated based on user input received at a user interface of the frame processing system, and being used to move an object position in 3D space subject to imposed geometric constraints to an elevation of the 3D object position relative to a road surface of the 3D road model.
[0088] The geometric constraints may constrain the known 3D positions to a predetermined zero elevation or non-zero elevation relative to the road surface of the 3D road model.
[0089] The 3D annotation element can be located at an initial 3D position, the 3D annotation element can be geometrically projected into the image plane of the first frame, and the 3D position can be adjusted while maintaining the geometric constraints to match the projection of the 3D annotation element with the moving object in the first frame, thereby determining the known 3D position of the moving object.
[0090] The annotation data may include 2D annotation data generated by projecting 3D annotation elements at the determined 3D object positions into an image plane of the frame.
[0091] The expected road model may be determined by reconstructing a path traveled by the vehicle while capturing a time series of images, and the 3D road model exhibits a road shape corresponding to a shape of the reconstructed vehicle path.
[0092] Another aspect of the present invention provides an image processing computer system, which includes one or more processors configured to perform the above steps, and a computer program including executable instructions, which, when executed on one or more processors of the image processing computer system, cause the computer system to perform the above steps.
[0093] The present invention provides a computer program product comprising code stored on a computer readable storage medium and configured to implement the method or system of any preceding claim when executed on one or more processors.
[0094] Yet another aspect of the present invention provides a processing computer system comprising one or more processors configured to perform any of the above steps.
[0095] Yet another aspect of the present invention is a computer program comprising executable instructions which, when executed on one or more processors of a processing computer system, cause the computer system to perform any of the above steps. BRIEF DESCRIPTION OF THE DRAWINGS
[0096] The present invention will be more clearly understood and the manner in which the embodiments of the present invention are implemented will be shown by way of non-limiting examples in conjunction with the following drawings, in which:
[0097] Figure 1 A highly schematic functional block diagram of a training system for training a structure detection component is shown;
[0098] Figure 2 A highly schematic block diagram showing an autonomous vehicle;
[0099] Figure 3 A highly schematic block diagram showing a vehicle for capturing road images to be annotated;
[0100] Figure 3A showing a schematic front view of a vehicle;
[0101] Figure 4 A schematic block diagram of an image processing system is shown;
[0102] Figure 4A Show Figure 4 Expanded diagram of the system in the middle;
[0103] Figure 5 A flowchart of the automatic image annotation process is shown on the left, and an example diagram of the corresponding steps of the method is shown on the right;
[0104] Figure 6 A schematic block diagram showing the functionality of an imaging system to facilitate adjustments during a manual fixer stage is shown;
[0105] Figure 7 , Fig. 7A and Figure 7B Demonstrate the principles of simultaneous localization and mapping (SLAM) for reconstructing vehicle paths;
[0106] Figure 8 showing a schematic perspective view of a vehicle;
[0107] Fig. 9 A schematic perspective view showing a vehicle equipped with at least one image capturing device;
[0108] Fig.10 Certain principles of image-based measurement of angular orientation of an image capture device are illustrated;
[0109] Fig.11 Some principles of image-based road width measurement are shown;
[0110] Fig.12 An example of manual adjustment during the manual fixing phase is shown;
[0111] Fig.13 shows an example of an annotated road image;
[0112] Fig.14 shows an example of SLAM output;
[0113] Fig.15 Some principles of image-based direction of travel are shown;
[0114] Fig.16 Some principles of image-based measurement of road normals are shown;
[0115] Fig.17 Example image showing SLAM reconstruction of a camera position moving through a point cloud world;
[0116] Figures 18 to 24 shows an example of an annotation tool user interface for annotating an image based on the techniques of the present disclosure;
[0117] Fig.25 shows continuous annotation applied to a road image;
[0118] Fig.26 showing reference points defined relative to an image capture device of the vehicle;
[0119] Fig. 27 Shows more details of the road normal estimation process;
[0120] Fig.28 A top view showing the ego lane estimation;
[0121] Fig.29 A flow chart illustrating a method of annotating a moving object;
[0122] Fig.30 Show Fig.29 A schematic diagram of a possible way in which the steps of the method can be applied in practice;
[0123] Fig.31 An example of an annotation interface for annotating a 3D frame is shown;
[0124] FIG. 32A to FIG. 32E shows an example of the extended annotation interface in use; and
[0125] Fig.33 Show Fig.29 The steps of the method are applied to 3D annotation elements to generate 2D and / or 3D annotation data. DETAILED DESCRIPTION
[0126] For example, reference is made to PCT International Patent Application entitled “Image Annotation” and Application Number PCT / EP2019 / 056356, the entire contents of which are incorporated herein by reference. The present invention discloses an image processing system, wherein, for image annotation, a 3D path (VP) of a moving vehicle (ego vehicle) is reconstructed using a simultaneous localization and mapping (SLAM) process. The SLAM process is applied to images captured by an image capture device (camera) of the vehicle. When processing the images, the corresponding position of the ego vehicle on the ego vehicle path VP when each image was captured (referred to as the capture position of the image) is determined.
[0127] Further details of the image processing system will now be described to provide relevant context for the embodiments of the invention described below under the heading "Annotating Objects". The invention uses a 3D road model as a basis for annotating static objects or objects moving between frames. In the embodiments of the invention described below, the 3D road model is inferred from the path of the host vehicle in 3D space using the techniques outlined and described above. However, the invention is not limited in this respect, and other methods (some examples are given later) may be used to determine the 3D road model.
[0128] Image processing system
[0129] Autonomous vehicles need to know the layout of the surrounding roads, which can be predicted by state-of-the-art CNNs. This work addresses the current lack of data for determining lane instances required for various driving maneuvers. The main problem is the time-consuming manual annotation process that is usually applied image by image.
[0130] The present disclosure recognizes that driving a car is itself a form of annotation. This is exploited in a semi-automatic method that allows an image sequence to be efficiently annotated by utilizing an estimated road plane in 3D and projecting annotations from that plane to all images in the sequence. The average annotation time per image is reduced to 5 seconds, and data capture requires only a low-cost dashcam.
[0131] Autonomous vehicles have the potential to revolutionize urban transportation. Mobility will be safer, always available, more reliable and less expensive.
[0132] An important issue is to give autonomous driving systems knowledge about the surrounding space: self-driving cars need to know the road layout around them in order to make smart driving decisions. The present disclosure addresses the problem of detecting driving lane instances from a camera mounted on the vehicle. Spatially limited individual lane instance areas are required to perform various challenging driving maneuvers, including lane changes, overtaking, and intersections.
[0133] Typical state-of-the-art CNN models require a large amount of annotated data to reliably detect lane instances. However, there are not many publicly available annotated datasets, primarily due to the time-consuming annotation process; the time to fully annotate an image for semantic segmentation tasks ranges from several minutes to over an hour per image. In contrast, our semi-automatic annotation process reduces the average time per image to a few seconds. This is accelerated by: (1) noting that driving a car is itself a form of annotation and that cars are mostly driven along lanes; (2) propagating manual annotation adjustments from a single view to all images in the sequence; and (3) accepting unannotated parts in ambiguous cases.
[0134] Some previous work aims to create semi-automatic object detection in autonomous driving scenarios. It proposes to detect and project the future driving path in the image, but does not address the issue of lane annotation. This means that the path does not fit the lane width, but crosses lanes and intersections. In addition, it requires a high-cost sensor suite that includes calibrated cameras and lidars. In contrast, the present method works on data from GPS-enabled dashcams. The contributions of this disclosure include:
[0135] - A semi-automatic annotation method for 3D lane instances that requires only low-cost dashcam equipment;
[0136] - Road annotation is possible despite occlusions in traffic jams;
[0137] - Experimental results on road, ego lane and lane instance segmentation using CNN.
[0138] The following describes a method that provides a fully automatic way to generate training data, whose quality is only slightly lower than conventional manual processing.
[0139] This paper also describes an extension to this that introduces a manual correction ("manual fixing") stage. With this extension, the method becomes a semi-automatic method that produces results of the same high quality as conventional manual methods, but with a hundred times less manual effort (measured in annotation time per image). This is based on the observation that typical annotation times (using the current type of fully manual annotation in use today) can be anywhere between 7 minutes and 90 minutes per image; however, using the method described below, annotation times of about 12 seconds per image can be achieved.
[0140] In the method described below, the training images are frames of video images. It is obvious that the method described in this article is particularly suitable for batch annotation of video frames captured by a moving training vehicle. For a typical training video (consisting of a series of static 2D images, i.e., frames), the annotation time for each typical training video sequence is up to 7 minutes, which is equivalent to about 12 seconds per image, while still obtaining high-quality results.
[0141] Figure 1 A highly schematic functional block diagram of a supervised training system for training a perception component 102 based on a set of annotated perception inputs 108 (i.e., the perception inputs together with associated annotation data) is shown. In the description below, the perception component 102 may be synonymously referred to as a structure detector, a structure detection component, or simply a structure detector. The perception inputs used for training purposes may be referred to herein as training examples or training inputs.
[0142] The perception input may be a 2D image, and a 2D image annotated with 2D annotation data (e.g., a 2D bounding box, a 2D segmentation map, etc.) may be used to train a 2D perception component, such as a 2D object detector, a 2D instance segmentation component, etc. However, the perception input may also be 3D, i.e., include 3D structure data, and a 3D structure detector (e.g., a 3D bounding box detector, a 3D pose estimator, a 3D object detector, etc.) may be trained using 3D annotation data (e.g., a 3D bounding box).
[0143] exist Figure 1 In the figure, the training examples are labeled with reference numeral 104, and a set of annotation data associated therewith is labeled with reference numeral 106. The annotation data 106 provides ground truth for the associated training examples 104. For example, for training examples in the form of images, the annotation data 106 may mark the locations of certain structural components (such as roads, lanes, intersections, non-drivable areas, etc.) within the image 104 and / or objects (such as other vehicles, pedestrians, street signs or other infrastructure, etc.) within the image.
[0144] The annotated perception input 108 may be divided into a training set, a test set, and a validation set, labeled 108a, 108b, and 108c, respectively. The annotated training examples may be used to train the perception component 102 without forming part of the training set 108a because they are used for testing or validation.
[0145] The perception component 102 receives a perception input, denoted as x, from one of the training set 108a, the test set 108b, and the validation set 108c, and processes the perception input x to provide a corresponding perception output, denoted as
[0146] y=f(x;w)
[0147] In the above, w represents a set of model parameters (weights) of the perception component 102, and f represents a function defined by the weights w and the architecture of the perception component 102. For example, in the case of 2D or 3D bounding box detection, the perception output y may include one or more detected 2D or 3D bounding boxes derived from the perception input x; in the case of instance segmentation, y may include one or more segmentation maps derived from the perception input. In general, the format and content of the perception output y depends on the choice of the perception component 102 and its chosen architecture, and these choices are made according to the one or more desired perceptual modalities to be trained on it.
[0148] The detection component 102 is trained based on the perceptual inputs of the training set 108a to match its output y=f(x) with the ground truth provided by the associated annotation data. The ground truth provided for the perceptual input x is denoted herein as y x . Thus, for a training example 104 , the ground truth is evidenced by the associated annotated data 106 .
[0149] This is a recursive process in which the input component 112 of the training system 110 systematically provides the perceptual inputs of the training set 108b to the perceptual component 102, and the training component 114 of the training system 110 adapts the model parameters w in an attempt to optimize an error (cost) function that compensates for the difference between each perceptual output y = f(x; w) and the corresponding ground truth y x The deviation is characterized by a defined metric such as mean squared error, cross entropy loss, etc. Thus, by optimizing the cost function to a defined degree, the overall error of the entire training set 108a measured relative to the ground truth can be reduced to an acceptable level. The perception component 102 may be, for example, a convolutional neural network, where the model parameters w are the weights between neurons, but the present disclosure is not limited thereto. It should be appreciated that there are several forms of perception models that can be usefully trained on appropriately annotated perception inputs.
[0150] The test data 108b is used to minimize overfitting, which refers to the fact that beyond a certain point, improving the accuracy of the detection component 102 on the training data set 108a is detrimental to its ability to generalize to perceptual inputs not yet encountered in training. Overfitting may be identified as the point at which improving the accuracy of the perceptual component 102 on the training data 108 reduces (or does not improve) its accuracy on the test data, where accuracy is measured according to an error function. The goal of training is to minimize the total error on the training set 108a to the point where it can be minimized without overfitting.
[0151] If necessary, the validation data set 108c may be used to provide a final estimate of the performance of the detection assembly.
[0152] The description initially focuses on 2D perceptual input in the form of annotated street view images (also referred to herein as road images), i.e., street view images with associated image 2D annotation data. 3D perceptual input and 3D annotation data are described later.
[0153] The following methods may be used to automatically or semi-automatically generate such annotated data 106 for training, testing, and / or validating the detection component 102 .
[0154] Figure 2A highly schematic block diagram of an autonomous vehicle 200 is shown, which is shown to include an example of a trained detection component 102, whose input is connected to an image capture device 202 of the vehicle 200 and whose output is connected to an autonomous vehicle controller 204. In use, the trained structure detection component 102 of the autonomous vehicle 200 detects structures within images captured by the image capture device 102 in real time according to its training, and the autonomous vehicle controller 204 controls the speed and direction of the vehicle based on the results without any human input or with limited human input.
[0155] although Figure 2 Only one image capture device 202 is shown in FIG. 1 , but the autonomous vehicle may include multiple such devices. For example, a pair of image capture devices may be arranged to provide a stereoscopic view, and the road structure detection method may be applied to the images captured from each image capture device.
[0156] It should be appreciated that this is a highly simplified description of some of the autonomous vehicle functionality. The general principles of autonomous vehicles are well known and will not be described in detail.
[0157] 2Video Collection
[0158] For later detailed experiments, a standard Nextbase 402G professional dash cam was used to record video and associated GPS data at 30 frames per second at a resolution of 1920x1080 and compressed using the H.264 standard (although any suitable low-cost image capture device could be used to the same advantage). The camera was mounted on the inside of the car windshield approximately along the centerline of the vehicle and approximately aligned with the axis of motion.
[0159] Fig.26 (Top left) shows an example image from the collected data. To eliminate sections where the car was moving very slowly or was stationary (which is quite common in urban environments), only frames that were at least 1 meter apart according to GPS were included. Finally, the recorded data was split into sequences of 200m in length.
[0160] Fig.25 An example road image (top left) is shown, including annotations of the road (top right), ego lane (bottom left), and lane instances (bottom right). The road and lanes below the vehicle are annotated despite occlusion. Uncolored portions are unannotated, i.e., their categories are unknown.
[0161] Figure 3 A simplified block diagram of a vehicle 300 is shown, which can be used to capture road images to be annotated, i.e., as described above with reference to Figure 1Preferably, these images are captured as frames of short video clips recorded while the vehicle 300 is driving along the road, in a manner that allows the path traveled by the vehicle during the recording of each video clip to be reconstructed from the frames of the video clip. The vehicle 300 may be referred to as a training vehicle, for ease of reference. Figure 2 300. Training vehicle 300 is shown as including an image capture device 302, which may be a forward or rearward image capture device, and coupled to a processor 304. Processor 304 receives captured images from image capture device 302 and stores them in memory 306, from which they can be retrieved for use in the manner described below.
[0162] In this example, vehicle 300 is a car, but it may be any form of transportation.
[0163] The invention is based on the assumption that the path traveled by a training vehicle 300 driven by a human follows a road, and therefore the location of the road can be inferred from any path taken by the training vehicle 300. When a particular image in a captured sequence of training images is to be annotated, the hindsight of the path subsequently taken by the training vehicle 300 after the image was captured allows for automatic annotation. In other words, the location of the road in the image can be inferred using the hindsight of the vehicle's behavior after the image was captured. The annotated road location is what would be expected given the path subsequently traveled by the training vehicle 300 and the underlying assumptions about its relevance to the road location.
[0164] As further described below, the path is determined by processing the captured image itself. Accordingly, when a particular image is annotated with an expected road location, for a forward (backward) image capture device 302, the expected road location in the image is determined based on the path that the vehicle traveled after (before) the image was captured, as reconstructed using at least one subsequent (previous) captured image in the sequence of captured images. That is, each annotated image is annotated using path information derived from one or more images captured after (before) the image being annotated.
[0165] 3 Video Annotations
[0166] Fig. 27 Shows the lane boundary points for frame i The estimation principle. i is the camera position at frame i (obtained via SfM), f is the heading, n is the normal vector to the road plane (both oriented relative to the camera), h is the height of the camera above the road, r is the horizontal vector across the lane, is the distance to the left and right lane boundaries.
[0167] The initial annotation step is automated and provides an estimate of the road surface in 3D space as well as an estimate of the ego lane (see Section 3.1). The estimates are then manually corrected and additional annotations are added in road surface space. The annotations are then projected into the 2D camera view, allowing all images in a sequence to be annotated at once (see Section 3.2).
[0168] 3.1 Automatic 3D ego lane estimation
[0169] Given a video sequence of N frames from a camera with unknown intrinsic and extrinsic parameters, the goal is to determine the 3D road surface and project an estimate of the ego lane onto it. To do this, we first apply OpenSfM
[28] , a “structure-from-motion” algorithm, to obtain the 3D camera position ci and set R in the global coordinate system for each frame i∈{1,...,N}. i , as well as camera focal length and distortion parameters.
[0170] The road is assumed to be a 2D manifold embedded in a 3D world. Furthermore, the local curvature of the road is low, so the orientation of the wheels provides a good estimate of the local road slope. The camera is fixed inside the vehicle and statically translated and rotated relative to the current road plane (i.e., the body is assumed to follow the road plane and suspension motion is ignored). Therefore, the ground point g on the road below the camera in frame i is i It can be calculated as:
[0171] g i =c i +hR i n
[0172] where h is the height of the camera above the road and n is the road normal relative to the camera (see Fig.26 Left side). Therefore, the left and right boundaries of the own lane can be derived as:
[0173]
[0174] Where r is the vector perpendicular to the driving direction in the road plane, For more information, see Fig.26 (right). A brief assumption is made that the road surface is flat and perpendicular to the direction of motion of the car (but no assumption is made that the road is generally flat - if the ego path goes over a hill, this is captured in the ego path).
[0175] Given a frame i, all future lane boundaries
[0176]
[0177] Projection into the local coordinate system can be done as follows:
[0178]
[0179] Lane annotations can then be drawn as polygons with corner points for adjacent future frames.
[0180]
[0181] This implicitly assumes that the lane is piecewise straight and flat between captured images. The following sections describe how to measure or otherwise obtain the following quantities:
[0182]
[0183] It should be noted that h, n and r need only be estimated once for all sequences with the same camera position.
[0184] The camera height ft above the road is easy to measure manually. However, in cases where this is not possible (e.g., for dashcam videos downloaded from the web), the camera height can be obtained using the estimated road surface grid obtained from OpenSfM. A rough estimate of h is sufficient, as it can be corrected by manual annotation, see the next paragraph.
[0185] Fig. 27 Shows the estimation of the road normal n and the heading f at a single frame i i The final estimate is the sum of all frames.
[0186] The estimation of the road normal n is based on the fact that when the car moves around a turn, the vectors m representing its motion will all lie in the plane of the road, so taking the cross product of these vectors will give the road normal, see Fig. 27 Let m i,j is the normalized motion vector between frame i and frame j, that is,
[0187]
[0188] The estimated road normal for frame i (in camera coordinates) is:
[0189]
[0190] in represents the cross product (see Fig. 27 ). The quality of this estimate depends heavily on the correctness of the previous assumptions. To obtain a more reliable estimate, the n i The average is taken, and weighted implicitly by the cross-product magnitude:
[0191]
[0192] The normal can be estimated only during turns, so this weighting scheme emphasizes the deflection during turns and ignores the straight portion of travel.
[0193] r is perpendicular to the direction of travel f and in the plane of the road, resulting in:
[0194]
[0195] The only quantity left is f, which can be derived using the following fact:
[0196] m i-l,i+l
[0197] If the turn rate is low, it is approximately parallel to c i处的切线 Therefore, the forward point at frame i can be estimated (see Fig. 27 ):
[0198]
[0199] As for the normal, we can adjust f in the whole stroke i Take the average to get a more reliable estimate:
[0200]
[0201]
[0202] In this case, the movement is weighted according to the inner product so that the weights of these parts are increased at low turn rates while maximizing forward movement.
[0203] quantity It is important to properly align the annotated lane boundaries with the visible boundaries.
[0204] To estimate these quantities, we can assume that the ego lane has a fixed width w,
[0205] And the car is already driving exactly in the center, that is:
[0206]
[0207] is constant for all frames. In an extension (see next paragraph), this assumption is relaxed to obtain improved estimates through manual annotation.
[0208] In practice, sequences with multiple turns in the road plane are chosen to estimate n, and straight line sequences are chosen to estimate f. The same value is then reused for all sequences with the same static camera position. Only the first part of the sequence is annotated until 100m from the end, otherwise not enough future lane boundary points can be projected. A summary of the automated lane annotation procedure is provided in Appendix A (Algorithm 1), Fig.28 A visualization of the automatic boundary point estimation is shown in (see below) and labeled accordingly.
[0209] See below for more details.
[0210] Figure 4 The present invention is shown to automatically generate a training method for the above reference. Figure 3 A schematic block diagram of an image processing system for annotating data of captured images in a manner depicting the manner in which the annotation data marks expected road locations within the images, wherein these locations are inferred using the above-described assumptions. Figure 4A An extension of this system is shown as follows. Figure 4A The systems in Figure 4 All components in the Figure 4 All descriptions apply equally to Figure 4A .
[0211] Figure 5 The left side shows a flow chart of the automatic image annotation process implemented by the image capture system, Figure 5 An example description of the corresponding steps is shown on the right.
[0212] Now we will describe in parallel Figure 4 and Figure 4A Image system and Figure 5 process.
[0213] Figure 4 The image processing system in FIG. 4 is shown to include a path reconstruction component 402 , a road modeling component 404 , and an image annotation component 406 .
[0214] The path reconstruction component 402 receives a sequence of captured two-dimensional (2D) images 400 and processes them to perform a three-dimensional (3D) reconstruction of the path traveled by the vehicle based on the captured image sequence. Figure 5 , step 502).
[0215] On the right side of step 502, Figure 5 Two reconstruction paths in three-dimensional space are shown.
[0216] The first path, labeled CP (Camera Path), is a 3D reconstruction of the path traveled by the image capture device (camera) 302 of the training vehicle 300. Techniques for reconstructing the path traveled by a moving camera based on a sequence of images captured by the camera while moving are well known in the art and will not be described in detail herein.
[0217] The path can be reconstructed from other data (accelerometers, high-precision GPS, etc.), but reconstruction from video alone can advantageously reduce the cost of capturing the data, as a basic dash cam can be used for data collection rather than a high-priced car containing high-cost accelerometers, high-cost precision GPS, etc.
[0218] The second path, labeled VP (Vehicle Path), is a reconstruction of the path traveled by the training vehicle 300 and is defined as the path traveled by a point located approximately halfway across the width of the training vehicle 300, i.e., equidistant to the left and right sides of the training vehicle, and approximately at road level below the image capture device 300 (at Figure 3 and Figure 3A That is, along the center line C ( Figure 3A ), the center line C extends vertically and is located at the midpoint along the width of the vehicle 300.
[0219] It should be noted that the points do not have to be exactly equidistant - the method will give accurate results even if the points are not equidistant.
[0220] If the height of the camera 302 above the road surface is known (in Figure 3 , Figure 3A and Figure 5 , the horizontal offset of the camera 302 relative to the center line C (in Figure 3A and Figure 5 The vehicle path VP on the ground plane can be directly determined from the camera path CP, given the camera's orientation relative to the vehicle (labeled S in the figure). The relative orientation of the camera can be captured as a "forward point" and an "eye level", defined as follows.
[0221] It should be pointed out that Figure 5 The examples shown on the right are illustrative and non-exhaustive and are intended only to illustrate the general principle that the position of road structures can be inferred from the 3D camera path CP using a small number of reference parameters.
[0222] Based on the 3D path reconstruction, the road modeling component 404 creates ( Figure 5 504 in the above example) assumes that the 3D model 602 ( Figure 6 ).
[0223] In the example shown on the right side of step 504, the road model is formed by two geometric curves R1 and R2 corresponding to the assumed lane boundary positions, which are defined as parallel to the vehicle path VP, at the same height as the vehicle path VP, located on either side of the vehicle path VP, and at a distance of W / 2 from the vehicle VP, where W is the assumed road or lane width.
[0224] This is based on the following assumptions: the training vehicle 300 travels along a road lane, the travel direction of the training vehicle 300 is approximately parallel to the actual lane boundary of the lane, and the vehicle is driven approximately in the center of the lane.
[0225] The expected road position may, for example, correspond to the edge of the road itself (in order to mark a non-drivable area on or beside the road) or to a lane position of a lane within the road.
[0226] In general, automatic annotation techniques can be used to mark the locations of what are referred to herein as "parallel structures", i.e. structures that are expected to be at least approximately parallel to the path of vehicle driving, or more generally structures that are expected to have a particular orientation relative to the vehicle path. This can be road structures, such as roads, lanes, non-drivable areas that are assumed to be parallel to the vehicle path, intersections that are assumed to extend perpendicular to the vehicle path, or real-life parallel structures that vehicles other than cars may encounter, such as runways (in the case of autonomous drones, aircraft, etc.). Accordingly, the entire description of the 3D road model is equally applicable to any other form of 3D structure model of the vehicle driving environment, for which the expected parallel structures can be automatically determined using the methods described herein.
[0227] In addition to the basic assumption that they are parallel to the path that the training vehicle 300 actually travels, different assumptions can be used to accommodate different types of parallel structures.
[0228] Here, the description may specifically refer to roads or lanes and related parameters such as road or lane width. However, it should be appreciated that the description applies to any parallel road structure, and any reference to road or lane width (etc.) generally applies to the width parameters determined for the intended parallel structure.
[0229] The image annotation component 406 uses the 3D road model to generate 2D annotation data to individually label each 2D training image with the expected road location ( Figure 5 , step 506).
[0230] like Figure 5 As shown on the right side of step 506, annotation data A(n) for the nth 2D image in the sequence (image n, which can be any image in the captured training image sequence) is generated by performing a projection of the 3D road model geometry onto the image plane, which is labeled I(n). In this example, this means projecting the 3D road boundaries R1, R2 onto the image plane I(n) so that the projections of these boundaries in the image are labeled as assuming that the road boundaries are located in image n.
[0231] Image plane I(n) is the plane corresponding to the field of view of camera 302 when image n is captured, and is therefore located at a point corresponding to the vehicle position when image n is captured.
[0232] The parameter calculation component 408 calculates various "reference parameters" that are used to build a 3D model and perform geometric projection based on the reconstructed vehicle path. These reference parameters include the following image capture parameters, which are related to the position and orientation of the image capture device 302 of the training vehicle 300:
[0233] 1) The height H of the camera 302 above the road;
[0234] 2) The horizontal offset S of the camera relative to the center line C;
[0235] 3) The orientation of the camera 302 relative to the vehicle, which can be captured as:
[0236] a. The "forward point" of camera 302 (defined below); and
[0237] b. The camera's "eye level" (defined below);
[0238] and additionally:
[0239] 4) The width W of the road / lane.
[0240] These parameters are used in the manner briefly described above, and further details of how they are used are described below. Suffice it to say that these parameters are calculated from the captured training images 400 themselves, which has various advantages, as described later.
[0241] The annotation data is stored in electronic storage 414 from which it can be accessed or retrieved for use in the above references. Figure 1 In this process, eventually, the automatic structure detection component 102 is learning from the driver's ability to keep the vehicle 300 approximately in the center of the road most of the time while capturing training images.
[0242] 3.2 Manual Correction and Additional Annotations
[0243] The annotation interface provides human annotators with the ability to individually review frames in a batch with the currently rendered lanes projected into the image. The annotator can widen, narrow, and move these lanes in the most recent frame, and these changes to the lane structure will be projected to all other frames in the batch, providing a clear advantage over annotating all images individually.
[0244] Fig.28 A top view showing the visualization of the ego lane estimation. The automatic estimation and manual correction are marked as:
[0245] w i left (automatic)
[0246] wi left (Manual)
[0247] Initially, all and is set to a constant value, so the estimate follows the driving path, but it can be seen that the lane boundaries cannot be annotated correctly. After that, it is estimated that it will be improved.
[0248] Figures 18 to 24 shows an example of the annotation interface used by annotators. Fig.18 In the center of the image, the ego path can be seen projected into the frame. At the bottom left, the annotator is provided with controls for manipulating the currently rendered lanes (narrowing, widening, moving left or right, moving lane boundaries, etc.) and adding new lanes. Fig.19 In , the annotator is provided with the mechanism to adjust the camera height so that the reconstruction matches the road surface, and to crop the height to exclude the dashboard or hood of the vehicle. All annotation is performed in the estimated 3D road plane, but the projection in the 2D camera view provides immediate feedback. The annotator can easily skip forward and backward in the sequence to determine whether the annotations are aligned with the image, and make corrections if necessary. An example of a corrected sequence can be seen in Fig. 27 (See above).
[0249] In addition to the lanes on the road, the land along the side of the road is annotated with reference NR1 as "non-road." In addition, the entire upper portion of each image (upper land NR2) is annotated as non-road, with manual resizing by the annotator. Fig.25 An example of the rendered annotations is shown.
[0250] The above process is Figure 4A , wherein a presentation component 416 is provided that can present a single training image (img n) labeled with expected road annotations determined for this image via a user interface (UI) 412. Thus, a user can check how well the expected road structure corresponds to the actual road structure visible in the image and make corrections if necessary.
[0251] The output of UI 412 is shown connected to the input of the image annotation component via connector 418 to represent the fact that the user can modify the annotation data A(n) of a single image via UI 412 to better align the hypothesized lane boundaries R1, R2 (only) with the actual lane boundaries in that image. This may be useful to resolve "one-off" discrepancies, e.g., those caused by unusual local road structure.
[0252] A model adaptation component 410 is also provided to allow efficient manual adaptation of annotation data across multiple images simultaneously via UI 412. This is achieved by allowing the user to adapt the 3D road model with reference to a single image and the annotation data presented therein to better align road boundaries in that image. However, because the 3D road model is being adapted (and not just the annotation data A(n) for a single image), these adaptations may also be applied to the annotation data for other images without the user having to manually adapt the annotation data for those other images. For example, a user may adjust an assumed lane width W to better fit one image, and because the road model is being adapted to the new lane width, the annotation data for other images may be automatically adapted based on the adapted model with the new lane width W. This will be described below with reference to FIG. Figure 6 This is elaborated.
[0253] Figure 4 and Figure 4A The above components of the image processing system in are functional components of the image processing system representing functions performed by the image processing system. These functions are implemented in software, i.e., by executable instructions (code) stored in a memory and executed by one or more processors of the image processing system (not shown), which may be formed by one or more general-purpose processing units (such as CPUs), special-purpose processing units (such as GPUs), or any combination thereof. It should be appreciated that these components represent high-level functions that can be implemented using various code structures. Figure 4 The arrows in the diagram generally represent high-level interrelationships between these high-level functions and do not necessarily correspond to any particular arrangement of physical or logical connections. Figure 5 The steps in are computer-implemented steps, and the instructions, when executed on one or more processors, cause the image processing system to perform these steps.
[0254] Example implementation details:
[0255] Fig.13 An example of annotated images is shown in Fig.13 There are two drivable lanes marked in the image: the ego lane towards the middle of the image, and another drivable lane on the left side of the image. There are two non-drivable areas marked on either side of these lanes, with a rightmost lane for traffic in the other direction separated from the ego lane by one of the non-drivable areas.
[0256] As mentioned above, this approach makes use of the following observations and assumptions:
[0257] 1) When a person drives a car, they typically follow a path that stays close to the center of a lane on the road. So if a video camera is placed forward (or backward) in a person's car, the path traced by that camera will (mostly) outline where the lane is as the camera is cast down toward the ground and expanded to the width of the lane.
[0258] 2) Once we have found the outline of where the lane is, for each image in the video, we can project the outline of the lane under the car back into that image and provide (almost) exactly the annotations needed to mark where the lane is.
[0259] As will be described later, preferred embodiments of the present invention may be implemented using low-cost equipment.
[0260] Determine the path and reference parameters:
[0261] In order to generate the lane positions and project them back to each image, as described above, the following quantities need to be known or inferred:
[0262] The path CP followed by the camera 302 in 3D space ( Figure 5 );
[0263] The orientation of the camera 302 within the vehicle 300, which can be represented using the following parameters:
[0264] o Forward points, which are defined as the pixels that the car appears to be facing when driving in a straight line;
[0265] о Eye level, defined as the line on the image that appears parallel to the road;
[0266] The height of the camera above the road surface (H) Figure 3 , Figure 3A and Figure 5 );
[0267] · The left / right position of the camera in the car (in Figure 3A and Figure 5 centered as interval S); and
[0268] Lane width W( Figure 3A and Figure 5 ).
[0269] The image processing techniques described below allow all of the above quantities to be determined using low cost and uncalibrated equipment (such as a mobile phone or dash cam or other low cost consumer device that costs a few hundred pounds.) This is because these quantities can be calculated from images captured by the training vehicle 300 using the image processing techniques described below.
[0270] 3) The 3D path of the camera CP can be derived by post-processing the video captured by the uncalibrated camera using existing visual SLAM (Simultaneous Localization and Mapping) techniques. Software packages such as OpenSFM and ORB-SLAM can be used to generate accurate camera path information.
[0271] Such techniques may be used to construct a 3D map of the environment in which the training vehicle 300 is traveling, while simultaneously tracking the vehicle's position within that environment.
[0272] Reference Figure 7 Some basic principles of image-based SLAM are briefly explained with examples. Figure 7 The top part shows three images of real-world environments, img(n-1), img(n), and img(n+1), captured at different times and different but similar locations. By identifying unique reference features present in the images (such as those labeled F1 and F2) and matching them across the images, a 3D model of these features can then be constructed using triangulation, as shown in Figure 1. Figure 7 For each captured image img(n-1), img(n), img(n+1), the 3D position of the camera 302 at the time of capturing the image is estimated, which is marked as x n-1 、x n , x n+1 , and the orientation of the camera's field of view in 3D space. For each image img(n-1), img(n), img(n+1), define the respective image plane I(n-1), I(n), I(n+1) as the plane that is orthogonal to and intersects the camera's field of view when the image was captured, at any non-zero distance x from the camera position n-1 , , x n+1 The position and orientation of the camera are chosen according to the principle of ray tracing applied to the reference features F1 and F2 so that for each camera position x n-1 , , x n+1 , the position where the light (straight line) from each reference feature F1, F2 to the camera position intersects the corresponding image plane I(n-1), I(n), I(n+1) at least approximately corresponds to the position where the reference feature appears in the corresponding image, that is, the pixel position of the feature in the image. Then, based on the obtained camera position x n-1 、x n , x n+1 The sequence reconstructs the camera path CP.
[0273] It should be noted that the 3D model constructed as part of the SLAM process is different from the 3D model constructed by the modeling component 404. The former is in the form of a road map constructed based on image triangulation, while the latter is an expected structure based on the inference of the vehicle path, which in turn is determined based on the image capture device positions derived when performing the SLAM process.
[0274] Reference Fig. 7A and Figure 7B , the 3D orientation of the camera 302 can be represented using the pitch angle (P'), the roll angle (R'), and the yaw angle (Y'). L 'It can be simply defined as parallel to the camera optical axis, with the horizontal and vertical axes a' T , a V' corresponds to the horizontal and vertical directions in the image plane (i.e., corresponds to the directions that appear horizontal and vertical in the captured image), where the camera's pitch angle P', roll angle R' and roll angle Y' are defined as the angles a around the camera's transverse axis, longitudinal axis and pivot axis (camera axis), respectively. T 'a' L , a V '. Here we use vector notation, where a T 'a' L , a V ' is a unit vector along the corresponding axis. Under this definition, the image plane I(n) is perpendicular to the horizontal axis of the camera and is horizontally / vertically aligned with the horizontal / vertical axis of the camera.
[0275] In practice, this type of SLAM technology does generate a surface reconstruction of the environment. Although in theory it is possible to use this surface reconstruction to directly determine the road / lane position, in practice the quality of this surface reconstruction is not sufficient to do this. However, the camera's motion in the 3D scene is of good quality and thus provides a sufficiently accurate reconstruction for the camera path CP (this is because the camera position is based on hundreds of feature points, while the surface position is based on only a few feature points), which in turn allows the road / lane position to be inferred with sufficient accuracy using the present technique.
[0276] It should be noted that some of these techniques can also automatically infer camera characteristics, such as focal length and even lens distortion parameters. Such parameters can be used to optimize the present technique, but they are not a requirement.
[0277] Fig.14 An example image showing a SLAM reconstruction of the camera position moving through a point cloud world. The sequence of pyramid structures labeled 1400 represents the position and orientation of the image capture device when taking different images, as estimated by the SLAM process. Together they define the camera path CP. The blocks labeled 1402 constitute a 3D road map, generated as part of the SLAM process. Both aspects will be described in detail later.
[0278] The remaining reference parameters are calculated by the parameter calculation component 408 via image processing, as described below. First, some basic considerations are briefly introduced.
[0279] Reference Figure 8 The 3D orientation of the training vehicle 300 can be defined in the same way as the camera. Here, the pitch, roll, and roll angles of the vehicle are labeled P, R, and Y, respectively, while the lateral, longitudinal, and vertical axes (vehicle axes) of the training vehicle are represented by unit vectors a and b. T , a L , a VThe longitudinal axis corresponds to the direction of the vehicle when it is traveling in a straight line (the forward direction). The vertical axis is the direction orthogonal to the road surface (i.e., the plane where the contact points between the wheels and the road are located). The horizontal axis extends along the width of the vehicle 300 and is perpendicular to the longitudinal axis and the vertical axis.
[0280] It should be noted that the term "absolute" in this document is used to refer to the orientation and position within the environment in which the vehicle is traveling, unless otherwise noted, i.e. as defined by an (arbitrary) global coordinate system relative to the global reference frame of the environment in which the vehicle is moving. This can be defined as an inherent part of the SLAM process. This is the coordinate system that defines the reconstructed vehicle path, the orientation of the image capture device within the environment (absolute orientation) and the surface reconstruction.
[0281] The term "relative" is used to refer to a position and orientation relative to the vehicle, i.e., relative to a coordinate system of the vehicle (such as a coordinate system defined by the vehicle axes). T , a L , a V ) defined position and orientation.
[0282] Reference Fig. 9 , this method does not require that the orientation of camera 302 be perfectly aligned with the orientation of training vehicle 300. Angular misalignment of the camera axis relative to the vehicle axis is acceptable because such misalignment can be detected and accounted for using the techniques described below.
[0283] 4) Using the fact that the motion vector of the car will be parallel to the body of the car when it is moving straight, the angular orientation of the camera inside the car can be automatically derived, i.e., the longitudinal axis of the camera a L 'With the longitudinal axis of the vehicle a L (ie, the longitudinal axis of the vehicle as measured in the reference frame of the image capture device, ie, in the reference frame in which the orientation of the image capture device remains fixed) L ), so the camera orientation relative to the car body can be calculated. This orientation is captured as the camera's forward point, defined as the pixel that the car appears to be heading towards when driving straight ahead (as described above).
[0284] It is not possible to rely on those parts of the video where the vehicle is driving perfectly straight. To overcome the problem that the vehicle may never be driving perfectly straight, the following approach is used.
[0285] Pick three equidistant points on the path, connect the two outer equidistant points, and use that vector as an approximation of the positive direction at the center point. This method is weighted by the dot product of the vectors between the center point and the two outer points continuously. Therefore, the time when the car is nearly going straight has a higher weight than the time when the car is turning. Using this method can produce sub-pixel accuracy.
[0286] exist Fig.15 This is explained in .
[0287] In other words, this exploits the observation that when the vehicle 300 is traveling in a (nearly) straight line, the vector difference x t+1 -x t-1 (approximately) parallel to the longitudinal axis of the vehicle a L , in order to estimate the angular offset of the camera 302 relative to the vehicle 300. This can be achieved using a weighted average, by assigning higher weights to intervals of more linear (straight) motion.
[0288] It should be noted that strictly speaking, as long as the turning rate is approximately constant (which is not actually required for the vehicle to travel in a straight line), the vector difference x t+1 -x t-1 However, as the vehicle approaches a straight line, the assumption of an approximately constant turn rate becomes more likely to hold.
[0289] This calculation is performed as a vector average, where the vector represents the average forward motion of the car expressed in the camera coordinate system. When projected forward, this vector will slice the image plane at a specific pixel (the forward point pixel). This pixel is the pixel that the car would appear to be heading towards if it were driving on a straight and flat road (the term "forward point" is used here).
[0290] For example, the heading of the vehicle may be calculated as a weighted average of the vector differences between the camera and the longitudinal axis of the vehicle, respectively, according to Equation 1:
[0291]
[0292] where w L,t =v0·v1, as estimated at time t.
[0293] Assume that the longitudinal axis of the vehicle is L At time t, parallel to the vector difference x t+1 -x t-1 .
[0294] It should be noted that in this regard, it is necessary to estimate x in the reference frame of the image capture device 302 t+1 -x t-1 , in order to give a meaningful result, that is, to make the weighted average of equation (1) provide the vehicle longitudinal axis a L Relative to the camera's longitudinal axis a' L Accurate estimation of direction.
[0295] The SLAM process will deduce x t+1 and x t-1 However, the direction of the camera axis at time t in the global coordinate system (i.e., the absolute orientation of the image capture device) can also be derived through the SLAM process itself, which in turn can be used to transform the vector in the global reference frame into the reference frame of the image capture device 302.
[0296] The dot product v0·v1 depends on the magnitude of the change in direction between t-1 and t+1, so that the smaller the magnitude of the change in direction, the higher the weight given to the measurement at time t.
[0297] As mentioned above, strictly speaking, as long as the turning rate is approximately constant (which is not actually required for the vehicle to move in a straight line), the vector difference x t+1 -x t-1 Thus, an alternative weighting scheme may be used in which the vector difference at each time is weighted according to the change in curvature of the vehicle path between t-1 and t+1.
[0298] Of course, typically as the vehicle moves, the orientation of the vehicle axis remains fixed in the image capture device's reference frame because the camera 302 is expected to remain in a substantially fixed orientation relative to the vehicle 300 - the above-mentioned time dependency arises from inaccuracies in the estimates of the vehicle and camera axis.
[0299] The time t corresponds to a point on the camera path CP and may be the time when one of the images was captured or the time between captured images, to which the camera pose may be assigned by interpolating the camera poses of different images.
[0300] In an alternative embodiment, a L with a L ' can be expressed as a pair of angles.
[0301] Reference Fig.10 Once the angular orientation of the camera 302 relative to the vehicle 300 has been calculated in this way, the vehicle longitudinal axis a can be determined. L Orientation at any point along the camera path CP (including when the vehicle is not traveling in a straight line), based on the camera longitudinal axis a' known from SLAM calculations L Orientation at this point.
[0302] 5) The rotational orientation of the camera within the vehicle can be automatically inferred by noting that most roads are locally flat, so that when the car turns a corner, the vector perpendicular to the surface described by the camera path will point directly from the camera to the road surface - that is, the camera vertical axis a V 'With the vehicle vertical axis a V (i.e. the vehicle vertical axis a as measured in the reference frame of the image capture device V ). In more detail, again take three equidistant points on the path (but it is still not necessary for them to be equidistant), take the cross product of the two vectors from the center point to the two outer points, and weight the concatenated average by the cross product magnitude. Thus, the time the car is driving around a corner is weighted more heavily than the time the car is driving in a straight line (where there is no information about the normal).
[0303] exist Fig.16 This is explained in .
[0304] In other words, this exploits the observation that when a vehicle exhibits angular acceleration such that the vehicle path exhibits local curvature in 3D space, the plane in which the curved portion of the vehicle path lies is at least approximately parallel to the plane of the road surface under normal driving conditions; or equivalently, the normal to the local curved portion of the path is at least approximately parallel to the vertical axis a of the vehicle 300. V .
[0305] The average normal vector of the path is expressed in camera coordinates. A plane perpendicular to this road surface normal vector can be cut through the image plane, providing a line on the image that matches the horizon (the term "eye level" is used herein) if the car is traveling along a straight, level road.
[0306] For example, the rotational orientation of the camera within the vehicle may be calculated as a weighted average of the offset angles between the vertical axis of the camera and the vertical axis of the vehicle according to equation (2):
[0307]
[0308] in:
[0309] ·w V,t =|v0×v1|, as estimated at time t,
[0310] v0 = x t -x t-1 ,
[0311] v1 = x t+1 -x t .
[0312] Assume that the vehicle vertical axis a V Parallel to the vector cross product v0×v1 at time t.
[0313] As mentioned above, it is necessary to estimate v0×v1 in the reference frame of the image capture device 302 in order to give a meaningful result, that is, to enable the weighted average of the equation to provide the vehicle vertical axis a V Relative to the camera's longitudinal axis a' L Accurate Estimate of Orientation. As described above, the absolute orientation of the image capture device determined by the SLAM process can be used to transform vectors from the global reference frame into the image capture device's reference frame.
[0314] The magnitude of the cross product |v0×v1| increases as the component vectors move toward the perpendicular direction, since this is expected to produce the most accurate results. More generally, their weighting factors increase as the offset angle α between these vectors approaches 90 degrees, where |v0×v1|=|v0||v1|sin α.
[0315] It should be appreciated that many of the observations made above with respect to angular orientation measurements also apply to rotational orientation measurements.
[0316] 6) The height (H) of the camera above the road can be calculated by generating a road surface grid from the SLAM process (even if this is of low accuracy), and then averaging the height of all points on the camera path above this grid. Averaging over the entire path offsets the effect of the poor accuracy of the grid generated by the SLAM process.
[0317] The height H of a given point on the camera path CP is defined as the height of the point on the camera path CP and the height of the point along the vehicle vertical axis a V The reason for requiring rotational orientation of the camera 302 within the vehicle 300 is to determine the position of the vehicle's vertical axis at each point on the path so that a height estimate can be calculated. Fig.11 This is illustrated in , where the reconstructed road surface map is marked as SM.
[0318] 7) The left and right positions of the camera within the lane (S) and the width of the lane (W) can be estimated by detecting the lane boundaries using various computer vision techniques.
[0319] Detecting lane edges is obviously a hard problem (the whole point of generating this lane training data is to train a neural network to solve it). However, in this particular case, detecting lane edges becomes significantly easier thanks to two additional pieces of information. First, the known camera path (and lane shape) can be used to create a transformation that effectively straightens the road before any lane boundary detection is done. Second, the centerline of the lane is (almost) known from the camera path, and the width of the lane is relatively fixed, so the number of locations to search for lane boundaries is greatly reduced.
[0320] For example, the image can be projected into a top view, transformed to straighten the road (using the camera path information), and then a Hough transform can be used to detect the main wires with the correct approximate position and orientation.
[0321] The road width at each point on the camera path CP is defined as T The reason why the angular orientation of the camera 302 within the training vehicle 300 is required in addition to the rotational orientation is to determine the vehicle's lateral axis a T The location of , in order to estimate W and C at each point along the path. Fig.11 This is also explained in.
[0322] By utilizing the above, annotations of the car's driving lane can be fully automatically generated in all images of a video using only images captured from a low-cost and uncalibrated image capture device.
[0323] It is also desirable to extend these techniques to annotate multiple lanes. One approach is as follows:
[0324] 8) Multiple lanes (and the entire road surface) can be automatically annotated by driving along each lane and creating a single 3D SLAM reconstruction containing the path of each lane. In order to obtain the relative positioning of the lanes and the entire road surface, it is necessary to merge the separate videos of driving along each route into a single reconstruction.
[0325] Fig.17 An image showing a SLAM reconstruction of a camera position moving through a point cloud world, where there are two sets of camera images.
[0326] 9) It is difficult to merge images from multiple videos into a single 3D reconstruction unless the camera images are all facing the same direction; i.e., it is not practical to merge videos of a car driving in the opposite direction of the road into a single 3D reconstruction. One way to solve this problem is to have the forward-facing camera driving in one direction and the rear-facing camera driving in the other direction (in the other lane). This gives a set of images that are all pointing in the same direction along the road.
[0327] Extension-artificial fixing stage:
[0328] The fully automatic annotation system described in this article produces usable annotations, but it still contains many errors that can be easily fixed by human annotators. For example:
[0329] The calculated camera height may not be completely correct;
[0330] The left / right position within the lane may not be completely correct;
[0331] Lane width may not be exactly correct;
[0332] In addition to these fixes for imperfect automatic calculations, there are additional annotations that can be added manually that require minimal effort to add, but can provide significant advantages if they are present in the training data. For example:
[0333] Adjacent lanes can be added (you don't have to drive on them). These are easy to add because they generally have parallel boundaries to the lane the car is driving on.
[0334] Non-drivable parallel road features can be added. Areas of the road reserved for bicycles or buses / taxis can be added simply (again, these are generally just parallel areas on the ground that follow the driving lane).
[0335] · Sidewalk areas adjacent to roads can be added.
[0336] 10) A tool with a simple user interface has been developed that allows a human annotator to make these “repairs” and add these additional annotations. The main advantage of this tool is that the repairs and additional annotations can be made in one frame of the video, and the system automatically propagates these changes to all frames in the video.
[0337] This significant saving stems from the fact that the user can use one frame and its annotation data as a reference to adapt the 3D road model (via the model adaptation component 410), and in turn can apply the adapted 3D road model to multiple images.
[0338] It should also be noted that this technique is not limited to manually adjusting the automatically generated road / lane models. Any stationary object in the world can be marked and its position in each image of the video can be calculated (since the motion of the camera is known).
[0339] In this way, one frame of video can be provided to a person already pre-populated with nearly correct annotations, and with minimal inpainting approximately 50 to 100 well-annotated images can be generated. If longer video clips are used, more images may be generated, but as a rule of thumb, 50 to 100 images are sufficient length for easy 3D reconstruction and reasonably similar in construction.
[0340] 11) It should be noted that the images used as training data are sampled from the entire video, so they are reasonably spaced apart in space (not time), providing the network with some diverse training data. Taking every image from the video would not be very helpful, as most of them are extremely similar to each other.
[0341] Figure 6 Schematic block diagram showing examples of the types of adjustments that may be performed during the manual fixing phase. The presentation component may present the data of any captured image on the display of the user interface 412, overlaid with its associated annotation data A(n). One or more selectable options 600 are also presented, which the user may select to cause the model adaptation component 410 to adapt the 3D road model 602 in a quick and intuitive manner. Figure 6 As shown, as the road model 602 is updated, not only the annotation data A(n) currently being presented is modified according to the adapted model 602, but the adapted model 602 can also be used to update the annotation data 604 for other images in the same video sequence. Figure 6The top part shows a road or lane marked within the image. While the right boundary R2 coincides exactly with the real-world right boundary visible in the image, the left boundary R1 is slightly off. The user can quickly fix this by displacing the position of the left boundary in the 3D model, using the current image and the annotation data presented on the current image as a reference to manually adapt the position of the left boundary R2 until it corresponds to the real-world left boundary visible in the image. This change will be applied across multiple images in the sequence, meaning that the left road boundary will be accurately placed in these images as long as the road / lane remains the same width.
[0342] Utilizes a number of target road structures that are parallel to the vehicle path, providing a range of extremely fast manual annotation options:
[0343] - The width or position of the automatically determined road structure can be easily changed ("nudged");
[0344] - Additional road structures can be easily created parallel to the vehicle path / automatically determined road, and their width and position can be easily nudged in the same way. Additional road structures are automatically oriented parallel to the vehicle path / existing road structures. For example, an automatically determined road structure could be the "own lane", i.e. the lane the vehicle is driving in. It is easy to add additional structures on multiple images, such as:
[0345] o Other lanes (e.g. non-own lane, bus lane, bicycle lane), central road markings or left and right road markings or any other parallel road markings, non-drivable areas such as sidewalks, lawns, obstacles, roadside vegetation (trees, hedges, etc.).
[0346] -Vertical road structures, i.e. road structures that extend perpendicular to the automatically determined road structure / vehicle path, can also be added very easily, and their width and position can be nudged in the same way. Examples include things like junctions, yield lines or other perpendicular road signs, or other structures at an assumed angle to the path (e.g. a fork in the road).
[0347] - Changes in road width, e.g. due to a change in the number of lanes, can be accommodated by allowing the user to "split" the video sequence at a point of his / her choosing. After the video split point, the user's adaptation will only be applied to the 3D road model at some point after the split point - the part of the model before the split point will remain unchanged, which in turn means that the image annotation data before the split point remains unchanged.
[0348] As an example, consider the following workflow. The user starts at the beginning of the training sequence. The 3D ego lane has been automatically determined using the techniques described above. In addition, the positions of the center lane marker and left / right lane markers have been determined by assuming they have a certain width and are located in the center and rightmost / leftmost of the ego lane. At this point, the user can adapt the width and position of the ego lane and, if necessary, the width and position of the road signs so that they align with the actual ego lane / road signs visible in the image. The user can also add additional parallel structures at this point, such as non-ego lanes, non-drivable areas, etc.
[0349] Fig.18 An example of a pre-inpainted automatically annotated image is shown in , where only the ego lane (and its assumed centerline and lane boundaries) are marked. As can be seen, the markings do not completely correspond to the actual lane boundaries. Fig.19 Shown is the restored image, where the width and position of the own lane have been adjusted, and other lanes and non-drivable areas have been marked.
[0350] These changes are applied to the base 3D model and therefore to subsequent images in the sequence. The user can then cycle through the subsequent images to quickly verify that they all match.
[0351] When the user reaches a point in the video where the intersection is clearly visible in one of the images, he / she can add the intersection structure at that point and nudge its position / width if necessary. Again, this will be applied to the 3D model and therefore to all images where the intersection is visible. Non-drivable road structures at the edge of the road will automatically fit to the parallel intersection (i.e. any part of the non-drivable road structure that overlaps the new intersection will be automatically eliminated).
[0352] Fig. 22 This shows an example of the UI when an intersection is just added. The user has defined the intersection as perpendicular to the lanes to divide the left and right lanes.
[0353] Reference Fig.12 , when the user reaches a point in the video where the road / lane width changes (e.g. lane narrowing / widening, number of lanes changing or decreasing, etc.), s / he can split the video at that point and use that frame as a reference to modify the base model ( Fig.12 , step 1). The position of the current video frame is mapped to a point on the vehicle's path, which in turn is used to restrict the fit of the 3D model to only the portion after that point - points before that (where the user has verified that the annotations match the image) are not affected. At this point, the user can change the road width, add lanes, etc. The system is able to automatically interpolate parallel structures around the split point, such as linear interpolation. For example, when adding a lane, the road will typically widen gradually until it can accommodate the new lane, and the user can continue the video until the point where the new lane reaches its full width, and then add the lane at that point ( Fig.12 , step 2). The system automatically interpolates between intermediate regions where the width of the new lane is assumed to increase linearly until its final width (e.g. Fig.12 , step 3).
[0354] Fig.23 and Fig.24 An example thereof is shown. Fig.23 In the Fig. 22 Option to split the video shortly after the intersection in , to accommodate the fact that there is a wider non-drivable road area on the right at this point, Fig.24 , which has been annotated by the user.
[0355] As an extension, users can also mark sky areas in the images. This could be a simple case where the user defines a skyline above which all images must be sky. Even though this does not label all of the sky in the image, it can still be used to prevent a trained neural network from identifying sky as road structures. More sophisticated techniques can use 3D road structures to infer the skyline in each image, for example based on the highest point of the road in 3D space.
[0356] Extensions beyond lane training data:
[0357] Any features in the video that are parallel to the path followed can benefit from similarly accelerated annotation and single-click addition by human annotators.
[0358] 12) For example, road signs can be annotated in this way. Both the location and type are the same, including the center dashed or solid line, the road edge, double yellow lines, etc. For example, a bicycle lane can be added in this way, and a pedestrian path can also be added.
[0359] Fig. 20 An example image of a marked bicycle lane is shown on the left.
[0360] Annotation Object
[0361] A brief review Figure 4A In addition to the components described above, an object annotation component 420 is shown. This represents the object annotation functionality of the image processing system, which allows objects that do not belong to the road to be efficiently annotated, based on the certainty of the relationship between such objects and the 3D road model.
[0362] An object annotation component 420 is shown having inputs connected to the 3D road modeling component 404 and the UI 412 , respectively, and having outputs for providing object annotation data, which is both rendered by the rendering component 416 and stored in the electronic storage 414 .
[0363] The functionality of the object annotation component 420 will now be described.
[0364] Static Objects
[0365] Any feature in the video that is stationary can benefit from some degree of speedup. Even though these features may not be directly creatable because their shapes are parallel to the driving path, they can still benefit from the fact that one can label them in one frame of the video and automatically propagate them to all frames of the video.
[0366] 13) For example, a road sign can be annotated in this way: it can be specifically marked by humans in one frame (with or without any initial hints provided by low-quality detection), and then automatically propagated to the entire set of images in the video.
[0367] Annotations for stationary vehicles can be generated in the same way. Although this technique does not automatically allow annotation of moving objects to be accelerated in the same way, it can be used for stationary vehicles. Thus, it allows a subset of vehicles in the scene to be annotated with minimal effort. Moving vehicles can be annotated by marking the same object in multiple (at least two) images and assuming a constant velocity between annotations. The annotations can then be propagated to all images between the annotated images.
[0368] 14) It should be noted that a stationary vehicle looks the same as a moving vehicle, so a neural network trained on annotated images of stationary vehicles can be used to detect moving vehicles at runtime.
[0369] Fig.21 An image showing a 3D bounding box of a vehicle being added.
[0370] Another advantage of this technique is that when the camera is close to an object in a video image (for example, when the object is large), a person can choose to annotate the object. This accurate annotation is then propagated to all images, even if the object is small within the image.
[0371] 15) Since the 3D model is based on the entire sequence of images in the video, the annotations automatically generated when the object is farther away from the camera will have the same accuracy as when the object is larger and annotated by a human. This will usually be higher than what can be achieved by a human trying to annotate images of smaller objects.
[0372] Fig.21 Consider static objects. Extension of the technique to moving objects (ie, moving relative to the road) will now be described.
[0373] Annotate moving objects
[0374] An extension of the above techniques to moving objects (ie, moving relative to the road, so that, unlike static objects, moving objects do not maintain a fixed position relative to the road) will now be described.
[0375] Using the 3D vehicle path and certain reference parameters that can be derived from the image itself, a 3D road model is determined. The road is modeled as a 2D manifold embedded in 3D space. By projecting the road model geometry into the image plane of the captured images, those images can be effectively annotated with 2D road structure annotations.
[0376] The fast data annotation software allows to reconstruct a 3D model of the ego vehicle's path, from which the path of the road can be inferred. The basic idea is that the camera and the vehicle almost always follow the lane, so the vehicle trajectory implies possible lane paths in 3D space. A lane structure that already follows the shape of the road can be automatically generated, thus allowing to very efficiently annotate a set of parallel lanes of a road.
[0377] In some embodiments, a certain degree of manual annotation input is provided for a few 2D images via a user interface (UI) and used together with the 3D road model to automatically extrapolate these inputs to other images.
[0378] The structure of the road, such as individual lanes, is also modeled. Lanes can be effectively added to the model via the UI assuming that they run parallel to the ego vehicle path VP.
[0379] Assuming that the shape of the reconstructed vehicle path corresponds to the shape of the road and / or lane on which the vehicle is traveling (collectively referred to as the road shape), the reconstructed vehicle path VP is used to determine the 3D road model.
[0380] Annotate moving objects
[0381] The functionality of the above image processing system can be extended in the current context to provide effective annotation of other vehicles moving on the same road. These images can then be used, for example, to train object detectors. The process is not limited to vehicles and can be applied to any object that is assumed to move in a way that follows the shape of the road, such as pedestrians walking along the sidewalk next to their own road.
[0382] In addition to understanding road lanes, autonomous vehicles also need to identify other road users to avoid collisions. Training data for this task can be generated by annotating other vehicles, cyclists, and pedestrians in videos captured from cars.
[0383] like Fig.21 As shown, it is relatively simple to annotate static objects (such as parked cars) because once the correct position, orientation, and range are established in the 3D model (by annotating the object for one frame), the annotation is valid for all frames in which the object is visible. However, vehicles moving along the road have different 3D spatial positions in each frame, which would require manual annotation of each vehicle in each frame.
[0384] Building on the knowledge that the ego vehicle path implies the road path, this is now flipped in the opposite direction to establish the assumption that the inferred road path is a good prediction of the likely paths of other vehicles. Once the annotator has annotated a moving vehicle at a single point in time, they can mark it as “coming soon” or “heading ahead,” and the system will then animate it along the road trajectory at a constant speed. To correct for speed, the animation is moved to an earlier or later frame, and the annotation box is repositioned to coincide with the target vehicle’s position at that point in time. The average speed between the two points is then derived; additional positions can be marked when the target vehicle significantly speeds up or slows down within the field of view. Similarly, small lateral deviations can be annotated by slightly repositioning the annotation, such as when the target vehicle driver takes a slightly different route on a stretch of road, and the system will interpolate between the different given offsets along the road route.
[0385] For many common situations of passing and oncoming traffic, the system is able to accurately annotate other vehicles quickly and easily.
[0386] Fig.29 A flow chart showing a method for automatically generating annotation data for a moving object implemented in the above image processing system. Fig.30 Shows how to apply Fig.29 A graphic illustration of the steps in Fig.30 The numbered circles are used to represent Fig.29 The corresponding relationship of the steps in .
[0387] In step 1, a 3D road structure model is determined for a time series S of captured images using the above-described image processing method. As shown in the figure, the road is modeled as a 2D manifold (surface) in 3D space. This is a 3D reconstruction based on the own vehicle path VP. The process also estimates the heading f (the direction the own vehicle is traveling at this time) and the road surface normal n (perpendicular to the road surface - vertical here) at different points along the reconstructed vehicle path VP. They define the direction r = f × n perpendicular to the heading f and the road surface normal n (lateral).
[0388] Fig.30 Three images IMG1 , IMG2 and IMG3 of the time series S are shown, but the time series may contain any number of images. The time series S may be formed, for example, by frames of a short video sequence captured by a camera of the host vehicle.
[0389] In step 2, at least one image is manually annotated to mark the location of the moving objects in the image.
[0390] exist Fig.30In the example of , IMG1 is manually annotated via the UI to define a 2D bounding box B1 around a moving object visible in IMG1 (which is another vehicle).
[0391] Under the assumption that the mobile object travels along a path OP parallel to the reconstructed ego vehicle path VP, it is possible to estimate the mobile object path OP from only the manually annotated bounding box B1 and the reconstructed ego vehicle path VP (step 3).
[0392] This can be formulated assuming that the moving object's path OP maintains a fixed lateral offset (ie, in the r direction) and a fixed vertical offset (ie, in the road normal n direction) from the own vehicle path VP (elevation from the road surface).
[0393] exist Fig.30 In the example of , two 3D paths OP1 and OP2 are determined for the other vehicle, which are the paths tracked by reference points C1, C2 on the bottom edge of the bounding box B1 - in this case, at the two lower corners of the bounding box B1. It is assumed that the reference points C1 and C2 are located in the road plane, that is, the vertical offset with respect to the ego vehicle path VP is zero. Therefore, the object paths OP1 and OP2 can be uniquely defined as paths in 3D space, which have a fixed lateral offset and zero vertical offset with respect to the ego vehicle path VP, and intersect with the reference points C1, C2 of the manually annotated bounding box B1 when they are geometrically projected back to the image plane of IMG1. The initial positions of C1 and C2 in 3D space are also referred to as the points in 3D space corresponding to those points in the 2D image plane of IMG1 where C1 and C2 intersect OP1 and OP2 respectively.
[0394] It should be appreciated that the lower corner of the bounding box is only one example of a suitable reference point at the height of the road, and other suitable reference points may be used instead. For example, it may be assumed that the upper corner position of the bounding box B1 of IMG1 is located vertically above C1 and C2. In general, by utilizing the geometry of the 3D road model, the manually annotated bounding box B1 may be mapped from the 2D image plane of IMG1 to any suitable representation in 3D space.
[0395] As another example, another way to establish the initial position in 3D space is as follows. The animation starts with a 2D annotation, which is projected into 3D space to create a 3D annotation feature that represents the moving object, in this case a cuboid (3D bounding box) in 3D space, but it can take a different geometry. Fig.33 This is illustrated in FIG. 8 , where the 3D annotation element is labeled with reference numeral 800 .
[0396] Alternatively, the 3D annotation elements may be placed at multiple locations in 3D space and projected back into the image plane of the first frame.
[0397] The cuboid 800 is located at an initial position in 3D space under the assumption that the 3D element is at a predetermined elevation from the road surface. For example, the four lowest corners of the cuboid may be assumed to be at zero spacing from the defined road surface in the direction of the road surface normal n. The road surface is defined to be laterally flat (r) at each point along the reconstructed vehicle path VP (but it is not assumed to be flat overall - in the forward direction f, it is defined to follow the shape of the vehicle path VP). The annotator then provides manual annotation input via UI 412, in response to which the cuboid may be repositioned but maintaining this fixed elevation from the road surface (fixed vertical elevation from the vehicle path VP). The cuboid is projected back into the plane of the current image IMG1 so that the annotator can see when the cuboid is correctly placed. By imposing geometric constraints at a fixed elevation from the road surface, annotation elements can be accurately placed in 3D space using only a 2D view. The annotator can also edit the size, shape, etc. of the annotation element in 3D space to manually match it to the 2D image.
[0398] For example, for a moving vehicle, it can be assumed that the moving vehicle always stays on the road surface when moving along the path, for example, the vertical spacing between one or more predetermined reference points of the object annotation element (for example, the four bottom corners of a cuboid) and the road surface is zero or essentially zero.
[0399] After determining the bounding box representation of IMG1 in 3D space, the knowledge of the assumed 3D path of the moving object can be used to represent other images in 3D space (e.g., Fig.33 By moving the 3D representation of the bounding box (e.g., cuboid) 800 to its assumed position relative to the capture position of another image IMG2, annotation data can be automatically generated for this image IMG2 by geometrically projecting the moved representation 800 into its image plane.
[0400] At step 4, the path information determined at step 3 is used to automatically annotate additional images in the sequence. To annotate an additional image (e.g., IMG2 or IMG3), it is assumed that the reference points C1 and C2 are moved a certain distance along the paths OP1 and OP2, respectively, relative to their initial positions known from the manual annotation of IMG1. This provides the positions of those reference points C2, C3 relative to the capture positions of the additional image, which in turn allows the automatic generation of a bounding box (e.g., OP1) by geometrically projecting the reference points C1, C2 back to the image plane of the additional image (e.g., IMG2, IMG3). Fig.30 B2 or B3 of IMG2 and IMG3 in the .
[0401] according to Fig.30 In the example in , it is assumed that reference points C1 and C2 remain on object paths OP1 and OP2 respectively, and the only unknown is the distance traveled along these paths.
[0402] In the above alternative example, assuming that the cuboid is placed at a fixed elevation from the road surface in 3D space, a single expected path for the cuboid can be defined in 3D space in exactly the same manner relative to the cuboid's path reference points (e.g., center points, corners, etc.). The path reference points are moved along the expected 3D path in the same manner to determine the new position of the cuboid. Determining the expected path is based on the following assumptions: (i) following the expected shape of the road (as derived from the vehicle path VP), and (ii) maintaining a predetermined elevation from the road surface (e.g., the four lowest corners always remain at the road height, or applicable reference points maintain a predetermined elevation from the road surface). It should be appreciated that this operates on the same basic principle, and all descriptions of C1 or C2 on OP1 and OP2 herein are equally applicable to reference points, such as reference points for 3D annotation elements (such as established cuboids) and 3D definitions of their paths. In that case, once the cuboid is moved to the relevant expected position using the above assumptions, additional annotation data is automatically generated by projecting the cuboid geometry into an image plane such as IMG2 or IMG3.
[0403] In the case of manually annotating only a single image, it is assumed that the other vehicles are moving at the same speed as the ego vehicle. Therefore, in this case, when annotating additional images (e.g., IMG2 or IMG3), if the ego vehicle has moved a distance D along the ego vehicle path VP relative to the capture position of IMG1 V , assuming that the other vehicles have also moved a distance D along their path V , so the reference points C1 and C2 have moved a distance D along the paths OP1 and OP2 respectively V .
[0404] When at least two images in the sequence S are manually annotated, this allows the speed of other vehicles to be estimated via the speed estimation component 422 of the object annotation component 420. This can be a simple linear extrapolation, where if it is determined that the other vehicle has moved a distance D between two manually annotated images at two separated image capture times T, O , it can be assumed that the reference points C1 and C2 have traveled a certain distance along OP1 and OP2 respectively between the two interval image capture times T'
[0405] This can be refined via UI 412. For example, when a user first annotates an image with an object bounding box, the system will initially estimate the speed of other vehicles to match the speed of the ego vehicle.
[0406] When the user moves to the next (or following) image in the sequence, that image will be automatically annotated with the assumption that the other vehicles are traveling at the same speed as this vehicle. If that seems correct, the user can simply continue through the sequence. However, if the annotation is wrong, the user can manually adjust it. In this way, the user has now accurately annotated two images, which in turn allows the system to refine the speed estimate of the other vehicles. When the user moves again, the next image will automatically be annotated with the refined speed estimate and will therefore be of acceptable accuracy (unless the speed of the other vehicles has changed significantly).
[0407] Regardless, the knowledge of the road layout learned from the 3D road model can be used to infer the direction of travel. For example, when another vehicle is determined to be in the lane marked as the oncoming lane of the vehicle, it can be assumed that the vehicle is traveling in the opposite direction of the host vehicle.
[0408] As another example, a 3D annotation element can be a 3D model of a moving object (not a cube, more accurate than a 3D bounding box). The advantage of a 3D model is that it can provide more accurate 2D annotation data, such as a tighter fitting 3D bounding box, when projected back to the image location. The 3D model can also be used to generate 2D annotation data in the form of a segmentation mask, the purpose of which is to essentially track the outline of the object in the image plane. This can be used, for example, for instance segmentation training.
[0409] Verify the image
[0410] Another extension to the above approach comes from the realization that localizing objects with respect to the surrounding road structure provides a useful layer of information for validating trained object detectors. Validation is the process by which a trained model is able to perform with acceptable accuracy on images not encountered during training, and is therefore a structured test of the model's ability to generalize.
[0411] In the context of autonomous driving, there are special cases where accurate object detection is critical, and there are other cases where object detection, while potentially useful, is not critical. A simple example is that it is critical for an AV to be able to accurately detect and localize other vehicles on the road, while it is less important to detect a parked vehicle on the side of the road alone.
[0412] As described above, as part of the above-described image processing system, a road model is created with a detailed road structure, which may include, for example, individual lanes, parking areas on the side of the road, non-drivable areas (sidewalks, bus lanes, etc.). As described above, when object annotation is performed using this model as a basis, the result is that for any given image, the location of the annotated object in relation to the road structure can be directly inferred, for example, whether it is in the ego lane (the lane in which the ego vehicle is traveling), or in a different lane, or in a parking area, etc.
[0413] This in turn allows the images to be labeled or otherwise classified with such information, i.e., labels or other classification data can be associated with the images to provide information about what objects are contained therein and, importantly, how those objects relate to the surrounding road structure. This in turn allows specific types of images to be localized for validation purposes. In particular, in images that capture background where accurate object detection is critical due to the way other vehicles are positioned relative to the surrounding road structure, these images can be identified and used for validation purposes to ensure that the object detection model is able to achieve acceptable results on such critical images.
[0414] User interface extensions
[0415] Several user interface features are also provided to facilitate rapid annotation of static and dynamic objects, which will now be described.
[0416] The above image processing only considers 2D images and annotates them using 3D road models, but the present technique can be extended to 3D structural data. For example, 3D structural data can be obtained from stereo image pairs, LiDAR, RADAR, etc. or a combination of multiple such sensor modalities.
[0417] In the context of the above image processing examples, the frame takes the form of a 2D image. By extension to 3D structure data, the term "frame" also encompasses 3D frames, and Fig.29 The steps in can also be applied to 3D frames.
[0418] In this 3D context, a 3D frame can be any captured representation of a 3D structure, i.e., it includes capture points (3D structure points) that define the structure of the 3D space, and provides a static "snapshot" of the 3D structure captured in the frame (i.e., a static 3D scene). The frame can be said to correspond to a single moment in time, but this does not necessarily imply that the frame needs to be captured immediately or the underlying sensor data from which the frame is derived - for example, LiDAR measurements can be captured in a LiDAR scan by a mobile object in an "untwisted" manner for a short period of time (e.g., about 100ms) to account for any motion of the mobile object, thereby forming a single point cloud. In that case, despite the manner in which the underlying sensor data is captured, the single point cloud can still be said to correspond to a single moment in time in the sense of providing a useful static snapshot due to this untwisting. In the context of a time series of frames, the moment in time to which each frame corresponds is the time index (timestamp) of the frame within the time series (and each frame in the time series corresponds to a different moment in time).
[0419] In the following example, each frame is in the form of an RGBD (Red Green Blue Depth) image captured at a particular moment in time. An RGBD image has four channels, three of which (RGB) are color channels (color components) encoding a "regular" image, and the fourth channel is a depth channel (depth component) encoding the depth values of at least some pixels of the image. RGB is used as an example, but the description is more generally applicable to any image with color components and depth components (or indeed images with only depth components). In general, the color components of an image can be encoded in any appropriate color space using one or more color channels (including grayscale / monochrome). The point cloud computing component 302 converts each frame into a point cloud form to allow annotation of the frame in 3D space. More generally, a frame corresponds to a particular moment in time and can reference any data set (such as multiple RGBD images, one or more point clouds, etc.) that has captured a static "snapshot" structure (i.e., a static 3D scene) for that moment in time. Therefore, all descriptions below about RGBD images are equally applicable to frames of other forms. In the case where a frame is received at the annotation system 300 in the form of a point cloud, no point cloud conversion is required. Although the following examples are described with reference to point clouds derived from RGBD images, the annotation system can be applied to point clouds of any modality, such as monocular depth, stereo depth, LiDAR, radar, etc. Point clouds can also be derived from two or more such sensing modalities and / or from multiple sensor components of the same or different modalities by merging the outputs of different sensors. Therefore, the term "point cloud of a frame" can refer to any form of point cloud corresponding to a particular moment in time, including a frame received in the form of a point cloud at the annotation computer system 300, a point cloud derived from a frame by the point cloud computing component 302 (e.g., in the form of one or more RGBD images), or a merged point cloud.
[0420] As described above, although a frame corresponds to a specific moment in time, the underlying data used to derive the frame may be captured at (usually shorter) time intervals and transformed if necessary to account for temporal changes. Therefore, a frame corresponding to a specific moment in time (e.g., represented by a timestamp) does not necessarily imply that the underlying data has all been captured at the same time. Therefore, the term "frame" includes point clouds received at timestamps different from the frame, for example, a lidar scan captured within 100ms is "untwisted" into a single point cloud at a specific moment in time (such as the time when the image was captured). The time sequence of frames 301 may also be referred to as a video clip (it should be noted that the frames of a video clip do not have to be images, and can, for example, be point clouds).
[0421] Fig.31 and FIG. 32A to FIG. 32E Shown can be used in Fig.29 Schematic diagram of an example annotation interface 300 for effectively placing an initial 3D bounding box in step 3 of the method of FIG. That is, based on the initial two bounding boxes placed around the moving object, the bounding boxes of other frames can be extrapolated based on the following assumptions: for example, (i) a constant (or otherwise known, such as assumed or measured) speed, and (ii) the moving object follows the road shape (as described above, if the speed of the external vehicle can be assumed to match the speed of the own vehicle, it is sufficient to manually place a single 3D bounding box).
[0422] Fig.31 A schematic diagram of an annotation interface 700 that may be presented via UI 412 is shown.
[0423] In the annotation interface 700, the color components of the RGBD image 702 (current frame) are displayed on the left, and a top view 704 of the point cloud 400 of the frame is displayed on the right.
[0424] In addition, a projection 706a of the 3D road model into the image plane of the RGBD image is superimposed on the displayed image 702. Likewise, a projection 706b of the 3D road model into the top view is displayed as superimposed on the top view of the point cloud 400.
[0425] A selectable option 708 is provided for creating a new 3D bounding box for the current frame. Once created, selectable options 710 and 712 are provided for moving the bounding box and resizing the bounding box, respectively.
[0426] Options 710 for moving the bounding box include options for moving the bounding box longitudinally along the road in either direction (±R, as shown on the right side of the top view), and options for moving the bounding box laterally across the road (±L).
[0427] Options 712 for resizing the bounding box include options for changing the width (w), height (h), and length (1) of the bounding box.
[0428] Although depicted as displayed UI elements, associated inputs may alternatively be provided using keyboard shortcuts, gestures, or the like.
[0429] An example workflow for placing a 3D annotation object will now be described. It should be appreciated that this is just one example way in which an annotator may utilize the annotation functionality of the annotation interface 700.
[0430] Fig.32A The annotation interface is shown once a new bounding box 800 is created. The bounding box 800 is placed at an initial position at the height of the road in the 3D annotation space and oriented parallel to the road direction at that position (as captured in the 3D road model). To assist the annotator, the 3D bounding box 800 is projected into the image plane of the display image 702 and the top view 704.
[0431] like Fig.32B As shown, when the annotator moves the bounding box 800 along the road in the +R direction, the bounding box 800 is automatically reoriented to keep it parallel to the road direction. In this example, the annotator's goal is to manually fit the bounding box 800 to the vehicle that is visible in the right half of the image and facing the image plane.
[0432] like Fig.32C As shown, once the annotator has moved the bounding box 800 to the desired position along the road, it is then moved laterally (ie, perpendicular to the direction of the road) to the desired lateral position - in this case, in the +L direction.
[0433] like Fig.32D and Fig.32E As shown, the annotator then appropriately adjusts the width (in this case, "-w" indicates a decrease) and height ("+h" indicates an increase) of the bounding box 800. It happens that no length adjustment is required in this example, but the length of the bounding box can be adjusted in the same manner as needed. The width dimension of the bounding box 800 remains parallel to the road direction at the location of the bounding box 800, while the height dimension remains perpendicular to the road surface at the location of the bounding box 800.
[0434] The above example assumes that bounding box 800 remains bound to the 3D road model when adjustments are made. Although not shown, the annotation interface may also allow for "free" adjustments that are not constrained by the 3D road model, i.e., the annotator is also free to move or rotate bounding box 800 as needed. This may be useful, for example, for vehicles whose annotated behavior sometimes deviates from the assumed behavior (e.g., during a turn or lane change).
[0435] After placing a 3D bounding box for at least one moving object in this manner, Fig.29 Steps 3 to 4 of the method can be applied in substantially the same manner as described above. Assuming that the other vehicles follow the shape of the 3D road model, the position and orientation of the 3D bounding box 800 can be extrapolated in the same manner. FIG. 32A to FIG. 32E When manually adjusted of the type shown, it is at that point that 3D bounding box 800 may be "snapped" to the 3D road model, and when automatically extrapolated between frames, 3D bounding box 800 may remain snapped to the 3D road model. The system moves the 3D bounding box longitudinally along the road between frames in the same manner as if the user had manually selected the ±R option of selectable option 710, but this time it may do so automatically by an automatically determined amount determined based on the speed of the vehicle and the time interval between frames.
[0436] It should be noted that in this regard, the desired 3D annotation data may be 3D annotation data. That is, in this regard, the 3D annotation data is not only used to determine the 2D annotation data of the moving object in order to train the 2D machine learning perception component, but as an alternative or in addition, the 3D annotation data may be used to train the machine learning component to interpret the 3D structure data.
[0437] As mentioned above, the 3D annotation element 800 may alternatively take the form of a 3D model.With 3D frames, a 3D model may be derived from one or more 3D frames themselves, for example by selectively extracting points from the 3D frames belonging to the 3D model.
[0438] See, for example, UK patent applications GB 1910382.9, GB 1910395.1, GB 1910390.2 and GB1910392.8, the entire contents of which are incorporated herein by reference. These documents disclose 3D frame-based object modeling techniques that can be applied in this context. A 3D bounding box can be placed around an object in one or more 3D frames (e.g. manually), and then a subset of points can be extracted from the 3D bounding box in that frame / each frame to build a 3D model of the object, which can then be applied to other frames. Based on the assumed motion of the object along the road, the 3D model can be placed in other frames.
[0439] Alternative road modeling methods
[0440] In the above, the road modeling component 404 applies a "Structure from Motion (SfM)" process, which is applied to a series of 2D images in order to reconstruct the 3D path (ego path) of the vehicle VP that captured the images. This in turn is used as a basis for extrapolating the 3D surface of the road the vehicle traveled on. It can be assumed that moving objects at least approximately follow the same road surface. This is based on 2D feature matching between images of the video sequence (segment).
[0441] The road model may also be determined in alternative ways, such as point cloud fitting, for example the own path may be based on 3D structure matching applied to a depth map or point cloud and / or using high precision satellite positioning (e.g. GPS). Alternatively, an existing road model may be loaded and the frame may be positioned within the existing road model as required.
[0442] More generally, examples of technologies that can be used to reconstruct the ego vehicle path VP include accelerometers / IMUs (internal measurement units - the path can be reconstructed based on internal sensor data), visual odometry, LIDAR odometry, RADAR odometry, wheel encoders, etc.
[0443] The aforementioned reference uses a 3D road model extrapolated from the vehicle's own path to efficiently generate 2D annotation data to annotate the road structure in the original image. In the current context, this technique is extended to allow 3D bounding boxes to be effectively placed around other objects on the road, such as other vehicles, cyclists, etc., across multiple frames in a video clip, by assuming that other road users generally follow the road shape over time.
[0444] References
[0445] 5.G·J·Brostow, J·Fauqueur, R·Cipolla: "Semantic object classes in video: A high-definition ground truth database" Pattern Recognition Letters, 30(2)(2009)88-97
[0446] 6. M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Pranke, S. Roth, B. Schiele: "The cityscapes dataset for semantic urban sceneunderstanding" CVPR (2016)
[0447] 7.G. Neuhold, T. Ollmann, S. R. Bulo, P. Kontschieder: "The mapillaryvistas dataset for semantic understanding of street scenes" Proceedings of the International Conference on Computer Vision (ICCV), Venice, Italy (2017) 22-29
[0448] 14. G.J. Brostow, J. Shotton, J. Fauqueur, R. Cipolla: Segmentation and recognition using structure from motion point clouds, European conference on computer vision, Springer (2008), 44-57
[0449] 15. S. Sengupta, P. Sturgess, P. H. S. Torr, et al.: Automatic dense visual semantic mapping from street-level imagery, Intelligent Robots and Systems (IROS), 2012 IEEE / RSJ International Conference on Computer Vision, IEEE (2012) 857-862
[0450] 16. T. Scharwachter, M. Enzweiler, U. Franke, S. Roth: "Efficient multi-cue scene segmentation" German Conference on Pattern Recognition, Springer Publishing House (2013)
[0451] 17. J·Fritsch, T·Kuehnl, A·Geiger: "A new performance measure and evaluation benchmark for road detection algorithms", 16th International IEEE Conference on Intelligent Transportation Systems (ITSC2013), IEEE (2013) 1693 - 1700
[0452] 18. M·Aly: "Real time detection of lane markers in urban streets", IEEE Intelligent Vehicles Symposium, Proceedings, IEEE (2008) 7 - 12
[0453] 19. TuSimple: "Lane Detection Challenge (Dataset)", http: / / benchmark.tusimple.ai (2017)
[0454] 27. D·Barnes, W·Maddern, I·Posner: "Find Your Own Way: Weakly-Supervised Segmentation of Path Proposals for Urban Autonomy", ICRA (2017)
[0455] 28. Mapillary: "OpenSfM (Software)", https: / / github.com / mapillary / OpenSfM (2014)
[0456] 29. V·Badrinarayanan, A·Kendall, R·Cipolla: Segnet: "A deep convolutional encoder-decoder architecture for image segmentation", arXiv preprint, arXiv:1511.00561 (2015)
[0457] 32. K. He, G. Gkioxari, P. Dollar, R. B. Girshick: Mask R-CNN, CoRR abs / 1703.06870 (2017)
[0458] 33. B. D. Brabandere, D. Neven, L. V. Gool: "Semantic instance segmentation with a aki discriminative loss function", CoRR abs / 1708.02551 (2017)
[0459] 34. S. Li, B. Seybold, A. Vorobyov, A. Fathi, Q. Huang, C. C. J. Kuo: "Instance embedding transfer to unsupervised video object segmentation" (2018)
[0460] 35. A. Fathi, Z. Wojma, V. Rathod, P. Wang, H. O. Song, S. Guadarrama, K. P. Murphy: "Semantic instance segmentation via deep metric learning", CoRR713abs / 1703.10277 (2017)
[0461] 36. S. Kong, O. Fowlkes: "Recurrent pixel embedding for instance grouping" (2017)
[0462] 37. D. Comaniciu, P. Meer: "Mean shift: A robust approach toward feature space analysis", IEEE Trams. Pattern Amal. Mach. Intell. 24(5) (May 2002) 603 - 61
[0463] Appendix A
[0464]
Claims
1. A method for annotating frames in a time sequence of frames captured by at least one moving vehicle, the method comprising, in a frame processing system: determining a three-dimensional (3D) road model of the area captured in the time sequence of frames; receiving first annotation data representing a known 3D position of a moving object for a first frame of a temporal sequence of frames; and Second annotation data is automatically generated for marking the expected mobile object position in at least a second frame of the frame time sequence by assuming that the mobile object moves along an expected path determined based on the known 3D position and the 3D road model.
2. The method according to claim 1, in, The frame is a 2D image.
3. The method according to claim 2, in, The 3D position of the mobile object is determined for the first frame based on geometric constraints imposed on the elevation of the known 3D position relative to the road surface of the 3D road model.
4. The method according to claim 3, in, The geometric constraints constrain the known 3D position to a predetermined zero elevation or non-zero elevation relative to a road surface of the 3D road model.
5. The method according to claim 3 or 4, in, The 3D annotation feature is located at an initial 3D position, the 3D annotation feature is geometrically projected into an image plane of the first frame, and the 3D position is adjusted while maintaining the geometric constraints to match the projection of the 3D annotation feature with the moving object in the first frame, thereby determining a known 3D position of the moving object.
6. The method according to claim 5, in, The 3D position was manually adjusted while maintaining the geometric constraints to manually match the projection to the moving object in the image plane.
7. The method according to claim 5 or 6, in, The second annotation data is determined by transforming the 3D annotation element so as to move it along an expected path to an expected moving object position in the 3D space.
8. The method according to claim 7, in, The second annotation data includes 2D annotation data for marking a position of the expected moving object in an image plane of the second frame, wherein the 2D annotation data is determined by projecting a 3D annotation element at the position of the expected moving object into the image plane of the second frame.
9. The method according to any one of claims 5 to 8, in, First 2D annotation data is determined for the first frame by projecting 3D annotation elements at known 3D locations into an image plane of the first frame.
10. A method according to any preceding claim, in, The expected path is determined based on a speed of the moving object and a time interval between the first frame and the second frame.
11. The method according to claim 10, in, It is assumed that the speed matches the speed of the moving vehicle.
12. The method according to claim 10, in, For one-third of the frames, an additional known 3D position of the moving object is determined, and based on the time interval between the first frame and the third frame, the known 3D position and the additional known 3D position are used to estimate the velocity.
13. A method according to any preceding claim, in, The 3D road model is determined by reconstructing a path traveled by a vehicle while capturing a time series of images, and the 3D road model exhibits a road shape corresponding to a shape of the reconstructed vehicle path.
14. The method according to claim 13, in, The path is reconstructed by processing the frame time sequence.
15. The method according to claim 13, in, The path is determined based on satellite positioning data or other sensor data captured simultaneously with the frames.
16. The method according to any one of claims 13 to 15, in, The expected path of the moving object is defined as having zero vertical offset relative to the reconstructed vehicle path in the road surface normal direction.
17. The method according to any one of claims 13 to 16, in, The expected path of the mobile object is defined with a fixed lateral offset relative to the reconstructed vehicle path.
18. A method according to any preceding claim, in, Based on the position of the mobile object relative to the road structure of the 3D road model, the moving direction of the mobile object is inferred.
19. A method according to any preceding claim, further comprising: include: Classification data is generated for classifying the frames based on the positions of the moving objects in the frames relative to the road structure of the 3D road model.
20. The method according to claim 19, in, Classification frames are used to validate machine learning models.
21. A method according to any preceding claim, in, The annotated frames are used to train the machine learning model in a supervised training process.
22. A method according to any preceding claim, in, The position data captured in association with the frame time sequence is used to match the frame time sequence with a map area in order to determine the road shape of the 3D road model.
23. A method according to any preceding claim, in, The second annotation data takes the form of 3D annotation elements located at locations of expected moving objects in 3D space.
24. A method for annotating a road image, the method comprising the following steps implemented in an image processing system: receiving a time sequence of images captured by an image capture device of a moving vehicle; determining a three-dimensional (3D) road model of the area captured in the image time series; receiving annotation data representing a known position of a moving object in at least one image of a time sequence of images; as well as By assuming that the mobile object moves along an expected path determined from the known position and the 3D road model, further annotation data is automatically generated for marking the expected mobile object position in at least one other image of the image time series.
25. The method according to claim 24, in, The further annotation data comprises 2D annotation data generated by moving an object annotation element from a known position in 3D space along an expected path to an expected position in 3D space and geometrically projecting the object annotation element at the expected position to a two-dimensional image plane of a second frame.
26. The method according to claim 24 or 25, in, The known position and the expected mobile object position are such that both positions maintain a predetermined zero elevation or non-zero elevation from a road surface.
27. The method according to claim 26, in, Annotation data representing a known position of a moving object is generated in response to annotation input received at a user interface of the image processing system, wherein the annotation input causes the object annotation element to move while maintaining a predetermined elevation from a road surface.
28. The method according to claim 27, in, Annotation data is generated in response to the annotation input by geometrically projecting the object annotation features into a 2D image plane of the image.
29. The method according to claim 28, in, The annotation data is superimposed on or otherwise displayed with the image and updated as annotation input is received so that the annotator can see when the object annotation elements are correctly positioned based on the projection of the 2D annotation elements displayed with the image.
30. The method according to any one of claims 24 to 29, in, The annotation data is generated in response to annotation input received at a user interface of the image processing system.
31. The method according to any one of claims 24 to 29, in, The further annotation data is generated based on an expected speed of the moving object along the expected path.
32. The method according to claim 31, in, Assume that the expected speed of the moving object matches the speed of the moving vehicle.
33. The method according to claim 31, in, The annotation data represent respective known positions of the moving object in at least two images, and an expected speed of the moving object is determined based thereon.
34. The method according to any one of claims 30 to 33, in, For images between two images in the sequence, the moving object position is interpolated along the expected path based on the expected velocity to generate additional annotation data for these images.
35. A method of annotating frames in a time sequence of frames captured by at least one moving vehicle, the method comprising, in a frame processing system: determining a three-dimensional (3D) road model of the area captured in the time sequence of frames; and Annotation data is generated for marking a 3D object position in at least one frame of a frame time sequence, the annotation data being generated based on user input received at a user interface of a frame processing system, and being used to move the object position in 3D space, subject to imposed geometric constraints, to an elevation of the 3D object position relative to a road surface of a 3D road model.
36. The method according to claim 35, in, The geometric constraints constrain the known 3D position to a predetermined zero elevation or non-zero elevation relative to a road surface of the 3D road model.
37. The method according to claim 35 or 36, in, The 3D annotation feature is located at an initial 3D position, the 3D annotation feature is geometrically projected into an image plane of the first frame, and the 3D position is adjusted while maintaining the geometric constraints to match the projection of the 3D annotation feature with the moving object in the first frame, thereby determining a known 3D position of the moving object.
38. The method according to any one of claims 35 to 37, in, The annotation data includes 2D annotation data generated by projecting 3D annotation elements at determined 3D object positions into an image plane of the frame.
39. The method according to any one of claims 35 to 38, in, The 3D road model is determined by reconstructing a path traveled by a vehicle while capturing a time series of images, and the 3D road model exhibits a road shape corresponding to a shape of the reconstructed vehicle path.
40. A method according to any one of claims 1 to 34 except claim 2, in, The frame is a 3D frame.
41. The method according to claim 40, in, Each 3D frame includes at least one of a depth map and a point cloud.
42. The method according to claim 5, in, The 3D annotation element is a 3D bounding box.
43. The method according to claim 5, in, The 3D annotation element is a 3D model, and the 3D model is derived from one or more 3D frames.
44. A processing computer system comprising one or more processors configured to perform the steps of any preceding claim.
45. A computer program comprising executable instructions which, when executed on one or more processors of a processing computer system, cause the computer system to perform the steps of any one of claims 1 to 43.
