structural annotation

Abstract

The invention relates to a computer-implemented method of creating one or more annotated perception inputs, the method comprising, in an annotation computer system: receiving a plurality of captured frames, each frame comprising a set of 3D structure points, wherein at least a portion of a common structure component is captured; computing a reference position within at least one reference frame of the plurality of frames; generating a 3D model of the common structure component by selectively extracting 3D structure points of the at least one reference frame based on the reference position within the frame; determining an aligned model position of the 3D model within a target frame of the plurality of frames based on one or more manual alignment inputs received at a user interface in relation to the target frame, by rendering the 3D model to manually align the 3D model with the common structure component in the target frame; storing annotation data of the aligned model position in a computer memory in association with at least one perception input of the target frame so as to annotate the common structure component therein.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to annotating structures captured in images, point clouds, and other forms of perception input. Such annotation can be applied to create annotated perception input for use in training machine learning (ML) perception components. BACKGROUND

[0002] Structure perception refers to a class of data processing algorithms that can meaningfully interpret structures captured in perception input. Such processing can be applied to different forms of perception input. Perception input generally refers to any structure representation, i.e., any dataset in which structures are captured. Structure perception can be applied to two-dimensional (2D) and three-dimensional (3D) space. The result of applying a structure perception algorithm to a given structure input can be encoded as structure perception output.

[0003] One form of perception input is two-dimensional (2D) images, i.e., images having only color components (one or more color channels). The most basic form of structure perception is image classification, i.e., a simple classification of an image as a whole with respect to a set of image classes. More complex forms of structure perception applied to 2D space include 2D object detection and / or localization (e.g., orientation, pose, and / or distance estimation in 2D space), 2D instance segmentation, etc. Other forms of perception input include three-dimensional (3D) images, i.e., images having at least a depth component (a depth channel); 3D point clouds, e.g., 3D point clouds captured using RADAR or LIDAR or derived from 3D images; voxel- or mesh-based structure representations, or any other form of 3D structure representation. Perception algorithms applicable to 3D space include, e.g., 3D object detection and / or localization (e.g., distance, bearing, or pose estimation in 3D space), etc. A single perception input can also be formed from multiple images. For example, stereo depth information can be captured in a pair of stereo 2D images, which can be used as a basis for 3D perception. 3D structure perception can also be applied to a single 2D image, e.g., monocular depth extraction, extracting depth information from a single 2D image (note that even without any depth channel, some degree of depth information can be captured in one or more of its color channels). Such forms of structure perception are examples of different “perception modalities,” as that term is used herein. Structure perception applied to 2D or 3D images can be referred to as “computer vision.”

[0004] Object detection refers to detecting any number of objects captured in perception input, typically involving characterizing each such object as an instance of an object class. Such object detection can involve or incorporate one or more forms of position estimation, such as 2D or 3D bounding box detection (a form of object localization whose goal is to define an area or volume in 2D or 3D space that bounds an object), distance estimation, pose estimation, etc.

[0005] In the context of machine learning (ML), a structure perception component can comprise one or more trained perception models. For example, machine vision processing is often implemented using convolutional neural networks (CNNs). Such networks require a large number of training images that are annotated with information that the neural network is to learn (a form of supervised learning). When training, the network is presented with thousands, or preferably tens of thousands, of such annotated images, and the network learns, on its own, a correlation between features captured in the images and the annotations with which they are associated. Each image is annotated in the sense of being associated with annotation data. The image as the perception input, and the associated annotation data provides the “Ground Truth” for the image. CNNs and other forms of perception model architecture can be architected to receive and process other forms of perception input, such as point clouds, voxel tensors, etc., and perceive structure in 2D and 3D space. In the training context, a perception input can be generally referred to as a “training example” or “training input.” In contrast, a training example captured by a trained perception component for processing at runtime can be referred to as a “runtime input.” Annotation data associated with a training input provides the Ground Truth for that training input, in that the annotation data encodes the expected perception output for that training input. In a supervised training process, the parameters of the perception component are systematically tuned to minimize, within defined limits, an overall measure of the difference between the perception output generated by the perception component when applied to training examples in a training set (the “actual” perception output) and the corresponding Ground Truth provided by the associated annotation data (the “expected” perception output). In this way, the perception input is “learned” from the training examples, and this learning is “generalized” so that, once trained, the perception component is able to provide meaningful perception outputs for perception inputs that it has not encountered during training.

[0006] Such perception components are the cornerstone of many mature and emerging technologies. For example, in the field of robotics, mobile robotic systems capable of autonomously planning paths in complex environments are increasingly popular. Such rapidly developing technologies are exemplified by autonomous vehicles (AVs) that can navigate themselves on urban roadways. Such vehicles must not only perform complex maneuvers between people and other vehicles, but must also do so while ensuring that the probability of a poorly constrained event (e.g., a collision with other such agents in the environment) is low. In order to allow an AV to plan safely, it is critical that it be able to accurately and reliably observe its environment. This includes the need to accurately and reliably detect real-world structures in the vicinity of the vehicle. An autonomous vehicle (also known as a self-driving vehicle) is a vehicle that has a sensor system for monitoring its external environment and a control system that is capable of using these sensors to make and implement driving decisions automatically. This includes, in particular, the ability to automatically adapt the speed and direction of travel of the vehicle based on perception inputs from the sensor system. A fully autonomous or "driverless" vehicle has sufficient decision-making capability to operate without any input from a human driver. However, the term "autonomous vehicle" as used herein also applies to semi-autonomous vehicles, which have greater autonomous decision-making capability and thus still require some level of oversight by a human driver. Other mobile robots are under development, such as those used to carry supplies in and around industrial areas. Such mobile robots do not carry people and belong to a class of mobile robots known as UAVs (unmanned autonomous vehicles). Autonomous aerial mobile robots (drones) are also under development.

[0007] Thus, in the more general field of autonomous driving and robotics, it can be desirable for one or more perception components to interpret perception inputs, i.e., to determine information about real-world structures captured in a given perception input.

[0008] Complex robotic systems such as AVs can increasingly need to implement multiple perception modalities in order to accurately interpret multiple forms of perception inputs. For example, an AV can be equipped with one or more pairs of stereo optical sensors (cameras) from which associated depth maps are extracted. In this case, the data processing system of the AV can be configured to apply one or more forms of 2D structure perception to the images themselves (e.g., 2D bounding box detection and / or other forms of 2D localization, instance segmentation, etc.) plus one or more forms of 3D structure perception to the associated depth map data (e.g., 3D bounding box detection and / or other forms of 3D localization). Such depth maps can also come from LiDAR, RADAR, etc. or be derived from fusing multiple sensor modalities.

[0009] To train a perception component for a desired perception modality, the perception component architecture is architected to enable it to receive perception input in the desired form and provide perception output in the desired form in response. In addition, to train a perception component of the appropriate architecture based on supervised learning, annotation conforming to the desired perception modality needs to be provided. For example, to train a 2D bounding box detector, 2D bounding box annotation is needed; likewise, to train a segmentation component to perform image segmentation (pixel-wise classification of individual image pixels), the annotation needs to encode a suitable segmentation mask from which the model can learn; a 3D bounding box detector needs to be able to receive 3D structure data as well as annotated 3D bounding boxes, etc. SUMMARY

[0010] Conventionally, annotated training examples are created by having human annotators manually annotate training examples. Even in the case of 2D images, each image can take tens of minutes. Thus, creating hundreds of thousands of training images takes a huge amount of time and human cost, which in turn makes it a high-cost training. In practice, it limits the actual number of training images that can be provided, which in turn can be detrimental to the performance of training a perception component on the limited number of images. Manual 3D annotation is significantly more tedious and time-consuming. Moreover, when multiple perception modalities need to be accommodated, the problem only gets exacerbated, as then multiple forms of annotation data (e.g., two or more 2D bounding boxes, segmentation masks, 3D bounding boxes, etc.) can be needed for one or more forms of training input (e.g., one or more 2D images, 3D images, point clouds, etc.).

[0011] The present disclosure generally relates to an annotation tool of a form having annotation functionalities that facilitate fast and efficient annotation of perception input. Such an annotation tool can be used to create annotated perception input for training a perception component. The term “annotation tool” refers broadly to a computer system programmed or otherwise configured to implement those annotation functionalities, or to a set of one or more computer programs for programming a programmable computer system to perform those functionalities.

[0012] In one aspect, the invention provides a computer-implemented method of creating one or more annotated perception inputs, the method comprising, in an annotation computer system: receiving a plurality of captured frames, each frame comprising a set of 3D structure points, wherein at least a portion of a common structural component is captured; computing a reference position within at least one reference frame of the plurality of frames; generating a 3D model of the common structural component by selectively extracting 3D structure points of the at least one reference frame based on the reference position within the frame; determining an aligned model position of the 3D model within a target frame of the plurality of frames based on one or more manual alignment inputs received at a user interface with respect to the target frame, by rendering the 3D model to manually align the 3D model with the common structural component in the target frame; storing annotation data of the aligned model position in association with at least one perception input of the target frame in a computer memory, so as to annotate the common structural component therein.

[0013] This allows the perception input for the target frame to be annotated more quickly and accurately compared to placing a 3D bounding box or other 3D bounding object around the common structural component in the target frame. This is because it is easier for a human annotator to see when the 3D model of the common structural component (as generated from the reference frame) is correctly aligned with the common structural component captured in the target frame. The method takes advantage of the fact that the position (location and / or orientation) of the common structural component in the reference frame can be used to generate a 3D model of the common structural component from the reference frame, which in turn can be aligned with the common structural component in the target frame to determine the position of the common structural component in the target frame.

[0014] The method uses "self-contained" modeling, whereby a 3D model is generated using 3D structure points from one or more frames themselves. Thus, the flexibility of the method is sufficient to apply to any form of structural component captured in multiple 3D frames.

[0015] In some embodiments, the method can be applied in an iterative manner to build up a 3D model of increasing quality as the frames are annotated, while generating usable annotation data for the frames. For example, an annotator can initially create a 3D model by positioning a 3D bounding box in a single initial reference frame, or the 3D bounding box can be placed automatically (see below). In this example, the "reference position" is the position of the 3D bounding box within that frame. In addition to providing annotation data for that frame, a subset of 3D structure points from that frame can also be extracted from within the volume of the 3D bounding box and used to generate an initial 3D model. This 3D model can then be "propagated" to the next frame to be annotated (the target frame) and quickly and accurately aligned to common structure components in the target frame, which in turn provides a position for the 3D bounding box within the target frame as the aligned model position (i.e., the position of the 3D bounding box in the target frame is the 3D model position in the target frame that is aligned to the common structure components). From there, the method can be performed in an iterative manner: the 3D bounding box position within the target frame provides usable annotation data for the target frame; in addition, a subset of 3D structure points from the target frame (now considered a second reference frame) can be extracted from within the volume of the 3D bounding box and aggregated with those from the initial reference frame to provide a more complete 3D model, which can then be propagated to the next frame to be annotated (a new target frame) for manual or semi-automatic alignment, and so on.

[0016] The annotation data can be 2D or 3D annotation data or a combination of both.

[0017] The 3D annotation data can include position data for the aligned model position to define the position of the 3D bounding object (e.g., 3D bounding box) in 3D space relative to the perceived input of the target frame.

[0018] For example, the 2D annotation data can be created based on a computed projection of the 3D model to the image plane of the target frame or based on a computed projection of a new 3D model generated using 3D structure points selectively extracted from the target frame based on the aligned model position (e.g., an aggregated 3D model generated by aggregating the selectively extracted points from the target frame and the reference frame).

[0019] 2D annotation data can include a 2D bounding box (or other 2D bounding object) for the structural component, fitted to a projection of the computed 3D model into the image plane. Advantageously, fitting the 2D bounding box to a projection of the 3D model itself can provide a tight 2D bounding box, i.e. a 2D bounding box that closely aligns with the outer boundary of the structural component when the structural component appears in the image plane. An alternative approach is to determine a 3D bounding box (or other 3D bounding object) for the common structural component in 3D space, and project the 3D bounding object into the image plane; however, in most practical scenarios, this does not provide a tight 2D bounding box in the image plane: even if the 3D bounding object closely fits in 3D space, there is no guarantee that the projected 3D bounding box will align with the outer boundary of the structural component itself when it appears in the image plane.

[0020] Additionally or alternatively, the 2D annotation data can include a segmentation mask for the structural component. By projecting the 3D model itself into the image plane, a segmentation mask can be provided that labels (annotates) pixels of the image plane within the model projection outer boundary as belonging to the structural component, and labels (annotates) pixels outside the outer boundary as not belonging to the structural component. It should be noted that some degree of processing can be applied to the computed projection to provide a useful segmentation mask. For example, the projection region can be "smoothed" or "filled" to reduce artefacts caused by noise, sparsity, etc. in the underlying frames from which the 3D model was generated. In this case, the outer boundary of the computed projection refers to the outer boundary after such processing has been applied.

[0021] The 3D model can include the selectively extracted structural points themselves and / or a 3D mesh model or other 3D surface model fitted to the selectively extracted points. When a 3D surface model is used to create a segmentation mask, the outer boundary of the projected 3D surface model defines the segmentation mask (subject to any post-processing of the computed projection). A 3D surface model is a way of providing a higher quality segmentation mask.

[0022] In general, denoising and / or refinement can be applied to one or both of the 3D model (before it is projected) and the computed projection. This can involve one or more of noise filtering (to filter out noisy points / pixels, e.g. having insufficient number of neighbouring points / pixels within a defined threshold distance), predictive modelling, etc. For predictive modelling, given that the 3D model (the corresponding projection) is incomplete or incorrect, a predictive model is applied to the existing points (the corresponding pixels) of the 3D model (the corresponding projection). The existing points (the corresponding pixels) act as a "prior" from which a correct 3D model (the corresponding projection) can be inferred. This has the effect of adding additional points / pixels or removing points / pixels that are determined to be incorrect (thus, predictive modelling can be used to predict missing points / pixels and / or used as a noise filtering means).

[0023] In some embodiments, the 3D model is an aggregated model derived from multiple frames. For each frame of the multiple frames, a reference position is computed for the frame for selectively extracting 3D structure points from the frame. The 3D structure points extracted from the multiple frames are aggregated to generate an aggregated 3D model. The aggregated 3D model has a number of advantages. For example, such a model can account for occlusions or other forms of partial data capture as well as data sparsity and noise. With respect to the latter, aggregation over multiple frames means that noise artifacts are more easily identified and corrected (e.g., spurious noise points are more sparsely distributed than those that actually belong to relevant structure components, and thus can be more reliably filtered out; the increased density of points that do belong to relevant structure components also provides a stronger prior for predictive modeling, etc.). Thus, there are particular benefits to applying aggregation to 3D models in conjunction with denoising.

[0024] In preferred embodiments, the annotation tool provides annotation data (e.g., 2D and 3D annotations) for multiple perception modalities and / or for multiple frames based on a common set of annotation operations (which can be manual, automatic, or semi-automatic operations). That is, the same operations are used to provide annotation data for multiple perception modalities and / or for multiple frames. There are various ways in which this annotation functionality can be utilized, including but not limited to the examples described below.

[0025] An example annotation workflow is provided for illustrative purposes. For simplicity, the examples described below consider two frames. It will be appreciated, of course, that aggregated models can be generated across a larger number of frames. Indeed, by aggregating across many frames, a high-quality aggregated model (dense and low noise) can be obtained, which in turn can be used to efficiently generate high-quality annotation data for those frames (and / or other frames in which the common structure components are captured).

[0026] Example 1 - Context: A 3D bounding box (or other bounding object) is accurately positioned within a first frame to bound a particular structure component. Assume that one or more forms of "external" input are used to position (e.g., position and orient) the 3D bounding object, i.e., the perception component is not available at runtime, then useful Ground Truth is provided for supervised training purposes. In the simplest case, the 3D bounding object can be positioned manually (i.e., the external input is provided by human input); however, the 3D bounding object can also be positioned automatically by utilizing contextual information such as a measured or assumed structure component path (in which case the external input comes from the known or assumed path). The position and dimensions (as applicable) of the 3D bounding box within the first frame can in turn be stored as 3D annotation data for the first perception input of the first frame (i.e., the included perception input originates from or otherwise corresponds to at least a portion of the frame).

[0027] Example 1 - 3D to 2D; same frame: In addition, at this point the 3D bounding box has been accurately located, which can be used as a basis for generating a 3D model of the desired structural component. For a 3D bounding object that fits tightly and is placed accurately, the intersection of the first frame with the 3D bounding box volume can be used as a basis for the model (i.e., 3D structural points within the 3D bounding box volume can be extracted to generate the 3D model). In this case, the "reference position" described above is calculated as the position of the 3D bounding box within the first frame (it should be noted that, as described below, the bounding box need not initially fit tightly - at least a "rough" bounding box can also be used initially).

[0028] At this point 2D annotation data can be generated for the first frame immediately: the aligned position of the 3D model within the first frame (i.e., the position at which the 3D model is aligned with the structural component in the frame) is known as the position of the 3D bounding box. Thus, the 3D model can be projected into the desired image plane based on this position, and the resulting projection used as a basis for one or more forms of 2D annotation data. Such 2D annotation data can be stored in association with the first perception input of the first frame described above or another perception input of the first frame. For example, the 3D annotation data can be stored in association with the point cloud of the first frame, and the 2D annotation data can be stored in association with the 2D image of the first frame (e.g., color components associated with depth components, where the first frame includes or is derived at least in part from depth components).

[0029] Example 1 - model propagation: More importantly, at this point the same 3D model can be propagated into a second frame that at least partially captures the same structural component. By aligning the 3D model with the common structural component in the second frame (manually or semi-automatically), i.e., positioning and / or re-orienting the 3D model so that its structural elements and / or visible features, etc., align with corresponding structural elements, features, etc., of the common structural component in the second frame, an aligned position of the 3D model within the second frame is determined.

[0030] Example 1 - Model propagation; 3D to 3D: The position of the 3D bounding box relative to the 3D model is originally known as a result of deriving the 3D model based on the position of the 3D bounding box within the first frame. For example, the 3D structure points of the 3D model can be defined in the frame of reference of the 3D bounding box, see below. Thus, by accurately positioning the 3D model within the second frame (aligning it with the common structural components), the accurate position of the 3D bounding box within the second frame at this time is taken as the aligned model position in the second frame. For rigid objects, the same bounding box dimensions can be applied in the first and second frames. In other words, by propagating the 3D model into the second frame and aligning it with the common structural components, the bounding box positioned in the first frame is propagated into the second frame and accurately positioned in the second frame. The position of the bounding box in the second frame at this time has been accurately determined and can be stored (as applicable, along with the bounding box dimensions) as 3D annotation data for the second frame of the second perception input (i.e. including perception input derived from or otherwise corresponding to at least a portion of the second frame). As mentioned above, the alignment can be manual, automatic or semi-automatic. In the case of manual alignment, it is significantly easier to accurately align the 3D model with the corresponding structural components by eye (based on characteristic structural elements, features, etc.) than to "start from scratch" and position a second bounding box in the second frame. Furthermore, in the case of rigid objects, the same bounding box dimensions can be applied across all frames, so there is no need to define these bounding box dimensions separately for each frame in the event.

[0031] Example 1 - Model propagation; 3D to 2D: Finally, with the aligned model position (or, in this context, equivalently, the 3D bounding box position) taken to be within the second frame, 2D annotation data can be created for the second frame in a similar manner. This can be based on: (i) a projection of the model to the desired image plane; (ii) a projection of the second 3D model determined in the same manner, but with the 3D structure points of the second frame selectively extracted from within the volume of the bounding box in the second frame (which has been accurately positioned using the 3D model derived from the first frame); or (iii) an aggregated model generated by aggregating these selectively extracted points from the second frame with those from the first frame (possibly with many other frames to build a dense aggregated model). Such 2D annotation data can be stored in association with the second perception input for the second frame or another perception input for the second frame as described above.

[0032] As can be seen from Example 1, with respect to the first frame, 2D and 3D annotation data is generated using the operations of positioning the 3D bounding object within the first frame. Furthermore, using those same operations, in conjunction with the operations of aligning the 3D model in the second frame, 3D and 2D annotation data is additionally provided for the second frame. Two frames are exemplified, but it will be appreciated that these principles can be applied to a greater number of frames, providing a more significant performance advantage in terms of annotation time and efficiency.

[0033] As should be appreciated, the above Example 1 is one of many effective annotation workflows facilitated by certain embodiments of the annotation tool. This example is merely illustrative of certain features of the annotation tool and does not limit or restrict the scope of the present invention. For the same purpose, more examples are described below.

[0034] The annotation tool is particularly well suited for annotating a time-sequential frame, i.e., a time-sequential series of one or more frames captured over a certain time interval, typically at relatively short regular time intervals. In this context, such a time-sequential series of frames can be referred to as a “3D video sequence,” with the understanding that each frame includes 3D structure points, i.e., points capturing structure in 3D space. One example application is the annotation of a 3D video sequence captured by a vehicle or other mobile object in motion to provide an annotated perception input well suited for training one or more perception components for use in an autonomous driving vehicle or other mobile robot. For example, such frames can capture urban or non-urban road scenes, which in turn can be annotated to mark road structures, other vehicles, pedestrians, bicyclists, and any other form of structure component that an autonomous driving vehicle needs to be able to perceive and respond to.

[0035] In this context, a “frame” refers to any captured 3D structure representation, i.e., including captured points (3D structure points) defining a 3D spatial structure, providing a static “snapshot” of the 3D structure captured in that frame (i.e., a static 3D scene). It can be said that the frame corresponds to a single moment in time, but this does not necessarily imply that the frame needs to be captured immediately or that the underlying sensor data from which the frame is derived needs to be captured immediately—e.g., LiDAR measurements can be captured in an “untwisted” fashion over a short time period (e.g., about 100 ms) by a mobile object in a LiDAR scan to account for any motion of the mobile object, thereby forming a single point cloud. In that case, despite the manner in which the underlying sensor data is captured, the single point cloud can still be said to correspond to a single moment in time in the sense that such untwisting provides a useful static snapshot. In the context of a time-sequential series of frames, each frame corresponds to a time index (timestamp) within the time sequence (and each frame in the time sequence corresponds to a different moment in time).

[0036] In the context of the annotation tool, the terms “object” and “structure component” are used synonymously to refer to an identifiable piece of structure within the static 3D scene of a 3D frame modeled as an object. It should be noted that, in accordance with this definition, an object in the context of the annotation tool can in fact correspond to only a portion of a real-world object or to multiple real-world objects, etc. That is, the term “object” is broadly applicable to any identifiable piece of structure captured in any 3D scene.

[0037] With respect to more terminology employed herein, the terms "orientation" and "angular position" are used synonymously and refer to a rotational configuration of an object in 2D or 3D space, as applicable, unless otherwise specified. As will be clear from the foregoing description, the term "position" is used in a broad sense to encompass both position and / or orientation. Thus, a position of an object as determined, calculated, assumed, etc. can have only a positional component (one or more positional coordinates), only an orientation component (one or more orientation coordinates), or both a positional component and an orientation component. In general, then, a position can include at least one of a positional coordinate and an orientation coordinate. The term "pose" refers to a combination of position and orientation of an object, e.g., a full six-dimensional (6D) pose vector, fully defining the position and orientation of an object in 3D space (the term "6D pose" can also be used as shorthand for full pose in 3D space), unless otherwise specified.

[0038] The terms "2D perception" and "3D perception" can be used as shorthand to refer to structure perception applied to 2D and 3D space, respectively. For the avoidance of doubt, such terms do not necessarily imply the dimensionality of the resulting structure perception output, e.g., the output of a full 3D bounding box detection algorithm can be in the form of one or more nine-dimensional vectors, each defining a 3D bounding box (cuboid) as a 3D position, 3D orientation, and size (height, width, length - bounding box dimensions); by way of another example, an object depth can be estimated in 3D space, but in this case a single-dimensional output can be sufficient to capture the estimated depth (as a single depth dimension). Moreover, 3D perception can also be applied to 2D images, e.g., in monocular depth perception.

[0039] In some embodiments, the annotation data for the aligned model position can include position data for the aligned model position to annotate a position of a common structural component in the at least one perception input of the target frame.

[0040] The position data can be 3D position data to annotate a position of the common structural component in 3D space.

[0041] The annotation data for the aligned model position can include annotation data derived from the 3D model using the aligned model position.

[0042] The derived annotation data can be 2D annotation data derived by projecting the 3D model into an image plane based on the aligned model position. Alternatively, the 2D annotation data can be derived by projecting a single-frame or aggregated 3D model generated by selectively extracting 3D structure points from the target frame based on the aligned model position into the image plane.

[0043] The 2D annotation data can include at least one of: a 2D bounding object fitted to a projection of the 3D model or the single-frame or aggregated 3D model into the image plane; and a segmentation mask for the common structural component.

[0044] The reference position of the reference frame can be computed based on one or more positioning inputs received at the user interface with respect to the reference frame while rendering a visual indication of the reference position within the reference frame for manual adjustment of the reference position within the reference frame.

[0045] The reference position can be automatically or semi-automatically computed for the reference frame.

[0046] The reference position can be automatically or semi-automatically computed by applying a perception component to the reference frame.

[0047] The frame can be a time-series frame, and the reference position can be automatically or semi-automatically computed based on a path of a common structural component within a time interval of the time-series frame.

[0048] One or more boundary object dimensions can be determined for the common structural component, wherein the one or more boundary object dimensions can be one of: (i) manually determined based on one or more dimension inputs received at the user interface with respect to the reference frame; (ii) automatically determined by applying a perception component to the reference frame; (iii) semi-automatically determined by applying a perception component to the reference frame and further based on one or more dimension inputs received with respect to the reference frame; (iv) assumed.

[0049] The selectively extracted structural points from the reference frame 3D can be selectively extracted therefrom to generate the 3D model based on the computed reference position within the reference frame and the one or more boundary object dimensions.

[0050] The selectively extracted 3D structural points can be a subset of points within a 3D volume defined by the reference position and the one or more boundary object dimensions.

[0051] The annotation data of the at least one perception input of the target frame can further include: one or more boundary object dimensions for annotating a rigid common structural component, or a transformation of one or more boundary object dimensions for annotating a non-rigid common structural component.

[0052] The one or more boundary object dimensions can be manually or semi-automatically determined based on one or more dimension inputs received with respect to the reference frame, wherein the visual indication of the reference position can take the form of rendering a 3D boundary object at the reference position within the reference frame and such that the one or more boundary object dimensions are for manual adjustment of those one or more boundary object dimensions.

[0053] The one or more boundary object dimensions can additionally be calculated based on one or more adjustment inputs received at the user interface in respect of the target frame, while rendering the 3D boundary object at the aligned model position within the target frame; the 3D boundary object can be rendered at the reference position within the reference frame, simultaneously or subsequently, with the one or more boundary object dimensions adjusted in accordance with the one or more adjustment inputs received in respect of the target frame.

[0054] The annotation data can comprise 2D annotation data of the aligned model position and 3D annotation data of the aligned model position stored in association with the one or more perception inputs of the target frame, whereby the aligned model position is used for 2D annotation and 3D annotation.

[0055] Further annotation data of the reference position can be stored for annotating the common structural component in the at least one perception input of the reference frame, whereby the calculated reference position within the reference frame is usable for annotating the perception inputs of the target frame and the reference frame.

[0056] The further annotation data can comprise the same one or more boundary object dimensions used for annotating the rigid common structural component, or a transformation thereof used for annotating the non-rigid common structural component.

[0057] The 3D model can be an aggregated 3D model determined by aggregating data points selectively extracted from the reference frame with data points extracted from at least one of: the target frame; and a third frame other than the target frame and the reference frame.

[0058] The perception component can be a trained perception component that is retrained using the at least one perception input of the target frame and the annotation data.

[0059] The method can comprise the step of training at least one perception component using the or each perception input in a training process, wherein the annotation data of the perception input provides Ground Truth for the perception input in the training process.

[0060] The 3D structure point set of each frame can be in the form of a point cloud.

[0061] The 3D structure point set of the reference frame can be transformed into a frame of reference of the reference position to selectively extract 3D structure points of the 3D model.

[0062] The 3D volume defined by the reference position and the one or more boundary object dimensions can be a cuboid aligned with coordinate axes of the frame of reference.

[0063] The aligned model position can be determined semi-automatically by automatically initially estimating a model position and then aligning the estimated model position in accordance with one or more manual alignment inputs.

[0064] The model position can be initially estimated based on an automatic alignment of the 3D model to common structural components in the target frame.

[0065] The frame can be a time-sequential frame, and the model position can be initially estimated based on a path of common structural components within a time interval of the time-sequential frame.

[0066] The model position can be initially estimated based on an automatic alignment of the 3D model to common structural components in the target frame.

[0067] Other aspects of the disclosure provide a computer system comprising one or more computers programmed or otherwise configured to perform any of the steps disclosed herein, and one or more computer programs embodied on a transitory or non-transitory medium for programming a computer system to perform these steps.

[0068] The computer system can be embodied in a robotic system (e.g., an autonomous vehicle or other mobile robot) or as a simulator. BRIEF DESCRIPTION OF DRAWINGS

[0069] Embodiments of the present application will now be described, by way of example only, with reference to the accompanying drawings in which:

[0070] Figure 1 A highly schematic functional block diagram of a training system for training a perception component is shown;

[0071] Figure 2 A highly schematic block diagram of an autonomous vehicle is shown;

[0072] Figure 3 A schematic functional block diagram of an annotation computer system is shown;

[0073] Figure 4 A schematic perspective view of a frame in the form of a point cloud is shown;

[0074] Figure 5 A block diagram of a stereo image processing system is shown;

[0075] Figure 6 Certain principles of stereo depth extraction are schematically illustrated;

[0076] Figure 7 A to Figure 8E Various examples of a graphical user interface (GUI) rendered by the annotation computer system showing a time-sequential sequence of annotated 3D road scenes are shown;

[0077] Figure 9A A flowchart of a method of generating an object model is shown;

[0078] Figure 9B A schematic diagram of a method applied to a point cloud is shown;

[0079] Figure 10 A to Figure 12C More examples of the annotation system GUI are shown, in particular showing ways in which the generated object models can be applied to create annotations in time series of 3D road scenes;

[0080] Figures 13A-13C Illustrating ways in which vehicle path information can be incorporated into an automated or semi-automated annotation process;

[0081] Figure 14 Flowchart showing a method of iteratively generating and propagating aggregated 3D object models. DETAILED DESCRIPTION

[0082] The embodiments of the invention will be described in detail below. First, some mechanisms that are beneficial to the described embodiments are provided.

[0083] Figure 1 A highly schematic functional block diagram of a supervised training system for training a perception component 102 based on a set of annotated perception inputs 108 (i.e. perception inputs together with associated annotation data) is shown. In the following description, the perception component 102 can synonymously be referred to as a structure detector, structure detection component or simply as a structure detector. As mentioned above, the perception inputs used for training purposes can be referred to herein as training examples or training inputs.

[0084] In Figure 1 In the following, the training examples are labeled with reference numeral 104 and the set of annotation data associated therewith is labeled with reference numeral 106. The annotation data 106 provides Ground Truth for the training example 104 associated therewith. For example, for training examples in the form of images, the annotation data 106 can label the location of certain structure components within the image 104 (such as roads, lanes, intersections, non-drivable areas, etc.) and / or objects within the image (such as other vehicles, pedestrians, street signs or other infrastructure, etc.).

[0085] The annotated perception inputs 108 can be divided into a training set, a test set and a validation set, labeled 108a, 108b and 108c, respectively. Annotated training examples can be used to train the perception component 102 without necessarily forming part of the training set 108a for testing or validation.

[0086] The perception component 102 receives a perception input, denoted x, from one of the training set 108a, the test set 108b and the validation set 108c and processes the perception input x so as to provide a corresponding perception output, denoted y.

[0087] y = f(x; w).

[0088] In the above, w denotes a set of model parameters (weights) of the perception component 102, f denotes a function defined by the weights w and the architecture of the perception component 102. For example, in the case of 2D or 3D bounding box detection, the perception output y can comprise one or more detected 2D or 3D bounding boxes derived from the perception input x; in the case of instance segmentation, y can comprise one or more segmentation maps derived from the perception input. In general, the format and content of the perception output y depends on the choice of the perception component 102 and its chosen architecture, and these choices are made in accordance with the one or more desired perception modalities for which it is to be trained.

[0089] The detection component 102 is trained on the perception inputs of the training set 108a to match its output y = f(x) to the Ground Truth provided by the associated annotation data. The Ground Truth provided for a perception input x is denoted here as y x Thus, for the training examples 104, the Ground Truth is evidenced by the associated annotation data 106.

[0090] This is a recursive process in which the input component 112 of the training system 110 systematically provides the perception inputs of the training set 108b to the perception component 102, and the training component 114 of the training system 110 adapts the model parameters w in an attempt to optimize an error (cost) function that penalizes the discrepancy between each perception output y = f(x; w) and the corresponding Ground Truth y x The discrepancy is characterized by a defined metric, such as mean squared error, cross-entropy loss, etc. Thus, by optimizing the cost function to a defined extent, the overall error measured against the Ground Truth for the entire training set 108a can be reduced to an acceptable level. The perception component 102 may, for example, be a convolutional neural network in which the model parameters w are weights between neurons, but the present disclosure is not limited thereto. It should be appreciated that there are several forms of perception models that can be usefully trained on appropriately annotated perception inputs.

[0091] The test data 108b is used to try to reduce overfitting, which refers to the fact that beyond a certain point, increasing the accuracy of the detection component 102 on the training data set 108a does not favor its ability to generalize to perception inputs not encountered in training. Overfitting can be identified as a point at which increasing the accuracy of the perception component 102 on the training data 108 decreases (or does not increase) its accuracy on the test data, where accuracy is measured according to the error function. The purpose of training is to reduce the total error of the training set 108a to a minimum such that it can be minimized without overfitting.

[0092] If necessary, the validation data set 108c can be used to provide a final estimate of the detection component performance.

[0093] Training is not the only application of the present annotation technique. For example, another available application is scene extraction, in which the annotation is applied to 3D data in order to extract a scene that can be run in a simulator. For example, the present annotation technique can be used to extract the trajectories (path and motion data) of the annotated objects, allowing the behavior of these objects to be replayed in a simulator.

[0094] Figure 2 A highly schematic block diagram of an autonomous vehicle 200 is shown, the autonomous vehicle being shown as including an instance of a trained perception component 102, the input of which is connected to at least one sensor 202 of the vehicle 200 and the output of which is connected to an autonomous vehicle controller 204.

[0095] In use, the trained structure perception component 102 (of the instance) of the autonomous vehicle 200 interprets structures within the perception input captured by the at least one sensor 202 in real time according to its training, and the autonomous vehicle controller 204 controls the speed and direction of the vehicle based on the results, without any or with limited driver input.

[0096] Although Figure 2 Although only one sensor 202 is shown in the figure, the autonomous vehicle 102 can be equipped with multiple sensors. For example, a pair of image capture devices (optical sensors) can be arranged to provide stereoscopic vision, and the road structure detection method can be applied to the images captured from each image capture device. Other sensor modalities can alternatively or additionally be provided on the AV 102, such as LiDAR, RADAR, etc.

[0097] It will be appreciated that this is a highly simplified description of certain autonomous vehicle functionality. The general principles of autonomous vehicles are well known and so will not be described in further detail.

[0098] In Figure 2In the context of the AV 200, to train the perception component 102 for use, training examples can be captured using the same vehicle or a similarly equipped vehicle in order to capture training examples that closely correspond to one or more forms of runtime input that the trained perception component 102 will need to be able to interpret on the AV 200 at runtime. Such training examples can be captured using autonomous or non-autonomous vehicles having the same or only similar sensor arrangements. In this context, 3D frames are used as the basis for creating annotated training examples, requiring at least one 3D sensor modality, but it should be noted that the term is broadly applicable to any form of sensor data that can capture a large amount of available depth information, including LiDAR, RADAR, stereo imaging, time-of-flight, or even monocular imaging (where depth information is extracted from a single image - in this case, a single optical sensor is sufficient to capture the underlying sensor data for the perception input to be annotated).

[0099] Further, the techniques described herein can be implemented off-board, i.e., in a computer system such as a simulator that will perform path planning for modeling or experimental purposes. In this case, the sensor data can be obtained from a computer program running as part of a simulation stack. In either context, the perception component 102 can operate on the sensor data to identify objects. In the simulation context, a simulated agent can use the perception component 102 to navigate a simulated environment, and the agent behavior can be recorded, e.g., for flagging safety issues or as a basis for redesigning or retraining the simulated component.

[0100] Embodiments of the present application will now be described.

[0101] Figure 3 A functional block diagram of an annotation computer system 300 is shown, which for brevity can be referred to as an annotation system 300. The purpose of the annotation system 300 is to create data that can be used to train a machine learning component, such as a 2D or 3D structure detector (e.g., a 2D segmentation component, a 2D bounding box detector, or a 3D bounding box detector). Such data can be referred to as training data, which the annotation system 300 provides in the form of a set of training examples having annotation data associated therewith.

[0102] Each training example 321 is in a structured representation form, such as a 2D or 3D image, point cloud, or other sensor dataset in which structure is captured. Each training example 321 is associated with 2D annotation data 313 and / or 3D annotation data 309 created using the annotation system 300. 2D annotation data refers to annotation data defined in a 2D plane (or other 2D surface). For example, 2D annotation data can be defined in an image plane so as to annotate 2D structure within the image plane. 3D annotation data refers to annotation data defined in 3D space so as to annotate 3D structure captured in a depth map, point cloud, or other 3D structure representation. Each training example 321 and the 2D / 3D annotation data 313 / 309 associated therewith is stored in an electronic storage 322 accessible to the annotation computer system 300. The electronic storage 322 is a form of computer memory, in a persistent area of which each training example 321 and its associated annotation data is stored, persistently existing in the persistent area from which it can be exported or otherwise obtained for other uses, such as training one or more perception components (e.g., in an external training system).

[0103] As described below, various annotation functionalities are provided that allow such annotation data to be automatically or semi-automatically generated, thereby improving the speed at which such data is created and reducing the human effort required.

[0104] In Figure 3 , the annotation functionalities are generally represented by a point cloud computing component 302, a road modeling component 304, a rendering component 306, a 3D annotation generator 308, an object modeling component 310, and a 2D annotation data generator 312. The annotation system 300 is also shown to include a user interface (UI) 320 via which a user (human annotator) can interact with the annotation system 300. Via the UI 320, an annotation interface (also referred to herein as an annotation tool) is provided for accessing the annotation functionalities.

[0105] The annotation system 300 is also shown to have an input for receiving data to be annotated, in this example in the form of a time series of frames 301.

[0106] In the following example, each frame takes the form of an RGBD (Red Green Blue Depth) image captured at a particular time instant. An RGBD image has four channels, of which three channels (RGB) are color channels (color components) encoding a "regular" image, and the fourth channel is a depth channel (depth component) encoding depth values for at least some pixels of the image. RGB is exemplified, but the description applies more generally to any image having color components and depth components (or indeed to images having only depth components). In general, the color components of an image can be encoded using one or more color channels (including grayscale / monochrome) in any appropriate color space. The point cloud computing component 302 converts each frame into point cloud form to allow annotation of the frame in 3D space. More generally, a frame corresponds to a particular time instant, and can refer to any dataset (such as multiple RGBD images, one or more point clouds, etc.) that has captured a static "snapshot" structure (i.e. a static 3D scene) for that time instant. Thus, all the following description regarding RGBD images applies equally to other forms of frames. Where frames are received in point cloud form at the annotation system 300, no point cloud conversion is required. Although the following example is described with reference to point clouds derived from RGBD images, the annotation system can be applied to point clouds of any modality, such as monocular depth, stereo depth, LiDAR, radar, etc. Point clouds can also be derived from two or more such sensing modalities and / or from multiple sensor components of the same or different modalities, by merging the outputs of different sensors. Thus, the term "point cloud of a frame" can refer to any form of point cloud corresponding to a particular time instant, including frames received in point cloud form at the annotation computer system 300, point clouds derived from frames (e.g. in the form of one or more RGBD images) by the point cloud computing component 302, or merged point clouds.

[0107] As noted above, although a frame corresponds to a particular time instant, the underlying data used to derive the frame can be captured over a (typically short) time interval, and transformed as necessary to account for temporal changes. Thus, the correspondence of a frame to a particular time instant (e.g. indicated by a timestamp) does not necessarily imply that the underlying data has all been captured simultaneously. Thus, the term "frame" includes point clouds received at a different timestamp to the frame, e.g. laser radar scans captured within 100ms of a particular time instant (such as the time at which the images were captured) "untwisted" into a single point cloud. A time series of frames 301 can also be referred to as a video segment (it should be noted that the frames of a video segment need not be images, and can for example be point clouds).

[0108] Each training example 321 comprises data of at least one frame of the video segment 301. For example, each training example can comprise data of (a portion and / or component of) an RGBD image of at least one frame or a point cloud of at least one frame.

[0109] Figure 10 Two example frames in the video sequence 301 are shown, as detailed below. The depicted frames are of a road scene captured by a driving vehicle. The annotation system 300 is particularly well suited for annotating road scenes, which in turn can be used to effectively train structure detection components for autonomous driving vehicles. However, many of the annotation functionalities can equally well be applied in other contexts.

[0110] A number of annotation functionalities provided by the annotation system 300 are briefly outlined herein.

[0111] Certain annotation functionalities are based on an “object model”, which is a 3D model of an object, i.e. a structure (structure component) to be annotated. As mentioned above, in the annotation tool context, the term “object” is generally applicable to any form of identifiable structure modelled as an object within the annotation tool (such as a portion of a real-world object, multiple real-world objects, etc.). Thus, the use of the term “object” in the following description does not affect this generality.

[0112] The object model is determined as the intersection of the 3D bounding box with the point cloud of a frame (or multiple frames).

[0113] In other words, the 3D modelling component 310 derives an object model of the object to be annotated from one or more frames of the video sequence 301 itself: a 3D bounding box (or other 3D bounding object, such as a template) is placed around the points of the relevant object in a particular frame, and the object model is obtained by isolating the subset of points within the 3D bounding box volume (or equivalently, the intersection of the 3D bounding box with the point cloud) of the point cloud of that frame. This inherently provides the positioning and orientation of the 3D bounding box relative to the 3D object model, which can be encoded as a reference point and an orientation vector fixed in the frame of reference of the object model. This will be referred to as the “object model pose” in the following. Figure 9A And Figure 9B This is detailed further below.

[0114] This is achieved quickly by transforming all points in the point cloud to align with the axes of the bounding box, so a simple magnitude comparison can be used to determine if the point is enclosed. This can be implemented efficiently on a GPU (Graphics Processing Unit).

[0115] Once these points are isolated into the object model in this way, they can be used, for example, to:

[0116] 1. generate tight 2D bounding boxes for the relevant object;

[0117] 2. perform instance segmentation;

[0118] 3. manually improve the pose of distant boxes.

[0119] The responsive noise filtering is achieved by sorting the point pairs by K-neighbors within a fixed radius (using a 3D tree to first find the K-neighbors).

[0120] Points can also be accumulated (or otherwise propagated) across frames in order to generate more complete / dense object models. Using accumulated models can yield improved noise filtering results, as it is easier to separate isolated noise points from points of the object itself captured across multiple frames.

[0121] For example, a refined 3D annotation pose estimate can be obtained by fitting the model to the point cloud of other frames, e.g., using an Iterative Closest Point (ICP) algorithm.

[0122] Model propagation can also provide improved instance segmentation for distant objects, and can also provide improved segmentation for close objects (e.g., in areas where depth data is lacking).

[0123] In addition to generating annotations for training data, object models can also be used to augment training examples (i.e., a form of "synthetic" training data). For example, objects can be artificially introduced into training examples and annotated to provide an additional knowledge base from which the structure detection component can learn. This can be used, for example, to create more "challenging" training examples (for which existing models perform poorly), which in turn can provide performance improvements for more challenging inputs during inference (i.e., when the model is in operation).

[0124] Expanding on item 3 above, 3D annotation data can be automatically generated for a second frame by fitting the 3D object model to a point cloud of the second frame (the second point cloud). The position and orientation of the 3D bounding box relative to the 3D object model is known, so the position and orientation of the 3D bounding box relative to the second point cloud (i.e., in the second point cloud frame of reference) can be automatically determined by fitting the 3D object model to the second point cloud. This is illustrated below with reference to Figure 1 A to Figure 1 D. This is an example of one way to "propagate" a 3D object model from one frame into a second frame in order to automatically or semi-automatically generate annotation data for the second frame. One potential assumption is that the object can be treated as a rigid body.

[0125] The object model can also be propagated from one frame to a second frame based on a 3D bounding box that is manually placed or adjusted in the second frame. This provides a visual aid to assist the annotator in placing / adjusting the 3D bounding box in the second frame. In this case, the human annotator sets the position and / or orientation of the 3D bounding box in the second frame. This can in turn be used to position and / or orient the 3D object model in the second frame based on the fixed position and orientation of the object model relative to the 3D bounding box. When the annotator adjusts the pose (orientation and / or position) of the 3D bounding box in the second frame, the orientation / position of the 3D object model is rendered to match the change to maintain the fixed position and orientation relative to the 3D bounding box. This provides an intuitive way for the annotator to fine-tune the position / orientation of the 3D bounding box in the second frame in order to align the 3D object model with the actual object to the extent visible in the second frame: the annotator can see if the 3D model is not perfectly aligned with the actual object in the second frame and fine-tune the 3D bounding box as needed until alignment is achieved. This is obviously much easier than trying to align the 3D bounding box with the relevant object visually directly, especially in cases where the object is partially occluded. Details of this will be provided later with reference to Figures 11E-11G Details of this will be provided later with reference to

[0126] These two forms of object propagation are not mutually exclusive: the 3D bounding box can be automatically positioned and oriented in the second frame by fitting the 3D object model to the point cloud of the second frame, and then the annotator can manually fine-tune the 3D bounding box to minimize any visible discrepancies between the 3D model and the actual object in the second frame (thus fine-tuning the position / orientation of the 3D bounding box in the second frame).

[0127] In this document, the ability to generate a model and propagate the model to different frames can be referred to as an "x-ray vision feature" (the name derives from a particular use case where a model from another frame(s) can be used to "fill in" a partially occluded object region, but is more generally applicable to model propagation as described herein).

[0128] Extending items 1 and 2 above, 2D annotation data for an RGBD image (or e.g. a color component of the image) is generated by projecting the 3D object model into the image plane of the image. In the simplest case, a subset of points within the intended frame is isolated as described above, this subset of points is projected into the image plane and processed to generate the 2D annotation data. The 2D annotation data can for example be in the form of a segmentation mask or 2D bounding box fitted to the projected points. In some cases, it can be useful to generate 2D annotation data in this way, but this is based on a projection of a 3D model propagated from another frame in the manner described above. Details of the generation of 2D annotation data will be provided later with reference to Figures 12A-12C Details of the generation of 2D annotation data will be provided later with reference to

[0129] To further assist the annotator, the 3D road model provided by the road modeling component 304 can be used to guide the placement of 3D bounding boxes when annotating a road scene. This will be described below with reference to Figures 8A-8E This is described in more detail.

[0130] Some useful scenarios for the described embodiments will first be explained.

[0131] Figure 4 A highly schematic perspective view of a point cloud 400 is shown, which is a set of 3D spatial points defined in a frame of reference. The frame of reference is defined by a coordinate system and a coordinate system origin 402 within a "3D annotation space". In the present example, the frame of reference has an orthogonal coordinate system (Cartesian coordinate system), such that each point in the point cloud is defined by a triple of Cartesian coordinates (x, y, z).

[0132] Several examples are described herein in connection with "volumetric" point clouds, i.e. point clouds derived from one or more volumetric depth maps (but as mentioned above, the annotation system 300 is not limited in this regard and can be applied to any form of point cloud).

[0133] Figure 5 A highly schematic block diagram of a stereo image processing system 500 is shown. The stereo image processing system 500 is shown to comprise an image corrector 504, a depth estimator 506 and a depth transform component 508.

[0134] The stereo image processing system 500 is shown to have an input for receiving left and right images L, R which together constitute a stereo image pair. The stereo image pair is composed of left and right images captured simultaneously by left and right optical sensors (cameras) 502L, 502R of a stereo camera system 502. The cameras 502L, 502R are arranged in a stereo arrangement, in which the cameras are displaced from each other with overlapping fields of view. This mirrors the geometry of the human eye, which enables a person to perceive three-dimensional structure.

[0135] A depth map D extracted from the left and right image pair L, R is shown to be provided as an output of the stereo image processing system 500. The depth map D assigns an estimated depth d ij to each pixel (i, j) of the "target" image of the stereo image pair. In the present example, the target image is the right image R, and thus an estimated depth is assigned to each pixel of the right image R. The other image (in the present example, the left image L) is used as a reference image. The stereo depth map D can for example be in the form of a depth image or image channel, in which the value of a particular pixel in the depth map is the depth assigned to the corresponding pixel of the target image R.

[0136] Referring back to Figure 6 , the pixel depth is estimated by the depth estimator 506 which applies stereo imaging principles.

[0137] Figure 6The upper part shows a schematic view of the image capture system 502, illustrating the basic principle of stereo imaging. The left side shows a plan view (in the xz-plane) of the cameras 502L, 502R, which are shown as being horizontally (i.e., in the x-direction) separated by a distance b (the baseline). The right side shows a side view (in the xy-plane), in which the cameras 502L, 502R are substantially aligned in the vertical (y) direction, so that only the right camera 502R is visible. It should be noted that in the present context, the terms "vertical" and "horizontal" are defined in the frame of reference of the camera system 502, i.e., vertical refers to the direction in which the cameras 502L, 502R are aligned, independent of the direction of gravity.

[0138] For example, a pixel (i,j') in the left image L and a pixel (i,j) in the right image R are shown as corresponding to each other, since they each correspond to substantially the same real-world scene point P. The reference sign I denotes the image plane of the captured images L, R, in which the image pixels are shown to lie. Due to the horizontal misalignment between the cameras 502L, 502R, those pixels in the left and right images exhibit a relative "parallax", as indicated by the arrows. Figure 6 The lower part shows. Figure 6 The lower part shows a schematic view of the left and right images L, R captured by the rectified cameras 502L, 502R, and a depth map D extracted from these images. The parallax associated with a given pixel (i,j) in the target image R refers to the offset between that pixel and the corresponding pixel (i,j') in the reference image L, which is due to the separation of the cameras 502L, 502R and depends on the depth of the corresponding scene point P in the real world (distance from the camera 502R along the z-axis).

[0139] The depth can thus be estimated by searching for matching pixels between the left and right images L, R of a stereo image pair: for each pixel in the target image R, a matching pixel in the reference image L is searched for. The search for a matching pixel can be simplified by inherent geometric constraints, i.e., given a pixel in the target image, the corresponding pixel will lie on a known "epipolar line" in the reference image. For an ideal stereo system with vertically aligned image capture units, the epipolar lines are all horizontal, so that given any pixel (i,j) in the target image, the corresponding pixel (assuming it exists) will be vertically aligned, i.e., in the same pixel row (j) in the reference image L as the pixel (i,j) in the target image R. In practice, this can not be the case, since the stereo cameras can not be perfectly aligned. However, image rectification is applied by the image rectifier 504 to the images L, R to account for any misalignment, so as to ensure that the corresponding pixels are always vertically aligned in the images. Thus, in Figure 5In this case, depth estimator 506 is shown receiving corrected versions of left and right images L, R from image corrector 504 from which a depth map can be extracted. Matching can be assessed based on relative intensity, local features, etc. Several stereo depth extraction algorithms can be applied to estimate pixel disparity, such as Global Matching, Semi-Global Matching, and Local Matching algorithms. In real-time scenarios, Semi-Global Matching (SGM) generally provides an acceptable tradeoff between accuracy and real-time performance.

[0140] In this case, depth estimator 506 is shown receiving corrected versions of left and right images L, R from image corrector 504 from which a depth map can be extracted. Matching can be assessed based on relative intensity, local features, etc. Several stereo depth extraction algorithms can be applied to estimate pixel disparity, such as Global Matching, Semi-Global Matching, and Local Matching algorithms. In real-time scenarios, Semi-Global Matching (SGM) generally provides an acceptable tradeoff between accuracy and real-time performance.

[0141] D ij = j' - j.

[0142] In this way, a disparity is assigned to each pixel of the target image for which a matching pixel can be found in the reference image (this is not necessarily all pixels in the target image: there will generally be a region of pixels on one edge of the target image that are outside the field of view of the other camera, and thus have no corresponding pixel in the reference image; the search can also fail to find a match, or they can be removed if the depth values do not satisfy certain criteria).

[0143] The depth of each such target image pixel is initially computed in disparity space. Each disparity can in turn be converted to distance units using the camera intrinsic parameters (focal length f and baseline b) as follows:

[0144]

[0145] where d ij is the estimated depth of pixel (i,j) in distance units in the target image R, i.e. the distance between camera 502R and the corresponding real-world point P along the optical axis (z-axis) of the stereo camera system 502, D ij is the disparity assigned to pixel (i,j) of the target image R in the pixel matching search. Thus, in Figure 5 In this case, depth transform component 508 is shown receiving the output of depth extraction component 506 in disparity space and transforming that output to the distance space described above in order to provide a depth map D in distance units. In Figure 6 the lower half, pixel (i,j) of the depth map is shown having a value d ij , which is the estimated depth in distance units.

[0146] As mentioned above, in this example, the right image R is the target image and the left image L is used as the reference image. However, more generally, either image can be used as the target image, and the other as the reference image. The choice of which image to use as the target image can be context-dependent. For example, in the context of an autonomous vehicle, a stereo camera captures images of the road ahead of the vehicle; the image captured by the camera closest to the center line of the road can be used as the target image (i.e., the right image of a vehicle driving on the left and the left image of a vehicle driving on the right).

[0147] Brief Review Figure 4 The origin 402 of the coordinate system corresponds to the position of the optical sensor 502R that captures the target image R when capturing the target image R (in this case, the right camera 502R). The z-axis is parallel to the optical axis of the camera 502R, and the x-axis and y-axis are aligned with the pixel row and column directions of the target image R, respectively (i.e., the pixel row represented by the subscript i is parallel to the x-axis, and the pixel column represented by the subscript j is parallel to the y-axis).

[0148] Point cloud computing component 302 can compute point cloud 400 from stereo depth map D based on the known field of view of camera 502R. For example... Figure 6 As shown in the upper part, the i-th column of pixels in the target image R corresponds to a set of angular directions α defined by angles on the xz plane within the camera's field of view. j Similarly, the j-th row of pixels in the target image R corresponds to a set of angular directions β defined by angles on the xy plane within the camera's field of view. j Therefore, the pixel (i,j) of the target image R corresponds to the angular direction (α) defined by the angle pair. j ,α i ).

[0149] Once the depth of pixel (i,j) is known, it can be used as a basis for... ij and the angular direction α corresponding to this pixel j ,α i To calculate the position of the corresponding real-world point in 3D space (using a 3D space point (x)). ij ,y ij ,z ij (Representation). In this example, angle α j and β i It is defined relative to the z-axis, therefore:

[0150] x ij =d ij tan α j ;

[0151] y ij =d ij tan β i ;

[0152] zij = d ij .

[0153] More generally, the x and y components are determined as a function of the depth of the pixel and the angular direction to which the pixel corresponds.

[0154] As shown, the 3D space point (x ij ,y ij ,z ij ) is the point in the point cloud 400 that corresponds to the pixel (i,j) in the target image R.

[0155] In addition, each point in the point cloud can be associated with color information derived from the target image R itself. For example, for an RGB target image, each point in the point cloud can be associated with the RGB values based on the corresponding pixel in the target image R.

[0156] In review Figure 3 , the point cloud computation component 302 is shown with an input that receives an RGBD image 301 and processes this RGBD image 301 as described above to determine the corresponding 3D point cloud 400 depicted in Figure 6 . The point cloud can be determined from a single image or from multiple images that are merged in a common frame of reference.

[0157] The 3D annotation generator 308 allows an annotator to place (i.e., position, orient, and size) a 3D bounding object in the frame of reference of the point cloud 400 to be annotated. In the following example, the 3D bounding object takes the form of a 3D bounding box (a cuboid), but the description applies equally to other forms of 3D bounding objects, such as 3D object templates, etc. This can be a manual or semi-automatic process.

[0158] Alternatively, all the steps performed by the annotator can also be implemented automatically, as described below.

[0159] The annotator places the 3D bounding box via the UI 320 of the annotation system 300. Thus, the 3D annotation generator 308 is shown with a first input coupled to the UI 320 of the annotation system 300 for receiving user input therefrom. The annotator can manually place the 3D bounding box via the UI 320 in order to define a desired structural element (such as a vehicle, a cyclist, a pedestrian, or other object) within the point cloud 400. This is a form of 3D annotation that can be used, for example, to train a 3D structure detection component in the manner described above, for which it is provided as part of the 3D annotation data 309.

[0160] The road modeling component 304 is also shown with an input for receiving at least the color components of the RGBD image and processing these color components to determine a 3D road model. To this end, it is assumed that the vehicle (like the vehicle 100) is equipped with a stereo camera device that provides a 3D point cloud 400 as described above. Figure 2The series of images 301 are captured while driving along the road, so that a 3D model of the road along which the vehicle is driving can be reconstructed based on the captured series of images. To this end, the method applied by the road modelling component 304 can be found in international patent application PCT / EP2019 / 056356, which is incorporated herein in its entirety. This is based on a "Structure from Motion - SfM" process, which is applied to the series of images in order to reconstruct the 3D path of the vehicle that captured the images (ego path). This is in turn used as a basis to extrapolate the 3D surface of the road along which the vehicle is driving. This is based on 2D feature matching between the images of the video sequence 301.

[0161] The road model can also be determined in alternative ways, such as point cloud fitting, e.g. the ego path can be based on 3D structure matching applied to depth maps or point clouds and / or using high precision satellite positioning (e.g. GPS). Alternatively, an existing road model can be loaded and the frames can be positioned within the existing road model as needed.

[0162] The aforementioned references use a 3D road model extrapolated from the ego path of the vehicle to effectively generate 2D annotation data to annotate road structures in the original images. In the present context, this technique is extended to allow 3D bounding boxes to be effectively placed around other objects on the road, such as other vehicles, cyclists, etc. across multiple frames in the video sequence 301 by assuming that other road users generally follow the shape of the road over time.

[0163] The 3D annotation generator 308 is thus shown with a second input coupled to the output of the 3D road modelling component 304. The 3D annotation generator 308 uses the 3D road model as a reference to allow the annotator to "tie" the 3D bounding boxes to the 3D road model. That is, the 3D bounding boxes are moved in a way that is controlled by the 3D road model, which is particularly useful for annotating other road users, such as vehicles, cyclists, etc. For example, the annotator can be provided with the option to move the 3D bounding box along the road, which will automatically re-orient to match the shape and slope of the road or to cross the road perpendicular to the current direction of the road. This will be detailed later.

[0164] The 3D annotation data 309 is also shown as being provided back to the 3D annotation component 309 as a third input. This means that 3D annotation data defined for one frame can be used to automatically generate 3D annotation data for another frame. This will be detailed later.

[0165] The rendering component 306 is shown with inputs connected to the outputs of the point cloud computation component 302, the 3D road modeling component 304, and the 3D annotation generator 308, as well as an input for receiving an RGBD image. The rendering component 306 renders the 3D annotation data 309 within an annotation interface in a manner that is meaningfully interpretable by a human annotator.

[0166] 1. Annotation interface:

[0167] Figure 7 A schematic diagram showing an example annotation interface 700 that can be rendered by the rendering component 306 via the UI 320.

[0168] Within the annotation interface 700, the color component of the RGBD image 702 (current frame) is displayed on the left. The overhead view 704 of the point cloud 400 for this frame is displayed on the right.

[0169] In addition, a projection 706a of the 3D road model onto the image plane of the RGBD image is superimposed on the displayed image 702. Likewise, a projection 706b of the 3D road model into the overhead view is displayed as superimposed on the overhead view of the point cloud 400.

[0170] A selectable option 708 is provided for creating a new 3D bounding box for the current frame. Once creation is complete, selectable options 710 and 712 are provided for moving the bounding box and resizing the bounding box, respectively.

[0171] The option 710 for moving the bounding box includes options for moving the bounding box longitudinally along the road in either direction (±R, as indicated on the right side of the overhead view), as well as options for moving the bounding box laterally across the road (±L).

[0172] The option 712 for resizing the bounding box includes options for changing the width (w), height (h), and length (l) of the bounding box.

[0173] Although depicted as displayed UI elements, the associated inputs can alternatively be provided using keyboard shortcuts, gestures, etc.

[0174] An example workflow for placing a 3D annotation object will now be described. It should be appreciated that this is merely one example way in which an annotator can utilize the annotation functionality of the annotation interface 700.

[0175] Figure 8A The annotation interface is shown once a new bounding box 800 has been created. The bounding box 800 is placed in an initial position at the road height in the 3D annotation space, and oriented parallel to the road direction at that position (as captured in the 3D road model). To assist the annotator, the 3D bounding box 800 is projected into the image plane of the displayed image 702 and the overhead view 704.

[0176] As shown in Figure 8B Fig. 8, as the annotator moves the bounding box 800 along the road in the +R direction, the bounding box 800 is automatically reoriented so that it remains parallel to the road direction. In this example, the annotator’s goal is to manually fit the bounding box 800 to a vehicle visible in the right half of the image and facing the image plane.

[0177] As shown in Figure 8C Fig. 9, once the annotator has moved the bounding box 800 along the road to the desired location, the annotator then moves it laterally (i.e., perpendicular to the road direction) to the desired lateral location - in this case, in the +L direction.

[0178] As shown in Figure 8D and Figure 8E Fig. 10, the annotator then adjusts the width (in this case, to decrease, denoted by “-w”) and height (to increase, denoted by “+h”) of the bounding box 800 as appropriate. In this example, no length adjustment happens to be needed, but the length of the bounding box can be adjusted in the same way as needed. The width of the bounding box 800 remains parallel to the road direction at the location of the bounding box 800, while the height remains perpendicular to the road surface at the location of the bounding box 800.

[0179] The example above assumes that the bounding box 800 remains bound to the 3D road model while adjustments are made. Although not shown, the annotation interface can also allow for “free” adjustments that are not constrained by the 3D road model, i.e., the annotator can also freely move or rotate the bounding box 800 as needed. This can be beneficial, for example, in cases where the annotation behavior deviates from the assumed behavior for vehicles at times (e.g., during turns or lane changes).

[0180] 2.3D Object Modeling:

[0181] Referring back Figure 3 , the object modeling component 310 implements a form of object modeling based on the output from the 3D annotation generator 308. As described above, an object model is a 3D model of a desired 3D structure (modeled as an object) created by isolating a subset of the point cloud defined within a 3D bounding object defined in the point cloud’s frame of reference (or equivalently determined as the intersection of the bounding box and the point cloud). The modeled object can correspond, for example, to a single real-world object (such as a vehicle, cyclist, or pedestrian to be annotated for training a structure detection component for use in an autonomous driving vehicle), a portion of a real-world object, or a group of real-world objects.

[0182] Figure 9A A flowchart showing a method of creating an object model from a point cloud.

[0183] At step 902, a point cloud capturing a structure to be modeled is received.

[0184] At step 904, a 3D bounding object in the form of a 3D bounding box is manually fitted to the required structure (object) captured in the 3D point cloud.

[0185] In this example, the 3D bounding box is manually adapted to fit the structure, for example in the manner described above with reference to Figures 8A-8E The inputs are provided by a human annotator, with the aim of achieving as tight a fit of the 3D bounding box to the required structure as possible, based on user inputs provided at the user interface 320.

[0186] Alternatively, the bounding box can be placed automatically. For example, the bounding box can be placed automatically based on a bounding box defined for another frame in the manner described above.

[0187] In another example, the bounding box can be generated automatically by a 3D structure detection component, such as a trained neural network.

[0188] Once the 3D bounding box has been placed, at step 906, a subset of the 3D point cloud is determined to be within the 3D bounding box. The 3D bounding box is defined in the point cloud frame of reference, so it is meaningful to determine which points in the 3D point cloud lie within the interior volume of the 3D bounding box. In most cases, these points will correspond to the required structure. As described above, this can be computed efficiently on the GPU by transforming the points to a coordinate frame whose axes are perpendicular to the faces of the bounding box.

[0189] Figure 9B A schematic diagram illustrating step 908 of the method described above is shown. In this example, a point cloud 400 has been captured from a first vehicle and a second vehicle, each of which is visible in the point cloud (labelled 902 and 904 respectively). In addition, points 906 have been captured of the surrounding road structure. By placing a tightly fitted 3D bounding box 800 around the second vehicle to the extent that it is visible in the point cloud 800, a subset of points within the 3D bounding box 800 can be isolated to provide a 3D model 912 of the second vehicle 902. It is often easier for an annotator to define a 3D bounding box around a structure element (in this case, the second vehicle) than to select a subset of points that belong to the required structure element.

[0190] In review Figure 9A Additional processing can be applied to the object model to refine and improve it.

[0191] For example, noise filtering can be applied to the determined subset of points (as indicated by step 910a). The purpose of noise filtering is to filter out "noise points", i.e. points that are unlikely to correspond to a desired structure. These points can be due to noise in the underlying sensor measurements used to derive the 3D point cloud, for example. The filtering can be K-Nearest Neighbour (K-NN) filtering to remove points that have an insufficient number of neighbouring points, for example (e.g. a point can be removed if the number of points within a defined radius of that point is below a threshold). The filtering can be applied according to filtering criteria that can be manually adjusted via the user interface 412 (e.g. the radius and / or threshold can be adjustable). More generally, one or more parameters of the modelling process, such as the filtering parameters, can be manually configured, as indicated in Figure 3 by the input from the UI 320 to the object modelling component 310.

[0192] In another example, the object model can be aggregated across multiple frames (as indicated by step 910b) in order to construct an aggregated object model.

[0193] In this regard, it is noted that the object modelling component 310 is capable of constructing "single-frame" and "aggregated" object models.

[0194] A single-frame object model refers to an object model derived from sensor data captured at a single time instant, i.e. from a single frame in the above sense. This can include an object model derived from a single point cloud, but also an object model derived from multiple point clouds captured at the same time. For example, multiple RGBD images can be captured simultaneously by multiple pairs of stereo cameras, and merged to provide a single merged point cloud.

[0195] A multi-frame object model refers to an object model derived from sensor data captured at multiple time instants, e.g. from RGBD images captured at different time instants, in which the object to be modelled is at least partially captured. An aggregated object model can be determined from two or more point clouds corresponding to different time instants by isolating a subset of points from each point cloud in the above sense, and then aggregating the point cloud subsets in a common frame of reference. This can provide a denser object model, but also effectively "fill in" parts of the object to be modelled that are obscured in one point cloud by using points from another point cloud captured at a different time instant.

[0196] In order to generate 2D or 3D annotation data, the object model can be applied to the frame or frames from which the object model was derived in order to generate annotation data for this frame or frames.

[0197] However, the object modelling component 310 is also capable of propagating the object model across frames. An object model that has been created using data from a point cloud of one frame is said to be propagated when it is applied to another frame (effectively propagating the point cloud data from one frame to another) in order to generate annotation data for this other frame. Single-frame and aggregated object models can be propagated in this sense.

[0198] As mentioned above, the purpose of propagating the object model can also be to generate augmented training examples.

[0199] Figure 9A Reference 910c in the figure indicates an optional surface reconstruction step, in which a surface mesh or other 3D surface model is fitted to the selective points. Such a 3D surface model can be fitted to the points selectively extracted from a single frame (single-frame object model) or multiple frames (aggregated object model). This effectively "smooths" the point cloud subset (single-frame or aggregated) into a continuous surface in 3D space. For this, a known surface fitting algorithm can be used, which can for example be based on a signed distance function (SDF) to minimize a distance metric between the extracted points and the reconstructed 3D surface. It will be appreciated that this description relates to an object model, which can include a 3D surface model generated in this way.

[0200] Figure 10 Two example frames are depicted, labeled 1001 (first frame) and 1002 (second frame), respectively. A first object and a second object (both vehicles) are visible in both frames, labeled 1021 (first vehicle) and 1022 (second vehicle), respectively. For each frame, a camera view (in the image plane of the relevant frame) and a top view (of the associated point cloud) are depicted from the left and right side, respectively.

[0201] In the second frame, the first vehicle is partially occluded by the second vehicle. In the first frame, which is captured at a later time instant in this example, the first vehicle is no longer occluded.

[0202] It can also be seen that in the second frame, both vehicles are at a greater distance. As a result, as schematically shown in the top view of the second frame, the point cloud data captured for each vehicle in the second frame (i.e. the number of points within the associated point cloud corresponding to the vehicle) is expected to be relatively less. Point cloud data for distant objects is also generally noisier and less accurate. One factor is that due to the inverse relationship between parallax and distance, a given error in parallax space translates into a larger error for a more distant point in distance space.

[0203] However, in the first frame, the first vehicle is clearly closer to the camera. As a result, as schematically shown in the top view of the first frame, the point cloud data for the first vehicle in the first frame is substantially denser and of higher quality (less error, less noise, etc.). Furthermore, the first vehicle is also more complete (i.e. encompasses more parts of the first vehicle) due to no longer being occluded.

[0204] 3. Object model propagation:

[0205] Two examples of object model propagation will now be described - namely automatic bounding box alignment (3.1) and manual bounding box alignment (3.2). They are illustrated usingFigure 10 The frames are described with reference to FIG. 10.

[0206] 3.1 Automatic bounding box alignment:

[0207] Figure 11A An annotation interface 700 is shown, with a first frame (1001, Figure 10 ) currently selected for annotation. Using the tools provided within the annotation interface 700, the annotator accurately places a tight bounding box 800 around the first vehicle (labelled 1021) in the manner described above. The tight bounding box 800 is defined in the point cloud frame of reference of the first frame.

[0208] Figure 11B The annotation interface 700 is shown, but this time a second frame (1002, Figure 10 ) is currently selected for annotation. The bounding box 800 defined in the first frame has been imported (propagated) into the second frame, but at this time only a coarse estimate pose 1121 (position and orientation) has been determined within the second frame. This coarse estimate pose 1121 is defined in the global reference but within the point cloud of the second frame.

[0209] The coarse estimate pose 1121 can be defined manually by the annotator. This process is simple and places minimal burden on the annotator.

[0210] Alternatively, the coarse estimate pose 1121 can be determined automatically, for example using a trained perception component - a form of Model in the Loop (MITL) processing.

[0211] Alternatively or additionally, the coarse estimate pose 1121 can be determined by interpolation based on an assumed or measured path of the first vehicle (1021). See below for details.

[0212] For rigid objects (i.e. objects modelled as rigid), the size and dimensions of the bounding box 800 remain constant throughout the annotation (in the sense that they are the same throughout the frames - the dimensions can be reflected throughout the frames by applying adjustments for one frame).

[0213] Non-rigid objects, such as pedestrians and cyclists, can be accommodated by applying appropriate transformations to the bounding object dimensions across frames. This can take into account information about the type or class of the relevant object, for example.

[0214] The coarse estimate of the bounding box pose 1121 is most conveniently obtained automatically by interpolation or MITL, such that it has the same "true" dimensions (width, length and height) of the coarse pose and orientation.

[0215] It should be noted that while tight bounding boxes are referenced in this context, it is not necessary that an initial tight 3D bounding box exist: one or more "loose" bounding boxes (i.e., they do not tightly fit to the object 1021 to be annotated) - which can be generated automatically or manually - can be sufficient to determine the vehicle structure that exists across multiple frames (and in turn apply the annotation functionality of the present disclosure). Thus, while the bounding box 800 can be referred to as a tight bounding box in the following description, the bounding box 800 need not be tight to implement the features described.

[0216] It should be noted that there is a distinction between a loose bounding box that is not accurately positioned or sized in any frame, and a loose pose of a bounding box in a given frame - for the latter, when a loose position is defined in a given frame, the accurate pose can or can not have been determined for a different frame.

[0217] If a tight bounding box is not initially provided, the annotator can at some time need to correct the orientation of the axes relative to the "optimized" box pose, and this can be done either before or after the bounding box 800 is propagated to other frames, and need only be corrected for one frame, as the corrections will automatically be applied to all frames to which the bounding box 800 is propagated.

[0218] Figure 11C A flowchart showing the object model propagation method is shown, along with a graphical illustration of the method steps. At step 1142, a point cloud subset of the first frame is extracted within the bounding box 800 of the first frame to provide an object model 1143 of the first vehicle. At step 1144, the object model 1143 from the first frame is fitted to a point cloud subset of the second frame. As described above, the fitting can be performed based on ICP or any other automatic alignment algorithm that attempts to match the structure of the object model to the structure of the point cloud. Any color information associated with the points in the point cloud can also be used as a basis for the fitting (in this case, attempting to fit the points of the model to points in the point cloud of similar color, in addition to the structural match). The alignment process can also be referred to as "registering" the 3D model 1143 with the point cloud of the second frame. The algorithm searches within the point cloud for matching structure to which the model 1143 can be aligned (i.e., to which it can be registered).

[0219] The loose bounding box pose 1121 can be used to limit the search range, e.g., to within a search volume defined in the 3D space defined by the loose bounding box pose 1121. The search volume can additionally be defined by the size / dimensions of the bounding box 800. However, the search volume need not be limited to the volume within the bounding box 800. For example, the search volume can be extended by an additional "buffer" around the 3D bounding box 800. Alternatively, the search volume can be manually defined, e.g., by a 2D rectangle or freeform "lasso" selection in the image, or by one of the projected 3D views. Alternatively, the search can be performed over the full extent of the point cloud of the second frame, although this can be inefficient.

[0220] Image features can also be used to assist in point cloud registration, such as edges, corners, or other feature descriptors like the Scale Invariant Feature Transform (SIFT) algorithm.

[0221] In general, while the object model 1143 is aligned in 3D space to the object (the first vehicle), this can or can not be based on 3D structure matching, i.e., adjusting the 3D pose of the object model 1142 to match the 3D features of the object model 1143 to corresponding 3D features of the first vehicle (e.g., using the ICP or other automatic 3D registration process described above). For example, instead or in addition, the alignment in 3D space can be based on 2D feature matching, i.e., adjusting the 3D pose of the object model 1142 to match the 2D features of the object model 1143 to corresponding 2D features of the first vehicle (e.g., using the type of image feature matching described above)

[0222] In another example, instead or in addition, the alignment in 3D space can be based on reprojection error or other photometric cost function. This involves projecting the 3D object model 1143 into the image plane and adjusting the 3D pose of the object model 1143 to match the computed projection to the first vehicle as appearing in the image. This can also be based on image feature matching between the image and the projection of the object model 1143 into the image plane.

[0223] In the case that noise filtering is applied to the points of the 3D object model, the 3D surface model can be fitted to the filtered points (i.e., the points remaining after the noise is filtered out).

[0224] Both the camera and the bounding box have a position and orientation (pose) relative to the world (global) coordinate system. Thus, once the pose of the bounding box 800 has been determined within the first frame, it is possible to find the bounding box 800 relative to another camera (i.e., the same camera, but at a different time instance - e.g., corresponding to the second frame), which in turn allows the bounding box 800 to be placed (positioned and oriented) in the point cloud coordinate system of that camera. The point cloud can in turn be transformed into the coordinate system of the bounding box in order to effectively isolate the point cloud subset within the bounding box (see above).

[0225] Referring to Figure 11D, the tight bounding box 800 and the 3D object model 1143 are derived in the same frame of reference (in this case, the frame of reference of the bounding box 800), so the position and orientation of the tight bounding box 800 relative to the object model 1143 is known at the time the model is created at step 1142. Thus, when fitting the 3D object model 1143 to the point cloud of the second frame, at step 1144, the position and orientation of the tight bounding box 800 in the frame of reference of the point cloud of the second frame is automatically determined. This is encoded as a reference point (position) 1152 and an orientation vector 1154 of the bounding box 800, where the reference point 1152 and the orientation vector 1154 are fixed relative to the points of the object model 1143. Assuming the relevant object (in this case, the first vehicle) can be treated as a rigid body, the tight bounding box 800 initially defined in the first frame will now be reasonably accurately positioned and oriented so as to tightly fit the subset of points in the second frame that belong to the first vehicle (the accuracy will depend on the degree of fitting of the object model to the point cloud).

[0226] In this way, high quality 3D annotation data is semi-automatically generated for the second frame, which in turn can be used to train a machine learning 3D bounding box detector, orientation network, or for example, distance estimation or any other form of 3D structure perception component.

[0227] Further steps can also be taken to automatically generate 2D annotation data for the underlying image of the second frame, as described below.

[0228] For the reasons described above, the rough estimate of the pose of the bounding box 1121 is also used as a rough estimate of the pose of the 3D object model 1143. This is in turn refined by automatically, manually or semi-automatically aligning the 3D object model 1143 better with the first vehicle in the second frame.

[0229] Although in the above example, the bounding box 800 is placed manually in the first frame, this step can be automatic. In the MITL approach, the bounding box can be placed automatically by an automatic object detector (such as a trained neural network), possibly subject to manual fine-tuning. For example, a bounding box detector can perform well on the first frame for the first vehicle, but perform poorly when applied directly to the second frame. In this case, taking advantage of the good performance of the bounding box detector on the first frame, high quality training data can be automatically or semi-automatically generated for the second frame. This in turn can provide high quality training data for the second frame, which in turn can be used to train / retrain to improve the object detection performance. Additionally or alternatively, the pose can be estimated by interpolation based on the measured or assumed path of the object being annotated (see below).

[0230] It should also be understood that this is merely one example of the workflow an annotator can employ using the provided annotation capabilities. The potential efficiency stems from the fact that changes made to the 3D bounding box relative to the object model in one frame are automatically applied to one or more other frames to keep the 3D bounding box of a rigid object consistent across frames. Thus, for example, an annotator might initially perform a coarse annotation on the first frame, apply the above steps to position and orient the coarse bounding box of the first frame in the second frame, and then apply adjustments to the positioning and / or orientation of the bounding box in the first frame, which are automatically reflected in the second frame. In this case, adjustments to the bounding box in one frame are automatically applied across multiple frames.

[0231] It should be noted that, in the sense that the orientation of bounding box 800 relative to object model 1143 remains consistent across frames, the reference point 1152 and orientation vector 1154 of bounding box 800 relative to object model 1143 are fixed—however, the annotator can adjust the position and orientation of the bounding box relative to the object model as needed (i.e., he can change the reference point 1152 and orientation vector 1154), and any such adjustments will be automatically implemented in all frames to which the object model is applied. In this sense, 3D bounding box 800 is "locked" to 3D object model 1143.

[0232] 3.2 Manual Bounding Box Alignment

[0233] Following the previous example, Figure 11E The expanded view of the annotation interface 700 is shown, with the second frame selected for annotation. It can be seen that the object model 1143 is superimposed on the camera view and the top view through projection (1152) so that the annotator can see the relative position of the object model 1143 with respect to the related object 1021 (the first vehicle) in the reference frame of the second frame.

[0234] Figure 11F This demonstrates how commentators utilize this feature. For example... Figure 11F As shown in the upper part, when object model 1143 is first fitted to the point cloud of the second frame, it may not be perfectly aligned with the actual points of object 1021. This is in Figure 11F It is easy to see that the reason is the visual misalignment between object model 1143 and the actual object 1021. Therefore, as Figure 11F As shown in the lower part, the annotator can fine-tune the pose (localization and orientation) of bounding box 800 to correct misalignment: in the sense described above, as bounding box 800 is adjusted, object model 1143 remains locked to bounding box 800, and any changes in the pose of object model 1143 in the current point cloud reference frame apply to the pose of 3D bounding box 800 in that reference frame. Therefore, when no perceptible misalignment exists, the annotator knows that bounding box 800 has been correctly localized and oriented in the reference frame of the second frame. Although Figure 11FAlthough not shown in the diagram, object model 1143 is also projected into the top view, allowing the annotator to simultaneously correct any visual misalignments in that top view.

[0235] In comparison, Figure 11G Showing with Figure 11F The upper half is the same view, but the object model 1143 is not overlaid. At this point, the bounding box 800 is still misaligned, but this misalignment is more difficult to perceive without model projection. This situation is partly due to partial occlusion of the first vehicle. By propagating the object model 1143 from frames where the object is not occluded, the occlusion in the second frame is corrected, thus assisting the annotator in fine-tuning the bounding box 1143.

[0236] Besides correcting occlusion, propagating object models in this way also helps address sparsity, noise, and inaccuracies in point clouds, especially for distant objects. (Review) Figure 10 To reiterate, the first vehicle is farther away in the second frame, so the point cloud subset corresponding to the second vehicle will typically be sparser and of lower quality in the second frame. This is another reason why it can be difficult to manually place the bounding box accurately in the second frame, as this will be reflected in the quality of the top view. Propagating object model 1143 from the first frame to the second frame in the manner described above helps annotators compensate for this.

[0237] Model propagation can also be used to address "gap" in sensor coverage. For example, with stereo depth images, depth estimation relies on locating matching pixels between a target image and a reference image. The target image often contains regions where some pixels have no corresponding pixels in the reference image. This corresponds to a portion of the scene within the field of view of the camera capturing the target image but outside the field of view of the camera capturing the reference image. For the object portion within this region in a given frame, no depth information will be available. However, this depth information can be obtained from another frame by propagating the object model from other frames as described above. This could be, for example, a frame with the closest temporal proximity, where the relevant object is fully visible in the depth channel.

[0238] 4. Aggregate object model:

[0239] To create an aggregated object model, bounding boxes are placed around the relevant objects across multiple frames (e.g., as described above, or simply placed manually). For each frame, a subset of the point cloud is isolated within the bounding box of that frame, and then aggregated in a common frame of reference. As described above, this provides a denser and less noisy object model, which can then be applied to one or more frames to generate high-quality annotation data.

[0240] Following Figures 11A-11GIn the above example, having placed the bounding box 800 accurately in the second frame, a subset of points within the bounding box of the second frame can be extracted and merged (aggregated) with the corresponding subset of points extracted from within the bounding box 800 in the first frame. This provides a denser model of the first vehicle, which can in turn be propagated into other frames in the manner described above.

[0241] The annotator can also use the aggregated model to guide manual annotation. When the annotator adjusts the position / pose of the bounding box 800 in the target frame, the aggregated model using data from the target frame and at least one other frame (reference frame) can be rendered. If the bounding box is not positioned or oriented correctly, this can result in visible artefacts in the aggregated model, such as "duplicated" or misplaced features, because points taken from the target frame are not correctly registered with points taken from the reference frame. The user can then fine-tune the pose of the bounding box 800 in the target frame as necessary until there are no longer visual artefacts.

[0242] 4.1 Iterative propagation and generation of aggregated models

[0243] The aggregated 3D model can be generated and applied in an iterative manner.

[0244] An example will now be described with reference to Figure 14 The figure shows a method flow diagram for the iterative generation and application of increasingly dense aggregated 3D models across multiple frames (potentially a large number of frames).

[0245] First, at step 1402, a single-frame object model is generated for an initial single frame (the current object model at this point in the process), by placing a 3D bounding box around the object in the frame (automatically, semi-automatically or manually) and extracting object points from within the 3D bounding box in the frame.

[0246] At step 1404, the current object model is propagated to the next frame and aligned with the object in the frame in 3D space (1406) (manually, automatically or semi-automatically). In doing so, the pose of the 3D bounding box in the frame is derived, so at step 1408, points belonging to the object can be extracted from the 3D bounding box in the frame. At this point, these points can again be aggregated (1410) with the object points of the current object model to generate a new aggregated 3D model incorporating the point information extracted from the most recent frame.

[0247] The process can now be repeated iteratively for the next frame, starting at step 1404 - note that from this point onwards, the current object model propagated to the next frame is an aggregated object model incorporating point information from multiple frames. Thus, from this point onwards, it is the current aggregated object model that is aligned with the object in the next frame. With each iteration of the process, points from yet another frame are added, allowing a progressively denser and complete aggregated object model to be generated.

[0248] 5. Automatic / semi-automatic 2D annotation data generation:

[0249] As mentioned above, in addition to quickly generating 3D annotation data and alleviating the human effort, the work of the annotators can also be used to generate high quality 2D annotation data, without or with only minimal additional human input.

[0250] Figure 12A A flowchart showing the method of automatically generating 2D annotation data, as well as a graphical illustration of the method steps. The method is implemented by the 2D annotation generator 312.

[0251] For ease of reference, the first frame is depicted in the top left corner. The first vehicle is the object to be annotated in this example, labeled with reference numeral 1021. The 3D model of this vehicle is determined by isolating a subset of the point cloud 400 within the bounding box, as described above in step 1002, which is projected into the image plane of the relevant frame. The projection of the point cloud subset (i.e. the 3D model of the relevant vehicle) is indicated by reference numeral 1204 in the bottom left image. As can be seen, the projection of the point cloud 1004 coincides with the expected object 1000.

[0252] In step 1206, 2D annotation data for annotating the image 702 is automatically generated based on the projection 1204 of the 3D model into the image plane.

[0253] The 2D annotation data can be in the form of a 2D segmentation mask 1208 (top right), which substantially matches the region of the object within the image plane (i.e. it at least approximately depicts the object in the image plane, tracking one or more boundaries of the object). Such annotation data can be used to train a segmentation component to perform instance segmentation, i.e. pixel-level classification of an input image, where each pixel of the input image is individually classified. In this example, the annotated object 1021 belongs to a particular object class (such as "car" or "vehicle"), and the image 702 in combination with the segmentation mask 1208 can be used to train a segmentation component to, for example, label image pixels as "car / non-car" or "vehicle / non-vehicle", depending on whether these pixels lie within the region of the segmentation mask 1208.

[0254] The projection 1204 of the 3D object model is a point-based projection, which can be too sparse to be used directly as a usable segmentation mask (but this problem can be mitigated by using an aggregated object model). To generate the segmentation mask 1208, a prediction model such as a conditional random field (CRF) can be applied to the projected points in order to fill and smooth the projection 1204, converting it into a usable segmentation mask that accurately defines the object region in the image plane. In this context, the projection 1204 of the 3D model is used as a sparse prior from which the prediction model is inferred in order to predict the complete segmentation mask of the object. Optionally, the parameters of the prediction model can be adjusted via the UI 320 in order to achieve a desired result. More generally, the annotator can adjust the parameters of the CRF and / or superpixel segmentation parameters, etc. to achieve manual corrections. This can be a post-processing step applied after the annotation data has been generated.

[0255] As another example, the 2D annotation data can be in the form of a tight-fitting 2D bounding box 1210 (bottom left). This is generated by fitting a 2D bounding box (a rectangle in the image plane) to the projection 1204 of the 3D model. This can in turn be used in conjunction with the image 702 itself to train a 2D bounding box detector. Once trained, such a component can automatically detect and localize structures within an image by automatically generating a 2D bounding box for an image received at inference time.

[0256] An alternative approach is to simply project the 3D bounding box into the image plane, and then fit a 2D bounding box to the projection of the 3D bounding box. However, this generally does not result in a tight 2D bounding box: as shown in the top right image of Figure 12A the shape of the 3D bounding box 800 does not match the shape of the vehicle, so the edges of the projected 3D bounding box do not generally coincide with the edges of the vehicle as appearing in the 2D image plane.

[0257] Once generated, the annotator can fine-tune the 2D annotation data as needed, as shown by the input from the UI 320 to the 2D annotation generator 312 in Figure 3 .

[0258] As mentioned above, the 3D object model can be in the form of a 3D surface model fitted to the relevant extracted points. In this case, the 3D surface model is projected into the image plane to create the 2D annotation data. Thus, the projection is of a continuous 3D surface, which can provide higher quality 2D annotation data compared to the projection of a discrete (possibly sparse) set of 3D structure points.

[0259] 5.1 Occluded objects:

[0260] 2D annotation data can be generated for occluded objects using either a single-frame or an aggregated object model. The annotator can choose between these two options as appropriate via the UI 320.

[0261] Figure 12B Taking the second frame as an example, in which the first vehicle is partially occluded by the second vehicle. In this example, the single-frame 3D model of the first vehicle is determined based on the point cloud of the second frame only by isolating the subset of the point cloud that lies within the 3D bounding box placed around the first vehicle (not shown). Thus, the single-frame model of the first vehicle comprises only the non-occluded points of the first object 1220. Figure 12B Figure 12B The single-frame model of the first vehicle represented by reference numeral 1224 in the lower left of the figure comprises only the non-occluded points of the first object 1220. Thus, when the single-frame 3D model 1224 is projected back onto the image plane and used to automatically generate 2D annotation data, this 2D annotation data will only label the visible portion of the occluded object 1220. The projection of the model is represented by reference numeral 1204a towards the lower left of the figure. The effect of this on the 2D annotation data is shown on the right of the figure, showing a segmentation mask 1232 (upper right) generated based on the projection 1204a of the single-frame model. As can be seen, this only covers the area of the visible portion of the occluded object 1232. Similarly, when a 2D bounding box 1234 (lower right) is fitted to the projection of the single-frame model, this bounding box will fit tightly to the visible portion of the occluded object 1220.

[0262] Figure 12C A second example is shown, in which the object model 11 propagated from the first frame is used instead. In this case, the occluded portion of the object 1220 is "filled in" using point cloud data from one or more related frames in which this portion of the object is not occluded. From the left of the figure, it can be seen that when such a model is projected back onto the image frame, the occluded portion of the object is "restored". The projection of the propagated object model 1143 is labelled 1204b. Figure 12C From the left of the figure, it can be seen that when such a model is projected back onto the image frame, the occluded portion of the object is "restored". The projection of the propagated object model 1143 is labelled 1204b.

[0263] Thus, when this projection 1204b is used as the basis for a segmentation mask represented by reference numeral 1242 (upper right), this covers the entire area of the object 1220, including the occluded portion. This can not be ideal in practice, but can still be desirable given the other advantages (greater density, less noise, etc.) that using a propagated (e.g. aggregated) model would provide. In this case, the aggregated model can be determined in the same way for the occluded object, the 3D model of the occluded object can be used to "block" the projection of the 3D model onto the image plane, and ensure that the segmentation mask only covers the non-occluded portion of the object that is visible in the image plane.

[0264] Similarly, when a 2D bounding box 1244 (lower right) is fitted to the projection of the propagated object model, this bounding box will fit to the overall object, including the occluded portion; depending on the positioning of the occluded portion, this can result in the 2D bounding box extending beyond the visible portion of the object - for example, in the case of the object 1220, the 2D bounding box 1244 extends beyond the visible portion of the object 1220. Figure 12C ​At the bottom right, the left edge of the bounding box 1244 can be seen extending beyond the leftmost visible part of the object 1220 to encompass the leftmost occluded part of the object 1220.

[0265] Similar effects can be achieved by using aggregated object models, e.g. generated by aggregating points from the point clouds corresponding to the first and second frames of the first vehicle in the manner described above and applying the aggregated model to the second frame.

[0266] 6. Interpolation based on vehicle path

[0267] Reference is made to UK patent application GB1815767.7, the entire contents of which are incorporated herein by reference. This document discloses a method of inferring a path of an external vehicle based on a reconstructed path of the vehicle itself.

[0268] In the present context, the reconstructed path of the vehicle itself can be used to automatically determine an initial coarse estimate of the pose 1121 of the bounding box 800 Figure 11B ).

[0269] With reference to Figure 13A and Figure 13B , the optional additional feature allows the vehicle path precision to increase as the position estimate is refined.

[0270] Figure 13A A camera is shown at known poses at respective time instants t1, t2 along a reconstructed path of the vehicle itself, denoted EP (Ego Path). Based on the ego path EP, an object path (OP) has been inferred for the object to be annotated (the first vehicle described above). Based on the time instants t1 and t2, the respective poses of the object can be preliminarily inferred by interpolating from the object path OP, as P1 and P2. This provides a starting point for manually or automatically registering a 3D model of the first vehicle with the point clouds at time instants t1 and t2 respectively.

[0271] Figure 13B Refined (more accurate) poses P1 ', P2' are shown determined by aligning the 3D model 1143 with the point clouds at time instants t1 and t2 respectively in the manner described above.

[0272] Additionally, as shown in Figure 13B , once those more accurate poses P1 ', P2' are known, an updated vehicle path OP' can be determined based on them.

[0273] This can be used for various purposes - for example, to provide more accurate initial pose estimates for other frames.

[0274] Information about the vehicle path can also be incorporated into the structure matching process to penalize changes in the pose of the 3D bounding box 800 / 3D model 1143 that deviate from an expected vehicle behaviour model, i.e. that result in unexpected changes in the vehicle path.

[0275] In the example case in Figure 13C , the poses P1" and P2" at t1 and t2 respectively can happen to provide a good registration of the 3D model 1143 with the points at t1 and t2 respectively. However, these poses P1" and P2" imply an unrealistic path OP" for the first vehicle, which should be penalized according to the expected behavior model.

[0276] This dependence on expected behavior can be incorporated into a cost function that rewards good registration but penalizes unexpected changes in the vehicle path. An automated alignment process that applies the model 1143 to the point clouds of the relevant frames to optimize the cost function. Thus, in the example of Figure 13C , if the cost function assigns too high a penalty to the path OP", an alternative pose can be selected instead.

[0277] 6.1 Iterative path refinement

[0278] The above principles can be applied in an iterative fashion, by iteratively building from an initial coarse annotation, i.e. by creating an initial annotation, aligning poses, refining the motion model, repeating the refinement of more poses based on the refined motion model, etc.

[0279] As mentioned above, the coarse annotation can be provided in the following ways:

[0280] 1. A model in 2D or 3D in a loop (e.g. a neural network or a moving object detector);

[0281] 2. A coarse kinematic model of the bounding box (e.g. constant velocity);

[0282] 3. By providing a "single-click" object function point or a 2D "lasso" function around points;

[0283] d. Minimizing a cost function that takes into account 2D error, 3D error and possible behavior.

[0284] Referring to Figure 2An instance of a perception component 102 refers to any tangible embodiment of one or more underlying perception models of the perception component 102, which can be a software or hardware instance or a software and hardware combined instance. Such an instance can be embodied using programmable hardware such as a general purpose processor (e.g. a CPU, an accelerator such as a GPU, or the like) or a field programmable gate array (FPGA) or any other form of programmable computer. Thus, a computer program for programming a computer can take the form of program instructions on a general purpose processor, circuit description codes for programming an FPGA, or the like. An instance of a perception component can also be implemented using non-programmable hardware such as an application specific integrated circuit (ASIC), which can be referred to herein as a non-programmable computer. In general, a perception component can be embodied in one or more computers, which can be programmable or non-programmable, and programmed or otherwise configured to perform the perception component 102.

[0285] Referring to Figure 3 , Figure 3 The components 302-314 in the annotation computer system 300 are functional components that can take various forms for implementation in hardware: although Figure 3 not shown, the annotation computer system 300 includes one or more processors (computers) that perform the functions of the components described above. The processors can take the form of general purpose processors such as central processing units (CPUs) or accelerators (e.g. GPUs), or more specialised hardware processors such as field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs). Although not shown separately, the UI 320 typically includes at least one display and at least one user input device for receiving user input to allow an annotator to interact with the annotation system 300, such as a mouse / trackpad, touchscreen, keyboard, or the like. Referring to Figure 5 , the image corrector 504, the depth estimator 506 and the depth transform component 508 are functional components of the stereo image processing system 500 that can take different forms for implementation in hardware. For example, the functions of the stereo image processing system 500 can be implemented in software, i.e. by computer code executed on one or more processors such as CPUs, accelerators (e.g. GPUs), or in hardware (e.g. FPGA fabric) and / or application specific integrated circuits (ASICs) or using a combination of hardware and software. By way of example, UK patent applications GB1807392.4 and GB1817390.6 disclose examples of FPGA-based implementation in semi-global matching, which are incorporated herein in their entirety and are applicable in the present context.

[0286] It will be appreciated that the above is by way of example only. Other aspects and embodiments of the disclosure are described below.

[0287] 3D to 2D

[0288] A first aspect (Aspect A) of the present disclosure provides a computer-implemented method of creating 2D annotation data for annotating one or more perception inputs, the method comprising: in an annotation computer system, receiving at the annotation computer system at least one captured frame (a first frame) comprising a set of 3D structure points, the frame capturing at least a portion of a structure component; computing a reference position of the structure component within the frame; generating a 3D model of the structure component by selectively extracting 3D structure points of the frame based on the reference position; computing a projection of the 3D model onto an image plane; storing the computed 2D annotation data of the projection in a persistent computer storage device to annotate the structure component within the image plane.

[0289] Embodiments of Aspect A can provide one or more of: manual annotation, automatic annotation, and semi-automatic annotation.

[0290] In an embodiment of Aspect A (Embodiment Al), the 2D annotation data can be stored in association with at least one perception input of the frame to annotate the structure component therein, and the projection can be computed based on the computed reference position within the frame. That is, the 2D annotation data can be created for the first frame used to generate the 3D model by applying the 3D model to the same frame.

[0291] Some such embodiments can further create 3D annotation data for the first frame, wherein the 3D annotation data comprises or is derived from the reference position. Preferably, the 2D and 3D annotation data are created for the first frame using a common set of annotation operations.

[0292] In an alternative embodiment of Aspect A (Embodiment A2), the 2D annotation data can be stored in association with at least one perception input of a second frame to annotate the structure component in the at least one perception input of the second frame, the second frame capturing at least a portion of the structure component. That is, the 3D model can be generated from the first frame (or a combination of the first frame and the second frame in the case of an aggregated model) and applied to the second frame to create the 2D annotation data for the second frame. As the term is used herein, this is an example of "model propagation".

[0293] In the context of Embodiment Al, the structure component can be referred to as a common structure component (shared by both frames). It should be noted in this regard that all descriptions relating to a common structure component captured in multiple frames equally apply to a structure component captured in one or more frames as described in Embodiment Al, unless the context otherwise requires.

[0294] In the general context of aspect A, the first frame used to generate the 3D model can be referred to as a "reference frame", and the term "target frame" can be used to refer to the frame for which annotation data is created. It should be noted that in the context of embodiment A1, the first frame is both a target frame and a reference frame. In the context of embodiment A2, the second frame is a target frame.

[0295] In embodiments of aspect A, the 3D model can also be used to create 3D annotation data to annotate structural components in the 3D space.

[0296] For example, 3D annotation data can be created to annotate structural components in at least one perception input of the second frame in which at least a portion of the structural components is captured. That is, the 3D model can be generated from the first frame and applied to the second frame in order to create 3D annotation data for the second frame.

[0297] 2D or 3D annotation data can be created to annotate at least one perception input of the second frame (the frame for which 2D and / or 3D annotation data is generated) by computing an aligned model position of the 3D model within the second frame (see below).

[0298] Alternatively, 2D annotation data can be created for the target frame by projecting the 3D model generated from the reference frame into the image plane associated with the target frame based on the aligned model position determined within the target frame. This means that the projection derived from the selectively extracted points of the reference frame is used to create 2D annotation data for the target frame.

[0299] Alternatively, creating 2D annotation data for the target frame can be by generating a second 3D model using the aligned model position (as determined using the 3D model generated from the reference frame), selectively extracting 3D structure points of the target frame based on the aligned model position, and then projecting the second 3D model (as generated from the target frame) into the image plane associated with the target frame. In this case, the 2D annotation data comprises or is derived from the projection of the second 3D model, which is generated from the target frame but positioned using the 3D model generated from the reference frame.

[0300] In another example, the second 3D model can be an aggregated 3D model generated by aggregating the selectively extracted 3D structure points from the target frame and the reference frame.

[0301] The selectively extracted 3D structure points can be selectively extracted from the frame used to generate the 3D model based on the reference position and one or more boundary object dimensions.

[0302] The one or more boundary object dimensions can be one of:

[0303] (i) manually determined based on one or more size inputs received at a user interface;

[0304] (ii) automatically determined by applying a perception component to the frame;

[0305] (iii) semi-automatically determined by applying a perception component to the frame and further based on one or more dimensional inputs; and

[0306] (iv) assumed.

[0307] The selectively extracted 3D structure points can be a subset of points within a 3D volume defined by the reference location and one or more bounding object dimensions.

[0308] The 3D annotation data described above can further include one or more bounding object dimensions used to generate the 3D model or a transformation thereof (thereby defining a 3D bounding box for the applicable perception input).

[0309] The second model described above can be generated from the target frame based on the aligned model location and the same one or more bounding object dimensions (for annotating a rigid common structure component) or a transformation thereof (for annotating a non-rigid common structure component).

[0310] Model propagation

[0311] The second and third aspects of the disclosure (Aspects B and C, respectively) each provide a computer-implemented method of creating one or more annotated perception inputs, the method comprising: in an annotation computer system, receiving a plurality of captured frames, each frame comprising a set of 3D structure points, wherein at least a portion of a common structure component is captured; computing a reference location within a reference frame of the frames; generating a 3D model of the common structure component by selectively extracting 3D structure points of the reference frame based on the reference location within the frame; determining an aligned model location of the 3D model within a target frame; storing annotation data of the aligned model location in a computer memory in association with at least one perception input of the target frame so as to annotate the common structure component therein.

[0312] According to Aspect B, the determination of the aligned model location is based on:

[0313] (i) one or more manual alignment inputs received at a user interface in relation to the target frame, while rendering the 3D model to manually align the 3D model with the common structure component in the target frame.

[0314] According to Aspect C, the determination of the aligned model location is based on:

[0315] (ii) an automatic alignment of the 3D model with the common structure component in the target frame.

[0316] In some embodiments, the automatic alignment can match features (2D or 3D) of the 3D model to features (2D or 3D) of the common structural component. However, the subject matter of aspect C is not limited as such and automatic alignment forms are possible (see more examples below).

[0317] Example annotation data

[0318] The term "annotation data of the aligned model position" refers to annotation data that includes or is otherwise derived using the aligned model position.

[0319] For example, the annotation data of the aligned model position can include position data of the aligned model position for annotating a position of the common structural component in at least one perception input of the target frame. Such position data is "directly" derived from the aligned model position (subject to any geometric transformation to a suitable frame of reference as needed), i.e. once the aligned model position is determined using the 3D model, it no longer plays a role in creating such annotation data.

[0320] The position data may, for example, be 3D position data (a form of 3D annotation data) for annotating a position of the common structural component in 3D space.

[0321] Alternatively or additionally, the annotation data of the aligned model position can include annotation data derived from the 3D model using the aligned model position (derived annotation data). That is, the 3D model can be used both to determine the aligned model position and, once the aligned model position is determined, to derive annotation data from the 3D model itself.

[0322] As another example, a 3D model generated from the reference frame (a first 3D model) can be used to determine the aligned model position in the target frame. Then, a second 3D model can be generated from the target frame using the aligned model position (see above). Thus, in this case, the annotation data of the aligned model position can include annotation data derived from the second 3D model using the aligned model position.

[0323] An example of derived annotation data is 2D annotation data derived by projecting the applicable 3D model into the image plane based on the aligned model position. Such 2D annotation data may, for example, include a 2D bounding object fitted to the projection of the 3D model into the image plane, or include or be derived from a computed segmentation mask of the projection.

[0324] In embodiments of aspect B and aspect C, the annotation data can be 2D annotation data, 3D annotation data, or a combination of 2D annotation data and 3D annotation data stored in association with one or more perception inputs of the target frame (i.e. each form of annotation data can be stored in association with the same perception input in a respective different perception input of the target frame).

[0325] The annotation data can comprise refined annotation data computed by applying a prediction model from the 3D point cloud subset.

[0326] In the case of 3D annotation data, the prediction model can be applied to the 3D model itself.

[0327] In the case of 2D annotation data, the prediction model can be applied to the 3D model itself (before its projection) or to the computed projection of the 3D model in the image plane.

[0328] Whether the prediction model is applied to the 3D model or to the computed projection in the case of 2D annotation data, the prediction model leads to such refined annotation data.

[0329] The refined annotation data can for example have the effect of providing a "filled" or "smoothed" annotation of the structural components.

[0330] The prediction model can be a Conditional Random Field (CRF).

[0331] Model alignment

[0332] Embodiments of aspect B can provide one or both of manual annotation (i.e. only (i)) and semi-automatic annotation (i.e. based on a combination of (i) and (ii)) by propagating the 3D model into the target frame.

[0333] Embodiments of aspect C can provide one or both of automatic annotation (i.e. only (ii)) and semi-automatic annotation (i.e. based on a combination of (i) and (ii)) by propagating the 3D model into the target frame.

[0334] That is, the alignment model position is determined manually (only based on manual alignment input), automatically (only based on automatic alignment) or semi-automatically (based on manual alignment input and automatic alignment) by aligning the 3D model with the (partial) common structural components captured in the second frame, as appropriate.

[0335] Embodiments of aspect A A2 can be manual, automatic or semi-automatic, i.e. based on (i), (ii) or a combination of (i) and (ii).

[0336] In either case, the 3D model can be an aggregated 3D model determined by aggregating 3D structural points selectively extracted from two or more frames.

[0337] Aggregated model

[0338] The 3D model can be an aggregated 3D model determined by aggregating data points selectively extracted from the reference frame with data points extracted from the target reference frame, wherein the automatic alignment causes the aggregated 3D model to match the common structural component in the target frame by matching 3D structure points of the 3D model extracted from the reference frame with the common structural component in the target frame.

[0339] The method can include generating an aggregated 3D model by selectively extracting 3D structure points from the target frame based on the aligned model position and aggregating them with the points selectively extracted from the first frame.

[0340] Alternatively or additionally, the 3D model can be an aggregated 3D model determined by aggregating data points selectively extracted from the reference frame with data points extracted from at least a third frame in the set of frames other than the target frame and the reference frame.

[0341] It will of course be appreciated that an aggregated model can be generated from more than two frames (and potentially many more to build a dense aggregated 3D model).

[0342] The method can include the step of applying noise filtering to the aggregated 3D structure points to filter out noise points therefrom in generating the aggregated 3D model.

[0343] Alternatively or additionally, the aggregated 3D model comprises a 3D surface model fitted to the aggregated 3D structure points (in the case that noise filtering is applied, this can be fitted to the filtered 3D structure points, i.e. the 3D structure points from which noise points have been filtered out).

[0344] Alternatively or additionally, the method can include the step of applying a prediction model in dependence on the aggregated 3D surface points. For example, applying a prediction model to the aggregated 3D surface points to generate a 3D model and / or applying a prediction model to a 2D projection of the aggregated 3D model to create a segmentation mask or other 2D annotation data.

[0345] Whilst noise filtering, prediction modelling and / or surface fitting can be applied to both single frames and aggregated 3D models, it is particularly advantageous to apply one or more of these to the aggregated 3D model. For noise filtering, noise points within the aggregated set of structure points are more sparse than points that actually belong to the common structural component, enabling more accurate filtering. For prediction modelling, the aggregated points provide a stronger prior.

[0346] Further examples of aggregated model features are provided below.

[0347] Manual / semi-automatic alignment

[0348] The aggregated 3D model can be rendered effectively to assist in manually aligning the aggregated 3D model with the common structural component in the target frame.

[0349] For example, in a manual or semi-automatic alignment scenario, when one or more manual alignment inputs are received at the user, the aggregated 3D model can be updated and re-rendered to align the second reference location with the common structural component in the second frame, with the effect of correcting visual artifacts in the rendered aggregated 3D model due to the initial misalignment of the second reference location.

[0350] Such visual artifacts are caused by misalignment of the model location within the target frame relative to the reference location in the reference frame. For example, the annotator can see repeated or misaligned structural elements, features, etc. in the aggregated 3D model. By adjusting the model location until those artifacts are no longer visible, the annotator can find the correct model location within the target frame.

[0351] In this scenario, it can be sufficient to simply render the 3D aggregated model for manual alignment of the aggregated 3D model with the common structural component in the target frame - it is not actually necessary to actually render any portion of the target frame itself with the aggregated 3D model. In practice, it can be convenient to render the aggregated 3D model within the target frame so that the annotator can see the effect of the adjustment by another means. In some cases, it can be provided an option to render an enlarged version of the aggregated 3D model, which the annotator can elect to use for the final adjustment.

[0352] Aligning the model location can be determined based on the one or more manual alignment inputs, rather than using any automatic alignment.

[0353] Further disclosure is provided below regarding the aggregated 3D model.

[0354] Automatic / semi-automatic model alignment

[0355] The automatic alignment can include Iterative Closest Point.

[0356] Additionally or alternatively, the automatic alignment can use at least one of: color matching, 2D feature matching, and 3D feature matching.

[0357] Additionally or alternatively, the automatic alignment can include: computing a projection of the 3D model to a 2D image plane associated with the target frame, and adjusting the model location in 3D space to match the projection to 2D features of the common structural component within the 2D image plane.

[0358] For example, the model location can be adjusted to minimize a re-projection error or other photometric cost function.

[0359] For example, the target frame can include depth component data of a 3D image, and the projection matches to 2D features of the common structural component captured in a color component of the 3D image.

[0360] The aligned model position can be determined automatically without any manual alignment input.

[0361] Some such embodiments can still operate based on an initial coarse estimate followed by fine tuning.

[0362] That is, determining the aligned model position can be by initially estimating the model position within the target frame and then applying automatic alignment to adjust the estimated model position.

[0363] While in semi-automatic alignment, the model position can be initially estimated as a manually defined position, represented by one or more manual position inputs received at the user interface, in full automatic alignment, the model position is automatically initially estimated.

[0364] The model position can be initially estimated by applying the structure perception component to the target frame.

[0365] Automatic / semi-automatic model alignment based on structure component path

[0366] In another example, the frames can be time-sequential frames and the model position can be automatically initially estimated based on a (common) structure component path within a time interval of the time-sequential frames.

[0367] Further, the common structure component path can be updated based on the automatic alignment applied to the target frame.

[0368] The updated common structure component path can be used to compute a position of the common structure in one of the frames other than the target frame.

[0369] The method can comprise the step of storing the computed position for said one frame as 2D or 3D annotation data for annotating the common structure component in at least one perception input of the frame.

[0370] The automatic alignment can be performed to optimize a defined cost function that rewards matching of the 3D model to the common structure component while penalizing unexpected behavior of the common structure component as defined by an expected behavior model of the common structure component.

[0371] For example, the defined cost function can penalize unexpected changes in the common structure component path as defined by the expected behavior model.

[0372] This advantageously combines knowledge of measured or assumed behavior to provide more reliable alignment across frames.

[0373] The common structure component path can also be used to compute a reference position within a reference frame to generate the 3D model (before updating the path).

[0374] Semi-automatic model alignment

[0375] The aligned model position can be determined semi-automatically by initially estimating the model position automatically and then aligning the estimated model position according to one or more manual alignment inputs.

[0376] That is, a rough alignment can be performed automatically and then adjusted manually.

[0377] Alternatively or additionally, the aligned model position can be initially estimated by applying a structure perception component to the target frame.

[0378] Alternatively or additionally, in case the frames are time-sequential frames, the model position can be initially estimated based on a common structure component path within a time interval of the time-sequential frames.

[0379] Alternatively or additionally, the model position can be initially estimated based on an automatic alignment of the 3D model with a common structure component in the target frame.

[0380] Alternatively or additionally, the aligned model position can be determined semi-automatically by initially estimating the model position according to one or more manual alignment inputs and then aligning the estimated model position according to an automatic alignment procedure.

[0381] That is, a rough alignment can be performed manually and then adjusted automatically.

[0382] Computing the reference position

[0383] In embodiments of manual or automatic alignment, i.e. (i) or (i) and (ii), the reference position of the reference frame can be computed based on one or more positioning inputs received at the user interface with respect to the reference frame, while rendering a visual indication of the reference position within the reference frame for manually adjusting the reference position within the reference frame.

[0384] The reference position can be computed automatically or semi-automatically for the reference frame.

[0385] The reference position can be computed based on one or more positioning inputs received at the user interface, while rendering a visual indication of the reference position within the frame for manually adjusting the reference position within the frame.

[0386] The reference position can be computed automatically or semi-automatically for the reference frame.

[0387] The reference position can be computed automatically or semi-automatically based on a (common) structure component path within a time interval of the time-sequential frames.

[0388] The reference position can be computed automatically or semi-automatically by applying a perception component to the reference frame.

[0389] Alternatively or additionally, the reference position can be computed automatically or semi-automatically based on a common structure component path within a time interval of the time-sequential frames.

[0390] Iterative generation and propagation of aggregated 3D models

[0391] The reference position can be computed within the reference frame as an aligned model position of the existing 3D model of the structural component based on at least one of: (i) one or more manual alignment inputs received at the user interface in respect of the frame while rendering the existing 3D model to manually align the existing 3D model with the structural component in the reference frame; and (ii) an automatic alignment of the existing 3D model with the structural component in the reference frame.

[0392] The existing 3D model can have been generated from one or more other frames in which at least a portion of the structural component is captured.

[0393] The 3D model can be an aggregated 3D model determined by aggregating the selectively extracted 3D structural points with 3D structural points of the existing 3D model.

[0394] The automatic alignment can comprise: computing a projection of the existing 3D model onto a 2D image plane associated with the reference frame, and adjusting the model position in 3D space to match the projection to 2D features of the common structural component within the 2D image plane.

[0395] Defining object dimensions

[0396] One or more boundary object dimensions can be determined for the common structural component.

[0397] The one or more boundary object dimensions can be one of:

[0398] (i) manually determined based on one or more size inputs received at the user interface in respect of the reference frame;

[0399] (ii) automatically determined by applying a perception component to the reference frame;

[0400] (iii) semi-automatically determined by applying a perception component to the reference frame and further based on one or more size inputs received in respect of the reference frame;

[0401] (iv) assumed.

[0402] The selectively extracted structural points from the reference frame 3D can be selectively extracted therefrom to generate the 3D model based on the computed reference position within the reference frame and the one or more boundary object dimensions.

[0403] The selectively extracted 3D structural points can be a subset of points within a 3D volume defined by the reference position and the one or more boundary object dimensions.

[0404] The annotation data of the at least one perceptual input of the target frame can further comprise one or more bounding object dimensions for annotating a rigid common structural component, or a transformation of one or more bounding object dimensions for annotating a non-rigid common structural component.

[0405] The one or more bounding object dimensions can be manually or semi-automatically determined based on one or more size inputs received in respect of the reference frame, wherein the visual indication of the reference location takes the form of rendering a 3D bounding object at the reference location within the reference frame, and such that the one or more bounding object dimensions are for manual adjustment of those one or more bounding object dimensions.

[0406] The one or more bounding object dimensions can additionally be computed based on one or more adjustment inputs received at a user interface in respect of the target frame, while rendering a 3D bounding object at the aligned model location within the target frame. The 3D bounding object can be rendered at the reference location within the reference frame simultaneously or subsequently, wherein the one or more bounding object dimensions are adjusted in accordance with the one or more adjustment inputs received in respect of the target frame (to allow the annotator to see the effect of any adjustments in the context of the reference frame in the target frame).

[0407] 2D / 3D annotation data

[0408] Reference is made below to 2D annotation data and 3D annotation data. This refers to 2D or 3D annotation data created for annotating a structural component in the at least one perceptual input of the target frame, unless otherwise stated.

[0409] It is noted, however, that embodiments of any of the above aspects can additionally create and store further annotation data for annotating a common structural component in the at least one perceptual input of the reference frame. These embodiments can advantageously utilize a common set of annotation operations (manual, automatic or semi-automatic operations) to create the annotation data for the target frame and the reference frame.

[0410] Furthermore, in any of the above embodiments, 2D annotation data can be created in addition to 3D annotation data. In some such embodiments, one type of annotation data can be created to annotate one or more perceptual inputs of the target frame, and another type of annotation data can be created to annotate one or more perceptual inputs of the target frame. This can similarly utilize a common set of annotation operations to create both types of annotation data.

[0411] By way of example, the common annotation operations can be used to create:

[0412] - 2D annotation data for the target frame and the reference frame;

[0413] - 3D annotation data for the target frame and the reference frame;

[0414] - 2D annotation data of the reference frame and 3D annotation data of the target frame;

[0415] - 2D annotation data and 3D annotation data of the target frame.

[0416] The above examples are for illustration only and are not intended to be exhaustive.

[0417] The annotation data can comprise 2D annotation data of the aligned model position stored in association with the one or more perception inputs of the target frame and 3D annotation data of the aligned model position, whereby the aligned model position is used for 2D annotation and 3D annotation of the one or more perception inputs of the target frame.

[0418] Alternatively or additionally, further annotation data of the reference position can be stored for annotating the common structural component in the at least one perception input of the reference frame, whereby the computed reference position within the reference frame is used for annotating the perception inputs of the target frame and the reference frame.

[0419] The further annotation data can comprise the same one or more boundary object dimensions for annotating the rigid common structural component or a transformation thereof for annotating the non-rigid structural component.

[0420] In some embodiments, the 2D annotation data can comprise a 2D bounding box of the structural component, the 2D bounding box being fitted to the computed 3D model projection in the image plane.

[0421] Alternatively or additionally, the 2D annotation data can comprise a segmentation mask of the structural component.

[0422] Aggregated 3D model (continued)

[0423] A fourth aspect (Aspect D) of the present disclosure provides a computer- implemented method of modeling a common structural component, the method comprising: receiving, in a modeling computer system, a plurality of captured frames, each frame comprising a set of 3D structure points, wherein at least a portion of the common structural component is captured; computing a first reference position within a first one of the frames; selectively extracting first 3D structure points of the first frame based on the computed first reference position for the first frame; computing a second reference position within a second one of the frames; selectively extracting second 3D structure points of the second frame based on the computed second reference position for the second frame; aggregating the first 3D structure points and the second 3D structure points, thereby generating an aggregated 3D model of the common structural component based on the first reference position and the second reference position.

[0424] In embodiments of Aspect D, the aggregated 3D model can be used to generate annotation data for annotating the common structural component in a training example of one of the plurality of frames, the one of the plurality of frames being the first frame, the second frame, or the third frame.

[0425] It is noted, however, that aspect D is not limited in this regard, and the aggregated 3D model can alternatively (or additionally) be used for other purposes - see below.

[0426] In some embodiments, the annotation data can be generated in accordance with any of aspects A to C, or any embodiment thereof.

[0427] The annotation data can comprise at least one of: 2D annotation data derived by projecting the 3D model into the image plane; and 3D annotation data.

[0428] The one frame can be a third frame, and the method can comprise the step of computing a registered model position of the 3D model within the third frame, the annotation data being annotation data of the computed position, wherein the registered model position is based on at least one of:

[0429] (i) automatic alignment of the 3D model to the common structural component in the third frame;

[0430] (ii) one or more manual alignment inputs received at the user interface in respect of the third frame, while rendering the 3D model to manually align the 3D model to the common structural component in the third frame.

[0431] The second reference position within the second frame can be initially estimated to generate the aggregated 3D model, and the method can comprise the subsequent step of aligning the second reference position to the common structural component in the second frame, based on at least one of:

[0432] (i) automatic alignment of the first 3D structural points extracted from the first frame to the common structural component in the second frame to automatically align the aggregated 3D model to the common structural component in the second frame;

[0433] (ii) one or more manual alignment inputs received at the user interface in respect of the second frame, while rendering the aggregated 3D model to manually align the aggregated 3D model to the common structural component in the second frame;

[0434] The aggregated 3D model can be updated based on the second frame and the aligned second reference position within the second frame.

[0435] The first 3D model can be generated by selectively extracting the first 3D structural points, and the second reference position can be aligned to the common structural component in the second frame to generate the aggregated 3D model based on at least one of: (i) automatic alignment of the first 3D model to the common structural component in the second frame; (ii) one or more manual alignment inputs received at the user interface in respect of the second frame, while rendering the first 3D model to manually align the first 3D model to the common structural component in the second frame.

[0436] At least a portion of the common structural component can be captured in a third frame, the method can include aligning a third reference position with the common structural component in the third frame based on at least one of (i) an automatic alignment of the 3D aggregated model with the common structural component in the third frame, and (ii) one or more manual alignment inputs received at the user interface with respect to the third frame, while rendering the aggregated 3D model to manually align the aggregated 3D model with the common structural component in the third frame; selectively extracting third 3D structure points of the third frame based on the third reference position; aggregating the first 3D structure points, the second 3D structure points, and the third 3D structure points to generate a second aggregated 3D model of the common structural component based on the first reference position, the second reference position, and the third reference position.

[0437] The set of 3D structure points of the third frame can be transformed to the frame of reference of the third reference position to selectively extract the third 3D structure points.

[0438] The second reference position within the second frame can be initially estimated to generate the aggregated 3D model, and the aggregated 3D model can be updated based on the second frame and the aligned second reference position within the second frame.

[0439] The aggregated 3D model can be rendered via the user interface and updated and re-rendered with one or more manual alignment inputs received at the user interface with respect to the second frame to manually align the second reference position with the common structural component to align the second reference position with the common structural component in the second frame, with the effect of correcting visual artifacts in the rendered aggregated 3D model due to the initial misalignment of the second reference position.

[0440] It should be noted that in this context, the aligned second reference position is equivalent to the "aligned model position" referred to in other paragraphs of this disclosure, where the second frame serves as the target frame. All of the above descriptions regarding the model position apply equally to the second reference position in this context (e.g., including initially estimating the second reference position and then adjusting it using any of the above manual, automatic, or semi-automatic processes).

[0441] The aggregated 3D model can be updated and re-rendered when the one or more manual alignment inputs are received at the user to align the second reference position with the common structural component in the second frame, with the effect of correcting visual artifacts in the rendered aggregated 3D model due to the initial misalignment of the second reference position.

[0442] As described above, this provides the annotator with a means to manually align (or adjust the alignment of) the second reference position and has the advantages described above.

[0443] The one frame can be the second frame, and the annotation data is annotation data of the aligned second reference position.

[0444] The annotation data can comprise position data of the aligned second reference position for annotating the position of the common structural component in at least one training example of the target frame, e.g. 3D position data for annotating the position of the common structural component in 3D space.

[0445] Additionally or alternatively, for example, the annotation data can comprise data derived from the aggregated 3D model using the aligned second reference position, e.g. 2D annotation data derived by projecting the 3D model into the image plane based on the aligned second reference position.

[0446] The first 3D structure points can be selectively extracted from the first frame used to generate the 3D model based on the first reference position and one or more boundary object dimensions. The second 3D structure points can be selectively extracted from the frame used to generate the 3D model based on the second reference position and one of:

[0447] (a) the same one or more boundary object dimensions used to model rigid objects;

[0448] (b) a transformation of the one or more boundary object dimensions used to model non-rigid objects.

[0449] The one or more boundary object dimensions can be one of:

[0450] (i) manually determined based on one or more size inputs received for at least one of the first frame and the second frame;

[0451] (ii) automatically determined by applying a perception component to at least one of the first frame and the second frame;

[0452] (iii) semi-automatically determined by applying a perception component to at least one of the first frame and the second frame and further based on one or more size inputs received for at least one of the first frame and the second frame;

[0453] (iv) assumed.

[0454] The first 3D structure can be a subset of points within a first 3D volume defined by the first reference position and the one or more boundary object dimensions. The second 3D structure points can be a subset of points within a second 3D volume defined by the second reference position and the same one or more boundary object dimensions or a transformation thereof.

[0455] The method can comprise the step of applying noise filtering to the aggregated 3D structure points to filter out noise points therefrom to generate the aggregated 3D model.

[0456] The aggregated 3D model can comprise a 3D surface model fitted to the aggregated 3D structure points.

[0457] The method can comprise the step of applying a prediction model to the aggregated 3D surface points to generate the 3D model.

[0458] The method can comprise the step of training the at least one perception component using the annotated perception input, wherein the annotation data of the annotated perception input provides Ground Truth for the perception input during the training.

[0459] That is, in the notation of Figure 1 the perception input is x and the annotation data provides Ground Truth y x .

[0460] Training data augmentation

[0461] As mentioned above, using the aggregated 3D model is not limited to creating annotated perception input. For example, the aggregated 3D model determined according to aspect D can alternatively or additionally be used for one or more of:

[0462] (a) training data augmentation;

[0463] (b) simulation.

[0464] Training data augmentation

[0465] The aggregated 3D model can be used to augment data of one of the multiple frames by means of model data of the aggregated 3D model, thereby creating at least one augmented perception input comprising data of said one of the multiple frames and model data of the 3D model, said one of the multiple frames being the first, second or third frame.

[0466] The model data can comprise at least one of: 2D augmentation data created by projecting the 3D model into an image plane; 3D model data.

[0467] The method can comprise the step of training the at least one perception component using the augmented perception input, whereby during the training the combination of model data and data of said one of the multiple frames is provided to the perception component as part of the same perception input.

[0468] That is, in the notation of Figure 1 the frame data and the model data each form part of the same perception input x.

[0469] The augmented perception input can be used for one of:

[0470] (a) an unsupervised training process, wherein no Ground Truth is provided for the augmented perception input (i.e. in the notation of Figure 1 there is no y x );

[0471] (b) a supervised training process, wherein the annotation data of claim 2 or any of its dependent claims provides Ground Truth for the augmented perception input.

[0472] simulation

[0473] Additionally or alternatively, the aggregated 3D model can be input to a simulator for rendering in a simulated environment, wherein at least one autonomous agent is executed to autonomously navigate the simulated environment and record behaviors of the autonomous agent in response to the simulated environment in an electronic behavior log.

[0474] The autonomous agent can use simulated instances of trained perception components applied to simulated perception inputs to navigate the simulated environment, wherein data of the electronic behavior log can be used to retrain and / or redesign the perception components for application to real-world perception inputs.

[0475] The method can include a step of embodying the retrained or redesigned perception components in a real-world autonomous robot control system for autonomous decision making based on real-world perception inputs.

[0476] Efficient model generation

[0477] To efficiently (and thus quickly) generate a 3D model, a set of 3D structure points of a reference frame can be transformed into a frame of reference of a reference location so as to selectively extract 3D structure points of the 3D model.

[0478] Note the distinction between the term “frame of reference” as used in the geometric sense and the term “reference frame” as used here.

[0479] For example, the 3D volume defined by the reference location and one or more boundary object dimensions can be a cuboid aligned with the coordinate axes of the frame of reference. This allows for efficient computation of the subset of 3D structure points within the volume, e.g., on a GPU.

[0480] For example, the 3D structure points can be selectively extracted from the frame of reference by scalar comparisons performed in the reference frame.

[0481] For the aggregated 3D model, a set of 3D structure points of a first frame can be transformed into a frame of reference of a first reference location so as to selectively extract first 3D structure points; a set of 3D structure points of a second frame can be transformed into a frame of reference of a second reference location so as to selectively extract second 3D structure points.

[0482] The first 3D volume can be aligned with the coordinate axes of the frame of reference of the first reference location, and the second 3D volume can be aligned with the coordinate axes of the frame of reference of the second reference location.

[0483] Perception inputs - examples

[0484] The at least one perception input can comprise 2D image data of the or a second frame or 2D image data associated with the target frame, the image plane being an image plane of the image data.

[0485] The target frame can comprise data of a depth component of a 3D image, the image data of the perception input can be image data of a colour component of the 3D image.

[0486] The method can comprise the step of applying noise filtering to at least one of: the extracted 3D structure points used to generate the 3D model, wherein the 3D model comprises or is derived from the filtered 3D structure points in the event; the computed projections (in the case of 2D annotation data), wherein the 2D annotation data is 2D annotation data for the filtered projections in the event, to filter out noise therefrom.

[0487] The noise filtering can be applied in accordance with filtering criteria that can be manually adjusted by a user of an annotation computer system.

[0488] The or each frame can be one of a plurality of time-sequential frames.

[0489] Annotated perception input - use cases

[0490] Any of the above perception components used to facilitate automatic or semi-automatic annotation can be a trained (machine learning) perception component. In this case, the trained perception component can be retrained using any of the above annotated training inputs. Using a trained perception in this context can be referred to as "Model in the Loop".

[0491] More generally, the method can comprise the step of using the or each perception input in a training process to train at least one perception component, wherein the annotation data of the perception input provides Ground Truth for the perception input in the training process.

[0492] For example, the perception component can be one of: a 2D bounding box detector, a 3D bounding box detector, an instance segmentation component, a localisation estimation component, an orientation estimation component and a distance estimation component.

[0493] For example, 3D annotation data can be used to train a 3D perception component, 2D annotation data can be used to train a 2D perception component.

[0494] Example 3D frames

[0495] The set of 3D structure points of the or each frame can be in the form of a point cloud.

[0496] The set of 3D structure points can have been captured using one or more sensors having one or more sensor modalities.

[0497] Each frame can correspond to a different single time instant.

[0498] At least one of the frames can comprise 3D structure points captured at multiple time instants, which have been transformed to correspond to the single time instant to which the frame corresponds.

[0499] Each frame can be one of a plurality of time-sequential frames. For example, the target frame and the reference frame can be frames of a time-sequential series of frames corresponding to different time instants in the series.

[0500] The set of 3D structure points of a frame can be a union generated by merging at least two sets of 3D structure points captured by different sensors.

[0501] Example 3D model

[0502] Any of the above 3D models can comprise a 3D surface model fitted to the selectively extracted 3D structure points. This can be a single-frame model fitted to 3D structure points selectively extracted from a single frame, or an aggregated 3D model fitted to points selectively extracted from multiple frames and aggregated.

[0503] This can be used, for example, to create 3D annotation data comprising or based on projections of the 3D surface model.

[0504] Another aspect of the inventive subject matter provides a computer-implemented method of creating one or more annotated perception inputs, the method comprising: in an annotation computer system, receiving at least one captured frame comprising a set of 3D structure points, the frame capturing at least a portion of a structure component; receiving a 3D model of the structure component; determining an aligned model position of the 3D model within the frame based on automatic alignment of the 3D model with the structure component in the frame; storing annotation data of the aligned model position in computer memory in association with at least one perception input of the frame so as to annotate the structure component therein.

[0505] In some embodiments, the 3D model can be generated by selectively extracting 3D structure points of at least one reference frame based on a computed reference position within the reference frame.

[0506] However, the 3D model can also be a CAD (Computer Aided Design) model or other externally generated model of the structure component.

[0507] That is, the above automatic or semi-automatic model alignment features can also be applied to externally generated models. It will thus be appreciated that all the above description regarding the 3D model generated from at least one reference frame equally applies to an externally generated 3D model in this context.

[0508] Further examples

[0509] To further illustrate the ways in which various annotation features of the present disclosure can be used, either individually or in combination, some more example use cases and workflows supported by these features are listed below. It should be understood in light of the teachings presented herein that they are by no means intended to be exhaustive.

[0510] 1. 3D to 2D annotation: Given a complete or partial 3D model:

[0511] a. Generate tight 2D bounding boxes;

[0512] b. Generate instance segmentation masks (or priors for CRF / annotator / other methods to refine the segmentation masks);

[0513] c. Use projected points from different frames to assist annotation (using the X-ray vision features described above), thereby improving consistency and accuracy of annotation.

[0514] 2. 3D model extracted from a single frame: Can be efficiently generated by transforming all points of the frame to align with the axes of the 3D bounding box, which means that a simple scalar comparison can be used to determine whether any given point is enclosed.

[0515] 3. 3D model extracted across multiple frames (aggregated 3D model): Use bounding boxes from multiple frames to extract and aggregate all enclosed point cloud points (to generate an aggregated point cloud). By applying the transformation described above to each frame with a 3D bounding box located in that frame, extraction can be efficiently performed for each frame.

[0516] a. Noise can then be filtered out from the aggregated point cloud.

[0517] b. The aggregated point cloud to be generated can be smoothed with a surface, e.g., using SDF.

[0518] c. Advantages of accumulated points include:

[0519] i. Improved instance segmentation priors, with predictive models (e.g., CRF) applied; or even the aggregated point cloud itself can be sufficient for instance segmentation (i.e., without applying predictive models or fitting surface models, etc.);

[0520] ii. Improved noise filtering / surface, as there will be more points than a single frame;

[0521] iii. Aggregated models are more beneficial than single-frame models for augmenting training data, as there will be higher quality and greater range of applicable viewpoints;

[0522] iv. 2D bounding boxes can be drawn to include occluded portions of partially occluded objects.

[0523] 4. Automatic 3D annotation: With one or more of the following:

[0524] a. generate a 3D model from one frame, match it to the point cloud in another frame (e.g. by iterative closest point), then combine into a single model, and repeat for the next frame (using the combined model as the matching reference);

[0525] b. project the 3D model into the image and minimize photometric error (i.e. allow the annotator to automatically "eyeball" the x-ray visual alignment);

[0526] c. feature matching in 2D and / or 3D, and minimize re-projection error of the matching features;

[0527] d. iterative build from initial coarse annotation, i.e. determine initial annotation, align pose, refine motion model, and repeat for more poses. Coarse annotation can be provided by:

[0528] i. model-in-the-loop (e.g. neural network or moving object detector) in 2D or 3D;

[0529] ii. initial dynamic model (expected behavior model) across frame boundary boxes, e.g. assume objects travel at constant velocity;

[0530] iii. "single-click" object point (i.e. by selecting a single object point) or "2D lasso" around object points (i.e. by selecting a set of object points in a 2D plane such as the image plane or top view).

[0531] e. minimize alignment cost function that takes into account e.g. 2D error, 3D error, and possible (expected) behavior.

[0532] For the sake of illustration, the above describes specific embodiments of the application, but it should be understood that they are not necessarily exhaustive. The scope of the application is not limited by the described embodiments, but only by the claims that follow.

Claims

1. A computer-implemented method of creating one or more annotated perception inputs, each perception input being a dataset capturing a structure, the method comprising: In an annotation computer system: receiving a plurality of captured frames, each frame comprising a set of 3D structure points, wherein at least a portion of a common object is captured in each frame, wherein the common object is an object common to the plurality of captured frames; computing a reference position within at least one reference frame of the plurality of frames; locating a 3D bounding object in the reference frame to define the common object, wherein the reference position is a position of the 3D bounding object within the reference frame; generating a 3D model of the common object by selectively extracting 3D structure points of the at least one reference frame based on the reference position within the frame, wherein the selectively extracted 3D structure points of the reference frame are a subset of the 3D structure points extracted from within a volume of the 3D bounding object; determining an aligned model position of the 3D model within a target frame of the plurality of frames based on one or more manual alignment inputs received at a user interface with respect to the target frame, while rendering the 3D model to manually align the 3D model with the common object in the target frame; and storing annotation data of the aligned model position in the computer memory in association with at least one perceptual input of the target frame to annotate the common object therein.

2. The method of any preceding claim, wherein, The annotation data of the aligned model position comprises position data of the aligned model position to annotate a position of the common object in the at least one perceptual input of the target frame.

3. The method of claim 1 or 2, wherein, The annotation data of the aligned model position comprises annotation data derived from the 3D model using the aligned model position.

4. The method of claim 3, wherein, The derived data is 2D annotation data derived by: projecting the 3D model into an image plane associated with the target frame based on the aligned model position; or projecting a single-frame 3D model generated by selectively extracting 3D structure points from the target frame by locating the 3D bounding object in the target frame at the aligned model position into an image plane associated with the target frame; wherein the selectively extracted 3D structure points of the target frame are a subset of the 3D structure points extracted from within a volume of the 3D bounding object located in the target frame, or projecting an aggregated 3D model generated by: selectively extracting 3D structure points from the target frame by locating the 3D bounding object in the target frame at the aligned model position, wherein the selectively extracted 3D structure points of the target frame are a subset of the 3D structure points extracted from within a volume of the 3D bounding object located in the target frame, and aggregating the selectively extracted 3D structure points of the target frame with the selectively extracted 3D structure points of the reference frame.

5. The method of any preceding claim, wherein, computing the reference position of the reference frame based on one or more positioning inputs received at a user interface with respect to the reference frame, while rendering a visual indication of the reference position within the reference frame for manually adjusting the reference position within the reference frame.

6. The method of any preceding claim, wherein, The reference position is automatically or semi-automatically computed for the reference frame.

7. The method of claim 6, wherein, The plurality of frames are time-sequential frames, and the reference position is automatically or semi-automatically computed based on a path of the common object within a time interval of the time-sequential frames.

8. The method of any preceding claim, wherein, determining one or more bounding object dimensions for the public object, wherein the one or more bounding object dimensions are one of: (i) manually determined based on one or more size inputs received at the user interface with respect to the reference frame; (ii) automatically determined by applying an object detector to the reference frame; (iii) semi-automatically determined by applying the object detector to the reference frame and further based on one or more size inputs received with respect to the reference frame; (iv) assumed.

9. The method of claim 8, wherein, 3D structure points selectively extracted from the reference frame are selectively extracted therefrom to generate the 3D model based on a reference position computed within the reference frame and one or more bounding object dimensions.

10. The method of claim 5, 8, or any claim dependent therefrom, wherein, The one or more bounding object dimensions are manually or semi-automatically determined based on one or more size inputs received with respect to the reference frame, wherein the visual indication of the reference position takes the form of rendering a 3D bounding object at the reference position within the reference frame and such that the one or more bounding object dimensions are available for manual adjustment of those one or more bounding object dimensions.

11. The method of claim 10, wherein, The one or more bounding object dimensions are additionally computed based on one or more adjustment inputs received at the user interface with respect to the target frame while rendering the 3D bounding object at an aligned model position within the target frame; wherein the 3D bounding object is simultaneously or subsequently rendered at a reference position within the reference frame, wherein the one or more bounding object dimensions are adjusted according to the one or more adjustment inputs received with respect to the target frame.

12. The method of any preceding claim, comprising the step of using the each perception input to train at least one object detector in a training process, wherein, The annotation data of the perception input provides Ground Truth of the perception input during the training process.

13. The method of any preceding claim, wherein, The aligned model position is semi-automatically determined by automatically initially estimating the model position and then aligning the estimated model position according to the one or more manual alignment inputs.

14. A computer system comprising one or more computers programmed or otherwise configured to perform the steps of any of the preceding claims.

15. A computer readable medium containing instructions which, when executed by a computer system, cause the computer system to perform the steps of any of claims 1 to 13.

Images