structural annotation

Abstract

The invention relates to a computer-implemented method of creating 2D annotation data for annotating one or more perception inputs, comprising: receiving, at an annotation computer system, at least one captured frame comprising a set of 3D structure points, the frame capturing at least a portion of a structure component; computing a reference position of the structure component within the frame; generating a 3D model of the structure component by selectively extracting 3D structure points of the frame based on the reference position; computing a projection of the 3D model to an image plane; and storing the computed 2D annotation data of the projection in a persistent computer storage device to annotate the structure component within the image plane.

Description

Technical Field

[0001] This disclosure relates to annotating structures captured in images, point clouds, and other forms of perceptual input. Such annotations can be applied to create annotated perceptual inputs for training machine learning (ML) perceptual components. Background Technology

[0002] Structure-aware refers to a class of data processing algorithms that can meaningfully interpret the structure captured in a perceptual input. This processing can be applied to different forms of perceptual input. Perceptual input generally refers to any structural representation, that is, any dataset in which structure is captured. Structure-aware can be applied to two-dimensional (2D) and three-dimensional (3D) spaces. The result of applying a structure-aware algorithm to a given structural input can be encoded as a structure-aware output.

[0003] One form of perceptual input is a two-dimensional (2D) image, i.e., an image with only color components (one or more color channels). The most basic form of structure perception is image classification, i.e., classifying an image as a whole relative to a set of image classes. More complex forms of structure perception applied to 2D space include 2D object detection and / or localization (e.g., orientation, pose, and / or distance estimation in 2D space), 2D instance segmentation, etc. Other forms of perceptual input include three-dimensional (3D) images, i.e., images with at least a depth component (depth channel); 3D point clouds, such as 3D point clouds captured using RADAR or LIDAR or derived from 3D images; voxel- or mesh-based structural representations, or any other form of 3D structural representation. Perceptual algorithms that can be applied to 3D space include, for example, 3D object detection and / or localization (e.g., distance, orientation, or pose estimation in 3D space). A single perceptual input can also be formed from multiple images. For example, stereo depth information can be captured in a pair of stereo 2D images, which can be used as the basis for 3D perception. 3D structure awareness can also be applied to a single 2D image, such as monocular depth extraction, which extracts depth information from a single 2D image (it should be noted that even without any depth channels, some depth information can still be captured in one or more color channels). This form of structure awareness is an example of different “perceptual modalities,” as the term is used in this paper. Structure awareness applied to 2D or 3D images can be called “computer vision.”

[0004] Object detection refers to detecting any number of objects captured in a perceptual input, typically involving representing each such object as an instance of an object class. This object detection may involve or combine one or more forms of location estimation, such as 2D or 3D bounding box detection (a form of object localization aimed at defining the area or volume of an object in 2D or 3D space), distance estimation, pose estimation, etc.

[0005] In machine learning (ML) contexts, a structure-aware component can include one or more trained perception models. For example, convolutional neural networks (CNNs) are commonly used in machine vision processing. Such networks require a large number of training images annotated with information the neural network needs to learn (a form of supervised learning). During training, thousands, preferably hundreds of thousands, of such annotated images are presented to the network, and the network learns on its own how to associate the features captured in the images with their associated annotations. Each image is annotated in the sense of its association with the annotated data. The image serves as the perception input, and the associated annotated data provides the "Ground Truth" of the image. CNNs and other forms of perception models can be architected to receive and process other forms of perception input, such as point clouds, voxel tensors, etc., and perceive structure in 2D and 3D space. In the training context, the perception input is generally referred to as a "training example" or "training input." In contrast, at runtime, the training examples captured by the trained perception component for processing are referred to as "runtime input." The annotated data associated with the training input provides the Ground Truth for that training input because the annotated data encodes the expected perception output of that training input. During supervised training, the parameters of the perceptual component are systematically tuned to an overall metric that minimizes, within defined limits, the difference between the perceptual output (“actual” perceptual output) generated by the perceptual component when applied to training examples in the training set and the corresponding Ground Truth (“expected” perceptual output) provided by the associated annotated data. In this way, perceptual input is “learned” from the training examples, and this learning can be “generalized” so that, once trained, it can provide meaningful perceptual outputs for perceptual inputs not encountered during training.

[0006] Such perception components form the cornerstone of many mature and emerging technologies. For example, in robotics, mobile robot systems capable of autonomously planning paths in complex environments are becoming increasingly prevalent. This rapidly developing technology includes, for instance, autonomous vehicles (AVs) that can navigate city roads on their own. These vehicles not only need to perform complex maneuvers between humans and other vehicles, but also need to maintain strict constraints on the probability of adverse events (such as collisions with other media in the environment) while performing such maneuvers frequently. For an AV to plan safely, it is crucial that it can accurately and reliably observe its environment. This includes the need to accurately and reliably detect real-world structures near the vehicle. Autonomous vehicles (also known as self-driving vehicles) are vehicles equipped with sensor systems for monitoring their external environment and control systems capable of automatically making and implementing driving decisions using these sensors. This specifically includes the ability to automatically adjust the vehicle's speed and direction of travel based on perceived input from the sensor systems. Fully autonomous or "driverless" vehicles have sufficient decision-making capabilities to operate without any driver input. However, the term "autonomous vehicle" as used herein also applies to semi-autonomous vehicles, which have a greater degree of autonomous decision-making capability and therefore still require some level of driver supervision. Other mobile robots are under development, such as those for transporting goods within and outside industrial areas. These mobile robots do not carry people and belong to a category called UAVs (Unmanned Autonomous Vehicles). Autonomous aerial mobile robots (drones) are also under development.

[0007] Therefore, in the more general fields of autonomous driving and robotics, one or more sensing components may be needed to interpret the sensing input, that is, to determine information about the real-world structure captured in a given sensing input.

[0008] Complex robotic systems such as AVs (Analog Devices) may increasingly require the implementation of multiple perceptual modalities to accurately interpret diverse forms of perceptual input. For example, an AV may be equipped with one or more pairs of stereo optical sensors (cameras) from which associated depth maps are extracted. In this case, the AV's data processing system can be configured to apply one or more forms of 2D structure perception to the image itself (e.g., 2D bounding box detection and / or other forms of 2D localization, instance segmentation, etc.) plus one or more forms of 3D structure perception of the associated depth map data (e.g., 3D bounding box detection and / or other forms of 3D localization). Such depth maps may also be derived from LiDAR, RADAR, etc., or obtained by combining multiple sensor modalities.

[0009] To train a perceptual component for a desired perceptual modality, the perceptual component architecture is designed to receive perceptual input in the desired form and provide perceptual output in the desired form as a response. Furthermore, to train a perceptual component with an appropriate architecture based on supervised learning, annotations conforming to the desired perceptual modality are required. For example, training a 2D bounding box detector requires 2D bounding box annotations; similarly, training a segmentation component to perform image segmentation (pixel-by-pixel classification of individual image pixels) requires annotations that encode a suitable segmentation mask from which the model can learn; a 3D bounding box detector needs to be able to receive 3D structural data and annotated 3D bounding boxes, and so on. Summary of the Invention

[0010] Typically, annotated training examples are created manually by human annotators. Even with 2D images, each image can take tens of minutes. Therefore, creating hundreds of thousands of training images requires enormous time and manpower, making it a costly training process. In practice, this limits the number of training images actually available, which can negatively impact the performance of training perceptual components on a limited number of images. Manual 3D annotation is significantly more tedious and time-consuming. Furthermore, the problem is exacerbated when adapting to multiple perceptual modalities, as this may require multiple forms of annotated data (e.g., two or more 2D bounding boxes, segmentation masks, 3D bounding boxes, etc.) for one or more forms of training input (e.g., one or more 2D images, 3D images, point clouds, etc.).

[0011] This disclosure generally relates to a form of annotation tool with annotation capabilities that facilitate the rapid and efficient annotation of perceptual input. Such annotation tools can be used to create annotated perceptual input for training perceptual components. The term "annotation tool" refers generally to a computer system programmed or otherwise configured to implement those annotation capabilities, or generally to one or more computer programs for programming a programmable computer system to perform those capabilities.

[0012] One aspect of the present invention provides a computer-implemented method for creating 2D annotation data for annotating one or more perceptual inputs, the method comprising: receiving, in an annotation computer system, at least one capture frame comprising a set of 3D structural points, wherein at least a portion of a structural component is captured in the frame; calculating a reference position of the structural component within the frame; generating a 3D model of the structural component by selectively extracting the 3D structural points of the frame based on the reference position; calculating the projection of the 3D model onto an image plane; and storing the calculated projection of the 2D annotation data in a persistent computer storage device to annotate the structural component in the image plane.

[0013] Advantageously, this allows for the automatic or semi-automatic generation of 2D annotation data based on a 3D model extracted from the frame itself. The 2D annotation data is generated based on 3D modeling, and thus such 2D annotation data can be generated for the frame itself (i.e., the frame used as the basis for the 3D model) and / or different frames that at least partially capture the same structural components (in this context, the structural components can be referred to as common structural components, i.e., shared by both frames). That is, the 3D model can be applied to the frame from which it was generated, or it can be propagated to another frame and aligned in 3D space with the common structural components in that different frame to generate 2D annotation data for that other frame (by projecting the 3D model from its alignment position, or by determining a second model that can selectively extract 3D structural points from that other frame based on that alignment position, and then projecting the second model; the second model can be, for example, an aggregated model determined by aggregating the selectively extracted points from the two frames).

[0014] The method employs "self-contained" modeling, thereby generating a 3D model using 3D structural points from one or more frames themselves. Therefore, the method is flexible enough to be applied to any form of structural component captured across multiple 3D frames.

[0015] 2D annotation data can include 2D bounding boxes (or other 2D boundary objects) of structural components, fitted to their projections onto the computed 3D model image plane. Advantageously, fitting 2D bounding boxes to the projection of the 3D model itself provides tight 2D bounding boxes, meaning that the 2D bounding boxes are closely aligned with the outer boundaries of the structural components when they appear in the image plane. An alternative approach is to determine the 3D bounding boxes (or other 3D boundary objects) of common structural components in 3D space and project these 3D boundary objects onto the image plane; however, in most practical scenarios, this does not provide tight 2D bounding boxes in the image plane: even if the 3D boundary objects are tightly fitted in 3D space, there is no guarantee that the boundaries of the projected 3D bounding boxes will align with the outer boundaries of the structural components themselves when they appear in the image plane.

[0016] Alternatively, 2D annotation data can include segmentation masks for structural components. By projecting the 3D model itself onto an image plane, a segmentation mask can be provided that labels pixels in the image plane within the outer boundary of the model projection as belonging to structural components and pixels outside the outer boundary as not belonging to structural components. It should be noted that certain processing can be applied to the calculated projection to provide a useful segmentation mask. For example, the projected region can be "smoothed" or "filled" to reduce artifacts caused by noise, sparsity, etc., in the underlying frames that generate the 3D model. In this case, the outer boundary of the calculated projection refers to the outer boundary after such processing has been applied.

[0017] 3D models can include the selectively extracted structural points themselves and / or 3D mesh models or other 3D surface models fitted to the selectively extracted points. When using a 3D surface model to create a segmentation mask, the outer boundary of the projected 3D surface model defines the segmentation mask (subject to any post-processing of the computational projection). 3D surface models are a way to provide higher quality segmentation masks.

[0018] In general, denoising and / or thinning can be applied to one or both of the 3D model (before it is projected) and the computed projection. This may involve one or more of noise filtering (to filter out noisy points / pixels, such as those with an insufficient number of neighboring points / pixels within a defined threshold distance), predictive modeling, etc. For predictive modeling, assuming the 3D model (and corresponding projection) is incomplete or incorrect, the predictive model is applied to the existing points (and corresponding pixels) of the 3D model (and corresponding projection). The existing points (and corresponding pixels) act as "prior" from which the correct 3D model (and corresponding projection) can be inferred. This has the effect of adding extra points / pixels or removing points / pixels determined to be incorrect (therefore, predictive modeling can be used to predict missing points / pixels and / or as a noise filtering means).

[0019] In some embodiments, the 3D model is an aggregated model derived from multiple frames. For each frame in the multi-frame set, a reference position is computed for that frame to selectively extract 3D structural points from that frame. The 3D structural points extracted from the multi-frame set are aggregated to generate an aggregated 3D model. Aggregated 3D models offer several advantages. For example, such models can account for occlusion or other forms of partial data capture, as well as data sparsity and noise. Regarding the latter, aggregation across multiple frames means it is easier to identify and correct noise artifacts (e.g., stray noise points are on average sparser than those that actually belong to relevant structural components, and therefore can be filtered out more reliably; the increased density of points that do belong to relevant structural components also provides stronger priors for predictive modeling, etc.). Therefore, combining aggregated 3D models with denoising applications has particular benefits.

[0020] In a preferred embodiment, the annotation tool provides annotation data for multi-sensory modalities (e.g., 2D and 3D annotations) and / or for multiple frames based on a set of general annotation operations (which may be manual, automatic, or semi-automatic). That is, the same operations are used to provide annotation data for multi-sensory modalities and / or multiple frames. There are various ways in which this annotation functionality can be utilized, including but not limited to the examples below.

[0021] For simplicity, the following example considers two frames. It is understood, of course, that aggregated models can be generated across a much larger number of frames. In fact, by aggregating across many frames, it is possible to obtain high-quality aggregated models (dense and low-noise), which can then be used to efficiently generate high-quality annotation data for those frames (and / or other frames that capture common structural components).

[0022] Example 1—Context: Accurately locating a 3D bounding box (or other boundary object) within the first frame to define a specific structural component. Assuming one or more forms of “external” input are used to locate (e.g., localize and orient) the 3D boundary object, i.e., when the perceptual component is unavailable at runtime, this provides useful Ground Truth for supervised training purposes. In the simplest case, the 3D boundary object can be manually located (i.e., external input is provided by human input); however, it can also be automatically located by utilizing contextual information such as measured or hypothetical structural component paths (in this case, the external input comes from a known or hypothetical path). The position and size of the 3D bounding box within the first frame (if applicable) can then be stored as 3D annotation data of the first perceptual input of the first frame (i.e., the included perceptual input originates from or otherwise corresponds to at least a portion of that frame).

[0023] Example 1—3D to 2D; Same Frame: Furthermore, the 3D bounding box is now accurately positioned and can be used as the basis for generating the 3D model of the desired structural components. For a closely fitted and accurately placed 3D bounding object, the intersection of the first frame and the 3D bounding box volume can be used as the basis for the model (i.e., 3D structural points within the 3D bounding box volume can be extracted to generate the 3D model). In this case, the aforementioned "reference position" is calculated as the position of the 3D bounding box within the first frame (it should be noted that, as described below, the bounding box does not actually need to be closely fitted initially—at least a "coarse" bounding box can be used initially).

[0024] At this point, 2D annotation data can be generated immediately for the first frame: the alignment position of the 3D model within the first frame (i.e., the position that aligns the 3D model with the structural components in the frame) is known as the position of the 3D bounding box. Therefore, the 3D model can be projected onto the desired image plane based on this position, and the resulting projection can be used as the basis for one or more forms of 2D annotation data. Such 2D annotation data can be stored in association with a first perceptual input or another perceptual input of the first frame. For example, 3D annotation data can be stored in association with the point cloud of the first frame, and 2D annotation data can be stored in association with a 2D image of the first frame (e.g., color components are associated with depth components, where the first frame includes or is derived from the depth component).

[0025] Example 1—Model Propagation: More importantly, the same 3D model can then be propagated to a second frame that at least partially captures the same structural components. The alignment position of the 3D model within the second frame is determined by aligning the 3D model with the common structural components in the second frame (automatically, manually, or semi-automatically), i.e., locating and / or redirecting the 3D model so that its structural elements and / or visible features align with the corresponding structural elements and features of the common structural components in the second frame.

[0026] Example 1—Model Propagation; 3D to 3D: The position of the 3D bounding box relative to the 3D model is already known, derived from the position of the 3D bounding box within the first frame. For example, the 3D structural points of the 3D model can be defined in the reference frame of the 3D bounding box, as described below. Therefore, by accurately positioning the 3D model within the second frame (aligning it with common structural components), the accurate position of the 3D bounding box within the second frame is now considered the aligned model position in the second frame. For rigid objects, the same bounding box size can be applied in both the first and second frames. In other words, by propagating the 3D model into the second frame and aligning it with common structural components, the bounding box positioned within the first frame is propagated to the second frame and correctly positioned within it. The position of the bounding box is now accurately determined within the second frame, and this position can be stored (if applicable, along with the bounding box size) as a second perceptual input for the second frame (i.e., including perceptual input derived from or otherwise corresponding to at least a portion of the second frame). As mentioned above, alignment can be manual, automatic, or semi-automatic. In manual alignment scenarios, accurately aligning the 3D model with its corresponding structural components by eye (based on characteristic structural elements, features, etc.) is significantly easier than locating the second bounding box "from scratch" in the second frame. Furthermore, in the case of rigid objects, the same bounding box size can be applied across all frames, eliminating the need to define these bounding box sizes separately for each frame of the event.

[0027] Example 1—Model Propagation; 3D to 2D: Finally, the aligned model position (or, in this context, equivalently, the 3D bounding box position) is considered to be within the second frame, and the 2D annotation data can be created in a similar manner by projecting onto the second frame. This can be based on: (i) the projection of the model onto the desired image plane; (ii) the projection of the second 3D model determined in the same way, but by selectively extracting the 3D structural points of the second frame (which have been correctly located using the 3D model derived from the first frame) from the volume of the bounding box in the second frame; or (iii) a composite model generated by aggregating these selectively extracted points from the second frame with those from the first frame (there may be many other frames to construct the dense composite model). Such 2D annotation data can be stored in association with a second perceptual input or another perceptual input of the second frame.

[0028] As can be seen from Example 1, for the first frame, the 2D and 3D annotation data for the first frame are generated by locating 3D boundary objects within the first frame. Furthermore, using the same operations, combined with the operation of aligning the 3D model in the second frame, additional 3D and 2D annotation data is provided for the second frame. While this example illustrates two frames, it should be understood that these principles can be applied to a larger number of frames, thus providing a more significant performance advantage in terms of annotation time and efficiency.

[0029] It should be understood that Example 1 above is one of many effective annotation workflows facilitated by certain embodiments of this annotation tool. This example is only used to illustrate certain features of the annotation tool and does not limit or restrict the scope of the invention. For the same purpose, more examples are described below.

[0030] This annotation tool is particularly suitable for annotating time-series frames, that is, a sequence of one or more frames captured within a certain time interval, typically at relatively short, regular time intervals. In this paper, such a time-series frame may be referred to as a “3D video sequence.” It should be noted that each frame includes 3D structure points, i.e., points capturing structures in 3D space. An example application is annotating 3D video sequences captured of moving vehicles or other moving objects to provide an annotated perceptual input ideally suited for training one or more perceptual components for use in autonomous vehicles or other mobile robots. For example, such frames could capture urban or non-urban road scenes, which could then be annotated to label road structures, other vehicles, pedestrians, cyclists, and any other form of structural components that an autonomous vehicle needs to perceive and respond to.

[0031] In this context, a "frame" refers to any captured 3D structural representation, including capture points that define the 3D spatial structure (3D structural points), providing a static "snapshot" (i.e., a static 3D scene) of the captured 3D structure within that frame. It can be said that the frame corresponds to a single moment in time, but this does not necessarily imply the need to immediately capture the frame or derive the underlying sensor data—for example, LiDAR measurements can be captured in a LiDAR scan in an "untwisted" manner for a short period (e.g., approximately 100 ms) to illustrate any motion of the moving object, thus forming a single-point cloud. In that case, despite the use of underlying sensor data for capture, the single-point cloud can still be said to correspond to a single moment in the sense of providing a useful static snapshot due to this untwisting. In the context of frame time series, the moment corresponding to each frame is the time index (timestamp) of that frame within the time series (and each frame in the time series corresponds to a different moment).

[0032] In the context of annotation tools, the terms "object" and "structural component" are used synonymously, referring to identifiable structural elements within a static 3D scene modeled as objects in 3D frames. It should be noted that, according to this definition, in the context of annotation tools, an object may actually correspond to only a portion of a real-world object or to multiple real-world objects, etc. That is, the term "object" is broadly applicable to identifiable structural elements captured in any 3D scene.

[0033] Regarding further terminology used herein, the terms “orientation” and “angular position” are used synonymously to refer to the rotational configuration of an object in 2D or 3D space (where applicable), unless otherwise stated. As can be clearly seen from the foregoing description, the term “position” is used broadly to encompass localization and / or orientation. Therefore, the position of an object, as determined, calculated, or assumed, may have only localization components (one or more localization coordinates), only orientation components (one or more orientation coordinates), or both localization and orientation components. Thus, generally speaking, position can include at least one of the following: localization coordinates and orientation coordinates. The term “pose” refers to a combination of the localization and orientation of an object, such as a full six-dimensional (6D) pose vector, which comprehensively defines the localization and orientation of an object in 3D space (the term “6D pose” can also be used as an abbreviation to represent the full pose in 3D space), unless otherwise stated.

[0034] The terms "2D perception" and "3D perception" can be used as abbreviations to refer to structure perception applied to 2D and 3D spaces, respectively. For the avoidance of ambiguity, these terms do not necessarily imply the dimensionality of the resulting structure-perceived output. For example, the output of a full 3D bounding box detection algorithm may be in the form of one or more nine-dimensional vectors, each defining a 3D bounding box (cube), such as 3D localization, 3D orientation, and size (height, width, length—bounding box dimensions). Alternatively, object depth can be estimated in 3D space, but in this case, a single-dimensional output may be sufficient to capture the estimated depth (as a single-dimensional depth). Furthermore, 3D perception can also be applied to 2D images, such as in monocular depth perception.

[0035] In some embodiments, the 2D annotation data may include 2D boundary objects of structural components, which are fitted onto the calculated 3D model projection in the image plane.

[0036] Alternatively or additionally, 2D annotation data may include segmentation masks of structural components.

[0037] 2D annotation data can be stored in association with at least one perceptual input of the frame to annotate structural components therein, and projections can be calculated based on reference positions computed within the frame.

[0038] 2D annotation data can be stored in association with at least one perceptual input of a second frame to annotate structural components in at least one perceptual input of the second frame, wherein at least a portion of the structural components is captured in the second frame.

[0039] 3D models can also be used to create 3D annotation data to annotate structural components in 3D space.

[0040] 3D annotation data can be created to annotate at least one structural component in a second frame of perceptual input, wherein at least a portion of the structural component is captured in the second frame.

[0041] 2D or 3D annotation data can be created to annotate at least one perceptual input of the second frame by calculating the alignment model position of the 3D model within the second frame, wherein the alignment model position is determined based on at least one of the following: (i) one or more manual alignment inputs received in the user interface with respect to the second frame, while rendering the 3D model to manually align the 3D model with the structural components in the second frame; (ii) automatic alignment of the 3D model with the structural components in the second frame.

[0042] Annotation data may include refined annotation data calculated by applying a predictive model based on selectively extracted 3D structural points.

[0043] The prediction model can be a conditional random field (CRF) or other probabilistic graphical model.

[0044] At least one perceptual input may include 2D image data of the frame or the second frame or 2D image data associated with the frame or the second frame, wherein the image plane is the image plane of the image data.

[0045] The frame or the second frame may include depth component data of the 3D image, and the image data of the perceived input is image data of the color component of the 3D image.

[0046] The method may include the following steps: applying noise filtering to at least one of the following to remove noise points: extracted 3D structural points for generating a 3D model, wherein the 3D model includes or originates from the filtered 3D structural points in the event; and calculated projections, wherein the 2D annotation data is the 2D annotation data of the filtered projections in the event.

[0047] Noise filtering can be applied based on filtering criteria that can be manually adjusted via the user interface of the computer system.

[0048] The reference position can be calculated based on one or more positioning inputs received at the user interface, while rendering visual indications of the intra-frame reference position to manually adjust the intra-frame reference position.

[0049] The reference position can be calculated automatically or semi-automatically for the reference frame.

[0050] The reference position can be calculated automatically or semi-automatically by applying the sensing component to the reference frame.

[0051] A frame can be one of multiple time-series frames, and the reference position can be calculated automatically or semi-automatically based on the structural component path within the time interval of the time-series frame.

[0052] Selectively extracted 3D structural points can be selectively extracted from frames used to generate 3D models based on reference locations and one or more boundary object dimensions, wherein the one or more boundary object dimensions are one of the following: (i) manually determined based on one or more dimension inputs received at the user interface; (ii) automatically determined by applying a perceptual component to the frame; (iii) semi-automatically determined by applying a perceptual component to the frame and further based on one or more dimension inputs; or (iv) assumed.

[0053] Selectively extracted 3D structural points can be a subset of points within a 3D volume defined by a reference location and the dimensions of one or more boundary objects.

[0054] 3D annotation data may also include one or more boundary object dimensions used to generate a 3D model or its transformations.

[0055] The method may include the following steps: training at least one perceptual component during training using the or each perceptual input, wherein annotation data of the perceptual input provides the Ground Truth of the perceptual input during training.

[0056] 3D annotated data can be used to train 3D perception components, and 2D annotated data can be used to train 2D perception components.

[0057] The 3D model may include a 3D surface model fitted to selectively extracted structural points, and the projection may include the projection of the 3D surface model onto the image plane.

[0058] 3D structural point sets can be presented in the form of point clouds.

[0059] The 3D structural point set can be transformed into a reference frame at the reference location in order to selectively extract the 3D structural points of the 3D model.

[0060] The 3D volume defined by the reference position and the dimensions of one or more boundary objects can be a cuboid aligned with the coordinate axes of the reference system.

[0061] A 2D annotation can be created for the second frame, in which a second 3D model is generated by using an alignment model, 3D structural points of the second frame are selectively extracted based on the alignment model position, and then the second 3D model is projected onto the image plane.

[0062] The second 3D model can be an aggregated 3D model generated by aggregating 3D structural points selectively extracted from the first and second frames.

[0063] The reference position can be calculated within the frame as the alignment model position of the existing 3D model of the structural component, based on at least one of the following: (i) one or more manual alignment inputs received in the user interface with respect to the frame, while rendering the existing 3D model to manually align the existing 3D model with the structural component in the frame; and (ii) automatic alignment of the existing 3D model with the structural component in the frame.

[0064] An existing 3D model may have been generated from one or more other frames that capture at least a portion of the structural components.

[0065] A 3D model can be an aggregated 3D model determined by aggregating selectively extracted 3D structural points with the 3D structural points of an existing 3D model.

[0066] Automatic alignment may include calculating the projection of an existing 3D model or 3D model onto a 2D image plane associated with the frame or second frame, and adjusting the position of the model in 3D space so that the projection matches the 2D features of common structural components in the 2D image plane.

[0067] 2D annotation data may include an instance segmentation mask that is essentially determined as an energy function, wherein the instance segmentation mask assigns an instance classification label to each pixel in a set of pixels in an image plane, wherein the energy function has: unary terms of a unary function aggregated on the segmentation mask, the unary function being applied to the instance classification label of each pixel, wherein the unary function is defined by one or more ray projection values, the ray projection values ​​being determined in the projection step by projecting rays through the pixel and determining whether and where the rays intersect the surface of the 3D model; and non-unary terms of a non-unary function aggregated on the segmentation mask, the non-unary function being applied to the instance classification labels of at least two pixels.

[0068] A non-univariate function may have one or more terms, each term being the product of a label compatibility function and a kernel, wherein the kernel measures at least one of the following differences: pixel position, pixel value, and pixel feature, such that the effect of the label compatibility function decreases as the one or more differences increase.

[0069] Pixel values ​​may include depth values ​​or disparity values, and the kernel of at least one item measures the difference between at least two depth values ​​or disparity values ​​of at least two pixels, such that the effect of the annotation compatibility function decreases as the difference between depth values ​​or disparity values ​​increases.

[0070] The annotation compatibility function can depend on the size of the instance to which at least one of the two pixels belongs.

[0071] An instance can be defined by an instance segmentation mask.

[0072] Although the techniques described above for determining instance segmentation masks were proposed within the context of object modeling techniques described above, they can be more broadly applied to any situation (online or offline) that requires instance segmentation masks.

[0073] Another aspect of the present invention provides a computer-implemented method for determining an instance segmentation mask for an image, the method comprising: receiving an image to be annotated in an annotation computer system; determining the relative pose between the image and a 3D object model; projecting light rays through each pixel of an image plane and determining whether and where the light rays intersect with the surface of the 3D model, thereby determining one or more ray projection values ​​for the pixel; determining the instance segmentation mask as a substantially optimized energy function, wherein the instance segmentation mask assigns an instance classification label to each pixel in a set of pixels in the image plane, wherein the energy function has: aggregating unary terms of a unary function on the segmentation mask, the unary function being applied to the instance classification label of each pixel, wherein the unary function is defined by one or more ray projection values, the ray projection values ​​being determined as in the projection step by projecting light rays through the pixel and determining whether and where the light rays intersect with the surface of the 3D model; and aggregating non-unary terms of a non-unary function on the segmentation mask, the non-unary function being applied to the instance classification labels of at least two pixels.

[0074] In another aspect, the present invention provides a computer-implemented method for determining an instance segmentation mask for an image, comprising: substantially determining the instance segmentation mask as an energy function, wherein the instance segmentation mask assigns instance classification labels to each pixel in a set of pixels within the image, wherein the energy function has: aggregating unary terms of a unary function on the segmentation mask, the unary function being applied to the instance classification labels of each pixel; and aggregating non-unary terms of a non-unary function on the segmentation mask, the non-unary function being applied to the instance classification labels of at least two pixels, wherein the non-unary function has one or more terms, each term being the product of a label compatibility function and a kernel, wherein pixel values ​​include depth values ​​or disparity values, and the kernel of at least one term measures the difference between at least the depth values ​​or disparity values ​​of at least two pixels, such that the effect of the label compatibility function decreases as the difference between the depth values ​​or disparity values ​​increases.

[0075] The present invention also provides a computer-implemented method for determining an instance segmentation mask for an image, comprising: substantially determining the instance segmentation mask as an energy function, wherein the instance segmentation mask assigns an instance classification label to each pixel in a set of pixels within the image, wherein the energy function has: aggregating unary terms of a unary function on the segmentation mask, the unary function being applied to the instance classification label for each pixel; and aggregating non-unary terms of a non-unary function on the segmentation mask, the non-unary function being applied to the instance classification labels for at least two pixels, wherein the non-unary function has one or more terms, each term determining at least one of the following difference measures: pixel position, pixel value, and pixel feature, such that the effect of a label compatibility function decreases as the one or more differences increase, wherein the label compatibility function depends on the size of the instance to which at least one of the at least two pixels belongs.

[0076] An instance can be defined by an instance segmentation mask.

[0077] Please refer to UK patent applications GB1910392.8, GB1910390.2, GB1910395.1 and GB1910382.9 (an earlier application) filed on July 19, 2019, and UK patent application GB2004842.7 filed on April 1, 2020, all of which are incorporated herein by reference in their entirety. All the subjects taught herein (including the foregoing aspects and embodiments) can be practiced in any of the foregoing contexts.

[0078] Other aspects of this disclosure provide a computer system comprising one or more computers programmed or otherwise configured to perform any of the steps disclosed herein, and one or more computer programs embodied on transient or non-transient media for programming the computer system to perform those steps.

[0079] The computer system may be embodied in a robotic system (e.g., an autonomous vehicle or other mobile robot) or as a simulator. Attached Figure Description

[0080] The embodiments of the present invention are illustrated below with reference to the accompanying drawings to provide a clearer understanding of the invention. In the drawings:

[0081] Figure 1 A highly schematic functional block diagram of a training system for training the perceptual components is shown.

[0082] Figure 2 A schematic block diagram showing the height of an autonomous vehicle is provided.

[0083] Figure 3 A schematic functional block diagram of the annotated computer system is shown;

[0084] Figure 4 A schematic perspective view showing a frame in the form of a point cloud;

[0085] Figure 5 A block diagram of a stereoscopic image processing system is shown.

[0086] Figure 6 This illustration demonstrates some of the principles of stereo depth extraction.

[0087] Figure 7 A to Figure 8E Various examples of graphical user interfaces (GUIs) rendered by the annotation computer system when annotating a time series of 3D road scenes are shown;

[0088] Figure 9A A flowchart illustrating the method for generating an object model;

[0089] Figure 9B A schematic diagram illustrating the method applied to point clouds is shown;

[0090] Figure 10 A to Figure 12C Showing more examples of the annotation system GUI, specifically demonstrating how the generated object model can be applied to create annotations for a time series of 3D road scenes;

[0091] Figures 13A to 13C This illustration demonstrates how vehicle routing information can be incorporated into an automatic or semi-automatic annotation process.

[0092] Figure 14 A flowchart illustrating the method for iteratively generating and propagating aggregated 3D object models;

[0093] Figure 15A and Figure 15B This demonstrates certain geometric principles of a form of SDF modeling;

[0094] Figure 16A This demonstrates the principle of dynamic voxel scaling;

[0095] Figure 16B Figure 16C shows the physical representation and data structure representation of the voxel tree;

[0096] Figure 17 This demonstrates some principles of SDF calculation based on maximum likelihood estimation;

[0097] Figure 18 An example of annotated pipelines is shown;

[0098] Figure 19 This demonstrates some of the principles of ray tracing. Detailed Implementation

[0099] The embodiments of the present invention will now be described in detail. First, some mechanisms that are beneficial to the embodiments are provided.

[0100] Figure 1 A highly schematic functional block diagram of a supervised training system for training a perceptual component 102 based on a set of annotated perceptual inputs 108 (i.e., perceptual inputs together with associated annotation data) is shown. In the following description, the perceptual component 102 may be referred to synonymously as a structure detector, a structure detection component, or simply a structure detector. As mentioned above, the perceptual inputs used for training purposes may be referred to herein as training examples or training inputs.

[0101] exist Figure 1 In the diagram, training examples are labeled with reference numeral 104, and a set of associated annotation data is labeled with reference numeral 106. Annotation data 106 provides ground truth for the associated training examples 104. For example, for training examples in image form, annotation data 106 can label the location of certain structural components (such as roads, lanes, intersections, non-driving areas, etc.) and / or objects (such as other vehicles, pedestrians, street signs, or other infrastructure, etc.) within image 104.

[0102] The annotated perceptual input 108 can be divided into a training set, a test set, and a validation set, labeled 108a, 108b, and 108c, respectively. The annotated training examples, being used for testing or validation, do not need to be part of the training set 108a to train the perceptual component 102.

[0103] The perception component 102 receives a perception input, denoted as x, from one of the training set 108a, the test set 108b, and the validation set 108c, and processes the perception input x to provide a corresponding perception output, denoted as .

[0104] y = f(x; w).

[0105] In the above text, w represents a set of model parameters (weights) of the perception component 102, and f represents the function w defining the weights and the architecture of the perception component 102. For example, in the case of 2D or 3D bounding box detection, the perception output y may include one or more detected 2D or 3D bounding boxes derived from the perception input x; in the case of instance segmentation, y may include one or more segmentation maps derived from the perception input. In general, the format and content of the perception output y depend on the selection of the perception component 102 and its chosen architecture, and these selections are made based on one or more desired perception modalities to be trained.

[0106] The detection component 102 is trained based on the perceptual input of training set 108a to match its output y = f(x) with the ground truth provided by the associated annotation data. The ground truth provided for the perceptual input x is denoted here as y. x Therefore, for training example 104, the Ground Truth is proved by the associated annotated data 106.

[0107] This is a recursive process in which the input component 112 of the training system 110 systematically provides the perceptual input of the training set 108b to the perceptual component 102, and the training component 114 of the training system 110 tunes the model parameters w in an attempt to optimize the error (cost) function, compensating for the difference between each perceptual output y = f(x; w) and the corresponding Ground Truth y. x The deviation is characterized by a defined metric (such as mean squared error, cross-entropy loss, etc.). Therefore, by optimizing the cost function to a limited extent, the overall error of the entire training set 108a relative to the Ground Truth measurement can be reduced to an acceptable level. The perceptual component 102 may be, for example, a convolutional neural network, where the model parameters w are the weights between neurons, but this disclosure is not limited thereto. It should be understood that several forms of perceptual models exist that can be usefully trained on appropriately annotated perceptual inputs.

[0108] Test data 108b is used to minimize overfitting, which means that beyond a certain point, increasing the accuracy of detection component 102 on training dataset 108a is detrimental to its ability to generalize to perceptual inputs not yet encountered during training. Overfitting can be identified as a point where increasing the accuracy of perceptual component 102 on training data 108a will decrease (or not increase) its accuracy on test data, where accuracy is measured according to an error function. The goal of training is to minimize the total error of training set 108a so as to minimize it without overfitting.

[0109] If necessary, the validation dataset 108c can be used to provide a final estimate of the performance of the detection component.

[0110] Figure 2 A schematic block diagram of an autonomous vehicle 200 is shown, which is shown as an example including a trained perception component 102 whose input is connected to at least one sensor 202 of the vehicle 200 and whose output is connected to an autonomous vehicle controller 204.

[0111] In use, the trained structure perception component 102 (an instance) of the autonomous vehicle 200 interprets the structure within the perceived input captured by at least one sensor 202 in real time according to its training, and the autonomous vehicle controller 204 controls the speed and direction of the vehicle based on the results in the absence of any driver input or with limited driver input.

[0112] although Figure 2 Only one sensor 202 is shown, but the autonomous vehicle 102 may be equipped with multiple sensors. For example, a pair of image capture devices (optical sensors) may be arranged to provide a stereo view, and a road structure detection method may be applied to the images captured from each image capture device. Other sensor modalities, such as LiDAR, RADAR, etc., may be alternatively or additionally provided on the AV 102.

[0113] It should be understood that this is a highly simplified description of some functions of autonomous vehicles. The general principles of autonomous vehicles are well known and will not be elaborated upon here.

[0114] exist Figure 2 In this context, to train the perception component 102 for use, the same vehicle or a similarly equipped vehicle can be used to capture training examples to capture training examples closely corresponding to one or more forms of runtime input. The trained perception component 102 will need to be able to be interpreted on the AV 200 at runtime. Autonomous or non-autonomous vehicles with the same or only similar sensor arrangements can be used to capture such training examples. In this context, 3D frames are used as the basis for creating annotated training examples, requiring at least one 3D sensor modality. However, it should be noted that the terminology is broadly applicable to any form of sensor data capable of capturing a large amount of available depth information, including LiDAR, RADAR, stereo imaging, time-of-flight imaging, or even monocular imaging (where depth information is extracted from a single image—in this case, a single optical sensor is sufficient to capture the underlying sensor data of the perception input to be annotated).

[0115] Furthermore, the techniques described herein can be implemented off-site, i.e., in a computer system such as a simulator, which performs path planning for modeling or experimental purposes. In this case, sensor data can be acquired from a computer program running as part of a simulation stack. In either scenario, the sensing component 102 can process the sensor data to identify objects. In a simulated environment, a simulated agent can use the sensing component 102 to navigate the simulated environment, and the agent's behavior can be recorded, for example, to flag security issues or as a basis for redesigning or retraining simulated components.

[0116] Embodiments of the present invention will now be described.

[0117] Figure 3 A functional block diagram of annotated computer system 300 is shown; for simplicity, this computer system may be referred to as annotation system 300. The purpose of annotation system 300 is to create data that can be used to train machine learning components, such as 2D or 3D structure detectors (e.g., 2D segmentation components, 2D bounding box detectors, or 3D bounding box detectors). Such data can be referred to as training data, and the training data output component 314 of annotation system 300 provides annotated training data in the form of a set of training examples with associated annotation data.

[0118] Each training example 321 is in the form of a structural representation (such as a 2D or 3D image, point cloud, or other sensor dataset in which structures are captured). Each training example 321 is associated with 2D annotation data 313 and / or 3D annotation data 309 created using the annotation system 300. 2D annotation data refers to annotation data defined in a 2D plane (or other 2D surface). For example, 2D annotation data can be defined in an image plane to annotate 2D structures within the image plane. 3D annotation data refers to annotation data defined in 3D space to annotate 3D structures captured in depth maps, point clouds, or other 3D structural representations. Each training example 321 and its associated 2D / 3D annotation data 313 / 309 are stored in an electronic storage device 322 accessible to the annotation computer system 300. Electronic storage device 322 is a form of computer memory from which each training example 321 and its associated annotation data are stored in a persistent region of the electronic storage device, thereafter persistently residing in the persistent region from which they can be derived or otherwise obtained for other uses, such as training one or more perceptual components (e.g., in an external training system).

[0119] As described below, various annotation features are provided that allow for the automatic or semi-automatic generation of such annotated data, thereby increasing the speed of creating such data and reducing the manpower required.

[0120] exist Figure 3 In this system, the annotation functionality is generally represented by a point cloud computing component 302, a road modeling component 304, a rendering component 306, a 3D annotation generator 308, an object modeling component 310, and a 2D annotation data generator 312. The annotation system 300 also includes a user interface (UI) 320 through which users (human annotators) can interact with the annotation system 300. The UI 320 provides an annotation interface (also referred to herein as an annotation tool) for accessing the annotation functionality.

[0121] The annotation system 300 is also shown to have an input for receiving data to be annotated (in this example, in the form of a time series frame 301).

[0122] In the following example, each frame is taken as an RGBD (Red Green Blue Depth) image captured at a specific moment. An RGBD image has four channels: three (RGB) are color channels (color components) that encode the “normal” image, and the fourth channel is a depth channel (depth component) that encodes the depth values ​​of at least some pixels of the image. RGB is used as an example, but this description is more generally applicable to any image with both color and depth components (or actually only a depth component). In general, the color components of an image can be encoded using one or more color channels (including grayscale / monochrome) in any suitable color space. The point cloud computing component 302 converts each frame into a point cloud format to allow the frame to be annotated in 3D space. More generally, a frame corresponds to a specific moment and can reference any dataset (such as multiple RGBD images, one or more point clouds, etc.) that has captured a static “snapshot” structure (i.e., a static 3D scene) for that moment. Therefore, the entire description of RGBD images below is equivalent to other forms of frames. When frames are received in point cloud form at annotation system 300, no point cloud conversion is required. Although the following example is described with reference to a point cloud derived from an RGBD image, the annotation system can be applied to point clouds of any form, such as monocular depth, stereo depth, LiDAR, radar, etc. Point clouds can also be derived from two or more such sensing modalities and / or from multiple sensor components of the same or different modalities by merging the outputs of different sensors. Therefore, the term "point cloud of a frame" can refer to any form of point cloud corresponding to a specific moment, including frames received in point cloud form at annotation computer system 300, point clouds derived from frames by point cloud computing component 302 (e.g., in the form of one or more RGBD images), or merged point clouds.

[0123] As described above, although a frame corresponds to a specific moment in time, the underlying data used to derive the frame can be captured within (typically shorter) time intervals and transformed as necessary to account for temporal changes. Therefore, a frame corresponding to a specific moment in time (e.g., represented by a timestamp) does not necessarily imply that all the underlying data has been captured simultaneously. Thus, the term "frame" encompasses point clouds received at different timestamps than the frame itself; for example, a LiDAR scan captured within 100 ms at a specific moment (such as the time of image capture) will be "deconstructed" into a single point cloud. The temporal sequence of frame 301 can also be referred to as a video segment (it should be noted that the frames of a video segment are not necessarily images and can be, for example, point clouds).

[0124] Each training example 321 includes data from at least one frame of video clip 301. For example, each training example may include at least one frame of RGBD image (a portion and / or components) or at least one frame of point cloud data.

[0125] Training is not the only application of current annotation techniques. Another available application is scene extraction, where annotations are applied to 3D data to extract scenes that can be run in a simulator. For example, this annotation technique can be used to extract the trajectories (path and motion data) of annotated objects, allowing the behavior of these objects to be replayed in a simulator.

[0126] Figure 10 Two example frames from video sequence 301 are shown below in detail. The depicted frames represent a road scene captured by a moving vehicle. The annotation system 300 is particularly well-suited for annotating road scenes, which can then be used to efficiently train a structure detection component for autonomous vehicles. However, many annotation features can also be applied usefully to other scenarios.

[0127] This article briefly summarizes the various annotation functions provided by the annotation system 300.

[0128] Some annotation features are based on an "object model," which is a 3D model of an object, i.e., a structural component (structural element) to be annotated. As mentioned above, in the context of annotation tools, the term "object" generally applies to any recognizable structure (such as a part of a real-world object, multiple real-world objects, etc.) modeled as an object within the annotation tool. Therefore, the term "object" is used in the following description without affecting this broad meaning.

[0129] The object model is defined as the intersection of a 3D bounding box and a point cloud of one (or more) frames.

[0130] In other words, the 3D modeling component 310 derives the object model of the object to be annotated from one or more frames of the video sequence 301 itself: a 3D bounding box (or other 3D bounding objects, such as templates) is placed around the points of the relevant object in a specific frame, and the object model is obtained by isolating a subset of points within the point cloud of that frame that lie within the volume of the 3D bounding box (or equivalently, the intersection of the 3D bounding box and the point cloud). This inherently provides the positioning and orientation of the 3D bounding box relative to the 3D object model, which can be encoded as reference points and orientation vectors fixed in the reference frame of the object model. The following will refer to... Figure 9A and Figure 9B This will be explained in detail.

[0131] This is achieved quickly by converting all points in the point cloud to be aligned with the axis of the bounding box, allowing a simple magnitude comparison to determine whether a point is enclosed. This can be implemented efficiently on a GPU (Graphics Processing Unit).

[0132] Once these points are isolated into an object model in this way, they can be used, for example:

[0133] 1. Generate tight 2D bounding boxes for relevant objects;

[0134] 2. Perform instance splitting;

[0135] 3. Manually improve the pose of the distant frame.

[0136] Responsive noise filtering is achieved by sorting points by their K nearest neighbors within a fixed radius (using a 3D tree to find the K nearest neighbors first).

[0137] Points can also be accumulated across frames (or otherwise propagated) to generate a more complete / dense object model. Using an accumulated model can yield improved noise filtering results because it is easier to separate isolated noise points from the points of the object itself captured across multiple frames.

[0138] For example, refined 3D annotation pose estimation can be obtained by fitting the model to point clouds of other frames, such as using the Iterative Closest Point (ICP) algorithm.

[0139] Model propagation can also provide improved instance segmentation for distant objects, and may also provide improved segmentation for nearby objects (e.g., in regions where depth data is lacking).

[0140] In addition to generating annotations for training data, object models can also be used to augment training examples (i.e., a form of "synthetic" training data). For example, objects can be manually introduced into training examples and annotated to provide an additional knowledge base from which the structure detection component can learn. This can be used, for example, to create more "challenging" training examples (which existing models perform poorly on), which in turn can provide performance improvements for more challenging inputs during inference (i.e., when the model is running).

[0141] Extending from item 3 above, by fitting the 3D object model to the point cloud of the second frame (the second point cloud), 3D annotation data can be automatically generated for the second frame. Since the location and orientation of the 3D bounding box relative to the 3D object model are known, the location and orientation of the 3D bounding box relative to the second point cloud (i.e., in the second point cloud reference frame) can be automatically determined by fitting the 3D object model to the second point cloud. See below for reference... Figure 1 A to Figure 1 D elaborates on this. This is an example of how a 3D object model from one frame can be "propagated" to a second frame to automatically or semi-automatically generate annotation data for the second frame. One potential assumption is that the object can be treated as a rigid body.

[0142] The object model can also be propagated from one frame to the second frame based on a 3D bounding box that is manually placed or adjusted in the second frame. This provides visual assistance to the annotator in placing / adjusting the 3D bounding box in the second frame. In this case, the human annotator sets the position and / or orientation of the 3D bounding box in the second frame. This can then be used to position and / or orient the 3D object model in the second frame based on the fixed position and orientation of the object model relative to the 3D bounding box. When the annotator adjusts the pose (orientation and / or position) of the 3D bounding box in the second frame, the orientation / position of the 3D object model exhibits matching changes to maintain a fixed position and orientation relative to the 3D bounding box. This provides the annotator with an intuitive way to fine-tune the position / orientation of the 3D bounding box in the second frame to align the 3D object model with the actual object within the visible range of the second frame: the annotator can see if the 3D model is not perfectly aligned with the actual object in the second frame and fine-tune the 3D bounding box as needed until it is aligned. This is obviously easier than trying to visually align the 3D bounding box with the relevant object directly, especially when the object is partially occluded. See below. Figures 11E to 11G This will be explained in detail.

[0143] These two forms of object propagation are not mutually exclusive: first, by fitting the 3D object model to the point cloud of the second frame, the 3D bounding box can be automatically located and oriented in the second frame, and then the annotator can manually fine-tune the 3D bounding box to minimize any visible differences between the 3D model and the actual object in the second frame (thus fine-tuning the location / orientation of the 3D bounding box in the second frame).

[0144] In this paper, the ability to generate models and propagate them across different frames can be referred to as “x-ray visual features” (this name originates from a specific use case where models from another frame (or other frames) can be used to “fill” partially occluded object regions, but model propagation is more generally applicable as discussed in this paper).

[0145] Extending from items 1 and 2 above, 2D annotation data for an RGBD image (or, for example, its color components) is generated by projecting a 3D object model onto the image plane of the image. In the simplest case, as described above, a subset of the point cloud within a given frame is isolated, this subset of points is projected onto the image plane, and processed to generate 2D annotation data. The 2D annotation data can, for example, be in the form of a segmentation mask fitted to the projected points or a 2D bounding box. In some cases, generating 2D annotation data in this manner may be useful, but this is based on a projection of a 3D model propagated from another frame as described above. See below. Figures 12A to 12C The generation of 2D annotation data is described in detail.

[0146] To further assist annotators, the 3D road modeling component 304 provides a 3D road model that can guide the placement of 3D bounding boxes when annotating road scenes. The following will refer to... Figures 8A to 8E This will be explained in detail.

[0147] First, some useful scenarios of the embodiments will be explained.

[0148] Figure 4 This shows a schematic perspective view of the height of point cloud 400, which is a set of 3D spatial points defined in a reference frame. This reference frame is defined by a coordinate system within the "3D annotation space" and the origin 402 of the coordinate system. In this example, the reference frame has a Cartesian coordinate system, such that each point in the point cloud is defined by a triplet of Cartesian coordinates (x, y, z).

[0149] This article describes several examples in conjunction with “stereo” point clouds, namely point clouds derived from one or more stereo depth maps (but as mentioned above, the annotation system 300 is therefore unrestricted and can be applied to any form of point cloud).

[0150] Figure 5 A schematic block diagram of a stereo image processing system 500 is shown. The stereo image processing system 500 is shown to include an image corrector 504, a depth estimator 506, and a depth transformation component 508.

[0151] The stereo image processing system 500 is shown having input terminals for receiving left and right images L and R that together constitute a stereo image pair. The stereo image pair consists of left and right images simultaneously captured by the left and right optical sensors (cameras) 502L and 502R of the stereo camera system 502. The cameras 502L and 502R are arranged in a stereo configuration, with the cameras staggered relative to each other with overlapping fields of view. This reflects the geometry of the human eye, enabling the human to perceive three-dimensional structures.

[0152] The depth map D extracted from the left and right image pairs L and R is shown as the output of the stereo image processing system 500. The depth map D assigns an estimated depth d to each pixel (i,j) of the "target" image of the stereo image pair. ij In this example, the target image is the right image R, so an estimated depth is assigned to each pixel of the right image R. Another image (the left image L in this example) is used as a reference image. The stereo depth map D can be, for example, in the form of a depth image or an image channel, where the value of a particular pixel in the depth map is the depth assigned to the corresponding pixel in the target image R.

[0153] Reference Figure 6 The pixel depth is estimated by a depth estimator 506 that applies the principle of stereo imaging.

[0154] Figure 6The upper part shows a schematic diagram of the image capture system 502, illustrating the basic principle of stereoscopic imaging. The left side shows a plan view (in the xz plane) of cameras 502L and 502R, shown as horizontally (i.e., in the x-direction) separated by a distance b (baseline). The right side shows a side view (in the xy plane), where cameras 502L and 502R are substantially aligned in the vertical (y) direction, so only the right camera 502R is visible. It should be noted that in this context, the terms "vertical" and "horizontal" are defined in the frame of reference of camera system 502; that is, vertical means that cameras 502L and 502R are aligned in the direction independent of the direction of gravity.

[0155] For example, pixels (i,j') in the left image L and pixels (i,j) in the right image R are shown as corresponding to each other because they each correspond to essentially the same real-world scene point P. The reference numeral I in the attached diagram denotes the image plane of the captured images L and R, and the image pixels are shown as lying in this plane. Due to the horizontal misalignment between cameras 502L and 502R, those pixels in the left and right images exhibit relative "parallax," such as... Figure 6 As shown in the lower part. Figure 6 The lower part shows a schematic diagram of the left and right images L and R captured by the calibrated cameras 502L and 502R, and the depth map D extracted from these images. The parallax associated with a given pixel (i,j) in the target image R refers to the offset between that pixel and the corresponding pixel (i,j') in the reference image L. This is due to the separation of cameras 502L and 502R and depends on the depth of the corresponding scene point P in the real world (distance from camera 502R along the z-axis).

[0156] Therefore, depth can be estimated by searching for matching pixels between the left and right images L and R of a stereo image pair: for each pixel in the target image R, a matching pixel is searched in the reference image L. The search for matching pixels can be simplified by inherent geometric constraints, namely, given a pixel in the target image, the corresponding pixel will appear on a known "epipole line" in the reference image. For an ideal stereo system with vertically aligned image capture units, the epipole lines are all horizontal, so for any pixel (i,j) in the given target image, the corresponding pixel (assuming it exists) will be vertically aligned, i.e., located in the same pixel row (j) in the reference image L as pixel (i,j) in the target image R. In practice, this may not be the case, as stereo cameras cannot be perfectly aligned. However, image correction is applied to images L and R by image corrector 504 to address any misalignment, thus ensuring that the corresponding pixels are always vertically aligned in the images. Therefore, in Figure 5In the diagram, depth estimator 506 receives corrected versions of the left and right images L and R from image corrector 504, from which depth maps can be extracted. Matching can be evaluated based on relative intensity, local features, etc. Several stereo depth extraction algorithms can be applied to estimate pixel disparity, such as global matching, semi-global matching, and local matching algorithms. In real-time scenarios, semi-global matching (SGM) generally provides an acceptable trade-off between accuracy and real-time performance.

[0157] In this example, assume that in the pixel matching search, pixel (i,j) in the target image R correctly matches pixel (i,j') in the reference image L. Therefore, the disparity is assigned to pixel (i,j) in the right image R.

[0158] D ij =j′-j.

[0159] In this way, parallax is assigned to each pixel in the target image, and a matching pixel can be found in the reference image (this does not have to be all pixels in the target image R: there are usually pixel regions on one side of the target image that are outside the field of view of another camera, so there are no corresponding pixels in the reference image; the search may also fail to find a match, or they may be deleted if the depth values ​​do not meet certain criteria).

[0160] The depth of each pixel in such a target image is initially calculated in the parallax space. Each parallax can then be converted into a distance unit using the camera's intrinsic parameters (focal length f and baseline b), as follows:

[0161]

[0162] Where d ij Let D be the estimated depth of pixel (i,j) in the target image R, expressed in distance units, i.e., the distance between camera 502R and the corresponding real-world point P along the optical axis (z-axis) of the stereo camera system 502. ij Let be the disparity of pixels (i,j) assigned to the target image R during pixel matching search. Therefore, in Figure 5 In the diagram, depth transformation component 508 is shown receiving the output of depth extraction component 506 in parallax space and transforming that output to the aforementioned distance space to provide a depth map D in distance units. Figure 6 In the lower half, the pixel (i,j) of the depth map is shown as having the value d assigned to the pixel (i,j) in the target image R. ij This is an estimated depth in units of distance.

[0163] As mentioned above, in this example, the right image R is the target image and the left image L is used as the reference image. However, more generally, either image can be used as the target image, and the other as the reference image. The choice of which image to use as the target image can be context-dependent. For example, in the context of an autonomous vehicle, a stereo camera captures images of the road ahead of the vehicle; the image captured by the camera closest to the center line of the road can be used as the target image (i.e., the right image of a vehicle driving on the left and the left image of a vehicle driving on the right).

[0164] Brief Review Figure 4 The origin 402 of the coordinate system corresponds to the position of the optical sensor 502R that captures the target image R when capturing the target image R (in this case, the right camera 502R). The z-axis is parallel to the optical axis of the camera 502R, and the x-axis and y-axis are aligned with the pixel row and column directions of the target image R, respectively (i.e., the pixel row represented by the subscript i is parallel to the x-axis, and the pixel column represented by the subscript j is parallel to the y-axis).

[0165] Point cloud computing component 302 can compute point cloud 400 from stereo depth map D based on the known field of view of camera 502R. For example... Figure 6 As shown in the upper part, the i-th column of pixels in the target image R corresponds to a set of angular directions α defined by angles on the xz plane within the camera's field of view. j Similarly, the j-th row of pixels in the target image R corresponds to a set of angular directions β defined by angles on the xy plane within the camera's field of view. j Therefore, the pixel (i, j) of the target image R corresponds to the angular direction (α) defined by the angle pair. j ,β i ).

[0166] Once the depth of pixel (i, j) is known, it can be used as a basis for... ij and the angular direction (α) corresponding to this pixel j ,β i To calculate the position of the corresponding real-world point in 3D space (using a 3D space point (x)). ij y ij , z ij (This is represented by α). In this example, angle α j and β i It is defined relative to the z-axis, therefore:

[0167] x ij =d ij tanα j ;

[0168] y ij =d ij tanβ i ;

[0169] zij =d ij .

[0170] More generally, the x and y components are determined as functions of pixel depth and the angular direction corresponding to that pixel.

[0171] As shown in the figure, 3D spatial point (x ij y ij , z ij Let be the point within the point cloud 400 that corresponds to pixel (i, j) in the target image R.

[0172] Furthermore, each point in the point cloud can be associated with color information derived from the target image R itself. For example, for an RGB target image, each point in the point cloud can be associated with the RGB value of the corresponding pixel in the target image R.

[0173] review Figure 3 The point cloud computing component 302 is shown to have an input terminal that receives an RGBD image 301 and processes the RGBD image 301 as described above in order to determine... Figure 6 The corresponding 3D point cloud 400 is depicted in the image. The point cloud can be determined from a single image or from multiple images merged from a common reference frame.

[0174] The 3D annotation generator 308 allows annotators to place (i.e., position, orient, and resize) 3D boundary objects within a reference frame of the point cloud 400 to be annotated. In the example below, the 3D boundary object takes the form of a 3D bounding box (cube), but this description also applies to other forms of 3D boundary objects, such as 3D object templates. This can be a manual or semi-automatic process.

[0175] Alternatively, all steps performed by the commentator can be automated, as described below.

[0176] The annotator places 3D bounding boxes via the UI 320 of the annotation system 300. Therefore, the 3D annotation generator 308 is shown as having a first input terminal coupled to the UI 320 of the annotation system 300 for receiving user input. The annotator can manually place 3D bounding boxes via the UI 320 to define desired structural elements (such as vehicles, cyclists, pedestrians, or other objects) within the point cloud 400. This is a form of 3D annotation that can be used, for example, to train a 3D structure detection component as described above, for this purpose as part of the 3D annotation data 309.

[0177] The road modeling component 304 is also shown as having an input for receiving at least the color components of an RGBD image and processing these color components to determine a 3D road model. For this purpose, it is assumed that a vehicle equipped with a stereo camera (such as...) Figure 2As shown, a series of images 301 are captured while the vehicle is traveling along a road, allowing a 3D model of the road along which the vehicle travels to be reconstructed based on the captured series of images. For this purpose, the method applied by the road modeling component 304 is described in International Patent Application PCT / EP2019 / 056356, the entire contents of which are incorporated herein by reference. This is based on "Structure from Motion (SfM)" processing, which is applied to the series of images to reconstruct the 3D path (self-path) of the vehicle in the captured images. This is then used as the basis for extrapolating the 3D surface of the road traversed by the vehicle. This is based on 2D feature matching between images in the video sequence 301.

[0178] The road model can also be determined using alternative methods, such as point cloud fitting, or ego path based on 3D structure matching applied to the depth map or point cloud and / or using high-precision satellite positioning (e.g., GPS). Alternatively, an existing road model can be loaded, and frames can be positioned within the existing road model as needed.

[0179] The aforementioned references use a 3D road model derived from the vehicle's own path to efficiently generate 2D annotation data to annotate road structure in the original image. In this context, the technique is extended to allow 3D bounding boxes to be effectively placed around other objects on the road, such as other vehicles, cyclists, etc., across multiple frames in video clip 301 by assuming that other road users generally follow the road shape over time.

[0180] Therefore, the 3D annotation generator 308 is shown as a second input with an output coupled to the 3D road modeling component 304. The 3D annotation generator 308 uses the 3D road model as a reference to allow annotators to "bind" 3D bounding boxes to the 3D road model. That is, the 3D bounding box is moved in a manner controlled by the 3D road model, which is particularly useful for annotating other road users, such as vehicles, cyclists, etc. For example, annotators can be given the option to move the 3D bounding box along the road, and the 3D bounding box will automatically reorient itself to match the shape and slope of the road or cross the road perpendicular to its current direction. This will be described in detail below.

[0181] 3D annotation data 309 is also shown as being provided back to 3D annotation component 309 as a third input. This means that 3D annotation data defined for one frame can be used to automatically generate 3D annotation data for another frame. This will be described in detail later.

[0182] The rendering component 306 is shown as having an input terminal connected to the output terminals of the point cloud computing component 302, the 3D road modeling component 304, and the 3D annotation generator 308, as well as an input terminal for receiving RGBD images. The rendering component 306 renders 3D annotation data 309 in a manner that can be meaningfully interpreted by human annotators within the annotation interface.

[0183] 1. Comment interface:

[0184] Figure 7 A schematic diagram of an example annotation interface 700 that can be rendered by the rendering component 306 via the UI 320 is shown.

[0185] Within the annotation interface 700, the color components of the RGBD image 702 (current frame) are displayed on the left. A top-down view 704 of the point cloud 400 of that frame is displayed on the right.

[0186] Furthermore, a projection 706a of the 3D road model onto the image plane of the RGBD image is superimposed on the displayed image 702. Similarly, a projection 706b of the 3D road model onto the top view is displayed as a superimposition on the top view of the point cloud 400.

[0187] Option 708 is provided for creating a new 3D bounding box for the current frame. Once creation is complete, options 710 and 712 are provided for moving the bounding box and resizing it, respectively.

[0188] The options 710 for moving the bounding box include options for moving the bounding box longitudinally along the road in either direction (±R, as shown on the right side of the top view) and options for moving the bounding box laterally across the road (±L).

[0189] Option 712 for resizing the bounding box includes options for changing the width (w), height (h), and length (l) of the bounding box.

[0190] Although depicted as UI elements to be displayed, keyboard shortcuts, gestures, etc. can be used alternatively to provide associated input.

[0191] The following describes a sample workflow for placing 3D annotation objects. It should be understood that this is merely one example of how annotators can utilize the annotation functionality of the annotation interface 700.

[0192] Figure 8A The annotation interface is shown after a new bounding box 800 is created. The bounding box 800 is placed at an initial position at road height in the 3D annotation space and oriented parallel to the road direction at that position (such as the direction captured in the 3D road model). To assist the annotator, the 3D bounding box 800 is projected onto the image plane displaying image 702 and the top view 704.

[0193] like Figure 8B As shown, when the annotator moves bounding box 800 along the road in the +R direction, bounding box 800 is automatically reoriented to remain parallel to the road direction. In this example, the annotator's goal is to manually fit bounding box 800 to the vehicles visible in the right half of the image and facing the image plane.

[0194] like Figure 8C As shown, once the annotator moves the bounding box 800 to the desired position along the road, it then moves it laterally (i.e., perpendicular to the road direction) to the desired lateral position—in this case, along the +L direction.

[0195] like Figure 8D and Figure 8E As shown, the commentator then adjusts the width (in this case, "-w" indicates a decrease) and height ("+h" indicates an increase) of bounding box 800 as appropriate. Fortunately, no length adjustment is needed in this example, but the length of the bounding box can be adjusted in the same way as required. The width of bounding box 800 remains parallel to the road direction at its position, while the height remains perpendicular to the road surface at its position.

[0196] The example above assumes that bounding box 800 remains bound to the 3D road model when adjusted. Although not shown, the annotation interface can also allow for "free" adjustment without being constrained by the 3D road model, meaning the annotator can move or rotate bounding box 800 as needed. For example, this could be helpful when annotating vehicles whose behavior sometimes deviates from the assumed behavior (e.g., during turns or lane changes).

[0197] 2. 3D object modeling:

[0198] review Figure 3 The object modeling component 310 implements a form of object modeling based on the output from the 3D annotation generator 308. As described above, the object model is a 3D model of the desired 3D structural component (modeled as an object) created by isolating a subset of the point cloud within a 3D bounding object defined in a point cloud reference frame (or equivalently determined as the intersection of the bounding box and the point cloud). The modeled object can, for example, correspond to a single real-world object (such as a vehicle, cyclist, or pedestrian to be annotated for training a structure detection component for use in autonomous driving vehicles), a portion of a real-world object, or a group of real-world objects.

[0199] Figure 9A A flowchart illustrating the method for creating an object model from a point cloud.

[0200] In step 902, the point cloud of the structure to be modeled is captured.

[0201] In step 904, the 3D boundary object in the form of a 3D bounding box (cube) is manually fitted to the desired structure (object) captured in the 3D point cloud.

[0202] In this example, for instance, referring to the above... Figures 8A to 8E The method described involves manually adjusting the 3D bounding box to fit the structure based on user input provided at user interface 320. This input is provided by human annotators with the aim of achieving the closest possible fit between the 3D bounding box and the desired structure.

[0203] Alternatively, the bounding box can be placed automatically. For example, the bounding box can be placed automatically based on a bounding box defined for another frame in the manner described above.

[0204] Alternatively, bounding boxes can be automatically generated by 3D structure detection components, such as trained neural networks.

[0205] Once the 3D bounding box has been placed, in step 906, a subset of the 3D point cloud within the bounding box is determined. The 3D bounding box is defined in a point cloud reference frame, thus making it meaningful to determine which points in the 3D point cloud lie within the interior volume of the 3D bounding box. In most cases, these points will correspond to the desired structural elements. As described above, this can be efficiently computed on the GPU by transforming the points to a coordinate system whose axes are perpendicular to the faces of the bounding box.

[0206] Figure 9B A schematic diagram of step 908 of the method described above is shown. In this example, point cloud 400 has captured spatial points (labeled 902 and 904 respectively) from the first and second vehicles. Additionally, points 906 of the surrounding road structure have been captured. By placing a tightly fitted 3D bounding box 800 around the second vehicle to a degree that makes it visible within point cloud 800, a subset of points within the 3D bounding box 800 can be isolated to provide a 3D model 912 of the second vehicle 902. Annotators often find it easier to define a 3D bounding box around a structural element (in this case, the second vehicle) than to individually select a subset of points belonging to the desired structural element.

[0207] review Figure 9A Additional processing can be applied to the object model to refine and improve it.

[0208] For example, noise filtering can be applied to the determined subset of points (as shown in step 910a). The purpose of noise filtering is to filter out “noisy points,” that is, points that are unlikely to actually correspond to the desired structural elements. These points may be due to noise in the underlying sensor measurements used to derive the 3D point cloud, for example. For example, the filtering could be K-nearest neighbor (K-NN) filtering to remove points with insufficient numbers of other neighbors (e.g., if the number of points within a point’s defined radius is less than a threshold, then that point can be removed). Filtering is applied according to filtering criteria that can be manually adjusted via user interface 412 (e.g., the radius and / or threshold can be adjusted). More generally, one or more parameters of the modeling process (such as filtering parameters) can be manually configured, which in Figure 3 The input representation is from UI 320 to object modeling component 310.

[0209] Alternatively, an aggregated object model can be constructed across multiple frames (as shown in step 910b).

[0210] In this regard, it should be noted that the object modeling component 310 is capable of constructing both "single-frame" and "aggregate" object models.

[0211] A single-frame object model refers to an object model derived from sensor data captured at a single moment, i.e., an object model derived from a single frame in the sense described above. This includes object models derived from a single point cloud as well as object models derived from multiple point clouds captured simultaneously. For example, multiple pairs of stereo cameras can simultaneously capture multiple RGBD images and merge them to provide a single merged point cloud.

[0212] A multi-frame object model refers to an object model derived from sensor data captured at multiple times, such as an object model derived from RGBD images captured at different times, where at least part of the object to be modeled is captured. By isolating a subset of each point cloud in the aforementioned manner and then aggregating the point cloud subsets in a common reference frame, an aggregated object model can be determined from two or more point clouds corresponding to different times. This provides a denser object model and addresses partial occlusion of the object to be modeled by effectively “filling” occluded parts of one point cloud with points from another point cloud captured at different times.

[0213] To generate 2D or 3D annotation data, an object model can be applied to one or more frames from which the object model is derived, so as to generate annotation data for that frame or frames.

[0214] However, the object modeling component 310 can also propagate object models between frames. Propagation (effectively transferring point cloud data from one frame to another) is the process of applying an object model created using data from a point cloud in one frame to another to generate annotation data for that new frame. Single-frame and aggregated object models can be propagated in this sense.

[0215] As mentioned above, the purpose of propagating object models can also be to generate enhanced training examples.

[0216] Figure 9A Reference numeral 910c in the accompanying figure indicates an optional surface reconstruction step in which a surface mesh or other 3D surface model is fitted to selective points. This 3D surface model can be fitted to points selectively extracted from a single frame (single-frame object model) or multiple frames (aggregated object model). This effectively "smooths" a subset of the point cloud (single frame or aggregate) into a continuous surface in 3D space. For this purpose, well-known surface fitting algorithms can be used, which can be, for example, based on a directed distance function (SDF) to minimize the distance metric between the extracted points and the reconstructed 3D surface. It should be understood that this description relates to an object model, which may include the 3D surface model generated in this manner.

[0217] Figure 10 Two example frames are depicted, labeled 1001 (first frame) and 1002 (second frame). A first and a second object (both vehicles) are visible in both frames, labeled 1021 (first vehicle) and 1022 (second vehicle), respectively. For each frame, a camera view (in the image plane of the relevant frame) and a top view (of the associated point cloud) are depicted from the left and right sides, respectively.

[0218] In the second frame, the first vehicle is partially obscured by the second vehicle. In this example, in the first frame captured at a later time, the first vehicle is no longer obscured.

[0219] It can also be seen that in the second frame, both vehicles are relatively far apart. As a result, as schematically shown in the top view of the second frame, the point cloud data captured for each vehicle in the second frame (i.e., the number of points in the associated point cloud corresponding to the vehicle) is expected to be relatively small. Point cloud data for distant objects will generally also be noisier and less accurate. One factor is that, due to the inverse relationship between parallax and distance, a given error in parallax space will translate into a larger error for more distant points in distance space.

[0220] However, in the first frame, the first vehicle is significantly closer to the camera. Therefore, as schematically shown in the top view of the first frame, the point cloud data of the first vehicle in the first frame is generally denser and of higher quality (less error, less noise, etc.). In addition, the first vehicle is also more complete because it is no longer occluded (i.e., more of the first vehicle is covered).

[0221] 3. Object model propagation:

[0222] Two examples describing object model propagation are presented below—automatic bounding box alignment (3.1) and manual bounding box alignment (3.2). They are achieved using... Figure 10 The frame is used as a reference for description.

[0223] 3.1 Automatic bounding box alignment:

[0224] Figure 11A Showing the first frame (1001) currently selected for annotation. Figure 10 The annotation interface 700 is used. Using the tools provided within the annotation interface 700, the annotator accurately places the tight bounding box 800 around the first vehicle (marked with reference numeral 1021) as described above. The tight bounding box 800 is defined in the point cloud reference frame of the first frame.

[0225] Figure 11B The annotation interface 700 is shown, but at this time the second frame (1002, Figure 10 Currently selected for annotation. The bounding box 800 defined in the first frame has been imported (propagated) into the second frame, but at this point, the coarse estimated pose 1121 (localization and orientation) is determined only within the second frame. This coarse estimated pose 1121 is defined in the global reference but within the point cloud of the second frame.

[0226] A rough estimate of pose 1121 can be manually defined by the annotator. This process is simple and requires minimal effort from the annotator.

[0227] Alternatively, a coarse estimate of the pose 1121 can be determined automatically, for example, using a trained perception component—a “Model in the Loop” (MITL) processing form.

[0228] Alternatively or additionally, the coarsely estimated pose 1121 may be determined by interpolation based on an assumed or measured path of the first vehicle (1021). See below for details.

[0229] For rigid objects (i.e., objects modeled as rigid), the size and dimensions of the bounding box 800 remain unchanged across all annotations (in the same sense across all frames—the dimensions can be reflected across all frames by applying adjustments to one frame).

[0230] By applying appropriate transformations to the dimensions of boundary objects across frames, non-rigid objects, such as pedestrians and cyclists, can be accommodated. For example, this can take into account information about the type or category of the relevant objects.

[0231] The most convenient way to obtain a coarse estimate of the bounding box pose 1121 automatically via interpolation or MITL is to give it a coarse pose and orientation with the same “true” dimensions (width, length, and height).

[0232] It should be noted that although a tight bounding box is referenced in this context, an initial tight 3D bounding box is not necessary: ​​one or more “coarse” bounding boxes (i.e., those that are not tightly fitted to the object to be annotated 1021) – which may be automatically or manually generated – are sufficient to determine the vehicle structure present across multiple frames (and thus apply the annotation features of this disclosure). Therefore, although bounding box 800 may be referred to as a tight bounding box in the following description, bounding box 800 does not need to be tight to implement the features.

[0233] It should be noted that there is a difference between a coarse bounding box that is not accurately positioned or resized in any frame and a coarse pose of a bounding box in a given frame—for the latter, when a coarse position is defined in a given frame, the accurate pose may or may not be determined for different frames.

[0234] If a tight bounding box is not initially provided, the annotator may need to correct the orientation of the axes relative to the "optimized" box pose at some point. This can be done before or after the bounding box 800 propagates to other frames, and only for one frame, as these corrections will be automatically applied to all frames to which the bounding box 800 propagates.

[0235] Figure 11C A flowchart illustrating the object model propagation method is provided, along with graphical illustrations of the method steps. In step 1142, a subset of the point cloud from the first frame is extracted within the bounding box 800 of the first frame to provide an object model 1143 for the first vehicle. In step 1144, the object model 1143 from the first frame is fitted to a subset of the point cloud from the second frame. As described above, the fitting can be performed based on ICP or any other automatic alignment algorithm that attempts to match the object model structure with the point cloud structure. Any color information associated with points in the point cloud can also be used as a basis for fitting (in this case, attempting to fit the points of the model to points in a similarly colored point cloud, plus structural matching). The alignment process can also be referred to as “registering” the 3D model 1143 with the point cloud of the second frame. The algorithm searches within the point cloud for matching structures to which the model 1143 can be aligned (i.e., registered).

[0236] The coarse bounding box pose 1121 can be used to limit the search range, for example, to a search volume within the 3D space defined by the coarse bounding box pose 1121. The search volume can be additionally defined by the size / dimension of the bounding box 800. However, the search volume does not necessarily have to be limited to the volume within the bounding box 800. For example, the search volume can be extended by an additional “buffer” around the 3D bounding box 800. Alternatively, the search volume can be manually defined, for example, by “lasso” selection of a 2D rectangle or freeform shape in the image or by one of the projected 3D views. Alternatively, the search can be performed across the entire range of the point cloud in the second frame, but this may be inefficient.

[0237] Image features can also be used to assist point cloud registration, such as edges, corners or other feature descriptors, such as the Scale Invariant Feature Transform (SIFT) algorithm.

[0238] In general, although object model 1143 is aligned with the object (first vehicle) in 3D space, this may or may not be based on 3D structural matching. That is, the 3D pose of object model 1142 is adjusted so that the 3D features of object model 1143 match the corresponding 3D features of the first vehicle (e.g., using the ICP or other automatic 3D registration process described above). For example, alternatively or additionally, alignment in 3D space can be based on 2D feature matching, that is, the 3D pose of object model 1142 is adjusted so that the 2D features of object model 1143 match the corresponding 2D features of the first vehicle (e.g., using image features of the type described above).

[0239] Alternatively, or additionally, alignment in 3D space can be based on reprojection error or other photometric cost functions. This involves projecting the 3D object model 1143 onto the image plane and adjusting the 3D pose of the object model 1143 so that the calculated projection matches the first vehicle appearing in the image. This can also be based on image feature matching between the image and the projection of the object model 1143 onto the image plane.

[0240] When noise filtering is applied to points of a 3D object model, the 3D surface model can be fitted to the filtered points (i.e., the points remaining after the noise points are filtered out).

[0241] All cameras and bounding boxes have a position and orientation (pose) relative to the world (global) coordinate system. Therefore, once the pose of bounding box 800 has been determined in the first frame, it is possible to find the bounding box 800 relative to another camera (i.e., the same camera, but at a different time—e.g., corresponding to the second frame), which allows the bounding box 800 to be placed (localized and oriented) in the point cloud coordinate system of that camera. This point cloud can then be transformed into the coordinate system of the bounding box to effectively isolate a subset of the point cloud within the bounding box (see above).

[0242] Reference Figure 11DThe tight bounding box 800 and the 3D object model 1143 are derived from the same reference frame (in this case, the reference frame of the bounding box 800). Therefore, when the model is created in step 1142, the location and orientation of the tight bounding box 800 relative to the object model 1143 are known. Consequently, when fitting the 3D object model 1143 to the point cloud of the second frame, in step 1144, the location and orientation of the tight bounding box 800 in the point cloud reference frame of the second frame are automatically determined. This is encoded as a reference point (location) 1152 and an orientation vector 1154 for the bounding box 800, where the reference point 1152 and the orientation vector 1154 are fixed relative to points on the object model 1143. Assuming the relevant object (in this example, the first vehicle) can be considered a rigid body, the tight bounding box 800 initially defined in the first frame will now be reasonably accurately located and oriented to tightly fit the subset of points belonging to the first vehicle in the second frame (the accuracy will depend on how well the object model fits the point cloud).

[0243] In this way, high-quality 3D annotation data is generated semi-autonomously for the second frame, which can then be used to train machine learning 3D bounding box detectors, orientation networks, or (for example) distance estimation or any other form of 3D structure-aware component.

[0244] Further steps can be taken to automatically generate 2D annotation data for the underlying image of the second frame, as described below.

[0245] For the reasons stated above, the coarse estimate of the bounding box pose 1121 is also used as a coarse estimate of the pose of the 3D object model 1143. This is further refined by automatically, manually, or semi-automatically aligning the 3D object model 1143 better with the first vehicle in the second frame.

[0246] Although in the example above, bounding box 800 was manually placed in the first frame, this step can be automated. In the MITL method, bounding boxes can be automatically placed by an automatic object detector (such as a trained neural network) (which may or may not undergo manual fine-tuning). For example, a bounding box detector might perform well for the first vehicle in the first frame but poorly when directly applied to the second frame. In this case, leveraging the good performance of the bounding box detector in the first frame, high-quality training data can be automatically or semi-automatically generated for the second frame. This, in turn, provides high-quality training data for the second frame, which can then be used to train / retrain to improve object detection performance. Alternatively or separately, pose can be estimated by interpolation based on measurements or hypothetical paths of positively annotated objects (see below).

[0247] It should also be understood that this is merely one example of the workflow an annotator can employ using the provided annotation capabilities. The potential efficiency stems from the fact that changes made to the 3D bounding box relative to the object model in one frame are automatically applied to one or more other frames to keep the 3D bounding box of a rigid object consistent across frames. Thus, for example, an annotator might initially perform a coarse annotation on the first frame, apply the above steps to position and orient the coarse bounding box of the first frame in the second frame, and then apply adjustments to the positioning and / or orientation of the bounding box in the first frame, which are automatically reflected in the second frame. In this case, adjustments to the bounding box in one frame are automatically applied across multiple frames.

[0248] It should be noted that, in the sense that the orientation of bounding box 800 relative to object model 1143 remains consistent across frames, the reference point 1152 and orientation vector 1154 of bounding box 800 relative to object model 1143 are fixed—however, the annotator can adjust the position and orientation of the bounding box relative to the object model as needed (i.e., he can change the reference point 1152 and orientation vector 1154), and any such adjustments will be automatically implemented in all frames to which the object model is applied. In this sense, 3D bounding box 800 is "locked" to 3D object model 1143.

[0249] 3.2 Manual bounding box alignment

[0250] Following the previous example, Figure 11E The expanded view of the annotation interface 700 is shown, with the second frame selected for annotation. It can be seen that the object model 1143 is superimposed on the camera view and the top view through projection (1152) so that the annotator can see the relative position of the object model 1143 relative to the related object 1021 (the first vehicle) in the reference frame of the second frame.

[0251] Figure 11F This demonstrates how commentators utilize this feature. For example... Figure 11F As shown in the upper part, when object model 1143 is first fitted to the point cloud of the second frame, it may not be perfectly aligned with the actual points of object 1021. This is in Figure 11F It is easy to see that the reason is the visual misalignment between object model 1143 and the actual object 1021. Therefore, as Figure 11F As shown in the lower part, the annotator can fine-tune the pose (localization and orientation) of bounding box 800 to correct misalignment: in the sense described above, as bounding box 800 is adjusted, object model 1143 remains locked to bounding box 800, and any changes in the pose of object model 1143 in the current point cloud reference frame apply to the pose of 3D bounding box 800 in that reference frame. Therefore, when no perceptible misalignment exists, the annotator knows that bounding box 800 has been correctly localized and oriented in the reference frame of the second frame. Although Figure 11FAlthough not shown in the diagram, object model 1143 is also projected into the top view, allowing the annotator to simultaneously correct any visual misalignments in that top view.

[0252] In comparison, Figure 11G Showing with Figure 11F The upper half is the same view, but the object model 1143 is not overlaid. At this point, the bounding box 800 is still misaligned, but this misalignment is more difficult to perceive without model projection. This situation is partly due to partial occlusion of the first vehicle. By propagating the object model 1143 from frames where the object is not occluded, the occlusion in the second frame is corrected, thus assisting the annotator in fine-tuning the bounding box 1143.

[0253] Besides correcting occlusion, propagating object models in this way also helps address sparsity, noise, and inaccuracies in point clouds, especially for distant objects. (Review) Figure 10 To reiterate, the first vehicle is farther away in the second frame, so the point cloud subset corresponding to the second vehicle will typically be sparser and of lower quality in the second frame. This is another reason why it can be difficult to manually place the bounding box accurately in the second frame, as this will be reflected in the quality of the top view. Propagating object model 1143 from the first frame to the second frame in the manner described above helps annotators compensate for this.

[0254] Model propagation can also be used to address "gap" in sensor coverage. For example, with stereo depth images, depth estimation relies on locating matching pixels between a target image and a reference image. The target image often contains regions where some pixels have no corresponding pixels in the reference image. This corresponds to a portion of the scene within the field of view of the camera capturing the target image but outside the field of view of the camera capturing the reference image. For the object portion within this region in a given frame, no depth information will be available. However, this depth information can be obtained from another frame by propagating the object model from other frames as described above. This could be, for example, a frame with the closest temporal proximity, where the relevant object is fully visible in the depth channel.

[0255] 4. Aggregate object model:

[0256] To create an aggregated object model, bounding boxes are placed around the relevant objects across multiple frames (e.g., as described above, or simply placed manually). For each frame, a subset of the point cloud is isolated within the bounding box of that frame, and then aggregated in a common frame of reference. As described above, this provides a denser and less noisy object model, which can then be applied to one or more frames to generate high-quality annotation data.

[0257] Following Figures 11A to 11GIn the example above, the bounding box 800 has been accurately placed in the second frame. A subset of points in the second frame can be extracted from this bounding box and merged (aggregated) with the corresponding subset of points extracted from the bounding box 800 in the first frame. This provides a denser model of the first vehicle, which can then be propagated to other frames in the manner described above.

[0258] Annotators can also use aggregated models to guide manual annotation. When an annotator adjusts the position / pose of bounding box 800 in the target frame, an aggregated model is rendered using data from the target frame and at least one other frame (reference frame). If the bounding box is incorrectly positioned or oriented, this can result in visible artifacts in the aggregated model, such as “repetitive” or misaligned features, because points obtained from the target frame are not correctly registered with points obtained from the reference frame. The user can then fine-tune the pose of bounding box 800 in the target frame as needed until visual artifacts no longer exist.

[0259] 4.1 Iterative Propagation and Generation of Aggregation Model

[0260] Aggregated 3D models can be generated and applied iteratively.

[0261] Now refer to Figure 14 To illustrate one example, this diagram shows a flowchart of a method for iteratively generating and applying increasingly dense aggregated 3D models across multiple frames (potentially a large number of frames).

[0262] First, in step 1402, a single-frame object model is generated for the initial single frame (the current object model at this point in the process), wherein a 3D bounding box is placed around the object in the frame (automatically, semi-automatically, or manually), and object points are extracted from the frame within the 3D bounding box.

[0263] In step 1404, the current object model is propagated to the next frame and aligned in 3D space with the objects in that frame (1406) (manual, automatic, or semi-automatic). This derives the pose of the 3D bounding box in that frame, so in step 1408, points belonging to the object can be extracted from the 3D bounding box in that frame. These points are then aggregated with the object points of the current object model (1410) to generate a new aggregated 3D model that incorporates the point information extracted from the most recent frame.

[0264] This process can then be repeated iteratively for the next frame, starting from step 1404—it should be noted that from this point onward, the current object model propagated to the next frame is an aggregated object model that incorporates point information from multiple frames. Therefore, from this point onward, the current aggregated object model is aligned with the objects in the next frame. With each iteration, points from yet another frame are added, allowing for the generation of an increasingly dense and complete aggregated object model.

[0265] 4.2 Object Modeling with Signed Distance Field (SDF)

[0266] In the simplest case, an object can be modeled as an aggregated point cloud. However, as mentioned above, there are various other forms of object modeling that can be applied to model the surface of a 3D object based on cross-frame point cloud aggregation. One of the examples above is SDF-based modeling, and an example of such an SDF implementation will now be described. The following description takes into account directed distance fields, but the same applies to other forms of distance fields that can be used to (implicitly) represent the surface of a 3D object.

[0267] In this specific case, a method for simultaneous localization and mapping of a directed range field using adaptive sampling is adopted.

[0268] This implementation generates an SDF model per object, which aggregates the object's point cloud data across a batch of frames (e.g., frames captured over a driving distance on the order of 10 or 100 meters). The SDF form is a discrete SDF that represents (models) the 2D surface of a 3D object as 3D voxels (volume pixels) in the object's reference frame, where each voxel encodes a "directed distance" to the 3D object surface; the directed distance to points on the surface itself is zero. For convenience, this description assumes a typical convention that points outside the object have positive distances and points inside the object have negative distances, but this is only one possible choice. Voxels within the SDF do not need to have a fixed size; in the example below, the size (3D dimensions) of voxels closer to the object's 2D surface is reduced (while the voxel density correspondingly increases) to provide more memory-efficient methods for modeling granular surfaces.

[0269] First refer to Figure 15A and Figure 15B This provides a brief overview of this SDF-based implementation. This implementation assumes an RGBD frame with a corresponding point cloud, but the description is equally applicable to any form of 3D image with pixels having depth and (optionally) color components.

[0270] The 3D bounding box of an object (placed manually, semi-automatically, or automatically) is used to isolate a subset of the 3D point cloud for each object in each frame, in a manner described in other paragraphs of this document. This, in turn, is used to determine the object and the RGBD image of the frame, which consists of a subset of the RGBD pixels of the frame corresponding to the subset of the 3D point cloud isolated for the object in that frame. That is, in each frame, based on its manually, automatically, or semi-automatically estimated 6D pose and 3D dimensions, a subset of depth (or more precisely, parallax) and color pixels is determined for each object to belong to that object. The term "parallax image" (or parallax map) refers to an image in which each pixel (at least some pixels) has a measured parallax, such as the depth component (D) of an RGBD image.

[0271] 4.2.1 Example SDF Implementation Overview

[0272] Figure 15A The bounding box 800 of the 3D object is shown, along with a subset of pixels 1504 within the image plane 1502 corresponding to the 3D points within the bounding box 800. For simplicity, only three points from the point cloud are shown, but in reality, each effective depth pixel has a 3D point, and the subset of those points within the bounding box 800 defines the pixel subset 1504 (object pixels) of the relevant object. The remaining pixels within the image plane 1504 are called the "mask" for the object and are the unmasked object pixels 1504 used to construct the SDF object model labeled by the bounding box 800. The orientation of the image plane 1502 in 3D space is defined by the estimated 6D camera pose (localization and orientation in 3D space) indicated by reference numeral 1500 in the object reference frame. As described below, across a batch of frames, the camera pose relative to each object is estimated and refined in each frame based on the SDF model. As described in other paragraphs of this document, this can be iterative, where the SDF and camera pose are refined sequentially, for example, by a predetermined number of iterations or until certain convergence criteria are met. More generally, it involves determining the relative pose between the camera and the object, which could be the pose of the camera in the object's frame of reference or the pose of the object in the camera's frame of reference.

[0273] In addition, for each object pixel, a “surface normal” is estimated at a point on the surface corresponding to that pixel. This is an estimate of the normal vector (i.e., the vertical direction) of the 2D surface of the object.

[0274] To estimate the surface normal for each pixel, a superpixel segmentation of the RGBD image is formed, and a slope is fitted to each superpixel to estimate the normal. That is, the slope is fitted to a subset (relatively small) of the 3D object points corresponding to the pixel and its nearest neighbors. For more details on a technique for estimating surface normals pixel by pixel, please refer to "Efficient Joint Segmentation, Occlusion Labeling, Stereo and Flow Estimation" by Yamaguchi et al., 2014, which is cited in full and incorporated herein by reference.

[0275] Figure 15B Two object pixels (i.e., two pixels in object pixel set 1504), j and k, are shown, where the subscript i represents the frame i to which these pixels belong. The measured disparity values ​​of these pixels are denoted as d. ij and d ik Pixel extras and their respective surface normals Correspondingly, as shown in the figure, these are estimates of the vertical (normal) direction of the object surface at the corresponding 3D object point. The surface normal is estimated for each object pixel based on the disparity value of object pixel 1504. For calculating the SDF, the following example assumes that the object surface is locally flat (planar) at each object point, as detailed below. Reference numerals 1506-j and 1506-k are used to denote segments j and k of the object surface plane at the point corresponding to the pixel, respectively.

[0276] 4.2.2 Directed distance field

[0277] The "Directed Distance Field" (SDF) is a model that represents a surface as a directed distance scalar field. At each point, the value of the field is the shortest distance from that point to the surface of the object; it is negative if the point is outside the surface and positive if the point is inside the surface.

[0278] For example, the equation of a two-dimensional sphere (2-sphere) with radius r is:

[0279] x 2 +y 2 +z 2 =r 2

[0280] The corresponding SDF value is denoted as F, as shown below:

[0281]

[0282] The value of field F at a certain point is negative when that point is outside the surface and positive when that point is inside the surface. The surface can be reconstructed as the zero set of the field, that is, the set of points where it is zero.

[0283] While the SDF can be continuously defined in the simple example above, in this context, for practical reasons, the SDF approximates a finite number of points. The minimum size of surface features that can be accurately recorded is then determined by the sampling rate. For example, sampling points on a regular 3D mesh cannot record details smaller than the distance between mesh points. Therefore, a higher sampling rate can record finer details. However, the amount of memory required to store this information is proportional to the cube of the sampling rate, so sampling on a regular mesh is not feasible for large or highly detailed surfaces. In practice, only points near the surface are needed to reconstruct the surface; areas with less detail on the surface can be accurately reconstructed using relatively sparse sampling. Only areas near the surface with a high degree of detail require dense sampling for reconstruction. Memory can be saved by using adaptive sampling methods, which change the sampling rate at different locations based on the required level of detail. This can be achieved by storing the samples in a "voxel tree," such as an octree or a 64-bit tree.

[0284] 4.2.3 Voxel Tree

[0285] A voxel tree is a data structure that allows variable-resolution 3D spatial positioning data to be stored in a memory-efficient manner. Specifically, a large 3D space is subdivided into several cubic voxels of different sizes, with smaller voxels in the region requiring higher resolution data storage.

[0286] Figure 16A An example of an octree is shown, where a cube voxel can be subdivided into 8 smaller voxels, each of which can be further subdivided into 8 even smaller voxels, and so on, continuing as needed until the desired resolution is achieved within the desired area. Data can then be associated with the voxel itself or its vertices. Each larger subdivided pixel can be called a "parent" voxel, and its smaller constituent voxels can be called "child" voxels (one or more of which can be further subdivided, thus having their own child voxels). In this sense, voxels form nodes in a hierarchical tree structure, with edges representing parent-child relationships. "Leaf" voxels (nodes) are voxels that are not subdivided themselves. Leaf nodes can appear at any tree level; for voxels closer to the object's surface, leaf nodes will appear at lower tree levels.

[0287] Each voxel in an octree can be identified by a unique bit sequence called a "Morton code". For each voxel, a 3-bit sequence is associated with each of its 8 child voxels based on the location of the child voxel and the chosen x, y, and z directions. If the child voxel is located on the negative x-side of the voxel, the first bit is 0; if it is located on the positive x-side, the first bit is 1. Similarly, the second bit relates to the location of the child voxel in the y-direction, and the third bit relates to its location in the z-direction. In this example, the convention used is: the positive x-axis direction is right, the positive y-axis direction is down, and the positive z-axis direction is forward.

[0288] Figure 16B An example of an SDF voxel mesh with dynamic voxel sizes is shown in Figure 16C, which illustrates the corresponding voxel tree hierarchy. Larger voxels (including those labeled Vx1 and Vx2) correspond to the first-level nodes under their parent nodes; in the figure, voxel Vx2 is subdivided into eight smaller voxels (including voxels Vx3 and Vx4), shown in the second level of the tree; in the figure, such voxels are further subdivided into eight even smaller voxels (including Vx5), shown in the third level of the tree.

[0289] For example, in Figure 16BIn the first-level subdivision, voxel Vx2 has a 3-bit sequence 100 because it is the rightmost (positive x-direction), top (negative y-direction), and front (negative z-direction) sub-voxel of the larger red voxel. Similarly, voxel Vx4 has a 3-bit sequence 101, and voxel Vx5 has a 3-bit sequence 101. The complete Morton code is obtained by concatenating the 3-bit sequences (from largest to smallest) from each voxel (including the selected voxel) and appending a 1 in front. Therefore, the complete Morton code for voxel Vx5 is 1-100-101-101. Similarly, Figure 16A The Morton code for the voxel labeled 1900 is 1-100-000.

[0290] The principle is the same in a 64-bit tree, but each voxel is subdivided into an 8x8x8 array of smaller voxels, instead of the 4x4x4 array used in an octree.

[0291] The SDF representation of an object is stored in voxel tree format. There exists a constituent data structure: voxels. Each voxel contains the following data:

[0292] • A pointer to its parent voxel;

[0293] • A pointer to its first child element (all of these are stored contiguously, so only one pointer is needed);

[0294] • The distance from the center of the voxel to the nearest point on the object's surface is denoted as α;

[0295] • Indicates the weight of the confidence level for that distance;

[0296] • The mean and variance of the RGB color channels in each voxel;

[0297] • The number of measurements has been used to update the voxel data;

[0298] • Coordinates of the voxel center;

[0299] • Voxel hierarchy within a voxel tree.

[0300] The mean and variance of each voxel are determined across a batch of frames. The color channels are modeled as a normal distribution, with their mean and variance updated per frame—using Bayes' theorem.

[0301] It should be understood that this is merely one example of a possible voxel data structure. For example, array subscripts can be used to reference other voxels instead of pointers.

[0302] Given a 3D point, the distance and color of the SDF can be estimated by finding a leaf voxel in the octree containing that point and reading the distance and color of that voxel. Alternatively, the distance and color can be found by interpolation between two or more adjacent voxels.

[0303] 4.2.4 SDF Modeling Algorithm

[0304] Perform the following processing on all frames within a batch of frames. Generate a color 3D point cloud using color images and parallax (application). Figure 6 The principle is that each point in the point cloud is associated with the color component of its corresponding image pixel, which is then used to estimate camera pose and refine the SDF structure. The SDF data is updated using the color image, parallax, and surface normals. The overall process refines the SDF structure across all frames, then updates the SDF data across all frames, and subsequently estimates the camera pose across all frames. It should be noted that these processes can be performed in a different order than described above.

[0305] 4.2.5 Estimating Camera Pose

[0306] This section describes how the refined camera pose is computed for each object in each frame once the object's SDF has been calculated. This is a form of pose refinement (achieving a similar effect to the ICP fitting described above, but specific to SDF in some respects). In this particular case, refinement is performed in the object reference frame for each object in each frame to compute the refined camera pose for that object and that frame.

[0307] Given a color image and parallax, the camera pose for each object is estimated using the corresponding color 3D point cloud. This is the camera pose relative to the object, i.e., the camera pose in the object's frame of reference. This is done for each object in a frame.

[0308] For each colored 3D point p, its RGB components are denoted as pr, pg, and pb, respectively. The average (mean) color stored in the SDF at point p can be read as the color value stored in the voxel containing (intersecting) that point; the RGB components of this color are denoted as r(p), g(p), and b(p), respectively. The photometric error term is calculated as the sum of color differences taken at all object points within the relevant frame (the set of object points is denoted as P below, which is a subset of points within the point cloud belonging to the relevant objects):

[0309]

[0310] This is an example of photometric matching error, which penalizes the color difference between an object point and its spatially corresponding voxel in the SDF.

[0311] Here, t represents the initial 6D camera pose (pre-refinement), and R represents the SE(3) transformation of t; R and t will determine the position p of the given point with respect to the voxel mesh, thus determining the values ​​of r(p), g(p) and b(p).

[0312] Similarly, the distance stored in the SDF can be read at point p, denoted as d(p). The geometric error term is calculated as the sum of directed distances taken at all points within the relevant frame:

[0313]

[0314] This definition utilizes the definition of SDF—for a point on an object surface, the directed distance should be zero; therefore, for each voxel intersecting with an object point, a penalty is imposed on the directed distance other than zero. This is an example of geometric matching error, which penalizes the offset of the object point from the object surface encoded in the SDF (as its zero set). Furthermore, R,t will determine the position of each point p relative to the voxel grid, and thus determine the value of d(p).

[0315] It should be understood that the above are merely examples of possible error functions that penalize color difference and deviation from the object surface, respectively. Other forms of error may be used. For example, in an alternative implementation,

[0316]

[0317] Where L δ This represents the Huber norm.

[0318] Perform the above steps for each object and each frame to calculate the object-by-object, frame-by-frame camera pose.

[0319] Although the above optimization is performed in the object reference frame to find the basic optimal SE(3) transformation of the camera pose and the object point P, optimization can also be performed in the camera reference frame to determine the basic optimal SE(3) transformation of the SDF voxel mesh (i.e., pose refinement can be performed in the camera reference frame instead).

[0320] The camera pose is estimated by finding the SE(3) transform R of the camera pose t and the object point P (represented in the camera reference frame), which essentially minimizes the overall matching error function, defined in this example as a weighted average of the photometric and geometric error terms:

[0321]

[0322] In other words, by translating and / or rotating the 6D camera pose, and consequently translating and / or rotating the target point P relative to the SDF, the overall matching error E(P) is minimized. In the above equation, W P W G Let P be the weight of each object, and |P| be the number of object points (the size of P).

[0323] The last term is the regularization term; |δR, δt| LieThis represents the changes in the six Lie space pose parameters from the original transformation to the current transformation. The rotation and translation parameters have different weights. This term represents the Tikhonov regularization (AKA Ridge Regression) damping factor.

[0324] For example, the estimation of this SE(3) transform can be achieved using the forward synthesis algorithm, with the Levenberg-Marquardt (LM) approximation applied to the gradient descent step and the Gauss-Newton approximation applied to the Hessian. However, given the above error definition, other suitable optimization methods are clearly apparent.

[0325] 4.2.6 Modify the SDF structure

[0326] As mentioned above, modifying the voxel tree structure of the SDF can increase or decrease the resolution of different regions.

[0327] As the algorithm moves between frames, the resolution increases around newly observed surface regions. It iterates over each point in the 3D point cloud, locating a leaf voxel in the octree containing that point and subdividing it into multiple smaller voxels (e.g., 8 or 64, depending on the form of the voxel tree). This ensures that, over multiple frames, voxels closer to the surface become small enough to record the surface geometry in detail, while voxels farther from the surface remain relatively large.

[0328] 4.2.7 Update SDF data

[0329] It should be noted that the symbols α and β, which are defined as follows, are used below. i They and Figure 6 The angles used are different α j ,β i It is important to distinguish this from the substitution symbol A. j B i It can be used to replace the latter to represent Figure 6 From the perspective of the middle. Furthermore, although in the above text, d ij D is used to represent depth in units of distance. ij This is used to express depth in units of parallax (proportional to the reciprocal of distance), but d will be used instead in the following text. i Representing parallax, in the accompanying Appendix B, D i This represents the normal distribution that characterizes disparity error.

[0330] Now refer to Figure 17 Describes a method for updating the voxel data of an object's SDF.

[0331] Figure 17 It is a geometric representation of some principles of the SDF update method, and uses the above Figure 15B As a context.

[0332] Reference Figure 17 Consider a single point within a single frame, where the subscript i indicates a specific frame. In this context, a point is a point p of an SDF voxel, such as its center point or other reference points of the voxel.

[0333] p i This represents the coordinates of point p in the camera's reference frame. Figure 17 In the middle, the coordinate p i This is shown as a vector extending from the 3D position of camera 1500. After projection, p i Returning to image plane 1502 (i.e., through ray tracing between point p and the camera), point p is associated with the image pixel where the traced ray intersects image plane 1502. This pixel has a corresponding disparity value d. i and surface normal Where the subscript 'i' represents the relevant frame, the subscript representing the pixel itself has been removed for simplicity. Figure 15B (j and k in the text).

[0334] It should be noted that point p is not necessarily located on the object surface – as mentioned above, SDF is defined on a 3D spatial volume, and the object surface is defined by a zero subset of the SDF (i.e., points where the SDF is zero). Point p has a disparity k in frame i. i This is to measure the parallax for point p (which can be determined by transforming the known depth of point p in units of distance, i.e., its distance from the camera |p). i |, based on the known properties of the camera, use the relationship between distance and parallax listed above). Because p i It may not be on the surface of the object, therefore its calculated disparity may not be the same as the measured disparity d of the corresponding pixel. i Matching; in any case, the calculated disparity (i.e., the directed distance is zero) of points on the surface of the modeling object may not match the measured disparity, because there may be disparity measurement errors.

[0335] exist Figure 17 In the diagram, x represents the direction p. i x is the distance from point p to the object's surface. In other words, x is the distance from p to point q on the object's surface, corresponding to the same number of image pixels (via ray tracing). It should be noted that this differs from the directed distance at point p in SDF because x is not assumed to be the minimum (shortest) distance to the object's surface. The directed distance is defined as the shortest distance to the object's surface, denoted as α.

[0336] Assuming the surface of a 3D object is a local plane at q, the object plane 1506 at point q is entirely composed of q and the surface normal calculated for the corresponding pixel. Let's define it. The directed distance α is also the minimum distance between point p and the object plane 1506, which is the distance between these two points on the surface normal. Distance in a direction.

[0337] To compute the directed distance α for a given voxel, perform maximum likelihood estimation (MLE) across all frames in a batch F as follows (where the sum is for all frames in a batch F):

[0338]

[0339] Summation is an example of the aggregation disparity error function (F-aggregation across multiple frames), where f i Let be a known function of an unknown directed distance α, defined as follows:

[0340]

[0341]

[0342] Where · represents the vector dot product.

[0343] The goal here is to find the directed distance α between the relevant voxels (defined by point p) such that the aggregation disparity error is substantially minimized.

[0344] Slight change Figure 17 From this perspective, α is treated as a variable, and the corresponding object point q can be considered as any point on the tracking ray that is a distance x from p (p intersects with the camera position 1500), where x is defined by the variable α. Reference numeral 1700 indicates the pixel corresponding to point p (via ray tracing), and reference numeral 1702 defines the actual point in the 3D point cloud, determined by the measured parallax d of pixel 1700. i To define it. The goal of MLE is to find the value of α such that the corresponding object point q (where q is determined by α, the tracing ray passing through p, and the surface normal at pixel 1700) is such that... (Definition) Across all frames, F should be as close as possible to the actual object point 1702.

[0345] The goal is to find a value of α such that the object point q is as close as possible to the disparity d across all frames F. i The points in the defined 3D point cloud, or similarly, the points whose calculated disparity f is maximized across all frames. i (α) and measurement disparity d i Matching.

[0346] Although the above formula is for MLE in parallax space, it should be understood that the same principle just described can be applied to express MLE in distance space, for example, based on a depth map where depth is expressed in units of distance.

[0347] function f i Based on Figure 17 The geometric structure is derived from it. For further explanation, Appendix A at the end of this specification lists f. iThe geometric derivation is clear. Although Appendix A assumes a locally flat object surface, the derived geometric principles can be extended to adapt to different assumptions about the object surface (i.e., applying the same principles under conditions where a flat surface is not assumed).

[0348] With respect to the reference point p of the pixel, perform MLE for each voxel in the SDF to calculate the directed distance α of that voxel.

[0349] In this parallax space formula, f i (α) Calculate the disparity of object point q based on the directed distance α from point p. In other words, given a specific directed distance α, assuming the object surface of the local plane is 1506, the function f i Return the disparity of the corresponding object point q. The optimization attempt seeks the directed distance value α. * This makes it consistent with the measured disparity d of the corresponding object points across all frames F. i Matching.

[0350] Of course, if each disparity map is error-free, each object point q will perfectly coincide with its corresponding point in the 3D point cloud in each frame, and the voxel at q can have a directed distance of exactly zero. However, aggregation across multiple frames F allows for disparity measurement errors, in which case the zero set of the SDF will not perfectly coincide with the 3D object points in the point cloud in each frame (or any frame). The way maximum likelihood estimation is defined has a built-in tolerance for disparity measurement errors, which allows for the determination of the substantially optimal object surface (as the zero set of the SDF) even in the presence of disparity measurement errors. In this sense, the disparity f i (α) can be called the "true" parallax corresponding to pixel 1700, and the parallax d is measured. i This may deviate from the original value due to measurement errors. As listed in Appendix B, the above MLE process measures the parallax d. i As derived from the mean f i A sample of Gaussian disparity distributions with (α) and (unknown but constant) covariance.

[0351] To make optimization easier to handle, f i It can be approximated as a Taylor series:

[0352]

[0353] in

[0354]

[0355] as well as

[0356]

[0357] Where λ is an adjustable parameter. As described in Appendix B, theoretically, the value of λ that approximates 1... More suitable when the overall disparity map error level is relatively low (e.g., the zero set of the SDF is well aligned with the 3D object points of the point cloud in each frame), while values ​​approaching 0 can provide a larger tolerance for measurement error. For λ≈0, regarding Perform a Taylor expansion, which is the disparity of point p determined by the system geometry, rather than the measured disparity d of the corresponding object point q. i -Therefore this is related to d i The measurement error is independent and applicable to points close to the surface, which are of most interest because they capture fine-grained surface structures. Experimental results demonstrate good performance across all voxels for λ = 0, but implementation is not limited to this.

[0358] For a more detailed discussion, Appendix B lists the derivations of maximum likelihood estimation and f i approximate.

[0359] 5. Automatic / Semi-automatic 2D annotation data generation:

[0360] As mentioned above, in addition to quickly generating 3D annotation data and reducing manpower, the annotator's work can also be used to generate high-quality 2D annotation data with little or no additional manual input.

[0361] Figure 12A A flowchart and graphical illustration of a method for automatically generating 2D annotation data are shown. This method is implemented by a 2D annotation generator 312.

[0362] For ease of reference, the first frame is depicted in the upper left corner. The first vehicle, the object to be annotated in this example, is marked with reference numeral 1021. The 3D model of this vehicle is determined by isolating a subset of the point cloud 400 within this bounding box, as described above, in step 1002, projecting this subset of the point cloud onto the image plane of the relevant frame. The projection of the subset of the point cloud (i.e., the 3D model of the relevant vehicle) is indicated by reference numeral 1204 in the lower left image. It can be seen that the projection of the point cloud 1004 corresponds to the expected object 1000.

[0363] In step 1206, 2D annotation data for annotating image 702 is automatically generated based on the projection 1204 of the 3D model onto the image plane.

[0364] 2D annotation data can take the form of a 2D segmentation mask 1208 (top right), which substantially matches the object region within the image plane (i.e., it at least approximately depicts the object in the image plane, tracing one or more boundaries of the object). Such annotation data can be used to train a segmentation component to perform instance segmentation, i.e., pixel-level classification of an image, where each pixel of the input image is classified individually. In this example, the annotated object 1021 belongs to a specific object class (such as "car" or "vehicle"), and image 702 combined with segmentation mask 1208 can be used to train the segmentation component to, for example, label image pixels as "car / non-car" or "vehicle / non-vehicle," depending on whether these pixels are located within the region of segmentation mask 1208.

[0365] The projection 1204 of the 3D object model is a point-based projection, which may be too sparse to be directly used as a usable segmentation mask (but this problem can be mitigated by using an aggregated object model). To generate the segmentation mask 1208, a predictive model such as a Conditional Random Field (CRF) can be applied to the projection points to fill and smooth the projection 1204, thereby converting it into a usable segmentation mask that accurately defines the object region within the image plane. In this context, the projection 1204 of the 3D model is used as a sparse prior from which the predictive model is inferred to predict the complete segmentation mask of the object. Optionally, the parameters of the predictive model can be adjusted via UI 320 to achieve the desired result. More generally, the annotator can adjust the parameters of the CRF and / or superpixel segmentation parameters, etc., to achieve manual correction. This can be a post-processing step applied after the annotated data is generated.

[0366] For another example, 2D annotation data can take the form of a tightly fitted 2D bounding box 1210 (lower left). This is generated by fitting the 2D bounding box (a rectangle in the image plane) to the projection 1204 of the 3D model. This can then be combined with the image 702 itself to train the 2D bounding box detector. Once trained, such a component can automatically detect and locate structures within an image by automatically generating 2D bounding boxes for the image received during inference.

[0367] An alternative approach is to simply project the 3D bounding box onto the image plane and then fit the 2D bounding box onto the projection of the 3D bounding box. However, this generally does not result in a tight 2D bounding box: such as Figure 12A As shown in the upper right image, the shape of the 3D bounding box 800 is different from the shape of the vehicle, so the edges of the projected 3D bounding box generally do not coincide with the edges of the vehicle appearing in the 2D image plane.

[0368] Once generated, annotators can choose to fine-tune the 2D annotation data as needed, such as... Figure 3 The input from UI 320 to 2D annotation generator 312 is shown in the figure.

[0369] As mentioned above, a 3D object model can take the form of a 3D surface model fitted to relevant extraction points. In this case, the 3D surface model is projected onto the image plane to create 2D annotation data. Therefore, the projection is a projection of a continuous 3D surface, which can provide higher quality 2D annotation data compared to the projection of a discrete (potentially sparse) set of 3D structural points.

[0370] 5.1 Occlusion Objects:

[0371] 2D annotation data can be generated for occluded objects using either a single-frame or aggregated object model. Annotators can appropriately choose between these two options via UI 320.

[0372] Figure 12B Taking the second frame as an example, the first vehicle is partially obscured by the second vehicle. Figure 12B In the example, the single-frame 3D model of the first vehicle is determined solely based on the point cloud of the second frame by isolating a subset of the point cloud (not shown) within a 3D bounding box placed around the first vehicle. Therefore, by Figure 12B The reference numeral 1224 in the lower left corner indicates that the single-frame model of the first vehicle only includes the unoccluded points of the first object 1220. Therefore, when the single-frame 3D model 1224 is projected back onto the image plane and used to automatically generate 2D annotation data, the 2D annotation data will only annotate the visible portion of the occluded object 1220. The projection of the model is indicated by reference numeral 1204a pointing towards the lower left corner of this figure. This effect on the 2D annotation data is shown on the right side of the figure, illustrating the segmentation mask 1232 (upper right) generated based on the projection 1204a of the single-frame model. It can be seen that this only covers the visible area of ​​the occluded object 1232. Similarly, when the 2D bounding box 1234 (lower right) is fitted to the projection of the single-frame model, the bounding box will fit tightly to the visible portion of the occluded object 1220.

[0373] Figure 12C A second example is shown, where instead of object model 11, object model 11 propagated from the first frame is used. In this case, the occluded portion of object 1220 is "filled" with point cloud data from one or more relevant frames where the object portion is not occluded. Figure 12C As can be seen on the left, when such a model is projected back into the image frame, the occluded parts of the object are "restored". The projection of the propagated object model 1143 is labeled with reference numeral 1204b.

[0374] Therefore, when this projection 1204b is used as the basis for the segmentation mask indicated by reference numeral 1242 (upper right), it covers the entire area of ​​object 1220, including the occluded portion. This may not be ideal in practice, but it may still be preferable given the other advantages (higher density, less noise, etc.) offered by using a propagation (e.g., aggregation) model. In this case, the aggregation model can be determined for the occluded object in the same manner, using a 3D model of the occluded object to "block" the projection of the 3D model onto the image plane, and ensuring that the segmentation mask only covers the unoccluded portion of the object visible in the image plane.

[0375] Similarly, when the 2D bounding box 1244 (bottom right) is fitted to the projection of the propagating object model, the bounding box will fit the entire object, including occluded portions; depending on the location of the occluded portions, this may cause the 2D bounding box to extend beyond the visible portion of the object—for example, in… Figure 12C In the lower right corner, the left edge of bounding box 1244 extends beyond the leftmost visible part of object 1220 and covers the leftmost occluded part of object 1220.

[0376] A similar effect can be achieved by using an aggregated object model, for example, by aggregating points from the point clouds corresponding to the first and second frames of the first vehicle in the manner described above and applying the aggregated model to the second frame.

[0377] 5.2 Example Annotation Pipeline

[0378] Figure 18 An example of an annotated pipeline 1800 for movable object reconstruction and instance segmentation is shown, which incorporates several techniques from this paper.

[0379] The Annotation Pipeline 1800 accepts annotated stereo data (which can be expanded with additional 3D structural data such as LiDAR point cloud data) and outputs automatically generated instance segments and object coordinate maps. Instance segments are images where each pixel contains an index of any objects visible at that pixel. Object coordinate maps are images where each pixel contains the 3D coordinates of the corresponding point on the object's surface, given in the object's frame of reference.

[0380] This figure illustrates the input database 1802, which stores each batch of frames to be passed through pipeline 1800. Each batch of frames in the input database 1802 has been coarsely annotated with 3D bounding boxes (manual, automatic, or semi-automatic), that is, each frame is associated with the size of each annotated object in the frame and the coarse 6D object pose.

[0381] Object reconstruction component 1804 receives a batch of frames from input database 1802 and applies one or more of the object modeling techniques described above to generate a 3D model for each annotated object in the batch of frames (i.e., applying one model per object across all frames in the batch). In this example, object reconstruction component 1804 applies the techniques described in Section 4.2 above to generate an SDF model for each object. As described in Section 4.2.5, this process also refines the relative pose between each object and the camera in each frame.

[0382] Object modeling and pose refinement may or may not be iterative processes. In the simplest non-iterative case, an object model (e.g., SDF) is generated, and the pose is refined; these are the final model and pose. In an iterative process, after the object model and refined pose are computed, the refined pose is used to update the object model, and (optionally) the updated object model is used to refine the pose again. This can continue for any number of iterations, such as a predetermined number of iterations, or until certain termination criteria are met (e.g., some convergence criteria for the pose and object model).

[0383] For a given batch of frames, the output of the object reconstruction component 180 is an object model (e.g., SDF) 1805a for each object and a refined pose 1805b for each object per frame (as described above, for example, could be a camera pose in object space or an object pose in camera space).

[0384] 5.2.1 Light Projection

[0385] The recalculation component 1806 calculates a "ray cast image" for each object in each frame (i.e., one ray cast image per object per frame). In this document, the terms "ray tracing" and "ray casting" are used synonymously. (Referring to the above references...) Figure 17 A similar principle is described, calculating the ray-projected image for a given object in a given frame. Figure 17 In this context, ray casting is used as part of the method for generating SDFs for objects.

[0386] In this context, once the object reconstruction component 1804 has reconstructed the object and refined the 3D bounding box pose, it extracts the data from the SDF voxel mesh into an image based on the refined pose via ray projection. This means that the projected ray passes through each pixel and finds the location where the ray hits the object surface represented by the SDF (implicitly).

[0387] Figure 19The diagram shows a perspective view of a frame (frame i) where ray casting is used to calculate the ray-cast image from the object's SDF 1802. Although described in the context of SDF, these principles can be applied to any form of object model, such as (aggregated) point clouds, mesh-based models, etc. In the SDF context, a surface is defined by points whose SDF is at least approximately zero (the zero set). However, the following description of the SDF zero set also applies to surface points of any other form of object model.

[0388] According to Figure 15 and Figure 19 , Figure 19 The image plane of frame i is represented by reference numeral 1502. In this example, Figure 19 The image plane 1502 is defined by the refined camera pose 1805b provided by the object reconstruction component 1804 in the reference frame of object SDF 1805a.

[0389] The figure shows a point p in 3D space. j This point is essentially (exactly or approximately) located on the surface of the object modeled by SDF1805a, meaning its directed distance is essentially (exactly or approximately) zero. Point p j The light projection corresponds to the image pixel p. ij In this example, point p ij The determination is made by tracking pixels p in the camera and image plane 1502. ij The ray is traced, and the point p where the ray 1902 intersects with the object surface is determined. ij Point p on the surface of the object ij It can be approximated as a reference point (e.g., the center point) where ray 1902 intersects voxel 1900 and has the lowest directed distance, or the point can be determined by interpolation between two or more adjacent voxels near the implicitly represented object surface.

[0390] pixel p ij The pixels corresponding to the original RGBD image of frame i. Ray tracing is preferably performed to associate each pixel of the RGBD image (or at least a defined subset of RGBD pixels) with a corresponding point on the surface represented by object SDF 1805a (or more generally, the object model). The purpose of ray tracing in the pipeline at this time is to associate each such pixel of the RGBD image with available annotation data. This is in the sense of the 3D annotation data generated above (in this case, the refined pose 1805b along with object SDF 1805a) (i.e., in the sense that it is defined in image plane 1502 and associated with the underlying image).

[0391] From voxel 1900, which is struck by light on the surface, extract the mean and variance of the color components of voxel 1900, along with the values ​​at point p.i The confidence weights and 3D coordinates in the object space (object reference frame). Alternatively, this data can be interpolated similarly between adjacent pixels. This data is recorded at the corresponding pixel in the ray projection output. Point p i The depth is also recorded (z-coordinate in the camera's frame of reference) and can be recorded in units of distance or as parallax. The value associated with a pixel in this way through ray casting can be called the "ray casting value," which will be referred to by the following notation later in this section.

[0392] For the sake of simplicity, the subscripts used to indicate specific frames are omitted in the following text. Therefore, in the following sections, the symbol p... j Represents pixel j in a given frame. Figure 19 p in ij ), p represents the corresponding point in 3D space ( Figure 19 p in j Unless otherwise stated.

[0393] ·w y (j) - Confidence weights of ray projection for pixel j and object y (add bar to indicate normalized weights, see below).

[0394] ·d y (j) - The ray projection depth of pixel j and object y.

[0395] Light projection depth d y (j) may not perfectly match the corresponding pixel p j Measurement of parallax d j This is not only due to parallax measurement errors (as described above in the context of SDF updates), but also because object y may be occluded, causing the measured parallax to actually be closer to the different objects corresponding to (partially) occluded object y. It should be noted that for occluded objects, pixels may be associated with multiple ray projection depths and weights for different (partially) occluded / occluded objects.

[0396] The refined camera-to-boundary pose 1805b is used to perform ray casting for each instance of each frame in a batch.

[0397] All ray-casting object data from a single frame in a batch are combined with the original RGBD data to create a Bootstrap estimate (“unary mask”) for instance segmentation.

[0398] review Figure 18The unary mask component 1808 receives the ray-projected image and uses the ray-projected image 1807 of each frame to calculate the unary mask 1811 for that frame. These masks are then refined by the mask refinement component 1810 to create a refined mask 1811 for each frame. This two-stage masking process uses a CRF (Conditional Random Field) model.

[0399] CRF models can be constructed in various ways, such as in "Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials" by Krahenbuhl et al. (arXiv: 1210.5644v1 [cs.CV] October 20, 2012), which is cited in full hereafter referred to as the DenseCRF paper. However, the CRF model in this paper differs in some aspects and provides performance improvements in the context of this annotation tool.

[0400] However, it should be noted that the application of this model is entirely different. While the purpose of the DenseCRF paper was to provide a CRF that can be trained to perform instance segmentation on unlabeled images, the aim of this paper is to refine the ray casting results based on an object model in order to generate higher-quality instance segmentation Ground Truth. The ray casting results are used as a prior (encoded as unary terms, see below), which is completely different from the original application envisioned in the DenseCRF paper.

[0401] In the context of instance segmentation, a Conditional Random Field (CRF) on an image is a random distribution of all possible instances for each pixel, conditioned on the image. Given a cost function for this distribution, graph optimization techniques are used to find the optimal posterior distribution. As detailed below, the dense CRF used in this paper has unary and pairwise potential terms:

[0402]

[0403] The second term sums over all pixels with a one-dimensional pixel index i less than j. For this purpose, the image is viewed as a one-dimensional array—for example, defined from left to right, pixels within a given row can be assigned incrementing pixel indices; the index of the first pixel in the next row will be assigned an index that increments the index of the last pixel in the previous row.

[0404] Maximum A Posteriori (MAP) segmentation can be used to obtain an estimated instance segmentation by selecting the most likely instance at each pixel, as detailed below.

[0405] 5.2.2. Unidimensional Mask

[0406] Following the notation described in Section 2 of the DenseCRF paper, the unary mask acts as the “prior” or “bootstrap” of instance segmentation, and it is the input to the CRF refinement process implemented by the mask refinement component 1810.

[0407] In this context, a novel prior form is used, defined as follows:

[0408]

[0409] Where the function ψ u This can be called a univariate potential function. For a given frame, this function is defined by the ray-projected image 1807 calculated for that frame. Here, x i Represents a specific instance category for pixel i, i.e., an instance that appears in the frame or background (here, i is used instead of j to represent a specific pixel, not a specific frame); Let be the "relative probability product". The sum is for all considered object instances and additional background instances (for any pixel not belonging to any considered object instance). It can be seen that the denominator is normalized over the relative probability product of all object instances and the background. Therefore, the univariate potential is defined as the negative logarithm of the normalized relative probability product.

[0410] It should be noted that, in this context, the annotation uniquely identifies one or more instances of the same object class. Pixels i and j belonging to different instances (even if they are instances of the same object class) will be labeled with different instance classifications x. i ≠x j In this context, the terms "object" and "instance" are synonymous. There exists a single background instance, and all pixels that do not belong to any object instance belong to that background instance.

[0411] For each frame in a batch, a unary mask is defined based on the output of the ray-projected image 1807 and the original RGBD sequence. The unary mask is an image in which each pixel is a discrete distribution of all instances across the frame and the background; that is, each pixel has n+1 channels, where n is the number of instances in the frame, and each channel represents the probability that the pixel is either the corresponding instance or the background.

[0412] The relative probability product is defined as:

[0413]

[0414] Here, 'hat' represents the "relative" probability compared to the background; that is, the probability that a given pixel is more likely to be a given instance than the background. It should be noted that these relative probabilities are renormalized when defining the univariate potential. When x... i For non-background conditions, the relative probability product is defined as the product of six factors as defined below. Each of them, except for α (adjustment parameter), is cropped by a small constant value ∈.

[0415] 5.2.2.1 Adjusting parameters

[0416] The parameter α is adjusted to a constant, and its value is determined by searching the Ground Truth dataset. High values ​​favor movable objects, while low values ​​favor the background.

[0417] 5.2.2.2 Normalized Ray Projection Weights

[0418] For the weights of the SDF projection of rays from the object at pixel i, normalize per frame per object so that the maximum weight of the ray projection image is 1, then crop ∈ using the smallest minimum value. At locations where the ray does not intersect the surface of the reconstructed object, set the weight to this minimum value. This is a proxy for confidence in the reconstruction.

[0419] 5.2.2.3 Color difference

[0420] Color difference factor Reduce the weight of instances where the reconstructed ray projection color statistics do not match the instantaneous color observation. This is very effective for removing static scene occlusion from instance segmentation.

[0421] For each non-background instance x i The instantaneous color observed at pixel i is modeled as a random variable with a Gaussian distribution in the RGB color space by using the mean and variance of the ray projection of the reconstructed color. Color difference factor To find the (ε-clipping) probability of colors that are more extreme than those observed:

[0422]

[0423] Where R x (i) represents instance x at pixel i. i The average RGB color of the light projection; I i σ is the RGB color of the observed pixel i (i.e., its RGB value in the original RGBD image); x (i) represents instance x at pixel i. i The RGB color light projection covariance. σ x (i) Take small regularization values ​​on the diagonal to ensure non-singularity and assume diagonal covariance.

[0424] 5.2.2.4 Color difference

[0425] Reconstruction inter-occlusion factor Use other reconstructions that are occluded on the same ray to reduce the weight of the reconstruction instance.

[0426] The reconstructed ray projection depth d of all objects y appearing at pixel i.y (i) and normalized ray projection weights Object reconstruction inter-occlusion factor x i Defined as:

[0427]

[0428] in For the indicator function under condition X:

[0429]

[0430] The following should be noted:

[0431]

[0432] The average normalized weights for occlusion reconstruction.

[0433] Then, the role of the inter-reconstruction occlusion factor is to reduce the weight of the unary mask of the occluded object by using the average normalized weight of the occlusion reconstruction.

[0434] 5.2.2.5 2D Bounding Box

[0435] object x i 2D bounding box factor at pixel i Defined as:

[0436]

[0437] This has the effect of cropping a unary mask to a 2D bounding box.

[0438] 5.2.2.6 3D Bounding Box

[0439] 3D bounding box factor Increase the weight of depth points that fall within the 3D bounding box, and decrease the weight of depth points that fall outside the 3D bounding box. Its definition is as follows:

[0440]

[0441] Where k is a constant greater than 1, and object x i The "parallax uncertainty margin" is defined as the factor c(x) i The spatial region within the bounding box that expands around its center but is not within the original bounding box, wherein

[0442]

[0443] Where b is the stereo baseline, f is the camera focal length, s is the slant distance from the base center of the bounding box, and δ is an adjustment parameter representing the uncertainty in disparity estimation. This means that when transformed to "slant distance space," the bounding box is scaled with a factor equivalent to disparity. In the definition of the expansion factor, possible division by zero is allowed to give an infinitely positive result, which is interpreted as an infinite disparity uncertainty margin.

[0444] 5.2.3 CRF Refinement

[0445] The mask refinement component 1810 takes the estimated "soft" segmentation as a unary mask 1809 and performs optimization on a cost function defined on the fully connected graph of the pixels. The cost function (Gibbs energy) is defined as smoothing the input unary mask and snapping instance boundaries to any depth and color boundaries in the input RGBD data:

[0446]

[0447] Where x represents a possible segmentation mask (labeling each pixel as either one of the object classes or as the background). The goal is to find a maximum a posteriori (MAP) segmentation mask that minimizes Gibbs's algorithm.

[0448]

[0449] ψ u The first term is the unary potential (combining prior information from the ray-projected image 1807), and the second term is the pairwise potential. The DenseCRF pairwise potential is improved by adding a depth-sensitive term and introducing instance size-aware compatibility (see the DenseCRF paper for more details on the notation):

[0450]

[0451] Where p i Represents the position of pixel i, z i θ represents the instantaneous depth of pixel i. α θ β θ γ θ δ θ ∈ θ ζ For the feature scale, μ maxα μ maxγ μ maxδ For maximum compatibility.

[0452] In the above formula, z i It can be the measured depth associated with pixel i in the RGBD depth map. Alternatively, it can be the ray-projected depth of the applicable object. The depth-sensitive kernel can be defined based on depth or parallax.

[0453] As can be seen, the size-perceptual coefficient is applied to each kernel with an applicable feature scale. The size-perceptual coefficient is defined as:

[0454]

[0455] Where α is the adjustment parameter. To distribute evenly on object x k The standard deviation of pixel position within the 2D bounding box.

[0456] The above equation uses a 2D bounding box x from the input dataset. Alternatively, the 2D bounding box can be defined by a segmentation mask x under consideration, where, for example, a 2D rectangle is fitted to x and labeled as belonging to the same object x. k (That is, a subset of pixels from the same instance—to reiterate, different instances have different annotations in this context.) In this case, This depends on the segmentation mask x under consideration, and in particular on the size of the 2D bounding box defined for each object instance (ignoring the background—see below).

[0457] Given the width w and height h of the bounding box,

[0458]

[0459] For more details and the above derivatives, please refer to Appendix C at the end of this specification.

[0460] The bounding box of the background class is defined as infinitesimal so that it does not contribute to compatibility calculations. The purpose of instance size-aware compatibility is to reduce the smoothing effect of small classes (proportional to their 2D bounding box size), so that these classes are not disproportionately over-smoothed and forgotten, which is the problem faced by traditional constant compatibility.

[0461] When defining a depth kernel, care must be taken to avoid invalid depths. For example, partially occluded areas or areas with almost no texture. To ensure that invalid areas receive a small, approximately constant smoothing cost, the invalid depth is replaced with a large random shuffle value.

[0462] In the above formula, the pixel features are taken from the original color vector I. i ,I j The form. However, CRF is not limited to this. Appendix C in the feature vector f i ,f j The method uses a more general formula, which can take any form that facilitates meaningful feature comparisons between pixels.

[0463] 5.2.4 OCS Masking

[0464] To reduce the size of the output database 1814, the OCS mask component 1812 outputs a single object coordinate map instead of an object coordinate map per frame per object, where each pixel contains the object coordinates of the corresponding object given by MAP segmentation. As described above, MAP segmentation is determined by obtaining the most probable instance class for each pixel from the refined CRF mask 1811.

[0465] 5.2.5 Reference Frame Convention

[0466] The reference frame convention for the output object coordinate graph is derived from using the first half of the x-axis, the left half of the y-axis, and the upper half of the z-axis, with the origin as the base center point.

[0467] 6. Vehicle path-based interpolation

[0468] See UK patent application GB1815767.7, the entire text of which is incorporated herein by reference. That document discloses a method for inferring the paths of outgoing vehicles based on reconstructed paths of the vehicle itself.

[0469] In this context, the reconstructed vehicle path can be used to automatically determine an initial coarse estimate of the pose 1121 of the bounding box 800. Figure 11B ).

[0470] Reference Figure 13A and Figure 13B Optional additional features allow vehicle path accuracy to increase as location estimation becomes more refined.

[0471] Figure 13A The diagram shows the known poses of the camera at corresponding times t1 and t2 along the reconstructed vehicle's own path, denoted as EP (Ego Path). Based on the Ego Path EP, the object path (OP) has been inferred for the object to be annotated (the aforementioned first vehicle). Based on times t1 and t2, the corresponding poses of the object, such as P1 and P2, can be initially inferred by interpolation from the object path OP. This provides a starting point for manually or automatically registering the 3D model of the first vehicle using point clouds at times t1 and t2, respectively.

[0472] Figure 13B The diagram shows how the refined (more accurate) poses P1' and P2' are determined by aligning the 3D model 1143 with the point cloud at times t1 and t2, respectively, in the manner described above.

[0473] Additionally, such as Figure 13B As shown, now that we know the more accurate poses P1' and P2', we can determine the updated vehicle path OP' based on them.

[0474] This can be used for a variety of purposes—for example, to provide a more accurate initial pose estimate for other frames.

[0475] Information about vehicle paths can also be incorporated into the structure matching process to penalize pose changes in the 3D bounding box 800 / 3D model 1143 that deviate from the expected vehicle behavior model, i.e., causing unexpected changes in the vehicle path.

[0476] exist Figure 13C In the example scenario, the poses P1” and P2” at t1 and t2 respectively might just provide a good registration between the 3D model 1143 and the points at t1 and t2 respectively. However, these poses P1” and P2” suggest an unrealistic path for the first vehicle, denoted as OP”, which should be penalized according to the expected behavior model.

[0477] This dependence on expected behavior can be attributed to the cost function, which rewards good registration but penalizes unexpected changes in the vehicle path. An automatic alignment process, registering model 1143 with the point clouds of relevant frames, is applied to optimize the cost function. Therefore, in Figure 13C In the example, if the penalty assigned to the cost function to the path OP is too high, an alternative pose can be chosen instead.

[0478] 6.1 Iteration Path Refinement

[0479] The above principle can be applied iteratively, which is to build iteratively from an initial coarse annotation, that is, by creating an initial annotation, aligning poses, refining the motion model, and repeatedly refining more poses based on the refined motion model.

[0480] As mentioned above, rough annotations can be provided in the following ways:

[0481] 1. Model-in-the-loop in 2D or 3D (e.g., neural networks or moving object detectors);

[0482] 2. A coarse dynamic model of the bounding box (e.g., constant velocity);

[0483] 3. By providing a 2D "lasso" function that allows you to "click" on an object or around a point;

[0484] d. Minimize a cost function that takes into account 2D error, 3D error, and possible behavior.

[0485] Reference Figure 2An instance of sensing component 102 refers to any tangible embodiment of one or more underlying sensing models of sensing component 102, which can be a software or hardware instance or a combination of software and hardware. Such instances can be embodied using programmable hardware such as a general-purpose processor (e.g., a CPU, an accelerator such as a GPU, etc.) or a field-programmable gate array (FPGA) or any other form of programmable computer. Therefore, a computer program for programming the computer can take the form of program instructions that execute on a general-purpose processor, circuit description code for programming an FPGA, etc. Instances of sensing components can also be implemented using non-programmable hardware such as application-specific integrated circuits (ASICs), which may be referred to herein as a non-programmable computer. More generally, a sensing component can be embodied in one or more computers, which may be programmable or non-programmable, and programmed or otherwise configured to execute sensing component 102.

[0486] Reference Figure 3 , Figure 3 Components 302-314 are functional components of the annotated computer system 300 that can be implemented at the hardware level in various ways: although Figure 3 Not shown, the annotation computer system 300 includes one or more processors (computers) that perform the functions of the components described above. The processor may take the form of a general-purpose processor, such as a central processing unit (CPU) or an accelerator (e.g., a GPU), or it may take the form of a more specialized hardware processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). Although not shown separately, the UI 320 typically includes at least one display and at least one user input device for receiving user input to allow the annotator to interact with the annotation system 300, such as a mouse / touchpad, touchscreen, keyboard, etc. See reference. Figure 5 Image corrector 504, depth estimator 506, and depth transformation component 508 are functional components of a stereo image processing system 500 that can be implemented at the hardware level in various ways. For example, the functionality of the stereo image processing system 500 can be implemented in software, i.e., by computer code executing on one or more processors, such as a CPU, accelerator (e.g., GPU), or in hardware (e.g., FPGA architecture) and / or application-specific integrated circuits (ASICs) or using a combination of hardware and software. For instance, UK patent applications GB1807392.4 and GB1817390.6 disclose examples of FPGA-based implementations in semi-global matching, both of which are incorporated herein by reference in their entirety and are applicable in this context.

[0487] It should be understood that the above description is for illustrative purposes only. Other aspects and embodiments of this disclosure are described below.

[0488] 3D to 2D

[0489] A first aspect of this disclosure (Aspect A) provides a computer-implemented method for creating 2D annotation data for annotating one or more perceptual inputs, the method comprising: receiving, in an annotation computer system, at least one capture frame (first frame) including a set of 3D structural points, wherein at least a portion of a structural component is captured in the frame; calculating a reference position of the structural component within the frame; generating a 3D model of the structural component by selectively extracting the 3D structural points of the frame based on the reference position; calculating a projection of the 3D model onto an image plane; and storing the calculated projection of the 2D annotation data in a persistent computer storage device to annotate the structural component in the image plane.

[0490] Implementations of aspect A may provide one or more of the following: manual annotation, automatic annotation, and semi-automatic annotation.

[0491] In an embodiment of aspect A (Embodiment A1), 2D annotation data can be stored in association with at least one perceptual input of the frame to annotate structural components therein, and projection can be calculated based on reference positions calculated within the frame. That is, 2D annotation data can also be created for the first frame used to generate the 3D model by applying the 3D model to that same frame.

[0492] Some such embodiments may further create 3D annotation data for the first frame, wherein the 3D annotation data includes a reference position or is derived from a reference position. Preferably, 2D and 3D annotation data are created for the first frame using a common set of annotation operations.

[0493] In an alternative embodiment of aspect A (Example A2), 2D annotation data may be stored in association with at least one perceptual input of a second frame to annotate structural components in at least one perceptual input of the second frame, wherein at least a portion of the structural components is captured in the second frame. That is, a 3D model can be generated from a first frame (or a combination of the first and second frames in the case of an aggregated model) and applied to the second frame to create 2D annotation data for the second frame. This is an example of "model propagation" as used herein.

[0494] In the context of Example A1, the structural component may be referred to as a common structural component (shared by both frames). It should be noted that the entire description of common structural components captured in multiple frames also applies to structural components captured in one or more frames as described in Example A1, unless the context requires otherwise.

[0495] In the general context of aspect A, the first frame used to generate the 3D model can be called the "reference frame," and the term "target frame" can be used to refer to the frame for which annotation data is created. It should be noted that in the context of embodiment A1, the first frame is both the target frame and the reference frame. In the context of embodiment A2, the second frame is the target frame.

[0496] In an embodiment of aspect A, the 3D model can also be used to create 3D annotation data to annotate structural components in 3D space.

[0497] For example, 3D annotation data can be created to annotate at least one structural component in a second frame of perceptual input, capturing at least a portion of the structural component. That is, a 3D model can be generated from a first frame and applied to a second frame to create 3D annotation data for the second frame.

[0498] 2D or 3D annotation data can be created to annotate at least one perceptual input of a second frame (for which 2D and / or 3D annotation data is generated), by calculating the alignment model position of the 3D model within the second frame (see below).

[0499] Alternatively, based on the aligned model position determined within the target frame, 2D annotation data is created for the target frame by projecting the 3D model generated from the reference frame onto the image plane associated with the target frame. This means that the projection derived from selectively extracted points of the reference frame is used to create 2D annotation data for the target frame.

[0500] Alternatively, creating 2D annotation data for a target frame can be achieved by generating a second 3D model using aligned model positions (e.g., determined using a 3D model generated from a reference frame), selectively extracting 3D structural points of the target frame based on the aligned model positions, and then projecting the second 3D model (e.g., generated from the target frame) onto an image plane associated with the target frame. In this case, the 2D annotation data includes or is derived from a projection of the second 3D model, which is generated from the target frame but located using a 3D model generated from the reference frame.

[0501] Alternatively, the second 3D model could be an aggregated 3D model generated by aggregating 3D structural points selectively extracted from the target frame and the reference frame.

[0502] Selectively extracted 3D structural points can be selectively extracted from the frames used to generate the 3D model based on reference locations and the dimensions of one or more boundary objects.

[0503] One or more boundary objects may have one of the following dimensions:

[0504] (i) Determined manually based on one or more size inputs received at the user interface;

[0505] (ii) Automatically determined by applying the sensing component to the frame;

[0506] (iii) Semi-automatic determination by applying a sensing component to the frame and further determining based on one or more size inputs; and

[0507] (iv) Assumption.

[0508] Selectively extracted 3D structural points can be a subset of points within a 3D volume defined by a reference location and the dimensions of one or more boundary objects.

[0509] The aforementioned 3D annotation data may also include one or more boundary object dimensions used to generate a 3D model or its transformations (thus defining a 3D bounding box for the applicable perceptual input).

[0510] The second model mentioned above can be generated from the target frame based on the alignment model position and the same size of one or more boundary objects (for annotating rigid common structural components) or its transformation (for annotating non-rigid common structural components).

[0511] Model propagation

[0512] The second and third aspects of this disclosure (Aspect B and Aspect C, respectively) both provide a computer implementation method for creating one or more annotated perceptual inputs, the method comprising: in an annotation computer system, receiving a plurality of capture frames, each frame including a set of 3D structural points, wherein at least a portion of a common structural component is captured; calculating reference positions within a reference frame in the frame; generating a 3D model of the common structural component by selectively extracting 3D structural points of the reference frame based on the reference positions within the frame; determining an alignment model position of the 3D model within a target frame; and storing annotation data of the alignment model position in association with at least one perceptual input of the target frame in a computer memory for annotating the common structural component therein.

[0513] According to aspect B, the determination of the alignment model position is based on:

[0514] (i) Render a 3D model to manually align the 3D model with common structural components in the target frame, based on one or more manual alignment inputs received in the user interface with respect to the target frame.

[0515] According to aspect C, the determination of the alignment model position is based on:

[0516] (ii) Automatic alignment of the 3D model with common structural components in the target frame.

[0517] In some embodiments, automatic alignment can match features (2D or 3D) of a 3D model with features (2D or 3D) of a common structural component. However, the subject matter of aspect C is not limited thereto, and automatic alignment is feasible (see further examples below).

[0518] Example annotation data

[0519] The term "annotation data with alignment model position" refers to annotation data that includes alignment model position or is otherwise derived from alignment model position.

[0520] For example, the annotation data for the alignment model position may include positional data of the alignment model position used to annotate the position of at least one common structural component in the perceptual input of the target frame. This positional data is derived "directly" from the alignment model position (subject to any geometric transformations to a suitable frame of reference as needed), meaning that once the alignment model position is determined using the 3D model, it no longer plays a role in creating such annotation data.

[0521] Location data can be, for example, 3D location data (a form of 3D annotation data) used to annotate the positions of common structural components in 3D space.

[0522] Alternatively or additionally, the annotation data for aligning the model position may include annotation data derived from the 3D model using the aligning model position (derived annotation data). That is, the 3D model can be used to determine the aligning model position, and once the aligning model position is determined, it can also be used to derive annotation data from the 3D model itself.

[0523] As another example, a 3D model generated from a reference frame (the first 3D model) can be used to determine the alignment model position in the target frame. This alignment model position can then be used to generate a second 3D model from the target frame (see above). Therefore, in this case, the annotation data for the alignment model position can include annotation data derived from the second 3D model using the alignment model position.

[0524] An example of derived annotation data is 2D annotation data derived by projecting an applicable 3D model onto an image plane based on the aligned model position. Such 2D annotation data may, for example, include 2D boundary objects fitted to the projection of the 3D model into the image plane, or include or derive from a segmentation mask of the calculated projection.

[0525] In embodiments of aspects B and C, the annotation data may be 2D annotation data, 3D annotation data, or a combination of 2D and 3D annotation data stored in association with one or more perceptual inputs of the target frame (i.e., each form of annotation data may be stored in association with the same perceptual input in each of the different perceptual inputs of the target frame).

[0526] Annotated data may include refined annotated data calculated by applying a predictive model based on a subset of 3D point clouds.

[0527] In the case of 3D annotated data, the prediction model can be applied to the 3D model itself.

[0528] In the case of 2D annotated data, the prediction model can be applied to the 3D model itself (before its projection) or to the calculated projection of the 3D model onto the image plane.

[0529] Whether the prediction model is applied to a 3D model or to the calculated projection in the case of 2D annotated data, the prediction model will produce such refined annotated data.

[0530] Refining annotation data can, for example, have the effect of providing "fill" or "smooth" annotations for structural components.

[0531] The prediction model can be a conditional random field (CRF).

[0532] Model Alignment

[0533] Implementations of aspect B can provide one or both of manual annotation (i.e., only (i)) and semi-automatic annotation (i.e., based on a combination of (i) and (ii)) by propagating the 3D model into the target frame.

[0534] Implementations of aspect C can provide one or both of automatic annotation (i.e., only (ii)) and semi-automatic annotation (i.e., based on a combination of (i) and (ii)) by propagating the 3D model to the target frame.

[0535] That is, the alignment of the model position is determined manually (based only on manual alignment input), automatically (based only on automatic alignment), or semi-automatically (based on both manual alignment input and automatic alignment) by aligning the 3D model with the (partial) common structural components captured in the second frame, depending on the situation.

[0536] Implementation A2 of aspect A can be manual, automatic or semi-automatic, i.e. based on (i), (ii) or a combination of (i) and (ii).

[0537] In either case, the 3D model can be an aggregated 3D model determined by aggregating 3D structural points selectively extracted from two or more frames.

[0538] Aggregation Model

[0539] The 3D model can be an aggregated 3D model determined by aggregating data points selectively extracted from a reference frame with data points extracted from a target reference frame, wherein automatic alignment is achieved by matching the 3D structural points of the 3D model extracted from the reference frame with common structural components in the target frame, thereby matching the aggregated 3D model with common structural components in the target frame.

[0540] This method may include selectively extracting 3D structural points from a target frame based on the alignment model position and aggregating them with points selectively extracted from a first frame to generate an aggregated 3D model.

[0541] Alternatively or additionally, the 3D model may be an aggregated 3D model determined by aggregating data points selectively extracted from a reference frame with data points extracted from at least a third frame in addition to the target frame and the reference frame.

[0542] It is understandable that aggregated models can be generated from more than two frames (and possibly more to build dense aggregated 3D models).

[0543] The method may include the step of applying noise filtering to aggregated 3D structural points to filter out noise points and generate an aggregated 3D model.

[0544] Alternatively or additionally, the aggregated 3D model includes a 3D surface model fitted to the aggregated 3D structure points (in the case of applying noise filtering, this can be fitted to filtered 3D structure points, i.e., 3D structure points from which noise has been filtered out).

[0545] Alternatively or additionally, the method may include the step of applying a prediction model based on aggregated 3D surface points. For example, applying the prediction model to aggregated 3D surface points to generate a 3D model and / or applying the prediction model to a 2D projection of the aggregated 3D model to create a segmentation mask or other 2D annotation data.

[0546] While noise filtering, predictive modeling, and / or surface fitting can be applied to both single frames and aggregated 3D models, it is particularly advantageous to apply one or more of these methods to aggregated 3D models. For noise filtering, noisy points within the aggregated structural point set are sparser than points that actually belong to common structural components, thus enabling more accurate filtering. For predictive modeling, aggregated points provide stronger priors.

[0547] More examples of aggregated model features are provided below.

[0548] Manual / Semi-automatic Alignment

[0549] It can effectively render aggregated 3D models to assist in manually aligning aggregated 3D models with common structural components in the target frame.

[0550] For example, in manual or semi-automatic alignment scenarios, when one or more manual alignment inputs are received at the user's location, the aggregated 3D model can be updated and re-rendered to align the second reference position with the common structural components in the second frame, thus correcting visual artifacts in the rendered aggregated 3D model caused by the initial misalignment of the second reference position.

[0551] These visual artifacts are caused by the misalignment of the model's position within the target frame relative to its reference position in the reference frame. For example, an annotator may see duplicate or misaligned structural elements or features in an aggregated 3D model. By adjusting the model's position until those artifacts are no longer visible, the annotator can find the correct model position within the target frame.

[0552] In this context, simply rendering the 3D aggregate model and manually aligning it with common structural components in the target frame is sufficient—it's not actually necessary to render any part of the target frame itself using the aggregate 3D model. In practice, it might be convenient to render the aggregate 3D model within the target frame so that the annotator can view the effect of the adjustments in another way. In some cases, an option to render a magnified version of the aggregate 3D model can be provided, which the annotator can then choose to use for final adjustments.

[0553] The alignment model position can be determined based on one or more manual alignment inputs, rather than using any automatic alignment.

[0554] Further disclosure regarding the aggregated 3D model is provided below.

[0555] Automatic / Semi-automatic Model Alignment

[0556] Automatic alignment can include iterative closest point.

[0557] Alternatively or concurrently, automatic alignment may use at least one of the following: color matching, 2D feature matching, and 3D feature matching.

[0558] Alternatively or concurrently, automatic alignment may include: calculating the projection of the 3D model onto a 2D image plane associated with the target frame, and adjusting the position of the model in 3D space to match the projection with the 2D features of common structural components within the 2D image plane.

[0559] For example, the model position can be adjusted to minimize reprojection error or other photometric cost functions.

[0560] For example, the target frame may include depth component data of a 3D image, and the projection is matched with 2D features of common structural components captured in the color components of the 3D image.

[0561] The alignment of the model position can be determined automatically without any manual alignment input.

[0562] Some of these implementations can still operate based on an initial rough estimate followed by fine-tuning.

[0563] That is, determining the alignment model position can be done by initially estimating the model position within the target frame and then applying automatic alignment to adjust the estimated model position.

[0564] While in semi-automatic alignment, the model position can be initially estimated as a manually defined position, represented by one or more manual position inputs received at the user interface, in fully automatic alignment, the model position is automatically initially estimated.

[0565] The model position can be initially estimated by applying a structure-aware component to the target frame.

[0566] Automatic / semi-automatic model alignment based on structural component paths

[0567] Alternatively, the frame can be a time-series frame, and the model location can be automatically initially estimated based on the (common) structural component paths within the time interval of the time-series frame.

[0568] In addition, the paths of common structural components can be updated based on the automatic alignment applied to the target frame.

[0569] The updated common structure component path can be used to calculate the position of the common structure in one of multiple frames other than the target frame.

[0570] The method may include the following steps: storing 2D or 3D annotation data for the calculated location of the frame, used to annotate at least one common structural component in the perceptual input of the frame.

[0571] Automatic alignment can be performed to optimize a defined cost function that rewards the matching of the 3D model with common structural components while penalizing unexpected behaviors of the common structural components, as defined by the expected behavior model of the common structural components.

[0572] For example, a defined cost function can penalize unexpected changes in the paths of common structural components, as defined by the expected behavior model.

[0573] This advantageously combines the behavior of known measurements or assumptions to provide more reliable cross-frame alignment.

[0574] Common structural component paths can also be used to calculate reference positions within a reference frame to generate a 3D model (before updating the paths).

[0575] Semi-automatic model alignment

[0576] Aligning the model position can be semi-automatically determined by automatically initializing the model position and then aligning the estimated model position based on one or more manual alignment inputs.

[0577] That is, it can automatically perform a rough alignment, and then be adjusted manually.

[0578] Alternatively or additionally, the alignment model position can be initially estimated by applying a structure-aware component to the target frame.

[0579] Alternatively or additionally, when the frame is a time-series frame, the model location can be initially estimated based on the path of common structural components within the time interval of the time-series frame.

[0580] Alternatively or additionally, the model position can be initially estimated based on the automatic alignment of the 3D model with common structural components in the target frame.

[0581] Alternatively, the aligned model position can be determined semi-automatically by initially estimating the model position based on one or more manual alignment inputs and then aligning the estimated model position according to an automatic alignment process.

[0582] That is, you can roughly align it manually and then adjust it automatically.

[0583] Calculate reference position

[0584] In the implementation of manual or automatic alignment, i.e. (i) or (i) and (ii), the reference position of the reference frame can be calculated based on one or more positioning inputs received at the user interface regarding the reference frame, while rendering a visual indication of the reference position within the reference frame for manual adjustment of the reference position within the reference frame.

[0585] The reference position can be calculated automatically or semi-automatically for the reference frame.

[0586] The reference position can be calculated based on one or more positioning inputs received at the user interface, while rendering visual indications of the intra-frame reference position to manually adjust the intra-frame reference position.

[0587] The reference position can be calculated automatically or semi-automatically for the reference frame.

[0588] Reference positions can be calculated automatically or semi-automatically based on the (common) structural component paths within the time interval of a time sequence frame.

[0589] The reference position can be calculated automatically or semi-automatically by applying the sensing component to the reference frame.

[0590] Alternatively or additionally, reference locations can be calculated automatically or semi-automatically based on the paths of common structural components within the time interval of a time sequence frame.

[0591] Iterative generation and propagation of aggregated 3D models

[0592] The alignment model position of an existing 3D model with a reference position as a structural component can be calculated within a reference frame, based on at least one of the following: (i) one or more manual alignment inputs received in the user interface with respect to the frame, while rendering the existing 3D model to manually align the existing 3D model with the structural component in the reference frame; and (ii) automatic alignment of the existing 3D model with the structural component in the reference frame.

[0593] An existing 3D model may have been generated from one or more other frames that capture at least a portion of the structural components.

[0594] A 3D model can be an aggregated 3D model determined by aggregating selectively extracted 3D structural points with the 3D structural points of an existing 3D model.

[0595] Automatic alignment may include: calculating the projection of an existing 3D model onto a 2D image plane associated with a reference frame, and adjusting the model's position in 3D space to match the projection with the 2D features of common structural components within the 2D image plane.

[0596] Define object size

[0597] One or more boundary object dimensions can be determined for common structural components.

[0598] One or more boundary object dimensions can be one of the following:

[0599] (i) Determined manually based on one or more size inputs received at the user interface regarding the reference frame;

[0600] (ii) Automatically determined by applying the sensing component to the reference frame;

[0601] (iii) Semi-automatic determination is achieved by applying a sensing component to a reference frame and further based on one or more size inputs received about the reference frame;

[0602] (iv) Assumption.

[0603] Structure points selectively extracted from a reference frame 3D can be selectively extracted based on reference positions calculated within the reference frame and the dimensions of one or more boundary objects in order to generate a 3D model.

[0604] Selectively extracted 3D structural points can be a subset of points within a 3D volume defined by a reference location and the dimensions of one or more boundary objects.

[0605] The annotation data for at least one perceptual input of the target frame may further include: a transformation for annotating one or more boundary object dimensions of a rigid common structural component, or a transformation for annotating one or more boundary object dimensions of a non-rigid common structural component.

[0606] One or more boundary object sizes can be determined manually or semi-automatically based on one or more size inputs received with respect to a reference frame, wherein the visual indication of the reference position takes the form of rendering 3D boundary objects at the reference position within the reference frame, and the one or more boundary object sizes are used to manually adjust those one or more boundary object sizes.

[0607] One or more boundary object dimensions can be additionally calculated based on one or more adjustment inputs received at the user interface with respect to the target frame, while the 3D boundary object is rendered at the aligned model position within the target frame. The 3D boundary object can be rendered simultaneously or subsequently at a reference position within the reference frame, where the dimensions of one or more boundary objects are adjusted according to one or more adjustment inputs received with respect to the target frame (to allow the annotator to see the effect of any adjustments made in the target frame within the context of the reference frame).

[0608] 2D / 3D annotation data

[0609] The following refers to 2D annotation data and 3D annotation data. This refers to 2D or 3D annotation data created to annotate at least one structural component in the perceptual input of a target frame, unless otherwise stated.

[0610] However, it should be noted that embodiments of any of the above aspects may additionally create and store additional annotation data for annotating at least one common structural component in the perceptual input of the reference frame. These embodiments may advantageously utilize a set of common annotation operations (manual, automatic, or semi-automatic operations) to create annotation data for the target frame and the reference frame.

[0611] Furthermore, in any of the above embodiments, both 2D and 3D annotation data can be created. In some such embodiments, one type of annotation data can be created to annotate one or more perceptual inputs of the target frame, and another type of annotation data can also be created to annotate one or more perceptual inputs of the target frame. This can be achieved by similarly utilizing a common set of annotation operations to create both types of annotation data.

[0612] For example, public comment operations can be used to create:

[0613] - 2D annotation data for the target frame and reference frame;

[0614] - 3D annotation data for the target frame and reference frame;

[0615] - 2D annotation data of the reference frame and 3D annotation data of the target frame;

[0616] - 2D and 3D annotation data for the target frame.

[0617] The above example is for illustrative purposes only and is not intended to be exhaustive.

[0618] The annotation data may include 2D annotation data and 3D annotation data of the aligned model positions stored in association with one or more perceptual inputs of the target frame, thereby enabling the aligned model positions to be used for 2D and 3D annotations of one or more perceptual inputs of the target frame.

[0619] Alternatively or additionally, additional annotation data for the reference position can be stored to annotate at least one common structural component in the perceptual input of the reference frame, thereby using the reference position calculated within the reference frame to annotate the perceptual input of the target frame and the reference frame.

[0620] Additionally, annotation data may include the same one or more boundary object dimensions used to annotate rigid common structural components, or their transformations used to annotate non-rigid structural components.

[0621] In some embodiments, the 2D annotation data may include 2D boundary objects of structural components, which are fitted onto the calculated 3D model projection in the image plane.

[0622] Alternatively or additionally, 2D annotation data may include segmentation masks of structural components.

[0623] Aggregated 3D Models (Continued)

[0624] This disclosure provides a fourth aspect (Aspect D) of a computer implementation method for modeling a common structural component, the method comprising: in a modeling computer system, receiving a plurality of capture frames, each frame including a set of 3D structural points, wherein at least a portion of the common structural component is captured; calculating a first reference position within a first frame; selectively extracting first 3D structural points of the first frame based on the first reference position calculated for the first frame; calculating a second reference position within a second frame; selectively extracting second 3D structural points of the second frame based on the second reference position calculated for the second frame; and aggregating the first 3D structural points and the second 3D structural points to generate an aggregated 3D model of the common structural component based on the first reference position and the second reference position.

[0625] In an embodiment of aspect D, the aggregated 3D model can be used to generate annotation data for annotating common structural components in training examples of a frame in a multi-frame dataset, wherein the frame is the first, second, or third frame in the multi-frame dataset.

[0626] However, it should be noted that aspect D is not limited in this respect, and aggregated 3D models can be used alternatively (or additionally) for other purposes—see below.

[0627] In some embodiments, the annotation data may be generated according to any aspect A through C or any embodiment thereof.

[0628] Annotation data may include at least one of the following: 2D annotation data and 3D annotation data derived by projecting a 3D model onto an image plane.

[0629] The frame can be a third frame, and the method can include the following steps: calculating the aligned model position of the 3D model within the third frame, where the annotation data is the annotation data for the calculated position, and the aligned model position is based on at least one of the following:

[0630] (i) Automatic alignment of the 3D model with common structural components in the third frame;

[0631] (ii) Render the 3D model to manually align the 3D model with common structural components in the third frame, based on one or more manual alignment inputs received in the user interface with respect to the third frame.

[0632] A second reference position within the second frame can be initially estimated to generate an aggregated 3D model. This method includes a step of subsequently aligning the second reference position with common structural components in the second frame, based on at least one of the following:

[0633] (i) Automatic alignment of the first 3D structural points extracted from the first frame with common structural components in the second frame to automatically align the aggregated 3D model with common structural components in the second frame;

[0634] (ii) At the user interface, for one or more manual alignment inputs received in the second frame, render the aggregated 3D model to manually align the aggregated 3D model with the common structural components in the second frame.

[0635] The aggregated 3D model can be updated based on the second reference position aligned within the second frame.

[0636] A first 3D model can be generated by selectively extracting first 3D structural points, wherein a second reference position is aligned with a common structural component in a second frame, to generate an aggregated 3D model based on at least one of the following: (i) automatic alignment of the first 3D model with the common structural component in the second frame; (ii) rendering the first 3D model while receiving one or more manual alignment inputs at the user interface with respect to the second frame to manually align the first 3D model with the common structural component in the second frame.

[0637] At least a portion of a common structural component can be captured in a third frame. The method may include aligning a third reference position with the common structural component in the third frame based on at least one of the following: (i) automatic alignment of the 3D aggregate model with the common structural component in the third frame, and (ii) one or more manual alignment inputs received at a user interface with respect to the third frame, while rendering the aggregate 3D model to manually align the aggregate 3D model with the common structural component in the third frame; selectively extracting third 3D structural points of the third frame based on the third reference position; aggregating the first 3D structural points, the second 3D structural points, and the third 3D structural points to generate a second aggregate 3D model of the common structural component based on the first reference position, the second reference position, and the third reference position.

[0638] The 3D structural point set of the third frame can be transformed to a reference frame at the third reference position to selectively extract the third 3D structural points.

[0639] The second reference position within the second frame can be initially estimated to generate the aggregated 3D model, and the aggregated 3D model can be updated based on the second frame and the second reference position aligned within the second frame.

[0640] The aggregated 3D model can be rendered via the user interface and updated and re-rendered with one or more manual alignment inputs received at the user interface regarding the second frame, so as to manually align the second reference position with the common structural components, thereby aligning the second reference position with the common structural components in the second frame. The effect is to correct visual artifacts in the rendered aggregated 3D model caused by the initial misalignment of the second reference position.

[0641] It should be noted that in this context, the aligned second reference position is equivalent to the “aligned model position” mentioned in other paragraphs of this disclosure, where the second frame serves as the target frame. All the above descriptions of the model position also apply to the second reference position in this context (e.g., including an initial estimation of the second reference position, followed by adjustment using any of the manual, automatic, or semi-automatic processes described above).

[0642] When one or more manual alignment inputs are received at the user, the aggregated 3D model can be updated and re-rendered to align the second reference position with the common structural components in the second frame. This corrects visual artifacts in the rendered aggregated 3D model caused by the initial misalignment of the second reference position.

[0643] As mentioned above, this provides annotators with a means to manually align (or adjust) the second reference position and has the aforementioned advantages.

[0644] The frame can be the second frame, and the annotation data is the annotation data aligned to the second reference position.

[0645] The annotation data may include position data of an aligned second reference position for annotating the position of common structural components in at least one training example of the target frame, such as 3D position data for annotating the position of common structural components in 3D space.

[0646] Alternatively or alternatively, for example, the annotation data may include data derived from the aggregated 3D model using an aligned second reference position, such as 2D annotation data derived by projecting the 3D model onto an image plane based on the aligned second reference position.

[0647] The first 3D structure point may be selectively extracted from the first frame used to generate the 3D model based on a first reference position and one or more boundary object dimensions. The second 3D structure point may be selectively extracted from the frame used to generate the 3D model based on a second reference position and one of the following:

[0648] (a) The same one or more boundary object dimensions used to model rigid objects;

[0649] (b) Transformation of the dimensions of one or more boundary objects used to model non-rigid objects.

[0650] One or more boundary objects may have one of the following dimensions:

[0651] (i) Determined manually based on one or more size inputs received for at least one of the first and second frames;

[0652] (ii) Automatically determined by applying the sensing component to at least one of the first and second frames;

[0653] (iii) Semi-automatic determination by applying a sensing component to at least one frame and further based on one or more size inputs received for at least one frame;

[0654] (iv) Assumption.

[0655] The first 3D structure can be a subset of points within a first 3D volume defined by a first reference position and one or more boundary object dimensions. The second 3D structure points can be a subset of points within a second 3D volume defined by a second reference position and the same one or more boundary object dimensions or their transformations.

[0656] The method may include the step of applying noise filtering to aggregated 3D structural points to filter out noise points and generate an aggregated 3D model.

[0657] The aggregated 3D model can include a 3D surface model fitted to the aggregated 3D structure points.

[0658] The method may include the step of applying a prediction model to aggregated 3D surface points to generate a 3D model.

[0659] The method may include the steps of training at least one perceptual component using annotated perceptual input, wherein the annotation data of the perceptual input provides the ground truth of the perceptual input during training.

[0660] That is, in Figure 1 In the symbol, the perceptual input is x, and the annotation data provides the Ground Truth y x .

[0661] Training data augmentation

[0662] As mentioned above, the use of aggregated 3D models is not limited to creating annotated perceptual inputs. For example, an aggregated 3D model determined according to aspect D can be used alternatively or additionally for one or more of the following:

[0663] (a) Training data augmentation;

[0664] (b) Simulation.

[0665] Training data augmentation

[0666] The aggregated 3D model can be used to enhance the data of one frame in a multi-frame set by leveraging the model data of the aggregated 3D model, thereby creating at least one augmented perception input, including the data of the frame and the model data of the 3D model, wherein the frame is the first, second, or third frame in a multi-frame set.

[0667] Model data may include at least one of the following: 2D augmented data created by projecting a 3D model onto an image plane; or 3D model data.

[0668] The method may include the steps of training at least one perceptual component using augmented perceptual input, thereby providing the perceptual component with a combination of model data and data from the frame as part of the same perceptual input during training.

[0669] That is, in Figure 1 In the symbols, frame data and model data each form a part of the same perceptual input x.

[0670] Augmented sensor input can be used for one of the following:

[0671] (a) An unsupervised training process in which no Ground Truth (i.e., in) is provided to the augmented perception input. Figure 1 The symbol does not contain y x );

[0672] (b) A supervised training process, wherein the annotation data of claim 2 or any of its dependent claims provides ground truth to the augmented perception input.

[0673] simulation

[0674] Alternatively, the aggregated 3D model can be input into a simulator for rendering in a simulated environment, wherein at least one autonomous agent is executed to autonomously navigate the simulated environment, and the behavior of the autonomous agent in response to the simulated environment is recorded in an electronic behavior log.

[0675] Autonomous agents can navigate a simulated environment using simulated instances of trained perception components applied to simulated perceptual inputs, where data from electronic behavior logs can be used to retrain and / or redesign the perception components for application to real-world perceptual inputs.

[0676] This method may include steps that incorporate retrained or redesigned perception components into a real-world autonomous robot control system for making autonomous decisions based on real-world perception inputs.

[0677] Efficient model generation

[0678] In order to generate 3D models efficiently (and thus quickly), the 3D structural point set of the reference frame can be transformed into a reference frame at the reference position in order to selectively extract the 3D structural points of the 3D model.

[0679] Please note the difference between "(3D) reference frame" and the geometrically used term "frame of reference".

[0680] For example, a 3D volume defined by a reference location and the dimensions of one or more boundary objects can be a cuboid aligned with the coordinate axes of the reference frame. This allows, for example, efficient computation of a subset of 3D structural points within that volume on a GPU.

[0681] For example, 3D structural points can be selectively extracted from the reference frame by performing a scalar comparison in the aforementioned reference frame.

[0682] For the aggregated 3D mode, the 3D structure point set of the first frame can be transformed to the reference frame of the first reference position to selectively extract the first 3D structure points; the 3D structure point set of the second frame can be transformed to the reference frame of the second reference position to selectively extract the second 3D structure points.

[0683] The first 3D volume can be aligned with the coordinate axes of the reference system at the first reference position, and the second 3D volume can be aligned with the coordinate axes of the reference system at the second reference position.

[0684] Perceptual Input—Example

[0685] At least one perceptual input may include 2D image data of the frame or the second frame or 2D image data associated with the target frame, wherein the image plane is the image plane of the image data.

[0686] The target frame may include depth component data of a 3D image, and the perceptual input image data may be color component image data of a 3D image.

[0687] The method may include the step of applying noise filtering to remove noise points from at least one of the following: extracted 3D structure points for generating a 3D model, wherein the 3D model includes or is derived from the filtered 3D structure points in the event; and the calculated projection (in the case of 2D annotation data), wherein the 2D annotation data is the 2D annotation data of the filtered projection in the event.

[0688] Noise filtering can be applied based on filtering criteria that can be manually adjusted via the user interface of the computer system.

[0689] The frame, or each frame, can be one of multiple time-series frames.

[0690] Annotated Input-Use Cases

[0691] Any of the aforementioned perception components used to facilitate automatic or semi-automatic annotation can be trained (machine learning) perception components. In this case, the trained perception component can be retrained using any of the aforementioned annotated training inputs. Using a trained perception component in this context can be referred to as "Model in the Loop."

[0692] More generally, the method may include the steps of training at least one perceptual component using the said or each perceptual input during training, wherein annotation data of the perceptual input provides the ground truth of the perceptual input during training.

[0693] For example, the perception component can be one of the following: a 2D bounding box detector, a 3D bounding box detector, an instance segmentation component, a localization estimation component, an orientation estimation component, and a distance estimation component.

[0694] For example, 3D annotated data can be used to train 3D perception components, and 2D annotated data can be used to train 2D perception components.

[0695] Example 3D frame

[0696] The frame or the 3D structural point set of each frame can be in the form of a point cloud.

[0697] A set of 3D structural points can be captured using one or more sensors with one or more sensor modes.

[0698] Each frame can correspond to a different single moment in time.

[0699] At least one frame may include 3D structure points captured at multiple times, which have been transformed to correspond to a single time point corresponding to that frame.

[0700] Each frame can be one of multiple time-series frames. For example, the target frame and the reference frame can be frames in a time-series frame that correspond to different times in the sequence.

[0701] The 3D structure point set of a frame can be a collection generated by merging at least two sets of 3D structure point sets captured by different sensors.

[0702] Example 3D model

[0703] Any of the aforementioned 3D models may include a 3D surface model fitted to selectively extracted 3D structural points. This could be a single-frame model fitted to selectively extracted 3D structural points from a single frame, or an aggregated 3D model fitted to points selectively extracted and aggregated from multiple frames.

[0704] This can be used, for example, to create 3D annotation data that includes or is based on projections of 3D surface models.

[0705] Another aspect of the subject matter of the present invention provides a computer implementation method for creating one or more annotated perceptual inputs, the method comprising: in an annotation computer system, receiving at least one capture frame comprising a set of 3D structural points, wherein at least a portion of a structural component is captured in the frame; receiving a 3D model of the structural component; determining an aligned model position of the 3D model within the frame based on automatic alignment of the 3D model with the structural component in the frame; and storing annotation data of the aligned model position in association with at least one perceptual input of the frame in a computer memory for annotating the structural component therein.

[0706] In some embodiments, a 3D model can be generated by selectively extracting 3D structural points of at least one reference frame based on reference positions calculated within the reference frame.

[0707] However, 3D models can also be CAD (computer-aided design) models of structural components or other externally generated models.

[0708] In other words, the aforementioned automatic or semi-automatic model alignment features can also be applied to externally generated models. Therefore, it can be understood that the entire description above regarding 3D models generated from at least one reference frame also applies to externally generated 3D models in this context.

[0709] More examples

[0710] To further illustrate how the various annotation features of this disclosure can be used individually or in combination, some additional exemplary use cases and workflows supported by these features are listed below. Given the teachings presented herein, it should be understood that they are by no means exhaustive.

[0711] 1. 3D to 2D Annotation: Given a complete or partial 3D model:

[0712] a. Generate tight 2D bounding boxes;

[0713] b. Generate instance segmentation masks (or priors for refining segmentation masks using CRF / annotators / other methods);

[0714] c. Use projection points from different frames to assist annotation (using the X-ray visual features mentioned above), thereby improving the consistency and accuracy of annotation.

[0715] 2. 3D models extracted from a single frame: can be efficiently generated by converting all points of the frame to be aligned with the axes of the 3D bounding box, which means that a simple scalar comparison can be used to determine whether any given points are enclosed.

[0716] 3. 3D Model Extracted Across Multiple Frames (Aggregated 3D Model): Using bounding boxes from multiple frames, extract and aggregate all enclosing point cloud points (to generate an aggregated point cloud). By applying the above transformations to each frame containing the 3D bounding box, extraction can be performed efficiently for each frame.

[0717] a. Then noise can be filtered out from the aggregated point cloud.

[0718] b. The aggregated point cloud to be generated can be smoothed using surfaces, such as SDF.

[0719] c. The advantages of accumulation points include:

[0720] i. Improved instance segmentation priors, in which a predictive model (e.g., CRF) is applied; or even the aggregated point cloud itself may be sufficient for instance segmentation (i.e., without applying a predictive model or fitting a surface model, etc.).

[0721] ii. Improved noise filtering / surface, as there will be more points than in a single frame;

[0722] iii. Aggregated models are more beneficial for augmenting training data than single-frame models because they have higher quality and a wider range of applicable viewpoints;

[0723] iv.2D bounding boxes can be drawn as containing the occluded portion of partially occluded objects.

[0724] 4. Automatic 3D annotation: using one or more of the following methods:

[0725] a. Generate a 3D model from one frame, match it with the point cloud in another frame (e.g., by iterating the nearest points), then combine them into a single model, and repeat the process for the next frame (using the combined model as a matching reference);

[0726] b. Project the 3D model onto the image and minimize photometric errors (i.e., allow the annotator to automatically “visually” align the X-rays visually);

[0727] c. Feature matching in 2D and / or 3D, and minimizing the reprojection error of the matched features;

[0728] d. Iteratively build from the initial coarse annotations; that is, determine the initial annotations, align the poses, refine the motion model, and repeat for more poses. Coarse annotations can be provided in the following ways:

[0729] i. Model-in-the-loop in 2D or 3D (e.g., neural networks or moving object detectors);

[0730] ii. Initial dynamic model (expected behavior model) of the cross-frame bounding box, such as assuming the object travels at a constant speed;

[0731] iii. "Click" on an object point (i.e., by selecting a single object point) or "2D lasso" around an object point (i.e., by selecting a set of object points in a 2D plane such as an image plane or a top view).

[0732] e. Minimize the alignment cost function that takes into account, for example, 2D errors, 3D errors, and possible (expected) behavior.

[0733] For illustrative purposes, specific embodiments of the invention have been described above, but it should be understood that they are not necessarily exhaustive. The scope of the invention is not limited to the described embodiments, but only to the appended claims.

[0734] Appendix A

[0735] Reference Figure 17 This appendix shows the function f from section 4.2.7. i How is it derived based on system geometry?

[0736] definition

[0737] Assuming a locally flat object surface 1506, the shortest distance t from this surface to point p is α = x cosθ, therefore:

[0738]

[0739] The parallax k of point p i and The parallax of point q is directly proportional to the parallax of point q. i (α) and Proportional, therefore:

[0740]

[0741] Appendix B

[0742] This appendix shows how the MLE in Section 4.2.7 is derived.

[0743] Consider a single point at position i in a single frame. Using p... i This represents the coordinates of the point in the camera's reference frame. Let p... i After being projected back onto the image plane, it can be associated with an image pixel. This pixel has a corresponding disparity value d. i and surface normal Assume the measured disparity values ​​are normally distributed with a mean of μ. i Corresponding to the true value, the standard deviation σ is constant. Therefore, for each frame, there exists a normally distributed random variable D. i Sample d i Its mean μ i The known function f as the unknown parameter α i , which is the distance from the point to the surface. The task here is to estimate the parameter α.

[0744] D i ~N(f i (α), σ).

[0745] If we assume that each of these random variables is an independent variable, then maximum likelihood estimation can be performed.

[0746]

[0747] Taking the negative logarithm of each side and discarding the constant term, we obtain the following result:

[0748]

[0749] Differentiating this expression with respect to α, and obtaining a result of 0, yields the following equation:

[0750]

[0751] If we assume the surface is planar, some simple geometric shapes with respect to f i ,f i -1 ,f i The following formula is given: Here, k i Point p i parallax, β i Representing value

[0752]

[0753]

[0754]

[0755] In this respect, we can simply put this f i Explicitly substituting the values ​​into the maximum likelihood estimate and solving the resulting equation is quite difficult. Instead, we can use its first-order Taylor series at a certain value... Approximate f i .

[0756]

[0757]

[0758] Substituting this approximation into the maximum likelihood estimation, we obtain the following equation:

[0759]

[0760] Because of its simplified form, the equation can be solved explicitly, yielding the following α formula:

[0761]

[0762] The rest is just selection. value. The closer the approximation is to the true value α, the better the linear approximation. We will examine two possible choices here.

[0763] For the first option, you can select... The following relation holds:

[0764]

[0765] This corresponds to the assumption that the parallax d is measured. i Approaching the true parallax μ i When the parallax is accurate, this should yield good results, but may be poor in response to noise. Achieving this... The values ​​are as follows:

[0766]

[0767] The α formula then transforms into the following weighted average:

[0768]

[0769]

[0770] For the second option, we can choose α to hold the following relationship:

[0771]

[0772] This corresponds to the hypothetical point p. i Approximate the surface. This should provide good results near the surface, making minimal assumptions about the accuracy of parallax. This achieves... The values ​​are as follows:

[0773]

[0774] The α formula then transforms into the following weighted average:

[0775]

[0776]

[0777] Alternatively, the first two solutions can be generalized by interpolating between them. Choose a value λ∈[0,1]; when λ=1 or λ=0, the previous solution will be recovered. (The last part, "can be selected," is incomplete and requires further context.) The following relation holds:

[0778]

[0779] To achieve this The values ​​are as follows:

[0780]

[0781] The α formula then transforms into the following weighted average:

[0782]

[0783]

[0784] For example, take As a compromise between the two extreme solutions, we can choose The following relation holds:

[0785]

[0786] To achieve this The values ​​are as follows:

[0787]

[0788] The α formula then transforms into the following weighted average:

[0789]

[0790]

[0791] Appendix C

[0792] CRF annotation compatibility derivation.

[0793] This section outlines how the DenseCRF is derived from Sections 5.2.2 and 5.2.3 above.

[0794] This attempt aims to minimize the following Gibbs energy on a fully connected pixel graph:

[0795]

[0796] Where x is the annotation set, i and j are one-dimensional pixel indices, and ψ n For the unary potential (in this case, the aggregated data from the input and reconstruction), ψ p To form a pair.

[0797] Regarding the eigenvector f i and f j And the positions p of pixels i and j i and p j In this regard, the paired potential has the form of a weighted sum of K Gaussian kernels:

[0798]

[0799] Each kernel m is composed of a symmetric positive definite precision matrix ∧ (m) Characterized by the symmetric annotation compatibility function μ (m) (x i ,x j Weighted.

[0800] Technical issues

[0801] The label x represents instance segmentation on the image. Pairwise potentials smooth the input univariate potential by aligning the boundary to the edge in the feature vector field. However, this smoothing has only a single parameter s. xy , which represents the standard deviation of the Gaussian potential. Ideally, it should correspond in some way to the standard deviation of pixels within the instance mask so that smoothness is scaled appropriately. If smoothness is not scaled appropriately, smaller instances (such as distant or occluded objects) will disappear because removing the entire instance is less costly than smoothing its boundaries. However, s xy This is a global parameter and cannot be accessed on an instance-by-instance basis.

[0802] Solution

[0803] Appropriate compatibility can be selected to reduce the smoothing effect on smaller instances. A potentially desirable approach might be to adjust the terms in the exponent. Multiply The denominator is about the instance. The standard deviation of the inner pixels. The reason this cannot be achieved is that, by multiplying the exponent by the compatibility per instance, the best result achievable is to set the compatibility as the exponent to effectively add a term to the exponent. A possible approach is to add a constant term to effectively increase the distance metric between pixels, which is inversely proportional to the variance of points in the instance mask. This ensures that as the instance mask size decreases, the effective distance between points increases, thus reducing the smoothing effect. In summary, the following options are possible:

[0804]

[0805] in To ensure compatibility between two instances is determined entirely by the smallest of the two instances, two tuning parameters are introduced: maximum compatibility μ. max And the smoothing scale α, as the standard deviation proportion. For example, set Intuitively, this corresponds to limiting the smoothing radius to half the standard deviation of the pixel position in the instance mask.

[0806] Substitute it into ψ p (x i ,x j The definition of ) is given as follows:

[0807]

[0808] Where k is the smaller of the two instance masks i and j.

[0809] Using a 2D bounding box as a proxy for the instance mask is ineffective because the Ground Truth mask is inaccessible. For a bounding box with width w and height h, this would cause the corresponding variance to be affected. Defined as the variance of a uniformly distributed rectangle defined on the 2D bounding box. It should be noted that this is equal to the expected distance between two independently selected points within the bounding box, where i > j, which is also reasonable in the case of normalization to the kernel. The formula is as follows:

[0810]

[0811]

[0812]

[0813]

Claims

1. A computer-implemented method of creating 2D annotation data for annotating a 2D image, the method comprising: receiving, at an annotation computer system, at least one captured frame comprising a set of 3D structure points associated with the 2D image, the frame capturing at least a portion of an object; computing a reference position of the object within the frame; positioning a 3D bounding object within the frame to bound the object, wherein the reference position is a position of the 3D bounding object within the frame; generating a 3D model of the object by selectively extracting 3D structure points of the frame based on the reference position, wherein the selectively extracted 3D structure points of the frame are a subset of 3D structure points extracted from within a volume of the 3D bounding object; computing a projection of the 3D model to an image plane of the 2D image; storing the computed 2D annotation data of the projection with the 2D image in a persistent computer storage device; and training, testing, and validating a 2D structure detector using the 2D image and the 2D annotation data. the 2D annotation data comprises one of:

2. The method of claim 1, wherein, a 2D bounding object of the object, the 2D bounding object fitted to the computed 3D model projection in the image plane; and a segmentation mask of the object. the 2D annotation data:

3. The method of claim 1 or 2, wherein, is stored in association with at least one perception input comprising, originating from, or otherwise corresponding to the frame, wherein the projection is computed based on the computed reference position within the frame; or is stored in association with at least one perception input comprising, originating from, or otherwise corresponding to a second frame, wherein at least a portion of the object is captured in the second frame. is stored in association with at least one perception input of a second frame, wherein the 2D annotation data is created to annotate the at least one perception input of the second frame by computing an aligned model position of the 3D model within the second frame, wherein the aligned model position is determined based on at least one of:

4. The method of claim 3, wherein, (i) one or more manual alignment inputs received at a user interface with respect to the second frame while rendering the 3D model to manually align the 3D model with the object in the second frame; and (ii) an automatic alignment of the 3D model with the object in the second frame.

5. The method of any preceding claim, comprising the step of: applying noise filtering to at least one of: the 3D model to filter out noise points from the filtered 3D structure points of this step; and extracted 3D structure points for generating a 3D model, wherein the computed projection, the 2D annotation data being 2D annotation data of the filtered projection of this step. the reference position is computed based on one or more positioning inputs received at a user interface while rendering a visual indication of the reference position within the frame to manually adjust the reference position within the frame. ​ 6. The method of any preceding claim, wherein, ​ 7. The method of any preceding claim, wherein, The selectively extracted 3D structure points are selectively extracted from frames used to generate the 3D model based on the reference position and one or more bounding object dimensions, wherein the one or more bounding object dimensions are one of: (i) manually determined based on one or more size inputs received at a user interface; (ii) automatically determined by applying a structure detector to the frames; (iii) semi-automatically determined by applying the structure detector to the frames and further based on one or more size inputs; and (iv) assumed.

8. The method of any preceding claim, wherein, The 2D images are captured by a camera.

9. The method of any preceding claim, wherein, The 3D model comprises a 3D surface model fitted to the selectively extracted structure points, and the projection comprises a projection of the 3D surface model to the image plane.

10. The method of any preceding claim, wherein, The set of 3D structure points are transformed into a frame of reference of the reference position in order to selectively extract 3D structure points of the 3D model.

11. The method of any preceding claim, wherein, The reference position is computed within the frames as an aligned model position of an existing 3D model of the object based on at least one of: (i) one or more manual alignment inputs received at a user interface with respect to the frames while rendering the existing 3D model to manually align the existing 3D model with the object in the frames; and (ii) an automatic alignment of the existing 3D model with the object in the frames. The automatic alignment comprises computing a projection of the existing 3D model or the 3D model to a 2D image plane associated with the frames or a second frame, and adjusting the model position in 3D space to match the projection to 2D features of a common object within the 2D image plane.

12. The method of claim 4 or 11, wherein, The 2D annotation data comprises an instance segmentation mask substantially determined to optimize an energy function, wherein the instance segmentation mask assigns an instance class label to each pixel in a set of pixels within the image plane, wherein the energy function has:

13. The method of any preceding claim, wherein, a unary term that aggregates a unary function over the segmentation mask, the unary function being applied to the instance class label of each pixel individually, wherein the unary function is defined by one or more raycast values determined by raycasting through the pixel and determining whether and where a ray intersects a surface of the 3D model as in the projection step; and a non-unary term that aggregates a non-unary function over the segmentation mask, the non-unary function being applied to instance class labels of at least two pixels.

14. A computer system comprising one or more computers programmed or otherwise configured to perform the steps of any of the preceding claims.

15. A transitory or non-transitory medium comprising one or more computer programs embodied on the transitory or non-transitory medium for programming a computer system to perform the steps of any of claims 1 to 13. ​

Images