A system and method for augmenting point cloud data using model injection.

Abstract

A device, system, method and medium are described for point cloud data augmentation using model injection to train machine learning models to perform point cloud segmentation and object detection. A library of surface models is generated from point cloud object instances in a point cloud frame generated by a LIDAR. The surface models can be used to inject new object instances into a target point cloud frame at any location in the target frame to generate new augmented point cloud data. The augmented point cloud data can then be used as training data to improve the accuracy of a machine learning model trained with a machine learning algorithm to perform segmentation and / or object detection tasks.

Description

[Technical Field]

[0001] This application relates, in general, to point cloud data augmentation for machine learning, and more particularly to devices, systems, methods, and media for point cloud data augmentation using model injection. [Background technology]

[0002] Light detection and ranging (LIDAR, also referred to herein as "LIDAR" or "LIDAR") sensors generate point cloud data representing a three-dimensional (3D) environment (also called a "scene") scanned by the LIDAR sensor. A single scan pass of a LIDAR sensor generates a "frame" of point cloud data (hereinafter referred to as a "point cloud frame") consisting of a set of points from which light is reflected from one or more points in space, within a period of time representing the time it takes for the LIDAR sensor to perform one scan pass. Some LIDAR sensors, such as rotational scanning LIDAR sensors, include a laser array that emits light in an arc, and the LIDAR sensor rotates around a single position to generate a point cloud frame; other LIDAR sensors, such as solid-state LIDAR sensors, include a laser array that emits light from one or more positions and integrates the reflected light detected from each position to form a point cloud frame. Each laser in a laser array is used to generate multiple points with each scan pass, and each point in the point cloud frame corresponds to an object that reflects the light emitted by the laser at a point in space in the environment. Each point is typically stored as a set of spatial coordinates (X, Y, Z) and other data indicating values ​​such as intensity (i.e., the reflectivity of the object reflecting the laser). In some implementations, the other data may be represented as an array of values. In a scanning rotation LiDAR sensor, the Z-axis of the point cloud frame is typically defined by the rotation axis of the LiDAR sensor, which is approximately perpendicular to the azimuth direction of each laser in most cases (however, some LiDAR sensors may angle some of the lasers slightly above or below a plane perpendicular to the rotation axis).

[0003] Point cloud data frames may also be generated by other scanning techniques, such as high-resolution radar or depth cameras, and theoretically, any technique using a scanning beam of energy, such as electromagnetic or acoustic energy, can be used to generate point cloud frames. An example is described herein with reference to a LiDAR sensor, but it will be understood that other sensor techniques for generating point cloud frames may be used in some embodiments.

[0004] LiDAR sensors are one of the primary sensors used in autonomous vehicles to detect the environment (i.e., the scene) surrounding the autonomous vehicle. Autonomous vehicles generally include an automated driving system (ADS) or advanced driver-assistance system (ADAS). An ADS or ADAS includes a perceptual submodule that processes point cloud frames to generate predictions that can be used by other subsystems for autonomous vehicle localization, autonomous vehicle path planning, autonomous vehicle motion planning, or autonomous vehicle trajectory generation. However, due to the sparse and unordered nature of point cloud frames, the cost of collecting and labeling point cloud frames at the point level is time-consuming and expensive. Multiple points within a point cloud frame must be clustered, segmented, or grouped (e.g., using object detection, semantic segmentation, instance segmentation, or panoramic segmentation) so that sets of points within the point cloud frame can be labeled with an object class (e.g., "pedestrian" or "motorcycle") or an instance of an object class (e.g., "pedestrian #3"), and these labels are used in machine learning to train models for predictive tasks on point cloud frames, such as object detection or various types of segmentation. This cumbersome labeling process has resulted in limited availability of labeled point cloud frames representing various road and traffic scenes, which are necessary for training high-accuracy models for predictive tasks on point cloud frames using machine learning.

[0005] Segmentation object searchExamples of such labeled point cloud datasets, including point cloud frames used to train models using machine learning for prediction tasks such as scene detection, are described in the SemanticKITTI dataset (Behley et al., "SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences," 2019 IEEE / CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019, pp. 9296-9306, doi: 10.1109 / ICCV.2019.00939), KITTI360 (J. Xie, M. Kiefel, M. Sun and A. Geiger, "Semantic Instance Annotation of Street Scenes by 3D to 2D Label Transfer," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 3688-3697). The Nuscenes-lidarseg (described by H. Caesar et al., "nuScenes: A Multimodal Dataset for Autonomous Driving," 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 11618-11628, doi: 10.1109 / CVPR42600.2020.01164) is the only available point cloud dataset containing semantic information, which may be point cloud frames labeled with semantic information for training a model on a prediction task on point cloud frames, such as segmentation or object detection.

[0006] However, these available point cloud datasets generally do not contain a sufficient number of point cloud frames containing objects from a particular object class, and the set of point cloud frames containing such objects indicates a lack of diversity in object instances ("object instances") within each such object class. Object classes that appear in limited numbers within a point cloud dataset may be referred to herein as unfavorable classes. Unfavorable classes in existing point cloud datasets are typically small, less common types of objects, such as pedestrians, bicycles, cyclists, motorcycles, motorcyclists, trucks, and other types of vehicles.

[0007] Unfavorable classes can cause one or both of two problems. The first problem arises from a lack of environmental or contextual diversity. When object instances of an unfavorable class appear in only a few point cloud frames within a point cloud dataset, and these instances appear in environments different from the point cloud frames in the dataset, a model (e.g., a deep neural network model) trained on a prediction task for point cloud frames (such as object detection or various types of segmentation) may not learn to recognize object instances of the unfavorable class (i.e., point clouds corresponding to objects of the unfavorable class). For example, if the point cloud frames in a point cloud dataset only contain object instances of "motorcyclists" (i.e., the unfavorable class "motorcyclists") in point cloud frames corresponding to parking lots, this model may not be able to identify motorcyclists in a road environment. The second problem arises from a lack of diversity in object instances. When object instances of an unfavorable class appear in very small numbers within a point cloud dataset, diversity of object instances itself cannot be guaranteed. For example, if the point cloud frames in a point cloud dataset only contain object instances of motorcyclists riding sports bikes (i.e., the disadvantaged class "motorcyclist"), this model may not be able to identify motorcyclists riding scooters.

[0008] Traditionally, the problem of using sparse point cloud datasets with unfavorable classes to train models for prediction tasks on point cloud frames (such as segmentation and object detection) has been addressed through data augmentation. Data augmentation can be seen as a process to achieve higher model accuracy (i.e., a model that produces better predictions) by generating new training samples (e.g., new semantically labeled point cloud frames) from an existing labeled point cloud dataset using any technique that can help improve the training of models for prediction tasks on point cloud frames. The environmental diversity problem identified above is typically addressed by a method that involves extracting objects from one point cloud frame and injecting the extracted objects into other point cloud frames to generate additional point cloud frames containing object instances of unfavorable classes, which can be used to further train the model. The point cloud frames into which object instances are injected can represent different environments, thus assisting the model in learning to recognize object instances of unfavorable classes in other environments. Examples of such technologies include Yan Yan, Yuxing Mao, Bo Li, "SECOND: Sparsely Embedded Convolutional Detection", Sensors 2018, 18(10), 3337; https: / / doi.org / 10.3390 / s18103337; Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, Oscar Beijbom, "PointPillars: Fast Encoders for Object Detection from Point Clouds", https: / / arxiv.org / abs / 1812.05784; and Yin Zhou, Oncel Tuzel, "VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection", https: / / arxiv.org / abs / 1711.06396.These existing approaches to data augmentation typically proceed as follows: First, a database of object instances is generated by extracting clusters (i.e., point clouds of objects) from point cloud frames annotated with bounding boxes around the object instances. Second, object instances are randomly selected from the database and injected into similar locations in other point cloud frames. Finally, collision testing is implemented to avoid object location conflicts (e.g., spatial overlap with another object in the target point cloud frame into which the object instance is injected). Object instances extracted from point cloud frames are typically half-side due to the directivity of the LIDAR sensor. Therefore, the original position and orientation of the object instances cannot be significantly altered during injection to avoid this side of the object instance without multiple points defining the surface facing the LIDAR sensor. These existing approaches can result in an increased number of unfavorable class object instances per point cloud frame, simulating object instances existing in different environments.

[0009] However, these existing approaches to addressing environmental diversity typically have three limitations. First, they cannot generate reasonable scan lines on the surface of injected object instances, nor can they generate realistic object shadows (i.e., occlusion of other objects in the scene located behind the injected object instance). Second, the position and orientation of the injected object instance are always identical or nearly identical within two point cloud frames (i.e., the original point cloud frame in which the object instance appears, and the target point cloud frame into which the object instance is injected). Third, these existing approaches ignore the context in which object instances appear in different environments. For example, a person typically appears on a sidewalk, but this context is not considered in existing approaches to addressing environmental diversity. Furthermore, since object instances must typically appear in the same orientation and position relative to the LIDAR sensor, these approaches do not allow object instances to be injected into the target point cloud frame at a position or orientation that would make the most sense in context; for example, if the entire target point cloud frame consists of sidewalks and buildings except for a small parking lot extending only 20 meters from the LIDAR sensor, and the injected object instance is a track located 50 meters from the LIDAR sensor in the original point cloud frame, then it is not possible to inject the object instance into the target point cloud frame at a position that would make sense in context.

[0010] The object instance diversity problem has typically been addressed using two different approaches. The first approach involves positioning a computer-aided design (CAD) model of an object at a spatial location within a point cloud frame, and then generating multiple points representing each object using the object's CAD model and LIDAR parameters of the target point cloud frame (e.g., the mounting orientation of the LIDAR sensor and the pitch angle of each ray emitted by the LIDAR sensor's laser). Examples of the first approach include Jin Fang, Feilong Yan, Tongtong Zhao, Feihu Zhang, "Simulating LIDAR Point Cloud for Autonomous Driving using Real-world Scenes and Traffic Flows"; and Sivabalan Manivasagam, Shenlong Wang, Kelvin Wong, Wenyuan Zeng, Mikita Sazanovich, Shuhan Tan, Bin Yang, Wei-Chiu Ma, Raquel Urtasun, "LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World".

[0011] An example of the first approach could allow for the rotation and transformation of a CAD model of an object without any limitations, and the generation of reasonable scan lines and shadows. In contrast to the object instance injection approach described above for dealing with environmental diversity, the context can be considered during injection without positional and orientation constraints. However, CAD model-based approaches typically have three limitations. Firstly, CAD models are typically obtained from LiDAR simulators such as GTAV (described in GTAV: A LiDAR Point Cloud Generator: from a Virtual World to Autonomous Driving, arXiv:1804.00103) or CARLA (described in CARLA: An Open Urban Driving Simulator, arXiv:1711.03938), or purchased from 3D model websites. The variety of CAD models of objects available from these sources is typically very limited. Secondly, the style of available CAD models of objects may differ from those of the real-world objects they likely correspond to. For example, if a CAD model of a European truck is injected into a point cloud frame corresponding to a North American road environment, trucks of that style may appear very realistic, even though they do not actually exist in the environment in which the object's CAD model is trained to recognize and navigate. Thirdly, an object's CAD model cannot provide accurate intensity values ​​for the injected object instance. The intensity of a point on an object's surface is a function of the angle between the ray emitted by the laser and the surface reflecting the ray, as well as the reflectance of the material reflecting the ray. However, the most readily available CAD models of objects do not provide any information regarding the reflectance of the surface material of the model.

[0012] A second approach to addressing the object instance diversity problem is Waymo TM This is outlined at https: / / blog.waymo.com / 2020 / 04 / using-automated-data-augmentation-to.html. Instead of injecting new object instances into a point cloud frame using a CAD model of the object, a dense and complete point cloud scan of the object is used to inject new object instances into the target point cloud frame. The advantages of a dense and complete point cloud scan of an object are similar to those of a CAD model of the object, and these CAD models can be rotated and transformed without any limitations during injection, and can also generate reasonable scan lines and shadows. The diversity of point cloud scans injected with objects can be attributed to eight different data augmentation methods: ground truth augmentation (i.e., adding two or more object instances of the same object together), random flip (i.e., flipping object instances horizontally, for example), world scaling (i.e., scaling the size of object instances), global transformation noise (i.e., transforming object instances to different locations), frustum dropout (i.e., removing areas of the visible surface of an object instance to simulate partial occlusion, for example), frustum noise (i.e., randomly disrupting the positions of multiple points in an object instance to simulate slightly different surfaces, for example), random rotation (i.e., rotating an object instance around an axis), and random drop point (i.e., removing a randomly selected subset of multiple points in an object instance to simulate scanning at a lower resolution, for example).

[0013] However, injecting new object instances into a target point cloud frame using dense point cloud object scanning also has several limitations. First, a dense and complete point cloud scan of an object is required to implement this approach. In contrast, object instances in point cloud frames generated by LIDAR are typically sparse and half-side. Therefore, a large dataset of carefully, densely, and completely scanned objects would need to be assembled before this approach can be implemented. Second, object symmetry is often used to generate a complete point cloud scan of an object based on half-side scanning. However, many small objects encountered in road or other environments, such as pedestrians, motorcyclists, and cyclists, are not symmetric. Therefore, relying on symmetry and simply extrapolating from existing point cloud datasets that contain point cloud frames with dense half-scans of objects does not address the need to assemble a large database of object point cloud scans. Third, since dense point cloud scans of objects are usually imaged from different locations to capture a complete point cloud scan of the object, the intensity of the dense point cloud scan of the object may not be accurate. For example, to generate a complete and dense scan of an object, a 3D scanner may be rotated around the object in at least one direction; this results in scanning the same point from multiple directions, generating competing intensity measurements for that point and generating intensity measurements for multiple different points that do not coincide with each other, for different scanning directions.

[0014] Therefore, there is a need for data augmentation techniques for point cloud datasets that overcome one or more of the limitations of the existing approaches described above. [Overview of the project]

[0015] This disclosure describes devices, systems, methods, and media for augmenting point cloud data using model injection, aimed at training machine learning models for predictive tasks on point cloud frames, such as segmentation or object detection. The exemplary devices, systems, methods, and media described herein can generate a library of surface models that can be used to generate a new augmented point cloud frame by injecting new point cloud object instances into the target point cloud frame at arbitrary locations within the target point cloud frame. The augmented point cloud frame can then be used as training data to improve the accuracy of a machine learning model trained on a predictive task on the point cloud frame (i.e., a machine learning model trained using a machine learning algorithm and the original point cloud dataset).

[0016] In this disclosure, the term “LIDAR” (also known as “LiDAR” or “Lidar”) refers to a sensing technique in which a sensor emits a laser beam and collects location and potentially other features from objects that reflect light in the surrounding environment.

[0017] In this disclosure, the terms “point cloud object instance,” or simply “object instance” or “instance,” refer to a point cloud of a single, definable object, such as a car, a house, or a pedestrian, which can be defined as a single object. For example, typically a road cannot be an object instance; instead, a road may be defined within a point cloud frame as defining a scene type or region of the frame.

[0018] In this disclosure, the term “injection” refers to the process of adding point cloud object instances to a point cloud frame. The term “frame” refers to a point cloud frame unless otherwise specified; the “original” frame is the frame containing labeled point cloud object instances that can be extracted for injection into the “target” frame; once object instances have been injected into the target frame, the target frame may be referred to as the “extended” frame, and any dataset of point cloud data to which the extended frame has been added may be referred to as the “extended point cloud data,” or simply “extended data.” The terms “annotation” and “label” are used interchangeably to indicate the association of semantic data with point cloud data, such as scene type labels attached to a point cloud frame or its regions, or object class labels attached to object instances within a point cloud frame.

[0019] In this disclosure, “complete point cloud object scan” means a point cloud corresponding to an object scanned from more than one location such that multiple surfaces of the object are represented in the point cloud. “Dense” point cloud means a point cloud corresponding to one or more surfaces of an object, where the number of points per unit area of ​​the surface is relatively large. “Surface model” means a three-dimensional model of one or more surfaces of an object; surfaces may be represented by polygons, points, texture mappings, and / or any other means of representing a three-dimensional surface.

[0020] The exemplary devices, systems, methods, and media described herein can enhance unfavorable classes within an original point cloud dataset (i.e., a dataset of labeled point cloud frames). Surface models are derived from point cloud frames with point-level labels (e.g., semantically segmented point cloud frames). Object instances labeled with semantic labels within the original point cloud frame may be incomplete (half-side) and sparse. However, the methods and systems described herein can derive dense half-side point cloud object instances from the incomplete and sparse object instances within the original point cloud frame. These dense point cloud object instances can be used as surface models for injecting new point cloud object instances into a target frame.

[0021] The exemplary devices, systems, methods, and media described herein do not inject new point cloud object instances into a target point cloud frame using a CAD model of an object or a complete dense point cloud scan of an object, as in existing approaches that attempt to address the object instance diversity problem. Instead, they inject point cloud object instances derived from the actual point cloud frames generated by a LIDAR sensor. However, the described methods and systems can also be utilized to inject point cloud object instances using a CAD model of an object or a dense and complete point cloud object scan. The injected point cloud object instances can be obtained from point cloud frames received from different types of LIDAR sensors used to generate the target point cloud frame (e.g., the range and scan line configuration of the laser arrays of the LIDAR sensors used to generate the original point cloud frame and the target point cloud frame need not be the same). The injected point cloud object instances generated using the exemplary methods and systems described herein have reasonable scan lines on the surface (e.g., realistic direction, density, and intensity), and realistic shadows. Generally, the extended point cloud frames generated using the exemplary methods and systems described herein can be very similar to the actual point cloud frames generated by a LIDAR sensor.

[0022] The exemplary methods and systems described herein can be configured to further improve the realism and usefulness of the generated extended point cloud frames using context. The object class, number, position, and distribution of the injected point cloud object instances can be fully controlled using parameters. For example, if the exemplary methods and systems described herein are instructed to inject five people into a target point cloud frame, five point cloud object instances can be injected using a distribution. Each point cloud object instance has a 90% probability of being located on a sidewalk and a 10% probability of being located on a road.

[0023] The exemplary methods and systems described herein may perform the following series of operations to expand a point cloud data frame or point cloud data set. First, by processing a point cloud data set that includes an existing point cloud frame generated by a LIDAR sensor and annotated with point-level labels, a library of surface models is generated. The library generation process may involve object extraction and clustering to extract object instances from the original point cloud frame, followed by point cloud upsampling with respect to the azimuth-elevation plane to derive high-density point cloud object instances from the extracted point cloud object instances. Second, the point cloud object instances selected from the library are injected into a target point cloud frame to generate an expanded point cloud frame. The injection process may involve anchor point selection to determine the position within the target point cloud frame where the point cloud object instance may be injected, object injection to position the surface model within the target point cloud frame, and scan line and shadow generation to downsample the surface model to simulate the scan line of the LIDAR sensor at the anchor position within the target point cloud frame and generate a shadow that hides other point cloud objects within the target point cloud frame.

[0024] Some examples of methods and systems described herein may demonstrate advantages over existing approaches. While a library of surface models may be obtained directly from labeled point cloud frames, it may also be populated using CAD models of objects and dense point cloud object scanning, still allowing access to the advantages of the injection techniques described herein. Surface models of target point cloud frames may be obtained from point cloud frames generated by different types of LiDAR sensors. For example, point cloud object instances extracted from a point cloud frame generated by a 32-beam LiDAR sensor may be inserted into a target point cloud frame generated by a 64-beam LiDAR sensor. The scan line characteristics of the injected point cloud object instances (including density, direction, and intensity), and the shadows cast by the injected point cloud object instances are realistically simulated. The type, number, and injection location (i.e., anchor location) of the injected point cloud object instances can be controlled by parameters. The time required to label the point cloud frame (i.e., the time required to label multiple points) can be substantially reduced. This is because only the target objects within the original point cloud frame need to be labeled before they are used to populate the library of high-density point cloud object instances and injected into the target point cloud frame; it may not be necessary to label all points within the original point cloud frame.

[0025] In some embodiments, this disclosure describes a method. A point cloud object instance is obtained. The point cloud object instance is upsampled using interpolation to generate a surface model.

[0026] In some embodiments, this disclosure describes a system for augmenting point cloud data. The system comprises a processor device and memory. The memory stores point cloud object instances, a target point cloud frame, and machine-executable instructions. The machine-executable instructions, when executed by the processor device, cause the system to perform a number of operations. Point cloud object instances are upsampled using interpolation to generate a surface model. Anchor positions are determined within the target point cloud frame. The surface model is transformed based on the anchor positions to generate a transformed surface model. Multiple scan lines of the transformed surface model are generated, each scan line containing multiple points aligned with multiple scan lines of the target point cloud frame. Multiple scan lines of the transformed surface model are added to the target point cloud frame to generate an augmented point cloud frame.

[0027] In some examples of methods and systems, a point cloud object instance may include orientation information indicating the orientation of the point cloud object instance relative to the sensor location. The point cloud object instance further includes point intensity information and point position information for each of the multiple points within the point cloud object instance. The surface model includes the orientation information, point intensity information, and point position information of the point cloud object instance.

[0028] In some examples of the method and system, a point cloud object instance may contain multiple scan lines, each scan line containing a subset of multiple points. The step of upsampling the point cloud object instance may include adding multiple points along at least one scan line using linear interpolation.

[0029] In some examples of methods and systems, the step of upsampling a point cloud object instance may further include the step of adding multiple points between at least one pair of scan lines using linear interpolation.

[0030] In some examples of the system, the step of adding a point using linear interpolation may include assigning point location information to the added point based on linear interpolation of the point location information of two existing points, and assigning point intensity information to the added point based on linear interpolation of the point intensity information of two existing points.

[0031] In some embodiments, this disclosure describes a method. A target point cloud frame is acquired. Anchor positions within the target point cloud frame are determined. A surface model of an object is acquired. The surface model is transformed based on the anchor positions to generate a transformed surface model. Multiple scan lines of the transformed surface model are generated, each scan line containing multiple points aligned with multiple scan lines of the target point cloud frame. Multiple scan lines of the transformed surface model are added to the target point cloud frame to generate an expanded point cloud frame.

[0032] In some examples of methods and systems, the surface model may contain dense point cloud object instances.

[0033] In some examples of methods and systems, the step of obtaining a surface model may include obtaining point cloud object instances and generating a surface model by upsampling the point cloud object instances using interpolation.

[0034] In some examples of methods and systems, the surface model may include a computer-aided design (CAD) model.

[0035] In some examples of methods and systems, the surface model may include a complete, dense point cloud object scan.

[0036] In some examples, the method may further comprise the steps of determining the shadow of the transformed surface model, identifying one or more hidden points of a target point cloud frame located within the shadow, and removing a plurality of hidden points from the expanded point cloud frame.

[0037] In some examples of methods and systems, the step of generating multiple scan lines of a transformed surface model may include the steps of generating a range image including a two-dimensional pixel array, where each pixel is a point in a target point cloud frame, projecting the transformed surface model onto the range image, and for each pixel in the range image, identifying the nearest neighbor point of the projection of the transformed surface model to the center of the pixel in response to the determination that the pixel contains at least one point of the projection of the transformed surface model, and adding the nearest neighbor point to the scan line.

[0038] In some examples of the method and system, the surface model may include object class information indicating the object class of the surface model. The target point cloud frame includes scene type information indicating the scene type of the region of the target point cloud frame. The step of determining the anchor position includes positioning the anchor position within the region in response to a determination based on the region's scene type and the surface model's object class that the surface model should be located within the region.

[0039] In some examples of methods and systems, the step of transforming the surface model based on anchor positions may include the steps of rotating the surface model between a reference direction and an anchor point direction about an axis defined by the sensor positions of the target point cloud frame, while maintaining the orientation of the surface model relative to the sensor positions, and transforming the surface model between a reference distance and an anchor point distance.

[0040] In some examples, the method may further include a step of training a machine learning model using an augmented point cloud frame.

[0041] In some embodiments, the disclosure describes a non-transient processor-readable medium in which a surface model generated by one or more of the methods described above is stored.

[0042] In some embodiments, the disclosure describes a non-temporary processor-readable medium in which an expanded point cloud frame generated by one or more of the methods described above is stored.

[0043] In some embodiments, the disclosure describes a non-transient processor-readable medium having stored machine-executable instructions that, when executed by the device's processor device, cause the device to perform one or more steps of the methods described above. [Brief explanation of the drawing]

[0044] Here, we refer to the attached drawings illustrating exemplary embodiments of the present application as an example.

[0045] [Figure 1A] This is an exemplary, simplified point cloud frame, viewed from the upper front right, providing an operational context for the embodiments described herein.

[0046] [Figure 1B] This is an upper front right perspective view of an exemplary point cloud object instance labeled with the "Bicycle Rider" object class, suitable for use in the embodiments described herein.

[0047] [Figure 1C] This is an upper front right perspective view of an exemplary surface model based on the point cloud object instance of Figure 1B, generated by an embodiment described herein.

[0048] [Figure 1D] Figure 1B is a top view of a point cloud object instance that is rotated, transformed, and scaled before being injected into the target point cloud frame, as described in the examples of this specification.

[0049] [Figure 2]This block diagram shows some components of an exemplary system for generating surface models and extended point cloud frames, as described in the examples herein.

[0050] [Figure 3] Figure 2 is a block diagram showing the operation of the library generation module, data augmentation module, and training module.

[0051] [Figure 4] Figure 3 is a flowchart illustrating the steps of an exemplary method for generating a surface model, which can be performed using the library generation module.

[0052] [Figure 5] This flowchart shows the steps of an exemplary method for generating an augmented point cloud frame, which can be performed using the data augmentation module shown in Figure 3.

[0053] [Figure 6] This flowchart shows the steps of an exemplary method for training a machine learning model using augmented point cloud data generated by the methods shown in Figures 4 and 5.

[0054] The same reference numbers may be used in different diagrams to represent similar components. [Modes for carrying out the invention]

[0055] This disclosure describes exemplary devices, systems, methods, and media for adaptive scene augmentation to train machine learning models to perform point cloud segmentation and / or object detection.

[0056] Figure 1A shows an exemplary simplified point cloud frame 100, where multiple points are mapped to a three-dimensional coordinate system 102 of X, Y, and Z. The Z dimension typically extends upward, as defined by the rotation axis of the LIDAR sensor or other panoramic sensor generating the point cloud frame 100. The point cloud frame 100 contains numerous points, each of which may be represented by a set of coordinates (x,y,z) within the point cloud frame 100 along a vector of other values, such as an intensity value indicating the reflectivity of the object corresponding to this point. Each point represents the reflection of light emitted by the laser at a point in space relative to the LIDAR sensor corresponding to the point coordinates. While the exemplary point cloud frame 100 is shown as a box-shaped or rectangular prism, it will be understood that a typical point cloud frame captured by a panoramic LIDAR sensor is typically a 360-degree panoramic view of the environment surrounding the LIDAR sensor, extending across the entire detection range of the LIDAR sensor. Therefore, the exemplary point cloud frame 100 is more typical of a small portion of point cloud frames generated by an actual LIDAR and is used for illustrative purposes.

[0057] Multiple points in the point cloud frame 100 are clustered in the space where light emitted by the laser of the LIDAR sensor is reflected by objects in the environment, resulting in clusters of points corresponding to the surfaces of objects visible to the LIDAR sensor. The first point cloud 112 corresponds to a reflection from a car. In the exemplary point cloud frame 100, the first point cloud 112 is enclosed in a bounding box 122 and associated with an object class label (in this example, the label "car" 132). The second point cloud 114 is enclosed in a bounding box 122 and associated with the object class label "cyclist" 134, and the third point cloud 116 is enclosed in a bounding box 122 and associated with the object class label "pedestrian" 136. Thus, each of the point clusters 112, 114, and 116 corresponds to an object instance, i.e., an instance of the object classes "car," "cyclist," and "pedestrian," respectively. The entire point cloud frame 100 is associated with the scene type label 140 "Intersection," indicating that the point cloud frame 100 as a whole corresponds to an environment near an intersection (and therefore cars, pedestrians, and cyclists are in close proximity to each other).

[0058] In some examples, a single point cloud frame may contain multiple scenes, each of which may be associated with a different scene type label 140. Thus, a single point cloud frame may be segmented into multiple regions, each associated with its own scene type label 140. Exemplary embodiments are generally described herein with reference to a single point cloud frame associated with only a single scene type; however, it will be understood that in some embodiments, each region within the point cloud frame may be considered separately for point cloud object instance injection using the data augmentation methods and systems described herein.

[0059] Each bounding box 122 is sized and positioned, each of the object labels 132, 134, and 136 is associated with each point cluster, and the scene label is associated with the point cloud frame 100 using data labeling techniques known in the field of machine learning for generating labeled point cloud frames. As described above, these labeling techniques are generally very time-consuming and resource-intensive; the data augmentation techniques described herein can be used in some cases to reduce the time and resources required to manually identify and label point cloud object instances within a point cloud frame by increasing the number of labeled point cloud object instances within the point cloud frame 100.

[0060] The labels and bounding boxes of the exemplary point cloud frame 100 shown in Figure 1A correspond to labels applied in the context of object detection, and therefore the exemplary point cloud frame may be included in a point cloud dataset used to train a machine learning model for object detection on the point cloud frame. However, the methods and systems described herein are equally applicable not only to models for object detection on point cloud frames, but also to models for segmentation on point cloud frames, including semantic segmentation, instance segmentation, or panoramic segmentation of point cloud frames.

[0061] Figures 1B to 1D are described below with reference to the exemplary methods and systems described herein.

[0062] Figure 2 is a block diagram of a computing system 200 (hereinafter referred to as System 200) for extending a point cloud frame (or extending a point cloud dataset containing a point cloud frame). Exemplary embodiments of System 200 are shown and described below, but other embodiments may be used to implement the examples disclosed herein, which may include components different from those shown. Figure 2 shows a single instance of each component of System 200, but multiple instances of each component shown may exist.

[0063] System 200 includes one or more processors 202, such as a central processing unit, microprocessor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), dedicated logic circuit, tensor processing unit, neural processing unit, dedicated artificial intelligence processing unit, or a combination thereof. One or more processors 202 may be collectively referred to as a “processor device” or “processor 202”.

[0064] System 200 includes one or more memories 208 (collectively referred to as “Memory 208”) which may contain volatile or non-volatile memory (e.g., flash memory, random access memory (RAM), and / or read-only memory (ROM)). Non-temporary memory 208 may store machine-executable instructions for execution by processor 202, such as the execution of the examples described herein. A set of machine-executable instructions 220 that define a library generation module 330, a data expansion module 340, and a training module 234, each of which may be executed by processor 202 to perform a step of the method described herein. The operation of system 200 when executing the set of machine-executable instructions 220 that define the library generation module 330, the data expansion module 340, and the training module 234 will be described below with reference to Figure 3. The machine-executable instructions 220 that define the scene extension module 300 are executable by the processor 202 to perform the functions of their respective submodules 312, 314, 316, 318, 320, and 322. Memory 208 may contain other machine-executable instructions, for example, to implement the operating system and other applications or functions.

[0065] Memory 208 stores a dataset containing a point cloud dataset 210. As described above with reference to Figure 1, the point cloud dataset 210 includes a plurality of point cloud frames 212 and a plurality of labeled point cloud object instances 214. In some embodiments, some or all of the labeled point cloud object instances 214 are contained within and / or derived from the point cloud frames 212. As described above with reference to Figure 1, for example, each point cloud frame 212 may contain zero or more labeled point cloud object instances 214. In some embodiments, some or all of the labeled point cloud object instances 214 are stored separately from the point cloud frames 212, and each of the labeled point cloud object instances 214 may or may not originate from within one of the point cloud frames 212. The library generation module 330, described below with reference to Figures 3 and 4, can perform operations to extract one or more labeled point cloud object instances 214 from one or more point cloud frames 212 in some embodiments.

[0066] Memory 208 may also store other data, information, rules, policies, and machine-executable instructions as described herein, including a machine learning model 224, a surface model library 222 containing one or more surface models, a target point cloud frame 226, a target surface model 228 (selected from the surface model library 222), a converted surface model 232, and an expanded point cloud frame 230.

[0067] In some examples, system 200 may also include one or more electronic storage units (not shown), such as solid-state drives, hard disk drives, magnetic disk drives, and / or optical disk drives. In some examples, one or more datasets and / or modules may be provided by external memory (e.g., an external drive in a wired or wireless connection to system 200) or by temporary or non-temporary computer-readable media. Examples of non-temporary computer-readable media include RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, CD-ROM, or other portable memory storage. The storage units and / or external memory may be used in conjunction with memory 208 to implement the data storage, retrieval, and caching functions of system 200.

[0068] The components of system 200 may communicate with each other, for example, via a bus. In some embodiments, system 200 is a distributed system, such as a cloud computing platform, and may include multiple computing devices communicating with each other over a network, and optionally one or more additional components. The various operations described herein may, in some embodiments, be performed by different devices of the distributed system.

[0069] Figure 3 shows the operation of an exemplary library generation module 330, data augmentation module 340, and training module 234 executed by the processor 202 of system 200. In the illustrated embodiment, the library generation module 330 includes several functional submodules or submodules (instance extraction submodule 312 and upsampling submodule 314), and the data augmentation module 340 includes several functional submodules (frame selection submodule 316, transformation submodule 318, instance injection submodule 320, and surface model selection submodule 322). In other examples, one or more of the submodules 312, 314, 316, 318, 320, and 322 may have one or more functions or operations that are combined, split into multiple submodules, and / or, in particular, redistributed among the submodules. In some examples, the library generation module 330, the data augmentation module 340, and / or the training module 234 may include additional operations or submodules, or one or more of the illustrated submodules 312, 314, 316, 318, 320, and 322 may be omitted.

[0070] Referring to the exemplary method 400 shown in Figure 4, the operation of various submodules of the library generation module 330 shown in Figure 3 is described here.

[0071] Figure 4 is a flowchart showing the steps of an exemplary method 400 for generating a surface model. As described, the steps of method 400 are performed by various submodules of the library generation module 330 shown in Figure 3. However, it will be understood that method 400 can be performed by any suitable information processing technique.

[0072] Method 400 begins in step 402. In step 402, the instance extraction submodule 312 generates extracted instances 306 by extracting point cloud object instances from the point cloud dataset 210.

[0073] Figure 1B shows a detail view of an exemplary labeled point cloud object instance 148 within a point cloud frame 212 generated by a LIDAR sensor (or other 3D sensor as described above). The illustrated point cloud object instance 148 (e.g., one of the labeled point cloud object instances 214 selected from point cloud dataset 210) consists of the second point cloud 114 in Figure 1A (i.e., the point cloud object instance of "cyclist"), with multiple points 142 arranged along scan lines 144. Thus, the labeled point cloud object instance 148 includes multiple scan lines 144, each scan line 144 containing a subset of the multiple points 142 of the labeled point cloud object instance 148. The scan lines 144 correspond to multiple points detected by the LIDAR sensor when light emitted by the laser of the LIDAR sensor, moving along the azimuth direction midway between multiple measurements, is reflected by an object (in this case, the cyclist). In the illustrated example, the azimuth direction defining the direction of scan line 144 is approximately horizontal (i.e., on the XY plane defined by the coordinate system 102 of the point cloud frame). As described above with reference to Figure 1A, the labeled point cloud object instance 148 includes the "cyclist" object class label 134 and a bounding box 122 enclosing several points.

[0074] In some embodiments, semantic information, such as an object class label 134 and a bounding box 122, may be generated by the instance extraction submodule 312 as part of the instance extraction step 402 using known techniques for point cloud object detection and / or point cloud frame segmentation. In other embodiments, the point cloud frame 212 of the point cloud dataset 210 already contains labeled point cloud object instances 214 that are labeled and annotated with semantic information.

[0075] The instance extraction submodule 312 retrieves a point cloud frame (for example, from a point cloud frame 212) and identifies multiple points within the point cloud frame labeled with a given object class label 134. If the frame is annotated using semantic segmentation so that multiple instances of an object are uniformly annotated only with the object class label and are not segmented into individual object instances, the instance extraction submodule 312 can cluster the multiple points annotated with the object class label 134 to generate individual object instances of the object class indicated by the label 134 (for example, using panoramic or instance segmentation, or using object recognition).

[0076] The labeled point cloud object instance 148 and the extracted instance 306 generated by the object extraction process may contain orientation information indicating the orientation of the labeled point cloud object instance 148 relative to the sensor position. For example, the projection direction of the rays emitted by the laser of the LIDAR sensor used to generate multiple points 142 in the point cloud frame 212 may be recorded as part of the extracted instance 306 and defined, for example, as a direction vector using coordinate system 102. Each point 142 may be recorded in a format that includes a set of (x,y,z) coordinates in coordinate system 102. Thus, the intensity value of point 142 may be understood as a function of the reflectance of the object surface at the point of reflection of light from the object surface, as well as the relationship between the direction vector defining the rays emitted by the LIDAR sensor used to generate this point and the spatial coordinates of point 142, i.e., the orientation information of the extracted instance 306. Thus, the orientation information is used to represent the relationship between the direction vector of the rays and the surface normal of the object reflecting the light at that point in space. Orientation information may be used during the injection process (described below with reference to Figure 5) to maintain the orientation of the injected point cloud object instance relative to the sensor position in the target point cloud frame (i.e., the point cloud frame into which the point cloud object instance is injected) so that occlusion and intensity values ​​are accurately represented.

[0077] As described above with reference to Figure 1A, the labeled point cloud object instance 148 and the extracted instance 306 generated by the object extraction process may include, for each point 144, point intensity information (e.g., intensity value) and point position information (e.g., spatial (x,y,z) coordinates), as well as potentially other types of information.

[0078] In 404, the upsampling submodule 314 upsamples the extracted point cloud object instance 306 to generate surface models, such as the bicycle rider surface model 152 shown in Figure 1C.

[0079] Figure 1C shows an exemplary surface model 152 of a cyclist generated by an upsampling submodule 314 based on an extracted point cloud object instance 306 of the cyclist object instance 148 shown in Figure 1B. The upsampling submodule 314 upsamples the point cloud cluster of the extracted point cloud object instance 306 (i.e., a second point cloud cluster 114 representing the cyclist) using linear interpolation to increase the number of points in the cluster along and between each scan line 144. Point cloud object instances captured by a rotational scanning LIDAR sensor typically have very different point densities in the vertical direction (e.g., elevation direction approximately parallel to the Z-axis) and the horizontal direction (e.g., azimuth direction approximately parallel to the XY plane 157). Conventional surface generation methods that use polygonal meshes to represent surfaces, such as greedy surface triangulation and De Launay triangulation algorithms, produce surfaces consisting of polygonal meshes with holes, which can result in multiple scan lines lacking holes and multiple points in areas corresponding to multiple points appearing in the shadow area of ​​the surface during scan line and shadow generation (described below with reference to Figure 5). In contrast, in the examples of methods and systems described herein, point cloud object instances can be directly upsampled by taking advantage of the properties of a rotational scanning LiDAR sensor. First, linear interpolation is performed on multiple points 142 of each scan line, increasing the point density of each scan line 144 in the horizontal direction by adding new multiple points 155 between existing multiple points 142 of the scan line 144. Second, a thin sliding window 156 is used to separate sets of multiple points 142 along an azimuth angle 157 (i.e., the window 156 separates multiple points 142 located on multiple scan lines 144 that are roughly aligned with each other in the vertical direction). By adding multiple new points 154 in the middle of scan line 144 using linear interpolation, the density of multiple points 142 in the vertical direction is increased.Therefore, the point cloud object instance 148 is upsampled by adding multiple points 155 along scan line 144 and by adding multiple points 154 between multiple pairs of scan lines 144 (linear interpolation is used in both cases).

[0080] The added points 155 and 154 are assigned both point position information and point intensity information using linear interpolation. This upsampling can be performed on an azimuth-elevation plane, i.e., a plane defined by a vertically separated laser sweep along the azimuth direction 157 (for example, in a vertically separated arc centered on the sensor position). The density of the surface model generated by the upsampling submodule 314 can be controlled by defining the interpolation interval, for example, a parameter defined by the user of the library generation module 330. When the surface model is sufficiently dense, as described below with reference to Figure 5, and when multiple points should be hidden by the surface model, shadow generation should result in no multiple points remaining in the point cloud frame.

[0081] The upsampling submodule 314 includes other information in the surface model, such as orientation information, point intensity information, and point position information of the point cloud object instances 148 used in generating the surface model. A reference point 158, which indicates a single point in the space in which the surface model can be manipulated, may also be included in the surface model. In some embodiments, the reference point 158 ​​is located on or near the bottom surface of the bounding box 122 at the center position in the horizontal dimension of the bounding box 122, [x mean , y mean , z min This can be calculated using the x and y values ​​at the horizontal center of the XY rectangle of the bounding box, and the lowest z value of the bounding box. The distance d from the sensor position of the original frame to the reference point 158 ​​projected onto the XY plane (for example,

number

[0082] In 406, the upsampling submodule 314 adds surface models to the surface model library 222. The surface models contained in the surface model library 222 may be stored associated with their respective object class labels 134 (e.g., by keying or indexing) so that all surface models of a given object class can be easily retrieved. The surface model library 222 may then be stored or distributed as needed, for example, in the memory 208 of the system 200, in a central location accessible by the system 200, and / or distributed on non-temporary storage media. The stored surface model library 222 may be accessible by the system 200 for use by the training module 234.

[0083] Referring to the exemplary method 500 shown in Figure 5, the operation of various submodules of the data expansion module 340 shown in Figure 3 is described here.

[0084] Figure 5 is a flowchart illustrating the steps of an exemplary method 500 for injecting a surface model into a target point cloud frame. As described, the steps of method 500 involve various submodules of the data augmentation module 340 shown in Figure 3. To It will be implemented more effectively. However, it will be understood that method 500 can be implemented using any suitable information processing technique.

[0085] This method begins in step 502. In step 502, the surface model library 222 is generated, for example, by using the surface model generation method 400 shown in Figure 4, which is performed by the library generation module 330. In some embodiments, step 502 may be omitted, and one or more pre-generated surface models may be obtained before the surface model injection method 500 is executed.

[0086] In step 504, the data augmentation module 340 acquires the target point cloud frame 226. The target point cloud frame 226 can be selected from the point cloud dataset 210 by the frame selection submodule 316. In some examples, all point cloud frames 212 of the point cloud dataset 210 may be provided to the data augmentation module 340 for augmentation, while in other examples, only a subset of point cloud frames 212 may be provided. One iteration of method 500 is used to augment a single selected target point cloud frame 226.

[0087] In 506, a surface model is selected and prepared for injection into the target point cloud frame 226. The instance injection submodule 320 may receive the target point cloud frame 226 and, in some embodiments, control parameters used to control the selection of the surface model and the injection of the surface model into the target point cloud frame 226. An exemplary format for control parameters is: {person,2,[road,sidewalk,parking lot],[5%,90%,5%]} This indicates that two instances of the “person” object class are injected into the target point cloud frame 226. Each “person” object instance may be injected into multiple regions within the target point cloud frame 226 labeled with scene type labels 140 of the scene types “road”, “sidewalk”, or “parking lot”, with probabilities of 5%, 90%, and 5%, respectively. In such an example, steps 506 and 516 of method 500 would be repeated twice (to select and inject a surface model for each of these two point cloud object instances).

[0088] Stage 506 includes substages 508, 510, and 512. In substage 508, the instance injection submodule 320 determines anchor points within the target point cloud frame 226 based, for example, on a scene type probability distribution indicated by control parameters. As described below with reference to substage 512, the anchor points are used to position the injected point cloud object instances within the target point cloud frame 226.

[0089] In some embodiments, anchor points may be generated in three stages. First, all possible anchor points are identified using the scene type label 140 and object class label of the target point cloud frame 226, thereby identifying suitable regions and locations within the area where point cloud object instances can realistically be injected into the target point cloud frame 226 (for example, based on collision constraints with other objects in the target point cloud frame 226). Second, the probability p of each possible anchor point is calculated based on control parameters and any other constraints or factors. Third, an anchor point is selected based on the calculated probabilities; for example, the potential anchor point with the highest calculated probability may be selected as the anchor point.

[0090] The probability p of each anchor point candidate is p = p pos ·p class It can be calculated as p. pos p is a probability coefficient used to uniformly select anchor points on the ground plane. In the case of a rotational scanning LiDAR sensor, each point corresponds to a different area of ​​an object that reflects the light emitted by the laser at that point, and multiple points close to the sensor position cover a smaller area than multiple points farther away from the sensor position. Anchor points are typically selected from multiple points in the target point cloud frame 226 that are reflected by the ground plane. The selection probability of each point may be proportional to the area it covers; otherwise, most of the anchor points will be generated near the sensor position. Therefore, p pos teeth,

Number

[0091] p class The value of p can be determined by a control parameter, i.e., the probability of an anchor point located within a region labeled with a given scene type label 140. Thus, the target point cloud frame 226 includes scene type information (e.g., scene type label 140) indicating the scene type of one or more regions of the target point cloud frame 226, and this scene type information is used in the calculation of the probability p for selecting an anchor point from anchor point candidates. The value of p can be used to determine the value of p. In some embodiments, based on the scene type of this region and the object class of the surface model, it is essentially determined by the calculation of the probability p that the surface model should be located within a given region. Once an anchor point is selected from a plurality of anchor point candidates within this region, as described below in sub - step 512, this anchor point is selected, and the corresponding position (referred to as the anchor position) on the ground surface of the target point cloud frame 226 within this region is used as the position for positioning and injecting the surface model. class can be used to determine.

[0092] In sub-stage 510, the surface model selection submodule 322 obtains the target surface model 228 by, for example, selecting a surface model associated with an object class identified in the control parameters described above from the surface model library 222. In some examples, the surface model library 222 includes surface models stored as dense point cloud object instances, such as those generated by method 400 described above. In some examples, the surface model library 222 includes surface models stored as computer-aided design (CAD) models. In some examples, the surface model library 222 includes surface models stored as a dense point cloud representing a complete dense point cloud object scan, i.e., an object scanned from multiple viewpoints. The examples described herein refer to the use of surface models consisting of dense point cloud object instances, such as those generated by method 400. However, it will be understood that the methods and systems described herein are also applicable to other surface model types, such as CAD models and complete dense point cloud object scans (even if the use of those surface model types does not demonstrate all of the advantages that can be demonstrated by the use of dense point cloud object instances generated by Method 400).

[0093] Each surface model stored in the surface model library 222 may include object class information indicating the object class of the surface model. The surface model selection submodule 322 may obtain a list of all surface models of a given object class in the library 222 that satisfy the control parameters and other constraints defined by the anchor point selection described above. For example, the surface model selection submodule 322 may have distance constraint |r R |≦|r A | is imposed, thereby indicating the distance from the sensor position to the anchor point in the target point cloud frame 226, the anchor point range |r A |A distance d (reference range |r) that is smaller than or equal to | RIt may be required that the selected target surface model 228 has associated distance information (also referred to as |). Once a list of all surface models satisfying constraints (e.g., object class and spatial constraints) is obtained or generated in library 222, a surface model can be selected from the list using any appropriate selection criteria, such as random selection.

[0094] In sub-stage 512, the selected target surface model 228 is transformed by the transformation submodule 318 based on the anchor positions, and the transformed surface model 232 is generated. An example of surface model transformation is shown in Figure 1D.

[0095] Figure 1D shows a top view of the transformation of the target surface model 228 to generate the transformed surface model 232. The target surface model 228 is represented as a bicycle surface model 152 with a bounding box 122, a "bicycle" object class label 134, a reference point 158, and orientation information represented as an orientation angle 168 between the edge of the bounding box 122 and a reference direction indicated by a reference vector 172 extending from the sensor position 166 to the reference point 158. The reference vector 172 is defined as the distance d (i.e., the reference range |r R It has a length equal to |).

[0096] The anchor point determined in the above sub-stage 508 is located at anchor position 160 in the target point cloud frame 226, which defines an anchor point vector 170 pointing from sensor position 166 in the direction of the anchor point. The length of the anchor point vector 170 is the anchor point range |r A | is.

[0097] The conversion submodule 318 calculates the rotation angle θ between the reference direction (i.e., the reference vector 172) and the anchor point direction (i.e., the anchor point vector 170). Next, the target surface model 228 rotates about the axis defined by the sensor position 166 of the target point cloud frame 226 by the amount of the rotation angle θ (i.e., between the surface model reference direction defined by the reference vector 172 and the anchor point direction defined by the anchor point vector 170), while maintaining the orientation of the surface model relative to the sensor position 166 (i.e., maintaining the same orientation angle 168).

[0098] Next, the range or distance of the surface model is adjusted using a transformation, i.e., linear translation. The transformation submodule 318 defines the reference range |r| R |) and the anchor point distance (i.e., the anchor point range defined by the length of the anchor point vector 170 |r) A Transform the surface model between |).

[0099] In some examples, the surface model may then be scaled vertically and / or horizontally by a small amount relative to the anchor positions 160, potentially increasing the effectiveness of the data augmentation process for training machine learning models by introducing greater diversity to object instance injection into the point cloud data.

[0100] The transformed surface model 232 is the final result of the rotation, transformation, and scaling operations described above performed on the target surface model 228. In some examples, collision tests may be performed on the transformed surface model 232 by the instance injection submodule 320; if the transformed surface model 232 competes with (e.g., collides with or intersects with) other objects in the target point cloud frame 226, method 400 may return to step 506 to determine new anchor points and select a new surface model for transformation, and this process may be repeated until a suitable transformed surface model 232 is generated and positioned within the target frame 226.

[0101] In step 516, the instance injection submodule 320 injects point cloud object instances into the target point cloud frame 226 based on the surface model. Step 516 includes substeps 518 and 520.

[0102] Prior to step 516, the instance injection submodule 320 has already obtained the target point cloud frame 226 from the frame selection submodule 316 and the transformed surface model 232 from the transformation submodule 318, as described above. The transformed surface model 232 is positioned within the coordinate system 102 of the target point cloud frame 226. However, the transformed surface model 232 has no scan lines 144 on its surface and does not cast shadows that obscure other points in the target point cloud frame 226.

[0103] In sub-stage 518, the instance injection submodule 320 generates scan lines 144 on the surface of the transformed surface model 232 to generate point cloud object instances to be injected into the target point cloud frame 226. By adding the scan lines 144 of the transformed surface model 232 to the target point cloud frame 226, an expanded point cloud frame 230 is generated, which contains injected point cloud object instances consisting of multiple points of the scan lines 144 mapped to the surface of the transformed surface model.

[0104] Each scan line 144 of the transformed surface model 232 is generated as a plurality of points 142 aligned with the scan lines of the target point cloud frame 226. In some embodiments, the scan lines of the target point cloud frame 226 can be simulated by projecting the transformed surface model 232 onto a range image corresponding to the resolution of the LIDAR sensor used to generate the target point cloud frame 226. Thus, for example, the range image can be considered to be a set of all points in the target point cloud frame 226, where the spatial (x,y,z) coordinates of each point are transformed into (azimuth, elevation, distance) coordinates, and each point is then used to define a pixel of a two-dimensional pixel array in the (azimuth, elevation) plane. This two-dimensional pixel array is the range image. The azimuth coordinates may indicate an angular rotation about the Z axis of the sensor position, and the elevation coordinates may indicate an angle of elevation or concavity with respect to the XY plane. By projecting multiple points of the transformed surface model 232 onto the range image of the target point cloud frame 226, the instance injection submodule 320 can identify those points of the transformed surface model 232 that are included in the area corresponding to multiple points of the scanning ray performed by the LIDAR sensor used to generate the target point cloud frame 226. For each pixel of the range image containing at least one point of the projection of the transformed surface model 232, only the transformed surface model 232 closest to the center of each pixel is retained, and the retained point is used to populate a scan line 144 on the surface of the transformed surface model 232. Multiple points of a given scan line 144 correspond to rows of pixels in the range image. The retained point is moved in the elevation direction to align with the elevation angle of the center of the pixels in the range image. This ensures that the scan line 144 rises precisely, with all points generated by the pixels in that row having the same elevation angle.

[0105] In some embodiments, the range image is derived from the actual (azimuth, elevation) coordinates of a plurality of transformed points in the target point cloud frame 226; however, in other embodiments, the range image can be generated in a computationally less burdensome manner by obtaining the resolution of the LIDAR sensor used to generate the target point cloud frame 226 (which may be stored as information associated with the target point cloud frame 226, or may be derived from two or more points in the target point cloud frame 226), and generating a range image of the corresponding resolution without mapping the pixels of the range image one:1 to a plurality of points in the target point cloud frame 226. In some embodiments, the resolution-based range image can be aligned with one or more points in the frame after it has been generated.

[0106] Within the expanded point cloud frame 230, the transformed surface model 232 is discarded, leaving only the scan lines 144 generated as described above. However, this can be used to generate a shadow in sub-stage 520 before discarding the transformed surface model 232. The instance injection subsystem 320 determines the shadow cast by the transformed surface model 232, identifies one or more hidden points of the target point cloud frame 226 located within the shadow, and removes the hidden points from the expanded point cloud frame 230. The range image is used to identify all existing points of the target point cloud frame 226 contained within the area of ​​each pixel. Each pixel containing at least one point of the scan line 144 generated in sub-stage 518 is considered to cast a shadow. All of the existing points contained within the pixel (i.e., those within the shadow cast by the pixel) are considered hidden points and are removed from the expanded point cloud frame 230.

[0107] Methods 400 and 500 in Figures 4 and 5 can be used together to achieve one or more advantages. First, in Method 400, the surface model obtained from the point cloud frame generated by the actual LIDAR (i.e., the point cloud frame generated by the LIDAR sensor) is typically half-side; the rotation of the surface model in Method 500 ensures that the side with multiple points always faces the sensor position 166. Second, in some embodiments, as described above, the anchor point range is constrained by the transformation submodule 318 to be greater than the reference range (i.e., |r R |≦|r A Therefore, the density of scanline points generated on the surface of the surface model does not increase to the extent that it increases any artifacts of the upsampling process (although the density of extracted object instances increases due to upsampling, this does not increase the information contained in the original point cloud object instances). Other advantages of the combination of methods 400 and 500 will become apparent to those skilled in the art.

[0108] The library generation method 400 and the data augmentation method 500 can further be combined with a machine learning process to train a machine learning model. Interoperability of the library generation module 330, the data augmentation module 340, and the training module 234 shown in Figure 3 is described here with reference to the exemplary method 600 shown in Figure 6.

[0109] Figure 6 is a flowchart illustrating the steps of an exemplary method 600 for augmenting a point cloud dataset used to train a machine learning model 224 for a prediction task. As described, the steps of method 600 are performed by various submodules of the library generation module 330, data augmentation module 340, and training module 234 shown in Figure 3. However, it will be understood that method 600 can be performed by any suitable information processing technique.

[0110] In 602, the library generation module 330 generates a library 222 of one or more surface models according to method 400.

[0111] In 604, the data augmentation module 340 generates one or more augmented point cloud frames 230 according to method 500.

[0112] In 606, the training module 234 trains the machine learning model 224 using the expanded point cloud frame 230.

[0113] Steps 604 and 606 may be repeated one or more times to perform one or more training iterations. In some embodiments, multiple expanded point cloud frames 230 are generated before they are used to train the machine learning model 224.

[0114] The machine learning model 224 may be an artificial neural network or another model trained using machine learning techniques such as supervised learning to perform a prediction task on point cloud frames. The prediction task may be any prediction task for recognizing objects in a frame by object class or for segmenting a frame by object class, and may include object recognition, semantic segmentation, instance segmentation, or panoramic segmentation. In some embodiments, an extended point cloud frame 230 is added to the point cloud dataset 210, and the training module 234 trains the machine learning model 224 using the point cloud dataset 210 as the training dataset. That is, the machine learning model 224 is trained using supervised learning and the point cloud frames 212 and the extended point cloud frames 230 included in the point cloud dataset 210 to perform a prediction task on the point cloud frame 212, such as object recognition or segmentation. The trained machine learning model 224 may be trained to perform object detection and predict object class labels, or to perform segmentation and predict instance labels and / or scene type labels to attach to zero or more subsets or clusters of points or regions within each point cloud frame 212, and the labels associated with each labeled point cloud object instance 214 or regions within a given point cloud frame 212 are used as ground truth labels for training. In other embodiments, the machine learning model 224 is trained using different training point cloud datasets.

[0115] While this disclosure describes methods and processes with specific steps in a particular order, one or more steps in the methods and processes may be omitted or modified as appropriate. If necessary, one or more steps may be performed in an order other than that described.

[0116] While this disclosure describes at least partially a method, those skilled in the art will understand that this disclosure also covers various components for performing at least some of the aspects and features of the described method by hardware components, software, or any combination of the two. Accordingly, the technical solutions of this disclosure may be embodied in the form of a software product. A suitable software product may be stored on a pre-recorded storage device or other similar non-volatile or non-temporary computer-readable medium, including, for example, a DVD, CD-ROM, USB flash disk, removable hard disk, or other storage medium. The software product includes tangibly stored instructions that enable a processing device (e.g., a personal computer, server, or network device) to perform an example of the method disclosed herein.

[0117] This disclosure may be embodied in other specific forms without departing from the subject matter of the claims. The exemplary embodiments described should be considered in all respects to be merely illustrative and not limiting. Features selected from one or more of the embodiments described above may be combined to create alternative embodiments not expressly described, and features suitable for such combinations are understood to be within the scope of this disclosure.

[0118] All values ​​and sub-ranges within the disclosed scope are also disclosed. Furthermore, while the systems, devices, and processes disclosed and illustrated herein may include a certain number of elements / components, these systems, devices, and assemblies may be modified to include more or fewer such elements / components. For example, any of the disclosed elements / components may be referred to singly, but embodiments disclosed herein may be modified to include multiple such elements / components. The subject matter described herein is intended to encompass and include all appropriate changes in the art. Other possible items [Item 1] The step of obtaining a point cloud object instance; and The step of generating a surface model by upsampling the point cloud object instance using interpolation. A method for providing this. [Item 2] The aforementioned point cloud object instance is Orientation information indicating the orientation of the point cloud object instance relative to the sensor position. Including; and For each of the multiple points in the aforementioned point cloud object instance, Point intensity information; and Point position information Including; and The surface model includes the orientation information, point intensity information, and point position information of the point cloud object instance. The method described in item 1. [Item 3] The point cloud object instance includes a plurality of scan lines, each scan line includes a subset of the plurality of points; and The step of upsampling the point cloud object instance includes adding a plurality of points along at least one scan line using linear interpolation. The method described in item 2. [Item 4] The method according to item 3, wherein the step of upsampling the point cloud object instance further comprises the step of adding a plurality of points between at least one pair of scan lines of the plurality of scan lines using linear interpolation. [Item 5] The step of adding points using linear interpolation is, A step of assigning point location information to the added point based on linear interpolation of the point location information of two existing points; and A step of assigning point intensity information to the added point based on linear interpolation of the point intensity information of the two existing points. including, The method described in item 4. [Item 6] The stage of acquiring the target point cloud frame; A step of determining the anchor position within the target point cloud frame; The stage of obtaining a surface model of an object; A step of transforming the surface model based on the anchor positions to generate a transformed surface model; A step of generating a plurality of scan lines of the converted surface model, wherein each scan line includes a plurality of points aligned with the plurality of scan lines of the target point cloud frame; and The step of adding the plurality of scan lines of the converted surface model to the target point cloud frame to generate an expanded point cloud frame. A method for providing this. [Item 7] The surface model is the method described in item 6, which includes dense point cloud object instances. [Item 8] The step of obtaining the aforementioned surface model is: The step of obtaining a point cloud object instance; and The step of generating the surface model by upsampling the point cloud object instance using interpolation. Having, The method described in item 7. [Item 9] The surface model is the method described in item 6, including a computer-aided design (CAD) model. [Item 10] The surface model is the method described in item 6, which includes a complete dense point cloud object scan. [Item 11] A step of determining the shadow of the transformed surface model; A step of identifying one or more hidden points of the target point cloud frame located within the shadow; and Steps to remove the plurality of hidden points from the expanded point cloud frame. The method described in item 6, further comprising: [Item 12] The step of generating the plurality of scan lines of the converted surface model is: A step of generating a range image including a two-dimensional pixel array, where each pixel corresponds to a point in the target point cloud frame; The steps include: projecting the converted surface model onto the range image; and For each pixel of the range image, in response to the determination that the pixel includes at least one point of the projection of the transformed surface model, A step of identifying the nearest neighbor of the projection of the transformed surface model onto the center of the pixel; and Step of adding the nearest neighbor point to the scan line. Having, The method described in item 7. [Item 13] The surface model includes object class information indicating the object class of the surface model; The target point cloud frame includes scene type information indicating the scene type of the region of the target point cloud frame; and The step of determining the anchor position includes positioning the anchor position within the region in response to a determination based on the scene type of the region and the object class of the surface model that the surface model should be located within the region. The method described in item 6. [Item 14] The step of transforming the surface model based on the anchor position is: A step of rotating the surface model between the surface model reference direction and the anchor point direction about an axis defined by the sensor position of the target point cloud frame, while maintaining the orientation of the surface model with respect to the sensor position; and Steps to convert the surface model between a reference distance and an anchor point distance. Having, The method described in item 6. [Item 15] The method of item 6, further comprising a step of training a machine learning model using the expanded point cloud frame. [Item 16] A system for expanding point cloud data, Processor devices; and A memory containing machine-executable instructions that, when executed by the aforementioned processor device, cause the system to perform the method described in any one of items 1 to 14. A system that includes these features. [Item 17] A computer-readable medium comprising machine-executable instructions, when executed by a processor device of a computing system, causing the computing system to perform the method described in any one of items 1 to 14. [Item 18] A computer program comprising machine-executable instructions, when executed by the processor device of a computing system, causing the computing system to perform the method described in any one of items 1 to 14.

Claims

1. The step of obtaining a point cloud object instance of an object; and The step of generating a surface model by upsampling the point cloud object instance using interpolation. Equipped with, The step of upsampling the point cloud object instance is a method comprising the step of performing linear interpolation on a plurality of existing points in the point cloud object instance obtained in the step of acquiring the point cloud object instance, thereby adding a plurality of new points between the plurality of existing points.

2. The aforementioned point cloud object instance is Orientation information indicating the orientation of the point cloud object instance relative to the sensor position. including; and For each of the existing points in the point cloud object instance, Point intensity information; and Point position information including; and The surface model includes the orientation information, point intensity information, and point position information of the point cloud object instance. The method according to claim 1.

3. The point cloud object instance includes a plurality of scan lines, each scan line including a subset of the existing plurality of points; and The step of upsampling the point cloud object instance includes the step of performing linear interpolation on the existing points to add the new points along at least one scan line. The method according to claim 2.

4. The method according to claim 3, wherein the step of upsampling the point cloud object instance further comprises the step of performing linear interpolation on the existing plurality of points to add the new plurality of points between at least one pair of scan lines of the plurality of scan lines.

5. The step of adding the aforementioned new points is, A step of assigning point location information to the new plurality of points based on linear interpolation of the point location information of the existing plurality of points; and A step of assigning point intensity information to the new plurality of points based on linear interpolation of the point intensity information of the existing plurality of points. including, The method according to any one of claims 2 to 4.

6. The point cloud object instance includes a bounding box surrounding the surface model, The surface model includes a reference point that indicates a single point within the bounding box. The method according to any one of claims 1 to 5.

7. The stage of acquiring the target point cloud frame; The step of determining the anchor position within the target point cloud frame; The stage of obtaining a surface model of an object; A step of transforming the surface model based on the anchor position to generate a transformed surface model; A step of generating a plurality of scan lines of the converted surface model, wherein each scan line includes a plurality of points aligned with the plurality of scan lines of the target point cloud frame; and The step of adding the plurality of scan lines of the converted surface model to the target point cloud frame to generate an expanded point cloud frame. A method for providing this.

8. The method according to claim 7, wherein the surface model includes dense point cloud object instances.

9. The step of obtaining the aforementioned surface model is: The step of obtaining a point cloud object instance; and The step of generating the surface model by upsampling the point cloud object instance using interpolation. Having, The method according to claim 8.

10. The method according to any one of claims 7 to 9, wherein the surface model includes a computer-aided design (CAD) model.

11. The method according to any one of claims 7 to 9, wherein the surface model includes a complete dense point cloud object scan.

12. A step of determining the shadow of the transformed surface model; A step of identifying one or more hidden points of the target point cloud frame located within the shadow; and Steps to remove the plurality of hidden points from the expanded point cloud frame. The method according to any one of claims 7 to 11, further comprising:

13. The step of generating the plurality of scan lines of the converted surface model is: A step of generating a range image including a two-dimensional pixel array, where each pixel corresponds to a point in the target point cloud frame; The step of projecting the converted surface model onto the range image; and With respect to each pixel of the range image, in response to the determination that the pixel includes at least one point of the projection of the transformed surface model, A step of identifying the nearest neighbor point of the projection of the transformed surface model onto the center of the pixel; and Step of adding the nearest neighbor point to the scan line. Having, The method according to claim 8.

14. The surface model includes object class information indicating the object class of the surface model; The target point cloud frame includes scene type information indicating the scene type of the region of the target point cloud frame; and The step of determining the anchor position includes positioning the anchor position within the region in response to a determination based on the scene type of the region and the object class of the surface model that the surface model should be located within the region. The method according to any one of claims 7 to 13.

15. The step of transforming the surface model based on the anchor position is: A step of rotating the surface model between the surface model reference direction and the anchor point direction about an axis defined by the sensor position of the target point cloud frame, while maintaining the orientation of the surface model with respect to the sensor position; and Steps to convert the surface model between a reference distance and an anchor point distance. Having, The method according to any one of claims 7 to 14.

16. The method according to any one of claims 7 to 15, further comprising the step of training a machine learning model using the expanded point cloud frame.

17. A system for expanding point cloud data, Processor devices; and A memory that, when executed by the processor device, stores a machine-executable instruction causing the system to perform the method according to any one of claims 1 to 16. A system equipped with these features.

Images